Patent ID: 12200341

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention, and limitation is not made an invention that requires all combinations of features described in the embodiments. Two or more of the multiple features described in the embodiments may be combined as appropriate. Furthermore, the same reference numerals are given to the same or similar configurations, and redundant description thereof is omitted.

Embodiment

<Configuration>

FIG.1is a block diagram of a vehicle1according to an embodiment of the present invention. InFIG.1, the vehicle1is schematically illustrated in a plan view and a side view. The vehicle1is, for example, a sedan type of four-wheeled passenger vehicle. The vehicle1may be such a four-wheeled vehicle, a two-wheeled vehicle, or another type of vehicle.

The vehicle1includes a vehicle control device2(hereinafter, simply referred to as a control device2) that controls the vehicle1. The control device2includes a plurality of electronic control units (ECUs)20to29communicably connected by an in-vehicle network. Each of the ECUs includes a processor such as a central processing unit (CPU), a memory such as a semiconductor memory, and an interface with an external device. The memory stores a program executed by the processor, data used by the processor for processing, and the like. Each of the ECUs may include a plurality of processors, memories, and interfaces. For example, the ECU20includes one or more processors20aand one or more memories20b. Each of the processors20aexecutes an instruction including the program stored in each of the memories20b, so that processing is executed by the ECU20. Alternatively, the ECU20may include a dedicated integrated circuit such as an application specific integrated circuit (ASIC) for executing the processing by the ECU20. The same applies to the other ECUs.

Hereinafter, functions and the like assigned to each of the ECUs20to29will be described. Note that the number of ECUs and functions to be handled can be designed as appropriate and can be subdivided or integrated as compared with the present embodiment.

The ECU20executes control related to automated traveling of the vehicle1. In automated driving, at least any one of steering and acceleration/deceleration of the vehicle1is automatically controlled. The automated traveling by the ECU20may include automated traveling that does not require a traveling operation by a driver (which may also be referred to as automated driving) and automated traveling for assisting the traveling operation by the driver (which may also be referred to as driving assistance).

The ECU21controls an electric power steering device3. The electric power steering device3includes a mechanism that steers front wheels according to a driving operation (steering operation) of a driver with respect to a steering wheel31. Furthermore, the electric power steering device3includes a motor that produces driving force for assisting the steering operation and automatically steering the front wheels, and a sensor that detects a steering angle. In a case where a driving state of the vehicle1is automated driving, the ECU21automatically controls the electric power steering device3in response to an instruction from the ECU20and controls a traveling direction of the vehicle1.

The ECUs22and23control detection units for detecting a peripheral situation of the vehicle and performs information processing on detection results. The vehicle1includes one standard camera40and four fisheye cameras41to44as detection units for detecting the peripheral situation of the vehicle. The standard camera40and the fisheye cameras42and44are connected to the ECU22. The fisheye cameras41and43are connected to the ECU23. The ECUs22and23can extract an outline of a target object or a lane division line (white line or the like) on a road by analyzing images captured by the standard camera40and the fisheye cameras41to44.

The fisheye cameras41to44are cameras provided with a fisheye lens. Hereinafter, a configuration of the fisheye camera41will be described. The other fisheye cameras42to44may have a similar configuration. The angle of view of the fisheye camera41is wider than the angle of view of the standard camera40. Therefore, the fisheye camera41can image a wider area than the area of the standard camera40. The image captured by the fisheye camera41has a larger distortion than the image captured by the standard camera40. Therefore, before analyzing the image captured by the fisheye camera41, the ECU23may perform conversion processing (hereinafter, referred to as “distortion correction processing”) for reducing distortion on the image. On the other hand, before analyzing the image captured by the standard camera40, the ECU22may not perform the distortion correction processing on the image. In this manner, the standard camera40is an imaging device that captures an image not to be subjected to the distortion correction processing, and the fisheye camera41is an imaging device that captures an image to be subjected to the distortion correction processing. Instead of the standard camera40, any of other imaging devices may be used, which captures an image not to be subjected to the distortion correction processing, for example, a camera attached with a wide-angle lens or a telephoto lens may be used.

The standard camera40is attached at a center of a front portion of the vehicle1and captures an image of the peripheral situation in front of the vehicle1. The fisheye camera41is attached at the center of the front portion of the vehicle1and captures an image of the peripheral situation in front of the vehicle1. InFIG.1, the standard camera40and the fisheye camera41are illustrated as being aligned horizontally. However, the arrangement of the standard camera40and the fisheye camera41is not limited to this, and for example, the standard camera40and the fisheye camera41may be aligned vertically. Furthermore, at least one of the standard camera40or the fisheye camera41may be attached to a front portion of a roof (for example, on a vehicle interior side of a windshield) of the vehicle1. The fisheye camera42is attached at a center of a right-side portion of the vehicle1and captures an image of the peripheral situation on the right side of the vehicle1. The fisheye camera43is attached at a center of a rear portion of the vehicle1and captures an image of the peripheral situation behind the vehicle1. The fisheye camera44is attached at a center of a left-side portion of the vehicle1and captures an image of the peripheral situation on the left side of the vehicle1.

The types, number, and attachment positions of the cameras provided to the vehicle1are not limited to the example described above. Furthermore, the vehicle1may include Light Detection and Ranging (LiDAR) or a millimeter-wave radar as a detection unit for detecting the target object around the vehicle1and measuring a distance to the target object.

The ECU22controls the standard camera40, and the fisheye cameras42and44, and performs information processing on detection results. The ECU23controls the fisheye cameras41and43and performs information processing on detection results. Reliability of the detection results can be improved by dividing the detection units for detecting the peripheral situation of the vehicle into two systems.

The ECU24controls a gyro sensor5, a GPS sensor24b, and a communication device24c, and performs information processing on a detection result or a communication result. The gyro sensor5detects a rotational motion of the vehicle1. A course of the vehicle1can be determined based on the detection result of the gyro sensor5, a wheel speed, and the like. The GPS sensor24bdetects a current location of the vehicle1. The communication device24cperforms wireless communication with a server that provides map information and traffic information and acquires these pieces of information. The ECU24can access a map information database24aconstructed in the memory, and the ECU24searches for a route from the current position to a destination. The ECU24, the map information database24a, and the GPS sensor24bconstitute a so-called navigation device.

The ECU25includes a communication device25afor vehicle-to-vehicle communication. The communication device25aperforms wireless communication with other surrounding vehicles to exchange information between the vehicles.

The ECU26controls a power plant6. The power plant6is a mechanism that outputs a driving force for rotating driving wheels of the vehicle1and includes, for example, an engine and a transmission. For example, the ECU26controls an output of the engine according to a driving operation (accelerator operation or acceleration operation) of the driver detected by an operation detection sensor7aprovided on an accelerator pedal7A and switches a gear ratio of the transmission based on information such as a vehicle speed detected by a vehicle speed sensor7c. In a case where the driving state of the vehicle1is automated driving, the ECU26automatically controls the power plant6in response to the instruction from the ECU20, and controls the acceleration/deceleration of the vehicle1.

The ECU27controls a lighting device (headlight, taillight, and the like) including a direction indicator8(blinker). In the example ofFIG.1, the direction indicators8are provided at the front portion, door mirrors, and the rear portion of the vehicle1.

The ECU28controls an input and output device9. The input and output device9outputs information to the driver and receives an input of information from the driver. A voice output device91notifies the driver of information by voice. A display device92notifies the driver of information by displaying an image. The display device92is disposed, for example, in front of a driver's seat, and constitutes an instrument panel. Note that, although the voice and the image display have been exemplified here, information notification may also be made by using vibration or light. Furthermore, information notification may be made by using a combination of some of voice, display, vibration, and light. Furthermore, the combination or the notification form may be changed in accordance with a level (for example, a degree of urgency) of information notification of which should be made. An input device93is a switch group that is disposed at a position where the driver can operate the input device and is used to input an instruction to the vehicle1. The input device93may also include a voice input device.

The ECU29controls a brake device10and a parking brake (not illustrated). The brake device10is, for example, a disc brake device, is provided on each wheel of the vehicle1, and applies resistance to the rotation of the wheel to decelerate or stop the vehicle1. The ECU29controls, for example, working of the brake device10in response to the driver's driving operation (brake operation) that has been detected by an operation detection sensor7bprovided on a brake pedal7B. In a case where the driving state of the vehicle1is automated driving, the ECU29automatically controls the brake device10in response to the instruction from the ECU20, and controls the deceleration and stop of the vehicle1. The brake device10and the parking brake also can work to maintain a stopped state of the vehicle1. Furthermore, in a case where the transmission of the power plant6is provided with a parking lock mechanism, the parking lock mechanism also can work to maintain the stopped state of the vehicle1.

<Imaging Range>

Next, imaging ranges of the standard camera40and the fisheye cameras41to44will be described with reference toFIGS.2A to2C.FIG.2Aillustrates a horizontal imaging range of each camera,FIG.2Billustrates a vertical imaging range of the fisheye camera42attached to the right-side portion of the vehicle1, andFIG.2Cillustrates a vertical imaging range of the fisheye camera43attached to the rear portion of the vehicle1.

First, the imaging range of the vehicle1in plan view (for example, in the horizontal direction of the vehicle1) will be described with reference toFIG.2A. The standard camera40images scenery included in an imaging range200. An imaging center200C of the standard camera40faces a directly forward side of the vehicle1. The horizontal angle of view of the standard camera40may be less than 90°, for example, may be about 45° or about 30°.

The fisheye camera41images scenery included in an imaging range201. An imaging center201C of the fisheye camera41faces the directly forward side of the vehicle1. The fisheye camera42images scenery included in an imaging range202. An imaging center202C of the fisheye camera42faces a directly right side of the vehicle1. The fisheye camera43images scenery included in an imaging range203. An imaging center203C of the fisheye camera43faces a directly rear side of the vehicle1. The fisheye camera44images scenery included in an imaging range204. An imaging center204C of the fisheye camera44faces a directly left side of the vehicle1. The horizontal angles of view of the fisheye cameras41to44may be, for example greater than 90°, greater than 150°, or greater than 180°, and for example may be about 180°.FIG.2Aillustrates an example in which the horizontal angles of view of the fisheye cameras41to44is 180°.

The imaging range201can be divided into an area201L on a diagonally forward left side of the vehicle1, an area201F on a directly forward side of the vehicle1, and an area201R on a diagonally forward right side of the vehicle1. The imaging range202can be divided into an area202L on a diagonally forward right side of the vehicle1, an area202F on a directly right side of the vehicle1, and an area202R on a diagonally rearward right side of the vehicle1. The imaging range203can be divided into an area203L on a diagonally rearward right side of the vehicle1, an area203F on a directly rear side of the vehicle1, and an area203R on a diagonally rearward left side of the vehicle1. The imaging range204can be divided into an area204L on a diagonally rearward left side of the vehicle1, an area204F on a directly left side of the vehicle1, and an area204R on a diagonally forward left side of the vehicle1. The imaging range201may be equally divided into the three areas201L,201F, and201R (that is, the angles of views of the respective areas are made equal to one another). The other imaging ranges202to204may also be each divided equally into three areas.

The standard camera40and the fisheye cameras41to44have the imaging ranges200to204as described above, so that the directly forward direction and the four oblique directions of the vehicle1are included in the imaging ranges of the two individual cameras. Specifically, the directly forward side of the vehicle1is included in both the imaging range200of the standard camera40and the area201F of the imaging range201of the fisheye camera41. The diagonally forward right side of the vehicle1is included in both the area201R of the imaging range201of the fisheye camera41and the area202L of the imaging range202of the fisheye camera42. The same applies to the other three oblique directions of the vehicle1.

Next, a vertical imaging range of the vehicle1will be described with reference toFIGS.2B and2C. InFIG.2B, the vertical imaging range of the fisheye camera42will be described, and inFIG.2C, the vertical imaging range of the fisheye camera43will be described. The same may apply to the vertical imaging ranges of the other fisheye cameras41and44.

The vertical angles of view of the fisheye cameras41to44may be, for example, greater than 90°, greater than 150°, or greater than 180°, and for example, may be about 180°.FIGS.2B and2Cillustrate an example in which the vertical angle of view of the fisheye cameras41to44is 180°. In the illustrated example, the imaging center203C of the fisheye camera43faces a lower side than a direction parallel to the ground (toward the ground side). Instead of this, the imaging center203C of the fisheye camera43may face a direction parallel to the ground, or may face an upper side than the direction parallel to the ground (an opposite side of the ground). Furthermore, the imaging centers201C to204C of the fisheye cameras41to44may face different directions from one another in the vertical direction.

The distortion correction processing of the images captured by the fisheye cameras41to44will be described with reference toFIG.3. An image300is an image of scenery on the rightward side of the vehicle1captured by the fisheye camera42. As illustrated, the image300has a significant distortion particularly in a peripheral portion.

The ECU22connected to the fisheye camera42performs distortion correction processing on the image300(processing of converting the fisheye image into a planar image). Specifically, the ECU22sets one point in the image300as a correction center point301. The ECU22cuts out a rectangular area302centered on the correction center point301from the image300. The ECU22generates an image303in which the distortion is reduced by performing the distortion correction processing on the area302. In the distortion correction processing, the closer a position is to the correction center point301, the more the distortion is reduced, and at a position far from the correction center point301, the distortion is not reduced or the distortion is increased. Therefore, in some embodiments, the ECU22sets the correction center point301in an area desired to focus on in the surrounding environment of the vehicle1, and generates an image in which the distortion is reduced for such an area.

Note that, in the example ofFIG.3, the correction center point301and the rectangular area302are set on the right side of (a vertical line352passing through) the imaging center351of the fisheye image, but the correction center point301and the rectangular area302may be set on the left side, or both imaging center351of the fisheye image and the correction center point301may be set on the line352.

<Orientation Change of Vehicle>

The orientation of the vehicle1is changed depending on the acceleration/deceleration at the time of traveling.FIGS.4A to4Care views illustrating an example of the orientation of the vehicle1in the acceleration/deceleration in the traveling direction.FIG.4Aillustrates the orientation of the vehicle1during acceleration,FIG.4Billustrates the orientation of the vehicle1at the time of constant speed traveling or stopping, andFIG.4Cillustrates the orientation of the vehicle1at the time of deceleration.

First, as illustrated inFIG.4B, the fisheye camera41and the fisheye camera43disposed in the front-and-rear direction of the vehicle1face a predetermined imaging direction (imaging center) at the time of the constant speed traveling or the stopping.

At the time of the acceleration, as illustrated inFIG.4A, a front position of the vehicle1is lifted vertically upward by the acceleration, and a rear position of the vehicle1is lowered vertically downward by the acceleration. Accordingly, the imaging direction (imaging center) of the fisheye camera41disposed on the front portion of the vehicle1is changed upward as compared with the horizontal state of the vehicle1. Furthermore, the imaging direction (imaging center) of the fisheye camera43disposed on the rear portion of the vehicle1is changed downward as compared with the horizontal state of the vehicle1.

On the other hand, at the time of the deceleration, as illustrated inFIG.4C, the front position of the vehicle1is lowered vertically downward by the deceleration and the rear position of the vehicle1is lifted vertically upward by the deceleration. Accordingly, the imaging direction (imaging center) of the fisheye camera41disposed on the front portion of the vehicle1is changed downward as compared with the horizontal state of the vehicle1. Furthermore, the imaging direction (imaging center) of the fisheye camera43disposed on the rear portion of the vehicle1is changed upward as compared with the horizontal state of the vehicle1.

Furthermore, the orientation of the vehicle1is also changed by steering (lateral acceleration) at the time of traveling. Specifically, acceleration is generated in the right-and-left direction (lateral direction) according to a speed of the vehicle1and a steering amount of the steering wheel31.FIGS.5A to5Care views illustrating an example of the orientation of the vehicle1at the time of steering.FIG.5Aillustrates the orientation of the vehicle1at the time of constant speed traveling or stopping without a steering operation,FIG.5Billustrates the orientation of the vehicle1during the steering operation in a left direction, andFIG.5Cillustrates the orientation of the vehicle1during the steering operation in a right direction.

At the time of steering in the left direction such as curving to the left (when the steering wheel31is operated counterclockwise), the acceleration is generated in the right direction, and as illustrated inFIG.5B, a left side position of the vehicle1is lifted vertically upward by the steering, and a right side position of the vehicle1is lowered vertically downward by the steering. Accordingly, the imaging direction (imaging center) of the fisheye camera44disposed on the left side of the vehicle1is changed upward as compared with the horizontal state of the vehicle1. Furthermore, the imaging direction (imaging center) of the fisheye camera42disposed on the right side of the vehicle1is changed downward as compared with the horizontal state of the vehicle1.

On the other hand, at the time of steering in the right direction such as curving to the right (when the steering wheel31is operated clockwise), the acceleration is generated in the left direction, and as illustrated inFIG.5C, the right side position of the vehicle1is lifted vertically upward by the steering, and the left side position of the vehicle1is lowered vertically downward by the steering. Accordingly, the imaging direction (imaging center) of the fisheye camera42disposed on the right side of the vehicle1is changed upward as compared with the horizontal state of the vehicle1. Furthermore, the imaging direction (imaging center) of the fisheye camera44disposed on the left side of the vehicle1is changed downward as compared with the horizontal state of the vehicle1.

In this manner, since the orientation of the vehicle1is changed by the acceleration/deceleration and the steering of the vehicle1, an acquisition range of the peripheral information is changed. In the present embodiment, each conversion center position for converting the fisheye image captured by each fisheye camera into the planar image is corrected according to the orientation of the vehicle1.

For example, a change in the orientation of the vehicle1according to the acceleration and deceleration at the time of the acceleration/deceleration (how much the imaging direction (imaging center) is changed) is held in advance as table data or a function, and the current acceleration or the current deceleration is acquired, so that it is possible to estimate how the orientation of the vehicle1is changed.

Similarly, a change in the orientation of the vehicle1according to the lateral acceleration at the time of the steering (how much the imaging direction (imaging center) is changed) is held in advance as table data or a function, and the current acceleration in the lateral direction is acquired, so that it is possible to estimate how the orientation of the vehicle1is changed. The lateral acceleration can be calculated based on, for example, the speed of the vehicle1and the steering amount.

As described above, by correcting the conversion center position for converting the fisheye image into the planar image according to the orientation of the vehicle1, it is possible to correct influence of the orientation change of the vehicle1, and it is possible to acquire desired peripheral information with high accuracy.

<Processing>

Next, a procedure of processing performed by the control device2according to the present embodiment will be described with reference to a flowchart ofFIG.6.

In S601, the ECU22and the ECU23detect the orientation of the vehicle1. Various methods can be applied as the orientation detection method. For example, the acceleration and deceleration of the vehicle1are calculated based on the change in the speed of the vehicle1detected by the vehicle speed sensor7c, and the calculated acceleration or deceleration and the orientation of the vehicle1at that time are held in advance as the table data or the function. It is possible to detect the orientation by calculating the orientation of the corresponding vehicle1according to the current acceleration or deceleration of the vehicle1. Similarly, the lateral acceleration of the vehicle1is calculated based on the speed of the vehicle1and the steering amount of the steering wheel31, and the calculated lateral acceleration and the orientation of the vehicle1at that time are held in advance as the table data or the function. It is possible to detect the orientation by calculating the orientation of the corresponding vehicle1according to the current lateral acceleration of the vehicle1. Alternatively, the orientation of the vehicle1may be detected from the detection result of the gyro sensor5.

In S602, the ECU22and the ECU23respectively control the conversion center positions for converting the fisheye image of each fisheye camera (fisheye cameras41to44) into the planar image based on the orientation of the vehicle1. For example, by storing a relationship between the orientation of the vehicle1and the conversion center position in association with each other in advance, it is possible to derive the conversion center position according to the orientation. Various cases of the processing of S602will be described below.

[Control Example at Time of Braking (Deceleration)]

<Fisheye Camera41>

First, a control example at the time of acceleration will be described. For example, in S602, at the time of the orientation change in which the front portion of the vehicle1is moved downward and the rear portion of the vehicle1is moved upward (at the time of braking (deceleration)), the ECU23sets the conversion center position of the fisheye camera41, which is disposed on the front portion of the vehicle1and captures a forward image of the vehicle1, to a position above a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. Since the front portion of the vehicle1is lowered, the imaging center of the fisheye camera41faces downward as compared with the horizontal state, but the direction in which the peripheral information is desired to be acquired is upward. Therefore, the conversion center position is corrected upward by the extent that the front portion of the vehicle1is lowered. In the case of large deceleration, the front portion of the vehicle1is also significantly lowered, and thus the conversion center position is corrected to be higher according to this.

<Fisheye Camera43>

Moreover, the ECU23sets the conversion center position of the fisheye camera43, which is disposed on the rear portion of the vehicle1and captures a rearward image of the vehicle1, to a position below a predetermined position (conversion center position when the vehicle1is in the horizontal state). Since the rear portion of the vehicle1is lifted, the imaging center of the fisheye camera43faces upward as compared with the horizontal state, but the direction in which the peripheral information is desired to be acquired is downward. Therefore, the conversion center position is corrected upward by the extent that the rear portion of the vehicle1is lifted. In the case of large deceleration, the rear portion of the vehicle1is also significantly lifted, and thus the conversion center position is corrected to be lower according to this.

<Fisheye Cameras42and44>

Furthermore, at the time of the orientation change in which the front portion of the vehicle1is moved downward and the rear portion of the vehicle1is moved upward (at the time of braking (deceleration)), the ECU22corrects the conversion center position of the fisheye camera42, which is disposed on the right-side portion of the vehicle1and captures a rightward image of the vehicle1, by rotating the conversion center position clockwise around the imaging center of the fisheye camera42based on an orientation after the orientation change. On the other hand, at the time of the orientation change in which the front portion of the vehicle1is moved downward and the rear portion of the vehicle1is moved upward (at the time of braking (deceleration)), the ECU22corrects the conversion center position of the fisheye camera44, which is disposed on the left-side portion of the vehicle1and captures a leftward image of the vehicle1, by rotating the conversion center position counterclockwise around the imaging center of the fisheye camera44based on an orientation after the orientation change.

Here,FIG.7Ais an example of the fisheye image (imaging center703) of the fisheye camera42that captures the rightward image in the horizontal state in which the acceleration/deceleration or the lateral acceleration is not be generated. A rectangular area702centered on a correction center point701is set. At the time of the orientation change in which the front portion of the vehicle1is moved downward and the rear portion of the vehicle1is moved upward (at the time of braking (deceleration)), a correction center point751and a rectangular area752are rotated to be changed in position as illustrated inFIG.7B, and in the illustrated example, the peripheral information is acquired from an area above a position in which the peripheral information is originally desired to be acquired. In order to correct this, the conversion center position is corrected by rotating the conversion center position clockwise around the imaging center703as indicated by an arrow753. Since the conversion center position of the fisheye camera44disposed on the left-side portion of the vehicle1and capturing the leftward image of the vehicle1is rotationally changed in the opposite direction, the opposite rotation correction is performed.

According to this, the peripheral information of a position originally desired to be acquired can be acquired with high accuracy.

[Control Example at Time of Acceleration]

Next, a control example at the time of acceleration will be described. At the time of the acceleration, the front portion of the vehicle1is lifted and the rear portion of the vehicle1is lowered unlike the time of the deceleration. Therefore, the conversion center position is corrected in a direction opposite to that at the time of the deceleration. The correction can be performed by a method similar to that at the time of the deceleration.

For example, in S602, at the time of the orientation change in which the front portion of the vehicle1is moved upward and the rear portion of the vehicle1is moved downward (at the time of acceleration), the ECU23sets the conversion center position of the fisheye camera41, which is disposed on the front portion of the vehicle1and captures the forward image of the vehicle1, to a position below a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. Furthermore, the ECU23sets the conversion center position of the fisheye camera43, which is disposed on the rear portion of the vehicle1and captures the rearward image of the vehicle1, to a position above a predetermined position (conversion center position when the vehicle1is in the horizontal state).

In the orientation change state in which the front portion of the vehicle1is moved upward and the rear portion of the vehicle1is moved downward (for example, a state of being tilted to the rear side at the time of acceleration), the ECU22corrects the conversion center position of the fisheye camera42, which is disposed on the right-side portion of the vehicle1and captures the rightward image of the vehicle1, by rotating the conversion center position counterclockwise around the imaging center of the fisheye camera42based on an orientation after the orientation change. Furthermore, at the time of the orientation change in which the front portion of the vehicle1is moved upward and the rear portion of the vehicle1is moved downward (at the time of acceleration), the ECU22corrects the conversion center position of the fisheye camera44, which is disposed on the left-side portion of the vehicle1and captures the leftward image of the vehicle1, by rotating the conversion center position clockwise around the imaging center of the fisheye camera44based on an orientation after the orientation change.

According to this, the peripheral information of a position originally desired to be acquired can be acquired with high accuracy.

[Control Example in Case where Lateral Acceleration is Generated]

In addition to the acceleration/deceleration, the acceleration is generated in the lateral direction by steering during traveling such as traveling in a curve, the left-side portion of the vehicle1is lowered while the right-side portion of the vehicle1is lifted (tilted to the left side), and the right-side portion of the vehicle1is lowered while the left-side portion of the vehicle1is lifted (tilted to the right side). For example, an acceleration is generated in the right direction by steering when the vehicle travels in a left curve, and the vehicle1is tilted to the right side. On the other hand, acceleration is generated in the left direction by steering when the vehicle travels in a right curve, and the vehicle1is tilted to the left side.

First, a control example in a case where the orientation of the vehicle1is tilted to the right side will be described.

<Fisheye Cameras42and44>

For example, in S602, in the orientation change state in which the right portion of the vehicle1is moved downward and the left portion of the vehicle1is moved upward (state of being tilted to the right side), the ECU22changes the conversion center position of the fisheye camera42, which is disposed on the right-side portion of the vehicle1and captures the rightward image of the vehicle1, to a position above a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. In a case where the vehicle1is tilted to the right side, since the conversion center position of the fisheye camera42capturing the rightward image is directed downward from the area in which the peripheral information is originally desired to be acquired, the change is corrected upward.

In the orientation change state in which the right portion of the vehicle1is moved downward and the left portion of the vehicle1is moved upward (state of being tilted to the right side), the ECU22changes the conversion center position of the fisheye camera44, which is disposed on the left-side portion of the vehicle1and captures the leftward image of the vehicle1, to a position below a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. In a case where the vehicle1is tilted to the right side, since the conversion center position of the fisheye camera42capturing the leftward image is directed upward from the area in which the peripheral information is originally desired to be acquired, the change is corrected downward.

<Fisheye Cameras41and43>

In the orientation change state in which the right portion of the vehicle1is moved downward and the left portion of the vehicle1is moved upward (state of being tilted to the right side), the ECU23corrects the conversion center position of the fisheye camera41, which is disposed on the front portion of the vehicle1and captures the forward image of the vehicle1, by rotating the conversion center position counterclockwise around the imaging center of the fisheye camera41based on an orientation after the orientation change. Furthermore, in the orientation change state in which the right portion of the vehicle1is moved downward and the left portion of the vehicle1is moved upward (state of being tilted to the right side), the ECU23corrects the conversion center position of the fisheye camera43, which is disposed on the rear portion of the vehicle1and captures the rearward image of the vehicle1, by rotating the conversion center position clockwise around the imaging center of the fisheye camera43based on an orientation after the orientation change.

In a case where the vehicle1is tilted to the right side, since the conversion center position of the fisheye camera41capturing the forward image is rotated clockwise, the conversion center position is corrected by being rotated counterclockwise by the extent of the change of the rotation. Similarly, in a case where the vehicle1is tilted to the right side, since the conversion center position of the fisheye camera43capturing the rearward image is rotated counterclockwise, the conversion center position is corrected by being rotated clockwise by the extent of the change of the rotation.

Next, a control example in a case where the orientation of the vehicle1is tilted to the left side will be described. This control example is a reverse processing of the control example in a case where the orientation of the vehicle1is tilted to the right side.

<Fisheye Cameras42and44>

For example, in S602, in the orientation change state in which the right portion of the vehicle1is moved upward and the left portion of the vehicle1is moved downward (state of being tilted to the left side), the ECU22changes the conversion center position of the fisheye camera42, which is disposed on the right-side portion of the vehicle1and captures the rightward image of the vehicle1, to a position below a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. In a case where the vehicle1is tilted to the left side, since the conversion center position of the fisheye camera42capturing the rightward image is directed upward from the area in which the peripheral information is originally desired to be acquired, the change is corrected downward.

In the orientation change state in which the right portion of the vehicle1is moved upward and the left portion of the vehicle1is moved downward (state of being tilted to the left side), the ECU22changes the conversion center position of the fisheye camera44, which is disposed on the left-side portion of the vehicle1and captures the leftward image of the vehicle1, to a position above a predetermined position (conversion center position when the vehicle1is in the horizontal state) based on an orientation after the orientation change. In a case where the vehicle1is tilted to the left side, since the conversion center position of the fisheye camera42capturing the leftward image is directed downward from the area in which the peripheral information is originally desired to be acquired, the change is corrected upward.

<Fisheye Cameras41and43>

In the orientation change state in which the right portion of the vehicle1is moved upward and the left portion of the vehicle1is moved downward (state of being tilted to the left side), the ECU23corrects the conversion center position of the fisheye camera41, which is disposed on the front portion of the vehicle1and captures the forward image of the vehicle1, by rotating the conversion center position clockwise around the imaging center of the fisheye camera41based on an orientation after the orientation change. Furthermore, in the orientation change state in which the right portion of the vehicle1is moved upward and the left portion of the vehicle1is moved downward (state of being tilted to the left side), the ECU23corrects the conversion center position of the fisheye camera43, which is disposed on the rear portion of the vehicle1and captures the rearward image of the vehicle1, by rotating the conversion center position counterclockwise around the imaging center of the fisheye camera43based on an orientation after the orientation change.

In a case where the vehicle1is tilted to the left side, since the conversion center position of the fisheye camera41capturing the forward image is rotated counterclockwise, the conversion center position is corrected by being rotated clockwise by the extent of the change of the rotation. Similarly, in a case where the vehicle1is tilted to the left side, since the conversion center position of the fisheye camera43capturing the rearward image is rotated clockwise, the conversion center position is corrected by being rotated counterclockwise by the extent of the change of the rotation.

As described above, various types of processing have been described in accordance with the orientation change of the vehicle1, for example, at the time of acceleration, at the time of deceleration, and at the time of lateral acceleration. However, some of these types of processing may be executed in combination. For example, in a case where the vehicle travels in a left curve while decelerating, both a change in height of the front portion and the rear portion of the vehicle1and a change in height of the left portion and the right portion of the vehicle1occur. Therefore, processing corresponding to each change may be executed together.

Image conversion from the fisheye image to the planar image is executed based on the conversion center position set (corrected) in S602, and various operations such as acquisition of the peripheral information and driving assistance are executed by using the converted planar image. Accordingly, the series of processing inFIG.6is ended.

As described above, in the present embodiment, the conversion center position for converting the fisheye images of the fisheye cameras disposed at the front portion, the rear portion, and the left and right side portions of the vehicle into the planar images is controlled based on the orientation of the vehicle.

As a result, it is possible to accurately acquire the peripheral information regarding a desired area regardless of the orientation change of the vehicle. Therefore, various processing such as driving assistance and automated driving using the acquired peripheral information can also be executed with high accuracy.

Modifications

An image conversion frequency of each of the fisheye cameras42and44disposed on the right and left sides of the vehicle1may be controlled to be lower than the image conversion frequency of each of the fisheye cameras41and43disposed on the front and rear sides of the vehicle1. For example, in a case where a vehicle is traveling on a single-lane road on one side or a lane change is not performed, a change in environment (peripheral information) in the front-and-rear direction often affects driving more than a change in environment (peripheral information) in the right-and-left direction. Therefore, by increasing the image conversion frequency (image acquisition frequency) in the front-and-rear direction rather than the right-and-left direction, more accurate information can be acquired in a timely manner, and by suppressing the image conversion frequency in the right-and-left direction, a processing load can be reduced.

Furthermore, in the embodiment described above, an example in which the ECU22and the ECU23perform processing separately has been described. However, these may be configured as a single ECU, and the single ECU may execute the processing of the embodiment described above or the modification.

Furthermore, a shape, a position, and a size of the rectangular area for the cutting out described in the embodiment are merely examples, and are not limited to the illustrated examples. A rectangular area having a larger size than the illustrated rectangular area may be used, or a rectangular area having a smaller size may be used.

Other Embodiments

Furthermore, a program for implementing one or more functions described in the embodiments is supplied to a system or apparatus through a network or a storage medium, and one or more processors in a computer of the system or apparatus can read and execute the program. The present invention is also achievable by such an aspect.

The invention is not limited to the foregoing embodiments, and various variations/changes are possible within the spirit of the invention.

Summary of Embodiments

According to a first aspect, there is provided a control device (for example,2) that controls imaging with fisheye cameras (41to44) disposed on front and rear portions and right and left side portions of a vehicle (for example,1), the control device including:a detection unit (for example,22,23,5, and7c) configured to detect an orientation of the vehicle; anda control unit (for example,22) configured to control a conversion center position (for example,301,701, and751) for converting a fisheye image of each of the fisheye cameras into a planar image based on the orientation of the vehicle.

As a result, it is possible to accurately acquire the peripheral information regarding a desired area regardless of the orientation change of the vehicle. Therefore, various processing such as driving assistance and automated driving using the acquired peripheral information can also be executed with high accuracy.

In the control device (for example,2) according to a second aspect,the control unit sets the conversion center position of a first fisheye camera (for example,41), which is disposed on the front portion of the vehicle and captures a forward image of the vehicle, to a position above a predetermined position (for example, a position when the vehicle is in a horizontal state), based on an orientation change state in which the front portion of the vehicle is moved downward and the rear portion of the vehicle is moved upward (for example, state of being tilted to the front side at the time of deceleration).

According to this, it is possible to reduce influence of the orientation change in the front-and-rear direction and accurately acquire peripheral information of a desired area for the fisheye camera that captures the forward image.

In the control device (for example,2) according to a third aspect,the control unit sets the conversion center position of a second fisheye camera (for example,43), which is disposed on the rear portion of the vehicle and captures a rearward image of the vehicle, to a position below a predetermined position (for example, a position when the vehicle is in the horizontal state), based on an orientation change state in which the front portion of the vehicle is moved downward and the rear portion of the vehicle is moved upward (for example, a state of being tilted to the front side at the time of deceleration).

According to this, it is possible to reduce influence of the orientation change in the front-and-rear direction and accurately acquire peripheral information of a desired area for the fisheye camera that captures the rearward image.

In the control device (for example,2) according to a fourth aspect,based on an orientation change state in which the front portion of the vehicle is moved downward and the rear portion of the vehicle is moved upward (for example, a state of being tilted to the front side at the time of deceleration),the control unit corrects the conversion center position of a third fisheye camera (for example,42), which is disposed on the right-side portion of the vehicle and captures a rightward image of the vehicle, by rotating the conversion center position clockwise around an imaging center of the third fisheye camera, andcorrects the conversion center position of a fourth fisheye camera (for example,44), which is disposed on the left-side portion of the vehicle and captures a leftward image of the vehicle, by rotating the conversion center position counterclockwise around the imaging center of the fourth fisheye camera.

According to this, it is possible to reduce influence of the orientation change in the front-and-rear direction and accurately acquire peripheral information of a desired area for the fisheye cameras that capture the rightward and leftward images, respectively.

In the control device (for example,2) according to a fifth aspect,the control unit sets the conversion center position of the first fisheye camera (for example,41), which is disposed on the front portion of the vehicle and captures the forward image of the vehicle, to a position below a predetermined position (for example, a position when the vehicle is in the horizontal state), based on an orientation change state in which the front portion of the vehicle is moved upward and the rear portion of the vehicle is moved downward (for example, a state of being tilted to the rear side at the time of acceleration).

According to this, it is possible to reduce influence of the orientation change in the front-and-rear direction and accurately acquire peripheral information of a desired area for the fisheye camera that captures the forward image.

In the control device (for example,2) according to a sixth aspect,the control unit sets the conversion center position of the second fisheye camera (for example,43), which is disposed on the rear portion of the vehicle and captures the rearward image of the vehicle, to a position above a predetermined position (for example, a position when the vehicle is in the horizontal state), based on an orientation change state in which the front portion of the vehicle is moved upward and the rear portion of the vehicle is moved downward (for example, a state of being tilted to the rear side at the time of acceleration).

According to this, it is possible to reduce influence of the orientation change in the front-and-rear direction and accurately acquire peripheral information of a desired area for the fisheye camera that captures the rearward image.

In the control device (for example,2) according to a seventh aspect,based on an orientation change state in which the front portion of the vehicle is moved upward and the rear portion of the vehicle is moved downward (state of being tilted to the rear side at the time of acceleration),the control unit corrects the conversion center position of the third fisheye camera (for example,42), which is disposed on the right-side portion of the vehicle and captures the rightward image of the vehicle, by rotating the conversion center position counterclockwise around the imaging center of the third fisheye camera, andcorrects the conversion center position of the fourth fisheye camera (for example,44), which is disposed on the left-side portion of the vehicle and captures the leftward image of the vehicle, by rotating the conversion center position clockwise around the imaging center of the fourth fisheye camera.

According to this, the conversion center positions of the fisheye cameras on the right and left side portions that are rotationally moved due to the influence of the orientation change in the front-and-rear direction can be corrected to appropriate positions. Therefore, it is possible to accurately acquire peripheral information of a desired area for the fisheye cameras that capture the rightward and leftward images, respectively.

In the control device (for example,2) according to an eighth aspect,the control unit changes the conversion center position of the third fisheye camera (for example,42), which is disposed on the right-side portion of the vehicle and captures the rightward image of the vehicle, to a position above a predetermined position (for example, a position when the vehicle is in the horizontal state), based on an orientation change state in which the right portion of the vehicle is moved downward and the left portion of the vehicle is moved upward (when lateral acceleration is generated in the right direction, that is, a state of being tilted to the right side).

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the right side) of the vehicle at the time of executing the steering operation when the vehicle travels in a left curve and acquire the appropriate rightward image.

In the control device (for example,2) according to a ninth aspect,the control unit changes the conversion center position of the fourth fisheye camera (for example,42), which is disposed on the left-side portion of the vehicle and captures the leftward image of the vehicle, to a position below a predetermined position (for example, a position when the vehicle is in the horizontal state), based on an orientation change state in which the right portion of the vehicle is moved downward and the left portion of the vehicle is moved upward (when lateral acceleration is generated in the right direction, that is, a state of being tilted to the right side).

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the right side) of the vehicle at the time of executing the steering operation when the vehicle travels in a left curve and acquire the appropriate leftward image.

In the control device (for example,2) according to a tenth aspect,based on an orientation change state in which the right portion of the vehicle is moved downward and the left portion of the vehicle is moved upward (when the lateral acceleration is generated in the right direction, that is, a state of being tilted to the right side),the control unit corrects the conversion center position of the first fisheye camera (for example,41), which is disposed on the front portion of the vehicle and captures the forward image of the vehicle, by rotating the conversion center position counterclockwise around the imaging center of the first fisheye camera, andcorrects the conversion center position of the second fisheye camera (for example,43), which is disposed on the rear portion of the vehicle and captures the rearward image of the vehicle, by rotating the conversion center position clockwise around the imaging center of the second fisheye camera.

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the right side) of the vehicle at the time of executing the steering operation when the vehicle travels in a left curve and acquire the appropriate forward and rearward image.

In the control device (for example,2) according to an eleventh aspect,the control unit changes the conversion center position of the third fisheye camera (for example,42), which is disposed on the right-side portion of the vehicle and captures the rightward image of the vehicle, to a position below a predetermined position, based on an orientation change state in which the right portion of the vehicle is moved upward and the left portion of the vehicle is moved downward (when the lateral acceleration is generated in the left direction, that is, a state of being tilted to the left side).

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the left side) of the vehicle at the time of executing the steering operation when the vehicle travels in a right curve and acquire the appropriate rightward image.

In the control device (for example,2) according to a twelfth aspect,the control unit changes the conversion center position of the fourth fisheye camera (for example,44), which is disposed on the left-side portion of the vehicle and captures the leftward image of the vehicle, to a position above a predetermined position, based on an orientation change state in which the right portion of the vehicle is moved upward and the left portion of the vehicle is moved downward (when the lateral acceleration is generated in the left direction, that is, a state of being tilted to the left side).

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the left side) of the vehicle at the time of executing the steering operation when the vehicle travels in a right curve and acquire the appropriate leftward image.

In the control device (for example,2) according to a thirteenth aspect,based on an orientation change state in which the right portion of the vehicle is moved upward and the left portion of the vehicle is moved downward (when the lateral acceleration is generated in the left direction, that is a state of being tilted to the left side),the control unit corrects the conversion center position of the first fisheye camera (for example,41), which is disposed on the front portion of the vehicle and captures the forward image of the vehicle, by rotating the conversion center position clockwise around the imaging center of the first fisheye camera, andcorrects the conversion center position of the second fisheye camera (for example,43), which is disposed on the rear portion of the vehicle and captures the rearward image of the vehicle, by rotating the conversion center position counterclockwise around the imaging center of the second fisheye camera.

According to this, for example, it is possible to correct the influence of the orientation change (being tilted to the left side) of the vehicle at the time of executing the steering operation when the vehicle travels in a right curve and acquire the appropriate forward and rearward images.

In the control device (for example,2) according to a fourteenth aspect,the detection unit detects the orientation of the vehicle based on at least one of the acceleration or the deceleration of the vehicle and the lateral acceleration perpendicular to a traveling direction of the vehicle.

According to this, it is possible to easily detect the orientation of the vehicle, which is changed in the front-and-rear direction and/or the right-and-left direction.In the control device (for example,2) according to a fifteenth aspect,the detection unit calculates the lateral acceleration based on a speed of the vehicle and a steering amount of the vehicle.

According to this, the lateral acceleration (acceleration in the right-and-left direction) can be easily acquired.

In the control device (for example,2) according to a sixteenth aspect,the detection unit detects the orientation of the vehicle by using a gyro sensor (for example,5).

According to this, it is possible to easily detect the orientation of the vehicle.

According to a seventeenth aspect, there is provided an operation method for a control device (for example,2) that controls imaging with fisheye cameras (for example,41to44) disposed on front and rear portions and right and left side portions of a vehicle (for example,1), the operation method including.detecting an orientation of the vehicle (for example, S601); andcontrolling a conversion center position (for example,301,701, and751) for converting a fisheye image of each of the fisheye cameras into a planar image based on the orientation of the vehicle (for example, S602).

As a result, it is possible to accurately acquire the peripheral information regarding a desired area regardless of the orientation change of the vehicle. Therefore, various processing such as driving assistance and automated driving using the acquired peripheral information can also be executed with high accuracy.

According to an eighteenth aspect, there is a non-transitory computer-readable storage medium that stores a program causing a computer to function as the control device according to any one of the first to sixteenth aspects.

According to this, the operation of the control device can be realized by the computer.

According to the present invention, it is possible to accurately acquire the peripheral information regarding a desired area regardless of the orientation change of the vehicle. Therefore, various processing such as driving assistance and automated driving using the acquired peripheral information can also be executed with high accuracy.