Vehicle control apparatus and program with rotational control of captured image data

A vehicle control apparatus of the embodiment includes an acquisition portion acquiring captured image data output from an imaging portion that is provided at a vehicle and that images a surrounding of the vehicle, and vehicle state data output from a vehicle state detection portion provided at the vehicle, and a control portion performing a rotation control on the captured image data based on an inclination in a left-right direction of the vehicle relative to a horizontal direction which serves as a direction included in a horizontal plane orthogonal to a direction of gravity, the inclination in the left-right direction of the vehicle being calculated from the vehicle state data, the control portion performing the rotation control in a manner that a horizontal line included in a subject captured in the captured image data is substantially parallel to a lateral-direction side with respect to a display region serving as an output destination.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a National Stage of International Application No. PCT/JP2014/050387 filed Jan. 10, 2014, claiming priority based on Japanese Patent Application No. 2013-039895 filed Feb. 28, 2013 and Japanese Patent Application No. 2013-062440 filed Mar. 25, 2013, the contents of all of which are incorporated herein by reference in their entirety.

TECHNICAL FIELD

An embodiment of the present invention relates to a vehicle control apparatus and program.

BACKGROUND ART

Conventionally, a technique for providing a vehicle driver with image data captured as surrounding environments of a vehicle by plural cameras which are installed at the vehicle is known as a technique for assisting a parking of the vehicle. A technique for correcting the captured image data depending on an operation of the vehicle in a case where the image data is provided to the driver is proposed. In addition, in order to easily recognize the surrounding environments, a technique for generating bird's eye view image data indicating a ground around the vehicle in an overhead view is proposed.

DOCUMENT OF PRIOR ART

Patent Documents

OVERVIEW OF INVENTION

Problem to be Solved by Invention

Nevertheless, in the conventional art, an issue that the image data is not corrected on a real-time basis is raised because the image data is corrected only when an instruction is made by the driver, for example, or an issue that an appropriate display of the image data is not sufficiently made is raised because an inclination of the vehicle is detected by a vehicle height sensor, the inclination of the vehicle is not accurately obtained in a case where wheels are not in contact with the ground, for example. In addition, an issue that it is difficult to accurately grasp a state around the vehicle by the display based on such the image data is raised.

Means for Solving Problem

A vehicle control apparatus according to embodiments of the present invention, as an example, includes an acquisition portion acquiring captured image data output from an imaging portion that is provided at a vehicle and that images a surrounding of the vehicle, and vehicle state data output from a vehicle state detection portion provided at the vehicle, and a control portion performing a rotation control on the captured image data based on an inclination in a left-right direction of the vehicle relative to a horizontal direction which serves as a direction included in a horizontal plane orthogonal to a direction of gravity, the inclination in the left-right direction of the vehicle being calculated from the vehicle state data, the control portion performing the rotation control in a manner that a horizontal line included in a subject captured in the captured image data is substantially parallel to a lateral-direction side at a display region serving as an output destination. Accordingly, as an example, an effect that it is easy to grasp a state around the vehicle on a basis of the captured image data is obtained.

In addition, in the aforementioned vehicle control apparatus, as an example, the vehicle state detection portion acquires acceleration data serving as the vehicle state data and output from an acceleration detection portion provided at the vehicle, and the control portion performs the rotation control on the captured image data depending on a roll angle indicating an inclination around a front-rear axis of the vehicle obtained from the acceleration data with an origin at a position coordinate within the display region of the captured image data, the position coordinate corresponding to a center of a lens used for imaging by the imaging portion. Accordingly, as an example, an effect that it is easy to grasp the state around the vehicle on a basis of the captured image data is obtained. In addition, an effect that a height difference is easily visually recognizable is obtained.

Further, in the aforementioned vehicle control apparatus, as an example, the control portion further performs an enlargement processing or a reduction processing on the captured image data. Accordingly, as an example, the captured image data is enlarged or reduced depending on the output destination, which obtains an effect where visibility improves.

Furthermore, in the aforementioned vehicle control apparatus, as an example, the control portion further moves the position coordinate corresponding to the center of the lens from a center of the display region relative to the captured image data. Accordingly, as an example, the movement control of the captured image data is performed depending on the output destination, which obtains an effect where visibility improves.

Furthermore, in the aforementioned vehicle control apparatus, as an example, the control portion further moves the position coordinate corresponding to the center of the lens from the center of the display region to an upper direction within the display region. Accordingly, as an example, because a lower region than the horizontal line included in the captured image data is displayed, an effect that the state around the vehicle may be easily grasped is obtained.

Furthermore, in the aforementioned vehicle control apparatus, as an example, the captured image data is displayed at a display device, the display device displaying information that represents at least one of a roll angle indicating an inclination around a front-rear axis of the vehicle and a pitch angle indicating an inclination around a left-right axis of the vehicle together with the captured image data. Accordingly, as an example, an effect that it is easy to grasp both the vehicle state and the state around the vehicle is obtained.

Furthermore, in the aforementioned vehicle control apparatus, as an example, the acquisition portion further acquires information indicating whether or not the vehicle is switched to a mode for off-road, and the control portion performs the rotation control on the captured image data depending on the vehicle state data in a case where the vehicle is switched to the mode for off-road. Accordingly, as an example, an effect that the state around the vehicle is visually recognizable in the mode for off-road is obtained.

Furthermore, the aforementioned vehicle control apparatus, as an example, further includes a generation portion generating bird's eye view image data indicating a ground in a surrounding of the vehicle in an overhead view based on the captured image data on which the rotation control is performed by the control portion. Accordingly, as an example, the surroundings of the vehicle are recognizable in the overhead view by referring to the bird's eye view image data on which a change of point of view is performed after the rotation control for leveling is conducted, which obtains an effect that the state around the vehicle is visually recognizable.

Furthermore, a program according to the embodiments of the present invention, as an example, is configured to cause a computer to execute an acquisition step acquiring captured image data output from an imaging portion that is provided at a vehicle and that images a surrounding of the vehicle, and vehicle state data output from a vehicle state detection portion provided at the vehicle and a control step performing a rotation control on the captured image data based on an inclination in a left-right direction of the vehicle relative to a horizontal direction which serves as a direction included in a horizontal plane orthogonal to a direction of gravity, the inclination in the left-right direction of the vehicle being calculated from the vehicle state data, the control step performing the rotation control in a manner that a horizontal line included in a subject captured in the captured image data is substantially parallel to a lateral-direction side at a display region serving as an output destination. Accordingly, as an example, an effect that it is easy to grasp the state around the vehicle on a basis of the captured image data is obtained.

The aforementioned program, as an example, is further configured to cause the computer to execute a generation step generating bird's eye view image data indicating a ground in a surrounding of the vehicle in the overhead view based on the captured image data on which the rotation control is performed by the control portion. As an example, the bird's eye view image data on which a change of point of view is performed after the rotation control for leveling is conducted is generated. As a result, because the surroundings of the vehicle are recognizable in the overhead view, an effect where the state around the vehicle is visually recognizable is obtained.

MODE FOR CARRYING OUT THE INVENTION

The following plural embodiments include the similar components to one another. Thus, the similar components bear the common reference numerals. In addition, duplicated explanation is omitted.

First Embodiment

In the embodiment, a vehicle1may be a car (internal combustion car) including an internal combustion engine (an engine not illustrated) as a driving source, a car (an electric car, a fuel cell car, or the like) including an electric motor (a motor not illustrated) as the driving source, or a car (a hybrid car) including the engine and the motor as the driving sources, for example. In addition, the vehicle1may include various kinds of transmissions and various kinds of apparatuses (systems, parts and the like) necessary for driving the internal combustion engine or the electric motor. Further, method, quantity, layout and the like of an apparatus related to driving of wheels3of the vehicle1may be variously specified.

As illustrated inFIG. 1, a vehicle body2forms a vehicle interior2awhere a passenger (not illustrated) gets in. A steering portion4, an acceleration operating portion5, a braking operating portion6, a speed change operating portion7and the like are provided within the vehicle interior2ain a state facing a seat2bof a driver as the passenger. In the present embodiment, as an example, the steering portion4is a steering wheel projecting from a dashboard (instrument panel) and the acceleration operating portion5is an accelerator pedal positioned at the feet of the driver. The braking operating portion6is a brake pedal positioned at the feet of the driver and the speed change operating portion7is a shift lever projecting from a center console. Nevertheless, the steering portion4, the acceleration operating portion5, the braking operating portion6and the speed change operating portion7are not limited to the aforementioned members.

In addition, a display device8(display output portion) and an audio output device9(audio output portion) are provided within the vehicle interior2a. The display device8is, for example, a LCD (liquid crystal display), an OELD (organic electroluminescent display) and the like. The audio output device9is, as an example, a speaker. In the present embodiment, the display device8is covered by a clear operation input portion10(for example, a touch panel and the like), for example. The passenger and the like may visually confirm a projected image (image) on a display screen of the display device8via the operation input portion10. The passenger and the like may perform an operation input (instruction input) by operating the operation input portion10, i.e., touching, pressing or moving the operation input portion10with one's finger at a position corresponding to the projected image (image) displayed on the display screen of the display device8. In the present embodiment, as an example, the display device8, the audio output device9, the operation input portion10and the like are provided at a monitor device11positioned at a center portion of the dashboard in a vehicle width direction (left-right direction). The monitor device11may include an operation input portion (not illustrated) such as a switch, a dial, a joy-stick and a pressing button, for example. An audio output device (not illustrated) may be provided at other position within the vehicle interior2a, i.e., position different from the monitor device11. In addition, sound may be output from other audio output device than the audio output device9of the monitor device11. In the present embodiment, as an example, the monitor device11is shared by a navigation system and an audio system. Alternatively, a monitor device of a surroundings monitoring apparatus may be separately provided from the aforementioned systems. It may be configured that, in addition to the audio output device9, a warning sound and the like may be output from an audio output portion such as a buzzer24(refer toFIG. 3), for example.

As illustrated inFIGS. 1 and 2, in the present embodiment, the vehicle1is a four-wheel vehicle (four-wheel car) as an example. The vehicle1includes two right and left front wheels3F and two right and left rear wheels3R. Further, in the present embodiment, these four wheels3are configured to be steered (capable of being steered). Specifically, as illustrated inFIG. 3, the vehicle1includes a front wheel steering system12steering the front wheels3F and a rear wheel steering system13steering the rear wheels3R. The front wheel steering system12and the rear wheel steering system13are electrically controlled by a surroundings monitoring ECU14(electronic control unit) and the like to operate respective actuators12aand13a. Each of the front wheel steering system12and the rear wheel steering system13is, for example, an electric power steering system, an SBW (steer by wire) system, and the like. The front wheel steering system12and the rear wheel steering system13assist a steering force by adding torque (assist torque) to the steering portion4by the actuators12aand13a, and steer the corresponding wheels3(the front wheels3F or the rear wheels3R), for example. Each of the actuators12aand13amay steer one of or more than one of the wheels3. In the present embodiment, as an example, the two front wheels3F are steered substantially parallel to each other at the same phases (same phases, same steering directions, same rotation directions) and the two rear wheels3R are steered substantially parallel to each other at the same phases. The driving wheels may be variously specified.

In the present embodiment, as an example, plural (in the embodiment, four, as an example) imaging portions16(16a-16d) are provided at the vehicle1(vehicle body2) as illustrated inFIG. 2. Each of the imaging portions16is, for example, a digital camera incorporating an imaging element such as a CCD (charge coupled device), a CIS (CMOS image sensor) and the like. The imaging portions16may output image data (moving image data, frame data) at a predetermined frame rate. Each of the imaging portions16includes a wide-angle lens to thereby take a picture in a range from 140° to 220° in a horizontal direction (view angle). An optical axis of the imaging portion16is specified to face downward (obliquely downward). Thus, the imaging portion16takes a picture of outside environment around the vehicle body2including a road surface on which the vehicle1is movable.

In the above, the horizontal direction is a direction included in a horizontal plane orthogonal to a direction of gravity (vertical direction).

In the embodiment, as an example, the imaging portion16ais positioned at an end portion2c(an end portion in a plan view) at a front side (a front side in a vehicle front-rear direction) of the vehicle body2and is provided at a front bumper, for example. The imaging portion16bis positioned at an end portion2dat a left side (a left side in a vehicle width direction) of the vehicle body2and is provided at a door mirror2g(projecting portion) at a left side. The imaging portion16cis positioned at an end portion2eat a rear side (a rear side in the vehicle front-rear direction) of the vehicle body2and is provided at a wall portion at a lower side of a door2hof a rear trunk. The imaging portion16dis positioned at an end portion2fat a right side (a right side in the vehicle width direction) of the vehicle body2and is provided at a door mirror2g(projecting portion) at a right side. In the present embodiment, the method of mounting the camera at the vehicle is not limited and the camera may be mounted so that the image data in a front direction, the image data in right and left side directions and the image data in a rear direction relative to the vehicle is obtainable.

The surroundings monitoring ECU14performs a calculation processing and an image processing based on the image data obtained by the plural imaging portions16. The surroundings monitoring ECU14is able to generate a wider view angle image and a virtual bird's eye view image (planar image) where the vehicle1(vehicle body2) is viewed from an upper side.

In the present embodiment, as an example, in a surroundings monitoring system100as illustrated inFIG. 3, a brake system18, a steering angle sensor19(angular sensor), an accelerator sensor20, a shift sensor21, a wheel speed sensor22, an acceleration sensor26, and the like are electrically connected, in addition to the surroundings monitoring ECU14, the monitor device11, the front wheel steering system12, the rear wheel steering system13, and the like, via an in-vehicle network23(electric telecommunication line). The in-vehicle network23is configured as a CAN (controller area network) as an example. The surroundings monitoring ECU14may send a control signal via the in-vehicle network23to control the front wheel steering system12, the rear wheel steering system13, the brake system18, and the like. The surroundings monitoring ECU14may also receive detection results of a torque sensor12b, a tire angle sensor13b(for the rear wheels3R), an actuator18a, a brake sensor18b, the steering angle sensor19(for the front wheels3F), the accelerator sensor20, the shift sensor21, the wheel speed sensor22, the acceleration sensor26, and the like and indicator signals (control signals, operation signals, input signals, data) of the operation input portion10and the like via the in-vehicle network23.

In the present embodiment, the two acceleration sensors26(26a,26b) are provided at the vehicle1. In the embodiment, the vehicle1is equipped with an ESC (electronic stability control). Then, the acceleration sensors26(26a,26b) as conventionally mounted to the vehicle equipped with the ESC (electronic stability control) are employed. In the present embodiment, no restriction is made on the acceleration sensor. The sensor that is able to detect the acceleration in the left-right direction of the vehicle1is acceptable.

FIG. 4is a diagram illustrating an example of detection directions of the acceleration sensors26a,26b. A detection direction401is the detection direction of the acceleration sensor26awhile a detection direction402is the detection direction of the acceleration sensor26b. The detection direction401illustrated inFIG. 4corresponds to a direction inclined by 45 degrees from a travelling direction (front-rear direction) of the vehicle1on a plane in parallel with a ground (a plane on which the vehicle1is movable). The detection direction402forms an angle of 90 degrees relative to the detection direction401on the plane in parallel with the ground. Because the two different detection directions are provided on the plane in parallel with the ground, the acceleration in the front-rear direction and the acceleration in the left-right direction may be obtained. In the present embodiment, no restriction is made on the detection direction and at least the acceleration in the left-right direction may be obtained. Calculations of the acceleration in the front-rear direction and the acceleration in the left-right direction are made at the surroundings monitoring ECU14.

The front-rear direction of the vehicle1indicates the travelling direction and an opposite direction from the travelling direction of the vehicle1. The left-right direction of the vehicle1is a direction included in a surface orthogonal to the travelling direction of the vehicle1.

Back toFIG. 3, the surroundings monitoring ECU14includes, as an example, a CPU14a(central processing unit), a ROM14b(read only memory), a RAM14c(random access memory), a display control portion14d, an audio control portion14e, a SSD14f(solid state drive, flush memory), and the like. The CPU14aperforms the image processing related to the image displayed at the display device8and the various calculation processing such as calculation of a moving path of the vehicle1and determination of whether or not interference with an object occurs, for example. The CPU14areads out program stored (installed) at a nonvolatile memory device such as the ROM14b, for example, and performs the calculation processing based on the aforementioned program.

The RAM14ctentatively stores various data used for the calculations at the CPU14a. The display control portion14dmainly performs the image processing using the image data obtained at the imaging portions16and the image processing (composition and the like, as an example) of the image data displayed at the display device8, for example, within the calculation processing at the surroundings monitoring ECU14. In addition, the audio control portion14emainly performs processing of audio data output at the audio output device9within the calculation processing at the surroundings monitoring ECU14. The SSD14fis a rewritable nonvolatile memory portion that is able to store data even in a case where a power source of the surroundings monitoring ECU14is turned off. The CPU14a, the ROM14b, the RAM14cand the like may be integrated within the same package. The surroundings monitoring ECU14may be configured to include other logic operation processor such as a DSP (digital signal processor) or a logic circuit, for example, than the CPU14a. In addition, instead of the SSD14f, a HDD (hard disk drive) may be provided. Further, the SSD14for the HDD may be provided separately from the surroundings monitoring ECU14.

FIG. 5is a block diagram illustrating a construction of a surroundings monitoring portion500realized within the surroundings monitoring ECU14according to the present embodiment. Each construction within the surroundings monitoring portion500illustrated inFIG. 5is realized in a case where the CPU14aconfigured as the surroundings monitoring ECU14inFIG. 4performs software stored within the ROM14b.

The surroundings monitoring portion500realizes an acquisition portion501, an angle calculation portion502, a filtering control portion503, an image processing portion504and an output portion505by performing software stored within the ROM14b(computer readable storage medium). At this time, software (program) may be provided via other computer readable storage medium.

Then, the surroundings monitoring portion500according to the present embodiment assists the driving of the driver by displaying the image data by which a state around the vehicle1is recognizable on the basis of the captured image data input from the imaging portions16in a case where the vehicle1moves to be parked, and the acceleration data as an example of a vehicle state data acquired by the acceleration sensor26(acceleration detection portion) functioning as an example of a vehicle state detection portion.

The acquisition portion501acquires various pieces of information from various sensors, for example, provided at the vehicle1. The acquisition portion501according to the present embodiment acquires the captured image data output from the imaging portions16ato16dprovided at the vehicle1to capture the images in the surroundings of the vehicle1and the acceleration data output from the acceleration sensors26a,26bprovided at the vehicle1. Further, the acquisition portion501acquires information indicating whether or not a mode specified by a switch provided at the operation input portion10is an off-road mode. The acquisition portion501outputs the acquired information to the angle calculation portion502and the image processing portion504.

The acquisition portion501also correlates the captured image data with the acceleration data where time when the image is captured in the captured image data and time when the acceleration is detected in the acceleration data substantially match each other.

The angle calculation portion502calculates an inclination angle (a pitch angle and a roll angle) of the vehicle1based on the acceleration data acquired by the acceleration sensors26a,26b. Here, the pitch angle is an angle indicating an inclination of the vehicle1around a left-right axis (axis412inFIG. 4) of the vehicle1. In a case where the vehicle1is present on the horizontal plane (ground), the pitch angle is zero degrees.

The roll angle is an angle indicating an inclination of the vehicle1around a longitudinal axis (axis411inFIG. 4) of the vehicle1. In a case where the vehicle1is present on the horizontal plane (ground), the roll angle is zero degrees. In order to calculate the pitch angle and the roll angle, the angle calculation portion502first calculates an acceleration a1in the front-rear direction and an acceleration a2in the left-right direction of the vehicle1.

The angle calculation portion502calculates the acceleration a1in the front-rear direction using the following equation (1). The acceleration in the detection direction401is specified to be GL1and the acceleration in the detection direction402is specified to be GL2. In the present embodiment, as an example, the acceleration a1in the front-rear direction turns to 0 G in a case where the pitch angle is 0° (in a case where the vehicle1is horizontal) and the acceleration a1in the front-rear direction turns to 1 G in a case where the pitch angle is 90° (in a case where the vehicle1is vertical).
a1=GL1×cos 45°−GL2×cos 45°  (1)

Next, the angle calculation portion502calculates the acceleration a2in the left-right direction using the following equation (2).
a2=−(GL1×sin 45°+GL2×sin 45°)  (2)

Further, the angle calculation portion502calculates a pitch angle PA using the following equation (3).
PA[deg]=90[deg]×a1[G]  (3)

Further, the angle calculation portion502calculates a roll angle RA using the following equation (4).
RA[deg]=90[deg]×a1[G]  (4)

The angle calculation portion502correlates the roll angle and the pitch angle calculated from the acceleration data with the captured image data that is correlated to the aforementioned acceleration data. Accordingly, the roll angle and the pitch angle of the vehicle1when the captured image data is captured are recognizable.

The filtering control portion503performs filtering by low-pass filter relative to the roll angle RA and the pitch angle PA calculated by the angle calculation portion502.

In the present embodiment, steep changes of the roll angle RA and the pitch angle PA, in other words, a steep switching of the image data displayed at the display device8is restrained by performing the low-pass filter. Accordingly, the driver may comfortably watch the image data displayed at the display device8. In the present embodiment, an example where digital filter is used by the filtering control portion503provided within the surroundings monitoring portion500is explained. Nevertheless, for example, analog filter, for example, may be performed relative to a signal output from the acceleration sensor26.

The image processing portion504includes a rotation control portion521, a reduction/enlargement control portion522, a movement control portion523and a composition portion524to generate the image data displayed at the display device8.

The rotation control portion521performs a rotation correction on the captured image data capturing the surroundings of a front side of the vehicle1. A subject of the rotation correction is not limited to the captured image data captured by the imaging portion16aand may be the captured image data captured by the imaging portion16ccapturing the surroundings of a rear side of the vehicle1, for example.

FIG. 6is an example of the captured image data captured by the imaging portion16a. The captured image data illustrated inFIG. 6is captured from the vehicle1that is inclined. The driver tends to recognize the image displayed at the display device8in an objective way and thus tends to recognize areas in the captured image data displayed at the display device8, if the areas include the same heights in a vertical axis direction, include the same heights in reality or heights with a smaller height difference than the actual height difference. In the example illustrated inFIG. 6, a region601and a region602are possibly recognized as the same heights.

Thus, the rotation control portion521according to the present embodiment performs the rotation correction on the captured image data depending on the roll angle obtained by the acceleration sensors26. In other words, the rotation control portion521performs the rotation correction (control) on the captured image data based on the inclination of the vehicle in the left-right direction relative to the horizontal direction serving as the direction included in the horizontal plane orthogonal to the direction of gravity calculated from the vehicle state data. For example, the rotation control portion521performs the rotation correction (control) so that a horizontal line included in a subject captured in the captured image data is substantially parallel to a lateral-direction side at a display region of an output destination.

The rotation control portion521according to the present embodiment performs the rotation correction with an origin at a position coordinate within the display region of the captured image data corresponding to a center of a lens used by the imaging portion16for image capturing depending on the roll angle correlated to the aforementioned captured image data.

FIG. 7is a diagram illustrating an example of a two-dimensional orthogonal coordinate system that indicates the display region of the captured image data in a case where the position coordinate corresponding to the center of the lens serves as the origin. For each position coordinate included in the coordinate system illustrated inFIG. 7, the rotation control portion521converts the position coordinate by an equation (5) indicated below so as to achieve the rotation correction of the captured image data. Here, dx0, dy0 is a coordinate value with the origin at the center of the lens. In addition, θ is the roll angle that is calculated.

FIG. 8is a diagram illustrating an example of the captured image data obtained after the rotation correction is performed by the rotation control portion521. In the example illustrated inFIG. 8, the rotation correction is performed so that the horizontal line included in the subject (environment outside the vehicle1) captured in the captured image data is substantially in parallel with the lateral-direction side of the display region of the display device8. In other words, the rotation correction is performed so that a lower direction of the captured image data corresponds to the direction of gravity of the subject (environment outside the vehicle1) captured in the aforementioned captured image data. At this time, the lower direction and the direction of gravity do not necessarily completely coincide with each other and may coincide with each other so that a height relation within the captured image data is recognizable.

For example, as for the region601and the region602which seem to include the same heights inFIG. 6, it is recognizable inFIG. 8that the region602is present at a higher position than the region601. Therefore, the driver may recognize an objective height in the surrounding environments of the vehicle1. Accordingly, an appropriate driving is achievable, which may improve safety.

The reduction/enlargement control portion522functioning as the control portion performs an enlargement processing or a reduction processing relative to the captured image data after the rotation correction is performed by the rotation control portion521. The reduction/enlargement control portion522converts the position coordinate by an equation (6) indicated below to achieve an enlargement correction or a reduction correction of the captured image data. Here, dx1, dy1 is a coordinate value with the origin at the center of the lens after the rotation correction is performed. Here, magX and magY are horizontal and vertical enlargement/reduction rates. The enlargement/reduction rate is decided on a basis of a relationship between a display size of the captured image data and number of pixels of the display region of the display device8.

The movement control portion523functioning as the control portion performs a control on the captured image data after the enlargement or reduction processing is performed by the reduction/enlargement control portion522so that the position coordinate corresponding to the center of the lens moves from the center of the display region of the display device8. In the present embodiment, the movement control portion523performs a control to move the position coordinate corresponding to the center of the lens from the center of the display region of the display device8to an upper direction within the display region.

That is, in a situation where the vehicle1is inclined, the driver tends to desire to confirm the ground conditions. Thus, the movement control portion523performs the processing to move the position coordinate corresponding to the center of the lens from the center of the display region of the display device8to the upper direction within the display region. Accordingly, conditions upper than the vehicle1such as the sky captured in the captured image data, for example, are not displayed and conditions lower than the vehicle1are displayed. Thus, the user may recognize the ground conditions around the vehicle1by referring to the captured image data displayed at the display device8. Accordingly, an appropriate steering assist is achievable.

The movement control portion523converts the position coordinate by an equation (7) indicated below to achieve the movement of the position coordinate of the captured image data. Here, dx2, dy2 is a coordinate value with the origin at the center of the lens after the enlargement/reduction correction is performed. Here, a destination of the position coordinate of the center of the lens before the movement is (cx, cy).

The composition portion524performs a cutout relative to the captured image data after the movement control is performed by the movement control portion523so as to conform to the display region of the display device8and thereafter combines display information for assisting the steering of the driver.

FIG. 9is a diagram illustrating an example of the image data after the composition is performed by the composition portion524. In the example illustrated inFIG. 9, conditions around the left front wheel of the vehicle1captured by the imaging portion16bis displayed at a display region901. In addition, conditions around the right front wheel of the vehicle1captured by the imaging portion16dis displayed at a display region902. Further, information by which the pitch angle and the roll angle of the vehicle1are recognizable is displayed at a display region903. That is, an inclination of an icon921representing the vehicle1indicates the roll angle while a distance between a center line912passing through the icon921and a line911indicates the pitch angle. Accordingly, in the present embodiment, information by which the roll angle and the pitch angle are recognizable is indicated, however, display method is not limited to the above and other display method is acceptable.

In the vehicle1according to the present embodiment, the roll state and the pitch state of the vehicle1during the off-road driving may be displayed in real time. Accordingly, the driver may easily and objectively recognize the conditions of the surroundings of the vehicle1.

In addition, the captured image data after cut out by the composition portion524is displayed at a display region904. The horizontal line within the image in the captured image data is corrected to be substantially in parallel with a lateral frame of the display device8. In other words, the lower direction of the image in the captured image data is corrected to match the direction of gravity. Accordingly, the driver may easily recognize the surrounding state.

Then, the output portion505outputs the image data that is composited by the composition portion524to the display device8. Accordingly, together with the aforementioned captured image data after the correction processing is performed, information by which the roll angle and the pitch angle are recognizable is displayed at the display device8.

In the example ofFIG. 8 or 9, an estimated course line905of each of the front wheels3F is included. The surroundings monitoring ECU14(CPU14a) is able to calculate a planned course based on detection results of the steering angle sensor19and the tire angle sensor13b, for example, and to include (overlap) the estimated course line905conforming to the planned course in the output image. The estimated course line905is an example of a display element indicating the course that is planned. The surroundings monitoring ECU14corrects the display position, size, posture (inclination) and the like of the estimated course line905depending on the aforementioned rotation, enlargement/reduction and movement corrections. In addition, in a case where the position of the estimated course line905is greatly deviated from a center of the screen, the surroundings monitoring ECU14is able to correct the display region and the estimated course line905in a direction where the deviation is reduced.

In the example ofFIG. 9, the inclination of the icon921relative to a lateral-direction side of the display region903or904(an upper side or a lower side inFIG. 9) corresponds to the roll angle of the vehicle1. Thus, the surroundings monitoring ECU14may constitute a tiltmeter906(roll angle display portion) using the icon921by including an angle scale906a(tilt scale) surrounding the icon921in the output image in a manner that an angle of the angle scale906aremains unchanged relative to the display region903. For example, only by the display of the display region904, it may be difficult to understand the horizontal direction, the vertical direction, and the posture (the roll angle or the pitch angle) of the vehicle1. In this point, as in the example ofFIG. 9, the icon921that performs rotation (rolling) and pitching is displayed and the tiltmeter906is displayed on the screen depending on the roll angle and the pitch angle so that the horizontal direction, the vertical direction and the posture (the roll angle) of the vehicle1may be easily understood, regardless of the state of the image of the display region904. Accordingly, the display region904and the display region903are displayed together (displayed within the same screen or displayed in parallel with each other) so that the state around the vehicle and the state of the posture of the vehicle may be further easily understood.

In addition, the present embodiment may not perform the aforementioned rotation, enlargement/reduction and movement corrections on a constant basis and may be specified to perform the aforementioned corrections in a case where the vehicle1is brought to the off-road mode. For example, the image processing portion504performs the aforementioned rotation, enlargement/reduction and movement corrections at the time of the off-road mode by referring to information acquired by the acquisition portion501indicating whether or not the vehicle1is in the off-road mode.

Here, the off-road mode corresponds to the mode for bringing out a four-wheel driving performance of the vehicle1during the off-road driving and the mode for specifying a total transfer gear to be low. That is, in the present embodiment, the captured image data displayed at the display device8is switched in association with the operation when the off-road driving is performed. At this time, in the present embodiment, the switching of the image displayed at the display device8is not limited to be performed in a case where the vehicle1is switched to the off-road mode. For example, in a case where the vehicle1is switched to the four-wheel driving in a two/four wheel drive switching, it may be controlled that the image after the rotation correction is performed is displayed.

Next, a display processing at the display device8in the surroundings monitoring portion500according to the present embodiment is explained.FIG. 10is a flowchart illustrating procedures of the aforementioned processing in the surroundings monitoring portion500according to the present embodiment.

First, the acquisition portion501acquires the captured image data from the imaging portions16(step S1001). Next, the acquisition portion501acquires the acceleration data from the acceleration sensors26(step S1002).

Then, the angle calculation portion502calculates the roll angle and the pitch angle of the vehicle1from the acceleration data (step S1003).

Next, the filtering control portion503performs filtering by low-pass filter relative to the calculated roll angle and the calculated pitch angle (step S1004).

Then, the rotation control portion521performs the rotation control relative to the captured image data depending on the roll angle (step S1005).

Next, the reduction/enlargement control portion522and the movement control portion523perform the enlargement control and the movement control on the captured image data after the rotation control is performed (step S1006).

Then, the composition portion524performs the cutout conforming to the display region displayed at the display device8relative to the captured image data after the enlargement control and the movement control are performed (step S1007).

Next, the composition portion524combines the captured image data indicating the state around the front wheels and the display information by which the pitch angle and the roll angle are recognizable relative to the captured image data that is cut out (step S1008).

Then, the output portion505outputs the image data after the composition by the composition portion524to the display device8(step S1009).

The surroundings monitoring portion500according to the present embodiment includes the aforementioned construction so as to easily recognize the difference in height in the surroundings of the vehicle1. Accordingly, load of steering may be reduced to thereby improve safety.

Second Embodiment

FIG. 11is a block diagram illustrating a construction of a surroundings monitoring portion700realized within the surroundings monitoring ECU14according to the present embodiment. The CPU14aconfigured as the surroundings monitoring ECU14inFIG. 4executes software stored within the ROM14bto thereby realize an acquisition portion701, an angle calculation portion702, a filtering control portion703, an image processing portion704and an output portion705illustrated inFIG. 11. In addition, the surroundings monitoring portion700realizes a bird's eye view image storage portion706on the RAM14c.

The acquisition portion701acquires various pieces of information from various sensors, for example, provided at the vehicle1. The acquisition portion701according to the present embodiment acquires the captured image data output from the imaging portions16ato16dprovided at the vehicle1to capture the images in the surroundings of the vehicle1and the acceleration data serving as an example of the vehicle state data output from the acceleration sensors26a,26b(acceleration detection portion) provided at the vehicle1and functioning as an example of the vehicle state detection portion. The acquisition portion701outputs the acquired information to the angle calculation portion702and the image processing portion704.

The acquisition portion701also correlates the captured image data with the acceleration data where time when the image is captured in the captured image data and time when the acceleration is detected in the acceleration data substantially match each other.

The angle calculation portion702calculates the inclination angle (the pitch angle and the roll angle) of the vehicle1based on the acceleration data acquired by the acceleration sensors26a,26b. Here, the pitch angle is an angle indicating an inclination of the vehicle1around the left-right axis (axis412inFIG. 4) of the vehicle. In a case where the vehicle1is present on the horizontal plane (ground), the pitch angle is zero degrees.

The roll angle is an angle indicating an inclination of the vehicle1around the longitudinal axis (axis411inFIG. 4) of the vehicle1. In a case where the vehicle1is present on the horizontal plane (ground), the roll angle is zero degrees. In order to calculate the pitch angle and the roll angle, the angle calculation portion702first calculates the acceleration a1in the front-rear direction and the acceleration a2in the left-right direction of the vehicle1.

The angle calculation portion702calculates the acceleration a1in the front-rear direction using the following equation (1). The acceleration in the detection direction401is specified to be GL1and the acceleration in the detection direction402is specified to be GL2. In the present embodiment, as an example, the acceleration a1in the front-rear direction turns to 0 G in a case where the pitch angle is 0° (in a case where the vehicle1is horizontal) and the acceleration a1in the front-rear direction turns to 1 G in a case where the pitch angle is 90° (in a case where the vehicle1is vertical).
a1=GL1×cos 45°−GL2×cos 45°  (1)

Next, the angle calculation portion702calculates the acceleration a2in the left-right direction using the following equation (2).
a2=−(GL1×sin 45°+GL2x)sin 45°  (2)

Further, the angle calculation portion702calculates the pitch angle PA using the following equation (3).
PA[deg]=90[deg]×a1[G]  (3)

Further, the angle calculation portion702calculates the roll angle RA using the following equation (4).
RA[deg]=90[deg]×a2[G]  (4)

The angle calculation portion702correlates the roll angle and the pitch angle calculated from the acceleration data with the captured image data that is correlated to the aforementioned acceleration data. Accordingly, the roll angle and the pitch angle of the vehicle1when the captured image data is captured are recognizable.

The filtering control portion703performs filtering by low-pass filter relative to the roll angle RA and the pitch angle PA calculated by the angle calculation portion702.

In the present embodiment, steep changes of the roll angle RA and the pitch angle PA, in other words, a steep switching of the image data displayed at the display device8is restrained by performing the low-pass filter. Accordingly, the driver may comfortably watch the image data displayed at the display device8. In the present embodiment, an example where digital filter is used by the filtering control portion703provided within the surroundings monitoring portion700is explained. Nevertheless, for example, analog filter, for example, may be performed relative to a signal output from the acceleration sensor26.

The image processing portion704includes a rotation control portion711, a bird's eye view image generation portion712(generation portion), a moving amount calculation portion713, a conversion portion714and a composition portion715each of which serves as the control portion. The image processing portion704generates the image data to be displayed at the display device8.

The rotation control portion711performs the rotation correction on the captured image data capturing the surroundings of a front side of the vehicle1(travelling direction) based on the inclination of the vehicle in the left-right direction relative to the horizontal direction calculated from the vehicle state data (in other words, depending on the roll angle). The horizontal direction is a direction orthogonal to the travelling direction, for example. In addition, the rotation correction may be performed on the captured image data in the same direction as a rotation direction where the vehicle becomes horizontal based on the inclination in the left-right direction of the vehicle calculated from the vehicle state data. In other words, the aforementioned rotation correction may be performed on the captured image data as if the image is captured in a state where the left-right direction of the vehicle1is horizontal (in a state where the vehicle1is arranged on the horizontal plane orthogonal to the direction of gravity). In the present embodiment, as an example, the acceleration is used as the vehicle state data. The vehicle state data, however, is not limited to the acceleration and may be information relevant to the state of the vehicle1. A subject of the rotation correction is not limited to the captured image data captured by the imaging portion16aand may be the captured image data captured by the imaging portion16cthat captures the surroundings of a rear side of the vehicle1.

FIG. 12is a diagram illustrating an example of a state where the vehicle1drives over a stone, for example, during the off-road driving. In the sample illustrated inFIG. 12, because the vehicle1drives over the stone, for example, the vehicle1is inclined by a roll angle θ. In a case where bird's eye view image data is generated from the captured image data captured by the imaging portion16ain the aforementioned state, distortion depending on the roll angle θ occurs.

Therefore, the rotation control portion711according to the present embodiment performs the rotation correction on the captured image data depending on the roll angle θ obtained from the acceleration sensor26.

The rotation control portion711according to the present embodiment performs the rotation correction with the origin at the position coordinate within the display region of the captured image data corresponding to the center of the lens used by the imaging portion16afor image capturing depending on the roll angle correlated to the captured image data.

FIG. 13is a diagram illustrating an example of a two-dimensional orthogonal coordinate system that indicates the display region of the captured image data in a case where the position coordinate corresponding to the center of the lens serves as the origin. For each position coordinate included in the coordinate system illustrated inFIG. 13, the rotation control portion711converts the position coordinate by the equation (5) indicated below so as to achieve the rotation correction of the captured image data. Here, dx0, dy0 is a coordinate value with the origin at the center of the lens. In addition, the angle θ illustrated inFIG. 13is the roll angled that is calculated.

The rotation control portion711performs the rotation correction on all pixels included in a display region801so as to generate a display region802obtained by the rotation of the display region801by the angle θ. Then, the surroundings monitoring portion700generates the bird's eye view image data based on the captured image data including the display region802which is obtained after the rotation control is performed. Accordingly, the bird's eye view image data where the inclination caused by the roll angle θ generated at the vehicle1is corrected may be generated.

In addition, the rotation control portion711according to the present embodiment does not limit the angle for the rotation control on the captured image data to the roll angle by which the vehicle1is inclined from the horizontal plane. The rotation control portion711may perform the rotation control on the captured image data depending on a difference between the roll angle previously calculated and the roll angle presently calculated. This is because the state of the ground around the vehicle1when the vehicle1is driven on the ground that is inclined by a predetermined angle (roll angle previously calculated) is more easily recognizable in a case where the bird's eye view image data is generated from an angle inclined by the predetermined angle from the upper side of the vehicle1in the vertical direction than a case where the bird's eye view image data is generated from the upper side of the vehicle1in the vertical direction. In the aforementioned case, the rotation control portion711performs the rotation control on the captured image data for the difference of the roll angles resulting from the inclination by driving on a stone, for example (difference between the previously calculated roll angle and the presently calculated roll angle), in a case where the vehicle1drives on a stone, for example.

The bird's eye view image generation portion712generates, on a basis of the captured image data which is obtained after the rotation control is performed, the bird's eye view image data obtained by looking down the ground in the travelling direction of the vehicle1serving as the ground around the vehicle1from the upper side. Here, any method for generating the bird's eye view image data from the captured image data is acceptable. For example, a mapping table may be used for conversion.

The generation of the bird's eye view image data is performed each time the captured image data is acquired. In other words, the bird's eye view image generation portion712generates first bird's eye view image data based on first captured image data on which the rotation control is performed by the rotation control portion711and thereafter generates second bird's eye view image data based on second captured image data which is captured by the imaging portions16after the first captured image data is captured and then the vehicle1moves and on which the rotation control is performed by the rotation control portion711.

In the present embodiment, the image data displayed at the display device8is updated each time the vehicle1moves by a predetermined moving amount. Thus, the moving amount calculation portion713compares the bird's eye view image data generated by the bird's eye view image generation portion712and the bird's eye view image data used upon previous updating so as to calculate the moving amount of the vehicle1.

Nevertheless, the comparison in the entire bird's eye view image data causes a great processing load. Thus, the moving amount calculation portion713according to the present embodiment compares predetermined areas within the bird's eye view image data generated by the bird's eye view image generation portion712.

Specifically, the moving amount calculation portion713according to the present embodiment cuts out the predetermined area (display range) from each of the first bird's eye view image data used upon the previous updating and the second bird's eye view image data generated after the first bird's eye view image data so as to calculate an optical flow.

FIG. 14is a diagram illustrating a concept of optical flow calculated by the moving amount calculation portion713. (A) ofFIG. 14is the image data cut out at the predetermined display range from the first bird's eye view image data used upon the previous updating while (B) ofFIG. 14is the image data cut out at the predetermined display range from the second bird's eye view image data generated presently by the bird's eye view image generation portion712.

Then, the moving amount calculation portion713calculates the optical flows indicating a shifting of (feature points of) each displayed object by vectors between the image data illustrated in (A) ofFIG. 14and the image data illustrated in (B) ofFIG. 14. (C) ofFIG. 14illustrates an example of calculated optical flows. In the example illustrated in (C) ofFIG. 14, a length of each vector corresponding to a movement of the feature point (indicated by “X”) in (A) ofFIG. 14to the feature point (indicated by “X”) in (B) ofFIG. 14is indicated.

Then, the moving amount calculation portion713calculates the moving amount of the vehicle1from an average value of the calculated optical flows.

FIG. 15is a diagram illustrating a relation between the average value of the optical flows and the moving amount of the vehicle1. In the example illustrated inFIG. 15, an arrow1901is specified to be the average value of the optical flows. The vehicle1turns about a rear wheel axis. Thus, in a case of the average value1901of the optical flows, the vehicle1turns by a turning angle1θ. Accordingly, the moving amount calculation portion713calculates the turning angle1θ of the vehicle1. Further, the moving amount calculation portion713calculates the moving amount of the vehicle1from the length of each of the optical flows. The moving amount calculation portion713may separately calculate the moving amount of the vehicle1in the front-rear direction and the moving amount of the vehicle1in the left-right direction.

The conversion portion714converts the bird's eye view image data generated presently by the bird's eye view image generation portion712into the bird's eye view image data for composition with the bird's eye view image data stored at the bird's eye view image storage portion706in a case where the moving amount calculated by the moving amount calculation portion713is equal to or greater than a predetermined distance.

In a case where the vehicle1is inclined, the inclination is corrected by the rotation control portion711, however, distortion resulting from the inclination remains in the captured image data captured by the imaging portions16. Thus, in order to reduce the distortion by the conversion portion714, a projective transformation is performed on the bird's eye view image data.

In the present embodiment, the projective transformation is employed so that a case where the torsion of the road surface within the captured image data is generated by the inclination of the vehicle1is converted to a case where the vehicle1is not inclined. For example, a so-called trapezoidal correction where a trapezoidal-shaped area within the captured image data is converted to a rectangular or square area, for example, is included.

FIG. 16is a diagram illustrating the distortion of the captured image data caused by the inclination of the vehicle1and the display range of the captured image data after the conversion is performed by the conversion portion714. In the example illustrated inFIG. 16, distortion states (a1) and (a2) of the bird's eye view image data generated in a roll state of the vehicle1are illustrated and distortion states (b1) and (b2) of the bird's eye view image data generated in a pitch state of the vehicle1are illustrated. In a case where the bird's eye view image data in the distorted state because of the inclination of the vehicle1is combined with the bird's eye view image data stored at the bird's eye view image storage portion706, the distortion is accumulated.

Therefore, the conversion portion714according to the present embodiment performs the projective transformation determined on a basis of a second roll angle inFIG. 16relative to the second captured image data cut out at the display range ((a1), (a2), (b1), (b2)) specified on a basis of the aforementioned roll angle obtained from the acceleration data acquired when the captured image data is captured. The display range ((a1), (a2), (b1), (b2)) is converted to the display range illustrated in (c) ofFIG. 16. Here, a method for specifying the position coordinates of four points indicating the display region serving as a subject of the projective transformation based on the roll angle may be any method regardless of whether it is a conventional method or not. For example, a correlation of the roll angle θ with each of the four position coordinates may be provided beforehand.

The composition portion715combines the bird's eye view image data stored at the bird's eye view image storage portion706and the bird's eye view image data obtained after the projective transformation is performed by the conversion portion714.

FIG. 17is a diagram illustrating an example of the bird's eye view image data obtained after the composition is performed by the composition portion715. Within the bird's eye view image data illustrated inFIG. 17, a display range1101is a range most recently composited. A display range1102is a range composited before the display range1101. A display range1103is a range composited before the display range1102. Accordingly, in the present embodiment, the bird's eye view image data is composited each time the vehicle1moves.

The bird's eye view image storage portion706stores the bird's eye view image data after the composition by the composition portion715. Accordingly, the bird's eye view image storage portion706composites and stores the bird's eye view image data newly generated each time the vehicle1moves by the predetermined moving amount so that the bird's eye view image data indicating the condition of the ground below the vehicle1is stored at the bird's eye view image storage portion706.

FIG. 18is a diagram illustrating an example of the bird's eye view image data stored at the bird's eye view image storage portion706. As illustrated inFIG. 18, besides the bird's eye view image data generated from the captured image data presently captured, the bird's eye view image data generated from the captured image data captured up to the previous time is composited and stored. As illustrated inFIG. 18, the bird's eye view image data stored at the bird's eye view image storage portion706includes the ground below the vehicle1. The configuration of the vehicle1illustrated inFIG. 18is indicated for easily explanation and is not included in the actual bird's eye view image data stored at the bird's eye view image storage portion706.

In addition, in a case where the composition portion715combines the bird's eye view image data stored at the bird's eye view image storage portion706with the bird's eye view image data obtained after the projective transformation is performed by the conversion portion714, the composition portion715performs a rotation processing with the turning angle81on the bird's eye view image data stored at the bird's eye view image storage portion706when the vehicle1turns at the turning angle θ1. Then, the composition portion715performs the composition with the bird's eye view image data obtained after the projective transformation is performed by the conversion portion714. Accordingly, the bird's eye view image data conforming to the turning of the vehicle1may be displayed.

The output portion705outputs, together with the bird's eye view image data stored at the bird's eye view image storage portion706, the image data where various pieces of information are composited to the display device8.FIG. 19is a diagram illustrating an example of screen information output by the output portion705. In the example of the screen information illustrated inFIG. 19, captured image data1302capturing the travelling direction of the vehicle1by the imaging portion16a, captured image data1303around the front left wheel of the vehicle1captured by the imaging portion16b, and captured image data1304around the front right wheel of the vehicle1captured by the imaging portion16dare displayed in addition to bird's eye view image data1301. Further, in a display region1305, the pitch angle and the roll angle of the vehicle1are displayed as recognizable information. That is, while the roll angle is indicated by an inclination of an icon, the pitch angle is indicated by a distance between a center line passing through the icon and a horizontal line. In the present embodiment, the roll angle and the pitch angle are recognizable in the aforementioned method, however, the display method is not limited to the above and the other display method is acceptable.

In addition, the captured image data1302where the travelling direction of the vehicle1is captured by the imaging portion16aserves as the captured image data obtained after the rotation control is performed. Accordingly, the driver may recognize a height relation within the captured image data.

Next, the display processing on the display device8in the surroundings monitoring portion700according to the present embodiment is explained.FIG. 20is a flowchart illustrating procedures of the aforementioned processing in the surroundings monitoring portion700according to the present embodiment.

First, the acquisition portion701acquires the captured image data from the imaging portions16(step S1401). Next, the acquisition portion701acquires the acceleration data from the acceleration sensors26(step S1402).

Then, the angle calculation portion702calculates the roll angle and the pitch angle of the vehicle1from the acceleration data (step S1403). The filtering control portion703performs filtering by low-pass filter relative to the calculated roll angle and the calculated pitch angle.

Then, the rotation control portion711performs the rotation control relative to the captured image data depending on the roll angle (step S1404).

Next, the bird's eye view image generation portion712generates the bird's eye view image data where a predetermined area in the travelling direction of the vehicle1which is present around the vehicle1is illustrated in an overhead view (step S1405).

The moving amount calculation portion713extracts the image data of a predetermined display range (area) from the generated bird's eye view image data (step S1406). In addition, the moving amount calculation portion713holds the image data extracted from the similar range from the bird's eye view image data in the past (for example, in a case where the previous moving amount is determined to reach or exceed a predetermined threshold value).

Then, the moving amount calculation portion713calculates the moving amount of the vehicle1based on the image data of the predetermined display range (area) extracted from the bird's eye view image data (step S1407).

Then, the image processing portion704determines whether or not the calculated moving amount is equal to or greater than the predetermined threshold value (step S1408). The threshold value is specified to be 10 cm, for example, however, the threshold value may be specified appropriately depending on an implementation mode

In a case where the image processing portion704determines that the moving amount is equal to or greater than the threshold value (Yes in step S1408), the bird's eye view image generation portion712generates the bird's eye view image data from the presently captured image data serving as the captured image data before the rotation control is performed as in step S1404(step S1409).

Afterwards, the conversion portion714performs the projective transformation on the bird's eye view image data depending on the present roll angle and pitch angle of the vehicle1(step S1410). The torsion of the bird's eye view image data generated by either one of the roll angle and the pitch angle is corrected by the projective transformation including the trapezoidal correction. When the roll angle is generated at the vehicle1, the vehicle1is inclined with the axis of the wheel3, for example, instead of being inclined with reference to a center of gravity. Therefore, a displacement occurs in the left-right direction. Thus, the conversion portion714according to the embodiment performs an offset correction on the displacement in the left-right direction. In the same manner, the offset correction in the front-rear direction is performed in a case where the pitch angle is generated.

Next, the composition portion715combines the present bird's eye view image data after the projective transformation is performed with the bird's eye view image data stored at the bird's eye view image storage portion (step S1411). The composite portion715performs the rotation control on the bird's eye view image data stored at the bird's eye view image storage portion706so as to conform to the turning angle θ1obtained before the composition in a case where the vehicle1turns at the turning angle θ1.

Then, the output portion705cuts out the bird's eye view image data displayed at the display device8from the bird's eye view image data stored at the bird's eye view image storage portion706(step S1412). Thereafter, the output portion705adds various pieces of information to the bird's eye view image data that is cut out and outputs the data to the display device8(step S1413).

Meanwhile, in a case where the image processing portion704determines that the moving amount is smaller than the predetermined threshold value (No in step S1408), the image processing portion704continues outputting the bird's eye view image data and the like already displayed at the display device8(step S1414).

Third Embodiment

In the second embodiment, the embodiment where the bird's eye view image data is displayed for confirming the state of the vehicle1is explained. Nevertheless, the embodiment is not limited to the display of only the bird's eye view image data and various pieces of information for confirming present state may be added to the bird's eye view image data. Thus, in the third embodiment, an example where various pieces of information are added to the bird's eye view image data is explained.

First, a construction of a surroundings monitoring portion1700according to the third embodiment is explained.FIG. 21is a block diagram illustrating the construction of the surroundings monitoring portion1700realized within the surroundings monitoring ECU14according to the present embodiment.

The surroundings monitoring portion1700illustrated inFIG. 21is different from the surroundings monitoring portion700in the first embodiment in a point where the acquisition portion701is changed to an acquisition portion1701performing a different processing from the acquisition portion701and the image processing portion704is changed to an image processing portion1702performing a different processing from the image processing portion704.

The acquisition portion1701acquires the captured image data and the acceleration data, in the same way as the second embodiment, and also acquires a suspension detection result indicating a depression degree of a suspension of the front wheels3F from a stroke sensor (not illustrated) and a detection result of the steering angle of each of the front wheels3F and the rear wheels3R from the steering angle sensor19. In the present embodiment, the acquired suspension detection result and steering angle detection result are output to the image processing portion1702.

The image processing portion1702is different from the image processing portion704in the second embodiment in a point where a tire outline calculation portion1711and a locus calculation portion1712are added and the composition portion715in the second embodiment is changed to a composition portion1713performing a different processing from the processing performed by the composition portion715.

The tire outline calculation portion1711calculates an outline of a tire that should be superimposed on the bird's eye view image data based on the suspension detection result and the detection result of the steering angle acquired by the acquisition portion1701. For example, in a case where the camera is placed on a basis of the bird's eye view, the front wheels3F and the rear wheels3R are shown largely as approaching the camera when the suspension is depressed, and are shown small when the suspension is extended. Thus, in the present embodiment, in order to display the tire outline of the vehicle1on the bird's eye view image data, the tire outline calculation portion1711calculates the tire outline configuration (size and angle of each of the front wheels3F) that should be superimposed on the bird's eye view image data based on the suspension detection result and the steering angle.

The locus calculation portion1712calculates an estimated moving locus in the travelling direction of the vehicle1based on the steering angle of the front wheels3F and the steering angle of the rear wheels3R. The locus calculation portion1712according to the present embodiment calculates the estimated moving locus that should be added to the present captured image data as the estimated moving locus of the front wheels3F and calculates the estimated moving locus that should be added to the bird's eye view image data as the estimated moving locus of the rear wheels3R. The estimated moving loca that are calculated are added to the captured image data and the bird's eye view image data and then output by the output705.

In addition, the composition portion1713combines the bird's eye view image data stored at the bird's eye view image storage portion706with the bird's eye view image data obtained after the projective transformation is performed by the conversion portion714in the same processing as the second embodiment, and thereafter adds a mark by which the steering angle and the size of each of the front wheels3F are recognizable to a position where each of the front wheels3is estimated to presently exist on the bird's eye view image data obtained after the composition is performed.

Each time the composition portion1713performs the composition, the mark is added to the position where each of the front wheels3R exists so that the moving locus of the front wheel3R is displayed at the bird's eye view image data. Then, the output portion705outputs the screen information at the display device8based on the bird's eye view image data composited by the composition portion1713.

FIG. 22is a diagram illustrating an example of the screen information output by the output portion705according to the third embodiment. In the example illustrated inFIG. 22, in addition to bird's eye view image data1601, captured image data1602capturing the travelling direction of the vehicle1by the imaging portion16ais shown, for example.

In the captured image data1602capturing the travelling direction, estimated moving loca1611,1612of the front wheels3F (display element indicating a planned course) calculated by the locus calculation portion1702are indicated.

Meanwhile, in the bird's eye view image data1701, moving loca1621,1622of the vehicle1generated because of the marks which are continuously added on a basis of the outlines of the front wheels3F are indicated. Based on the size of each of the marks included in the moving loca1621,1622, the driver may recognize protrusion and recess on the road surface. That is, at a portion where the large mark is added, the suspension is largely depressed. In other words, an obstacle such as a stone, for example, is highly possibly present. Thus, the driver may drive, while confirming the aforementioned marks, to operate so that the rear wheels3F and a differential (not illustrated) are inhibited from collision.

In addition, in the bird's eye view image data1602, estimated moving loca1631,1632of the rear wheels3F calculated by the locus calculation portion1703are indicated. Because the driver can recognize the estimated moving loca1631,1632, the driver may restrain the rear wheels3R from being collided with an obstacle by driving the vehicle so that the rear wheels3R overlap the moving loca of the front wheels3F which have not been collided with the obstacle.

Here, in a case where the bird's eye view image generation portion1702adds the marks indicating the positions of the front wheels3F to the bird's eye view image data in the past, the bird's eye view image generation portion1702may differentiate colors or shapes based on information other than the suspension detection results. For example, the color or shape at the positions where the front wheels3F slip may be differentiated. As a result, safety when the driver drives the vehicle may improve.

The surroundings monitoring portion in the aforementioned embodiments include the aforementioned construction so as to easily recognize the surrounding state including the ground below the vehicle1. Accordingly, a load of driving is reduced to thereby enhance safety.

The second embodiment or the third embodiment is an example of a vehicle control apparatus or program according to either of the followings [1]-[8].

[1]A vehicle control apparatus including:an acquisition portion acquiring captured image data output from an imaging portion that is provided at a vehicle and that images a surrounding of the vehicle and vehicle state data output from a vehicle state detection portion provided at the vehicle;a control portion performing a rotation control on the captured image data based on an inclination in a left-right direction of the vehicle relative to a horizontal direction which serves as a direction included in a horizontal plane orthogonal to a direction of gravity, the inclination in the left-right direction of the vehicle being calculated from the vehicle state data; anda generation portion generating bird's eye view image data indicating a ground in a surrounding of the vehicle in an overhead view based on the captured image data on which the rotation control is performed by the control portion.

[2]The vehicle control apparatus according to [1], whereinthe generation portion generates first bird's eye view image data based on first captured image data on which the rotation control is performed by the rotation control portion and generates second bird's eye view image data based on second captured image data which is captured by the imaging portion after the first captured image data is captured and then the vehicle moves and on which the rotation control is performed by the rotation control portion,the vehicle control apparatus further including a composition portion combines the first bird's eye view image data and the second bird's eye view image data.

[3]The vehicle control apparatus according to [1] or [2], whereinthe acquisition portion acquires an acceleration of the vehicle as the vehicle state data from the vehicle state detection portion,the rotation control portion further performs the rotation control on the captured image data depending on a roll angle indicating an inclination around a front-rear axis of the vehicle obtained from the acceleration data with an origin at a position coordinate within a display region of the captured image data, the position coordinate corresponding to a center of a lens used for imaging by the imaging portion.

[4]The vehicle control apparatus according to any one of [1] through [3], further including a conversion portion performing a projective transformation which is specified on a basis of a second roll angle relative to the second captured image data which is cut out at a display range specified on a basis of the second roll angle obtained from a second acceleration data which is acquired when the second captured image data is captured, whereinthe composition portion combines the first bird's eye view image data and the second bird's eye view image data which is converted by the conversion portion.

[5]The vehicle control apparatus according to any one of [1] through [4], wherein the rotation control portion performs the rotation control on the captured image data depending on a difference between a first roll angle obtained from a first acceleration data that is acquired when the first captured image data is captured and a second roll angle obtained from the second acceleration data that is acquired when the second captured image data is captured.

[6]The vehicle control apparatus according to any one of [1] through [5], wherein the composition portion combines the first bird's eye view image data captured before the vehicle moves and including a ground below the vehicle and the second bird's eye view image data.

[7]The vehicle control apparatus according to any one of [1] through [6], further including an output portion outputting information that represents either the roll angle or a pitch angle indicating an inclination around a left-right axis of the vehicle and bird's eye view image data composited by the composition portion.

[8]A program configured to cause a computer to execute;an acquisition step acquiring captured image data output from an imaging portion that is provided at a vehicle and that images a surrounding of the vehicle and vehicle state data output from a vehicle state detection portion provided at the vehicle;a rotation control step performing a rotation control on the captured image data based on an inclination in a left-right direction of the vehicle relative to a horizontal direction which serves as a direction included in a horizontal plane orthogonal to a direction of gravity, the inclination in the left-right direction of the vehicle being calculated from the vehicle state data; anda generation step generating bird's eye view image data indicating a ground in a surrounding of the vehicle in an overhead view based on the captured image data on which the rotation control is performed by the rotation control step.

The embodiments of the present invention have been explained, however, the present embodiments are proposed as examples and not intended to limit the scope of the invention. The above new embodiments may be performed in other various modes. Without departing from the spirit of the invention, various omissions, replacements and changes may be made. The embodiments and alternatives thereof are included within the spirit and scope of the invention and included in the invention described in the scope of claims and equivalents thereof.

EXPLANATION OF REFERENCE NUMERALS

500: surroundings monitoring portion,501: acquisition portion,502: angle calculation portion,503: filtering control portion,504: image processing portion,505: output portion,521: rotation control portion,522: reduction/enlargement control portion,523: movement control portion,524: composition portion,700: surroundings monitoring portion,701: acquisition portion,702: angle calculation portion,703: filtering control portion,704: image processing portion,705: output portion,706: bird's eye view image storage portion,711: rotation control portion,712: bird's eye view image generation portion,713: moving amount calculation portion,714: conversion portion,715: composition portion