Vehicle periphery monitoring device and vehicle periphery image display method

From a plurality of imaging units (110) which capture images of a vehicle periphery and from an image taken by any one of the plurality of imaging units (any one of 111, 121, 131, 141), a value corresponding to brightness of the image is calculated. Based on the calculated value, brightness of images obtained by the plurality of imaging units (110) is corrected by a brightness correction unit (120). After a predetermined coordinate transformation is performed on the brightness corrected images, the images are combined into a single image, and the single image is displayed together with the image taken by the one imaging unit (any one of 111, 121, 131, 141).

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the U.S. National Phase under 35. U.S.C. §371 of International Application PCT/JP2010/062571, filed Jul. 27, 2010, which claims priority to Japanese Patent Application No. 2009-176308, filed Jul. 29, 2009. The International Application was published under PCT Article 21(2) in a language other than English.

TECHNICAL FIELD

The present invention relates to a vehicle periphery monitoring device and a vehicle periphery image display method, and specifically relates to a method of obtaining a natural composite image by correcting differences in brightness among multiple images taken in directions different from each other.

BACKGROUND ART

In recent years, systems have grown popular which include a camera mounted on a vehicle and provide a driver with an image of a vehicle periphery which can easily be a blind spot.

In particular, recently, systems such as the followings have been put to practical use: a system which provides an overhead image viewed from right above a vehicle and obtained through coordinate transformation of an image taken by a camera, and a system which has multiple cameras mounted on a vehicle, and provides a 360° image of a vehicle periphery by combining overhead images obtained by the coordinate transformation.

Such systems which combine images taken by multiple cameras into a single image have a problem that a composite image brings a feeling of strangeness because brightness is discontinuous at seams of images due to a type difference among cameras which constitute the system. Even with cameras of the same type, this problem also happens because the cameras differently work in automatic exposure correction, or a light amount decreases in proportion to a distance from the center of each camera.

To solve such a problem, techniques have been conventionally proposed which correct brightness of images taken by respective cameras so that image areas taken in common by adjacent cameras can have the same average brightness, and display the brightness-corrected image. (Patent Document 1, Patent Document 2)

PRIOR ART DOCUMENT

Patent Document

Patent Document 2: Japanese Patent Application Publication No. 2007-141098

SUMMARY OF THE INVENTION

Problems to be Solved by the Invention

In these conventional proposals, brightness is corrected as a prerequisite to provision of an image obtained through coordinate transformation and combination of multiple images.

For this reason, if a vehicle periphery monitoring device is implemented which has a function of enabling checking conditions around a vehicle while checking an image behind the vehicle, by simultaneously displaying the image behind the vehicle and an overhead image of a vehicle periphery side-by-side, for example, or a function of enabling checking conditions around a vehicle while checking an image in front of the vehicle by displaying the image in front of the vehicle and an overhead image of a vehicle periphery side-by-side, the vehicle periphery monitoring device corrects brightness of a composite image and also inevitably corrects brightness of the other image, that is, the image behind or in front of the vehicle at the same time.

For example, if brightness frequently changes in some of images taken by multiple cameras, there arises a problem that an image behind or in front of the vehicle, which is displayed together with a composite image of a vehicle periphery, is also corrected in brightness by using a brightness correction value calculated to correct brightness of a single composite image obtained by combining the multiple images, and therefore becomes difficult to see.

The present invention has been made in light of the above circumstances, and an object of the present invention is to provide a vehicle periphery monitoring device and a vehicle periphery image display method which are capable of correcting brightness of a composite image without causing mutual influence of two or more types of images displayed side-by-side and including the composite image.

Means for Solving the Problem

The present invention can provide easy-to-see images even in an environment in which brightness strongly changes, by correcting brightness of a composite overhead image of the vehicle periphery on the basis a value corresponding to brightness of an image behind a vehicle, when the composite overhead image of a vehicle periphery and the image behind the vehicle are displayed side-by-side.

In other words, a vehicle periphery monitoring device according to the present invention includes multiple imaging units which are mounted on a vehicle and capture images of a vehicle periphery, an image brightness calculation unit which calculates a value corresponding to brightness of an image taken by any one of the multiple imaging units from the image taken by the one of the imaging units, a brightness correction unit which corrects brightness of the images obtained by the multiple imaging units on the basis of the value calculated by the image brightness calculation unit, an image transformation unit which performs a predetermined coordinate transformation process on the image with the brightness thereof corrected by the brightness correction unit, an image combining unit which combines multiple images after the coordinate transformation process into a single image, and an image display unit which simultaneously displays the composite image obtained by the image combining unit and the image taken by the one of the imaging units.

With such configured vehicle periphery monitoring device according to the present invention, the image brightness calculation unit calculates a value corresponding to brightness of an image taken by one of the multiple imaging units, and the brightness correction unit corrects brightness of the images taken by the multiple imaging units on the basis of the calculated value.

The brightness corrected images are coordinate transformed by the image transformation unit and further combined into a single image by the image combining unit. Since the single composite image and an image taken by one of the multiple imaging units are displayed side-by-side, only brightness of the composite image can be corrected without affecting brightness of the image taken by the one of the multiple imaging units.

In addition, it is desirable that the vehicle periphery monitoring device according to the present invention includes a traveling direction judgment unit which judges a traveling direction of a vehicle and an imaging-unit-selecting unit which selects one imaging unit from the multiple imaging units on the basis of the judgment result of the traveling direction judgment unit, in which the image brightness calculation unit calculates a value corresponding to brightness of an image taken by the one imaging unit selected by the imaging-unit-selecting unit, and the image display unit simultaneously displays the composite image obtained by the image combining unit and the image taken by the one imaging unit selected by the imaging-unit-selecting unit.

With such configured vehicle periphery monitoring device according to the present invention, one imaging unit is selected by the imaging-unit-selecting unit on the basis of the result judged by the traveling direction judgment unit, and a value corresponding to brightness of an image taken by the selected imaging unit. The brightness correction unit corrects brightness of images taken by multiple imaging units on the basis of the calculated value. The brightness corrected images are coordinate transformed by the image transformation unit and further combined into a single image by the image combining unit. Since the single composite image and the image taken by the one of the multiple imaging units are displayed side-by-side, information of a traveling direction can be presented to a driver with the result of brightness correction not affecting brightness of an image in the travelling direction of a vehicle, to which most attention should be paid during driving, thus being effective in conforming safety around the vehicle.

In addition, it is preferable that the vehicle periphery monitoring device according to the present invention is such that the imaging-unit-selecting unit selects a first imaging unit from multiple imaging units which capture images in three or more directions and which are mounted so that the ranges captured by at least two imaging units of the multiple imaging units partly overlap each other; the image brightness calculation unit calculates a value corresponding to brightness of a first overlapping imaging range included in an image taken by the selected first imaging unit and in an image taken by a second imaging unit which shares the first overlapping imaging range with the selected first imaging unit, or calculates a value corresponding to brightness of a neighborhood of the first overlapping imaging range; the brightness correction unit corrects brightness of the image taken by the second imaging unit on the basis of the calculated value; the image brightness calculation unit further calculates, based on the brightness corrected result, a new value corresponding to brightness of a second overlapping imaging range or corresponding to brightness of a neighborhood of the second overlapping imaging range, from the brightness corrected image and an image taken by a third imaging unit and sharing the second overlapping imaging range with the brightness corrected image; the brightness correction unit further corrects brightness of the image taken by the third imaging unit on the basis of the calculated new value; and the image brightness calculation unit and the brightness correction unit sequentially repeat the calculation of brightness and the correction of brightness in the same manner thereafter.

With such configured vehicle periphery monitoring device according to the present invention, since brightness of an overlapping imaging range of two images with mutually overlapping imaging areas or brightness of a neighborhood of the overlapping imaging range is corrected as well as the imaging-unit-selecting unit selects an imaging unit depending on a vehicle status, brightness of a different image captured for an area partly overlapping with the reference image can be recursively corrected based on brightness of a reference image. Thus, in the vehicle periphery monitoring device which combines a number of images into a single image and displays the image, a feeling of unevenness in brightness at the seams of images can be substantially reduced, thereby enabling provision of easy-to-see images to drivers.

Furthermore, since image brightness correction can be performed by adjustment of brightness when images taken by imaging units are decoded, the brightness correction can be performed in a short period of time.

In addition, it is preferable that a vehicle periphery monitoring device according to the present invention is such that the image brightness calculation unit calculates a value corresponding to brightness of a first overlapping imaging range included in an image taken by a first imaging unit of multiple imaging units which capture the images in three or more directions and which are mounted so that the ranges captured by at least two imaging units of the multiple imaging units partly overlap each other and included in an image taken by a second imaging unit sharing the first overlapping imaging unit with the first imaging unit, or calculates a value corresponding to brightness of a neighborhood of the first overlapping imaging range; the brightness correction unit corrects brightness of the image taken by the second imaging unit on the basis of the calculated value; the image brightness calculation unit further calculates, based on the brightness corrected result, a new value corresponding to brightness of a second overlapping imaging range or corresponding to brightness of a neighborhood of the second overlapping imaging unit, from the brightness corrected image and an image taken by a third imaging unit sharing the second overlapping imaging range with the brightness corrected image; the brightness correction unit further corrects brightness of the image taken by the third imaging unit on the basis of the calculated new value; and the image brightness calculation unit and the brightness correction unit sequentially repeat the calculation of brightness and the correction of the brightness in the same manner thereafter.

With such configured vehicle periphery monitoring device according to the present invention, in particular, in the vehicle periphery monitoring devices which combine a number of images into a single image and display the image, since image brightness can be sequentially corrected on the basis of preselected image brightness, a feeling of unevenness in brightness at the seams of images can be substantially reduced and easy-to-see images can be provided to drivers.

Furthermore, since image brightness correction can be performed by adjustment of brightness when images taken by imaging units are decoded, the brightness correction can be performed in a short period of time.

In addition, a vehicle periphery image display method calculates a value corresponding to brightness of one image from multiple images captured for a vehicle periphery, corrects brightness of the multiple images on the basis of the calculated value, performs a predetermined coordinate transformation process on the brightness corrected images, combines the multiple coordinate-transformed images into a single image, and displays the composite image and the one image simultaneously.

With such configured vehicle periphery image display method according to the present invention, brightness of only the composite image can be corrected with no influence on brightness of the one image, because of the operation described above.

In addition, it is preferable that the vehicle periphery image display method judges a traveling direction of a vehicle, selects one image from multiple images captured for a vehicle periphery on the basis of the judgment result of the traveling direction, calculates a value corresponding to brightness of the one selected image from the one image, corrects brightness of the multiple images on the basis of the calculated value, performs a predetermined coordinate transformation process on the brightness corrected images, combines the multiple coordinate-transformed images into a single image, and displays the composite image and the one selected image simultaneously.

With such configured vehicle periphery image display method according to the present invention, the operation described above can provide drivers information useful for safety confirmation around a vehicle, since the result of image brightness correction does not affect brightness of an image in a traveling direction of the vehicle to which most attention should be paid during driving.

In addition, it is preferable that a vehicle periphery image display method according to the present invention selects a first image from multiple images captured for three or more directions and set so that imaging ranges of at least two of the multiple images partly overlap each other, calculates a value corresponding to brightness of a first overlapping imaging range included in the selected first image and in a second image sharing the first overlapping imaging area with the selected first image, or a value corresponding to brightness of a neighborhood of the first overlapping imaging range, corrects brightness of the second image on the basis of the calculated value, calculates, based on the brightness corrected result, a new value corresponding to brightness of a second overlapping imaging range or corresponding to brightness of a neighborhood of the second overlapping imaging area from the brightness corrected image and a third image sharing the second overlapping imaging range with the brightness corrected image, corrects brightness of the third image on the basis of the calculated new value, and sequentially repeats the calculation of brightness and the correction of brightness in the same manner thereafter.

With such configured vehicle periphery image display method according to the present invention, the operation described above can substantially reduce a feeling of unevenness in brightness at the seams of images when multiple images are combined into a single image and provide drivers with easy-to-see images.

Furthermore, image brightness is corrected by adjustment of brightness when captured images are decoded, the brightness correction can be performed in a short period of time.

In addition, it is preferable that a vehicle periphery image display method according to the present invention calculates a value corresponding to brightness of a first overlapping imaging range included in a first image of multiple images captured for three or more directions and set so that imaging ranges of at least two images of the multiple images partly overlap and included in a second image in which the first overlapping imaging area that overlaps the first image is captured, or a value corresponding to brightness of a neighborhood of the first overlapping imaging range, corrects brightness of the second image on the basis of the calculated value, calculates, based on the brightness corrected result, a new value corresponding to brightness of a second overlapping imaging area or corresponding to brightness of a neighborhood of the second overlapping imaging range from the brightness corrected image and a third image sharing the second overlapping imaging range with the brightness corrected image, corrects brightness of the third image on the basis of the calculated new value, and sequentially repeats the calculation of brightness and the correction of brightness in the same manner thereafter.

With such configured vehicle periphery image display method according to the present invention, since brightness of images can sequentially be corrected with brightness of a preselected image as a reference, a feeling of unevenness at the seams of images can be substantially reduced and easy-to-see images can be provided to drivers.

Furthermore, since image brightness correction can be performed by adjustment of brightness when images taken by imaging units are decoded, the brightness correction can be performed in a short period of time.

Effects of the Invention

With a vehicle periphery monitoring device and a vehicle periphery image display method, when a composite overhead image of a vehicle periphery and an image behind a vehicle are displayed side-by-side, there is the effect of being able to provide easy-to-see images even in an environment in which brightness sharply changes, by correcting brightness of the composite overhead image of the vehicle periphery on the basis of a value corresponding to brightness of the image behind the vehicle.

MODE FOR CARRYING OUT THE INVENTION

Embodiments of a vehicle periphery monitoring device of the present invention will be described hereinafter with reference to the drawings.

FIG. 1is a block diagram showing a schematic configuration of a first embodiment of a vehicle periphery monitoring device according to the present invention.

FIG. 2is a block diagram showing a detailed configuration of an imaging unit110, a brightness correction unit120, a coordinate transformation and image combining unit200, an image brightness calculation unit300, and an image display unit400in the vehicle periphery monitoring device ofFIG. 1.

FIG. 3is a flow chart showing a series of processing steps in the vehicle periphery monitoring device shown inFIG. 2.

As shown in the block diagram ofFIG. 1, the vehicle periphery monitoring device according to the embodiment includes the imaging unit110such as N CCD cameras or C-MOS cameras which observe different directions respectively, the brightness correction unit120which corrects brightness of images inputted by the imaging units, the coordinate transformation and image combining unit200which performs coordinate transformation on the images outputted from the brightness correction unit120and further combines them into a single image, the image brightness calculation unit300which calculates brightness of captured images, the image display unit400including a monitor, etc., which shows the result of coordinate transformation and image combination performed, and a start/end instruction detection unit500which detects an instruction to start or end the vehicle periphery monitoring device. In addition, this vehicle periphery monitoring device is mounted on a vehicle not shown inFIG. 1.

The embodiment is described taking, as an example, a case where the imaging unit110includes four cameras. That is to say, the imaging unit110and the brightness correction unit120include a front camera111for observing the forward direction of a vehicle and a first decoder and brightness correction unit112and a first A/D converter113connected thereto, a left side camera121for observing the left direction of the vehicle and a second decoder and brightness correction unit122and a second A/D converter123connected thereto, a right side camera131for observing the right direction of the vehicle and a third decoder and brightness correction unit132and a third A/D converter133connected thereto, and a rear camera141for observing the backward direction of the vehicle and a fourth decoder and brightness correction unit142and a fourth A/D converter143connected thereto.

In addition, the coordinate transformation and image combining unit200includes a coordinate transformation unit201, a coordinate transformation table202, and an image combining unit203.

The image brightness calculation unit300includes a first brightness calculation unit301connected to the first A/D converter113, a second brightness calculation unit302connected to the second A/D converter123, a third brightness calculation unit303connected to the third A/D converter133, and a fourth brightness calculation unit304connected to the fourth A/D converter143.

Furthermore, the image display unit400includes a D/A converter401, an encoder402, and a monitor403.

As shown inFIG. 4, the vehicle periphery monitoring device in the embodiment is configured to display, as a single image, one composite image700in which 360° information pieces on a vehicle periphery are combined and an image630captured for a backward direction of the vehicle, on the monitor403.

Now, the composite image700is a single image obtained by combining an image710looking down at an image600taken by the front camera111shown inFIG. 6Bfrom an overhead virtual viewing point V, an image720looking down at an image610taken by the left side camera121shown inFIG. 6Cfrom the overhead virtual viewing point V, an image730looking down at an image620taken by the right side camera181shown inFIG. 6Dfrom the overhead virtual viewing point V, an image740looking down at an image630taken by the rear camera141as shown inFIG. 4from the overhead virtual viewing point V, and an image750looking down at the vehicle having this device attached from the overhead virtual viewing point V.

In addition, although the embodiment is configured to observe the four directions of a vehicle, the number of directions to observe is not limited to this, and the embodiment may be configured to observe more number of directions. Even in such a case, the present invention can still be implemented with the operation similar to the embodiment.

The operation of the vehicle periphery monitoring device according to the embodiment will be described hereinafter with reference to the flow chart ofFIG. 3.

The front camera111, the left side camera121, the right side camera131, and the rear camera141are attached to the vehicle in a layout in which imaging ranges of adjacent cameras (the front camera111and the left side camera121, the front camera111and the right side camera131, the left side camera121and the rear camera141, and the right side camera131and the rear camera141) partly overlap.

The start/end instruction detection unit500detects that a shift position is in a retreated position (S2ofFIG. 3), the coordinate transformation unit201generates a trigger signal. The trigger signal is inputted to the first decoder and brightness correction unit112, the first A/D converter113, the second decoder and brightness correction unit122, the second A/D converter123, the third decoder and brightness correction unit132, the third A/D converter133, the fourth decoder and brightness correction unit142, and the fourth A/D converter143.

When each camera receives the trigger signal, the front camera111, the left side camera121, the right side camera131, and the rear camera141simultaneously input an image (S3ofFIG. 3).

The image inputted from the front camera111is converted from composite signals into component signals by the first decoder and brightness correction unit112(S4ofFIG. 3). Furthermore, a luminance signal of the converted component signals is converted into a digital signal by the first A/D converter113and the image600is generated (S6ofFIG. 3). Similarly, the image inputted from the left side camera121is converted from the composite signals into the component signals by the second decoder and the brightness correction unit122(S4ofFIG. 3). Furthermore, the luminance signal of the converted component signals is converted into a digital signal by the second A/D converter123and the image610is generated (S6ofFIG. 3). The image inputted from the right side camera131is converted from the composite signals into the component signals by the third decoder and the brightness correction unit132(S4ofFIG. 3). Furthermore, the luminance signal of the converted composite signals is converted into a digital signal by the third A/D converter133, and the image620is generated (S6ofFIG. 3). The image inputted from the rear camera141is converted from the composite signals into the component signals by the fourth decoder and brightness correction unit142(S4ofFIG. 3). Furthermore, the luminance signal of the converted component signal is converted into a digital signal by the fourth A/D converter143, and the image630is generated (S6ofFIG. 3).

In addition, now, brightness of the captured image is corrected (S5ofFIG. 3) at the same time when the signal is converted from the composite signal into the component signal. However, since a correction amount for correcting brightness of the image has not been calculated, the brightness correction is not performed when an initial image after starting of the vehicle periphery monitoring device is inputted.

Then, the fourth brightness calculation unit304calculates an average value of a pixel value of predetermined areas in the image630taken by the rear camera141(S7ofFIG. 3). Here, as shown inFIG. 5, an area consisting of m pixels in a horizontal direction and n pixels in a vertical direction is set to a prescribed position (x0, y0) which has been determined in advance, and an average value I ave of all pixel values in that area is calculated.

Then, the first brightness calculation unit301calculates an average value of pixel values in a predetermined area in the image600taken by the front camera111(S8ofFIG. 3). Here, as shown inFIG. 6B, an area601and an area602of a predetermined size are set at prescribed positions which have been determined in advance, and an average value I 1 ave of all the pixel values in the area610and the area620is calculated.

Now, as shown inFIG. 6A, the area601is an area to be converted into a rectangular area711having a size of width A and length B at a position where the imaging ranges of the front camera111and the left side camera121overlap or in the vicinity thereof, when the image600is converted into the image710looking down from the overhead virtual viewing point V. Furthermore, the area602is an area to be converted into a rectangular area712having a size of width A and length B at a position where the imaging ranges of the front camera111and the right side camera131overlap or in the vicinity thereof when the image600is converted into the image710looking down from the overhead virtual viewing point V.

In addition, the positions of the area601and the area602can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the first brightness calculation unit301.

Next, the second brightness calculation unit302calculates an average value of pixel values in predetermined areas in the image610taken by the left side camera121(S8ofFIG. 3). Here, as shown inFIG. 6C, an area611and an area612of a predetermined size are set at prescribed positions in the image610which have been determined in advance, and an average value I 2ave of all the pixel values in the area611and the area612is calculated.

Now, as shown inFIG. 6A, the area611is an area to be converted into a rectangular area721having a size of width A and length B at a position where the imaging ranges of the front camera111and the left side camera121overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V. Furthermore, as shown inFIG. 6A, the area612is an area to be converted into a rectangular area722having a size of width A and length B at a position where the imaging ranges of the left side camera121and the rear camera141overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V.

In addition, the positions of the area611and the area612can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the second brightness calculation unit302.

Further, the third brightness calculation unit303calculates an average value of pixel values in predetermined areas in the image620taken by the right side camera131(S8ofFIG. 3). Here, as shown inFIG. 6D, an area621and an area622of a predetermined size are set at prescribed positions in the image620which have been determined in advance, and an average value I 3 ave of all the pixel values in the area621and the area622is calculated.

Now, as shown inFIG. 6A, the area621is an area to be converted into a rectangular area731having a size of width A and length B at a position where the imaging ranges of the front camera111and the right side camera131overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V. Furthermore, as shown inFIG. 6A, the area622is an area to be converted into a rectangular area732having a size of width A and length B at a position where the imaging ranges of the right side camera131and the rear camera141overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V.

In addition, the positions of the area621and the area622can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the third brightness calculation unit303.

Thus calculated average value I ave of the pixel values in the predetermined area of the image630and the average value I 1ave of the pixel values of the predetermined areas of the image600are transmitted to the first decoder and brightness correction unit112and stored therein. Similarly, the average value I ave of the pixel values in the predetermined area in the image630and the average value I 2ave of the pixel values of the predetermined areas of the image610are transmitted to the second decoder and the brightness correction unit122and stored therein, and the average value I ave of the pixel values in the predetermined area of the image630and the average value I 3ave of the pixel values in the predetermined area of the image620are transmitted to the third data and brightness correction unit132and stored therein.

The image600, the image610, the image620, and the image630which have been generated in S6ofFIG. 3are coordinate transformed by the coordinate transformation unit201and converted into an image looking down from the overhead virtual viewing point V (S9ofFIG. 3). The coordinate transformation process is calculated based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc. However, in general, the coordinate transformation process is performed by creating the coordinate transformation table202in advance based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc., and then replacing coordinates of the inputted images on the basis of this coordinate transformation table202.

Through the coordinate transformation, the image600taken by the front camera111is converted into the image710, the image610taken by the left side camera121is converted into the image720, the image620taken by the right side camera131is converted into the image730, and the image630taken by the rear camera141is converted into the image740. The converted results are combined into a single image by the image combining unit203, and further a virtual image750looking down at the own vehicle from the overhead virtual viewing point V is combined therewith and a composite image700is generated. Furthermore, the composite image700is combined with the image630taken by the rear camera141and a single image shown inFIG. 4is created (S10ofFIG. 3).

The composite image is reconstructed to component signals by the D/A converter401(S11ofFIG. 3), further converted into composite signals by the encoder402(S12ofFIG. 3), and displayed to the monitor403(S13ofFIG. 3).

The start/end instruction detection unit500detects whether or not a shift position is in a retreated position (S14ofFIG. 3). If the shift position is in any position other than the retreated position, the display on the monitor403is cancelled (S15ofFIG. 3).

If it is confirmed that the shift position is in the retreated position, the image is inputted again in S3ofFIG. 3.

The image inputted from the front camera111is converted from composite signals into component signals by the first decoder and brightness correction unit112(S4ofFIG. 3), and then, brightness is corrected based on the average value I ave of the pixel values in the predetermined area in the image630and the average value I 1ave of the pixel values in the predetermined areas in the image600(S5ofFIG. 3).

Specifically, if a currently set brightness value in the first decoder and brightness correction unit112is B1, and a newly set brightness value by brightness correction is B1′, correction of brightness is performed by changing a brightness value from B1 to B1′ based on (Formula 1).
B1′=B1+(Iave−I1ave)×w(Formula 1)

where w represents a weighted value of brightness correction (0≦w≦1). When brightness of an image sharply changes, this is a value which is provided to avoid a brightness corrected value sharply changing accordingly. In addition, w is set to a prescribed value which has been determined in advance.

The brightness correction similar to this is also performed on the image610inputted from the left side camera121and the image620inputted from the right side camera131.

That is to say, if a currently set brightness value in the second decoder and brightness correction unit122is B2, a newly set brightness value by the brightness correction is B2′, a currently set brightness value in the third decoder and brightness correction unit132is B3, and a newly set brightness value by the brightness correction is B3′, brightness of the image610is corrected by changing the brightness value from B2 to B2′ based on the following (Formula 2). In addition, brightness of the image620is corrected by changing the brightness value from B3 to B3′ based on the following (Formula 3).
B2′=B2+(Iave−I2ave)×w(Formula 2)
B3′=B3+(Iave−I3ave)×w(Formula 3)

In addition, since an image taken by the rear camera141is a reference image for performing image brightness correction, the image itself is not brightness corrected. Thus, w=0 is stored in the fourth decoder and brightness correction unit142. That is to say, if the currently set brightness value in the fourth decoder and brightness correction unit142is B4 and a newly set brightness value by the brightness correction is B4′, B4′ is expressed in (Formula 4).
B4′=B4  (Formula 4)

The images with the brightness corrected by (Formula 1), (Formula 2), (Formula 3), and (Formula 4) are converted into digital signals by the first A/D converter113, the second A/D converter123, the third A/D converter133, and the fourth A/D converter143, and an image600, an image610, an image620, and an image630, which are new, with the brightness thereof corrected are generated (S6ofFIG. 3).

For the image600, the image610, the image620, and image630, which are new, with the brightness thereof corrected, the image brightness calculation unit300calculates I ave, I 1ave, I 2ave, and I 3ave again.

Of I ave, I 1ave, I 2ave, and I 3ave which have been thus calculated, I ave and I 1ave are transmitted to the first decoder and brightness correction unit112and stored therein. In addition, I ave and I 2ave are transmitted to the second decoder and brightness correction unit122and stored therein. Furthermore, I ave and I 3ave are transmitted to the third decoder and brightness correction unit132and stored therein. Since no brightness correction is performed on any image inputted from the rear camera, w=0 is stored in the fourth decoder and brightness correction unit142.

Next, a coordinate transformation process and an image combining process are performed on the brightness corrected images. After being subjected to D/A conversion, the composite images are converted into composite signals by the encoding process and outputted to the monitor403.

Subsequently, as far as the shift position is in the retreated position, the processes mentioned above are continuously performed. Every time they are performed, brightness of the images is corrected based on the calculated newest brightness value.

With such configured vehicle periphery monitoring device according to the embodiment, when a 360° image that can look down at a vehicle periphery and an image captured for the backward direction of the vehicle are displayed simultaneously, easy-to-see images can be provided to drivers even if light and dark frequently changes, because the result of brightness correction of the overhead image does not affect the image captured for the backward direction of the vehicle.

In addition, in the embodiment, by setting an area consisting of m pixels in a horizontal direction and n pixels in a vertical direction at a prescribed position (x0, y0), as shown inFIG. 5, a reference value of image brightness correction was set as an average value I ave of all pixel values in the area. However a method for taking a reference value is not limited to this. For example, by setting an area631and an area632shown inFIG. 12Ein the image630taken by the rear camera141, an average value of all pixel values in the area631and632may be a reference for brightness correction. Now, the area631is an area to be converted into a rectangular area741having a size of width A and length B at a position where imaging ranges of the rear camera141and the left side camera121overlap or in the vicinity thereof when the image630is converted into the image740looking down from the overhead virtual viewing point V. In addition, the area632is an area to be converted into a rectangular area742in the image740having a size of width A and length B at a position where imaging ranges of the rear camera141and the right side camera131overlap or in the vicinity thereof. With the method for taking a reference, the effect that continuity of brightness at the seams of images is improved better than the method of the embodiment can be achieved.

In addition, the operation of the vehicle periphery monitoring device of the embodiment described above corresponds to an embodiment of a vehicle periphery image display method according to the present invention.

FIG. 7is a block diagram showing a schematic configuration of a second embodiment of a vehicle periphery monitoring device according to the present invention.

FIG. 8is a block diagram showing a detailed configuration of an imaging unit810, a brightness correction unit820, a coordinate transformation and image combining unit900, an image brightness calculation unit1000, and an image display unit1100in the vehicle periphery monitoring device inFIG. 7.

FIG. 9is a flow chart showing a series of processing steps in the vehicle periphery monitoring device shown inFIG. 8.

As shown in the block diagram ofFIG. 7, the vehicle periphery monitoring device according to the embodiment includes the imaging unit810such as N CCD cameras or C-MOS cameras which observe different directions respectively, the brightness correction unit820which corrects brightness of images inputted by the imaging units810, the coordinate transformation and image combining unit900which performs coordinate transformation on the images outputted from the brightness correction unit820and further combines the images to a single image, the image brightness calculation unit1000which calculates brightness of captured images, the image display unit1100including a monitor, etc., which shows the result of coordinate transformation/image combination performed, a start/end instruction detection unit1200which detects an instruction to start or end the vehicle periphery monitoring device, and an imaging-unit-selecting unit1300which selects one imaging unit from multiple imaging units810. In addition, this vehicle periphery monitoring device is mounted on a vehicle not shown inFIG. 7.

The embodiment is described taking, as an example, a case where the imaging unit810includes four cameras. That is to say, the imaging unit810includes a front camera811for observing the forward direction of a vehicle and a first decoder and brightness correction unit812and a first A/D converter813connected thereto, a left side camera821for observing the left direction of the vehicle and a second decoder and brightness correction unit822and a second A/D converter823connected thereto, a right side camera831for observing the right direction of the vehicle and a third decoder and brightness correction unit832and a third A/D converter833connected thereto, and a rear camera841for observing the backward direction of the vehicle and a fourth decoder and brightness correction unit842and a fourth A/D converter843connected thereto.

In addition, the coordinate transformation and image combining unit900includes a coordinate transformation unit901, a coordinate transformation table902, and an image combining unit908.

The image brightness calculation unit1000includes a first brightness calculation unit1001connected to the first A/D converter813, a second brightness calculation unit1002connected to the second A/D converter823, a third brightness calculation unit1003connected to the third A/D converter888, and a fourth brightness calculation unit1004connected to the fourth A/D converter843.

Furthermore, the image display unit1100includes a D/A converter1101, an encoder1102, and a monitor1103.

The vehicle periphery monitoring device in the embodiment is configured to display, as a single image on the monitor1108, one composite image700in which 360° information pieces on a vehicle periphery are combined and an image630taken by the rear camera841, as shown inFIG. 10A, or one composite image700in which 360° information pieces on the vehicle periphery are combined and an image600taken by the front camera811, as shown inFIG. 10B.

Now, as shown inFIG. 12A, the composite image700is a single image obtained by combining an image710looking down at the image600taken by the front camera811from an overhead virtual viewing point V, an image720looking down at an image610taken by the left side camera821from the overhead virtual viewing point V, an image730looking down at an image620taken by the right side camera831from the overhead virtual viewing point V, an image740looking down at an image630taken by the rear camera841from the overhead virtual viewing point V, and a virtual image750looking down at the vehicle from the overhead viewing point V.

In addition, although the embodiment is configured to observe the four directions of a vehicle, the number of directions to observe is not limited to this, and the embodiment may be configured to observe more directions. Even in such a case, the present invention can still be implemented with the operation similar to the embodiment.

The operation of the vehicle periphery monitoring device according to the embodiment will be described hereinafter with reference to the flow chart ofFIG. 9.

The front camera811, the left side camera821, the right side camera831, and the rear camera841are mounted on the vehicle in a layout in which imaging ranges of adjacent cameras (the front camera811and the left side camera821, the front camera811and the right side camera831, the left side camera821and the rear camera841, and the right side camera831and the rear camera841) partly overlap.

The start/end instruction detection unit1200detects that a shift position is in a retreated position or that the shift position is in an advance position and a vehicle speed is equal to or less than a predetermined value (S2, S3, S4ofFIG. 9), the coordinate transformation unit901generates a trigger signal. The trigger signal is inputted to the first decoder and brightness correction unit812, the first A/D converter813, the second decoder and brightness correction unit822, the second A/D converter823, the third decoder and brightness correction unit832, the third A/D converter833, the fourth decoder and brightness correction unit842, and the fourth A/D converter843.

When each camera receives the trigger signal, the front camera813, the left side camera821, the right side camera831, and the rear camera841simultaneously input an image (S7ofFIG. 9).

The image inputted from the front camera811is converted from composite signals into component signals by the first decoder and brightness correction unit812(S8ofFIG. 9). Furthermore, a luminance signal of the converted component signals is converted into a digital signal by the first A/D converter813and the image600is generated (S10ofFIG. 9). Similarly, the image inputted from the left side camera821is converted from the composite signals into the component signals (S8ofFIG. 9). Furthermore, the luminance signal of the converted component signals is converted into a digital signal by the second A/D converter823and the image610is generated (S10ofFIG. 9). The image inputted from the right side camera831is converted from the composite signals into the component signals by the third decoder and the brightness correction unit832(S8ofFIG. 9). Furthermore, the luminance signal of the converted composite signals is converted into a digital signal by the third A/D converter833, and the image620is generated (S10ofFIG. 9). The image inputted from the rear camera841is converted from the composite signals into the component signals by the fourth decoder and brightness correction unit842(S8ofFIG. 9). Furthermore, the luminance signal of the converted component signal is converted into a digital signal by the fourth A/D converter843, and the image630is generated (S10ofFIG. 9).

In addition, now, brightness of the inputted image is corrected (S9ofFIG. 9) at the same time when the signal is converted from the composite signal into the component signal. However, as a correction amount for correcting brightness of the image has not been calculated, the brightness correction is not performed when an initial image after starting of the vehicle periphery monitoring device is inputted, the bright correction is not performed.

Next, the imaging-unit-selecting unit1300determines whether to display the image630captured for the backward direction of the vehicle as inFIG. 10A, or the image600captured for the forward direction of the vehicle as inFIG. 10B, as an image to be outputted to the monitor1103.

This is determined based on the result of monitoring of a shift position and vehicle speed by the start/end instruction detection unit1200.

That is to say, if the shift position is in the retreated position, the image630captured for the backward direction of the vehicle is displayed in the area to the right of the image shown inFIG. 10A(S2ofFIG. 9). Or, if the shift position is in the advance position and the vehicle speed is equal to or less than the predetermined value, the image600captured for the forward direction of the vehicle is displayed in the area to the right of the image inFIG. 10B(S3, S4ofFIG. 9). Now, if it is determined that the image630captured for the backward direction of the vehicle is displayed, the rear camera841is selected (S6ofFIG. 9), while if it is determined that image600captured for the forward direction of the vehicle is displayed, the front camera811is selected (S5ofFIG. 9). A brightness correction procedure to be described later differs depending on which camera is selected.

First, the operation when the rear camera841is selected by the imaging-unit-selecting unit1300will be described.

The fourth brightness calculation unit1004calculates an average value of pixel values in a predetermined area in the image630taken by the rear camera841(S11ofFIG. 9). Now, as shown inFIG. 5, an area consisting of m pixels in a horizontal direction and n pixels in a vertical direction is set at a predetermined position (x0, y0) and an average value I ave of all pixels in that area is calculated.

Then, the first brightness calculation unit1001calculates an average value of pixels in the image600generated in S7ofFIG. 9(S12ofFIG. 9). Now, as shown inFIG. 6B, an area601and an area602of a predetermined size are set in prescribed positions which have been determined in advance, and an average value I 1ave of all the pixels in the area601, the area602is calculated.

Now, the area601is an area to be converted into a rectangular area711having a size of width A and length B at a position where the imaging ranges of the front camera811and the left side camera821overlap or in the vicinity thereof, when the image600is converted into the image710looking down from the overhead virtual viewing point V. Furthermore, the area602is an area to be converted into a rectangular area712having a size of width A and length B at a position where the imaging ranges of the front camera811and the right side camera831overlap or in the vicinity thereof, when the image600is converted into the image710looking down from the overhead virtual viewing point V.

In addition, the positions of the area601and the area602can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the first brightness calculation unit1001.

Next, the second brightness calculation unit1002calculates an average value of pixel values in the predetermined areas in the image610generated in S7ofFIG. 9(S12ofFIG. 9). Here, as shown inFIG. 6C, an area611and an area612of a predetermined size are set at prescribed positions in the image610which have been determined in advance, and an average value I 2ave of all the pixel values in the area611and the area612is calculated.

Now, the area611is an area to be converted into a rectangular area721having a size of width A and length B at a position where the imaging ranges of the front camera811and the left side camera821overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V. Furthermore, the area612is an area to be converted into a rectangular area722having a size of width A and length B at a position where the imaging ranges of the left side camera821and the rear camera841overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V.

In addition, the positions of the area611and the area612can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the second brightness calculation unit1002.

Further, the third brightness calculation unit1003calculates an average value of pixel values in predetermined areas in the image620generated in S7ofFIG. 9. Here, as shown inFIG. 6D, an area621and an area622of a predetermined size are set at prescribed positions which have been determined in advance, and an average value I 3ave of all the pixel values in the area621and the area622is calculated.

Now, the area621is an area to be converted into a rectangular area731having a size of width A and length B at a position where the imaging ranges of the front camera811and the right side camera831overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V. Furthermore, the area622is an area to be converted into a rectangular area732having a size of width A and length B at a position where the imaging ranges of the right side camera831and the rear camera841overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V.

In addition, the positions of the area621and the area622can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the third brightness calculation unit1003.

The calculated I ave and I 1ave are transmitted to the first decoder and brightness correction unit812and stored therein. Similarly, I ave and I 2ave are transmitted to the second decoder and brightness correction unit822and stored therein, and I ave and I 3ave are transmitted to the third decoder and brightness correction unit832and stored therein. In addition, since no brightness correction is performed on images to be inputted from the rear camera841, w=0 is stored in the fourth decoder and brightness correction unit842.

The image600, the image610, the image620, and the image630which have been generated in S7ofFIG. 3are coordinate transformed by the coordinate transformation unit901and converted into an image looking down from the overhead virtual viewing point V (S13ofFIG. 9). The coordinate transformation process is calculated based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc. However, in general, the coordinate transformation process is performed by creating the coordinate transformation table902in advance based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc., and then replacing coordinates of the inputted images on the basis of this coordinate transformation table902.

Through the coordinate transformation, the image600taken by the front camera811is converted into the image710, the image610taken by the left side camera821is converted into the image720, the image620taken by the right side camera831is converted into the image730, and the image630taken by the rear camera841is converted into the image740. The converted results are combined into a single image by the image combining unit903, and further a virtual image750looking down at the own vehicle from the overhead virtual viewing point V is combined therewith and a composite image700is generated. Furthermore, the composite image700is combined with the image630taken by the rear camera841and a single image shown inFIG. 10Ais created (S14ofFIG. 9).

The created image is reconstructed to component signals by the D/A converter1101(S15ofFIG. 9), further converted into composite signals by the encoder1102(S15ofFIG. 9), and displayed to the monitor1103(S17ofFIG. 9).

The start/end instruction detection unit1200detects whether or not a shift position is in a retreated position (S18, S19ofFIG. 9). If the shift position is in any position other than the retreated position, the display on the monitor1103is cancelled (S21ofFIG. 9).

If it is confirmed that the shift position is in the retreated position, the image is inputted again in S7ofFIG. 9.

The image inputted from the front camera811is converted from composite signals into component signals by the first decoder and brightness correction unit812(S8ofFIG. 9), and then, brightness is corrected based on I ave and I 1ave which have been stored in the first decoder and brightness correction unit812(S9ofFIG. 9).

Specifically, if a currently set brightness value in the first decoder and brightness correction unit812is B1, and a newly set brightness value by brightness correction is B1′, correction of brightness is performed by calculating a new brightness value B1′ by (Formula 1).

Images taken by the left side camera821or the right side camera831are also brightness corrected as described below.

That is to say, if a currently set brightness value in the second decoder and brightness correction unit822is B2, a newly set brightness value by the brightness correction is B2′, a currently set brightness value in the third decoder and brightness correction unit832is B3, and a newly set brightness value by the brightness correction is B3′, the image taken by the left side camera821is brightness corrected by the new brightness value B2′ to be calculated by (Formula 2), and the image taken by the right side camera831is brightness corrected by the new brightness value B3′ to be calculated by (Formula 3).

In addition, since an image taken by the rear camera is a reference image for performing image brightness correction, the image itself is not brightness corrected. Thus, w=0 is stored in the fourth decoder and brightness correction unit842. That is to say, if a currently set brightness value in the fourth decoder and brightness correction unit842is B4, and a newly set brightness value by the brightness correction is B4′, the new brightness value B4′ is calculated by (Formula 4) and brightness corrected.

The images with the brightness thereof corrected by (Formula 1), (Formula 2), (Formula 3), and (Formula 4) are converted into digital signals by the first A/D converter813, the second A/D converter823, the third A/D converter833, and the fourth A/D converter843, and an image600, an image610, an image620, and an image630, which are new, with the brightness thereof corrected are generated (S9ofFIG. 10).

For the image600, the image610, the image620, and image680, which are new, with the brightness thereof thus corrected, the image brightness calculation unit1000calculates I ave, I 1ave, I 2ave, and I 3ave again.

Of I ave, I 1ave, I 2ave, and I 3ave which have been thus calculated, I ave and I 1ave are transmitted to the first decoder and brightness correction unit812and stored therein. In addition, I ave and I 2ave are transmitted to the second decoder and brightness correction unit822and stored therein. Furthermore, I ave and I 3ave are transmitted to the third decoder and brightness correction unit832and stored therein. Since no brightness correction is performed on the image630, w=0 is stored in the fourth decoder and brightness correction unit842.

Next, a coordinate transformation process and an image combining process are performed on the brightness corrected images. After being subjected to D/A conversion, the composite images are converted into composite signals by the encoding process and outputted to the monitor1103.

Subsequently, as far as the shift position is in the retreated position, the processes mentioned above are continuously performed. Every time they are performed, brightness of the images is corrected based on the calculated newest brightness value.

The operation when the imaging-unit-selecting unit1300selects the front camera811will be described hereinafter.

The first brightness calculation unit1001calculates an average value of pixel values in the predetermined areas in the image600taken by the front camera811(S11ofFIG. 9). Now, as shown inFIG. 11, an area consisting of k pixels in a horizontal direction and 1 pixels in a vertical direction is set at a predetermined position (x1, y1) and an average value I ave of all pixels in that area is calculated.

Then, the second brightness calculation unit1002calculates an average value of pixels in the image610generated in S7ofFIG. 9(S12ofFIG. 9). Now, as shown inFIG. 12C, an area611and an area612of a predetermined size are set in prescribed positions which have been determined in advance, and an average value I 2ave of all the pixels in the area611, the area612is calculated.

Now, as shown inFIG. 12A, the area611is an area to be converted into a rectangular area721having a size of width A and length B at a position where the imaging ranges of the front camera811and the left side camera821overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V. Furthermore, the area612is an area to be converted into a rectangular area722having a size of width A and length B at a position where the imaging ranges of the left side camera821and the rear camera841overlap or in the vicinity thereof, when the image610is converted into the image720looking down from the overhead virtual viewing point V.

In addition, the positions of the area611and the area612can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the second brightness calculation unit1002.

Next, the third brightness calculation unit1003calculates an average value of pixel values in the predetermined areas in the image620generated in S7ofFIG. 9(S12ofFIG. 9). Here, as shown inFIG. 12D, an area621and an area622of a predetermined size are set at prescribed positions in the image620which have been determined in advance, and an average value I 3ave of all the pixel values in the area621and the area622is calculated.

Now, as shown inFIG. 12A, the area621is an area to be converted into a rectangular area731having a size of width A and length B at a position where the imaging ranges of the front camera811and the right side camera831overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V. Furthermore, the area622is an area to be converted into a rectangular area732having a size of width A and length B at a position where the imaging ranges of the right side camera831and the rear camera841overlap or in the vicinity thereof, when the image620is converted into the image730looking down from the overhead virtual viewing point V.

In addition, the positions of the area621and the area622can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the third brightness calculation unit1003.

Next, the fourth brightness calculation unit1004calculates an average value of pixel values in the predetermined areas in the image630generated in S7ofFIG. 9(S12ofFIG. 9). Here, as shown inFIG. 12E, an area631and an area632of a predetermined size are set at prescribed positions in the image630which have been determined in advance, and an average value I 4ave of all the pixel values in the area631and the area632is calculated.

Now, as shown inFIG. 12A, the area631is an area to be converted into a rectangular area741having a size of width A and length B at a position where the imaging ranges of the rear camera841and the left side camera821overlap or in the vicinity thereof, when the image630is converted into the image740looking down from the overhead virtual viewing point V. Furthermore, the area632is an area to be converted into a rectangular area742having a size of width A and length B at a position where the imaging ranges of the right side camera831and the rear camera841overlap or in the vicinity thereof, when the image630is converted into the image740looking down from the overhead virtual viewing point V.

In addition, the positions of the area631and the area682can be determined in advance by calculations. The calculated positions of the areas are stored in advance in the fourth brightness calculation unit1004.

I ave and I 1ave which have been calculated are transmitted to the second decoder and brightness correction unit822and stored therein. Similarly, I ave and I 3ave are transmitted to the third decoder and brightness correction unit832and stored therein, and I ave and I 4ave are transmitted to the fourth decoder and brightness correction unit842and stored therein. Since no brightness correction is performed on an image taken by the front camera811, w=0 is stored in the fifth decoder and brightness correction unit812.

The image600, the image610, the image620, and the image630which have been generated in S7ofFIG. 9are coordinate transformed by the coordinate transformation unit901and converted into an image looking down from the overhead virtual viewing point V (S13ofFIG. 9). The coordinate transformation process is calculated based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc. However, in general, the coordinate transformation process is performed by creating the coordinate transformation table902in advance based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc., and then replacing coordinates of the inputted images on the basis of this coordinate transformation table902.

Through the coordinate transformation, the image600is converted into the image710, the image610is converted into the image720, the image620is converted into the image730, and the image630is converted into the image740. The converted results are combined into a single image by the image combining unit903, and further a virtual image750looking down at the own vehicle from the overhead virtual viewing point V is combined therewith and a composite image700is generated. Furthermore, the composite image700is combined with the image600taken by the front camera811and a single image shown inFIG. 10Bis created (S14ofFIG. 9).

The created image is reconstructed to component signals by the D/A converter1101(S15ofFIG. 9), further converted into composite signals by the encoder1102(S15ofFIG. 9), and displayed to the monitor1103(S17ofFIG. 9).

The start/end instruction detection unit1200judges that the front camera811is in a selected state and a vehicle speed is equal to or less than a predetermined value (S18, S20ofFIG. 9). If the condition is not matched, the display on the monitor1103is cancelled (S21ofFIG. 9).

On the other hand, if it is judged that the condition is matched, image is inputted once again in S7ofFIG. 9.

The image taken by the rear camera841is converted from composite signals into component signals by the first decoder and brightness correction unit842(S8ofFIG. 9), and then, brightness is corrected based on I ave and I 4ave which have been stored in the fourth decoder and brightness correction unit842(S9ofFIG. 9).

Specifically, if a currently set brightness value in the fourth decoder and brightness correction unit842is B4, and a newly set brightness value by brightness correction is B4′, B4′ is calculated by (Formula 5) and thereby brightness is corrected.
B4′=B4+(Iave−I4ave)×w(Formula 5)

Images taken by the left side camera821or the right side camera831are also brightness corrected as described below.

That is to say, if a currently set brightness value in the second decoder and brightness correction unit822is B2, a newly set brightness value by the brightness correction is B2′, a currently set brightness value in the third decoder and brightness correction unit832is B3, and a newly set brightness value by the brightness correction is B3′, the image taken by the left side camera821is brightness corrected by the new brightness value B2′ to be calculated by (Formula 2), and the image taken by the right side camera831is brightness corrected by the new brightness value B3′ to be calculated by (Formula 3).

In addition, since no brightness correction is performed on an image taken by the front camera, w=0 is stored in the first decoder and brightness correction unit812. That is to say, if the currently set brightness value in the first decoder and brightness correction unit812is B1 and the newly set brightness value by the brightness correction is B1′, B1′ is calculated in the (Formula 6).
B1′=B1  (Formula 6)

The images with the brightness thereof corrected by (Formula 2), (Formula 3), (Formula 5), and (Formula 6) are converted into digital signals by the first A/D converter813, the second A/D converter823, the third A/D converter833, and the fourth A/D converter843, and an image600, an image610, an image620, and an image680, which are new, with the brightness thereof corrected are generated (S10ofFIG. 9).

For the image600, the image610, the image620, and image630, which are new, with the brightness thereof thus corrected, the image brightness calculation unit1000calculates I ave, I 2ave, I 3ave, and I 4ave again.

Of I ave, I 2ave, I 3ave, and I 4ave which have been thus calculated, I ave and I 2ave are transmitted to the second decoder and brightness correction unit822and stored therein. In addition, I ave and I 3ave are transmitted to the third decoder and brightness correction unit832and stored therein. Furthermore, I ave and I 4ave are transmitted to the fourth decoder and brightness correction unit842and stored therein. Since no brightness correction is performed on the image600inputted from the front camera811, w=0 is stored in the first decoder and brightness correction unit812.

Next, a coordinate transformation process and an image combining process are performed on the brightness corrected images. After being subjected to D/A conversion, the composite image signals are converted into composite signals by the encoding process and outputted to the monitor1103.

Subsequently, as far as the shift position is in the advance position and a vehicle speed is less than or equal to a predetermined value, the above process is continuously performed. Every time it is performed, brightness of the images is corrected based on the calculated newest brightness value.

Since the result of image brightness correction does not affect brightness of an image in a traveling direction of the vehicle to which most attention should be paid during driving, it is possible to provide drivers information useful for safety confirmation around a vehicle.

In addition, the operation of the vehicle periphery monitoring device of the embodiment described above corresponds to an embodiment of a vehicle periphery image display method according to the present invention.

FIG. 13is a block diagram showing a schematic configuration of a third embodiment of a vehicle periphery monitoring device according to the present invention.

FIG. 14is a block diagram showing a detailed configuration of an imaging unit1410, a brightness correction unit1420, a coordinate transformation and image combining unit1500, an image brightness calculation unit1600, and an image display unit1700.

FIG. 15is a flow chart showing a series of processing steps in a vehicle periphery monitoring device shown inFIG. 13andFIG. 14.

As shown in the block diagram ofFIG. 13, the vehicle periphery monitoring device according to the embodiment includes the imaging unit1410such as N CCD cameras or C-MOS cameras which are mounted on a vehicle and observe different directions respectively, the brightness correction unit1420which corrects brightness of images inputted by the imaging units1410, the coordinate transformation and image combining unit1500which performs coordinate transformation on the images outputted from the brightness correction unit1420and further combines the images to a single image, the image brightness calculation unit1600which calculates brightness of captured images, the image display unit1700including a monitor, etc., which shows the result of coordinate transformation/image combination performed, a start/end instruction detection unit1800which detects an instruction to start or end the vehicle periphery monitoring device, and an imaging-unit-selecting unit1900which selects one imaging unit from multiple imaging units1410.

The embodiment is described taking, as an example, a case where the imaging unit1410includes four cameras. That is to say, the imaging unit1410includes a front camera1411for observing the forward direction of a vehicle and a first decoder and brightness correction unit1412and a first A/D converter1413connected thereto, a left side camera1421for observing the left direction of the vehicle and a second decoder and brightness correction unit1422and a second A/D converter1423connected thereto, a right side camera1431for observing the right direction of the vehicle and a third decoder and brightness correction unit1432and a third A/D converter1433connected thereto, and a rear camera1441for observing the backward direction of the vehicle and a fourth decoder and brightness correction unit1442and a fourth A/D converter1443connected thereto.

In addition, the coordinate transformation and image combining unit1500includes a coordinate transformation unit1501, a coordinate transformation table1502, and an image combining unit1503.

The image brightness calculation unit1600includes a first brightness calculation unit1601connected to the first A/D converter1413, a second brightness calculation unit1602connected to the second A/D converter1423, a third brightness calculation unit1603connected to the third A/D converter1433, and a fourth brightness calculation unit1604connected to the fourth A/D converter1443.

Furthermore, the image display unit1700includes a D/A converter1701, an encoder1702, and a monitor1703.

The vehicle periphery monitoring device in the embodiment is configured to display, as a single image on the monitor1703, one composite image700in which 360° information pieces on a vehicle periphery are combined and an image630taken by the rear camera1411, as shown inFIG. 10A, or one composite image700in which 360′ information pieces on the vehicle periphery are combined and an image600taken by the front camera1441, as shown inFIG. 10B.

Now, the composite image700is a single image obtained by combining an image710looking down at the image600taken by the front camera1411from an overhead virtual viewing point V, an image720looking down at an image610taken by the left side camera1421from the overhead virtual viewing point V, an image730looking down at an image620taken by the right side camera1431from the overhead virtual viewing point V, an image740looking down at an image630taken by the rear camera1441from the overhead virtual viewing point V, and a virtual image750looking down at the vehicle from the overhead viewing point V.

In addition, although the embodiment is configured to observe the four directions of a vehicle, the number of directions to observe is not limited to this, and the embodiment may be configured to observe more directions. Even in such a case, the present invention can still be implemented with the operation similar to the embodiment.

The operation of the vehicle periphery monitoring device according to the embodiment will be described hereinafter with reference to the flow chart ofFIG. 15.

The front camera1411, the left side camera1421, the right side camera1431, and the rear camera1441are mounted on the vehicle in a layout in which imaging ranges of adjacent cameras (the front camera1411and the left side camera1421, the front camera1411and the right side camera1431, the left side camera1421and the rear camera1441, and the right side camera1431and the rear camera1441) partly overlap.

The start/end instruction detection unit1800detects that a shift position is in a retreated position or that the shift position is in an advance position and a vehicle speed is less than or equal to a predetermined value (S2, S3, S4ofFIG. 15), the coordinate transformation unit1501generates a trigger signal. The trigger signal is inputted to the first decoder and brightness correction unit1412, the first A/D converter1413, the second decoder and brightness correction unit1422, the second A/D converter1423, the third decoder and brightness correction unit1432, the third A/D converter1433, the fourth decoder and brightness correction unit1442, and the fourth A/D converter1443.

When the trigger signal is received, images are simultaneously inputted from the front camera1411, the left side camera1421, the right side camera1431, and the rear camera1411(S7ofFIG. 15).

The image inputted from the front camera1411is converted from composite signals into component signals by the first decoder and brightness correction unit1412(S8ofFIG. 15). Furthermore, a luminance signal of the converted component signals is converted into a digital signal by the first A/D converter1413and the image600is generated (S10ofFIG. 15). Similarly, the image inputted from the left side camera1421is converted from the composite signals into the component signals the second decoder and brightness correction unit1422(S8ofFIG. 15). Furthermore, the luminance signal of the converted component signals is converted into a digital signal by the second A/D converter1423and the image610is generated (S10ofFIG. 15). The image inputted from the right side camera1431is converted from the composite signals into the component signals by the third decoder and the brightness correction unit1432(S8ofFIG. 15). Furthermore, the luminance signal of the converted composite signals is converted into a digital signal by the third A/D converter1433, and the image620is generated (S10ofFIG. 15). The image inputted from the rear camera1441is converted from the composite signals into the component signals by the fourth decoder and brightness correction unit1441(S8ofFIG. 15). Furthermore, the luminance signal of the converted component signal is converted into a digital signal by the fourth A/D converter1443, and the image630is generated (S10ofFIG. 15).

In addition, now, brightness of the inputted image is corrected (S9ofFIG. 9) at the same time when the signal is converted from the composite signal into the component signal. However, as a correction amount for correcting brightness of the image has not been calculated, the brightness correction is not performed when an initial image after starting of the vehicle periphery monitoring device is inputted, the bright correction is not performed.

Next, the imaging-unit-selecting unit1900determines whether to display the image630captured for the backward direction of the vehicle or the image600captured for the forward direction of the vehicle, as an image to be outputted to the monitor1103, in the area to the right of the composite image700.

This is determined based on the result of monitoring of a shift position and vehicle speed by the start/end instruction detection unit1800.

That is to say, if the shift position is in the retreated position, the image630captured for the backward direction of the vehicle is displayed in the area to the right of the composite image700, as shown inFIG. 10A. Alternatively, if the shift position is in the advance position and the vehicle speed is less than or equal to the predetermined value, the image600captured for the forward direction of the vehicle is displayed in the area to the right of the composite image700, as shown inFIG. 10B. Now, if it is determined to display the image630, the rear camera1441is selected (S6ofFIG. 15), while if it is determined to display the image600, the front camera1411is selected (S5ofFIG. 15). A brightness correction procedure to be described later differs depending on which camera is selected.

First, the operation when the rear camera1441is selected by the imaging-unit-selecting unit1900will be described.

First, as shown inFIG. 12B,FIG. 12C,FIG. 12D, andFIG. 12E, the fourth brightness calculation unit1604calculates an average value I a_ave of all pixels in the area601, an average value I b_ave of all pixel values of the area602, an average value I c_ave of all pixel values in the area611, an average value I d_ave of all pixel values in the area612, an average value I e ave of all pixel values in the area621, an average value I f_ave of all pixel values in the area622, an average value I g_ave of all pixel values in the area631and an average value I h_ave of all pixel values in the area632, respectively, which have been set in prescribed size and in predetermined positions in the image600, the image610, the image620and the image630.

Now, as shown inFIG. 12A, the area601, the area60, the area611, the area612, the area621, the area622, the area631, and the area632are areas, to be respectively converted into a rectangular area711, a rectangular area712, a rectangular area721, a rectangular area722, a rectangular area731, a rectangular area732, a rectangular area741, and a rectangular area742which have a size of width A and length B and which are located in a position where imaging ranges of adjacent cameras overlap or in the vicinity thereof, when the image600, the image610, the image620, and the image630are, respectively, converted into the image710, the image720, the image730, the image740looking down from the overhead virtual viewing point V.

Positions of the area601, the area602, the area611, the area612, the area621, the area622, the area631, and the area632can be determined in advance by the calculation. Of the positions of the areas which have been calculated in advance, the positions of the area601and the area602are stored in the first brightness calculation unit1601, the positions of the area611and the area612are stored in the second brightness calculation unit1602, the positions of the area621and the area622are stored in the third brightness calculation unit1603, and the positions of the area631and the area632are stored in the fourth brightness calculation unit1604.

Furthermore, out of I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_ave, I g_ave, I h_ave, I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I which are average values of all pixel values of the areas, a_ave, I b_ave, I c_ave, I e_ave are stored in the first decoder and brightness correction unit1412, I d_ave and I g_ave are stored in the second decoder and brightness correction unit1422, and I f_ave, I h_ave are stored in the third decoder and brightness correction unit1432.

In addition, since no brightness correction is performed on the image630, w=0 is stored in the fourth data and brightness correction unit1442.

The image600, the image610, the image620, and the image630which have been generated in S7ofFIG. 15are coordinate transformed by the coordinate transformation unit1501, and converted into images looking down from the overhead virtual viewing position V (S12ofFIG. 15). The coordinate transformation process is performed by creating the coordinate transformation table1502in advance based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point V or observation range, etc., and then replacing coordinates of the inputted images on the basis of this coordinate transformation table1502.

Through the coordinate transformation, the image600taken by the front camera1411is converted into the image710, the image610taken by the left side camera1421is converted into the image720, the image620taken by the right side camera1431is converted into the image730, and the image630taken by the rear camera1441is converted into the image740. The converted results are combined into a single image by the image combining unit1503, and further a virtual image750looking down at the own vehicle from the overhead virtual viewing point V is combined therewith and a composite image700is generated. Furthermore, the composite image700is combined with the image630taken by the rear camera1441and a single image shown inFIG. 10Ais created (S13ofFIG. 15).

The composite image is reconstructed to component signals by the D/A converter1701(S14ofFIG. 16), further converted into composite signals by the encoder1702(S15ofFIG. 15), and displayed to the monitor1703(S16ofFIG. 15).

The start/end instruction detection unit1800detects whether or not a shift position is in a retreated position (S17, S18ofFIG. 15). If the shift position is in any position other than the retreated position, the display on the monitor1703is cancelled (S20ofFIG. 15).

If it is confirmed that the shift position is in the retreated position, the image is inputted again in S7ofFIG. 15.

The image taken by the left side camera1421is converted from composite signals into component signals by the second decoder and brightness correction unit1422(S8ofFIG. 15). Then, based on I g_ave and I d_ave stored in the second decoder and brightness correction unit1422, brightness is corrected (S9ofFIG. 15).

Specifically, brightness correction is performed with the following (Formula 7) by setting a currently set brightness value in the second decoder and brightness correction unit1422to B2 and a newly set brightness value to B2′ by the brightness correction:
B2′=B2+(I g_ave−I d_ave)×w(Formula 7)

Furthermore, the image taken by the right side camera1431is converted from composite signals into component signals by the third decoder and brightness correction unit1432(S8ofFIG. 15) and then brightness is corrected based on the I h_ave and I f_ave stored in the third decoder and brightness correction unit1432(S9ofFIG. 15).

That is to say, by setting a currently set brightness value in the third decoder and bright correction unit1432to B3 and a newly set brightness value by the brightness correction to B3′, brightness is corrected with the following (Formula 8):
B3′=B3+(1h_ave−1f_ave)×w(Formula 8)

Then, an image taken by the front camera1411is brightness corrected and the correction is performed based on the result of the brightness correction of images inputted from the left side camera1421and the right side camera1421. That is to say, the brightness correction of an image taken by the front camera1411is performed when an image is inputted next time, rather than being performed here.

In addition, an image taken by the rear camera1441is not brightness controlled. That is to say, if a currently set brightness value in the fourth decoder and brightness correction unit1442is B4 and a newly set brightness value by the brightness correction is B4′, B4′ is calculated with (Formula 4):

The image with the brightness thereof corrected with (Formula 4), (Formula 7), and (Formula 8) is converted into digital signals by the first A/D converter1412, the second A/D converter1423, the third A/D converter1423, and the fourth A/D converter1442, and an image600, an image610, an image620, and an image630, which are new, with the brightness thereof corrected is generated (S10ofFIG. 9).

For the brightness corrected images600, the image610, the image620, the image630which have thus been obtained, the image brightness calculation unit1600calculates I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_have, I g_ave, and I h_ave.

Out of I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_ave, I g_ave, and I h_ave which are calculated, I a_ave, I b_ave, I c_ave, and I e_have are stored in the first decoder and brightness correction unit1412, I d_ave and I g_ave are stored in the second decoder and brightness correction unit1422, and I f_ave and I h_ave are stored in the third decoder and brightness correction unit1432.

In addition, since no brightness correction is performed on an image inputted from the rear camera, w=0 is stored in the fourth decoder and brightness correction unit1442.

Next, a coordinate transformation process and an image combining process are performed on the brightness corrected images (S12, S13ofFIG. 15). After being subjected to D/A conversion, the composite image is converted into composite signals by the encoding process and outputted to the monitor1103(S14, S15, S16ofFIG. 15).

If it is determined in S17, S18ofFIG. 15that the shift position is in the retreated position, images are simultaneously inputted from the front camera1411, the left side camera1421, the right side camera1431, and the rear camera1411in S7ofFIG. 15.

After a series of processes described above are repeated again, the image brightness calculation unit1600calculates each value of brightness I a_ave, I b_ave, I c_ave, I d_ave I e_ave, I f_ave, I g_ave, I h_ave in the predetermined areas in each image, and these values are stored in the predetermined decoders and brightness correction units as described earlier, brightness of the image taken by the left side camera1421and brightness of the image taken by the right side camera1431are respectively corrected based on the brightness of the images taken by the rear camera1441.

Furthermore, brightness of the images taken by the front camera1411is corrected, using values of I a_ave, I b_ave, I c_ave, I e_ave stored in the first decoder and brightness correction unit1411. Now, by setting a currently set brightness value in the first decoder and brightness correction unit1412to B1 and a newly set brightness value by the brightness unit to B1′, the brightness correction is performed with (Formula 9):
B1′=B1+((I c_ave+I e_ave)/2−(Ia_ave+I b_ave)/2)×w(Formula 9)

Subsequently, the coordinate transformation/combining process is performed on the brightness corrected images. Then, they are subjected to the D/A process and encoded, and the result thereof is outputted to the monitor1703.

If it is confirmed that the shift position is in the retreated position, a next image is inputted. Then, brightness correction with (Formula 4), (Formula 7), (Formula 8), and (Formula 9) are performed. Thereafter, as far as the shift position is in the retreated position, a series of processes described above will be repeated.

Then, the operation when the imaging-unit-selecting unit1900selects the front camera1411will be described hereinafter.

First, as shown inFIG. 12B,FIG. 12C,FIG. 12D, andFIG. 12E, in the image600, the image610, the image620, the image630, the first brightness calculation unit1602calculates average values, I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_ave, I g_ave, I h_ave of all pixel values, respectively, in the area601, the area602, the area611, the area612, the area621, the area622, the area631, and the area632which have been set in a predetermined size in prescribed positions which have been determined in advance (S11ofFIG. 15).

Now, as shown inFIG. 12A, the area601, the area602, the area611, the area612, the area621, the area622, the area631, and the area632are areas to be converted, respectively, into a rectangular area711, a rectangular area712, a rectangular area721a rectangular area722, a rectangular area731, a rectangular area732, a rectangular area741and a rectangular area742which have a size of width A and length B and which are located at a position where imaging ranges of adjacent cameras overlap or in the vicinity thereof, when the image600, the image610, the image620, the image630are converted into the image710, the image720, the image730, and the image740looking down from the overhead virtual viewing point V.

The area601, the area602, the area611, the area612, the area621, the area622, the area631, and the area632can be determined in advance by calculations. Of the calculated positions of the areas, the positions of the area601and the area602are stored in the first brightness calculation unit1601, the positions of the area611and the area612are stored in the second brightness calculation unit1602, the positions of the area621and the area622are stored in the third brightness calculation1603, and the positions of the area631and the area632are stored in the fourth brightness calculation unit1604.

Furthermore, out of I a_ave, I b_ave I c_ave, I d_ave, I e_ave, I f_ave, I g_ave, and I h_ave which are calculated, I d_ave, I f_ave, I g_ave, and I h_ave are stored in the fourth decoder and brightness correction unit1422, I a_ave and I c_ave are stored in the second decoder and brightness correction unit1422, and I b_ave and I e_ave are stored in the third decoder and brightness correction unit1432.

In addition, since no brightness correction is performed on an image taken by the front camera1411, w=0 is stored in the first decoder and brightness correction unit1412.

The image600, the image610, the image620, the image630which have been generated in S7ofFIG. 15are coordinate transformed by the coordinate transformation unit1501and converted into images looking down from the overhead virtual viewing point V (S12ofFIG. 15). The coordinate transformation process is performed by creating the coordinate transformation table1502in advance based on camera parameters such as geometric layout of each camera mounted on the vehicle or a focal length of the camera, pixel size, etc. and parameters such as a position of the virtual viewing point or observation range, etc., and then replacing coordinates of the inputted images on the basis of this coordinate transformation table1502.

Through the coordinate transformation, the image600is converted into the image710, the image610is converted into the image720, the image620is converted into the image730, and the image630is converted into the image740. The converted results are combined into a single image by the image combining unit1503, and further a virtual image750looking down at the own vehicle from the overhead virtual viewing point V is combined therewith and a composite image700is generated. Furthermore, the composite image700is combined with the image600, and a single image shown inFIG. 10Bis created (S13ofFIG. 15).

The composite image is reconstructed to component signals by the D/A converter1701(S14ofFIG. 15), further converted into composite signals by the encoder1702(S15ofFIG. 15), and displayed to the monitor1703(S16ofFIG. 15).

The start/end instruction detection unit1800judges that the front camera1411is in a selected state and a vehicle speed is less than or equal to a predetermined value (S17, S18ofFIG. 15). If the condition is not matched, the display on the monitor1703is cancelled (S21ofFIG. 15).

On the other hand, if it is judged that the condition is matched, image is inputted once again in S7ofFIG. 15.

In S7ofFIG. 15, after images are inputted again, an image taken by the left side camera1421is converted from composite signals into component signals by the second decoder and brightness correction unit1422(S8ofFIG. 15). Then, brightness is corrected based on I a_ave and I c_ave stored in the second decoder and brightness correction unit1422(S9ofFIG. 15).

Specifically, brightness correction is performed with the following (Formula 10) by setting a currently set brightness value in the second decoder and brightness correction unit1422to B2 and a newly set brightness value to B2′ by the brightness correction:
B2′=B2+(I a_ave−I c_ave)×w(Formula 10)

Furthermore, the image taken by the right side camera1431is converted from composite signals into component signals by the third decoder and brightness correction unit1432(S8ofFIG. 15) and then brightness is corrected based on the I b_ave and I e_ave stored in the third decoder and brightness correction unit1432(S9ofFIG. 15).

That is to say, by setting a currently set brightness value in the third decoder and bright correction unit1432to B3 and a newly set brightness value by the brightness correction to B3′, brightness is corrected with the following (Formula 11):
B3′=B3+(1h_ave−1e_ave)×w(Formula 11)

Then, an image taken by the rear camera1441is brightness corrected and the correction is performed based on the result of the brightness correction of images inputted from the left side camera1421and the right side camera1431. That is to say, the brightness correction of an image taken by the rear camera1441is performed when an image is inputted next time, rather than being performed here.

In addition, since an image taken by the front camera1411is not brightness controlled, if a currently set brightness value in the first decoder and brightness correction unit1412is B1 and a newly set brightness value by the brightness correction is B1′, B1′ is calculated with (Formula 6):

The image with the brightness thereof corrected with (Formula 6), (Formula 10), and (Formula 11) is converted into digital signals by the first A/D converter1413, the second A/D converter1423, the third A/D converter1433, and the fourth A/D converter1443, and an image600, an image610, an image620, and an image630, which are new, with the brightness thereof corrected is generated (S10ofFIG. 9).

Thus, for the brightness corrected images600, the image610, the image620, the image630, the image brightness calculation unit1600calculates I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_have, I g_ave, and I h_ave.

Out of I a_ave, I b_ave, I c_ave, I d_ave, I e_ave, I f_ave, I g_ave, and I h_ave which are calculated, I a_ave, I b_ave, I c_ave, and I e_have are stored in the fourth decoder and brightness correction unit1442, I d_ave and I g_ave are stored in the second decoder and brightness correction unit1422, and I f_ave and I h_ave are stored in the third decoder and brightness correction unit1432.

In addition, since no brightness correction is performed on the image600, w=0 is stored in the first decoder and brightness correction unit1412.

Next, a coordinate transformation process and an image combining process are performed on the brightness corrected images (S12, S13ofFIG. 15). After being subjected to D/A conversion, the composite images are converted into composite signals by the encoding process and outputted to the monitor1103(S14, S15, S16ofFIG. 15).

If it is determined in S17, S18ofFIG. 15that the front camera1411has been selected and a vehicle speed is less than or equal to a predetermined value, images are simultaneously inputted from the front camera1411, the left side camera1421, the right side camera1431, and the rear camera1441in S7ofFIG. 15.

After a series of processes described above are repeated again, the image brightness calculation unit1600calculates each value of brightness I a_ave, I b_ave, I c_ave, I d_ave I e_ave, I f_ave, I g_ave, I h_ave in the predetermined areas in each image. The values are stored in the predetermined decoders and brightness correction units, brightness of the image taken by the left side camera1421and brightness of the image taken by the right side camera1431are respectively corrected based on the brightness of the images taken by the front camera1411.

Furthermore, brightness of the images taken by the rear camera1441is corrected, using values of I d_ave, I f_ave, I g_ave, I h_ave stored in the fourth decoder and brightness correction unit1442. Now, by setting a currently set brightness value in the first decoder and brightness correction unit1442to B4 and a newly set brightness value by the brightness unit to B4′, the brightness correction is performed with (Formula 12):
B4′=B4+((I d_ave+I f_ave)/2−(Ig_ave+I h_ave)/2)×w(Formula 12)

Subsequently, the coordinate transformation/combining process is performed on the brightness corrected images. Then, they are subjected to the D/A process and encoded, and the result thereof is outputted to the monitor1703.

If it is confirmed that the front camera1411has been selected and the vehicle speed is less than and equal to the predetermined value, a next image is inputted. Then, brightness correction with (Formula 6), (Formula 10), (Formula 11), and (Formula 12) are performed. Thereafter, as far as the vehicle speed is less than or equal to the predetermined value, a series of processes described above will be repeated. Every time they are performed, brightness of the images is corrected based on the calculated newest brightness value.

With the vehicle periphery monitoring device according to such configured embodiment, since it is possible to perform brightness connection so that two images are equal in brightness in an overlapping imaging range in which two images overlap in part or in brightness in the vicinity thereof, as well as to recursively correct brightness of a different image captured for a range overlapping with the image on the basis of brightness of a reference image, in the vehicle periphery monitoring device which combines a number of images into one image and displays it, a feeling of unevenness at the seams of images can be considerably reduced, and easy-to-see images can be provided to drivers.

Furthermore, since image brightness correction can be performed by adjustment of brightness when images taken by imaging units are decoded, the brightness correction can be performed in a short period of time.

In addition, the operation of the vehicle periphery monitoring device of the embodiment described above corresponds to an embodiment of a vehicle periphery image display method according to the present invention.

In addition, here, although the embodiment 3 includes the imaging-unit-selecting unit1900for selecting one imaging unit from the plurality of imaging units1410, the configuration of this part is not limited to this, and one specific imaging unit which has been determined in advance may be selected.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority on the basis of Japanese Patent Application No. 2009-176308 filed to Japan Patent Office on Jul. 29, 2009, which is incorporated in entirety herein by reference.