Abstract:
A vehicle drive assistant system which converts, into birds&#39; eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising a means for, when each overlapping portion in which two bird&#39;s-eye views overlap each other is synthesized, setting a border line which allows two regions to be alternately disposed with respect to the overlapping portion and adopting a bird&#39;s-eye view in a region separated by the border line in the overlapping portion while adopting the other bird&#39;s-eye view in the other region separated by the border line so as to synthesize the overlapping portion.

Description:
BACKGROUND OF THE INVENITON  
       [0001]     1. Field of the Invention  
         [0002]     The present invention relates to a vehicle drive assistant system.  
         [0003]     2. Description of the Related Art  
         [0004]     It is difficult for a vehicle driver or the like to confirm the backward direction because a blind corner occurs when he or she backs up. For the reason, there has been developed a system which is provided with a vehicle loaded camera for monitoring the backward scene which is likely to becomes a blind corner for a vehicle driver so as to display its photographed image on a screen of car navigation unit or the like.  
         [0005]     However, when a wide angle lens is used to display a wide range, lens distortion is generated in a displayed image. Further, as an object is farther from the camera, its image is displayed smaller than in case of an ordinary lens, so that it is difficult to recognize a distance or space backward of a vehicle from the photographed image.  
         [0006]     Thus, a research for not just displaying a camera image but presenting an image more gentle for the human using image processing technology has been conducted. One of them is to convert the coordinates of a photographed image and generate a bird&#39;s-eye view as seen from above the ground and display it. By displaying the bird&#39;s-eye view as seen from above the ground, the vehicle driver can grasp the distance or space backward of the vehicle easily.  
         [0007]     Further, as the vehicle drive assistant system for parking, an apparatus which converts images obtained from multiple cameras to all around bird&#39;s-eye view by geometric transformation and displays the all around bird&#39;s-eye view on a monitor has been developed (see Japanese Laid-Open Patent Publication NO.11-78692). This apparatus has an advantage that the all around of a vehicle can be covered over 360° without any blind corner because the all around scene of the vehicle can be presented to a vehicle driver as a view seen from above the ground.  
         [0008]     However, in the bird&#39;s-eye view, as shown in  FIG. 1 , an object 200 having a height is projected to the ground such that its image is deformed on an extension line when a camera  1  and the object  200  are connected. If the object  200  having a height exists obliquely backward of the left rear end of the vehicle when the vehicle  100  is provided with cameras  1 F,  1 B,  1 L,  1 R at its front, rear, left and right sides as shown in  FIG. 2 , a projection image by the left side camera  1 L is  200 L and a projection image by the rear side camera  1 B is  200 B.  
         [0009]     Both the projection images  200 L and  200 B have an overlapping portion in which a bird&#39;s-eye view obtained from an image photographed by the left side camera  1 L and a bird&#39;s-eye view obtained by an image photographed by the rear side camera  1 B overlaps each other. Then, if this overlapping portion is divided into left side camera region SL and rear camera region SB by a border line D extending obliquely backward from the left rear end of the vehicle as shown in  FIG. 3 , the projection image  200 B by the rear side camera  1 B exists in the left side camera region SL and the projection image  200 L by the left side camera  1 L exists in the rear camera region SB.  
         [0010]     If only a bird&#39;s-eye view obtained from the photographed image of the left side camera  1 L is adopted to the left side camera region SL and only a bird&#39;s-eye view obtained from the photographed image of the rear camera  1 B is adopted to the rear camera region SB when the overlapping portions are synthesized, both the projection images  200 L and  200 B disappear in the all around bird&#39;s-eye view obtained after synthesis.  
         [0011]     To solve this problem, it can be considered to blend the both bird&#39;s-eye views when the aforementioned overlapping portions are synthesized. However, because the both projection images  200 L and  200 B exist in the all around bird&#39;s-eye view obtained after synthesis, the object  200  appears as double image. Further, because both projection images  200 L and  200 B are blended as a background image, the projection images  200 L and  200 B become difficult to see depending on the colors of the object  200  and the background.  
         [0012]     According to another developed method (see Japanese Patent No.3372944), when a bird&#39;s-eye image obtained from the rear camera  1 B and a bird&#39;s-eye image obtained from the side cameras  1 L and  1 R are synthesized, only the bird&#39;s-eye view obtained from the side cameras  1 L and  1 R is adopted at the overlapping portion so as to generate a synthesized bird&#39;s-eye view taking preference to the side camera and only a bird&#39;s-eye view obtained from the rear camera  1 B is adopted at that overlapping portion so as to generate a synthesized bird&#39;s-eye view taking preference to the rear camera and these two kinds of the synthesized bird&#39;s-eye views are arranged side by side for display.  
         [0013]     However, according to this method, the vehicle driver has to grasp situations around the vehicle by comparing the two bird&#39;s-eye views and therefore, burden on the vehicle driver increases thereby possibly damaging the safety.  
       SUMMARY OF THE INVENTION  
       [0014]     An object of the present invention is to provide a vehicle drive assistant system capable of solving such a problem that an object having a height disappears on a synthesized bird&#39;s-eye view and which allows that object to be recognized easily.  
         [0015]     To achieve the above-mentioned object, according to a first aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising a means for, when each overlapping portion in which two bird&#39;s-eye views overlap each other is synthesized, setting a border line which allows two regions to be alternately disposed with respect to the overlapping portion and adopting a bird&#39;s-eye view in a region separated by the border line in the overlapping portion while adopting the other bird&#39;s-eye view in the other region separated by the border line so as to synthesize the overlapping portion.  
         [0016]     According to a second aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, comprising a means for, when each overlapping portion in which two bird&#39;s-eye views overlap each other is synthesized, setting a pectinate border line with respect to the overlapping portion and adopting a bird&#39;s-eye view in a region separated by the pectinate border line in the overlapping portion while adopting the other bird&#39;s-eye view in the other region separated by the pectinate border line so as to synthesize the overlapping portion.  
         [0017]     According to a third aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising: a first synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a first synthesized bird&#39;s-eye view obtained by adopting only a bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; a second synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a second synthesized bird&#39;s-eye view obtained by adopting only the other bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; and a control means for displaying the first synthesized bird&#39;s-eye view and the second synthesized bird&#39;s-eye view alternately on the display unit by changing over the first synthesized bird&#39;s-eye view generating means and the second synthesized bird&#39;s-eye view generating means alternately.  
         [0018]     According to a fourth aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising: a first synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a first synthesized bird&#39;s-eye view obtained by adopting only a bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; a second synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a second synthesized bird&#39;s-eye view obtained by adopting only the other bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; a determining means for determining whether or not an object having a height exists by comparing two bird&#39;s-eye views in each overlapping portion in which two bird&#39;s-eye views overlap each other; a first control means which, if it is determined that the object having the height exists in at least one overlapping portion by the determining means, displays the first synthesized bird&#39;s-eye view and the second synthesized bird&#39;s-eye view alternately on the display unit by changing over the first synthesized bird&#39;s-eye view generating means and the second synthesized bird&#39;s-eye view generating means alternately; and a second control means which, if it is determined that no object having a height exists in any overlapping portion by the determining means, generates a synthesized bird&#39;s-eye view by any one synthesized bird&#39;s-eye view generating means preliminarily set among the first synthesized bird&#39;s-eye view generating means and the second synthesized bird&#39;s-eye view generating means and displays a generated synthesized bird&#39;s-eye view on the display unit.  
         [0019]     According to a fifth aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising: a first synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a first synthesized bird&#39;s-eye view obtained by adopting only a bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; a second synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a second synthesized bird&#39;s-eye view obtained by adopting only the other bird&#39;s-eye view preliminarily set in each overlapping portion in which two bird&#39;s-eye views overlap; a selecting means for selecting any one of the first synthesized bird&#39;s-eye view generating means and the second synthesized bird&#39;s-eye view generating means depending on the advancement condition of the vehicle; and a control means for generating a synthesized bird&#39;s-eye view by the synthesized bird&#39;s-eye view generating means selected by the selecting means and displaying a generated synthesized bird&#39;s-eye view on the display unit.  
         [0020]     According to a sixth aspect of the present invention, there is provided a vehicle drive assistant system which converts, into bird&#39;s-eye images, images photographed by a plurality of image pickup devices loaded on a vehicle and for photographing the surrounding of the vehicle, generates a synthesized bird&#39;s-eye view by synthesizing each of the obtained bird&#39;s-eye images and displays a generated synthesized bird&#39;s-eye view on a display unit, the vehicle drive assistant system comprising: a preference bird&#39;s-eye view determining means for determining a bird&#39;s-eye view in which an object having a height appears larger among two bird&#39;s-eye views in each overlapping portion in which two bird&#39;s-eye views overlap as a preference bird&#39;s-eye view; a synthesized bird&#39;s-eye view generating means for, when each bird&#39;s-eye view is synthesized, generating a synthesized bird&#39;s-eye view by adopting only the preference bird&#39;s-eye view determined by the preference bird&#39;s-eye view determining means in each overlapping portion in which two bird&#39;s-eye views overlap; and a means for displaying, on the display unit, the synthesized bird&#39;s-eye view generated by the synthesized bird&#39;s-eye view generating means.  
         [0021]     The preference bird&#39;s-eye view determining means in the vehicle drive assistant system according to the sixth aspect comprises: for example, a means which picks up a difference between a bird&#39;s-eye view and other bird&#39;s-eye view in the overlapping portion in which two bird&#39;s-eye views overlap and determines a region in which a difference amount is larger than a predetermined amount as a difference region; and a means which calculates an integrated value of an edge intensity within the difference region between the two bird&#39;s-eye views and determines the bird&#39;s-eye view in which the integrated value of the edge intensity is larger as the preference bird&#39;s-eye view.  
         [0022]     It is preferable that the vehicle drive assistant system according to the first to sixth aspects comprises a determining means for determining whether or not an object having a height exists by comparing two bird&#39;s-eye views in each overlapping portion in which two bird&#39;s-eye views overlap each other, and a means for displaying a mark indicating the object having the height in the synthesized bird&#39;s-eye view if it is determined that the object having the height exists in at least one overlapping portion by the determining means.  
         [0023]     It is preferable that the vehicle drive assistant system according to the first to sixth aspects comprises a determining means for determining whether or not an object having a height exists by comparing two bird&#39;s-eye views in each overlapping portion in which two bird&#39;s-eye views overlap each other; and a means for producing an alarm sound if it is determined that the object having the height exists in at least one overlapping portion by the determining means.  
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0024]      FIG. 1  is a schematic view showing that in bird&#39;s-eye view, an object  200  having a height is projected to the ground such that its image is deformed on an extension line when a camera  1  and the object  200  are connected;  
         [0025]      FIG. 2  is a schematic view showing a projection image  200 L by a left side camera  1 L and a projection image  200 B by a rear camera  1 B when an object  200  having a height exists obliquely backward of the left rear end of a vehicle;  
         [0026]      FIG. 3  is a schematic view showing that a projection image  200 B by a rear camera  1 B exists in a left side camera region  5 L and a projection image  200 L by a left side camera  1 L exists in a rear camera region  5 B;  
         [0027]      FIG. 4  is a schematic view showing a camera  1  provided at the rear portion of a vehicle  100 ;  
         [0028]      FIG. 5  is a schematic view showing the relation among a camera coordinate system XYZ, a coordinate system Xbu, Ybu of an image pickup face  5  of a camera  1  and a world coordinate system XW, YW, ZW containing a two-dimensional ground coordinate system XW, Zw;  
         [0029]      FIG. 6  is a plan view showing an example of arrangement of cameras  1 F,  1 B,  1 L,  1 R;  
         [0030]      FIG. 7  is a side view of  FIG. 6 ;  
         [0031]      FIG. 8  is a schematic view showing bird&#39;s-eye views  10 F,  10 B,  10 L,  10 R obtained from images photographed with the respective cameras  1 F,  1 B,  1 L,  1 R;  
         [0032]      FIG. 9  is a schematic view showing that four bird&#39;s-eye views  10 F,  10 B,  10 L,  10 R are synthesized by converting three bird&#39;s-eye views  10 F,  10 L,  10 R to bird&#39;s-eye view coordinate of the rear camera  1 B by rotation and parallel translation with respect to the bird&#39;s-eye view  10 B to the rear view camera  1 B of  FIG. 8 ;  
         [0033]      FIG. 10  is a schematic view showing an example of a pectinate border line DBL for use in the embodiment 1 at an overlapping portion between the bird&#39;s-eye view  10 B and bird&#39;s-eye view  10 L;  
         [0034]      FIG. 11  is a schematic view showing an example of image at the overlapping portion after synthesis;  
         [0035]      FIG. 12  is a schematic view showing another example of pectinate border line DBL for use in the embodiment 1 at the overlapping portion between the bird&#39;s-eye view  10 B and bird&#39;s-eye view  10 L;  
         [0036]      FIG. 13  is a block diagram showing the electric configuration of a vehicle drive assistant system provided on a vehicle;  
         [0037]      FIG. 14  is a flow chart showing the procedure by the image processing unit  2 ;  
         [0038]      FIG. 15  is a flow chart showing the procedure by the image processing unit  2 ;  
         [0039]      FIG. 16  is a flow chart showing the procedure by the image processing unit  2 ;  
         [0040]      FIG. 17  is a flow chart showing the procedure by the image processing unit  2 ;  
         [0041]      FIG. 18  is a flow chart showing the detailed procedure of processing in step S 44  in  FIG. 17 ;  
         [0042]      FIG. 19   a  is a schematic diagram showing examples of gray images  40 L and  40 B;  
         [0043]      FIG. 19   b  is a schematic diagram showing a difference region between the gray regions  40 L and  40 B; and  
         [0044]      FIG. 19   c  is a schematic diagram showing an edge portion in the difference region of the gray image  40 L and an edge portion in the difference region of the gray image  40 B. 
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0045]     Hereinafter the preferred embodiments of the present invention will be described with reference to the accompanying drawings.  
       First Embodiment  
       [0046]     Description of generation method of bird&#39;s-eye view First, a method for generating a bird&#39;s-eye view from an image photographed by a camera will be described.  
         [0047]     Assume that a camera  1  is disposed to be directed obliquely backward at the rear portion of a vehicle  100  as shown in  FIG. 4 . Angles formed between the horizontal surface and the optical axis of the camera  1  includes two kinds, that is, an angle indicated with α and an angle indicated with β in  FIG. 4 . The a is generally called look-down angle or angle of depression. In this specification, the angle of β is assumed to be an inclination angle θ of the camera  1  to the horizontal surface.  
         [0048]      FIG. 5  shows the relationship among a camera coordinate system XYZ, a coordinate system Xbu, Ybu of an image pickup face  5  of the camera  1  and a world coordinate system XW, YW, ZW containing two-dimensional ground  
               [               x           y                 z         ]     =       [         1       0       0           0         cos   ⁢           ⁢   θ             -   sin     ⁢           ⁢   θ             0         sin   ⁢           ⁢   θ           cos   ⁢           ⁢   θ           ]     ⁢     {       [                 x   w               y   w                     z   w           ]     +     [               0           h                 0         ]       }               (   1   )             
coordinate system XW, ZW.  
         [0049]     In the camera coordinate system XYZ, when the optical center of a camera is supposed to be home position 0, the Z-axis is taken in the direction of the optical axis, the X-axis is taken in a direction perpendicular to the Z-axis and parallel to the ground surface and the Y-axis is taken as a direction perpendicular to the Z-axis and X-axis. In the coordinate system Xbu, Ybu of the image pickup image S, home position is set at the center of the image pickup face  5  and the Xb. axis is taken in the crosswise direction of the image pickup face  5  while the Ybu axis is taken in the lengthwise direction of the image pickup face  5 .  
         [0050]     In the world coordinate system XW, YW, ZW, an intersection between a vertical line passing through the home position  0  of the camera coordinate system XYZ and the ground surface is home position  0 w, the YW axis is taken in a direction perpendicular to the ground surface, the Xw axis is taken in a direction parallel to the X axis of the camera coordinate system XYZ and the ZW axis is taken in a direction perpendicular to the XW axis and Yw axis.  
         [0051]     The amount of parallel translation between the world coordinate system Xw, Yw, ZW and the camera coordinate system XYZ is [ 0 , h,  0 ] and the amount of rotation around the X-axis is 0.  
         [0052]     Therefore, conversion equation between the coordinates (x, y, z) of the camera coordinate system XYZ and the coordinates (xw, yw, zw) of the world coordinate system Xw, Yw, Zw is expressed in a following equation (1).  
         [0053]     Assuming that the focal distance of the camera  1  is f, the conversion equation between the coordinates (Xbu, ybu) of the coordinate system Xbu, Ybu of the image pickup face  5  and the coordinates (x, y, z) of the camera coordinate system XYZ can be expressed in a following equation (2).  
               [           x   bu               y   bu           ]     =     [           f   ⁢     x   z                 f   ⁢     y   z             ]             (   2   )             
 
 A conversion equation (3) between the coordinates (Xbu, Ybu) of the coordinate system Xbu, Ybu of the image pickup face  5  and the coordinates (xw, Zw) of the two-dimensional ground coordinate system XW, ZW is obtained from the above-described equations (1), (2).  
               [           x   bu               y   bu           ]     =     [             f   ⁢           ⁢     x   w           h   ⁢           ⁢   sin   ⁢           ⁢   θ     +       z   w     ⁢   cos   ⁢           ⁢   θ                       (       h   ⁢           ⁢   cos   ⁢           ⁢   θ     -       z   w     ⁢   sin   ⁢           ⁢   θ       )     ⁢   f         h   ⁢           ⁢   sin   ⁢           ⁢   θ     +       z   w     ⁢   cos   ⁢           ⁢   θ               ]             (   3   )             
 
 Projection from the two-dimensional ground coordinate system XW, ZW to the bird&#39;s-eye view coordinate system Xau, Yau of a virtual camera is carried out by parallel translation. Assuming that the focal distance of the camera  1  is f and the height of the virtual camera is H, the conversion equation between the coordinates (xe, zw) of the two-dimensional ground coordinate system Xw, Zw and the coordinates (Xau, yau) of the bird&#39;s eye coordinate system Xau, Yau is expressed in a following equation (4). The height H of the virtual camera is set up preliminarily.  
               [           x   au               y   au           ]     =       f   H     ⁡     [           x   w               z   w           ]               (   4   )             
 
         [0054]     A following equation (5) is obtained from the aforementioned equation (4).  
               [           x   w               z   w           ]     =       H   f     ⁡     [           x   au               y   au           ]               (   5   )             
 
 By assigning the obtained equation (5) to the aforementioned equation (3), a following equation (6) is obtained.  
               [           x   bu               y   bu           ]     =     [             f   ⁢           ⁢   H   ⁢           ⁢     x   au           f   ⁢           ⁢   h   ⁢           ⁢   sin   ⁢           ⁢   θ     +     H   ⁢           ⁢     y   au     ⁢   cos   ⁢           ⁢   θ                     f   ⁡     (       f   ⁢           ⁢   h   ⁢           ⁢   cos   ⁢           ⁢   θ     -     H   ⁢           ⁢     y   au     ⁢   sin   ⁢           ⁢   θ       )           f   ⁢           ⁢   h   ⁢           ⁢   sin   ⁢           ⁢   θ     +     H   ⁢           ⁢     y   au     ⁢   cos   ⁢           ⁢   θ               ]             (   6   )             
 
 An equation (7) for converting the coordinates (xbu, ybu) of inputted image I to the coordinates (xau, yau) of bird&#39;s eye coordinate system XaU, Yau is obtained from the aforementioned equation (6).  
               [           x   au               y   au           ]     =     [               x   bu     ⁡     (       f   ⁢           ⁢   h   ⁢           ⁢   sin   ⁢           ⁢   θ     +     H   ⁢           ⁢     y   au     ⁢   cos   ⁢           ⁢   θ       )         f   ⁢           ⁢   H                   f   ⁢           ⁢     h   ⁡     (       f   ⁢           ⁢   cos   ⁢           ⁢   θ     -       y   bu     ⁢   sin   ⁢           ⁢   θ       )           H   ⁡     (       f   ⁢           ⁢   sin   ⁢           ⁢   θ     +       y   bu     ⁢   cos   ⁢           ⁢   θ       )               ]             (   7   )             
 
 The inputted image I is converted to bird&#39;s-eye view using the aforementioned equation (7). 
 
         [0055]     Description of basic concept of generation method of all around bird&#39;s-eye view  
         [0056]     FIGS.  6 ,and  7  show cameras provided on a vehicle. The vehicle is provided with cameras (image pickup devices)  1 F,  1 B,  1 L,  1 R at its front portion, rear portion, left side portion and right side portion, respectively. The camera  1 F is disposed to be directed forward obliquely downward, the camera  1 B is disposed to be directed backward obliquely downward, the camera  1 L is disposed to be directed leftward obliquely downward and the camera  1 R is disposed to be directed rightward obliquely downward.  
         [0057]     As shown in  FIG. 8 , bird&#39;s-eye views  10 F,  10 B,  10 L, and  10 R are generated from images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R. Next, the bird&#39;s-eye views  10 F,  10 B,  10 L, and  10 R generated for the respective cameras  1 F,  1 B,  1 L, and  1 R as shown in  FIG. 9  are converted to a bird&#39;s-eye view coordinate of the rear camera  1 B by rotation and parallel translation of three bird&#39;s-eye views  10 F,  10 L, and  10 R with respect to the bird&#39;s-eye view  10 B to the rear camera  1 B. In this case, portions in which the two bird&#39;s-eye views overlap each other are generated as shown in  FIG. 9 . The feature of this embodiment exists in how the both bird&#39;s-eye views are synthesized.  
         [0058]     At an overlapping portion  20   FL  between the bird&#39;s-eye view  10 F and bird&#39;s-eye view  10 L, a line connecting its upper left crest with its lower right crest is assumed to be an ordinary border line DFL. At an overlapping portion  20   FR  between the bird&#39;s-eye view  10 F and bird&#39;s-eye view  10 R, a line connecting the upper right crest with the lower left crest is assumed to be an ordinary border line DFR. At an overlapping portion  20   BL  between the bird&#39;s-eye view  10 B and the bird&#39;s-eye view  10 L, a line connecting the upper right crest with the lower left crest is assumed to be an ordinary border line DBL. At an overlapping portion  20   BR  between the bird&#39;s-eye view  10 B and the bird&#39;s-eye view  10 R, a line connecting the upper left crest with the lower right crest is assumed to be an ordinary border line DBR. Because actually, the overlapping portion is not formed in a rectangular shape, usually, an appropriate border line dividing the overlapping portion into two portions is assumed to be an ordinary border line.  
         [0059]     Conventionally, at the overlapping portion in which two bird&#39;s-eye views overlap each other, one bird&#39;s-eye view is adopted in one region separated by the ordinary border line while in the other region, the other bird&#39;s-eye view is adopted. More specifically, at the overlapping portion  20   BL  in which the bird&#39;s-eye view  10 B and bird&#39;s-eye view  10 L overlap each other, the bird&#39;s-eye view  10 L is adopted in a region above the ordinary border line DBL and the bird&#39;s-eye view  10 B is adopted in a region below the ordinary border line DBL. Thus, there is such a problem that any object having a height disappears on a synthesized bird&#39;s-eye view.  
         [0060]     According to the first embodiment, a pectinate border line in which two different regions appear alternately in the form of a slit in two regions divided by the ordinary border line is provided at each overlapping portion. One bird&#39;s-eye view is adopted in one region separated by the pectinate border line while the other bird&#39;s-eye view is adopted in the other region.  
         [0061]     For example, a pectinate border line DBL in which teeth are arranged in the direction of the ordinary border line DBL while the teeth are parallel to a direction perpendicular to a monitor screen at the overlapping portion  20   BL  between the bird&#39;s-eye view  10 B and bird&#39;s-eye view  10 L as shown in  FIG. 10  is used. Then, the bird&#39;s-eye view  10 L is adopted in a region SL above the pectinate border line DBL within the overlapping portion  20   BL  while the bird&#39;s-eye view  10 B is adopted in a region SB below the pectinate border line DBL.  
         [0062]     When this synthesis method is used, for example, if an object  200  having a height exists on the left side obliquely backward of the left rear end of a vehicle as shown in  FIG. 2 , that object  200  appears on all around bird&#39;s-eye view after the synthesis as shown in  FIG. 11 . In  FIG. 11 , DBL indicates a pectinate border line and  200 L indicates a bird&#39;s-eye view of the object  200  obtained from an image photographed by the left side camera  1 L while  200 B indicates a bird&#39;s-eye view of the object  200  obtained from an image photographed by the rear camera  1 B. As evident from  FIG. 11 , the object  200  can be recognized clearly because a projection image of the object  200  having a height does not disappear and its object image is not blended with the background. Further, because the object image obtained from the image photographed by the left side camera  1 L and the object image obtained from the image photographed by the rear camera  1 B appear alternately, these two object images can be recognized easily as a single object image.  
         [0063]     In the meantime, as the pectinate border line, it is permissible to use a pectinate border line in which the teeth are arranged in the direction of the ordinary border line while the teeth are arranged in parallel to the horizontal direction of the monitor screen as shown in  FIG. 12 . Further, a pectinate border line in which the teeth intersect the ordinary border line may be used. Further, a pectinate border line in which the teeth are parallel to the ordinary border line may be used.  
         [0064]     The length of and interval between the teeth on the pectinate border line are preferably adjusted depending on the resolution of the monitor and such that the double image is not displayed easily.  
         [0065]     Description of a specific example of generation method of all around bird&#39;s-eye view  
         [0066]     The coordinate of an inputted image I (image produced by lens distortion correction to an image photographed by camera) on bird&#39;s-eye view corresponding to the coordinate of each pixel can be preliminarily obtained from the equation (7).  
         [0067]     Conversion of coordinate on bird&#39;s-eye views corresponding to the respective cameras  1 F,  1 B,  1 L, and  1 R to coordinate on the all around bird&#39;s-eye view is carried out by a predetermined rotation and a predetermined parallel translation. That is, all conversion parameters for converting the inputted image I after correction of distortion of photographed image by each camera to bird&#39;s-eye view and further converting the obtained bird&#39;s-eye view to all around bird&#39;s-eye view are of a fixed value. Thus, the coordinate of the inputted image I (image obtained by correcting lens distortion) obtained from the respective cameras  1 F,  1 B,  1 L, and  1 R on the all around bird&#39;s-eye view corresponding to the coordinate of each pixel can be preliminarily obtained.  
         [0068]     Because in the all around bird&#39;s-eye view, a region in which two bird&#39;s-eye views overlap each other and a region divided by the aforementioned pectinate border line at each overlapping portion are already known, which of the two bird&#39;s-eye views is adopted as each coordinate within each overlapping portion on the all around bird&#39;s eye view can be determined preliminarily.  
         [0069]     In this embodiment, a coordinate reverse conversion table indicating which image of which pixel is to be allocated of images of respective pixels in the inputted image I (image obtained by correcting the lens distortion) obtained from the respective cameras  1 F,  1 B,  1 L, and  1 R is prepared preliminarily for each coordinate on the all around bird&#39;s-eye view. Data for specifying an image to be embedded into each coordinate on the all around bird&#39;s-eye view is memorized in the coordinate reverse conversion table. The data for specifying the-image to be allocated to each coordinate on the all around bird&#39;s-eye view comprises data for specifying a camera and data (coordinate data) for specifying the pixel position of the inputted image I (image obtained by correcting lens distortion) obtained from a camera. As the inputted image I, images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R may be used by considering lens distortion.  
         [0070]     Description of the structure of vehicle drive assistant system  FIG. 13  shows the electric structure of the vehicle drive assistant system provided on a vehicle. The. vehicle drive assistant system is provided with four cameras  1 L,  1 R,  1 F, and  1 B, an image processing unit  2  for generating an all around bird&#39;s-eye view from images photographed by the cameras  1 L,  1 R,  1 F, and  1 B and a monitor (display unit)  3  which displays an all around bird&#39;s-eye view generated by the image processing unit  2 . The image processing unit  2  includes a memory which memorizes the aforementioned coordinate reverse conversion table.  
         [0071]     As the camera  1 L,  1 R,  1 F, and  1 B, for example, a CCD camera is used. The image processing unit  2  is constituted of, for example, a micro computer. As the monitor  3 , for example, monitor of navigation system is used.  
         [0072]     The image processing unit  2  generates the all around bird&#39;s-eye view using images photographed by the cameras  1 L,  1 R,  1 F, and  1 B and the coordinate reverse conversion table. The all around bird&#39;s-eye view generated by the image processing unit  2  is displayed on the monitor  3 .  
         [0073]      FIG. 14  shows the procedure by the image processing unit  2 . First, images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R are read (step S 1 ). Next, lens distortion correction is carried out to each read image (step S 2 ). Hereinafter, an image obtained by the lens distortion correction is called an inputted image I.  
         [0074]     Next, the all around bird&#39;s-eye view is generated using the inputted image I obtained from the images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R and the coordinate reverse conversion table (step S 3 ). The obtained all around bird&#39;s-eye view is displayed on the monitor  3  (step S 4 ). Then, the procedure returns to step S 1 .  
       Second Embodiment  
       [0075]     The electric configuration of the vehicle drive assistant system of the second embodiment is the same as that of the first embodiment. The processing content of the image processing unit  2  is different between the second embodiment and the first embodiment. According to the second embodiment, the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front/rear cameras are displayed alternately on the monitor. The all around bird&#39;s-eye view taking preference to the side camera refers to an all around bird&#39;s-eye view obtained by adopting only a bird&#39;s-eye view obtained from images photographed by the right and left cameras at each overlapping portion in which two bird&#39;s-eye views overlap each other in an all around bird&#39;s-eye view coordinate system shown in  FIG. 9 . More specifically, it refers to an all around bird&#39;s-eye view obtained by adopting only the bird&#39;s-eye image  10 L obtained from the left side camera  1 L at the overlapping portions  20   FL  and  20   BL  in  FIG. 9  and only the bird&#39;s-eye view  10 R obtained from the right side camera  1 R at the overlapping portions  20   FR  and  20   BR  in  FIG. 9 .  
         [0076]     The all around bird&#39;s-eye view taking preference to the front and rear cameras refers to an all around bird&#39;s-eye view obtained by adopting only a bird&#39;s-eye view obtained from images photographed by the front and rear cameras at each overlapping portion in which two bird&#39;s-eye views overlap each other in the all around bird&#39;s-eye view coordinate system shown in  FIG. 9 . More specifically, it refers to an all around bird&#39;s-eye view obtained by adopting only the bird&#39;s-eye view  1 OF obtained by the front camera iF at the overlapping portions  20   FL  and  20   FR  in  FIG. 9  and only the bird&#39;s-eye view  10 B obtained by the rear camera  1 B at the overlapping portions  20   BL  and  20   BR  in  FIG. 9 .  
         [0077]     The image processing unit  2  comprises a first coordinate reverse conversion table for generating the all around bird&#39;s-eye view taking preference to the side cameras and a second coordinate reverse conversion table for generating the all around bird&#39;s-eye view taking preference to the front and rear cameras as the coordinate reverse conversion table.  
         [0078]      FIG. 15  shows the procedure by the image processing unit  2 . First, the flag F is reset (F=0) (step S 11 ). Images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R are read (step S 12 ). Next, lens distortion correction is carried out to each read image (step S 13 ). Hereinafter, the image obtained by the lens distortion correction is called the inputted image I.  
         [0079]     Next, whether or not the flag F is set is determined (step S 14 ). If the flag F is reset (F=0), after the flag F is set (F=1) (step S 15 ), an all around bird&#39;s-eye view taking preference to the side camera is generated using the inputted image I and the first coordinate reverse conversion table (step S 16 ). The obtained all around bird&#39;s-eye view taking preference to the side camera is displayed on the monitor  3  (step S 17 ). Then, the procedure returns to step S 12 .  
         [0080]     If the flag F is set in the above-described step S 14  (F=1), after the flag F is reset (F=0) (step  18 ), an all around bird&#39;s-eye view taking preference to the front and rear cameras is generated using the inputted image I and the second coordinate reverse conversion table (step S 19 ). The obtained all around bird&#39;s-eye view taking preference to the front and rear cameras is displayed on the monitor  3  (step S 20 ). Then, the procedure returns to step S 12 .  
         [0081]     Because the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front and rear cameras are displayed alternately on the monitor in the second embodiment, if the object  200  having a height exists on the left side obliquely backward of the left rear end of a vehicle as shown in  FIG. 2 , for example, a projection image of the object  200  having a height does not disappear but displayed. Further, because the synthesized bird&#39;s-eye view taking preference to the side camera and the synthesized bird&#39;s-eye view taking preference to the front/rear cameras are directed in different directions, the projection image of this object  200  having a height looks to move at a timing when those images are changed over. Thus, a vehicle driver can recognize the object  200  more easily.  
         [0082]     Although it is assumed that the fetch-in interval of photographed images is relatively long in  FIG. 15 , if the fetch-in interval of the photographed images is as short as for every frame, for example, the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye image taking preference to the front/rear cameras may be changed over for display every several frames (for example, every 15 frames).  
         [0083]     Although the first coordinate reverse conversion table for generating the all around bird&#39;s-eye view taking preference to the side camera and the second coordinate reverse conversion table for generating the all around bird&#39;s-eye view taking preference to the front and rear cameras are provided as the coordinate reverse conversion table, it is permissible to use one coordinate reverse conversion table instead of these two coordinate reverse conversion tables. In this case, for example, it is permissible to memorize data (data for specifying a camera and coordinate data) indicating the pixel positions corresponding to both the bird&#39;s-eye views  10 L and  10 B for each coordinate within an overlapping portion between the bird&#39;s-eye view  10 L obtained from the left side camera  1 L and the bird&#39;s-eye view  10 B obtained from the rear camera  1 B, adopt data indicating the pixel position corresponding to the bird&#39;s-eye view  10 L when generating the all around bird&#39;s-eye view taking preference to the side camera and adopt data indicating the pixel position corresponding to the bird&#39;s-eye view  10 B when generating the all around bird&#39;s-eye view taking preference to the front/rear cameras.  
         [0084]     If the object  200  having a height exists on the left side obliquely backward of the left rear end of a vehicle when the cameras  1 F,  1 B,  1 L, and  1 R are provided on the front, rear, right and left sides of the vehicle as shown in  FIG. 2 , a projection image by the left side camera  1 L turns to  200 L and a projection image by the rear camera  1 B turns to  200 B.  
         [0085]     If both the projection images  200 L and  200 B exist at an overlapping portion between a bird&#39;s-eye view obtained from the photographed image by the left side camera  1 L and a bird&#39;s-eye view obtained from the photographed image by the rear camera  1 B in the all around bird&#39;s-eye view coordinate system shown in  FIG. 9 , those projection images  200 L and  200 B appear at different positions. Therefore, both the projection images  200 L and  200 B are detected as a difference value when a difference between both the bird&#39;s-eye views is obtained at this overlapping portion.  
         [0086]     When at an overlapping portion in which two bird&#39;s-eye views overlap each other, a difference between both the gray images is obtained after those bird&#39;s-eye views are converted to gray images, a difference region in which an absolute value of the difference value is over a predetermined threshold value is extracted if an object (obstacle) having a height exists. Therefore, it is possible to determine whether or not any obstacle having a height exists in each overlapping portion depending on whether or not the difference region is extracted from each overlapping portion.  
         [0087]     If whether or not an object (obstacle) having a height exists at each overlapping portion in which the bird&#39;s-eye views overlap each other in the all around bird&#39;s-eye view coordinate system shown in  FIG. 9  and any obstacle exists in at least one overlapping portion, the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front/rear cameras are generated alternately for display and if no obstacle exists in any overlapping portion, a predetermined one all around bird&#39;s-eye view of the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front and rear cameras may be generated and displayed.  
         [0088]     If any obstacle exists in at least one overlapping portion, the bird&#39;s-eye view taking preference to the side camera and the bird&#39;s-eye view taking preference to the front/rear cameras may be generated and displayed alternately for only an overlapping portion in which the obstacle exists while the same kind of the bird&#39;s-eye view may be generated for the other overlapping portion and displayed.  
       Third Embodiment  
       [0089]     Although according to the second embodiment, the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front/rear cameras are displayed on the monitor alternately, the all around bird&#39;s-eye view taking preference to the side camera and the all around bird&#39;s-eye view taking preference to the front/rear cameras may be changed over depending on the travel condition of a vehicle.  
         [0090]     If the object  200  having a height exists on the left side obliquely backward of the left rear end of the vehicle as shown in  FIG. 2 , this object  200  moves out of the photographing area of the rear camera  1 B when the vehicle backs up straight. Then, in this case, the all around bird&#39;s-eye view taking preference to the side camera is generated and displayed. On the other hand, if the object  200  having a height exists on the left side obliquely backward of the left rear end of the vehicle as shown in  FIG. 2 , this object  200  moves out of the photographing area of the left side camera  1 L when the vehicle backs up while curving to the left obliquely backward. Then, in such a case, the all around bird&#39;s-eye view taking preference to the front/rear cameras is generated and displayed.  
         [0091]     The electric structure of the vehicle drive assistant system according to the third embodiment is equal to that of the first embodiment. In the meantime, the travel condition of the vehicle is judged based on for example, vehicle gear sensor, operating direction of the steering wheel, vehicle velocity pulse and the like.  
         [0092]     The image processing unit  2  includes a first coordinate reverse conversion table for generating an all around bird&#39;s-eye image taking preference to the side camera and a second coordinate reverse conversion table for generating the all around bird&#39;s-eye view taking preference to the front/rear cameras as the coordinate reverse conversion table.  
         [0093]      FIG. 16  shows the procedure by the image processing unit  2 . First, images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R are read (step S 31 ). Next, lens distortion correction is carried out to each read image (step S 32 ). Hereinafter, an image obtained by the lens distortion correction is called the inputted image I.  
         [0094]     Next, the travel condition of a vehicle is judged (step S 33 ). More specifically, whether a vehicle is in a first travel condition in which it moves forward or backs up straight or in a second travel condition in which it moves while curving obliquely forward or backs up while curving obliquely backward is determined.  
         [0095]     If it is determined that the travel condition of the vehicle is the first travel condition, an all around bird&#39;s-eye view taking preference to the side camera is generated using the inputted image I and first coordinate reverse conversion table (step S 34 ). The obtained all around bird&#39;s-eye view taking preference to the side camera is displayed on the monitor  3  (step S 35 ). Then, the procedure returns to step S 31 .  
         [0096]     If it is determined that the travel condition of the vehicle is the second travel condition in step S 33 , the all around bird&#39;s-eye view taking preference to the front/rear cameras is generated using the inputted image I and the second coordinate reverse conversion table (step S 36 ). The obtained all around bird&#39;s-eye view taking preference to the front/rear cameras is displayed on the monitor  3  (step S 37 ). Then, the procedure returns to step S 31 .  
         [0097]     Also in the third embodiment, as in the second embodiment, one coordinate reverse conversion table may be used instead of the first coordinate reverse conversion table and second coordinate reverse conversion table.  
       Fourth Embodiment  
       [0098]     According to the fourth embodiment, a bird&#39;s-eye view in which the obstacle (object having a height) appears larger is determined among two bird&#39;s-eye views which overlap with each other in each overlapping portion in which the two bird&#39;s-eye views overlap each other in the all around bird&#39;s-eye view coordinate system shown in  FIG. 9 , and only a bird&#39;s-eye view in which the obstacle appears larger is adopted when that overlapping portion is synthesized.  
         [0099]     The electric configuration of the vehicle drive assistant system according to the fourth embodiment is equal to that of the first embodiment. The image processing unit  2  includes a coordinate reverse conversion table. As the coordinate reverse conversion table, one reverse conversion table is prepared. Two kinds of data indicating the pixel positions corresponding to two bird&#39;s-eye views are memorized for each coordinate in the overlapping portion in which two bird&#39;s-eye views overlap each other.  
         [0100]     For example, data indicating the pixel positions corresponding to the bird&#39;s-eye views  10 L and  10 B are memorized for each coordinate in the overlapping portion between the bird&#39;s-eye view  10 L obtained from the left side camera  1 L and the bird&#39;s-eye view  10 B obtained form the rear camera  1 B. If the obstacle appears larger in, for example, the bird&#39;s-eye view  10 L among both the bird&#39;s-eye views  10 L and  10 B in this overlapping portion, data indicating the pixel position corresponding to the bird&#39;s-eye view  10 L is selected.  
         [0101]      FIG. 17  shows the procedure by the image processing unit  2 . First, images photographed by the respective cameras  1 F,  1 B,  1 L, and  1 R are read (step S 41 ). Next, lens distortion correction is carried out to each read image (step S 42 ). Hereinafter, the image obtained by the lens distortion correction is called the inputted image I.  
         [0102]     Next, a bird&#39;s-eye view at a portion in which two bird&#39;s-eye views overlap on the all around bird&#39;s-eye view coordinate is generated for each of the cameras  1 F,  1 B,  1 L, and  1 R using the inputted image I and the coordinate reverse conversion table (step S 43 ).  
         [0103]     Which of the two bird&#39;s-eye views is taken with preference is determined for each overlapping portion in which two bird&#39;s-eye views overlap based on the bird&#39;s-eye view obtained in step S 43  (step S 44 ). That is, a preference bird&#39;s-eye view is determined for each overlapping portion. The detail of this processing will be described later.  
         [0104]     Next, an all around bird&#39;s-eye view adopting only the bird&#39;s-eye view determined to be taken with preference in step S 44  in each overlapping portion is generated using a determination result in step S 44 , the inputted image I and the coordinate reverse conversion table (step S 45 ). The obtained all around bird&#39;s-eye view is displayed on the monitor  3  (step S 46 ). Then, the procedure returns to step S 41 .  
         [0105]      FIG. 18  shows the detailed procedure of processing of the aforementioned step S 44 . An overlapping portion between the bird&#39;s-eye view  10 L obtained from the left side camera IL and the bird&#39;s-eye view  10 B obtained from the rear camera  1 B will be exemplified. Among the bird&#39;s-eye views at the overlapping portion, an image obtained from an image photographed by the left side camera  1 L is expressed in  30 L and an image obtained from the rear camera  1 B is expressed in  30 B.  
         [0106]     The bird&#39;s-eye views  30 L and  30 B at the overlapping portion are converted to gray images  40 L and  40 B (step S 51 ).  FIG. 19   a  shows an example of the gray images  40 L and  40 B.  
         [0107]     A difference region between the gray images  40 L and  40 B is obtained (step S 52 ). More specifically, a difference between the gray images  40 L and  40 B is obtained and a region in which an absolute value of a difference value is over a predetermined threshold value is regarded as the difference region. If the gray images  40 L and  40 B are as indicated in  FIG. 19   a , the difference region is as indicated in  FIG. 19   b.    
         [0108]     Edge extraction processing is carried out within the difference region obtained in step S 52  for each of the gray images  40 L and  40 B (step S 53 ). That is, edge intensity is calculated for each pixel within the difference region for each of the gray images  40 L and  40 B. Next, a sum of the edge intensities in the difference region is calculated for each of the gray images  40 L and  40 B (step S 54 ). Then, a bird&#39;s-eye view having a larger sum of the edge intensities is determined to be preference bird&#39;s-eye view (step S 55 ). In the meantime, it is permissible to use the number of detected edges and area of a region surrounded by the edge portions instead of the sum (integrated value) of the edge intensities.  
         [0109]     If the gray images  40 L and  40 B are as indicated in  FIG. 19   a , the edge portion within the difference region in the gray. image  40 L is as indicated in a left diagram of  FIG. 19   c  and the edge portion within the difference region in the gray image  40 B is as indicated in a right diagram of  FIG. 19   c . Because the area of the edge portion within the difference region in the gray image  40 L is larger than the area of the edge portion within the difference region in the gray image  40 B, a bird&#39;s-eye view obtained form the left side camera  1 L is regarded as the preference bird&#39;s-eye view.  
       Fifth Embodiment  
       [0110]     As described above, whether or not an object (obstacle) having a height exists in the overlapping portion is determined depending on whether or not a difference region is extracted for each overlapping portion in which two bird&#39;s-eye views overlap each other.  
         [0111]     In the above first, second, third and fourth embodiments, whether or not any obstacle exists in each overlapping portion is determined and if any obstacle exists in at least one overlapping portion, preferably, a mark indicating the obstacle is displayed on the all around bird&#39;s-eye view or an alarm sound is produced. As the mark indicating the obstacle, for example, a mark which surrounds the obstacle is used.