Abstract:
This vehicle-mounted image recognition device achieves improved performance in flashing light source recognition at low cost and without loss of performance in image recognition applications such as preceding vehicle recognition and white line recognition. At staggered exposure timing, imaging is performed alternately with a first imaging unit and a second imaging unit, which form a pair in a vehicle-mounted camera. A flashing light source is detected from one or both of the first image obtained from the first imaging unit and a second image obtained from the second imaging unit, and the following times, exposure of the first imaging unit and the second imaging unit is matched to when the flashing light source is on. Thereby, flashing light sources are recognized with high accuracy without loss of performance of the image recognition function.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to an in-vehicle camera, and particularly to an in-vehicle image recognition device having a function of simultaneously recognizing a plurality of targets such as a preceding vehicle, a white lane, a pedestrian, and a light source. 
       BACKGROUND ART 
       [0002]    In recent years, light sources used in a traffic signal and a traffic sign of an electric light display type are replaced from the conventional incandescent lamps to LEDs. The LED has a merit on less power consumption and long life span compared to the incandescent lamp. In addition, there is no pseudo lighting considered as emission caused by the reflection of sunlight, so that the visibility is high and also the stability is improved. 
         [0003]    The LED used in the traffic signal and the traffic sign is flickered to be repeatedly on and off at a high speed when being turned on. A cycle of the flickering is 100 Hz in a region of 50 Hz commercial power source or 120 Hz in a region of 60 Hz commercial power source. The flickering is performed at a high speed, and thus it appears to person as always turned on. 
         [0004]    In a case where the exposure timing of an in-vehicle camera is matched to the OFF state of the flickering light source, the emission light of the light source is not recorded in the image obtained from the camera. In addition, in a case where the capturing cycle of the camera is close to an integer times the cycle of the flickering light source, the above problem continuously occurs in a plurality of frames. 
         [0005]    In order to solve the above problem, for example, PTL 1 discloses a technique in which a number of imaging units are provided to capture an image while deviating the exposure timings in a time shorter than a flickering cycle of the flickering light source and to observe a change in brightness of the light source in order to recognize the flickering light source. 
       CITATION LIST 
     Patent Literature 
       [0006]    PTL 1: Japanese Patent Application Laid-Open No. 2007-286943 
       SUMMARY OF INVENTION 
     Technical Problem 
       [0007]    However, in the conventional technique described above, an image is acquired in a cycle Tled shorter than the flickering cycle of the light source, and a recognition process is performed on each image, thereby significantly increasing a processing load. In addition, since a number of images are transferred from an imaging unit to a recognition unit, a transfer amount of a bus may be significantly expanded. An in-vehicle camera having an image recognition function is requested for simultaneously recognizing a number of objects such as a preceding vehicle, a pedestrian, and a white lane on a road. Therefore, when the processing load and the bus transfer amount are increased, it becomes difficult to be compatible with the other recognition processing in addition, since more imaging units are required, the cost is increased. 
         [0008]    The invention has been made in view of the above problems, and an object thereof is to provide an in-vehicle imaging device which can improve a performance of recognizing the flickering light source at a low cost without causing a degradation of an image recognition application such as a preceding vehicle recognition and a white lane recognition. 
       Solution to Problem 
       [0009]    There is provided an in-vehicle image recognition device according to the present invention in order to solve the above issue which processes an image captured by a pair of imaging units, wherein the same light source is captured by the pair of imaging units in a capturing cycle which is an integer times a predetermined flickering cycle, and a capturing cycle where exposure timings of the pair of imaging units are deviated by a half of the flickering cycle, and wherein, in a case where a difference between a pixel value of the light source captured by one imaging unit and a pixel value of the light source captured by the other imaging unit is equal to or more than a threshold, the light source is determined as a flickering light source. 
       Advantageous Effects of Invention 
       [0010]    According to the invention, the visibility of a flickering light source can be improved at a low cost without causing a degradation of the performance of an image recognition application such as a preceding vehicle recognition and a white lane recognition. Further, other objects, configurations, and effects will become clear through the explanation of the following embodiments. 
     
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         [0011]    [ FIG. 1 ]  FIG. 1  is a block diagram for describing functions of an in-vehicle image recognition device. 
           [0012]    [ FIG. 2 ]  FIG. 2  is a diagram illustrating an example of flickering of a flickering light source and exposure timings of a first imaging unit and a second imaging unit. 
           [0013]    [ FIG. 3 ]  FIG. 3  is a flowchart illustrating an exemplary operation of an exposure timing adjustment unit. 
       
    
    
     DESCRIPTION OF EMBODIMENTS 
       [0014]    Next, embodiments of the invention will be described below using the drawings.  FIG. 1  is a functional block diagram illustrating an example of an in-vehicle image recognition device in this embodiment. 
         [0015]    An in-vehicle image recognition device  10  is a device which alternately captures an image at a different timing using a first imaging unit  11   a  and a second imaging unit  11   b  which are paired in an in-vehicle camera, detects a flickering light source from one or both of a first image  12   a  obtained from the first imaging unit  11   a  and a second image  12   b  obtained from the second imaging unit  11   b,  and performs exposure using the first imaging unit  11   a  and the second imaging unit  11   b  in synchronization with timing of turning on the flickering light source from next time, so that the flickering light source is recognized with high accuracy without causing degradation of the performance of an image recognition function. 
         [0016]    The in-vehicle image recognition device  10  is a stereo camera which is mounted in a vehicle and, as described in the drawing, provided with the first imaging unit  11   a  and the second imaging unit  11   b,  a distance data generation unit  13  which generates distance information based on the first image  12   a  and the second image  12   b  acquired by the first imaging unit  11   a  and the second imaging unit  11   b,  a distance data storage unit  14  which stores the distance information indicating a distance to a capturing target, an image recognition unit  15  which performs image recognition, a light source recognition unit  16  which performs recognition of a light source, an exposure timing adjustment unit  17  which adjusts each timing of exposure, an external information reception unit  18  which receives external information, and a first exposure control unit  19   a  and a second exposure control unit  19   b  which control the exposure of the first imaging unit  11   a  and the second imaging unit  11   b.    
         [0017]    The first imaging unit  11   a  and the second imaging unit  11   b  are directed in the same direction and capture the same range. For example, the imaging units are separately disposed on right and left in the vicinity of a rear-view mirror of the vehicle to capture the forward side of the vehicle through a front class 
         [0018]    The distance data generation unit  13  generates the distance information such as a parallax image stereoscopically obtained by the first image  12   a  and the second image  12   b.  The image recognition unit  15  recognizes an object body such as a traffic signal, a traffic sign, a preceding vehicle, a pedestrian, and a bicycle using at least one of the first image  12   a  and the second image  12   b.  As a recognition method, the well-known pattern matching may be used. 
         [0019]    The light source recognition unit  16  recognizes a light source from at least one of the first image  12   a  and the second image  12   b,  and recognizes whether the recognized light source is a flickering light source. The exposure timing adjustment unit  17  adjusts the exposure timings of the first imaging unit  11   a  and the second imaging unit  11   b  based on an image recognition result of the image recognition unit  15  and the external information received by the external information reception unit  18 . The first exposure control unit  19   a  and the second exposure control unit  19   b  control the exposure timings of the first imaging unit  11   a  and the second imaging unit  11   b  according to, for example, an instruction from the exposure timing adjustment unit  17 . In addition, the exposure timings are adjusted to be exposure parameters optimized to the image recognition based on the information of the images acquired from the first imaging unit  11   a  and the second imaging unit  11   b.  The external information reception unit  18  is connected to a vehicle network  20  and acquires the external information such as vehicle speed information  21 , a yaw rate  22 , GPS information  23 , time information  24 , and map data  25  using the external information reception unit  18 . 
         [0020]    When capturing the traffic signal using an LED as the flickering light source and an indicator such as the traffic sign of an electric light display type, the vehicle image recognition device  10  captures the images at exposure timings of the first imaging unit  11   a  and the second imaging unit  11   b  in synchronization with a flickering period of the LED. 
         [0021]    Specifically, capturing cycles Tc of the first imaging unit  11   a  and the second imaging unit  11   b  are set to be Nc times (an integer) of a flickering cycle Tl, and the exposure timing of the first imaging unit  11   a  and the exposure timing of the second imaging unit  11   b  are set to be deviated by Ns (an integer)+0.5 times the flickering cycle, so that the first image and the second image deviated by a half cycle with respect to the flickering cycle are acquired. 
         [0022]    Therefore, the light source in the ON state is recorded in at least any one of the first image  12   a  and the second image  12   b.    
         [0023]    The light source recognition unit  16  performs a recognition process in which the light source is extracted from the first image  12   a  and the second image  12   b.  Then, in a case where there is a difference equal to or more than a predetermined value between pixel values of the light source extracted from the first image  12   a  and the second image  12   b,  it is determined that the subject light source is a flickering light source. In a case where the light source recognition unit  16  determines that the light source is a flickering light source, the exposure timing adjustment unit  17  changes the exposure timing such that the exposure timing of one imaging unit is changed to be matched to the exposure timing of the other imaging unit in which the flickering light source is brightly captured among the first imaging unit  11   a  and the second imaging unit  11   b,  and adjusts the exposure timings of the first imaging unit  11   a  and the second imaging unit  11   b  to be synchronized. The distance data generation unit generates distance data indicating a distance from the vehicle to the light source based on the first image and the second image captured by the first imaging unit and the second imaging unit. 
         [0024]    Next, an example of adjusting the exposure timing of the invention will be described using  FIG. 2 .  FIG. 2  is a diagram illustrating an example of flickering of the flickering light source and exposure timings of the first imaging unit and the second imaging unit. As illustrated in  FIG. 2 , in a case where the brightness of the LED is increased or decreased in the flickering cycle Tl and the flickering light source emits the light, the exposure timing adjustment unit  17  sets the capturing cycles Tc of the first imaging unit  11   a  and the second imaging unit  11   b  to be Nc (an integer) times the flickering cycle Tl. Therefore, even the flickering light source can be captured in the same brightness in every frame, and the brightness of the flickering light source illuminated on each frame can be made constant. 
         [0025]    Then, the exposure timing adjustment unit  17  sets a deviation time between the exposure timing of the first imaging unit and the exposure timing of the second imaging unit to be Ns (an integer)+0.5 times the flickering cycle Tl (Deviation Time=Tl×(Ns+0.5)). Therefore, the first imaging unit  11   a  and the second imaging unit  11   b  can acquire brightnesses deviated by the half flickering cycle Tl of the LED. Herein, in a case where the exposure timing of one of the first imaging unit  11   a  and the second imaging unit  11   b  is a timing when the flickering light source is dark, the flickering light source at the exposure timing of the other imaging unit necessarily comes to be bright since the exposure timing of the other imaging unit is deviated by the half cycle with respect to the flickering cycle Tl. 
         [0026]    The light source recognition unit  16  can recognize that the light source is flickering by recognizing that there is a light source in the image obtained by an imaging unit and there is no light source in the image obtained by the other imaging unit. Therefore, it is possible to reliably capture the flickering light source from next time by causing the first imaging unit  11   a  and the second imaging unit  11   b  to perform exposure in synchronization with timing when the flickering light source is bright. Therefore, the flickering light source can be stereoscopically viewed, and accurate distance information up to the flickering light source can be obtained. 
         [0027]    Even at a timing when the flickering light source is captured in the same brightness by the first imaging unit  11   a  and the second imaging unit  11   b,  the traffic signal and the indicator such as the traffic sign using the LED are sufficiently bright compared to the preceding vehicle and a road surface in order to secure visibility. Therefore, even in a case where an optimal exposure is performed in order to recognize the preceding vehicle and the road surface, the flickering light source can be captured with a sufficient pixel value, and can be recognized similarly to an unflickering light source (a light source always turned on). 
         [0028]    In this case, the light source can be recognized in synchronization with the exposure timing of any one of the first imaging unit  11   a  and the second imaging unit  11   b.  Therefore, the flickering light source can be recognized and the distance information can be obtained by performing the stereoscopic viewing in synchronization with the exposure timing of any one of the imaging units. In other words, the flickering light source can be recognized similarly to a case where the light always turned on is recognized. 
         [0029]    Further, as a method of recognizing the light source from an image, any method such as a template matching and a classification using color information may be used as long as a feature can be extracted from the image, and a well-known method may be used. 
         [0030]      FIG. 3  is a flowchart for describing an exemplary method of setting the exposure timing using the exposure timing adjustment unit. The exposure timing adjustment unit  17  determines whether the stereoscopic distance data is generated based on the recognition result of the image recognition unit  15  and the information acquired from the external information reception unit  18 . 
         [0031]    First, in Step S 1 , recognition result information is received from the image recognition unit  15  and the light source recognition unit  16 . Next, in Step S 2 , the external information such as the vehicle speed information  21 , the yaw rate  22 , the GPS information  23 , the time information  24 , and the map data  25  linked to the vehicle network  20  are acquired from the external information reception unit  18 . 
         [0032]    Then, in Step S 3 , it is determined whether the flickering cycle Tl of the flickering light source is 1/50 second. Since the capturing cycle Tc and a capture deviation time are determined based on the flickering cycle of the flickering light source, the flickering cycle of the flickering light source is necessarily determined. 
         [0033]    In Step S 3 , in a case where it is determined that the flickering cycle of the flickering light source is 1/50 second, the flickering cycle of the flickering light source is set to 1/50 second in Step S 4 , and if not (that is, it is determined that the flickering cycle is not 1/50 second), the flickering cycle of the flickering power source is set to 1/60 second in Step S 5 . 
         [0034]    At the first time, since the flickering cycle is not clear, the flickering cycle is set to any one of 1/50 and 1/60 second. For example, in a case where the pixel value of the light source is almost the same value in the image captured by the same imaging unit, it can be determined that the capturing cycle is an integer times an intended flickering cycle and the flickering cycle is correctly set. 
         [0035]    On the other hand, in a case where the pixel value of the light source is changed by stages in the image captured by the same imaging unit, it is determined that the flickering cycle of the light source is different from the current setting, and thus the setting of the flickering cycle is switched to the other value. The flickering cycle from next time is set to the cycle at the previous time. In a case where it is determined that the flickering cycle is different on the same basis, the setting is switched. 
         [0036]    The flickering cycle of the flickering light source may be determined based on a power source frequency of the flickering light source. Since the frequency of the commercial power source is determined depending on regions, the flickering cycle may be determined according to GPS information or a running location of a subject vehicle received in cooperation with a car navigation system. For example, in the case of Japan, the power source frequency in the east region of Japan is 50 Hz, and the power source frequency in the west region of Japan is 60 Hz. Therefore, the power source frequency of the flickering power source can be recognized by specifying the current location of the vehicle based on the external information such as the GPS information  23  and the map data  25 . For example, in a case where the power source frequency is 50 Hz, and in a case where the flickering cycle of the flickering light source is 1/50 second and the power source frequency is 60 Hz, the flickering cycle of the flickering light source can be set to 1/60 second. 
         [0037]    Next, in Step S 6 , it is determined whether a light emission timing of the flickering light source is estimated. In a case where the light emission timing of the flickering light source is estimated, the first imaging unit  11   a  and the second imaging unit  11   b  are caused to simultaneously perform the capturing in synchronization with the light emission timing of the flickering light source in Step S 8 . In this case, the flickering light source can reliably perform the capturing using both the first imaging unit  11   a  and the second imaging unit  11   b.  Further, the distance information up to the light source can be stereoscopically acquired. 
         [0038]    On the other hand, in a case where the light emission timing of the flickering light source is not estimated yet, it is determined whether the distance information is necessary in Step S 7 . While the acquirement of the distance information using the stereo camera is a feature, the distance information may be not necessary for all the frames. Therefore, the stereoscopic viewing (the imaging units are caused to perform the capturing at the same exposure timing) or the deviating of the exposure timings is selected according to a situation based on the recognition result acquired in Step S 1  and the external information acquired in Step S 2 . In a case where it is determined that the distance information is necessary, the first imaging unit and the second imaging unit are caused to simultaneously perform the capturing in synchronization with the light emission timing of the flickering light source in Step S 8 . In a case where it is determined that the distance information is not necessary, the capturing is performed by deviating the exposure timings of the first imaging unit and the second imaging unit by a predetermined time in Step S 9 . 
         [0039]    For example, in a case where it is a situation that the distance information from the stereo camera is necessary for the other applications such as a pedestrian detection and a preceding vehicle detection, the capturing is performed at the same exposure timing based on the other applications with priority higher than that of the light emission timing of the flickering light source. 
         [0040]    In the in-vehicle image recognition device  10 , the preceding vehicle and the pedestrian are recognized from the image, and the result is acquired from the image recognition unit  15 . In a case where the preceding vehicle is within a predetermined distance, or in a case where the pedestrian or the bicycle is near the advancing path of the subject vehicle, it is determined that the distance information is necessary for avoiding a risk. In addition, a three-dimensional object can be effectively detected in the stereoscope viewing. Therefore, a counter is provided in the device, and the capturing may be performed alternately using the stereoscopic viewing and the deviating of the exposure timings in a predetermined cycle even in a case where there is no change in an ambient environment. 
         [0041]    In a case where the traffic signal or the traffic sign of an electric light display type can be acquired from the map data  25 , the exposure timings may be deviated only when the subject vehicle comes close to a predetermined distance or more. It is desirable that a parameter related to the exposure be optimal to recognize the preceding vehicle and the pedestrian. However, for example, in a case where it is determined that the light source is recognized with priority, a shutter opening time may be lengthened. 
         [0042]    According to the in-vehicle image recognition device  10  described above, the shutter opening time and a pixel transfer amount of the imaging unit are not changed, so that the performance of the image recognition application is not damaged. In addition, since the extraction process of the light source is performed as in the conventional process, a processing load is not increased. Further, there is no increase in hardware, the recognition performance of the flickering light source can be improved at a low cost. 
         [0043]    Hitherto, the description has been made about embodiments of the invention, but the invention is not limited to the embodiments. Various changes in design can be made within a scope not departing from the spirit of the invention described in the accompanying claims. For example, the embodiments are described in a clearly understandable way for the invention, and thus the invention is not necessarily to include all the configurations described above. In addition, some configurations of a certain embodiment may be replaced with the configurations of another embodiment, and the configuration of the other embodiment may also be added to that of a certain embodiment. Furthermore, various additions, omissions, and substitutions may be partially made in some of the configurations of the respective embodiments. 
       REFERENCE SIGNS LIST 
       [0000]    
       
           10  in-vehicle image recognition device 
           11   a  first imaging unit 
           11   b  second imaging unit 
           12   a  first image 
           12   b  second image 
           13  distance data generation unit 
           14  distance data 
           15  image recognition unit 
           16  light source recognition unit 
           17  exposure timing adjustment unit 
           18  external information reception unit 
           19   a  first exposure control unit 
           19   b  second exposure control unit 
           20  vehicle network 
           21  vehicle speed information 
           22  yaw rate 
           23  GPS information 
           24  time information 
           25  map data