Abstract:
A system and method are provided for actively illuminating and monitoring a subject, such as a driver of a vehicle. The system includes a video imaging camera orientated to generate images of the subject eye(s). The system also includes first and second light sources offset from each other and operable to illuminate the subject. The system further includes a controller for controlling illumination of the first and second light sources such that when the imaging camera detects sufficient glare, the controller controls the first and second light sources to minimize the glare. This is achieved by turning off the illuminating source causing the glare.

Description:
TECHNICAL FIELD  
       [0001]     The present invention generally relates to imaging systems and, more particularly, relates to a system for illuminating and capturing images of a subject, such as a person&#39;s eye, in a manner that reduces glare and/or reflections.  
       BACKGROUND OF THE INVENTION  
       [0002]     A variety of video imaging systems have been proposed for use on-board vehicles to monitor the driver and/or passenger(s) in the vehicle. Some proposed video imaging systems include one or more cameras focused on the driver of the vehicle to capture images of the driver&#39;s face. The images are processed to determine various facial characteristics of the driver including position, orientation, and movement of the driver&#39;s eyes, face, and head. Given the determined driver facial characteristics, such as the driver&#39;s eye positioning and gaze, vehicle control systems can provide enhanced vehicle functions.  
         [0003]     One proposed vehicle control system can monitor one or both eyes of the driver and determine a condition in which the driver appears to be drowsy, and can initiate a countermeasure. Another proposed vehicle control system can also determine if the driver is distracted, and can act to minimize the distraction. Further, vehicle control systems can monitor the driver&#39;s eye gaze and control various vehicle systems, such as the radio and entertainment system, to provide enhanced control of such systems based on eye gaze monitoring.  
         [0004]     In order to maximize the effectiveness of a vehicle imaging system during low ambient light conditions, such as night time operation, active light illumination may be required. Proposed approaches for illuminating the driver&#39;s face with light have included providing a light source generally focused on the driver&#39;s eye(s). Other proposed techniques for illuminating the driver&#39;s face employ multiple light sources at different wavelengths, arranged in a generally concentric ring, for generating a bright eye pupil and dark eye pupil effect. A further approach provides first and second light sources located on opposite sides of the camera to achieve light and dark eye pupil effects. One example of the aforementioned approach is disclosed in U.S. application Ser. No. 10/705,541, filed on Nov. 11, 2003, which is hereby incorporated herein by reference.  
         [0005]     Many conventional approaches that employ active light illumination suffer from drawbacks. In particular, when the illuminated subject passenger is wearing corrective lens eye glasses, the geometry of the corrective lenses of the eye glasses may reflect the infrared illumination therefrom onto the image acquired by the imaging camera. This reflection of the illumination source is generally seen as a glare on the subject&#39;s eye glasses. The resultant glare may occur at the regions of interest near the eye(s) of the subject, thus inhibiting the ability to recognize imaged features.  
         [0006]     Relocation or repositioning of the light illumination source may modify appearance and location of the resultant glare; however, the optical advantage of the curvature of the corrective lenses generally imposes that a large relocation distance of the illuminator provides only a minor adjustment in the resultant position of the glare on the lenses. Thus, relocation of the illuminating source, particularly in a vehicle, may not adequately mitigate the disruptive glare. The glare problem is further aggravated in that nominal movements of the subject&#39;s head may shift the glare so that the glare again occludes the eye. In a vehicle eye monitoring system, nominal head movements should not result in interruption of the subject monitoring.  
         [0007]     It is therefore desirable to provide for an effective method of actively illuminating the eye(s) of a subject person and generating images of the eye(s) to allow for enhanced eye monitoring, particularly for a person wearing corrective lens glasses in a vehicle. In particular, it is desirable to provide for a more cost effective system and method that minimizes or eliminates glare that may result from a reflection from a reflecting surface, such as a corrective lens.  
       SUMMARY OF THE INVENTION  
       [0008]     In accordance with the teachings of the present invention, a system and method are provided for illuminating and monitoring a subject, such as one or more eyes. According to one aspect of the present invention, the system includes a video imaging camera orientated to generate images of a subject. The system also includes first and second light sources. The first and second light sources are offset from each other and are operable to illuminate the subject. The system further includes a controller for controlling operation of the first and second light sources such that when the imaging camera detects sufficient glare. The controller controls the first and second light sources to reduce the glare. This is achieved by turning off the illuminating source causing the glare.  
         [0009]     According to another aspect of the present invention, the method includes the steps of arranging a video imaging camera to generate images of a subject, and arranging first and second light sources at first and second locations. The method also includes the step of illuminating the subject with the first light source and generating a first image of the eye when the subject is illuminated with the first light source. The method further includes the steps of detecting the presence of sufficient glare and controlling the first and second illuminators to reduce the light provided by the first light source when sufficient glare is detected.  
         [0010]     Accordingly, the system and method effectively control first and second light illuminators in a manner to reduce glare that may be seen in a captured image so as to enhance the image processing eye recognition. The present invention is particularly useful for monitoring a subject passenger, such as a driver of a vehicle, wearing corrective lenses which can cause glare upon reflection of the illuminated light.  
         [0011]     These and other features, advantages and objects of the present invention will be further understood and appreciated by those skilled in the art by reference to the following specification, claims and appended drawings. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0012]     The present invention will now be described, by way of example, with reference to the accompanying drawings, in which:  
         [0013]      FIG. 1  is a front view of a vehicle passenger compartment equipped with an active illumination and imaging system according to the present invention;  
         [0014]      FIG. 2  is a top perspective view of the projection of the two IR illuminators toward the face of a subject driver of the vehicle;  
         [0015]      FIG. 3  is a side view of the two IR illuminators operable to illuminate the face of the driver;  
         [0016]      FIG. 4  is a block diagram further illustrating the illumination and imaging system according to the present invention;  
         [0017]      FIG. 5A  is an exemplary top view illustrating illumination of the subject face with the first IR illuminator;  
         [0018]      FIG. 5B  is an exemplary top view illustrating illumination of the subject face with the second IR illuminator;  
         [0019]      FIG. 6A  is a front view of the driver&#39;s face illustrating glare resulting from light reflection from the corrective lenses when illuminated by the first IR illuminator;  
         [0020]      FIG. 6B  is a front view of the driver&#39;s face illustrating the lack of glare when illuminated by the second IR illuminator;  
         [0021]      FIG. 7  is a flow diagram illustrating a routine for detecting glare and controlling the first and second IR illuminators according to a first embodiment of the present invention;  
         [0022]      FIG. 8  is a flow diagram illustrating a routine for detecting glare and controlling the first and second IR illuminators according to a second embodiment; and  
         [0023]      FIG. 9  is a flow diagram illustrating a routine for detecting glare and controlling the first and second IR illuminators according to a third embodiment. 
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS  
       [0024]     Referring now to  FIGS. 1 and 2 , the passenger compartment (cockpit)  12  of a vehicle  10  is generally shown equipped with a pair of infrared (IR) illuminators and a video imaging camera  32  for illuminating and monitoring the head portion  26  of the subject driver of the vehicle  10 . The vehicle passenger compartment  12  generally includes a dash  14  generally located in front of the driver and front passenger seats and below the front windshield  16 . The vehicle dash  14  is generally forward of the steering wheel  18  and steering column and includes accessories, such as an instrument panel and other electronic devices.  
         [0025]     Mounted in front of the driver&#39;s seat  24  generally in the upper region of the dash  14  is the video imaging camera  32  for acquiring video images of the subject driver&#39;s head  26 , including the face and eyes  28 , according to one embodiment. The imaging camera  32  is located and positioned sufficient to capture successive video image frames of the region where the driver&#39;s head  26  is expected to be located during normal vehicle driving. More particularly, the video imaging camera  32  captures at least a portion of the driver&#39;s head  26 , particularly the face including one or both eyes  28  and the surrounding ocular features generally found in the area referred to as the ocular adnexa. While a single video imaging camera  32  is shown mounted in dash  14  generally directly in front of the subject driver of the vehicle  10 , it should be appreciated the video imaging camera  32  may include one or more cameras mounted in any location suitable to capture images of the head  26  of a subject passenger in the vehicle  10 .  
         [0026]     Mounted within the passenger cockpit  12  of vehicle  10  are first and second infrared (IR) illuminators  20  and  30  for illuminating light onto the face of a subject passenger, such as the driver, so that lighted video images can be acquired by camera  32 . Light illuminators  20  and  30  are independently controlled to provide enhanced active light illumination in a manner that minimizes glare. The first IR illuminator  20  is shown mounted on or near the driver&#39;s side A-pillar  22  of the vehicle  10 . The second IR illuminator  30  is shown mounted on or in dash  14  generally in a region between the driver side and passenger side of the vehicle  10 . The first and second IR illuminators  20  and  30  are shown positioned on opposite sides of the subject driver and both illuminators  20  and  30  are oriented to illuminate floodlight on the subject driver&#39;s face to enable the imaging camera  32  to acquire sufficiently lighted video images.  
         [0027]     With particular reference to  FIG. 2 , the first IR illuminator  20  is located on or near the A-pillar at a lateral (side) angle θ A  from axis  70  which extends in the horizontal longitudinal axis of the vehicle  10  directly forward of the driver. The second IR illuminator  30  is located at a lateral angle θ B  from axis  70  on the opposite side of axis  70 . Each of the lateral angles θ A  and θ B  is at least fifteen degrees (15°), according to one embodiment. According to one example, lateral angles θ A  and θ B  may be equal to approximately forty-six degrees (46°) and thirty-three degrees (33°). While the IR illuminators  20  and  30  are shown located on opposite sides of the head  26  of the subject driver of the vehicle  10 , it should be appreciated that the IR illuminators  20  and  30  could be located at other locations on the vehicle  10  including the same side of the head  26  of the driver of the vehicle  10 .  
         [0028]     Referring to  FIG. 3 , the first and second IR illuminators  20  and  30  are shown in a side elevational view having inclination angles φ A  and φ B  relative to horizontal longitudinal axis  70 . The first IR illuminator  20  is located at an inclination angle φ A  relative to axis  70  of approximately twenty degrees (20°), while the second IR illuminator  30  is located at an inclination angle φ B  of approximately negative one degree (−1°), according to one example. However, it should be appreciated that the first and second IR illuminators  20  and  30  may be located at various other inclination angles φ A  and φ B .  
         [0029]     The first and second IR illuminators  20  and  30  may include any of a number of light illumination sources for illuminating light on the face of the subject driver, sufficient for the video imaging camera  32  to acquire video images of the illuminated subject&#39;s face. In one embodiment, first and second IR illuminators  20  and  30  include an array of multiple light emitting diodes (LEDs)  20 A and  30 A, respectively. According to one example, the first and second IR illuminators  20  and  30  may include an array of twenty ( 20 ) LEDs  20 A and twenty ( 20 ) LEDs  30 A, respectively.  
         [0030]     Referring to  FIG. 4 , the active illumination imaging system is generally illustrated having a CMOS imaging camera  32 , first and second IR illuminators  20  and  30 , a light illumination controller  34 , and a vision processor  40 . The video imaging camera  32  may include a CCD/CMOS active-pixel digital image sensor. One example of the CMOS active-pixel digital image sensor is Model No. PP-0330, commercially available from Photobit, which has a resolution of about 640 H×480 V. It should be appreciated that other cameras, including less costly and less sophisticated video cameras, may be employed.  
         [0031]     The vision processor  40  receives the captured video images from camera  32  on video output line  52 . The vision processor  40  processes the captured images and determines the presence of glare in the captured images. The presence of glare can be determined by detecting brightness in the image. The vision processor  40  further provides a control signal to the light illumination controller  34  when the determined glare may inhibit identification of subject features.  
         [0032]     The vision processor  40  includes a frame grabber  46  for receiving the video frames generated by the camera  32 . The vision processor  40  also includes a video processor  42  for processing the video frames and memory  44 , such as random access memory (RAM), read-only memory (ROM), and other memory as should be readily apparent to those skilled in the art. The video processor  42  is configured to perform one or more routines to identify one or more facial characteristics of the monitored subject driver and may make further determinations such as expected gaze vector of the driver, driver distraction, and driver drowsiness, for example.  
         [0033]     The vision processor  40  also includes an input/output (I/O) port  58  shown providing outputs to countermeasures  60 . The countermeasures  60  may include any of a number of control features such as may be found in a driver drowsiness system, driver distraction control system, and any other countermeasures that may utilize processed images from a video camera.  
         [0034]     The vision processor  40  further generates camera control signals on line  54  which are sent via RS-232  56  to the imaging camera  32 . The camera control signals allow for control of the video imaging camera  32 . The camera control signals may include automatic adjustment of the pointing orientation of the video imaging camera  30 . For example, the video imaging camera  32  may be repositioned to focus on an identifiable feature, and may scan a region in search of one or more identifiable features, including the subject&#39;s face and, more particularly, one or both eyes. The camera control signals may also include adjustment of focus and magnification as may be necessary to locate and track one or more identifiable features. Thus, the imaging system may automatically locate and track identifiable feature(s), such as one or both of the subject&#39;s eyes.  
         [0035]     The light illumination controller  34  includes a microprocessor-based controller having a microprocessor  36  and memory  38 . Memory  38  may include RAM, ROM, EEPROM, and other memory as should be evident to those skilled in the art. The controller  34  is programmed to include a glare detection and illuminator control routine  100  for controlling activation of the first and second IR illuminators  20  and  30  based on the detection of a sufficient amount of glare as described herein.  
         [0036]     While a separate light illumination controller  34  and vision processor  40  are shown and described herein, it should be appreciated that the glare detection and illuminator control routine  100  may be implemented in the vision processor  40  or in any other controller having adequate processing capability. The vision processor  40  and light illumination controller  34  may be integrated together or may be implemented in separate hardware and/or software as shown.  
         [0037]     The light illumination controller  34  provides control output signals to each of the first and second IR illuminators “A” and “B”  20  and  30  to control energization of the first and second illuminators  20  and  30 . The controller  34  may control activation of one of the first and second illuminators  20  and  30  at a time in a manner to minimize or reduce adverse effects caused by glare. When one of the first and second illuminators  20  and  30  is turned on and causes glare from a reflective surface, such as a corrective lens, the controller  34  de-energizes the one IR illuminator causing the glare and energizes the other of the two IR illuminators.  
         [0038]     According to one embodiment, the light illumination controller  34  alternates between the first and second IR illuminators  20  and  30  and, upon detecting glare when one of the illuminators is energized, the controller  34  energizes the other IR illuminator only. According to another embodiment, the light illumination controller  34  illuminates one of the first and second IR illuminators  20  and  30  and, upon detecting glare, switches to the other IR illuminator only. According to a further embodiment, the light illumination controller  34  illuminates both IR illuminators  20  and  30  and, upon detecting glare, turns off the glare causing illuminator. This independent control of the IR illuminators  20  and  30  results in a reduction of the glare that may otherwise be present in captured images that are processed by the vision processor  40 .  
         [0039]     Light illumination of the subject&#39;s face by the first and second IR illuminators  20  and  30  is illustrated in  FIGS. 5A and 5B . An example of captured images of the subject&#39;s face when illuminated with and without glare is illustrated in  FIGS. 6A and 6B . In  FIG. 5A , the first IR illuminator  20  is turned on (energized) to illuminate light on the subject head of the driver of the vehicle. The first IR Illuminator  20  illuminates the subject&#39;s head in the area of the eyes  28 , including the corrective lenses  72  worn by the subject driver. Images of the illuminated subject&#39;s head are then captured with the imaging camera  32 . In the example shown, illumination by the first IR illuminator  20  results in a glare  74  as shown in the captured image of  FIG. 6A . The presence of glare  74  prohibits or reduces the ability to process images and identify facial characteristics of the subject driver.  
         [0040]     When glare  74  is detected, the light illumination controller  34  switches to turn on the second IR illuminator  30  and turns off the first IR illuminator  20  as shown in  FIG. 5B . Energization of the second IR illuminator  30  only illuminates the head portion of the subject driver including the corrective lenses  72  worn in front of the eyes  28  to enable the camera  32  to capture images of the subject&#39;s head. With the second IR illuminator  30  turned on and the first IR illuminator  20  turned off, an image as shown in  FIG. 6B  is captured by camera  32  without the presence of any glare.  
         [0041]     By locating the IR illuminators  20  and  30  at different locations, particularly on opposite sides of the subject&#39;s head  26 , controlled activation of the IR illuminators  20  and  30  reduces the amount of glare that may otherwise interfere with processing of the captured image. As the subject driver&#39;s head  26  is rotated in one direction away from the forward line of sight (axis  70 ), there may exist two regions of glare occlusion with corrective lenses  72  before the driver&#39;s head  26  is turned ninety degrees (90°) forward. Since one side of the face of the driver&#39;s head  26  moves away from the imaging camera  32 , any resultant glare spot will move further away from the eye  28  and eventually off of the corrective lens  72 . Thus, the one side of the driver&#39;s face, as long as it is visible to the imaging camera  32 , will generally present a glare-free view of the eye  28 .  
         [0042]     The other side of the face, which is turned toward the imaging camera  32 , may have a glare spot moved inward which can eventually cover the eye  28 . At this first region of occlusion (i.e., left illuminator with left lens or right illuminator with right lens), the driver&#39;s head  26  is oriented away from the offending illuminator. By removing the offending illumination source (e.g.,  20 ), the alternate illuminator (e.g.,  30 ) may provide coverage to the driver&#39;s face with both eyes  28  not being occluded by a glare spot  74 . Thus, in the first occlusion region, the alternate illuminator will provide glare-free illumination of both eyes  28 .  
         [0043]     As the subject&#39;s head  26  is rotated through the occlusion region, it may enter into the second occlusion region wherein the IR illuminator left on casts a glare on the eye  28  as it crosses near the forward centerline (i.e., left illuminator on right lens or vice versa). In this situation, two possibilities may be available for monitoring the eyes  28 . One eye  28  is still illuminated and may be visible to the imaging camera  32 . If the offending (glare causing) IR illuminator is switched off and the other illuminator is turned back on, the driver&#39;s face will be illuminated but the glare may not be present. This is because the occlusion from the source exists at the first occlusion region and the glare has moved off the corrective lens  74  in the second occlusion region.  
         [0044]     Finally, as the head is rotated past the second occlusion region, there may be no further glare reflections experienced in a typical lens reflection scenario. Due to the symmetry of the subject&#39;s face and glasses, the first and second occlusion regions exist as the head is rotated laterally in the opposite direction, and any resulting glare  74  may be dealt with in the same way described above.  
         [0045]     Referring to  FIG. 7 , a glare detection and illuminator control routine  100  is illustrated for independently controlling activation of the first and second illuminators in a manner that minimizes the adverse effects of glare in the captured video images. The control routine  100  illuminates both the first and second IR illuminators and, upon detecting glare of a sufficient magnitude, turns off the IR illuminator causing the glare, according to a first embodiment.  
         [0046]     Control routine  100  begins at step  102  and proceeds to read the new video frame event in step  104 . Next, in step  106 , routine  100  energizes both the first and second IR illuminators. Thus, both the first and second IR illuminators are activated together at the same time.  
         [0047]     With both IR illuminators turned on, control routine  100  determines if sufficient occluding glare is detected in step  108 . If sufficient occluding glare is detected in step  108 , routine  100  proceeds to step  110  to turn the first IR illuminator “A” off. Thus, the second IR illuminator “B” remains on.  
         [0048]     With the first IR illuminator “A” turned off and the second IR illuminator “B” turned on, routine  100  proceeds to determine if there is sufficient occluding glare in step  112 . If there is sufficient occluding glare detected, routine  100  switches power to the opposite IR illuminator in step  114 . Thus, the second IR illuminator “B” is turned off and the first IR illuminator “A” is turned on. Thereafter, control routine  100  proceeds to process the current video frame in step  116  before waiting for the next new video frame in step  118  and then returning to step  104 .  
         [0049]     If there is not sufficient occluding glare detected in either of decision steps  108  and  112 , control routine  100  proceeds to process the current video frame in step  116 . Thereafter, routine  100  waits for the next video frame in step  118  and then returns to step  104 . Accordingly, control routine  100  advantageously turns off the IR illuminator causing the glare.  
         [0050]     Referring to  FIG. 8 , a glare detection and illuminator control routine  130  is illustrated according to a second embodiment for detecting glare and independently controlling the first and second illuminators  20  and  30  to reduce the presence of glare. Routine  130  begins at step  132  and proceeds to read a new frame event of the captured images in step  134 . In step  136 , routine  130  switches power to the opposite IR illuminator. In decision step  138 , routine  130  determines if there is an occluding glare present. An occluding glare can be detected if the brightness B is greater than a threshold brightness B T , according to one example.  
         [0051]     If there is occluding glare detected, routine  130  ignores the current frame in step  142 , and then proceeds to wait for the next frame in step  144  before returning to step  134 . If there is not sufficient occluding glare detected in step  138 , routine  130  proceeds to process the current frame in step  140 , and then waits for the next frame in step  144  before returning to step  134 . Accordingly, glare detection routine  130  processes captured images in the frame data and processes the frame data only when there is no sufficient occluding glare that interferes with the image recognition.  
         [0052]     Referring to  FIG. 9 , a glare detection and illuminator control routine  200  is illustrated according to a third embodiment. The control routine  200  turns on one of the first and second IR illuminators and, upon detecting glare of a sufficient magnitude, turns off the IR illuminator causing the glare and activates (turns on) the other IR illuminator.  
         [0053]     Control routine  200  begins at step  202  and proceeds to read a new video frame event in step  204 . In decision step  206 , routine  200  determines if there is sufficient occluding glare present in the captured image. If there is sufficient occluding glare, routine  200  switches power to the opposite IR illuminator in step  208 , and then proceeds to process the current frame data in step  210 . If there is not sufficient occluding glare, routine  200  processes a current video frame in step  210 . Following processing of the current video frame, routine  200  waits for the next video frame in step  212  before returning to step  204 . The alternate glare detection routine  200  switches the power between the first and second IR illuminators only when an occluding glare is detected when one of the IR illuminators is turned on, so as to reduce the effects of glare and to advantageously improve the image recognition.  
         [0054]     Accordingly, the active illumination and imaging system of the present invention advantageously controls illumination of a subject to enhance acquisition of video images of the illuminated subject. When glare, such as a reflection off of a corrective lens, is present, the system and method of the present invention control light illumination to turn off the illumination source that may cause the presence of glare. This allows for enhanced illumination in an imaging system, which is particularly advantageous for use in a vehicle where a subject driver of the vehicle may be wearing a corrective lens and the driver&#39;s head may be moving. While two IR illuminators  20  and  30  are shown and described herein, it should be appreciated that the illumination and imaging system and method may employ two or more light illuminators.  
         [0055]     It will be understood by those who practice the invention and those skilled in the art, that various modifications and improvements may be made to the invention without departing from the spirit of the disclosed concept. The scope of protection afforded is to be determined by the claims and by the breadth of interpretation allowed by law.