Abstract:
The presence of eyeglasses worn by a human subject is detected regardless of whether the subject&#39;s head is viewed frontally or in profile. Image data within search windows defined relative to the subject&#39;s eye is processed to form linear representations of edge boundaries. Angles formed by intersections of the linear representations are determined and used as a basis for determining if eyeglass lenses are present. When prescribed eyeglass components are detected, the eye detection and tracking routines are configured accordingly.

Description:
TECHNICAL FIELD 
       [0001]    The present invention relates to monitoring a human subject&#39;s eyes in a video image, and more particularly to a method of determining whether the subject is wearing eyeglasses. 
       BACKGROUND OF THE INVENTION 
       [0002]    Vision systems frequently entail detecting and tracking a subject&#39;s eyes in an image generated by a video camera. In the motor vehicle environment, for example, a camera can be used to generate an image of the driver&#39;s face, and portions of the image corresponding to the driver&#39;s eyes can be analyzed to assess drive gaze or drowsiness. See, for example, the U.S. Pat. Nos. 5,795,306; 5,878,156; 5,926,251; 6,097,295; 6,130,617; 6,243,015; 6,304,187; and 6,571,002, incorporated herein by reference. 
         [0003]    Since a high percentage of people wear some form of eyeglasses when driving, a vehicular vision system must be capable of reliably monitoring the driver&#39;s eyes regardless of whether the driver is wearing eyeglasses. And yet eyeglasses complicate eye monitoring because they can significantly change the appearance of the driver. Moreover, eyeglass lenses and frames can obscure the vision system&#39;s view of the driver&#39;s eyes, and produce glare spots due to reflected ambient and active illumination. While there are special lighting and image processing techniques for addressing the challenges posed by eyeglasses, using these techniques for all drivers would impose an unnecessary burden on the vision processor. Accordingly, what is needed is a simple but reliable way of detecting whether the driver is wearing eyeglasses so that the vision system can be configured accordingly. 
       SUMMARY OF THE INVENTION 
       [0004]    The present invention is directed to a method of monitoring the eyes of a human subject in a stream of video images, including a novel method of reliably determining if the subject is wearing eyeglasses, regardless of whether the subject&#39;s head is viewed frontally or in profile. Image data within search windows defined in relation to subject&#39;s eye is processed to form linear representations of edge boundaries. Angles formed by intersections of the linear representations are determined and used as a basis for determining if eyeglass lens are present. When prescribed eyeglass components are detected, eye detection and tracking routines are configured accordingly. 
     
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0005]      FIG. 1  depicts a diagram of a vehicle equipped with an eye monitoring apparatus according to the present invention; 
           [0006]      FIG. 2  is a block diagram of the eye monitoring apparatus of  FIG. 1 , including a video imaging device and a digital signal processor (DSP) for carrying out an eyeglass detection routine according to this invention; 
           [0007]      FIG. 3A  is diagram of a leftward facing human subject wearing eyeglasses and a set of search windows for eyeglass detection according to the present invention; 
           [0008]      FIG. 3B  is a representation of edge boundary data generated by the eye monitoring apparatus of  FIG. 1  for the human subject of  FIG. 3A  according to this invention; 
           [0009]      FIG. 3C  is a linear representation of the edge boundary data of  FIG. 3B  for purposes of eyeglass detection; 
           [0010]      FIGS. 4A and 4B  together depict a flow diagram representative of a software routine carried out by the DSP of  FIG. 2  for carrying out the eyeglass detection method of the present invention; 
           [0011]      FIG. 5  depicts a flow diagram detailing a lens detection routine called by the flow diagram of  FIGS. 4A-4B ; and 
           [0012]      FIG. 6  depicts a flow diagram detailing a portion of the flow diagram of  FIGS. 4A-4B  pertaining to determining eyeglass status and head pose. 
       
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0013]    Referring to  FIG. 1 , the method of the present invention is disclosed in the context of an eye monitoring apparatus  12  for the driver  16  of a motor vehicle  10 . However, it will be recognized that the method of this invention is equally applicable to other vision systems that monitor a human eye, whether vehicular or non-vehicular. In the illustration of  FIG. 1 , the eye monitoring apparatus  12  is mounted in the passenger compartment  14  of vehicle  10  forward of the driver  16  in a location that affords an unobstructed view of the driver&#39;s face when the driver  16  is reposed on the seat  20 , taking into account differences in driver height and orientation. In general, the eye monitoring apparatus  12  actively illuminates driver  16  with infrared (IR) light and produces a stream of video images that include the driver&#39;s eyes  22 . The images are processed to locate the driver&#39;s eyes in a given image  22  and to track the eye locations from one image to the next. The state of the eyes  22  can be characterized for various purposes such as detecting driver drowsiness and/or distraction, or even driver gaze. 
         [0014]    In the illustration of  FIG. 1 , the driver  16  is wearing eyeglasses  24 , which in general may include regular prescription eyeglasses, safety glasses, sunglasses, goggles, etc. According to this invention, the eye monitoring apparatus  12  processes video images of the driver  16  to determine whether the driver  16  is wearing eyeglasses  24 . 
         [0015]    Referring to the block diagram of  FIG. 2 , the eye monitoring apparatus  12  preferably includes upper and lower IR active illumination devices  28  and  30 , a solid-state imaging device  32  focused on the driver&#39;s face, and a vision processor  34 . In the illustrated embodiment, the apparatus  12  provides eye state information to a remote host processor  36 , and the host processor  36  selectively activates one or more counter-measure devices or systems  38  such as an alarm or a braking system if it is determined that the driver&#39;s lack of alertness or attention may possibly compromise vehicle safety. The active illumination devices  28  and  30  are individually activated by the vision processor  34  via I/O interface  46 , and each comprises an array of IR light emitting diodes as indicated. As described in co-pending U.S. patent application Ser. No. 11/______ (Attorney Docket No. DP-314360), the active illumination devices  28  and  30  may be individually and alternately activated for successive video frames to shift eyeglass glare away from the driver&#39;s eye in at least one-half of the video frames. If the eyeglass detection method of the present invention concludes that the driver  16  is not wearing eyeglasses  24 , the active illumination devices  28  and  30  may be activated concurrently for optimal driver illumination and system performance. The vision processor  34  comprises conventional components, including a frame grabber  40  for acquiring video images from imaging device  32 , a non-volatile memory  42  for storing various image processing routines, and a digital signal processor (DSP)  44  that selectively executes the routines stored in memory  42  for processing the video images acquired by frame grabber  40 . The DSP  44  outputs various control signals to illumination devices  28  and  30  and imaging device  32  via interface  46 , and communicates with host processor  37  via interface  48 . 
         [0016]    The signal processing routines residing in the non-volatile memory  42  include eye detection and tracking routines  50 , the eyeglass detection routine  52  of this invention, and an eye analysis routine  54 . In general, the detection and tracking routines  50  identify the location of the driver&#39;s eyes  22  in a given image frame, and then track the eye locations from one image frame to the next. The routine  52  processes the image data to determine if the driver  16  is wearing eyeglasses  24 , and the routine  54  characterizes the state of the driver&#39;s eyes (open vs. closed, for example). The eye detection and tracking routines  50 , the eye analysis routine  54 , and the routines executed by host processor  36  for using the eye state information may comprise any of a number of known processing techniques. The eyeglass detection routine  52  is the subject of the present invention, and is described below in reference to  FIGS. 3A-3C  and the flow diagrams of  FIGS. 4A-4B ,  5  and  6 . 
         [0017]      FIGS. 3A-3C  pictorially illustrate the eyeglass detection routine  52  for a condition in which the driver  16  is wearing eyeglasses  24  and facing to the left of the eye monitoring apparatus  12 . The eyeglasses  24  include a right lens  24   a , a right frame hoop  24   b , a right frame arm  24   c , a nose bridge  24   d , a left lens  24   e , a left frame hoop  24   f , and a left frame arm  24   g . It is assumed for purposes of the illustrations that the detection and tracking routines  50  have successfully located the subject&#39;s eyes  22  in a given image frame. 
         [0018]    In general, the eyeglass detection routine  52  selects the eye  22  that has been detected with the highest confidence (the driver&#39;s right eye in the illustration of  FIGS. 3A-3C ) and defines a primary eye-sized search window  60  centered on that eye. The image data within the primary search window  60  is processed to determine if an eyeglass frame hoop  24   b  is present. Next, the routine  52  defines a small secondary search window  62  laterally outboard of the primary search window  60 , and processes the image data within that window to determine if a frame arm  24   c  is present. Next, the routine  52  defines another secondary search window  64  laterally inboard of the primary search window  60 , and processes the image data within that window to determine if a nose bridge  24   d  is present. Next, the routine  52  defines another eye-sized search window  66 , and processes the image data within that window to determine if an eyeglass frame hoop  24   f  is present. The search window  66  can be defined with respect to the other search windows  60 - 64  as shown, or may be centered on the detected location of the un-selected eye  22  if desired. Finally, the routine  52  defines a small secondary search window  68  laterally outboard of the search window  66 , and processes the image data within that window to determine if a frame arm  24   g  is present. Depending on which eyeglass elements are deemed to be present, the routine  52  determines if the driver  16  is wearing eyeglasses  24 , and in some cases the driver&#39;s head pose (i.e., forward-facing, right-facing, or left-facing). 
         [0019]    Detecting the presence or absence of eyeglass components  24   a - 24   g  within the search windows  60 - 68  is accomplished by applying an edge detection function (such as a Sobel edge detector routine) to the image data within the search windows to identify edge boundaries of the imaged data as represented in  FIG. 3B , and then applying a line selection function (such as a Radon-Hough transformation) to the detected edges. If the pixel density of a constructed line is sufficiently high, the line selection function records the line coordinates for eyeglass component detection purposes. 
         [0020]    In the small search windows  62 ,  64  and  68 , the presence of a line of sufficient pixel density within a prescribed range of angles indicates the presence of the corresponding eyeglass component. For example, the presence of a frame arm  24   c  is detected based on the presence of line  88  within search window  62 , and the presence of a nose bridge  24   d  is detected based on the presence of line  86  in search window  64 . The search window  68  contains no edge data, and therefore no line segment. 
         [0021]    The eye search windows  60  and  66  are divided into sections and processed section-by-section to form a piecewise linear representation of the frame hoops  24   b  and  24   f , if present. In the illustrated embodiment, the search windows  60  and  66  are divided into quadrants as depicted by the dashed bisector lines in  FIG. 3C , and the line segments  70 - 84  represent the outputs of the line selection function for the various search window quadrants. In search window  60 , the lines  70 ,  72 ,  74  and  76  comprise a piecewise linear representation of the right frame hoop  24   b ; and in search window  66 , the lines  78 ,  80 ,  82  and  84  comprise a piecewise linear representation of the left frame hoop  24   f . If the interior angles subtended by the lines in the upper and lower search window quadrants are within a defined range such as 110°-160°, an eyeglass lens is detected. For example, the driver&#39;s right lens  24   a  is detected if the interior angle subtended between lines  70  and  72  is in the defined range of angles, and the interior angle subtended between lines  74  and  74  is also in the defined range. Since some eyeglasses lack an upper or lower frame hoop element, a lens can also be detected if only upper quadrant lines or lower quadrant lines are present, provided another eyeglass component such as nose bridge  24   d  is also detected. 
         [0022]    The flow diagrams of  FIGS. 4A-4B ,  5  and  6  describe the operation of the eye detection routine  52 . As mentioned above, it is assumed that the detection and tracking routines  50  have previously located the subject&#39;s eyes  22  in a given image frame. 
         [0023]    Referring to  FIGS. 4A-4B , the block  90  is executed to select one of the detected eyes  22  and to define the search windows  60 - 68  with respect to the coordinates of that eye. The selected eye may be a predetermined eye such as the driver&#39;s left eye, or alternately, the eye that has been detected with the highest confidence. For purposes of the flow diagram, the search window centered on the selected eye (i.e., search window  60  in the example of  FIGS. 3A-3C ) is designated as SW 1 . The frame arm search window laterally outboard of SW 1  (i.e., search window  62  in the example of  FIGS. 3A-3C ) is designated as SW 2 . The nose bridge search window laterally inboard of SW 1  (i.e., search window  64  in the example of  FIGS. 3A-3C ) is designated as SW 3 . The search window for the unselected eye (i.e., search window  66  in the example of  FIGS. 3A-3C ) is designated as SW 4 . The location of SW 4  can be defined with respect to SW 1  as shown in  FIGS. 3A-3C , or may be centered on the detected coordinates of the unselected eye. Finally, the frame arm search window laterally outboard of SW 4  (i.e., search window  68  in the example of  FIGS. 3A-3C ) is designated as SW 5 . 
         [0024]    Following definition of the search windows SW 1 -SW 5 , the block  92  is executed to apply an edge detection function (such as a Sobel edge detector) to the image data within the search windows to identify edge boundaries of the imaged data. An example of the result is depicted in  FIG. 3B , described above. The blocks  94 - 132  are then executed to construct linear representations of the edge data in search windows SW 1 -SW 5 , to select representative line segments based on the edge data, and to determine if the various eyeglass components are present based on the selected line segments. In the illustrated embodiment, a Radon-Hough transform converts the edge data from Cartesian coordinates to polar space to construct line segments representative of the edge boundaries. Alternately, the line segments may be generated using a least-squares method, or some other line-fitting technique. Line segments having at least a minimum pixel density and that fall within prescribed angles representative of the respective eyeglass component are identified, and the identified segment having the highest pixel density is selected and recorded. If the recorded line segment is within prescribed constraints, the respective eyeglass component is detected. 
         [0025]    As indicated at blocks  94  and  118 , the lens detection routine  140  of  FIG. 5  is executed to determine if an eyeglass lens is present within each of the eye search windows SW 1  and SW 4 . Referring to  FIG. 5 , the blocks  144 ,  146  and  148  are executed for each quadrant of the respective eye search window SW 1  or SW 4 , as indicated by blocks  142  and  150 . The block  144  applies a Radon-Hough transform to the edge data of a respective quadrant to construct and identify representative line segments, the block  146  selects the line segment having the highest pixel density, and block  148  records the selected line segment. Once the blocks  144 - 148  have been executed for each quadrant of the respective search window SW 1  or SW 4 , the blocks  152 - 172  are executed to determine if an eyeglass lens is present. Blocks  152  and  154  determine if line segments have been recorded for both lower quadrants of the search window and the angle subtended between them is between 110° and 160°. If so, block  156  sets the state of the LOWER ANGLES FOUND flag to True; if not, block  158  sets the state of LOWER ANGLES FOUND to False. Similarly, blocks  160  and  162  determine if line segments have been recorded for both upper quadrants of the search window and the angle subtended between them is between 110° and 160°. If so, block  164  sets the state of the UPPER ANGLES FOUND flag to True; if not, block  166  sets the state of UPPER ANGLES FOUND to False. Block  168  then determines if both LOWER ANGLES FOUND and UPPER ANGLES FOUND are True. If so, an eyeglass lens is detected, and block  170  sets the state of the LENS FOUND flag to True; otherwise, block  172  sets the state of LENS FOUND to False, indicating that a lens was not detected. 
         [0026]    Returning to the flow diagram of  FIGS. 4A-4B , the block  96  checks the status of the LENS FOUND flag following execution of the lens detection routine  140  with respect to eye search window SW 1 . If a lens was detected (i.e., if LENS FOUND=True), the block  98  sets the flag SW 1  to True. If a lens was not detected (i.e., if LENS FOUND=False), the block  100  sets the flag SW 1  to False. The block  102  then applies the line selection function (as described above in respect to blocks  144 - 148  of  FIG. 5 ) to the edge data of the frame arm search window SW 2 . Block  104  examines the line segment, if any, selected and recorded by the line selection function to determine if an eyeglass frame arm  24   c  is present. If the frame arm  24   c  is found, the block  106  sets the flag SW 2  to True; otherwise, the block  108  sets the flag SW 1  to False. The block  110  then applies the line selection function to the edge data of the nose bridge search window SW 3 . Block  112  examines the line segment, if any, selected and recorded by the line selection function to determine if an eyeglass nose bridge  24   d  is present. If the nose bridge  24   d  is found, the block  114  sets the flag SW 3  to True; otherwise, the block  116  sets SW 3  to False. The block  118  then calls the lens detection routine  140  of  FIG. 5  with respect to eye search window SW 4 . The block  120  checks the status of the LENS FOUND flag following execution of the lens detection routine  140 , and block  122  sets the flag SW 4  to True if a lens was detected (i.e., if LENS FOUND=True). If a lens was not detected (i.e., if LENS FOUND=False), the block  124  sets the flag SW 4  to False. Finally, block  126  applies the line selection function to the edge data of the frame arm search window SW 5 . Block  128  examines the line segment, if any, selected and recorded by the line selection function to determine if an eyeglass frame arm  24   g  is present. If the frame arm  24   g  is found, the block  130  sets the flag SW 5  to True; otherwise, the block  132  sets SW 5  to False. 
         [0027]    After the state of the search window flags SW 1 -SW 5  have been determined, the block  134  determines the eyeglass status and driver head pose, and the block  136  configures the detection and tracking routines  50  accordingly. The eyeglass status and head pose determination of block  134  is detailed by the flow diagram of  FIG. 6 . 
         [0028]    Referring to  FIG. 6 , the blocks  180 - 210  determine eyeglass status and driver head pose for a number of different situations. In certain cases, eyeglasses are deemed to be present and the EYEGLASS STATUS flag is set to True; in other cases, the information is deemed to be insufficient to confidently conclude that eyeglasses are present and EYEGLASS STATUS is set to Unknown. If EYEGLASS STATUS is True and driver head pose is detectable, the HEAD POSE flag is set to Right, Left or Forward. In some cases the routine can only conclude that the head pose is Non-Forward, and in other cases the head pose is Unknown. 
         [0029]    The block  180  checks for an ideal non-forward head pose situation in which both lenses  24   a ,  24   b , a nose bridge  24   d , and an ear frame  24   c  or  24   g  are detected. In other words, the flags SW 1 -SW 5  satisfy the Boolean statement: SW 1  AND SW 3  AND SW 4  AND (SW 2  OR SW 5 )=True. If the condition is met, block  184  sets EYEGLASS STATUS to True. If the condition is not met, block  182  checks for a non-forward head pose situation in which both lenses  24   a ,  24   b  and an ear frame  24   c  or  24   g  are detected. If the condition is met, block  184  likewise sets EYEGLASS STATUS to True. If either condition is met, the blocks  186 - 194  determine and designate head pose. The block  186  determines if the selected eye is the driver&#39;s right eye and the detected ear frame is in search window SW 2 . This is the condition shown in the example of  FIGS. 3A-3C ; if the condition is satisfied, block  188  sets HEAD POSE to Left to complete the routine. Conversely, the block  190  determines if the selected eye is the driver&#39;s left eye and the detected ear frame is in search window SW 5 . If the condition is satisfied, block  192  sets HEAD POSE to Right to complete the routine. If neither of the blocks  186  and  190  are answered in the affirmative, head pose cannot be confidently determined and the block  194  sets HEAD POSE to Unknown to complete the routine. 
         [0030]    If neither of the non-forward head pose conditions defined by blocks  180  and  182  are met, the block  196  checks for a forward-facing head pose condition in which both lenses  24   a  and  24   b  are detected (i.e., SW 1  AND SW 4 =True). If the condition is satisfied, block  198  sets EYEGLASS STATUS to True and HEAD POSE to Forward, completing the routine. If the condition is not satisfied, block  200  checks for a non-forward head pose condition in which one lens  24   a  or  24   e , a nose bridge  24   d , and one ear frame  24   c  or  24   g  are detected. That is: (SW 1  OR SW 4 ) AND SW 3  AND (SW 2  OR SW 5 )=True. If the condition is met, block  202  sets EYEGLASS STATUS to True and HEAD POSE to Non-Forward, completing the routine. If the condition is not satisfied, blocks  204  and  206  check for a special condition that can occur in eyeglasses where only the upper or lower half of lenses  24   a ,  24   e  are bounded by a frame hoop  24   b ,  24   f . Block  204  determines if just upper angles or just lower angles were found in both of the eye search windows SW 1  and SW 4 . If so, block  206  determines if a nose bridge  24   d  was also found. If both conditions are met, the block  208  sets EYEGLASS STATUS to True and HEAD POSE to Unknown, completing the routine. If either of the blocks  204  and  206  is answered in the negative, the block  210  is executed to set EYEGLASS STATUS to Unknown, completing the routine. 
         [0031]    In summary, the present invention provides a way of reliably detecting the presence of eyeglasses on a human subject. Knowing whether eyeglasses are present can significantly streamline eye detection and tracking routines, thereby enhancing eye monitoring performance and accuracy. For example, glare detection and elimination techniques need only be used when eyeglasses are detected. Also, the detection and tracking confidence may be lowered when eyeglasses are detected to reflect the reduced visibility of the subject&#39;s eyes through eyeglass lenses. Moreover, appearance models for eye vs. non-eye discrimination can be selected on the basis of the eyeglass status. Additionally, the head pose when detected can be used to confirm driver eye gaze determinations. 
         [0032]    While the present invention has been described with respect to the illustrated embodiment, it is recognized that numerous modifications and variations in addition to those mentioned herein will occur to those skilled in the art. For example, the prescribed angle ranges may be different than mentioned herein, the eye search windows may be divided into a different number of regions than shown, and so on. Accordingly, it is intended that the invention not be limited to the disclosed embodiment, but that it have the full scope permitted by the language of the following claims.