Method of tracking a human eye in a video image

A human eye is tracked between successively produced video image frames by consecutively applied eye location techniques. First, potential eye candidates within a local search window are identified using horizontal projection filtering, coupled with rudimentary appearance-based testing. The identified eye candidates are clustered and subjected to additional appearance-based testing to determine if one or more eye candidates should be accepted. If no eye candidates are accepted, a correlation/appearance technique is applied to the search window to identify the eye. If unsuccessful, the eye may be closed, and peripheral eye-related patches from the previous image frame are extracted and compared with the search window to identify the eye in the current frame. When comparable eye-related patches are identified in the search window for the current frame, the eye center is computed according to the mean of the eye periphery patches.

TECHNICAL FIELD

The present invention relates to a method of tracking a human's eyes in successively produced video image frames.

BACKGROUND OF THE INVENTION

Vision systems frequently entail detecting and tracking a person's eyes in a stream of images generated by a video camera. In the motor vehicle environment, for example, a camera can be used to generate an image of the driver's face, and portions of the image corresponding to the driver's eyes can be analyzed to assess drive gaze or drowsiness. See, for example, the U.S. Pat. Nos. 5,795,306; 5,878,156; 5,926,251; 6,097,295; 6,130,617; 6,243,015; 6,304,187; and 6,571,002, incorporated herein by reference.

While eye detection and tracking algorithms can work reasonably well in a controlled environment, they tend to perform poorly under real world imaging conditions where the lighting produces shadows and the person's eyes can be occluded by eyeglasses, sunglasses or makeup. As a result, pixel clusters associated with the eyes tend to be grouped together with non-eye features and discarded when subjected to appearance-based testing. This problem occurs both in eye detection routines that initially locate the eyes, and in eye tracking routines that track the eye from one image frame to the next. Problems that especially plague eye tracking include head movement and eye blinking, both of which can cause previously detected eyes to suddenly disappear. The usual approach in such cases is to abandon the tracking routine and re-initialize the eye detection routine, which of course places a heavy processing burden on the system and slows the system response. Accordingly, what is needed is an efficient method of reliably tracking a person's eyes between successively produced video image frames, even in situations where the person's head turns or the eyes momentarily close.

SUMMARY OF THE INVENTION

The present invention is directed to an efficient and reliable method of tracking a human eye between successively produced video image frames. Once the eye is detected, its location is used to define a search window in the next image frame. Potential eye candidates within the search window are identified using horizontal projection filtering, coupled with rudimentary appearance-based testing. The identified eye candidates are clustered and subjected to additional appearance-based testing to determine if one or more eye candidates should be accepted. If no eye candidates are accepted, a correlation/appearance technique is applied to the search window to identify the eye. If the correlation or appearance test results do not meet predefined criteria, it is assumed that the eye is closed, and peripheral eye-related patches from the previous image frame are extracted and compared with the search window to identify corresponding features in the search window for the current frame. When comparable eye-related patches are identified in the search window for the current frame, the eye center is computed according to the mean of the eye periphery patches. Only when all of these techniques fail to locate an eye in the search window is the tracking routine abandoned and the eye detection routine re-initialized.

DESCRIPTION OF THE PREFERRED EMBODIMENT

The method of the present invention is disclosed in the context of a system that monitors a driver of a motor vehicle. However, it will be recognized that the method of this invention is equally applicable to other vision systems that detect and track eye movement, whether vehicular or non-vehicular, and to systems that detect and track other types of features or targets.

Referring to the drawings, and particularly toFIG. 1, the reference numeral10generally designates a motor vehicle equipped with an eye monitoring apparatus12according to the present invention. In the illustration ofFIG. 1, the apparatus12is mounted in the passenger compartment14forward of the driver16in a location that affords an unobstructed view of the driver's face18when the driver16is reposed on the seat20, taking into account differences in driver height and orientation. In general, the eye monitoring apparatus12produces a series of video images of the driver's face18, and executes digital signal processing routines for detecting portions of a given image that correspond to the driver's eyes22and tracking the eye location between successively produced video images. The state of the eyes22can be characterized for various purposes such as detecting driver drowsiness and/or distraction, or even driver gaze.

Referring to the block diagram ofFIG. 2, the apparatus12includes an infrared (IR) light illumination device30, a solid-state imaging device32and a vision processor34. In the illustrated embodiment, the apparatus12provides eye state information to a remote host processor36via line37, and the host processor36selectively activates one or more counter-measure devices or systems38such as an alarm or a braking system if it is determined that the driver's lack of alertness or attention may possibly compromise vehicle safety. The illumination device30may be an array of light emitting diodes, for example, and the imaging device32may be a CCD or CMOS imaging chip. The vision processor34comprises conventional components, including a frame grabber40for acquiring video images from imaging device32, a non-volatile memory42for storing various signal processing routines, and a digital signal processor (DSP)44for selectively executing the routines stored in memory42processing the video images acquired by frame grabber40. The DSP44outputs various control signals to illumination device30and imaging device32via interface46, and communicates with host processor37via interface48.

The signal processing routines residing in the vision processor memory42include an eye detection routine50, an eye tracking routine52, and an eye analysis routine54. In general, the eye detection routine50identifies the regions of a video image that correspond to the driver's eyes22, the eye tracking routine52tracks the eye location from one video image to the next, and the eye analysis routine54characterizes the state of the driver's eyes (open vs. closed, for example). The present invention primarily concerns the eye tracking routine52; the eye detection routine50, the eye analysis routine54and the routines executed by host processor36for using the eye state information may comprise any of a number of known processing techniques.

The eye tracking routine52is broadly described by the main loop flow diagram ofFIG. 7, and is illustrated in part by the images ofFIGS. 3-6. The flow diagrams ofFIGS. 8-11detail various aspects of the mail loop flow diagram.

Referring toFIG. 7, the block60is first executed to acquire a new image frame and to extract a local search window (LSW) of predetermined size, centered on the last-identified location of the occupant's eye. In the image ofFIG. 6, for example, the local search window for the occupant's right eye is designated by a large solid-lined rectangle.

Once the LSW of the new image frame is defined, the blocks62and64are executed to extract eye candidates from the LSW using an inverse receptive hole extraction (IRHE) technique and to test the extracted eye candidates to determine if they should be accepted. As indicated, the functionality of block62is detailed in the flow diagram ofFIG. 8, and the functionality of block64is detailed in the flow diagram ofFIG. 9. If at least one eye candidate is accepted, the block62sets the status of a Tracking Flag to IRHE; otherwise, the status of the Tracking Flag is set to NONE. The block66then checks the status of the Tracking Flag. If block66determines that the Tracking Flag status is IRHE, block68is executed to update a state vector that marks the location of the eye, whereafter the routine is re-executed for the next image frame. If the Tracking Flag status is NONE, the IRHE technique was unsuccessful, and DSP44executes block70to perform an alternate eye extraction technique using correlation and appearance testing (CAT). As indicated, the functionality of block70is detailed in the flow diagram of FIG.10. The block70also sets the status of the Tracking Flag—if at least one eye candidate is extracted and accepted, block70sets the status of the Tracking Flag to CAT; otherwise, the status of the Tracking Flag is set to NONE. The block72then checks the status of the Tracking Flag. If the Tracking Flag status is CAT, block68is executed to update the eye location state vector as explained above. If the Tracking Flag status is NONE, the alternate eye extraction technique of block70was also unsuccessful. This can occur, for example, when the driver's eye is closed. In this case, DSP44executes the missed-observation routine of block74to determine if the eye location can be identified based on peripheral eye-related patches. As indicated, the functionality of block74is detailed in the flow diagram ofFIG. 11. Block74also sets the status of the Tracking Flag—if at least one eye candidate is identified and accepted, the status of the Tracking Flag is set to EPT; otherwise, the status of the Tracking Flag is set to NONE. Finally, the block76checks the status of the Tracking Flag. If the Tracking Flag status is EPT, block68is executed to update the eye location state vector as explained above. If the Tracking Flag status is NONE, the occupant's previously detected eye could not be successfully tracked, and the block78is executed to re-start the eye detection routine50.

Referring toFIG. 8, the process of extracting eye candidates from the local search window (i.e., block62ofFIG. 7) is initiated at blocks80and82which apply a morphological bottom-hat filter to the LSW and binarize the filtered image. The bottom-hat filter is implemented with a mask of predefined size, and generally serves to suppress extraneous pixel information and emphasize the edges of objects and other definable features and regions within the LSW. The filtered image is initially binarized to form a “white image”—i.e., an image in which the dark pixels of the filtered image are represented by white pixels, and light pixels of the filtered image are represented by black pixels.FIG. 4depicts a “white image” version of a typical LSW—it is seen that the occupant's facial skin and other highly reflective regions are represented by black pixels, while dark regions such as the occupant's iris and pupil are represented by white pixels.

The block84scans the “white image” developed at block80to identify the largest “blob” of contiguous white pixels (referred to herein as an “eye-blob”) and to define a rectangular patch of the image (boundary box) that bounds the identified eye-blob. The filtered image of block80is then binarized at block86to form a “black image” which is the opposite of the “white image” depicted inFIG. 4. In other words, dark pixels of the filtered image (i.e., the occupant's iris and pupil) are represented by black pixels in the black image, and light pixels of the filtered image (i.e., the occupant's facial skin) are represented by white pixels. The block88extracts a patch of the binarized black image corresponding to the eye-blob boundary box defined at block84. An extracted black-image eye-blob patch based on the image ofFIG. 4is depicted in the upper left-hand corner ofFIG. 5.

Once the “black-image” eye-blob patch has been extracted, block90is executed to count the number of black pixels (receptive holes) in each horizontal line or row of the patch. InFIG. 5, the pixel counts are represented by the histogram to the right of the eye-blob patch. In most cases, the highest pixel count will generally occur in the pixel rows that correspond to the occupant's pupil, as is the case in the example ofFIG. 5. Block90identifies the pixel row having the highest pixel count or receptive hole content (MAX_COUNT_ROW) and block92identifies each black pixel in that row as an eye candidate center.

Block94extracts a grey-scale image patch of predetermined size centered about each eye candidate identified at block92, and performs an appearance-based test (eye vs. non-eye) of each patch. The appearance-based test utilizes an Eigen analysis in which Eigen-space distances between a respective candidate grey-scale patch and relatively low resolution (Level-I) eye and non-eye models are computed. Any candidate patch that resembles the LEVEL-I eye model will have a relatively short eye distance and a relatively long non-eye distance; any candidate patch that resembles the Level-I non-eye model will have a relatively short non-eye distance and a relatively long eye distance. The distances are compared to thresholds to classify the eye candidates as DESIRED, UNDESIRED or UNSURE. The DESIRED classification indicates a high confidence determination that an eye candidate is the center of the occupant's eye; i.e., that it resembles the Level-I eye model, and does not resemble the Level-I non-eye model. Conversely, the UNDESIRED classification indicates a high confidence determination that the eye candidate is a non-eye feature; i.e., that it resembles the Level-I non-eye model, and does not resemble the Level-I eye model. The UNSURE classification indicates that a high confidence determination cannot be made. The confidence metric is based on the separation between the eye distance and the non-eye distance, with larger separation signifying higher confidence in the appearance determination. Only those eye candidates classified as UNSURE or DESIRED are retained; eye candidates classified as UNDESIRED are eliminated from a list of identified eye candidates. The eye distances and classifications of the retained eye candidates are stored for use in a clustering portion of the extraction routine. As indicated by block96, the process defined by blocks92-94is repeated for a predefined number N of pixel rows above and below MAX_COUNT_ROW. The predefined number N depends on the application, and may be determined based on the available processing time. The end result is a list of potential eye candidates, each comprising an eye-sized patch of the grey-scale image.

The blocks98and100are then executed to cluster the extracted eye candidates. The block98pairs a selected eye candidate with neighboring eye candidates—i.e., with any eye candidate whose center is within a predetermined coordinate-space distance of the selected eye candidate. If such pairing is possible, the block100compares the stored Eigen eye distances for the respective eye candidates; the eye candidate having the smallest Eigen eye distance is retained, while the other eye candidate(s) is removed from the eye candidate list. As indicated at block102, the process defined by blocks98-100is then repeated for the next eye candidate until every eye candidate in the list has been processed.

Referring toFIG. 9, the process of appearance testing the retained eye candidates (i.e., block64ofFIG. 7) begins at block106where DSP44extracts a gray-scale patch of the LSW centered on a selected eye candidate. The block108then computes Eigen-space distances between the eye candidate and various Level-II models and compares the distances to thresholds to classify the respective eye candidate as DESIRED, UNDESIRED or UNSURE as described above in reference to block94ofFIG. 8. In the illustrated embodiment, there are three sets of Level-II appearance-based tests: eye vs. non-eye; closed-eye vs. non-eye; and eyeglasses vs. non-eye. Block110then checks the classification results. If the selected eye candidate is classified as UNDESIRED by all three tests, the blocks112and114are executed to eliminate the eye candidate from the list and to set the Tracking Flag to NONE. If the eye candidate is classified as DESIRED or UNSURE by at least one of the tests, the blocks116and118are executed to store the minimum Eigen distance attributed to the eye candidate and to set the Tracking Flag to IRHE. As indicated at block120, the process defined by blocks106-118is then repeated for the next eye candidate until every eye candidate in the list has been selected.

Referring toFIG. 10, the correlation/appearance testing (i.e., block70ofFIG. 7) involves attempting to locate an eye within the LSW by correlation (blocks122-128), and if successful, conducting appearance-based testing of each identified eye candidate (blocks132-140). The block122computes a similarity distance (i.e., a single normalized cross-correlation) between a grey-scale eye-box patch from the previous image frame and a corresponding patch from the current image frame. The block124compares the computed similarity distance to a threshold THR. If the distance is less than THR, the patches compare favorably, and block132is executed to begin appearance testing of the patch from the current image frame. If block124is answered in the negative, the block126is executed to compute a multiple, or pixel-by-pixel, correlation matrix between the grey-scale eye-box patch from the previous image frame and the LSW. This time, the block128checks if any of the correlation values exceed a threshold such as 90%. If so, the DSP44is directed to block132to begin appearance testing of the identified region(s) of the LSW. If not, an eye candidate could not verified by correlation, and the block130is executed to set the Tracking Flag to NONE. If blocks122or126do identify a high-correlation eye candidate, the block132computes Eigen-space distances between the eye candidate and various Level-II models and compares the distances to thresholds to classify the respective eye candidate as DESIRED, UNDESIRED or UNSURE as described above in reference toFIGS. 8-9. Here, block132compares the eye candidate to two different sets of Level-II appearance-based models: eye vs. non-eye; and eyeglasses vs. non-eye. Block134then checks the classification results. If the eye candidate is classified as DESIRED by either test, the block136is executed to set the Tracking Flag to CAT. Otherwise, the block138computes Eigen-space distances between the eye candidate and Level-II appearance models to judge closed-eye vs. non-eye and the corresponding classification. If the classification is DESIRED, the block140directs DSP44to set the Tracking Flag to CAT; otherwise the block130is executed to set the Tracking Flag to NONE.

Referring toFIG. 11, the missed observation routine (i.e., block74ofFIG. 7) is initiated at block142which extracts a set of eye-periphery image patches from the previous image frame and attempts to identify comparable patches in the LSW. The extracted image patches are clustered about, and geometrically symmetrical with respect to, the previously detected eye location. The image ofFIG. 6illustrates a mechanization in which DSP44extracts a set of four eye-periphery image patches. However, it should be understood that the number and placement of the eye-periphery patches may be different than shown. In order to identify comparable patches in the LSW, DSP44computes and evaluates a similarity distance between the extracted patches and each set of potentially matching patches in the LSW. The block144compares the similarity distances to a threshold THR. If less than a specified number of matches are found, the missed observation routine is unsuccessful, and the blocks146and148are executed to set the Tracking Flag to NONE and to reset a counter EPT_CTR to zero. However, if at least a specified number of matches are found, block144is answered in the affirmative, and blocks150and152are executed to compute the eye center according to the mean (center) of the matching eye-periphery patches and to set the Tracking Flag to EPT. Additionally, the block154increments EPT_CTR, and block156compares the count to a reference count REF such as sixty (which corresponds to two seconds at a frame rate of 30 Hz). If DSP44relies on eye-periphery tracking for REF successive image frames, block156is answered in the affirmative, and blocks158and160are executed to perform an appearance check to determine if the located eye region resembles a closed eye. The block158extracts a grey-scale eye patch centered on the computed eye center and computes Eigen-space distances between the eye patch and Level-II appearance models to judge closed-eye vs. non-eye and the corresponding classification. The block160checks the classification. If the classification is UNDESIRED, the blocks144and146are executed to set the Tracking Flag to NONE and to reset EPT_CTR to zero; otherwise, the status of the Tracking Flag is maintained as EPT and the block148is executed to reset EPT_CTR to zero.

In summary, the method of the present invention provides a multi-faceted approach to eye tracking that minimizes lost eyes even in situations where the person's head turns or the eyes momentarily close. While the present invention has been described with respect to the illustrated embodiment, it is recognized that numerous modifications and variations in addition to those mentioned herein will occur to those skilled in the art. For example, the method of this invention may be applied to a human facial feature other than an eye, or to a distinguishable target feature in general. Accordingly, it is intended that the invention not be limited to the disclosed embodiment, but that it have the full scope permitted by the language of the following claims.