Abstract:
The present disclosure describes compact remote eye tracking systems that include depth sensing capacity. In one aspect, a method of eye tracking includes illuminating a subject&#39;s eye with illumination; detecting, in a depth sensor, optical signals reflected from the subject&#39;s face in response to the illumination; generating a depth map based on the optical signals detected by the depth sensor; acquiring an intensity image based on optical signals reflected from the subject&#39;s face in response to the illumination; determining, based on a portion of the depth map corresponding to the subject&#39;s eye, an estimated distance between the subject&#39;s eye and a target screen; determining an estimated gaze direction of the subject&#39;s eye based on information obtained from the intensity image; and determining, based on the estimated gaze direction and the estimated distance between the subject&#39;s eye and the target screen, a gaze point for the subject&#39;s eye.

Description:
FIELD OF THE DISCLOSURE 
       [0001]    The present disclosure relates to compact remote eye tracking systems that include depth sensing capacity. 
       BACKGROUND 
       [0002]    Eye tracking refers to the process of determining eye movement and/or gaze point. Eye tracking is widely used, for example, in psychology and neuroscience, medical diagnosis, marketing, product and/or user interface design, and human-computer interactions. Some eye tracking methods use search coils, electrooculogram, or special contact lens. These methods are considered to be intrusive as the eye trackers need to be in contact with the subject&#39;s eyes or the face regions surrounding the eyes. 
         [0003]    Non-contact optical methods also have been developed for eye tracking. For example, light (e.g., infrared) can be directed to, and reflected by, the eye and sensed by a video camera or other optical sensor. The images or signals are analyzed to extract eye movement information. Video-based eye trackers usually utilize the corneal reflection and the center of the pupil, although some optical methods for eye tracking utilize image features from inside the eye, such as the retinal blood vessels. Corneal reflection and pupil center based eye tracking methods are widely used because they are non-invasive and relatively inexpensive. 
         [0004]    Eye tracking methods also can be divided into two categories: head-mounted and remote. Head-mounted eye trackers are worn on the subject&#39;s head, whereas remote eye trackers are placed at a certain distance from the subject, for example, on a table or computer display. 
         [0005]    The present disclosure relates to remote eye tracking, utilizing corneal reflection and pupil center. It is assumed there is a target screen plane at which the subject is looking (e.g., television screen, computer screen, electronic tablet or smart phone display). 
         [0006]      FIG. 1A  illustrates an example of a remote eye tracker utilizing corneal reflection and pupil center. Each illuminator  20  forms a bright spot in the image of the eye obtained by the camera  22  as a result of corneal reflection (see  FIG. 1B ). Each illuminator  20  can comprise a light source and may consist of multiple light emitting elements placed next to each other. If the subject&#39;s head is at a fixed pose, the direction of the subject&#39;s gaze can be determined by the vector formed between the center of corneal reflection and the center of pupil (see “OG” in  FIG. 1( c ) ), which can be mapped to a target screen. Thus the coordinates of the gaze point (G) can be determined using the horizontal and vertical components of the gaze direction and its distance from the subject&#39;s eye  24 . 
         [0007]    Using two or more illuminators  20  in video eye trackers can be advantageous for several reasons. First, when there is head movement during eye tracking, the gaze direction is dependent on the head pose, in addition to the pupil center and corneal reflection. Multiple corneal reflections from the illuminators provides additional information from which head pose can be determined, and thus allow head movements during eye tracking. 
         [0008]    Further, the separation between the corneal reflections appearing on the image ( FIG. 1B ) sensed by the camera  22  is a function of the distance (D) between the eye and the eye tracker. For a given camera-illuminator configuration, the corneal reflection separation in the image becomes smaller when the eye is further away from the eye tracker. Thus the eye to eye tracker distance (D) can be estimated from the corneal reflection separation. In general, as the eye to eye tracker distance (D) increases, the corneal reflection separation becomes less sensitive (i.e., diminishing differentiation power). In such instances, it is necessary to increase the illuminator separation (L) for larger eye to eye tracker distance (D). The illuminator separation (L), however, is one of the main factors affecting the size (e.g., length) of a remote video eye tracker. 
       SUMMARY 
       [0009]    The present disclosure describes compact remote eye tracking systems that include depth sensing capacity. 
         [0010]    For example, in one aspect, this disclosure describes a method of eye tracking including illuminating a subject&#39;s eye with illumination. The method includes detecting, in a depth sensor, optical signals reflected from the subject&#39;s face in response to the illumination, generating a depth map based on the optical signals detected by the depth sensor, and acquiring an intensity image based on optical signals reflected from the subject&#39;s face in response to the illumination. The method further includes determining, based on a portion of the depth map corresponding to the subject&#39;s eye, an estimated distance between the subject&#39;s eye and a target screen, determining an estimated gaze direction of the subject&#39;s eye based on information obtained from the intensity image, and determining, based on the estimated gaze direction and the estimated distance between the subject&#39;s eye and the target screen, a gaze point for the subject&#39;s eye. 
         [0011]    In another aspect, this disclosure describes an eye tracking apparatus that includes an illuminator operable to illuminate a subject&#39;s eye with illumination. The apparatus includes a depth sensor operable to detect optical signals reflected from the subject&#39;s face in response to the illumination, and an image sensor operable to acquire an intensity image based on optical signals reflected from the subject&#39;s face in response to the illumination. The apparatus also includes one or more processors configured to generate a depth map based on the optical signals detected by the depth sensor, determine, based on a portion of the depth map corresponding to the subject&#39;s eye, an estimated distance between the subject&#39;s eye and a target screen, determine an estimated gaze direction of the subject&#39;s eye based on information obtained from the intensity image, and determine, based on the estimated gaze direction and the estimated distance between the subject&#39;s eye and the target screen, a gaze point for the subject&#39;s eye. 
         [0012]    Some implementations include one or more of the following features. For example, in some instances, the depth sensor includes a time-of-flight sensor. Further, in some cases, both of the depth sensor and the image sensor are implemented by the same time-of-flight sensor. In some instances, the image sensor is implemented by an infra-red camera different from the depth sensor. 
         [0013]    In some implementations, integration timing of the depth sensor and the infra-red camera can be correlated to timing of the illumination by the illuminator. The processor(s) can be configured to apply a face segmentation to identify a portion of the intensity image representing the subject&#39;s eye, and to determine which signals acquired by the depth sensor correspond to the subject&#39;s eye based on a mapping between pixels of the depth sensor and pixels of the infra-red camera. The processor(s) also can be configured to determine the estimated gaze direction of the subject&#39;s eye using an estimated pupil center and an estimated corneal reflection obtained from information in the intensity image. 
         [0014]    In some implementations, a controller is configured to control a field of the illumination by the illuminator dynamically based on a location and distance of the subject&#39;s face and/or eye as determined in one or more previous frames. 
         [0015]    Some implementations can provide a relatively compact eye tracker that obviates one or more of the difficulties encountered in known eye tracking systems. Further, in some cases, the eye tracker can be implemented as a module having much smaller dimensions (e.g., half the size) as compared to prior eye-tracking modules. 
         [0016]    Other aspects, features and advantages will be apparent from the following detailed description, the accompanying drawings and the claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0017]      FIG. 1A  illustrates an example of a remote eye tracking system. 
           [0018]      FIG. 1B  illustrates an image acquired by the camera in the remote eye tracker of  FIG. 1A . 
           [0019]      FIG. 1C  illustrates geometrical relationships for the remote eye tracker of  FIG. 1A . 
           [0020]      FIG. 2  illustrates an example of an eye tracking system. 
           [0021]      FIG. 3  is an example of a timing diagram for the eye tracking system of  FIG. 2 . 
           [0022]      FIG. 4  is a flow chart showing an example of a method for eye tracking using depth sensing. 
           [0023]      FIG. 5  illustrates an example of mapping pixels from an infra-red camera to a time-of-flight depth sensor. 
           [0024]      FIGS. 6A and 6B  illustrate examples of arrangement of light emitting elements in an illuminator. 
           [0025]      FIG. 7  illustrates an example of an eye tracking system. 
           [0026]      FIG. 8  is an example of a timing diagram for the eye tracking system of  FIG. 7 . 
           [0027]      FIG. 9  illustrates an example of an eye tracking system. 
       
    
    
     DETAILED DESCRIPTION 
       [0028]    As shown in  FIG. 2 , a remote eye tracker  100  includes an illuminator  102  that is operable to emit light (e.g., infra-red) toward a subject&#39;s face including the subject&#39;s eye  110 . The illuminator  102  emits homogenous illumination  108  of a specific wavelength and can be modulated, for example, at a relatively high frequency (e.g., 10-100 MHz). The eye tracker  100  also includes a depth sensor such as an optical time-of-flight (TOF) sensor  104  that can detect optical signals indicative of distance to the subject&#39;s eye  110 . The TOF sensor  104  is operable to demodulate the acquired signals and to generate depth data. Thus, the TOF sensor  104  provides depth sensing capability for the eye tracker  100 . The eye tracker  100  of  FIG. 1  further includes an infra-red (IR) image sensor, such as a near-IR (NIR) camera  106 , which is operable to acquire an intensity image of the subject&#39;s eye  110 . The image sensor  106  can be implemented, for example, as a CMOS or CCD camera. The TOF sensor  104  and NIR camera  106  are operable to detect light having the same wavelength as the illumination  108  generated by the illuminator  102 . 
         [0029]    Preferably, the optical axes of the illuminator  102  and the NIR camera  106  should be positioned such that there is an angle between them of no less than about five degrees. Under such conditions, the pupil of the subject&#39;s eye appears as a black circle or ellipse in the image of the eye acquired by the NIR camera  106 . It also helps reduce the impact of specular reflections from spectacles or contact lens worn by the subject. 
         [0030]    Operations of both the NIR camera  106  and TOF sensor  104  are synchronized with the illuminator  102  such that their integration timings are correlated to the timing of the illuminator  102 . An example of the timing is illustrated in  FIG. 3 . 
         [0031]    A method of determining a subject&#39;s gaze point on the target screen (e.g., television screen, computer screen, electronic tablet or smart phone display) is described with respect to  FIGS. 2 and 4 . Preferably, the TOF sensor  104  and NIR camera  106  are calibrated so that geometric relationships are established between a depth map subsequently generated from the signals acquired by the TOF sensor  104 , on the one hand, and an image acquired by the NIR camera  106 : 
         [0000]        q ( u,v )= f{p ( x,y )}, 
         [0000]    where (u, v) and (x, y) are the pixel coordinates of point q and p in the TOF sensor  104  and NIR camera  106 , respectively. This provides a mapping between pixels of the TOF sensor  104  and pixels of the NIR camera  106  (see  201  in  FIG. 4 ). 
         [0032]    When the subject&#39;s face (including the subject&#39;s eye  110 ) is illuminated by the illumination  108 , light reflected by the subject&#39;s face is sensed by the TOF sensor  104  and the NIR camera  106 . The eye&#39;s cornea is highly reflective (i.e., spectrally reflective), and thus the homogenous illumination from the illuminator  102  reflects from the eye&#39;s cornea as a dot. The reflected dot is incident on, and sensed by, the TOF sensor  104  and the NIR camera  106 . Other parts of the subject&#39;s face are diffusively reflective, and thus are not as reflective as the subject&#39;s eye  110 . The NIR camera  106  acquires an image of the subject&#39;s face, including the eye  110  (see  202  in  FIG. 4 ). 
         [0033]    The process of  FIG. 4  then applies a face segmentation technique to detect the portion of the image representing the eye  110  (see  204  in  FIG. 4 ) and an eye segmentation technique to detect the portion of the image representing the eye&#39;s cornea (see  206  in  FIG. 4 ). Known techniques can be used for the face segmentation and eye segmentation. For example, in some cases, eye segmentation segments the image of the subject&#39;s eye into a multitude of features according to a segmentation model that includes the subject&#39;s eye and the region around the subject&#39;s eye. 
         [0034]    The process continues by estimating the center of the corneal reflection ( 208  in  FIG. 4 ) and estimating the center of the eye&#39;s pupil ( 210  in  FIG. 4 ). The process uses the estimated pupil center and the estimated corneal reflection to determine an estimated gaze direction of the subject&#39;s eye (see  212  in  FIG. 4 ). The estimated gaze line, which sometimes is referred to as a gaze vector or line of sight, runs from the cornea center through the pupil center. In particular, by analyzing the geometry and orientation of the subject&#39;s eye as captured in the image, the process estimates a gaze direction indicating where the subject&#39;s eye is looking. In some instances, the estimated gaze is adjusted based on eye tracker calibration. 
         [0035]    In parallel with performance of the foregoing steps, the process generates a depth map based on the signals acquired by the TOF sensor  104  (see  214  in  FIG. 4 ). Next, the process determines which signals acquired by the TOF depth sensor  104  correspond to the subject&#39;s face, including the subject&#39;s eye  110  (see  216  in  FIG. 4 ). To accomplish this, the process uses the mapping between the TOF sensor pixels and the IR camera pixels ( 201 ) and the output of the face and/or eye segmentations ( 204 ,  206 ). Thus, the locations of the subject&#39;s face and eye obtained from the face and eye detection carried out on the image from the NIR camera  106  are mapped to the depth map generated from the signals sensed by the TOF sensor  104 .  FIG. 5  illustrates graphically an example of the mapping between the portion of the image of the subject&#39;s face acquired by the NIR camera  106  and the corresponding part of the depth map based on the TOF sensor  104 . 
         [0036]    The process then proceeds to determine, based on the portion of the depth map corresponding to the subject&#39;s eye  110 , an estimate of the distance between the subject&#39;s eye  110  and the target screen (see  218  in  FIG. 4 ). Next, the process predicts the subject&#39;s gaze point on the target screen (see  220  in  FIG. 4 ) based on the target-screen eye distance (from  218 ) and the estimated gaze direction (from  212 ). 
         [0037]    If the subject&#39;s head moves during the eye tracking process, the gaze direction also is dependent on the head pose. In that case, the depth map information for the subject&#39;s face and/or eyes in  216  can be used to estimate the subject&#39;s head pose, which then can be provided as further input for estimating the gaze direction in  212 . 
         [0038]    Additional details for some implementations of the eye tracker are described in the following paragraphs. 
         [0039]    The illuminator  102  can include one or more light emitter such as light-emitting diodes (LEDs) or vertical-cavity surface-emitting lasers (VCSELs). In some cases, one illuminator  102  includes an array of light emitting elements and a beam shaping structure (such as lens) to shape the field of illumination. The light emitting elements can be arranged in such a way that dynamic control of the field of illumination is possible. For example, as shown in  FIG. 6A , either central section  302  or the entire light emitter array (i.e., central section  302  and peripheral section  304 ) can be enabled. On the other hand, in the example of  FIG. 6B , various combinations of sections  306 A,  306 B  306 C,  306 D of the array  300  can be enabled. Thus, in some instances, only two of the sections (e.g.,  306 A and  306 B; or  306 B and  306 C) of the array  100  are enabled, whereas in other instances, only three of the sections (e.g.,  306 A,  306 B and  306 C; or  306 A,  306 B and  306 D) of the array  100  are enabled. In some cases, the entire array is enabled. A respective electrical supply line  308  is provided for the light emitting elements  110  in each section of the array. The decision as to which sections of the array  100  to enable under such dynamic control can be based on the location and distance of the face and/or eyes as determined from one or more previous frames. A controller (e.g., a microcontroller or other control circuitry) can be included to provide the feedback for dynamic control of the field of illumination. 
         [0040]    In some implementations, for each frame during a period of time T, the illuminator  102  is on (i.e., emits illumination  108 ) only for a fraction of the time T. Further, in some cases, the intensity of illumination  108  is adjusted dynamically based on the eye-to-screen distance measurements from previous frames. For example, a higher intensity can be generated for larger distances D, whereas lower intensities can be generated for smaller distances D. The intensity adjustments can be achieved by changing the electrical current level to the light emitters in the illuminator  102 . In some instances, these features can help optimize, and thereby reduce, the power consumption of the illuminator  102 . 
         [0041]    The number of prior frames used to estimate the light emitting sections to be enabled, as well as the illumination levels, can be determined, for example, based on expected head motions and distance ranges in a specific application. In some implementations, using one to three prior frames can achieve suitable results. 
         [0042]    For safety and power consumption considerations, the illuminator  102  and the eye tracking system should be configured to shut off automatically if it is determined the subject&#39;s face/eye is too close to the eye tracker (e.g., within 10 cm). 
         [0043]    The field of illumination should be matched closely to the field of views of both the NIR camera  106  and TOF sensor  104 . In some cases, it is desirable to provide a rectangular field of illumination with the long edge along the horizontal direction and the short edge along the vertical direction because head movements tend to be more significant and frequent in the horizontal direction than in the vertical direction. 
         [0044]    When the process determines which signals acquired by the TOF depth sensor  104  correspond to the subject&#39;s face  110  (see  216  in  FIG. 4 ), it sometimes is desirable to exclude certain pixels from the determination. For example, pixels near the boundary of the face/eye portion of the depth map may represent mixed signals from both the face/eye and the more distant background. Such pixels should be excluded when the eye-to-screen distance (D) is estimated at  218  ( FIG. 4 ). 
         [0045]    In some cases, a confidence level is derived for depth measurement of each pixel in the depth sensor  104 . Depth pixels with low confidence levels, even though they are located within the face/eye region, can be excluded for eye-to-screen distance estimation. When the depth sensor resolution is low (e.g., when the number of pixels is low), it is preferable to use the face region to estimate the eye- to-screen distance (D). Otherwise, it is preferable to the eye region alone, which can provide a more accurate estimation in many situations. 
         [0046]    In some implementations, the gaze direction estimation is performed in two stages by machine learning. For example, a large set of pre-collected eye depth maps and corresponding eye images (or particular features thereof) are used to train a neural network or support vector machine such that an implicit model is established in the neural network or support vector machine. During eye tracking, a new eye depth map and corresponding eye image (or the same features thereof as used in the training stage) are provide to the trained neural network or support vector machine for it to make a prediction of the gaze direction. The training stage typically can be performed once for the eye tracking system. 
         [0047]    As described above, the implementation of  FIG. 2  includes two cameras (i.e., the TOF sensor  104  and the NIR camera  106 ). In some implementations, however, as shown in  FIG. 6 , a single TOF camera  104  is used to provide an intensity image as well as the depth information. In this case, the resolution of the TOF camera  104  should be high enough such that its intensity image can be used instead of an image from the NIR camera  106  of  FIG. 2 . 
         [0048]    In the n-phase TOF camera  104  of  FIG. 6 , the intensity image (B) can be calculated as follows: 
         [0000]        B =√{square root over (( P   i ) 2 +( P   2 ) 2 + . . . +( P   n ) 2  )}
 
         [0000]    where (P i ) 2  (for i=1, 2, . . . n) is the intensity at an individual phase. For eye tracking purpose, the pixel values of this intensity image (B) are adjusted using the corresponding distance image, as well as the illumination profile and camera vignette profile. The latter two characteristics are constant for a give TOF camera and can be obtained, for example, during camera characterization. Operations of the TOF sensor  104  of  FIG. 6  is synchronized with the illuminator  102  such that the integration timing is correlated to the timing of the illuminator  102 . An example of the timing is illustrated in  FIG. 7 . Also, in the implementation of  FIG. 6 , the depth map and face/eye intensity image are already perfectly aligned. Thus, 
         [0000]        q ( u,v )= p ( x,y ). 
         [0049]    Various implementations described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementations in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. 
         [0050]    These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” “computer-readable medium” refers to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. 
         [0051]    Thus, as shown in  FIG. 9 , in some cases, one or more processors (e.g., microprocessor or microcontroller)  200  are programmed or otherwise configured to control the illuminator  102 , receive signals generated by the TOF sensor  104  and NIR camera  106 , and process the signals, for example, in accordance with the method of  FIG. 4  so as to generate the gaze point prediction. 
         [0052]    The calculated gaze point can be used, for example, as input for various applications (e.g., diagnostic, human performance and/or control applications). Particular examples include assessing the effectiveness of television advertising or web-page designs by measuring ocular dwell times, monitoring the degree of fatigue of an individual, tracking human performance with age, determining the effectiveness of training or exercise on performance, assessing driver or pilot awareness, assessing effects of drugs or alcohol on an individual, diagnosing post-traumatic stress disorder, magnifying or changing the brightness of specific objects or images under observation, controlling aspects of games, diagnosing and monitoring degenerative eye conditions, allowing individuals with limited or no mobility below the neck to communicate by controlling a computer cursor using one or more eyes and eyelids, and acquiring foundational clinical data to assess neurological or cognitive disorders. The eye tracking systems and methods described above can be used in any of these applications, as well as others. In particular, output generated by the eye tracking system or eye tracking method can be used in any of the foregoing applications. 
         [0053]    As will be readily apparent, various modifications can be made to the foregoing examples within the spirit of the invention. For example, in some instances, some processes or steps may be omitted. Further, in some cases, additional processes or steps may be performed. Other modifications may be made as well. Thus, other implementations are within the scope of the claims.