Patent Description:
For example, any detail discussed with regard to one embodiment may or may not be present in all contemplated versions of that embodiment. Likewise, any detail discussed with regard to one embodiment may or may not be present in all contemplated versions of other embodiments discussed herein. Finally, the absence of discussion of any detail with regard to embodiment herein shall be an implicit recognition that such detail may or may not be present in any version of any embodiment discussed herein.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other elements in the invention may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail.

Also, it is noted that individual embodiments may be described as a process which is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. A process may be terminated when its operations are completed, but could have additional steps not discussed or included in a figure. Furthermore, not all operations in any particularly described process may occur in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

The term "machine-readable medium" or the like includes, but is not limited to transitory and non-transitory, portable or fixed storage devices, optical storage devices, wireless channels and various other mediums capable of storing, containing or carrying instruction(s) and/or data. A code segment or machine-executable instructions may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and/or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc..

Furthermore, embodiments of the invention may be implemented, at least in part, either manually or automatically.

The present disclosure generally relates to the field of eye tracking. In particular, the present disclosure relates to systems and methods for use in generating gaze tracking data indicating an eye direction and/or gaze direction of an eye.

Several different eye tracking systems are known in the art. Such systems may for example be employed to identify a location at a display at which a user is looking and/or the gaze direction of the user. Some eye tracking systems capture images of at least one of a user's eyes, and then employ image processing to extract key features from the eye, such as a pupil center together with glints (corneal reflections) from illuminators illuminating the eye. The extracted features may then be employed to determine where at a display the user is looking, the eye direction of the user and/or the gaze direction of the user.

<CIT>, family member of <CIT>, discloses an information processing apparatus including: a detection unit configured to detect a corneal reflection image corresponding to light from a light source reflected at a cornea from a captured image in which an eye irradiated with the light from the light source is imaged. The detection unit estimates a position of a center of an eyeball on the basis of a plurality of time-series captured images each of which is the captured image according to the above, estimates a position of a center of the cornea on the basis of the estimated position of the center of the eyeball, estimates a position of a candidate for the corneal reflection image on the basis of the estimated position of the center of the cornea, and detects the corneal reflection image from the captured image on the basis of the estimated position of the candidate for the corneal reflection image.

<CIT> discloses an apparatus and method for automatically determining a strabismus angle of the eyes of an individual by performing a reflection test on said eyes, and estimating for both eyes individually pupil center coordinates. It is disclosed that the corneal center coordinates and the pupil center coordinates of both eyes are used to estimate the strabismus angle. For the eye that is fixating an angle kappa is calculated representing an angle between the eyes' optical axis through the corneal center and the pupil center, and the visual axis through the corneal center and the eye's fovea, and the angle kappa is then used for determining the strabismus angle.

One known method of eye tracking includes the use of infrared light and an image sensor. The infrared light is directed towards the pupil of a user and the reflection of the light is captured by an image sensor. Through analysis of the reflection point, the direction of the user's gaze may be calculated. One such system is described in <CIT>.

Portable or wearable eye tracking devices have also been previously described. One such eye tracking system is described in <CIT>.

A wearable eye tracking device is described using illuminators and image sensors for determining gaze direction.

In applications of eye tracking for in portable or wearable eye tracking devices, such as in virtual reality (VR) devices and augmented reality (AR) devices, where head mounted devices are used which include an eye tracking system determining an eye direction and/or gaze direction based on a pupil center and glints from illuminators illuminating a user's eyes, situations can arise where a user's eyes are directed in relation to illuminators such that no or too few glints can be identified for eye tracking, or the glint or glints identified are difficult to pair with the respective illuminator causing the glint. In such situations it will be difficult or impossible to determine eye direction and/or gaze direction and or eye direction or at least not with desirable reliability.

It would be desirable to provide an eye tracking technology to account for such situations where with no or too few glints identified for eye tracking, or that the glint or glints identified are difficult to correlate to the illumination rays.

An object of the present disclosure is to address at least one of the issues described above.

According to a first aspect, there is provided a method comprising, which is defined in claim <NUM>, obtaining an estimated radius from an eyeball center to a pupil center in an eye, and determining an estimated eyeball center position in the eye in relation to an image sensor for capturing images of the eye. Furthermore, an image of the eye captured by means of the image sensor is obtained and a position of a representation of the pupil center in the eye in the obtained image is determined. Finally, an estimated pupil center position is determined based on the estimated eyeball center position, the estimated radius, and the identified position of the representation of the pupil center in the obtained image.

In some situations, a primary method for determining eye direction and/or gaze direction of an eye fails to provide reliable results. This may be caused by a current eye direction or other temporary factors affecting the possibility for the method to provide reliable results of eye direction and/or gaze direction. For some applications of eye tracking, another method based on an estimated eyeball center position can then be used instead of the primary method for determining an eye direction in such situations. More specifically, such methods using an estimated eyeball center position can be used in applications of eye tracking where the eyeball center position can be approximated to be constant in relation to an image sensor capturing images of the eye regardless of eye direction. The approximation will be valid at least for some period of time.

The estimated eyeball center position may be determined in a situation where it is possible for it to be determined with required reliability. The estimated eyeball center position may for example be determined based on the primary method for determining eye tracking and/or gaze tracking. However, it will be appreciated that any other method can be used as long as estimated eyeball center position may be determined with required reliability.

When the estimated eyeball center position has been determined, this estimation can then be used for situations when it is not possible to determine the eye direction and/or gaze direction with required reliability based on the primary method. An image of the eye captured by means of the image sensor is obtained and a position of a representation of the pupil center in the eye in the obtained image is determined. An estimated radius from an eyeball center to a pupil center of an eye is obtained. The estimated radius is an approximation of the distance from the eyeball center to the pupil center of a human eye. As the estimated eyeball center position in relation to the image sensor is known, and the estimated radius from the eyeball center to the pupil center is known, these can be combined with the identified representation of the pupil center in the image captured by the image sensor to determine an estimated pupil center position.

The determined pupil center position is the actual position of the pupil center in relation to the image sensor.

The image sensor can be any type of imaging sensor consisting of an integrated circuit containing an array of pixel sensors, each pixel containing a photodetector and an active amplifier. The image sensor is capable of converting light into digital signals. In reality, as an example, it could be:.

The shutter mechanisms of the image sensors can be either rolling shutter or global shutter.

In embodiments, the estimated radius is first selected as the approximation of the distance from the eyeball center to the pupil center of a human eye. When further data regarding the eyes of a current user are retrieved the estimated radius can be updated.

In embodiments implemented in a system as defined in claim <NUM>, where a primary method for determining eye direction and/or gaze direction of an eye is based on determining a pupil center in the eye and glints (corneal reflections) on the eye from one or more illuminators illuminating the eye, an example of a situation where a primary method does not provide reliable results, is when an angle between a direction of the eye in relation to one or more illumination rays from the one or more illuminators becomes large, e.g. when the eye is directed to points at the edge of a display of the system. In such a situation difficulties may arise in identifying any glints or at least a sufficient number of glints on the eye, and in particular on a cornea of the eye, or that an identified glint or identified glints cannot be associated to a corresponding illuminator of the one or more illuminators. In such a situation it may not be possible to determine with required reliability an eye direction and/or gaze direction of the eye based on the identified glint or glints using the primary method. If it is determined that it is not possible to determine the eye direction and/or gaze direction with required reliability based on the primary method, the method of determining eye direction and/or gaze direction of an eye may use a secondary method for determining eye direction and/or gaze direction of the eye based on an estimated eyeball center position and an estimated pupil center position, the estimated pupil center position being determined based on the estimated eyeball center position, an estimated radius, and an identified position of a representation of a pupil center in an image obtained by the image sensor.

One example of an application of eye tracking where another method based on an estimated eyeball center position can be used instead of the primary method for determining an eye direction in situations where the primary method fails is wearable devices, such as devices for VR and AR, where one or more image sensors are positioned on or in the wearable device, and hence, will not move in relation to the eyes when the user moves as long as the user is wearing the device. In such an application, an eyeball center position can be approximated to be constant in relation to an image sensor capturing images of the eye regardless of the eye direction and head position.

For wearable devices, such as devices for VR and AR, with a primary method for determining eye direction and/or gaze direction of an eye is based on determining a pupil center of and glints on the eye from one or more illuminators illuminating the eye, an estimated eyeball center position in the eye in relation to an image sensor for capturing images of the eye can for example be determined for a situation where sufficient number of glints are identified on the eye, and the glint or glints identified can be associated to the corresponding illuminator or illuminators, such that it is possible to determine the estimated eyeball center position with required reliability. For a situation where the primary method cannot identify any glints or at least not a sufficient number of glints on the eye, or where an identified glint or identified glints cannot be associated to a corresponding illuminator of the one or more illuminators, a method based on the estimated eyeball center position can be used to determine an eye direction and/or gaze direction.

Once the estimated pupil center position has been determined, according to some embodiments, an estimated eye direction is determined based on a vector from the estimated eyeball center position to the estimated pupil center position.

Furthermore, according to some embodiments, an estimated gaze direction is determined based on the estimated eye direction.

According to some embodiments, an estimated distance from a cornea sphere center in the eye to the pupil center and an estimated distance from the eyeball center to the cornea sphere center are obtained. The estimated radius from the eyeball center to the pupil center is then equal to the sum of the estimated distance from the cornea sphere center to the pupil center and the estimated distance from the eyeball center to the cornea sphere center.

The cornea of the eye is typically approximately spherical in a central region of the cornea located around the pupil, but deviates more from the spherical shape further away from the center of the cornea. The central region of the cornea may therefore be referred to as a spherical region, while the region of the cornea outside the spherical region may be referred to as a non-spherical region. The cornea sphere center is defined as a position at the center of an imaginary sphere of which the spherical region of the cornea forms part.

The estimated distance from the cornea sphere center to the pupil center, also known as pupil depth, can be a constant approximation or may be dynamically updated.

For example, in embodiments implemented in a system using a primary method for determining eye direction and/or gaze direction of an eye based on determining a pupil center in the eye and glints on the eye from one or more illuminators illuminating the eye, a position of the cornea sphere center is determined for a situation where there are sufficient number of glints on the eye, and the glint or glints identified can be associated to the corresponding illuminator of the one or more illuminators, such that it is possible to determine with required reliability the position of the cornea sphere center. From this the distance from the cornea center to the pupil center can be determined.

The estimated distance from the eyeball center to the cornea sphere center can be a constant approximation or may be dynamically updated.

According to some embodiments, the estimated pupil center position is determined based on the estimated eyeball center position, the estimated radius from the eyeball center to the pupil center in the eye, and the identified position of the representation of the pupil center in the obtained image by first defining an eyeball sphere having a center in the estimated eyeball center position. The eyeball sphere further has a radius of the estimated radius. Based on the identified position of the representation of the pupil center in the obtained image, a pupil ray along which the pupil center is positioned is determined. The pupil ray is a projection in three-dimensional space from the image sensor of the identified position of the representation of the pupil center in the obtained image. The estimated pupil center position is then determined as an intersection between the eyeball sphere and the pupil ray.

According to some embodiments, determining the estimated pupil center position using an iterative method taking into account refraction in a cornea of the eye from a surface of the cornea to the pupil center.

In these embodiments, refraction in the cornea is taken into account, i.e. the pupil center will not be positioned on the pupil ray being a linear projection in three-dimensional space, but the pupil ray will be refracted in the cornea of the eye.

According to some embodiments, an estimated distance from a cornea sphere center in the eye to the pupil center in the eye is obtained. An estimated cornea sphere center position is then determined as the obtained estimated distance from the cornea sphere center to the pupil center from the estimated pupil center position in the direction towards the estimated eyeball center position, i.e. starting in the estimated pupil center position and moving, in the direction from the pupil center towards the estimated eyeball center position, the obtained estimated distance from the cornea sphere center to the pupil center.

According to some embodiments the estimated cornea sphere center position can be used to predict positions of one or more glints on the eye.

In embodiments implemented in a system using a primary method for determining eye direction and/or gaze direction of an eye based on determining a pupil center in the eye and glints on the eye from one or more illuminators illuminating the eye, the estimated cornea sphere center position can be used to predict positions of one or more glints before the primary method has determined the cornea sphere center position. The prediction can then be used in associate each of the one or more glints to a corresponding illuminator of the one or more illuminators. This is useful even for a situation where there are sufficient number of glints on the eye for it is possible to determine with required reliability the position of the cornea sphere center based on the primary method.

According to some embodiments, the estimated eyeball center position is determined by determining a cornea sphere center position is determined based on one or more glint positions on the eye. Furthermore, a pupil center position is determined, and an estimated distance from the eyeball center to the cornea sphere center is obtained. The estimated eyeball center position is then determined as the estimated distance from the eyeball center to the cornea sphere center from the cornea sphere center position in a direction from the pupil center position to the cornea sphere center position, i.e. starting in the determined cornea sphere center position and moving, in the direction from the determined pupil center position to determined cornea sphere center position, the obtained estimated distance from the eyeball center to the cornea sphere center.

In embodiments implemented in a system defined in claim <NUM>, where a primary method for determining eye direction and/or gaze direction of an eye is based on determining a pupil center in and glints on the eye from one or more illuminators illuminating the eye, the estimated eyeball center position may be determined in a situation where a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the one or more illuminators, such that such that it is possible to determine with required reliability the cornea sphere center position and the eye direction. The determined sphere center position is then used together with the eye direction, and the estimated distance from the eyeball center to the cornea sphere center to determine the estimated eyeball center position. The estimated eyeball center position may then be used for situations when it is not possible to determine the eye direction and/or gaze direction with required reliability based on the primary method. Such situations are for example when an angle between a direction of the eye in relation to one or more illumination rays from the one or more illuminators becomes large, e.g. when the eye is directed to points at the edge of a display of the system. In such situations it may become difficult to identify sufficient number of glints identified on the eye and/or to associate identified glints to a corresponding illuminator of the one or more illuminators to determine with required reliability an eye direction and/or gaze direction of the eye using the primary method.

According to a second aspect, there is provided an eye tracking system comprising a circuitry configured to obtain an estimated radius from an eyeball center to a pupil center of an eye, and determine an estimated eyeball center position in the eye in relation to an image sensor for capturing images of the eye. The circuitry is further configured to obtain an image of the eye captured by means of the image sensor and determine a position of a representation of the pupil center in the eye in the obtained image is determined. The circuitry is further configured to determine an estimated pupil center position is based on the estimated eyeball center position, the estimated radius, and the identified position of the representation of the pupil center in the obtained image.

The system of the second aspect, or the circuitry comprised in such a system, is configured to perform any of the embodiments of the first aspect.

Furthermore, embodiments of the system according to the second aspect include features corresponding to the features of any of the embodiments of the system according to the first aspect.

According to embodiments, the eye tracking system further comprises the image sensor for capturing images of the eye.

According to embodiments, the image sensor is arranged in a wearable device.

In wearable devices, such as devices for VR and AR, where one or more image sensors are positioned on the wearable device, and hence, will not move in relation to the eye when the user moves as long as the user is wearing the device, and the eyeball center position can be approximated to be constant in relation to an image sensor capturing images of the eye regardless of the eye direction. Hence, another method based on an estimated eyeball center position can be used instead of the primary for determining an eye direction in situations where a primary method for determining eye direction and/or gaze direction is not able to produce reliable results.

According to embodiments, the circuitry is arranged in a device different from the wearable device.

According to embodiments, the circuitry is arranged in the wearable device.

According to a third aspect, there is provided one or more computer-readable storage media storing computer-executable instructions that, when executed by a computing system that implements eye/gaze data processing, cause the computing system to perform a method. The method may for example be the method according to the second aspect.

Embodiments of the one or more computer-readable storage media according to the third aspect may for example include features corresponding to the features of any of the embodiments of the method according to the first aspect.

The one or more computer-readable media may for example be one or more non-transitory computer-readable media.

It is noted that embodiments of the invention relate to all possible combinations of features recited in the claims.

Exemplifying embodiments will be described below with reference to the accompanying drawings:.

All the figures are schematic and generally only show parts which are necessary in order to elucidate the respective embodiments, whereas other parts may be omitted or merely suggested.

<FIG> shows a simplified view of an eye tacking system <NUM> (which may also be referred to as a gaze tracking system) in which embodiments of the invention may be implemented. The system <NUM> comprises illuminators <NUM> and <NUM> for illuminating the eyes of a user, and a light sensor <NUM> for capturing images of the eyes of the user. The illuminators <NUM> and <NUM> may for example be light emitting diodes emitting light in the infrared frequency band, or in the near infrared frequency band. The light sensor <NUM> may for example be an image sensor, such as a complementary metal oxide semiconductor (CMOS) image sensor or a charged coupled device (CCD) image sensor.

A first illuminator <NUM> is arranged coaxially with (or close to) the light sensor <NUM> so that the light sensor <NUM> may capture bright pupil images of the user's eyes. Due to the coaxial arrangement of the first illuminator <NUM> and the light sensor <NUM>, light reflected from the retina of an eye returns back out through the pupil towards the light sensor <NUM>, so that the pupil appears brighter than the iris surrounding it in images where the first illuminator <NUM> illuminates the eye. A second illuminator <NUM> is arranged non-coaxially with (or further away from) the light sensor <NUM> for capturing dark pupil images. Due to the non-coaxial arrangement of the second illuminator <NUM> and the light sensor <NUM>, light reflected from the retina of an eye does not reach the light sensor <NUM> and the pupil appears darker than the iris surrounding it in images where the second illuminator <NUM> illuminates the eye. The illuminators <NUM> and <NUM> may for example take turns to illuminate the eye, so that every second image is a bright pupil image, and every second image is a dark pupil image.

The eye tracking system <NUM> also comprises circuitry <NUM> (for example including one or more processors) for processing the images captured by the light sensor <NUM>. The circuitry <NUM> may for example be connected to the light sensor <NUM> and the illuminators <NUM> and <NUM> via a wired or a wireless connection and be co-located with the light sensor <NUM> and the illuminators <NUM> and <NUM> or located at a distance, e.g. in a different device. In another example, circuitry <NUM> in the form of one or more processors may be provided in one or more stacked layers below the light sensitive surface of the light sensor <NUM>.

<FIG> shows an example of an image of an eye <NUM>, captured by the light sensor <NUM>. The circuitry <NUM> may for example employ image processing (such as digital image processing) for extracting features in the image. The circuitry <NUM> may for example employ pupil center cornea reflection (PCCR) eye tracking to determine where the eye <NUM> is looking. In PCCR eye tracking, the processor <NUM> estimates the position of the center of the pupil <NUM> and the position of the center of a glint <NUM> at the eye <NUM>. The glint <NUM> is caused by reflection of light from one of the illuminators <NUM> and <NUM>. The processor <NUM> calculates where the user is in space using the glint <NUM> and where the user's eye <NUM> is pointing using the pupil <NUM>. Since there is typically an offset between the optical center of the eye <NUM> and the fovea, the processor <NUM> performs calibration of the fovea offset to be able to determine where the user is looking. The gaze directions obtained from the left eye and from the right eye may then be combined to form a combined estimated gaze direction (or viewing direction).

In the eye tracking system described with reference to <FIG>, the illuminators <NUM> and <NUM> are arranged in an eye tracking module <NUM>. This arrangement serves only as an example. It will be appreciated that more or less any number of illuminators and light sensors may be employed for eye tracking, and that such illuminators and light sensors may be distributed in many different ways relative to displays watched by the user. It will be appreciated that the eye tracking scheme described in the present disclosure may for example be employed for wearable eye tracking (such as in virtual reality (VR) glasses or augmented reality (AR) glasses).

<FIG> shows a cross section of different parts of an eye <NUM>. A cornea <NUM> has a central region <NUM> which in general is close to spherical, and an outer region <NUM> which is non-spherical and therefore more difficult to model. The central region <NUM> (which is also referred to herein as the spherical region of the cornea <NUM>) extends from a cornea surface center <NUM> to an edge <NUM> of the spherical region <NUM>. The cornea surface center <NUM> is a position at the spherical region <NUM> of the cornea <NUM> where a virtual line <NUM> extending from an eyeball center <NUM> through a pupil center <NUM> intersects the spherical region <NUM> of the cornea <NUM>. An outer region <NUM> of the cornea <NUM> extends from the edge <NUM> of the spherical region <NUM> of the cornea <NUM> to the edge <NUM> of the cornea <NUM>. The outer region <NUM> is also referred to herein as the non-spherical region of the cornea <NUM>. A cornea sphere center <NUM> is a position at the center of an imaginary cornea sphere <NUM> of which the spherical region <NUM> forms part. <FIG> also shows the sclera <NUM> of the eye <NUM>.

In an eye tracking system, a current eye direction or gaze direction of the eye <NUM> is determined by determining the position of glints and using the geometry of the eye. In this process the current position of the cornea sphere center <NUM> and the current position of the pupil center <NUM> are determined.

<FIG> each show a separate view of selected parts of a head mounted device in the form of a virtual reality (VR) device (VR glasses) <NUM> including an eye tracking system in which embodiments may be implemented.

<FIG> shows a view of selected parts of a head mounted device in the form of the VR glasses <NUM> including an eye tracking system in which embodiments may be implemented. In addition to the VR glasses <NUM>, eyes <NUM> and a head <NUM> of a user are shown. The VR portion of the VR glasses <NUM> shown comprises two VR displays <NUM> and two lenses <NUM>, one VR display <NUM> and one lens <NUM> for each eye <NUM>. The VR displays <NUM> are positioned in front of the eyes <NUM> and the lenses <NUM> are positioned between the eyes <NUM> and the VR displays <NUM>. The eye tracking portion of the VR glasses <NUM> comprises two hot mirrors <NUM> and two cameras <NUM>. In order to capture images of the eyes <NUM> for use in eye tracking, the hot mirrors <NUM> are arranged between the VR displays <NUM> and the lenses <NUM>. Furthermore, illuminators (not shown) are arranged on or in the VR glasses <NUM> such that illumination rays are directed towards the eyes <NUM>. Reflections from the eyes <NUM> of the illumination rays towards the hot mirrors <NUM> will reflect towards the cameras <NUM> in which the illumination rays are detected to produce an image of the eye. For example, the hot mirrors <NUM> may be of a type such that they will reflect light in the infrared frequency band but be transparent for light in the visible frequency band. The illuminators (not show) used would then produce illumination rays in the infrared frequency band and the cameras <NUM> will include image sensors able to detect light in the infrared frequency band.

<FIG> shows a side view of selected parts of the VR glasses <NUM>. Illumination rays from the illuminators (not shown) towards the eye <NUM> will reflect back and pass through the lens <NUM> towards the hot mirror <NUM> and reflect towards the camera <NUM> in which the illumination rays are detected to produce an image of the eye.

<FIG> shows an exploded view of selected parts of the VR glasses <NUM>. Selected parts for one eye are shown including an illuminator cover <NUM>, illuminators in the form of light emitting diodes (LEDs) <NUM>, the camera <NUM> including an image sensor, the lens <NUM>, a lens cup or lens tube <NUM>, the hot mirror <NUM>, the VR display <NUM> and an electronics board <NUM>. <FIG> shows an example arrangement of the illuminators in the form of LEDs <NUM>, where the LEDs are arranged along the periphery of the lens <NUM> to produce a pattern when illuminating the eye <NUM>. The illumination rays from the LEDs <NUM> reflected from the eye and the hot mirror is detected in the camera <NUM> to produce an image of the eye. Reflections on the eye (cornea) are called glints and using the position of glints, the geometry of the eye and the geometry of the setup of the VR glasses <NUM> are used to determine a current eye direction or gaze direction.

Head mounted devices, such as in VR glasses or augmented reality AR glasses, can be enhanced by including wearable eye tracking using illuminators and one or more light sensors arranged in the head mounted device for determining eye direction and/or gaze direction based on estimation of a position of a center of the pupil and a position of the center of one or more glints at the eye from the illuminators. A problem that can arise in such devices is that when the user watches a point close to the edge of a display, glints tend to fall off the cornea so that it becomes difficult to determine the eye direction and/or gaze direction of the user based on identification of glints.

However, even if the glints cannot be identified and used for determination of the eye direction and/or gaze direction, it may still be possible to identify the position of the center of the pupil (pupil center position).

In the following a method for determining eye direction and gaze direction will be described in relation <FIG> and <FIG>.

The method is implemented in a wearable device, such as VR glasses or AR glasses, where illuminators and image sensors are arranged on or in the wearable device, and hence, will not move in relation to the eye when the user moves as long as the user is wearing the device. In such an application, an eyeball center position can be approximated to be constant in relation to an image sensor capturing images of the eye regardless of the eye direction. The wearable device uses a primary method for determining eye direction and/or gaze direction of an eye is based on determining a pupil center in the eye and glints on the eye from the illuminators the eye detected by means of the image sensors. An example of such a system is described in relation to <FIG>.

In situations where a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the one or more illuminators, for it to be possible to determine with required reliability the cornea sphere center position and the eye direction based on the primary method, the primary method will be used for determining eye direction and/or gaze direction.

In situations where it is not possible to determine the eye direction and/or gaze direction with required reliability based on the primary method, a secondary method for determining eye direction and/or gaze direction is used. The secondary method makes use of an estimated eyeball center position and an estimated pupil center position to determine eye direction and/or gaze direction.

The estimated eyeball center position used in the secondary method, is determined in a situation where a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the illuminators, such that it is possible to determine with required reliability the cornea sphere center position and the eye direction using the primary method.

Situations where the secondary method is used are for example when an angle between a direction of the eye in relation to one or more illumination rays from the illuminators becomes large, e.g. when the eye is directed to points at the edge of a display of the system. In such situations difficulties may arise in identifying any glints or at least a sufficient number of glints on the eye, and in particular on the cornea of the eye, or that an identified glint or identified glints cannot be associated to a corresponding illuminator of the illuminators. In such a situation it may not be possible to determine with required reliability an eye direction and/or gaze direction of the eye based on the identified glint or glints using the primary method.

Situations where the primary method is used are where a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the illuminators, such that it is possible to determine with required reliability the cornea sphere center position and the eye direction using the primary method.

The estimated eyeball center position may be valid only for a period of time and may then be updated at times/ in situations when a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the illuminators, such that it is possible to determine with required reliability the cornea sphere center position and the eye direction using the primary method.

<FIG> shows a view of selected portions of the geometry of an eye in relation to parts of the method for determining eye direction and gaze direction where the position of an eyeball center is estimated based on the primary method for determining eye direction and/or gaze direction. A pupil center position p, a cornea sphere center position c, a cornea sphere sc, an estimated eyeball center position e, an estimated distance d from the eyeball center to the cornea sphere center, an eye direction g and a camera <NUM> comprising an image sensor are shown.

In relation to <FIG>, a situation is disclosed the angle between the between the eye direction ĝ and one or more illumination rays from illuminators (not shown) is such that a sufficient number of glints are identified on the eye and the identified glints can be associated to a corresponding illuminator of the illuminators, for it to be possible to determine with required reliability the cornea sphere center position c and the eye direction g using the primary method.

The primary method is used to determine the cornea sphere position c based on glints on the eye as identified in an image captured by the camera <NUM>. Furthermore, the pupil center position is determined based on a representation of the pupil center position identified in the image captured by the camera <NUM>.

The estimated distance d from the eyeball center to the cornea sphere center is obtained as an approximation used generally for a human eye.

The eye direction g is estimated using the determined cornea sphere position c and the pupil center position p. In other words, the eye direction g is the direction from the cornea sphere center position c to the pupil center position p.

The estimated eyeball center position e can then be found based on the results of the primary method as the point on the eye direction axis the estimated distance d from the eyeball center to the cornea sphere center behind the cornea sphere center c.

As the camera <NUM> is arranged on or in the wearable device and the geometry of the setup of the wearable device is known, the estimated eyeball center position e in relation to the camera is known.

<FIG> shows a view of selected portions of the geometry of an eye in relation to parts of the method for determining eye direction and gaze direction where the eye direction and/or gaze direction is determined based on the estimated eyeball center position using the secondary method. An estimated pupil center position p', an estimated cornea sphere center position c', an eyeball sphere se, an estimated eyeball center position e, an estimated radius r from the eyeball center to the pupil center, an estimated distance d from the eyeball center to the cornea center, an estimated distance dp from the cornea sphere center to the pupil center, an estimated eye direction g', an image ray Rp, and a camera <NUM> comprising an image sensor are shown.

In relation to <FIG>, a situation is disclosed the angle between the between the eye direction g' and one or more illumination rays from illuminators (not shown) is such that a sufficient number of glints are not identified on the eye and/or the identified glints cannot be associated to a corresponding illuminator of the illuminators, for it to be possible to determine with required reliability the cornea sphere center position c and the eye direction ĝ using the primary method.

The estimated eyeball center position e in the eye in relation to the camera <NUM> has been derived as described in relation to <FIG>.

The estimated distance d from the eyeball center to the cornea sphere center and the estimated distance dp from the cornea sphere center to the pupil center are obtained as approximations used generally for a human eye.

The estimated distance dp from the cornea sphere center to the pupil center is first selected as an approximation of the distance from the cornea sphere center to the pupil center of a human eye. When further data regarding the eyes of a current user are retrieved the estimated distance dp from the cornea sphere center to the pupil center can be updated.

The eyeball sphere se is defined as a sphere with a center in the estimated eyeball center position e with a radius r which is the sum of the estimated distance d from the eyeball center to the cornea sphere center and the estimated distance dp from the cornea sphere center to the pupil center.

An image of the eye is captured by means of the camera <NUM> and a position of a representation of the pupil center in the eye is identified in the image.

Based on the identified position of the representation of the pupil center in the obtained image, the pupil ray Rp, which is a projection in three-dimensional space, along which the pupil center is positioned. The estimated pupil center position p' is found as an intersection between the eyeball sphere se and the pupil ray Rp.

An estimated eye direction g' is determined based on a vector from the estimated eyeball center position e to the estimated pupil center position p', and an estimated gaze direction can then be determined based on the estimated eye direction g'.

It is to be noted that although the pupil ray Rp is represented as a straight line in <FIG>, the setup of the wearable device the method is implemented in is normally such that a lens and a hot mirror are arranged between the eye and the camera <NUM>. Hence, the projection of the identified representations in an image captured by the camera, such as the projection of the identified representation of the pupil center, in three-dimensional space need to take the known geometry of the setup and the known optical properties to identify the projection. Such a projection would then not be a straight line.

Furthermore, it is also to be noted that the use of the intersection between the pupil ray Rp and the eyeball sphere se is only an approximation. In order to more accurately determine an estimated pupil center position p', an iterative method taking into account refraction in the cornea of the eye from the cornea surface to the pupil center.

In alternative to determining an estimated eye direction and estimated gaze direction, an estimated cornea sphere center position c' can be determined using the determined estimated pupil center position p' and the obtained estimated distance dp from the cornea sphere center to the pupil center. The estimated cornea sphere center position c' is determined by moving the estimated distance dp from the cornea sphere center to the pupil center from the estimated pupil center position p' in the direction towards the estimated eyeball center position e.

The estimated cornea sphere center position c' can be used for predicting glint positions on the eye. The prediction is further based on the knowledge of the geometry of the setup of illuminators and cameras.

<FIG> shows a view of selected portions of the geometry of an eye in relation to parts of a method for determining an enhanced estimated cornea center position c". A camera <NUM> comprising an image sensor, an illuminator <NUM>, a glint ray Rg, a vector rg from the camera which is the source of the glint ray Rg, a cornea surface normal ng, a reflection point Prefl, an angle v between the cornea surface normal ns at the reflection point Prefl, an the vector rg from the camera which is the source of the glint ray Rg, the enhanced estimated cornea sphere center position c", a cornea sphere sc, an estimated cornea radius rc from the cornea sphere center to a cornea sphere surface, an estimated eyeball center position e, an estimated distance d from the eyeball center to the cornea sphere center, and an eyeball to cornea sphere sed are shown.

In relation to <FIG>, a situation is disclosed where one glint is identified in an image of the eye and the identified glint can be associated to a corresponding illuminator of the illuminators.

Usually a cornea center position c is estimated using two or more glints (corneal reflections) on the spherical part of the cornea surface. If there is only one glint that can be used, due to the position of the cornea relative to the camera and illuminators or due to some illuminators being obscured by the user's facial geometry, we can make use of an estimated eyeball center position e, e.g. determined such as described in relation to <FIG>, to determine an enhanced estimated cornea center position c". The enhanced estimated cornea center position c" will be more accurate than the estimated cornea center position c' as described in relation to <FIG>.

In cases where many of the glints used to position the cornea center position c are close to the edge of the cornea, where the spherical assumption is inappropriate, the enhanced estimated cornea center position c" (estimated using the glint closest to the spherical cornea "top") can also be more accurate than the cornea center position c determined by a method such as the primary method described in relation to <FIG>.

The estimated distance d from the eyeball center to the cornea sphere center is an approximation used generally for a human eye.

To calculate the enhanced estimated cornea center position c", the process below may for example be followed:.

<FIG> shows a method according to an embodiment. The method comprises obtaining <NUM> an estimated radius r from an eyeball center to a pupil center of in an eye. The method further comprises determining <NUM> an estimated eyeball center position e of in the eye in relation to a camera an image sensor for capturing images of the eye, obtaining <NUM> an image of the eye captured by means of the camera image sensor, and identifying <NUM> a position of a representation of the pupil center of in the eye in the obtained image. An estimated pupil center position p' is determined <NUM> based on the estimated eyeball center position e, the estimated radius r, and the identified position of the representation of the pupil center in the obtained image.

<FIG> shows a method according to another embodiment similar to the embodiment described in relation to <FIG>. An estimated distance dp from a cornea sphere center of in the eye to the pupil center is obtained <NUM>, and an estimated distance d from the eyeball center to the cornea sphere center is obtained <NUM>. An estimated radius r is equal to the sum of the estimated distance dp from the cornea sphere center to the pupil center and the estimated distance d from the eyeball center to the cornea sphere center.

A cornea sphere center position c is determined <NUM> based on one or more glint positions on the eye, a pupil center position p is determined <NUM>, and an estimated eyeball center position e in relation to an image sensor is determined <NUM> as the estimated distance d from the eyeball center to the cornea sphere center from the cornea sphere center position c in a direction from the pupil center position p to the cornea sphere center position c. An image of the eye captured by means of the image sensor is obtained <NUM>, and a position of a representation of the pupil center in the eye in the obtained image is identified <NUM>. Furthermore, an eyeball sphere se having a center in the estimated eyeball center position e and having a radius of the estimated radius r is determined <NUM>. Based on the identified position of the representation of the pupil center in the obtained image, a pupil ray Rp along which the pupil center is positioned is determined <NUM> and the estimated pupil center position p' is determined <NUM> as an intersection between the eyeball sphere se and the pupil ray Rp. Based on a vector from the estimated eyeball center position e to the estimated pupil center position p', an estimated eye direction ĝ' is determined <NUM> and based on the estimated eye direction ĝ' an estimated gaze direction is determined <NUM>.

The determining <NUM> the estimated pupil center position p' may comprise using an iterative method taking into account refraction in a cornea of the eye from a surface of the cornea to the pupil center.

<FIG> shows a further method according to an embodiment. The method comprises obtaining <NUM> an estimated radius r from an eyeball center to a pupil center of in an eye. The method further comprises determining <NUM> an estimated eyeball center position e of in the eye in relation to a camera an image sensor for capturing images of the eye, obtaining <NUM> an image of the eye captured by means of the camera image sensor, and identifying <NUM> a position of a representation of the pupil center of in the eye in the obtained image. An estimated pupil center position p' is determined <NUM> based on the estimated eyeball center position e, the estimated radius r, and the identified position of the representation of the pupil center in the obtained image. A first estimated distance dp from a cornea sphere center of in the eye to the pupil center is obtained <NUM>, and an estimated cornea sphere center position c' is determined <NUM> as the first estimated distance dp from the cornea sphere center to the pupil center from the estimated pupil center position p' in the direction towards the estimated eyeball center position e. Positions of one or more glints on the eye are then predicted <NUM> based on estimated cornea sphere center position c'.

<FIG> is a block diagram illustrating a specialized computer system <NUM> in which embodiments of the present invention may be implemented. This example illustrates specialized computer system <NUM> such as may be used, in whole, in part, or with various modifications, to provide the functions of components described herein.

Specialized computer system <NUM> is shown comprising hardware elements that may be electrically coupled via a bus <NUM>. The hardware elements may include one or more central processing units <NUM>, one or more input devices <NUM> (e.g., a mouse, a keyboard, eye tracking device, etc.), and one or more output devices <NUM> (e.g., a display device, a printer, etc.). Specialized computer system <NUM> may also include one or more storage device <NUM>. By way of example, storage device(s) <NUM> may be disk drives, optical storage devices, solid-state storage device such as a random access memory ("RAM") and/or a read-only memory ("ROM"), which can be programmable, flash-updateable and/or the like.

Specialized computer system <NUM> may additionally include a computer-readable storage media reader <NUM>, a communications system <NUM> (e.g., a modem, a network card (wireless or wired), an infra-red communication device, Bluetooth™ device, cellular communication device, etc.), and working memory <NUM>, which may include RAM and ROM devices as described above. In some embodiments, specialized computer system <NUM> may also include a processing acceleration unit <NUM>, which can include a digital signal processor, a special-purpose processor and/or the like.

Computer-readable storage media reader <NUM> can further be connected to a computer-readable storage medium, together (and, optionally, in combination with storage device(s) <NUM>) comprehensively representing remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing computer-readable information. Communications system <NUM> may permit data to be exchanged with a network, system, computer and/or other component described above.

Specialized computer system <NUM> may also comprise software elements, shown as being currently located within a working memory <NUM>, including an operating system <NUM> and/or other code <NUM>. It should be appreciated that alternate embodiments of specialized computer system <NUM> may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Furthermore, connection to other computing devices such as network input/output and data acquisition devices may also occur.

Software of specialized computer system <NUM> may include code <NUM> for implementing any or all of the function of the various elements of the architecture as described herein. For example, software, stored on and/or executed by a specialized computer system such as specialized computer system <NUM>, can provide the functions of components of the invention such as those discussed above. Methods implementable by software on some of these components have been discussed above in more detail.

A person skilled in the art realizes that the present invention is by no means limited to the preferred embodiments described above. For example, the person skilled in the art realizes that the eye/gaze tracking methods described herein may be performed by many other eye/gaze tracking systems than the example eye/gaze tracking system <NUM> shown in Figure <NUM>, for example using multiple illuminators and multiple image sensors such as shown e.g. in <FIG>.

Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. The division of tasks between functional units referred to in the present disclosure does not necessarily correspond to the division into physical units; to the contrary, one physical component may have multiple functionalities, and one task may be carried out in a distributed fashion, by several physical components in cooperation. A computer program may be stored/distributed on a suitable non-transitory medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems. The mere fact that certain measures/features are recited in mutually different dependent claims does not indicate that a combination of these measures/features cannot be used to advantage. Method steps need not necessarily be performed in the order in which they appear in the claims or in the embodiments described herein, unless it is explicitly described that a certain order is required.

Claim 1:
A method of determining eye direction and/or gaze direction of an eye, the method implemented on a wearable device on which illuminators and image sensors are arranged which do not move in relation to the eye when a user moves, the method executed by a computing system, the method comprising:
using a primary method for determining eye direction and/gaze direction of an eye based on determining a pupil center and at least one glint on the eye from at least one illuminator, the pupil center and at least one glint being detected by means of an image sensor; and
if it is determined that it is not possible to determine the eye direction and/or gaze direction with required reliability based on the primary method, using a secondary method for determining eye direction and/or gaze direction of the eye based on an estimated eyeball center position (e) and an estimated pupil center position (p'), the estimated pupil center position (p') being determined based on the estimated eyeball center position (e), an estimated radius (r), and an identified position of a representation of a pupil center in an image obtained by the image sensor.