Patent Description:
Determining what a user looks at is of interest in a number of different fields. For example, a disabled person may be equipped with a gaze tracker enabling her to input characters to form words and sentences. As another example, an ambulance driver may be enabled to safely operate equipment of his ambulance or a nuclear power station operator may have his gaze tracked to detect episodes of fatigue during a shift.

Gaze tracking may be performed based on a number of different techniques, which have as a common feature that the position of one eye, or both eyes, is measured to obtain input information to the gaze tracking system to control a device, for example.

Document <CIT> discloses a gaze detection mechanism in near-eye mixed reality solutions, wherein a first plane is fitted to an illuminator, a glint and a pupil centre of a sensor and a second plane is fitted to a second illuminator, a second glint and the pupil centre of the sensor. Document <CIT> discloses a portable eye tracking device, wherein one illuminator selectively illuminates at least a portion of an eye of the user, and an image sensor captures image data representing at least a portion of the eye. A movement sensor may further detect movement of a frame comprised in the device. Finally, the <NPL>, timestamp <NUM>:<NUM>, discloses that eye tracking is a process of measuring either the point of gaze of the motion of an eye relative to the head.

According to some aspects, there is provided the subject-matter of the independent claims. Some embodiments are defined in the dependent claims.

According to a first aspect of the present disclosure, there is provided an apparatus comprising at least one processing core, at least one memory including computer program code, the at least one memory and the computer program code being configured to, with the at least one processing core, cause the apparatus at least to determine a reference point in a three-dimensional space based at least in part on locations of first and second features of a user's eye and on the user's gaze distance, and
perform a mapping of the reference point into a viewed scene of a near-to-eye optical device to obtain an estimated gaze point and/or gaze direction of the user who is using the near-to-eye optical device, the mapping being based at least in part on calibration information associated with the user.

According to a second aspect of the present disclosure, there is provided a method comprising determining a reference point in a three-dimensional space based at least in part on locations of first and second features of a user's eye and on the user's gaze distance, and performing a mapping of the reference point into a viewed scene of a near-to-eye optical device to obtain an estimated gaze point and/or gaze direction of the user who is using the near-to-eye optical device, the mapping being based at least in part on calibration information associated with the user.

According to a third aspect of the present disclosure, there is provided a non-transitory computer readable medium having stored thereon a set of computer readable instructions that, when executed by at least one processor, cause an apparatus to at least determine a reference point in a three-dimensional space based at least in part on locations of first and second features of a user's eye and on the user's gaze distance, and perform a mapping of the reference point into a viewed scene of a near-to-eye optical device to obtain an estimated gaze point and/or gaze direction of the user using the near-to-eye optical device, the mapping being based at least in part on calibration information associated with the user.

Disclosed herein are gaze tracking methods for ocular near-eye optical devices, , such as microscopes and binoculars. To obtain dependable gaze tracking in an ocular device which is invariant, at least partly, to movement of the user's head, a reference point is determined in a three-dimensional space based on the direction and distance of the gaze of the user, and the reference point is mapped to a point in viewed scene based on a calibration matrix obtained from the user in a calibration process. Therefore, advantageously, characteristics of a coordinate transformation between a coordinate system of an eye-tracking sensor, such as a camera, and a coordinate system of the user's viewed scene, which may be obtained with a scene camera, for example, need not be determined. The scene viewed may be obtained using various technologies, such as, but not limited to, retinal projection, image projection, image fusion and injection, being displayed or captured with a scene camera as in presented example. Indeed, determining such characteristics could be difficult in a near-to-eye optical devices using plural optical components, such as lenses, filters, beam splitters, light guides and/or mirrors. An example of an ocular device, is a microscope. A desired feature in a gaze tracking method is its invariance against the movement of the eye with respect to the measuring sensor.

<FIG> illustrates an example system in accordance with at least some embodiments of the present invention. Illustrated is a user's eye <NUM>, which comprises the pupil <NUM>. In the illustrated example, the user uses a microscope <NUM>, which is an example of an ocular device or in general a near-to-eye optical device. Microscope <NUM> comprises a plurality of optical components such as lenses, filters, beam splitters, light guides and/or mirrors, which are schematically illustrated using reference number <NUM>. A sample is visible through the microscope on plate <NUM>. Plate <NUM> may be of glass, for example, enabling illumination of the sample from below. Two possible gaze targets <NUM> are schematically illustrated on plate <NUM>.

Tracking the gaze of a user of a microscope gives valuable information concerning how the user observes the sample under study, for example what parts of the sample the user focuses on, for how long and what kind of gaze strategy the user uses. In addition, the pupil size may be estimated. This information may be used to estimate the user's awareness, workload and level of expertise, in educational purposes, and in assisting annotation of image areas. Gaze information may be used in controlling the functioning of an optical device, such as a microscope, for example. Thus manual controlling may be reduced.

A pathologist or laboratory worker may also use his gaze point on a sample plate to guide the microscope to move the sample. As a further example, a digital viewfinder may provide a visual indication when the user is looking at a target which may be a human. Further, a sample may be automatically associated based on the gaze point identification with metadata indicating an extent to which it has been analysed, for example, whether a threshold has been reached. An example is a sample which must be analysed at least as to <NUM>% of its contents.

In the system of <FIG>, an eye-tracking camera, which is not shown for the sake of clarity, is configured to image eye <NUM>. The eye-tracking camera may image eye <NUM> from below, wherein the eye-tracking camera may be comprised in the ocular device being used, for example such that it may image the eye via a semipermeable mirror in the ocular device which is transparent to visible light but not the infra-red. The eye may be illuminated for this purpose by guide lights or light shaping or light generating devices, for example visible-light or infra-red, IR, lights, which may be generated using lightemitting diodes, LEDs, for example. An advantage of IR lights is that the human eye does not detect it, making the light unobtrusive to the user. IR light also allows for filtering out lighting and reflections of visible light sources the user is looking at or are visible in the environment, helping to control the lighting scheme. The corneal shape, illustrated in <FIG>, enables extracting information on the direction the eye is turned based on glints of the lights on the moist surface of the eye. The eye-tracking sensor, such as, for example, a camera sensor, which is enabled to capture information of the user's eye may be based at least partly on charge-coupled device, CCD, technology, complementary metal-oxide semiconductor, CMOS, technology, and/or photosensitive photodiodes, for example, to produce a digital video or still data of the eye. Such digital video or still data of the eye may include including reflections, glints, of the guide lights.

Likewise in the system of <FIG>, a scene camera - not seen in the figure for the sake of clarity - is arranged to image plate <NUM>. The scene camera may likewise be based on CCD technology, for example, to produce digital video or still data of the sample. The scene camera and the eye-tracking camera may be synchronized with each other to produce time-aligned images of eye <NUM> and plate <NUM>.

In gaze tracking in general, a transformation may be sought from a 3D coordinate system of an eye-tracking camera to a 3D coordinate system of a scene camera. The eye-tracking camera may be assisted by light-guides to generate glints, as described above, to enable determining a direction where the eye is turned toward. In an ocular device, however, the presence of optical equipment <NUM> makes it more difficult to project a 3D gaze point into a scene camera's 2D coordinates.

Some gaze tracking solutions are sensitive to movement of the user's head relative to the eye-tracking camera. In these cases, if the user moves after her calibration the gazetracking results will be inaccurate. Similarly, if the optical device in question moves during operation, it will result in a similar misalignment error. A typical usage session of an ocular device contains constant small head movements for adjusting the optimal viewing angle, or head movement due to the user leaving the operational position to view objects outside the ocular view and returning to it. The inaccuracy resulting from such head movement hinders gaze tracking. Methods disclosed herein enable robust gaze tracking despite the afore-mentioned unwanted head movements and entering and leaving an operational position.

Much prior work in gaze tracking has used a two-dimensional, 2D, mapping from eye-tracking coordinates to scene camera coordinates which is less robust than a three-dimensional, 3D, algorithm. A 3D approach, however, is complex to implement as such because the projection between the eye-tracking camera coordinate system, where a 3D gaze point may be computed in, and the 2D scene camera coordinate system is non-trivial to determine due to effects of the optical equipment <NUM>.

To overcome the difficulty in defining the transformation from the coordinate system of the eye-tracking camera to the coordinate system of the scene camera, a two-phase gaze tracking method may be employed. In the first phase, data from the eye-tracking camera may be processed to determine a centre of the user's pupil and the centre of the user's cornea. These will be referred to herein as the pupil centre, Pc, and corneal centre, Cc. These may be determined as points in a three-dimensional space, enabling an optical vector L to be determined as a normalized vector traversing these points, establishing a direction of the gaze. An optical point, Op, also referred to herein as a reference point, may be determined by moving from the corneal centre towards the optical vector by the gaze distance d: Op = Cc + d*L. In some embodiments, different 3D features of the eye than the cornea and the pupil are used to compute a 3D optical point relating to the eye. In more general terms a first eye feature and a second eye feature may be used to obtain the optical point. An example of an eye feature other than the cornea and the pupil is the iris, wherefore the first and second eye feature may comprise the iris and the pupil, for example.

The gaze distance may be determined as follows: firstly, a virtual plane may be fitted to light-guide, such as IR LED locations. Secondly, the plane may then be shifted away from the user by a constant value, in the virtual plane's normal direction. The constant value is guessed or it is estimated during the user calibration. Thirdly, an intersection between the shifted plane and the (parameterized) vector Cc + d*L is computed, that is, d is found. Thus the gaze distance may be estimated per measurement and may be at least slightly different each time. Other ways of determining the gaze distance are also possible. The gaze distance is the distance from the eye to where the user's gaze is focused.

<FIG> illustrates an example of gaze distance estimation. A LED plane is fitted into LEDs by minimizing distances between the LEDs and the plane. To estimate the gaze distance, the location of this plane is mathematically varied such that the plane is moved away from the eye by a distance D_fixed in a direction which is orthogonal to the LED plane when it is fitted to the LED elements. This results in the plane at the viewing plane location illustrated in <FIG>. The distance D_fixed may be guessed, or it may be estimated during user calibration, or it may be known based on prior knowledge. The optical point, Op, is defined as the intersection between the vector (Cc + dL) and the viewing plane. Illustrated are three possible optical vectors and the corresponding optical points Op1, Op2 and Op3.

Once the three coordinates of the reference point are determined, the 3D reference point may be mapped in a second phase of the overall two-phase process into the two-dimensional scene camera coordinate system to obtain the gaze point, Gp, using the user's calibration matrix K:
<MAT>.

Calibration matrix K may be determined by conducting a calibration procedure with the user. The user fixates several target points with her gaze. The number of target points may be at least three or more than three. The target points should for ideal operation not be in a linear arrangement, for example they may form a grid of target points which avoids the linear arrangement of the target points being disposed on a straight line. For each target point, a calibration sample is collected from the eye-tracking camera which constitutes an annotated target point, Gp_truth, and the optical point. These calibration samples taken together form a matrix equation from which the calibration matrix K can be solved. A calibration reference point in the three-dimensional space where the optical point will be determined corresponds to each of the target points. The larger the number of target points and samples, the better will be the expected accuracy of gaze tracking based on the calibration matrix. While a calibration matrix is discussed herein, more generally calibration information may be used.

Overall, the procedure may be characterized as obtaining first and second eye features from a two-dimensional pixel scene of the eye-tracking camera. Examples of the first and second eye features include the corneal centre and pupil centre. From locations of the first and second eye features in 3D space the reference point in a three-dimensional coordinate system is determined, and from the reference point the gaze point in a two-dimensional scene camera view is obtained. The coordinates of the gaze point may be determined, in the coordinate system of the scene camera, even in the event the gaze point is outside the view of the scene camera.

The obtaining of the corneal centre and pupil centre from the output of the eye-tracking camera may be performed, for example, using the physical eye model described in<NPL>. Alternatively, a neural network-based method may be used where the neural network is trained to output the optical vector when given an eye-tracking camera image as input.

At least some embodiments of the herein described gaze tracking process are beneficial and advantageous in that the reference point is determined based on the 3D characteristics of the eye, wherefore its location is to a degree invariant to small movements of the head. Further, alternatively or in addition, the gaze point may be determined also where it is disposed outside the view of the scene camera.

<FIG> illustrates coordinate mapping in accordance with at least some embodiments of the present invention. Eye <NUM> is disposed on the left, with pupil <NUM> and the corneal centre <NUM> marked thereon. The optical vector multiplied by the gaze distance, d*L, is denoted as element <NUM>, pointing to the reference point <NUM>.

Reference point <NUM> is located in three-dimensional coordinate system, <NUM>. A mapping <NUM> from coordinate system <NUM> to two-dimensional coordinate system <NUM> of the scene camera is denoted as mapping <NUM> in <FIG>. The mapping associates reference point <NUM> to gaze point <NUM> in the coordinate system <NUM> of the scene camera.

For example, in terms of a practical implementation, the ocular part(s) of a microscope may be supplemented with a module comprising the eye-tracking camera, suitable circuitry, and at least one light source for structured light, such as infrared light emitting elements. If the microscope contains an ocular part for both eyes, both ocular parts may be provided with a similar module. In that case, the estimated gaze point may be determined as a weighted combination of separately determined left and right eye gaze points, weights being assigned score values of the estimation. The microscope may also contain a scene camera that views the sample and sees at least part of the view the user sees. A light path to the scene camera can be directed through a beam splitter which directs the same view to the user and to the scene camera, with different optical elements.

The cameras may be connected into a computer using suitable interfaces, such as universal serial bus, USB, connectors, or integrated electrical leads. The computer may be furnished with a computer program which reads the camera streams and estimates the gaze point in the scene camera coordinates, as is depicted in <FIG>. The computer program may be equipped with a graphical user interface which, for example, may be configured to show the scene camera view with a superimposed estimated gaze point. The computer program may also be configured to run the configuration process with the user to determine the calibration matrix K.

While described herein in terms of utilizing pupil and glint locations and a physical eye model, the optical point may alternatively be determined by other means than with a physical eye model. For example, machine learning approaches based on deep convolutional networks can be taught to automatically translate pupil and glint locations into gaze points and/or gaze directions. Where the ocular device used is a gun sight, for example, the two-dimensional output of the scene camera corresponds to plural gaze directions where the user is gazing.

<FIG> illustrates an example apparatus capable of supporting at least some embodiments of the present invention. Illustrated is device <NUM>, which may comprise, for example, a gaze tracking module for an ocular device. Comprised in device <NUM> is processor <NUM>, which may comprise, for example, a single- or multi-core processor wherein a single-core processor comprises one processing core and a multi-core processor comprises more than one processing core. Processor <NUM> may comprise, in general, a control device. Processor <NUM> may comprise more than one processor. Processor <NUM> may be a control device. A processing core may comprise, for example, a Cortex-A8 processing core manufactured by ARM Holdings or a Steamroller processing core designed by Advanced Micro Devices Corporation. Processor <NUM> may comprise at least one Qualcomm Snapdragon and/or Intel Atom processor. Processor <NUM> may comprise at least one application-specific integrated circuit, ASIC. Processor <NUM> may comprise at least one field-programmable gate array, FPGA. Processor <NUM> may be means for performing method steps in device <NUM>, such as determining a reference point and performing a mapping. Processor <NUM> may be configured, at least in part by computer instructions, to perform actions.

Memory <NUM> may comprise randomaccess memory and/or permanent memory.

Device <NUM> may comprise a transmitter <NUM>. Device <NUM> may comprise a receiver <NUM>. Transmitter <NUM> and receiver <NUM> may be configured to transmit and receive, respectively, information in accordance with at least one communication standard. Transmitter <NUM> may comprise more than one transmitter. Receiver <NUM> may comprise more than one receiver. Transmitter <NUM> and/or receiver <NUM> may be configured to operate in accordance with global system for mobile communication, GSM, wideband code division multiple access, WCDMA, <NUM>, long term evolution, LTE, IS-<NUM>, wireless local area network, WLAN, USB, Ethernet and/or worldwide interoperability for microwave access, WiMAX, standards, for example.

Device <NUM> may comprise user interface, UI, <NUM>. UI <NUM> may comprise at least one of a display, a keyboard, a touchscreen, a vibrator arranged to signal to a user by causing device <NUM> to vibrate, a speaker and a microphone. A user may be able to operate device <NUM> via UI <NUM>, for example to perform a calibration process and/or gaze tracking operations.

Device <NUM> may comprise further devices not illustrated in <FIG>. Device <NUM> may comprise a fingerprint sensor arranged to authenticate, at least in part, a user of device <NUM>. In some embodiments, device <NUM> lacks at least one device described above. For example, some devices <NUM> may lack the NFC transceiver <NUM> and/or at least one other unit described herein.

Processor <NUM>, memory <NUM>, transmitter <NUM>, receiver <NUM>, NFC transceiver <NUM>, and/or UI <NUM> may be interconnected by electrical leads internal to device <NUM> in a multitude of different ways. For example, each of the aforementioned devices may be separately connected to a master bus internal to device <NUM>, to allow for the devices to exchange information. However, as the skilled person will appreciate, this is only one example and depending on the embodiment various ways of interconnecting at least two of the aforementioned devices may be selected without departing from the scope of the present invention.

<FIG> is a flow graph of a method in accordance with at least some embodiments of the present invention. The phases of the illustrated method may be performed in device <NUM>, an auxiliary device or a personal computer, for example, or in a control device configured to control the functioning thereof, when installed therein.

In phase <NUM>, at least one eye image is obtained from the eye-tracking camera. In phase <NUM>, the pupil and glints of guide lights are located in the at least one eye image. In phase <NUM>, a 3D pupil centre, Pc, and 3D corneal centre, Cc, of the user are determined based on the pupil and glints, for example with the assistance of a model of the physical shape of the eye. In phase <NUM>, the optical vector L is determined, as described herein above. In phase <NUM>, the gaze distance is determined.

In phase <NUM>, the three-dimensional reference point, also known as the optical point, Op, is determined as Op = Cc + d*L. Finally, in phase <NUM>, the gaze point, Gp, is obtained by mapping the optical point into the two-dimensional scene camera view using the calibration matrix K: Gp = K*Op. The gaze point is the point on the plate <NUM> the user is looking at.

It is to be understood that the embodiments of the invention disclosed are not limited to the particular structures, process steps, or materials disclosed herein, but
are extended to equivalents thereof as would be recognized by those ordinarily skilled in the relevant arts.

At least some embodiments of the present invention find industrial application in tracking user gaze in ocular devices.

Claim 1:
An apparatus (<NUM>) comprising at least one processing core (<NUM>), at least one memory (<NUM>) including computer program code, the at least one memory (<NUM>) and the computer program code being configured to, with the at least one processing core (<NUM>), cause the apparatus (<NUM>) at least to:
- determine (<NUM>) a reference point in a three-dimensional space based at least in part on locations of first and second features of a user's eye and on the user's gaze distance, and
- perform a mapping (<NUM>) of the reference point into a viewed scene of a near-to-eye optical device to obtain an estimated gaze point and/or gaze direction of the user who is using the near-to-eye optical device, the mapping being based at least in part on calibration information associated with the user, the near-to-eye optical device comprising light guides,
characterized in that
- the near-to-eye optical device comprises an ocular device, and in that the apparatus is configured to obtain the gaze distance (<NUM>) by fitting a virtual plane to locations of the light guides by minimizing distances between the light guides and the virtual plane, shifting the virtual plane in its normal direction by a distance which is randomly generated or based on user calibration, and determining an intersection of the shifted virtual plane and a vector which traverses the first and second features of the user's eye.