Patent Description:
Different techniques have been developed for monitoring in which direction (or at which point on a display) a user is looking. This is often referred to as gaze tracking.

One example is given in <CIT> that describes eye tracking systems and methods comprising at least one image capture device detecting a user's gaze location relative to the display device. A processing device may electronically analyze the location of user elements within the user interface relative to the user's gaze location and dynamically determine whether to initiate the display of a zoom window.

Another example is given in <CIT> that describes calibration techniques for systems which include eye tracking devices and display devices. The calibration techniques may involve a calibration process which utilizes a plurality of visible calibration targets that each defines a gaze point for a user. Increased information regarding system setup that is useful in calibrating the eye tracking system may be obtained from the calibration process.

Yet another example is given in <CIT> that discloses a head mounted display (HMD) comprising an eye tracking system configured to enable eye-tracking using polarization. The eye tracking system determines eye tracking information based on an updated model in order to improve eye tracking performance, wherein the determined 3D shape of the eye is used to update a stored model of the eye in response to the one or more model parameter values extracted from the determined depth map of the corneal surface.

Such techniques often involve detection of certain features in images of the eye, and a gaze direction or gaze point is then computed based on positions of these detected features. An example of such a gaze tracking technique is pupil center corneal reflection (PCCR). PCCR-based gaze tracking employs the position of the pupil center and the position of glints (reflections of illuminators at the cornea) to compute a gaze direction of the eye or a gaze point at a display.

Another term which is often employed in this context is eye tracking. While the term eye tracking may in many cases is employed as an alternative name for gaze tracking, eye tracking need not necessarily involve tracking of the user's gaze (for example in the form of a gaze direction or a gaze point). Eye tracking may for example relate to tracking of the position of an eye in space, without actually tracking a gaze direction or gaze point of the eye.

As an alternative (or complement) to conventional techniques such as PCCR-based eye tracking, machine learning may be employed to train an algorithm to perform eye tracking. For example, the machine learning may employ training data in the form of images of the eye and associated known gaze points to train the algorithm, so that the trained algorithm can perform eye tracking in real time based on images of the eye. Plenty of training data is typically needed for such machine learning to work properly. The training data may take quite some time and/or resources to collect. In many cases, certain requirements may be put on the training data. The training data should for example preferably reflect all those types of cases/scenarios that the eye tracking algorithm is supposed to be able to handle. If only certain types of cases/scenarios are represented in the training data (for example only small gaze angles, or only well-illuminated images), then the eye tracking algorithm may perform well for such cases/scenarios, but may not perform that well for other cases/scenarios not dealt with during the training phase.

It would be desirable to provide new ways to address one or more of the abovementioned issues.

Methods, systems and computer-readable storage media having the features defined in the independent claims are provided for addressing one or more of the abovementioned issues. Preferable embodiments are defined in the dependent claims.

In what follows, example embodiments will be described in greater detail with reference to the accompanying drawings, on which:.

All the figures are schematic, not necessarily to scale, and generally only show parts which are necessary in order to elucidate the respective embodiments, whereas other parts may be omitted or merely suggested. Any reference number appearing in multiple drawings refers to the same object or feature throughout the drawings, unless otherwise indicated.

Throughout the present disclosure, the term eye tracking sensor relates to a sensor which is adapted to obtain sensor data for use in eye tracking. While an eye tracking sensor may for example be an imaging device (such as a camera), several other types of sensors could be employed for eye tracking. For example, an eye tracking sensor may employ light, sound, a magnetic field, or an electric field to obtain sensor data which may be employed (for example in combination with sensor data from other sensors) for determining where the eye is located and/or in which direction the eye is gazing. An eye tracking sensor may for example be arranged to (or configured to) monitor an eye. An eye tracking sensor may for example be arranged to (or configured to) perform measurements (or to obtain sensor data) when instructed to do so. In other words, an eye tracking sensor need not necessarily perform a constant/continuous monitoring of the eye.

Throughout the present disclosure, the term imaging device relates to a device, which is adapted to capture images. An imaging device may for example be an image sensor or a camera, such as a charge-coupled device (CCD) camera or a Complementary Metal Oxide Semiconductor (CMOS) camera. However, other types of imaging devices may also be envisaged.

Embodiments of methods, systems, and associated storage media will be described below with reference to <FIG>. First, certain features of an eye will be described with reference to <FIG>.

<FIG> is a front view of an eye <NUM>. <FIG> is a cross sectional view of the eye <NUM> from the side of the eye <NUM>. While <FIG> shows more or less the entire eye <NUM>, the front view presented in <FIG> only shows those parts of the eye <NUM> which are typically visible from in front of a person's face. The eye <NUM> has a pupil <NUM>, which has a pupil center <NUM>. The eye <NUM> also has an iris <NUM> and a cornea <NUM>. The cornea <NUM> is located in front of the pupil <NUM> and the iris <NUM>. The cornea <NUM> is curved and has a center of curvature <NUM>, which is referred to as the center <NUM> of corneal curvature, or simply the cornea center <NUM>. The cornea <NUM> has a radius of curvature <NUM> referred to as the radius <NUM> of the cornea <NUM>, or simply the cornea radius <NUM>. The eye <NUM> also has a sclera <NUM>. The eye <NUM> has a center <NUM> which may also be referred to as the center <NUM> of the eye ball, or simply the eye ball center <NUM>. The visual axis <NUM> of the eye <NUM> passes through the center <NUM> of the eye <NUM> to the fovea <NUM> of the eye <NUM>. The optical axis <NUM> of the eye <NUM> passes through the pupil center <NUM> and the center <NUM> of the eye <NUM>. The visual axis <NUM> forms an angle <NUM> relative to the optical axis <NUM>. The deviation or offset between the visual axis <NUM> and the optical axis <NUM> is often referred to as the fovea offset <NUM>. In the example shown in <FIG>, the eye <NUM> is looking towards a display <NUM>, and the eye <NUM> is gazing at a gaze point <NUM> at the display <NUM>. <FIG> also shows a reflection <NUM> of an illuminator at the cornea <NUM>. Such a reflection <NUM> is also known as a glint <NUM>.

Training of machine learning (ML) based eye tracking algorithms typically requires a very large number of images of the eye that are annotated with ground truth information, such as gaze origin (3D eye position), gaze direction, gaze point on screen, etc. In traditional data collections, test subjects are asked to look at points with a known location on a display to gather ground truth gaze data. There are several problems with this approach:.

If one instead uses a conventional, calibrated eye-tracker to supply this ground truth data, then the data collection would cost almost nothing and result in large quantities of natural (real life-like situation) training data. One could let a user work as normal in front of his/her computer while a ML-based eye tracker collects time-stamped sensor data (such as images) and a reference eye tracker collects ground truth information (such as gaze points, 3D positions of the eye, gaze directions etc.). Such a system can run in the background on a test subject's computer. At the end of a working day a large amount of annotated data will have been collected that can be used to train ML-based algorithms.

Hence, a method for training an eye tracking model is proposed. <FIG> is a flow chart of an embodiment of this method <NUM>. In the present embodiment, the eye tracking model is adapted to predict eye tracking data based on sensor data from a first eye tracking sensor. In other words, the eye tracking model is configured to predict or estimate eye tracking data using sensor data from the first eye tracking sensor, or using information derived from such sensor data. The eye tracking model may for example be regarded as a function (or a mapping) which receives sensor data from the first eye tracking sensor as input (and which optionally also receives further input data) and which provides predicted eye tracking data as output.

The eye tracking model, which is trained in the method <NUM> may for example be a machine learning (ML) based eye tracking model. The eye tracking model may for example be based on an artificial neural network, such as a convolutional neural network. However, the eye tracking model could also be a more traditional model, which may for example be trained by traditional optimization of values of a set of parameters.

The method <NUM> comprises receiving <NUM> sensor data obtained by the first eye tracking sensor at a time instance (or at a point in time). In other words, the sensor data is received <NUM> after having been obtained (or generated) by the first eye tracking sensor at a certain time instance or point in time. The first eye tracking sensor may for example be an imaging device. However, as described above, several other types of eye tracking sensors may also be envisaged.

The method <NUM> comprises receiving <NUM> reference eye tracking data for the time instance generated by an eye tracking system comprising a second eye tracking sensor. The reference eye tracking data is generated by the eye tracking system based on sensor data obtained by the second eye tracking sensor at the time instance (in other words, at the point in time when the received <NUM> sensor data was obtained by the first eye tracking sensor). The second eye tracking sensor may for example be an imaging device. However, as described above, several other types of eye tracking sensors may also be envisaged. It will be appreciated that the second eye tracking sensor is distinct from the first eye tracking sensor. In other words, the first and second eye tracking sensors do not coincide, but they could for example of a similar type.

The method <NUM> comprises training <NUM> the eye tracking model based on the sensor data obtained by the first eye tracking sensor at the time instance and the generated reference eye tracking data. The training may for example comprise adapting values for one or more parameters of the eye tracking model.

It will be appreciated that the sensor data received at step <NUM> and the sensor data on which the reference eye tracking data received at step <NUM> is based need not necessarily be obtained by the first and second eye tracking sensors at exactly the same time instance. In other words, these two sets of sensor data may be obtained by the respective eye tracking sensors at approximately the same time instance, but there may of course be a slight deviation or timing mismatch between these two sets of sensor data. It will be appreciated that as long as such a deviation is so small that the eye has not moved (or has not been redirected) too much during this very short time period, the step of training <NUM> the eye tracking model will not be significantly affected by the mismatch.

<FIG> is a schematic overview of a system <NUM> for training an eye tracking model, according to an embodiment. The system <NUM> may for example perform the method <NUM> described above with reference to <FIG>.

Consider the following scenario. You have a well-functioning eye tracking system <NUM> which comprises an eye tracking sensor <NUM> and means for analyzing the sensor data to generate eye tracking data, such as an estimated position of the eye <NUM> in space, or an estimated gaze point of the eye <NUM>. You have a new eye tracking system <NUM>, which comprises an eye tracking sensor <NUM>, but the new eye tracking system is not yet able to generate accurate gaze tracking data based on sensor data from the eye tracking sensor <NUM>. The software or algorithm employed in the old eye tracking system <NUM> is not that useful for the new eye tracking system <NUM> for a reason such as:.

Therefore, instead of reusing software from the old eye tracking system <NUM> in the new eye tracking system <NUM>, the old eye tracking system <NUM> is employed to provide ground truth data for training of the new eye tracking system <NUM>. The new eye tracking system <NUM> is equipped with an eye tracking model adapted to predict eye tracking data based on sensor data obtained by the eye tracking sensor <NUM>. The method <NUM> described above with reference to <FIG> may then be employed for training the eye tracking model of the new eye tracking system <NUM>, using the old eye tracking system <NUM> to generate reference eye tracking data. In the terminology of the method <NUM>, the eye tracking sensor <NUM> is an example of the first eye tracking sensor, which obtained the sensor data received at step <NUM>, and the old eye tracking system <NUM> is an example of the eye tracking system, which generated the reference eye tracking data received at step <NUM>. Further, the eye tracking sensor <NUM> is an example of the second eye tracking sensor referred to in the method <NUM>.

The system <NUM> comprises processing circuitry <NUM> configured to perform the method <NUM> to train the eye tracking model of the new eye tracking system <NUM>.

The processing circuitry <NUM> may for example comprise one or more processors <NUM>. The processor(s) <NUM> may for example be application-specific integrated circuits (ASIC) configured to perform a specific method (such as the method <NUM>). Alternatively, the processor(s) <NUM> may be configured to execute instructions (for example in the form of a computer program) stored in one or more memories <NUM>. Such one or more memories <NUM> may for example be comprised in the processing circuitry <NUM> of the system <NUM>, or may be external to (for example located remotely from) the system <NUM>. The one or more memories <NUM> may for example store instructions for causing the system <NUM> to perform the method <NUM>.

The processing circuitry <NUM> may be communicatively connected to the old eye tracking system <NUM> and the new eye tracking system <NUM> (or at least to the eye tracking sensor <NUM> in the new eye tracking system <NUM>), for example via wired and/or wireless connections.

The old eye tracking system <NUM> may for example be a PCCR-based eye tracking system. In other words, the reference eye tracking data received at step <NUM> in the method <NUM> may have been generated by the eye tracking system <NUM> based on an image position of a corneal reflection of an illuminator <NUM> at a known position in relation to the eye tracking sensor <NUM> (which in this case is an imaging device) and an image position of a pupil center.

The old eye tracking system <NUM> may for example comprise more eye tracking sensors, or more advanced eye tracking sensors, or more illuminators than the new eye tracking system <NUM>. By training the new eye tracking system <NUM> using a more advanced eye tracking system <NUM>, an eye tracking system <NUM> with relatively cheaper components could be obtained, which is able to perform almost as well as the more advanced eye tracking system <NUM>.

In the example implementation shown in <FIG>, the eye tracking sensor <NUM> in the old eye tracking system <NUM> is an imaging device (such as a camera), and one or more illuminators <NUM>-<NUM> are provided for illuminating the eye <NUM>. In the present example implementation, the eye tracking sensor <NUM> in the new eye tracking system <NUM> is also an imaging device. Light <NUM> from the illuminator <NUM> in the old eye tracking system <NUM> reaches the imaging device <NUM> in the old eye tracking system <NUM> via a reflection at the cornea of the eye <NUM>. However, light <NUM> from the illuminator <NUM> in the old eye tracking system <NUM> may also reach the imaging device <NUM> in the new eye tracking system <NUM>, which may cause interference in images captured by the imaging device <NUM> in the new eye tracking system <NUM>. A filter <NUM> may therefore be employed to prevent light <NUM> from the illuminator <NUM> from reaching the imaging device <NUM>.

Hence, according to some embodiments, the old eye tracking system <NUM> comprises an illuminator <NUM> which outputs light <NUM> within a wavelength range for illuminating an eye <NUM>, and the eye tracking sensor <NUM> of the old eye tracking system <NUM> provides sensor data based on light within the wavelength range. The eye tracking sensor <NUM> of the new eye tracking system <NUM> may be provided with a filter <NUM> for suppressing light within the wavelength range.

Light emitted by the illuminator <NUM> may for example be light of a relatively long wave length, and the filter <NUM> may be a short pass filter. Alternatively, light emitted by the illuminator <NUM> may for example be light of a relatively short wave length, and the filter <NUM> may be a long pass filter. If light emitted by the illuminator <NUM> is not restricted to a certain wavelength range, a filter <NUM> may for example be provided in front of the illuminator <NUM> for suppressing light outside a certain wavelength range.

Similarly, if the new eye tracking system <NUM> comprises one or more illuminators <NUM>-<NUM> for illuminating the eye <NUM>, the eye tracking sensor <NUM> in the old eye tracking system <NUM> may be provided with a filter <NUM> for suppressing light from the illuminator in the new eye tracking system <NUM>. Light emitted by the illuminator <NUM> may for example be light of a relatively long wave length, and the filter <NUM> may be a short pass filter. Alternatively, light emitted by the illuminators <NUM> may for example be light of a relatively short wave length, and the filter <NUM> may be a long pass filter. If light emitted by the illuminator <NUM> is not restricted to a certain wavelength range, a filter <NUM> may be provided in front of the illuminator <NUM> for suppressing light outside a certain wavelength range.

In this way, the two eye tracking systems <NUM> and <NUM> are prevented from interfering with each other. In other words, the old eye tracking system <NUM> may employ light in a first wavelength range (for example around <NUM>), while the new eye tracking system employs light in a second wavelength range (for example about <NUM>) which does not overlap the first wavelength range.

As shown in <FIG>, the old eye tracking system <NUM> and/or the new eye tracking system <NUM> may comprise one or more illuminators. The illuminators may for example be infrared or near infrared illuminators, for example in the form of light emitting diodes (LEDs). However, other types of illuminators may also be envisaged. As shown in <FIG>, the old eye tracking system <NUM> may for example comprise a member <NUM> (for example a circuit board, such as a printed circuit board, PCB) at which the eye tracking sensor <NUM> and the illuminators <NUM>-<NUM> are mounted. Similarly, the new eye tracking system <NUM> may for example comprise a member <NUM> (for example a circuit board, such as a PCB) at which the eye tracking sensor <NUM> and the illuminators <NUM>-<NUM> are mounted.

It will be appreciated that the system <NUM> need not necessarily comprise all those components shown in <FIG>. For example, the system <NUM> could comprise only the processing circuitry <NUM>, and the rest of the components shown in <FIG> could be regarded as external to the system <NUM>. In some embodiments, the system <NUM> comprises the eye tracking system <NUM> employed to generate the reference eye tracking data received at step <NUM> in the method <NUM>. In some embodiments, the system <NUM> comprises the eye tracking sensor <NUM> employed to obtain the sensor data received at step <NUM> in the method <NUM>.

The eye tracking system <NUM> and the eye tracking system <NUM> may for example be provided in the form of two separate units or devices, which may for example be mountable at a display device for performing eye tracking.

<FIG> is a schematic overview of an example eye tracking system <NUM>. The old eye tracking system <NUM> and/or the new eye tracking system <NUM> in <FIG> may for example be of the type described below with reference to <FIG>.

The system <NUM> comprises one or more illuminators <NUM> for illuminating the eye <NUM> and one or more imaging devices <NUM> for capturing images of the eye <NUM> while the eye <NUM> looks at a display <NUM>. The system <NUM> also comprises processing circuitry <NUM> configured to estimate where the eye <NUM> is located and/or where the eye <NUM> is looking. The processing circuitry <NUM> may for example estimate eye tracking data such as a gaze direction (or gaze vector) of the eye <NUM> (corresponding to a direction of the visual axis <NUM>), or a gaze point <NUM> of the eye <NUM> at the display <NUM>. In other words, the eye tracking system <NUM> may for example be a gaze tracking system.

The processing circuitry <NUM> is communicatively connected to the illuminators <NUM> and the imaging devices <NUM>, for example via a wired or wireless connection. The processing circuitry <NUM> may also be communicatively connected to the display <NUM>, for example for controlling (or triggering) the display <NUM> to show test stimulus points for calibration of the eye tracking system <NUM>.

<FIG> shows example illuminators <NUM> located at either side of the display <NUM>, but the illuminators <NUM> could be located elsewhere. <FIG> shows example imaging devices <NUM> located above the display <NUM>, but the imaging devices <NUM> could be located elsewhere, for example below the display <NUM>.

The display <NUM> may for example be a liquid-crystal display (LCD) or a LED display. However, other types of displays may also be envisaged. The display may <NUM> may for example be flat or curved. The display <NUM> may for example be a TV screen, a computer screen, or may be part of a head-mounted device (HMD) such as a virtual reality (VR) or augmented reality (AR) device. The display <NUM> may for example be placed in front of one of the user's eyes. In other words, separate displays <NUM> may be employed for the left and right eyes. Separate eye tracking equipment (such as illuminators <NUM> and imaging devices <NUM>) may for example be employed for the left and right eyes.

The processing circuitry <NUM> may be employed for eye tracking for both eyes, or there may be separate processing circuitry <NUM> for the left and right eyes. The eye tracking system <NUM> may for example perform eye tracking for the left and right eyes separately, and may then determine a combined gaze point as an average of the gaze points for the left and right eyes.

The processing circuitry <NUM> may for example comprise one or more processors <NUM>. The processor(s) <NUM> may for example be application-specific integrated circuits (ASIC) configured to perform a specific eye tracking method. Alternatively, the processor(s) <NUM> may configured to execute instructions (for example in the form of a computer program) stored in one or more memories <NUM>. Such a memory <NUM> may for example be comprised in the processing circuitry <NUM> of the eye tracking system <NUM>, or may be external to (for example located remotely from) the eye tracking system <NUM>. The memory <NUM> may store instructions for causing the eye tracking system <NUM> to perform an eye tracking method.

It will be appreciated that the eye tracking system <NUM> described above with reference to <FIG> is provided as an example, and that many other eye tracking systems may be envisaged. For example, the illuminators <NUM> and/or the imaging devices <NUM> need not necessarily be regarded as part of the eye tracking system <NUM>. The eye tracking system <NUM> may for example consist only of the processing circuitry <NUM>. There are even eye tracking systems that do not employ illuminators at all. Further, some eye tracking systems employ other types of eye tracking sensors than imaging devices. In other words, the eye tracking system <NUM> could employ other types of sensor data than images to perform eye tracking. The display <NUM> may for example be comprised in the eye tracking system <NUM>, or may be regarded as separate from the eye tracking system <NUM>.

The method <NUM> described above with reference to <FIG> may for example receive data passively at the steps <NUM> and <NUM> from the first eye tracking sensor (exemplified in <FIG> by the sensor <NUM>) and the eye tracking system (exemplified in <FIG> by the system <NUM>). However, the method <NUM> may further comprise using the first eye tracking sensor to obtain sensor data at the time instance (in other words, the sensor data received at step <NUM>), and/or using the eye tracking system to generate the reference eye tracking data for the time instance (in other words, the eye tracking data received at step <NUM>). In other words, the method <NUM> may actively use the first eye tracking sensor <NUM> and the eye tracking system <NUM>, for example by controlling (or instructing) them to provide the necessary data.

<FIG> is a flow chart of method <NUM> for training an eye tracking model, including such explicit use of the first eye tracking sensor <NUM> and the eye tracking system <NUM>, according to an embodiment. Although the method <NUM> is described below with reference to the eye tracking sensor <NUM> and the eye tracking system <NUM> shown in <FIG>, it will be appreciated that a different eye tracking sensor and/or a different eye tracking system may be employed in the method <NUM>.

The method <NUM> comprises using <NUM> the first eye tracking sensor <NUM> to obtain sensor data at a time instance. This corresponds to the sensor data received at step <NUM> in the method <NUM>.

The method <NUM> comprises using <NUM> an eye tracking system <NUM> to generate reference eye tracking data for the time instance. The eye tracking system <NUM> comprises a second eye tracking sensor <NUM>. The reference eye tracking data is generated by the eye tracking system <NUM> based on sensor data obtained by the second eye tracking sensor <NUM> at the time instance. In other words, the generated reference eye tracking data corresponds to the generated reference eye tracking data received at step <NUM> in the method <NUM>.

The method <NUM> comprises training <NUM> the eye tracking model based on the sensor data obtained by the first eye tracking sensor <NUM> at the time instance and the generated reference eye tracking data. In other word, the method <NUM> comprises the same training step <NUM> as the method <NUM>.

According to some embodiments, the eye tracking data predicted by the eye tracking model in the method <NUM> or the method <NUM> indicates a predicted gaze point of an eye, and the generated reference eye tracking data (received at step <NUM> of the method <NUM> or obtained at step <NUM> of the method <NUM>) indicates a reference gaze point of the eye. The predicted gaze point and the reference gaze point may for example be gaze points at a display. This is exemplified in <FIG> where a predicted gaze point <NUM> and a reference gaze point <NUM> are shown at the display <NUM>. A distance <NUM> between these two gaze points is also shown in <FIG>.

According to some embodiments, the eye tracking data predicted by the eye tracking model in the method <NUM> or the method <NUM> indicates a predicted gaze ray of an eye, and the generated reference eye tracking data (received at step <NUM> or obtained at step <NUM>) indicates a reference gaze ray of the eye. This is exemplified in <FIG>, which shows two example gaze rays. A first gaze ray <NUM> starts at a first estimated eye position <NUM> and is directed along a first gaze vector <NUM>. A second gaze ray <NUM> starts at a second estimated eye position <NUM> and is directed along a second gaze vector <NUM>. The first gaze ray <NUM> may for example be a gaze ray predicted by the eye tracking model in the method <NUM> or the method <NUM>, and the second gaze ray <NUM> may for example be a reference gaze ray indicated by the generated reference eye tracking data received at step <NUM> in the method <NUM> or obtained at step <NUM> in the method <NUM>.

<FIG> also shows that a deviation between the gaze rays <NUM> and <NUM> may for example be measured via an angle <NUM> formed between the gaze vectors <NUM> and <NUM>. A distance <NUM> may also be formed between the estimated eye positions <NUM> and <NUM>. <FIG> also shows that sensor data (such as an image <NUM> of an eye) may be employed by an eye tracking model <NUM> to predict eye tracking data such as the gaze ray <NUM>.

According to some embodiments, the eye tracking data predicted by the eye tracking model in the method <NUM> or the method <NUM> indicates a predicted position of an eye in space, and the generated reference eye tracking data (received at step <NUM> or obtained at step <NUM>) indicates a reference position of the eye in space. This is exemplified in <FIG>. The gaze origin <NUM> of the first gaze ray <NUM> in <FIG> may be an eye position predicted by the eye tracking model in the method <NUM> or the method <NUM>. The gaze origin <NUM> of the second gaze ray <NUM> in <FIG> may be a reference eye position indicated by the generated reference eye tracking data received at step <NUM> in the method <NUM> or obtained at step <NUM> in the method <NUM>.

<FIG> shows a scheme for how an eye tracking model may be trained in the methods <NUM> and <NUM> described above with reference to <FIG> and <FIG>, according to an embodiment. In the present embodiment, the step of training <NUM> the eye tracking model comprises:.

In other words, an objective function (such as a cost function or loss function) is employed to evaluate whether or not the predictions made by the eye tracking model seem to be compatible with the reference eye tracking data. The eye tracking model is updated <NUM> to improve its ability to make accurate predictions.

The step <NUM> of applying the objective function may include inserting the eye tracking data predicted by the eye tracking model for the time instance and the generated reference eye tracking data into the objective function. It will be appreciated that the step <NUM> of applying the objective function may for example also comprise inserting additional data into the objective function.

The step <NUM> of updating the eye tracking model may for example comprise modifying a value of at least one parameter of the eye tracking model. If the objective function is a cost function (or a loss function), which is supposed to have a low value if the prediction <NUM> is accurate, then the eye tracking model may for example be modified for reducing a value of the objective function (for example via graduate descent). If, on the other hand, the objective function is a function that should be maximized (for example if the objective function is a cost function multiplied by -<NUM>), then the eye tracking model may for example be modified for increasing a value of the objective function.

According to some embodiments, the step of applying <NUM> the objective function comprises forming a distance between a predicted gaze point indicated by the predicted eye tracking data for the time instance and a reference gaze point indicated by the generated reference eye tracking data. This is exemplified in <FIG> where a distance <NUM> between a predicted gaze point <NUM> and a reference gaze point <NUM> is illustrated. If the prediction provided by the eye tracking model is accurate, this distance <NUM> should be small.

According to some embodiments, the step of applying <NUM> the objective function comprises forming a deviation between a predicted gaze ray indicated by the predicted eye tracking data for the time instance and a reference gaze ray indicated by the generated reference eye tracking data. This is exemplified in <FIG>, where the first gaze ray <NUM> corresponds to a gaze ray predicted by the eye tracking model in the method <NUM> or the method <NUM>, and the second gaze ray <NUM> corresponds to a reference gaze ray indicated by the generated reference eye tracking data received at step <NUM> in the method <NUM> or obtained at step <NUM> in the method <NUM>. The deviation between the predicted gaze ray <NUM> and the reference gaze ray <NUM> may for example be expressed in form of the angle <NUM> formed between the respective gaze vectors <NUM> and <NUM>. If the prediction provided by the eye tracking model is accurate, this angle <NUM> should be small.

According to some embodiments, the step of applying <NUM> the objective function comprises forming a distance between a predicted eye position indicated by the predicted eye tracking data for the time instance and a reference eye position indicated by the generated reference eye tracking data. This is exemplified in <FIG>, where the gaze origin <NUM> of the first gaze ray <NUM> corresponds to an eye position predicted by the eye tracking model in the method <NUM> or the method <NUM>, and where the gaze origin <NUM> of the second gaze ray <NUM> corresponds to a reference eye position indicated by the generated reference eye tracking data received at step <NUM> in the method <NUM> or obtained at step <NUM> in the method <NUM>. If the prediction provided by the eye tracking model is accurate, the distance <NUM> between the predicted eye position <NUM> and the reference eye position <NUM> should be small.

<FIG> shows a scheme for how training of the eye tracking model in the methods <NUM> and <NUM> described above with reference to <FIG> may be performed, according to an embodiment. In the present embodiment, the step <NUM> of training the eye tracking model comprises predicting <NUM> eye tracking data for the time instance using the eye tracking model and the sensor data obtained by the first eye tracking sensor at the time instance. A check <NUM> is performed to see whether a deviation between the eye tracking data predicted <NUM> by the eye tracking model for the time instance and the generated reference eye tracking data (which is received at step <NUM> or generated at step <NUM>) exceeds a threshold. If the deviation exceeds the threshold, then the scheme/method proceeds by training <NUM> the eye tracking model based on the eye tracking data predicted by the eye tracking model for the time instance and the generated reference eye tracking data. If, on the other hand, the deviation does not exceed the threshold, then the eye tracking data predicted by the eye tracking model for the time instance and the generated reference eye tracking data may not be employed for training of the eye tracking model (as indicated by the step <NUM> in <FIG>).

In other words, as long as the predicted gaze tracking data checks out with (or matches) the reference gaze tracking data, there may be no need to train the eye tracking model. If, on the other hand, it is detected that the predicted eye tracking data deviates from the reference eye tracking data, then training may be needed. The eye tracking model may for example perform well for some types of input data (or for some scenarios or user activities) but worse for other types of input data. The scheme described above with reference to <FIG> allows the eye tracking model to be trained for such types of input data where further training is actually needed, rather than training blindly for all types of input data.

The deviation employed at step <NUM> in <FIG> may for example be a deviation between a predicted gaze point and a reference gaze point, a deviation between a predicted gaze ray and a reference gaze ray, or a deviation between a predicted eye position and a reference eye position.

The threshold employed at step <NUM> in <FIG> may for example be predefined. However, embodiments may also be envisaged in which this threshold may be changed or modified.

<FIG> shows a method <NUM> of training an eye tracking model, according to an embodiment. The method <NUM> is similar to the method <NUM> described above with reference to <FIG>, but further comprises using <NUM> the eye tracking system (exemplified in <FIG> by the eye tracking system <NUM>) to detect a certain trigger action of an eye <NUM>. The certain trigger action comprises:.

In the method <NUM>, the step <NUM> of receiving of the sensor data obtained by the first eye tracking sensor at the time instance (which may be referred to as a data acquisition step) and/or the step <NUM> of training of the eye tracking model may be performed in response to detection of the certain trigger action of the eye. In other words, if the trigger action is detected, then the data acquisition step <NUM> and the training step <NUM> may be performed. If, on the other hand, the trigger action is not detected, then the data acquisition step <NUM> and the training step <NUM> may be skipped, or the data acquisition step <NUM> may be performed and the training step <NUM> may skipped.

The method <NUM> described above with reference to <FIG> allows the eye tracking model to be trained to handle certain scenarios better, such as fixations, saccades or smooth pursuits, rather than training the eye tracking model blindly for all types of input data. Certain scenarios may for example be particularly difficult for the eye tracking model to handle, or some scenarios may require better accuracy than other scenarios. It may therefore be useful to train the model specifically for such scenarios. The training of the eye tracking model may for example be based on training data collected in connection with the trigger action, such as during the trigger action, and/or shortly before the trigger action, and/or shortly after the trigger action.

In the method <NUM>, the trigger action may for example be detected by analyzing eye tracking data received from the eye tracking system, or an explicit indication of the trigger action may for example be received from the eye tracking system.

The method <NUM> described above with reference to <FIG>, is provided in the context of the method <NUM> described above with reference to <FIG>. It will be appreciated that the method <NUM> could easily be modified to instead be employed in the context of the method <NUM> described above with reference to <FIG>. In other words, the step <NUM> and/or the step <NUM> in the method <NUM> may be made conditional upon detection of a certain trigger action of the eye <NUM>, just like the steps <NUM> and <NUM> from the method <NUM>.

<FIG> is a flow chart of a method <NUM> for training an eye tracking model, according to an embodiment. The method <NUM> is similar to the method <NUM> described above with reference to <FIG>, but the method <NUM> includes new steps <NUM>-<NUM>, and the step <NUM> is expressed in terms of a new step <NUM>. In the present embodiment, the eye tracking model is one of several eye tracking models. The eye tracking models are associated with respective potential users or persons.

The method <NUM> comprises detecting <NUM> presence of a user (or of a person), and selecting <NUM> the eye tracking model associated with the user (or person). Presence of the user may for example be detected <NUM> by the eye tracking system, or via the first eye tracking sensor. Presence of the user may for example be detected via biometric data (such as facial recognition, or a finger or, or an iris scan) or via some kind of credential (such as a smart card or a wireless sensor tag). The eye tracking model may for example be selected <NUM> from a database of potential users and their respective eye tracking models.

The method <NUM> comprises training <NUM> the selected eye tracking model based on the sensor data obtained by the first eye tracking sensor at the time instance (in other words, the sensor data received at step <NUM>) and the generated reference eye tracking data (in other words, reference eye tracking data received at step <NUM>).

The method <NUM> described above with reference to <FIG>, is provided in the context of the method <NUM> described above with reference to <FIG>. It will be appreciated that the method <NUM> could easily be modified to instead be employed in the context of the method <NUM> described above with reference to <FIG>. More specifically, the step <NUM>-<NUM> in the method <NUM> could for example be replaced by the steps <NUM>-<NUM> from the method <NUM>.

According to some embodiments, the method <NUM> described above with reference to <FIG> may be extended to perform training for sensor data from a sequence of time instances. More specifically, the method <NUM> may comprise:.

In other words, data for the time instances may be employed for training the eye tracking model, or may be stored for use in training later on. The training data may for example be stored in a database or may be uploaded to the cloud. Training of the eye tacking data may for example be performed at a remote location from where the training data was collected. Training of the eye tracking model may for example be performed gradually as training data becomes available. Alternatively, plenty a training data may be collected first and training may then be performed using the collected training data. After the eye tracking model has been sufficiently trained, it may for example be employed in an eye tracking system to predict eye tracking data in real time.

The methods and schemes described above with reference to <FIG> represent a first aspect of the present disclosure. The system <NUM> described above with reference to <FIG> represents a second aspect of the present disclosure. The system <NUM> (or the processing circuitry <NUM> of the system <NUM>) may for example be configured to perform the method of any of the embodiments of the first aspect described above. The system <NUM> may for example be configured to perform the method <NUM> described above with reference to <FIG> or the method <NUM> described above with reference to <FIG>.

The system <NUM> may for example comprise processing circuitry <NUM> (or one or more processors <NUM>) and one or more memories <NUM>, the one or more memories <NUM> containing instructions executable by the processing circuitry <NUM> (or one or more processors <NUM>), whereby the system <NUM> is operable to perform the method of any of the embodiments of the first aspect disclosed herein.

As described above with reference to <FIG>, the system <NUM> need not necessarily comprise all the elements shown in <FIG>.

A third aspect of the present disclosure is represented by embodiments of a non-transitory computer-readable storage medium <NUM> storing instructions which, when executed by the system <NUM> (or by processing circuitry <NUM> of the system <NUM>), cause the system <NUM> to perform the method of any of the embodiments of the first aspect described above (such as the method <NUM> described above with reference to <FIG>, or the method <NUM> described above with reference to <FIG>).

As described above with reference to <FIG>, the storage medium <NUM> need not necessarily be comprised in the system <NUM>.

The person skilled in the art realizes that the proposed approach presented in the present disclosure is by no means limited to the preferred embodiments described above. For example, the embodiments described above with reference to <FIG> may be combined to form further embodiments. Further, it will be appreciated that the system <NUM> shown in Fig. <NUM> is merely intended as an example, and that other systems may also perform the methods described above with reference to <FIG>.

It will be appreciated that processing circuitry <NUM> (or one or more processors) may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and/or encoded logic operable to provide computer functionality, either alone or in conjunction with other computer components (such as a memory or storage medium).

It will also be appreciated that a memory or storage medium <NUM> (or a computer-readable medium) may comprise any form of volatile or non-volatile computer readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and/or any other volatile or non-volatile, non-transitory device readable and/or computer-executable memory devices that store information, data, and/or instructions that may be used by a processor or processing circuitry. Additionally, variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims. In the claims, the word "or" is not to be interpreted as an exclusive or (sometimes referred to as "XOR").

Claim 1:
A method (<NUM>) for training an eye tracking model (<NUM>), wherein the eye tracking model (<NUM>) is adapted to estimate eye tracking data based on sensor data (<NUM>) from a first eye tracking sensor (<NUM>) of a new eye tracking system (<NUM>), wherein the first eye tracking sensor (<NUM>) is arranged to monitor an eye (<NUM>), the method comprising:
receiving (<NUM>) sensor data, wherein the sensor data is obtained by the first eye tracking sensor (<NUM>) at a time instance;
characterized by:
receiving (<NUM>) reference eye tracking data for said time instance generated by an old eye tracking system (<NUM>) comprising a second eye tracking sensor (<NUM>), arranged to monitor the eye (<NUM>), wherein the reference eye tracking data is generated by the old eye tracking system (<NUM>) based on sensor data obtained by the second eye tracking sensor (<NUM>) at said time instance; and
training (<NUM>) the eye tracking model (<NUM>) based on the sensor data obtained by the first eye tracking sensor (<NUM>) at said time instance and the generated reference eye tracking data for said time instance,
wherein the eye tracking model (<NUM>) is one of several eye tracking models, the eye tracking models being associated with respective potential users, the method comprising:
detecting (<NUM>) presence of a user;
selecting (<NUM>) the eye tracking model (<NUM>) associated with the user; and
training (<NUM>) the selected eye tracking model (<NUM>) based on the sensor data obtained by the first eye tracking sensor (<NUM>) at said time instance and the generated reference eye tracking data for said time instance.