Disclosed are systems and methods for extracting high resolution oculometric parameters. A video stream having a video of a face of a user is processed to obtain a set of oculometric parameters, such as eyelid data, iris data (e.g., iris translation, iris radius and iris rotation), and pupil data (e.g., pupil center and pupil radius) at a first resolution. A deconvolution process is performed on the video stream to improve accuracy or resolution of the oculometric parameters, based on stimulus information of a video stimulus displayed on a client device associated with the user, environment data of an environment in which the user is located, device data of the client device, etc. The oculometric parameters are then processed using a prediction model that is trained based on high resolution oculometric parameters obtained using eye tracking devices to predict oculometric parameters at a resolution greater than the first resolution.

BACKGROUND

Several papers have been published which demonstrate the connections between minute eye movements and the progression of neurological disorders (see, for example, References section below). Typically, these eye movements are measured in well-controlled lab settings (e.g., no movements, controlled ambient light, or other such parameters) using dedicated devices (e.g., infrared eye trackers, pupilometers, or other such devices), which is a challenge to set up, cost prohibitive, or may involve a significant amount of time and effort to create or maintain the controlled setup. Some prior technologies use neural networks to obtain these eye movements but do not attain the needed resolution for measurements (e.g., around 0.1 mm or greater resolution). Additionally, prior technologies focus on measuring reactions of the eye to specific standardized stimuli (e.g., controlled light stimulus, stimulus provided to measure specific parameters, stimulus that a patient needs to me made aware of, or other such stimulus) with controlled ambient conditions. These and other drawbacks exist.

SUMMARY

Disclosed embodiments relate to systems and methods for facilitating measurement of minute eye parameters or movements with video captured by standard cameras (e.g., smartphone camera, webcam, or another video capturing device) and without the need for controlled settings. The embodiments obtain various high resolution oculometric parameters. For example, the embodiments may obtain eyelid data, such as coordinates of eyelid boundaries. In another example, the embodiments obtain iris data such as iris translation or iris center, iris rotation, iris radius, iris visible fraction, or iris coverage asymmetry. In yet another example, the embodiments obtain pupil data such as a pupil center, a pupil radius, a pupil visible fraction, or a pupil coverage asymmetry. The oculometric parameters are obtained at a high resolution, for example, at a sub-pixel level, 0.1 mm, or other higher resolution. In some embodiments, the oculometric parameters may be used in determining eye movement measurements, such as pupillary response parameter, gaze, saccade, fixation, etc., which may be used to produce digital biomarkers that may further be used to diagnose or measure progress of a neurological condition or disorder. The digital biomarkers are objective, sensitive, accurate, correlated with disease progression, and may be conducted remotely and even in a distributed manner. The oculometric parameters or eye movement parameters may be obtained using non-standard stimulus presented to the user on a user device (e.g., stimulus that a patient need not be made aware of, stimulus that may be used to measure multiple parameters, stimulus such as a video playing on a display device being observed by the user, or other such stimulus that does not need controlled ambient lighting conditions).

In some embodiments, the disclosed embodiments may use probabilistic methods (e.g., maximum likelihood estimate, or other such method) or signal processing methods (e.g., blind deconvolutions, or other such method) to increase resolution and ascertain oculometric parameters, eye movement parameters at a very high resolution (e.g., at a sub-pixel level, 0.1 mm, or other higher resolution) on layers of data present in video frame sequences of eye movements captured by standard cameras in uncontrolled settings. The disclosed embodiments may employ prediction models (e.g., machine learning (ML) models such as neural networks) to obtain one or more oculometric parameters at high resolution.

Various other aspects, features, and advantages of the inventions will be apparent through the detailed description of the invention and the drawings attached hereto. It is also to be understood that both the foregoing general description and the following detailed description are exemplary and not restrictive of the scope of the inventions. As used in the specification and in the claims, the singular forms of “a”, “an”, and “the” include plural referents unless the context clearly dictates otherwise. In addition, as used in the specification and the claims, the term “or” means “and/or” unless the context clearly dictates otherwise.

DETAILED DESCRIPTION

FIG.1Ashows a system100for facilitating measurement of oculometric parameters and eye movements of a user, consistent with various embodiments.FIG.1Bshows a process175of extracting high resolution (HR) oculometric parameters associated with an eye, consistent with various embodiments. As shown inFIG.1A, system100may include computer system102, client device106, or other components. By the way of example, computer system102may include any computing device, such as a server, a personal computer (PC), a laptop computer, a tablet computer, a hand-held computer, or other computer equipment. Computer system102may include video management subsystem112, oculometric parameter subsystem (OPS)114, marker subsystem116, or other components. The client device106may include any type of mobile terminal, fixed terminal, or other device. By way of example, client device106may include a desktop computer, a notebook computer, a tablet computer, a smartphone, a wearable device, or other client device. A user, e.g., user140, may, for instance, utilize the client device106to interact with one or more servers, or other components of system100. The camera120may capture still or video images of the user140, e.g., a video of the face of the user140. The camera120may be integrated with the client device106(e.g., smartphone camera) or may be a standalone camera (e.g., webcam or other such camera) that is connected to the client device106.

A component of system100may communicate with one or more components of system100via a communication network150(e.g., Internet, a mobile phone network, a mobile voice or data network, a cable network, a public switched telephone network, or other types of communications network or combinations of communications networks). The communication network150may be a wireless or wired network. As an example, the client device106and the computer system102may communicate wirelessly.

It should be noted that, while one or more operations are described herein as being performed by particular components of computer system102, those operations may, in some embodiments, be performed by other components of computer system102or other components of system100. As an example, while one or more operations are described herein as being performed by components of computer system102, those operations may, in some embodiments, be performed by components of client device106.

It should be noted that, although some embodiments are described herein with respect to machine learning models, other prediction models (e.g., statistical models or other analytics models) may be used in lieu of or in addition to machine learning models in other embodiments (e.g., a statistical model replacing a machine learning model and a non-statistical model replacing a non-machine-learning model in one or more embodiments).

In some embodiments, system100facilitates determination of oculometric parameters170-178of a user140at a high resolution (e.g., sub-pixel level, 0.1 mm or higher resolution) using a video stream125(e.g., a sequence of frames of an image of the eye) of the eye movements captured by the camera120. The video management subsystem112obtains the video stream125from the client device106and stores the video stream125in the database132. The OPS114processes the video stream125to extract the oculometric parameters, such as (i) pupil center176, (ii) pupil radius178, (iii) iris radius174, (iv) iris translation172, and (v) iris rotation170of each of the eyes, among other parameters, at a high resolution. These oculometric parameters may be used to determine eye movement parameters (e.g., pupillary response parameters and gaze parameters) that are indicative of eye movements responsive to a non-standard stimuli (e.g., a video or image displayed on the client device106). The marker subsystem116may produce digital biomarkers (e.g., parameters) based on the eye movement parameters that may act as an indicator of one or more neurological disorders.

The video management subsystem112may store the video stream125in the database132in any of several formats (e.g., WebM, Windows Media Video, Flash Video, AVI, QuickTime, AAC, MPEG-4, or another file format). The video management subsystem112may also provide the video stream125as an input (e.g., in real-time) to the OPS114for generating the oculometric parameters. In some embodiments, the video management subsystem112may perform preprocessing191of the video stream125to convert the video stream125to a format that is suitable for oculometric parameter extraction by the OPS114. For example, the video management subsystem112may perform an extract, transform and load (ETL) process on the video stream125to generate a preprocessed video stream151for the OPS114. The ETL process is typically a multi-phase process where data is first extracted then transformed (e.g., cleaned, sanitized, scrubbed) and finally loaded into a target system. The data may be collated from one or more sources, and it may also be outputted to one or more destinations. In one example, the ETL process performed by the video management subsystem112may include adjusting color and brightness (e.g., white balance) in the video stream125. In another example, the ETL process may include reducing noise from the video stream125. In yet another example, the ETL process may include improving resolution of the video stream125based on multi-frame data, for example, by implementing one or more multi-frame super-resolution techniques that recover a high-resolution image (or a sequence) from a sequence of low-resolution images. In some embodiments, super-resolution is a digital image processing technique to obtain a single high-resolution image (or a sequence) from multiple blurred low-resolution images. The basic idea of super-resolution is that the low-resolution images of the same scene contain different information because of relative subpixel shifts; thus, a high-resolution image with higher spatial information can be reconstructed by image fusion. The video management subsystem112may perform the above preprocessing operations and other such similar preprocessing operations to generate a preprocessed video stream151that is of a format suitable for the OPS114to perform the oculometric parameter extraction.

In some embodiments, the video management subsystem112may obtain the preprocessed video stream151via a prediction model (e.g., based on the video stream125obtained from the client device106). As an example, the video management subsystem112may input the video stream125obtained from the client device106to the prediction model, which then outputs the preprocessed video stream151. In some embodiments, the system100may train or configure a prediction model to facilitate the generation of a preprocessed video stream. In some embodiments, system100may obtain a video stream from a client device associated with a user (e.g., video stream125having a video of a face of the user140) and provide such information as input to a prediction model to generate predictions (e.g., preprocessed video stream151in which color or brightness are adjusted, resolution of the video is improved, etc.). System100may provide reference feedback to the prediction model and the prediction model may update one or more portions of the prediction model based on the predictions and the reference feedback. As an example, where the prediction model generates predictions based on video stream obtained from a client device, one or more preprocessed video streams associated with such input video streams may be provided as reference feedback to the prediction model. As an example, a particular preprocessed video stream may be verified as a suitable video stream for oculometric parameter extraction by the OPS114(e.g., via user confirmation of the preprocessed video, via one or more subsequent actions demonstrating such goal, etc.) based on one or more user responses or one or more other user actions via one or more services. The foregoing user input information may be provided as input to the prediction model to cause the prediction model to generate predictions of the preprocessed video stream, and the verified preprocessed video stream may be provided as reference feedback to the prediction model to update the prediction model. In this way, for example, the prediction model may be trained or configured to generate more accurate predictions. In some embodiments, the training dataset may include several data items in which each data item includes at least a video obtained from a client device and a corresponding preprocessed video as the ground truth.

In some embodiments, the foregoing operations for updating the prediction model may be performed with a training dataset with respect to one or more users (e.g., a training dataset associated with a given user to specifically train or configure the prediction model for the given user, a training dataset associated with a given cluster, demographic, or other group to specifically train or configure the prediction model for the given group, a training dataset associated with any given set of users, or other training dataset). As such, in some embodiments, subsequent to the updating of the prediction model, system100may use the prediction model to facilitate generation of a preprocessed video stream for facilitating oculometric parameter extraction by the OPS114.

The video management subsystem112may generate the preprocessed video stream151prior to inputting the video stream125to the OPS114, or after storing the video stream125in the database132. Further, the video management subsystem112may also store the preprocessed video stream151in the database132along with the video stream125.

The OPS114may process the video stream125or the preprocessed video stream151to extract, determine, derive, or generate the oculometric parameters (e.g., oculometric parameters170-178), which are characteristic of the eyes145of the user140. In some embodiments, the oculometric parameters include parameters such as (i) a pupil center176, (ii) a pupil radius178, (iii) an iris radius174, (iv) an iris translation172, and (v) an iris rotation170. The pupil center176is a center of the pupil and may be represented using coordinates (e.g., three-dimensional (3D) lateral displacement vector (x,y,z)) that identifies a location of the pupil center in a coordinate system. The pupil radius178may be a radius of the pupil and may be represented as a scalar with distance units. The iris translation172(or iris translation160) is a center of the iris and may be represented using coordinates (e.g., 3D lateral displacement vector (x,y,z)) that identify a location of the iris center in a coordinate system. The iris radius174(or iris radius162) may be a radius of the iris and may be represented as a scalar with distance units. In some embodiments, the iris radius174is assumed to have a small variance in the human population (e.g., 10%-15% considering iris radius varies from 11 mm-13 mm among the human population). The iris rotation170(or iris rotation158) is representative of a rotation of the iris and is retrieved as the iris normal vector (e.g., perpendicular to the iris circle plain). In some embodiments, the iris rotation170may be projected to a 3D vector in a device coordinate system (e.g., x, y, z) or represented by a 2D vector in a head coordinate system (e.g., azimuth angle and elevation angle). In some embodiments, the OPS114may obtain the coordinates of the oculometric parameters in one or more coordinate systems, such as the device coordinate system or the head coordinate system illustrated inFIGS.3A and3B, respectively.

FIG.3Aillustrates a device coordinate system, consistent with various embodiments. In some embodiments, the device coordinate system is a cartesian coordinate system, with a center of a display302of a client device (e.g., the client device106) considered as an origin304having the (x,y,z) coordinates as (0,0,0). The device coordinate system defines the three orthogonal axes, e.g., x-axis in the direction of the horizontal edge of the display302, y-axis in the direction of the vertical edge of the display302and perpendicular to the x-axis, and z-axis in a direction perpendicular to the plane of the display302.

FIG.3Billustrates a head coordinate system, consistent with various embodiments. In some embodiments, the head coordinate system is defined by a plane that contains the following points in the face bounding polygon: most left, most right and most upper (points 1, 2 and 3 inFIG.3B) detectable points in the face polygon. For example, the upper most point 3 may be defined as the most upper detectable point in the face bounding polygon and may be located in the cross-section between the y-axis and the head. When obtaining a face image, it is The horizontal axis (x) may be defined as the line between the most left and the most right points, and the perpendicular vector to that line on the defining plane may be defined as the vertical axis (y). The depth axis (z) may be orthogonal to both the x and y axes. The origin of the head coordinate system (point 0 in theFIG.3B) is located in the center between the eyes, as projected on the defining plane by displacement on the z-direction. In some embodiments, the iris rotation may be represented on the head coordinate system as the eyeball rotation, given two angles (e.g., azimuth and elevation).

Referring to the OPS114, the OPS114may be configured to represent the coordinates of the oculometric parameters in any of the coordinate systems, including the head coordinate system or the device coordinate system. Further, the OPS114may be configured to convert the coordinates from one coordinate system to another, such as from the head coordinate system to the device coordinate system or vice versa.

The OPS114may also obtain eyelid data164, such as coordinates of the upper eyelid boundary and lower eyelid boundary of both the eyes. The eyelid data164may be used in determining, or in improving the accuracy of, the oculometric parameters. In some embodiments, the OPS114may obtain the above data (e.g., oculometric parameters170-178, eyelid data164or other data) as time series data. For example, the OPS114may extract a first set of values of the oculometric parameters170-178for a first moment in time of the video stream125, a second set of values of the oculometric parameters170-178for a second moment in time of the video stream125and so on. That is, the OPS114may continuously extract the oculometric parameters170-178at a specified time interval (e.g., every millisecond, every few milliseconds, or other temporal resolution). Further, the OPS114may obtain the above data for one or both the eyes145of the user140.

The OPS114may perform several processes to extract the above oculometric parameters (e.g., oculometric parameters170-178) from the preprocessed video stream151. For example, in an extraction process192, the OPS114may obtain a first set of oculometric parameters, such as iris data153, eyelid data155and pupil center157. In some embodiments, the OPS114may use computer vision techniques on the preprocessed video stream151to model the translation and rotation of a face of the user140with respect to the camera120and to identify the eyes and the mouth in the face. After identifying the position of the eyes, the OPS114may obtain the eyelid data155(e.g., coordinates of an array of points that describe a polygon representing the eye lids) for each frame (e.g., using the fact that their edge may be approximated with quadratic functions). In some embodiments, the OPS114may determine the iris data153(e.g., a shape, such as an ellipsoid, representing the iris) by determining the pixels that are in or out of the iris radius at any moment in time based on the eyelid's location as a time series. In some embodiments, the OPS114may be configured to consider that the iris and pupil are ellipsoids with a relatively uniform and dark color compared to the sclera (e.g., the white part of the eye) in determining the pixels that are in or out. The iris data153may include several coordinates such as a first coordinate and a second coordinate, which correspond to right and left coordinates of a shape representing the iris (e.g., an ellipse); a third coordinate and fourth coordinate, which correspond to top and bottom coordinates of the shape; and a fifth coordinate corresponding to a center of the iris. The pupil center157may also be determined based on the time series of the eyelid data155and iris data153. In some embodiments, the pupil center157may include a coordinate representing a location of the center of the pupil.

In some embodiments, the OPS114may implement a prediction model in the extraction process192and may obtain the first set of oculometric parameters via the prediction model (e.g., based on the video stream125or the preprocessed video stream151). As an example, the OPS114may input the video stream125or the preprocessed video stream151to the prediction model, which then outputs the first set of oculometric parameters. In some embodiments, the system100may train or configure a prediction model to facilitate the generation of the first set of oculometric parameters. In some embodiments, system100may obtain a video stream having a video of the face of a user (e.g., video stream125or preprocessed video stream151) and provide such information as input to a prediction model to generate predictions (e.g., first set of oculometric parameters such as iris data, eyelid data, pupil center, etc.). System100may provide reference feedback to the prediction model and the prediction model may update one or more portions of the prediction model based on the predictions and the reference feedback. As an example, where the prediction model generates predictions based on the video stream125or the preprocessed video stream151, the first of oculometric parameters associated with such input video streams may be provided as reference feedback to the prediction model. As an example, a particular set of oculometric parameters may be verified as an appropriate set of oculometric parameters (e.g., via user confirmation of the set of oculometric parameters, via one or more subsequent actions demonstrating such goal, etc.). The foregoing user input information may be provided as input to the prediction model to cause the prediction model to generate predictions of the first set of oculometric parameters, and the verified set of oculometric parameters may be provided as reference feedback to the prediction model to update the prediction model. In this way, for example, the prediction model may be trained or configured to generate more accurate predictions.

In some embodiments, the foregoing operations for updating the prediction model may be performed with a training dataset with respect to one or more users (e.g., a training dataset associated with a given user to specifically train or configure the prediction model for the given user, a training dataset associated with a given cluster, demographic, or other group to specifically train or configure the prediction model for the given group, a training dataset associated with any given set of users, or other training dataset). As such, in some embodiments, subsequent to the updating of the prediction model, system100may use the prediction model to facilitate generation of a first set of oculometric parameters.

In some embodiments, the OPS114may also further adjust, modify, or correct the first set of parameters based on other factors, such as optometric data associated with a user. For example, the user140may have certain vision conditions such as myopia, hyperopia, astigmatism or other vision conditions and the user may wear corrective lenses such as eyeglasses or contact lens. The OPS114may consider such optometric conditions of the user140and correct, adjust or modify, values of one or more of the first set of oculometric parameters. In some embodiments, the OPS114may obtain the optometric data associated with the user140from user profile data (e.g., stored in the database132).

In some embodiments, the OPS114may further process193the iris data153to extract iris related parameters such as iris rotation158, iris translation160and iris radius162. In some embodiments, the above iris related parameters are obtained using an ML technique. For example, the OPS114may implement a maximum likelihood estimation (MLE) based curve fitting method to estimate the iris related parameters. The iris data is provided as input to the MLE-based curve fitting method (e.g., oval fit—based on an assumption that iris is circular and so when projected to a 2D plane it is oval/ellipse), which generates the above iris related parameters. In some embodiments, in statistics, MLE is a method of estimating the parameters of a probability distribution by maximizing a likelihood function, so that under the assumed statistical model the observed data is most probable. The point in the parameter space that maximizes the likelihood function is called the maximum likelihood estimate. In some embodiments, curve fitting operation is a process of constructing a curve, or mathematical function, that has the best fit to a series of data points, subject to constraints (e.g., oval fit). Curve fitting may involve either interpolation, where an exact fit to the data is required, or smoothing, in which a “smooth” function is constructed that approximately fits the data. Fitted curves may be used as an aid for data visualization, to infer values of a function where no data are available, and to summarize the relationships among two or more variables.

In some embodiments, by performing the MLE-based curve fitting operation the accuracy of at least some of the iris related parameters may be further improved. For example, the shape depicting the iris boundary is further improved from the coordinates of the shape in the iris data153, and therefore an improved or more accurate iris radius162may be determined. In some embodiments, the iris translation160, also referred to as iris center, is similar to the coordinates of the iris center in the iris data153. In some embodiments, in the process193, the OPS114may consider different iris centers, construct an iris shape for each of the candidate iris centers, and assign a confidence score for each of the shapes. Such a method may be repeated for different iris centers, and the shape having a score satisfying a score criterion (e.g., the best score) is selected. In some embodiments, the score for each iteration may reflect a log-likelihood of the examined parameter set (e.g., coordinates of the iris shape, radius and center), based on the known physical constraints and assumptions. In some embodiments, the physical constraints and assumptions may improve the efficiency of the MLE process by introducing better estimated initial conditions before iterating towards convergence. For example, some assumptions may include that: the iris is circular (which when projected on a plane as 2D is represented as ellipse/oval, iris center is expected at the centroid of the circles, high contrast between iris and sclera, degree of similarity of orientation between left and right eye, stimulus-based assumption on the expected gaze point, and brightness of a display of the client device on which a video stimulus is shown to the user. In some embodiments, examples of physical constraints may include: pupil dilation dependence in overall brightness (light conditions), limited range of face orientation as users look at the display, blinking may initiate uncertainty due to readjustment time, or physical possible range of motion (both face and eyeball).

The OPS114may perform a deconvolution process194on the video stream125or the preprocessed video stream151to adjust (e.g., improve the resolution or increase the accuracy of) some oculometric parameters, such as the eyelid data155and pupil center166, and determine other oculometric parameters, such as the pupil radius. In some embodiments, the OPS114may perform a blind deconvolution process to improve the accuracy of the oculometric parameters. In image processing, blind deconvolution is a deconvolution technique that permits recovery of the target scene from a single or a set of “blurred” images in the presence of a poorly determined or unknown point spread function (PSF). Regular linear and non-linear deconvolution techniques may utilize a known PSF. For blind deconvolution, the PSF is estimated from the image or image set, allowing the deconvolution to be performed. Blind deconvolution may be performed iteratively, whereby each iteration improves the estimation of the PSF and the scene, or non-iteratively, where one application of the algorithm, based on exterior information, extracts the PSF. After determining the PSF, the PSF may be used in deconvolving the video stream125or the preprocessed video stream151to obtain the adjusted oculometric parameters (e.g., adjusted eyelid data164, adjusted pupil center166or pupil radius168).

In some embodiments, applying blind deconvolution algorithm may help in decreasing or removing the blurring in the images/video, and in estimating irradiance (e.g., based on parameters of the deduced convolution vector (e.g., PSF)). In some embodiments, the estimated irradiance is an estimated irradiance reflected from the eye of the user140, which is an indication of the irradiance the eyes are exposed to. After the blurring is reduced or removed and the irradiance is obtained, the deconvolution process may be repeated one or more times to obtain adjusted oculometric parameters (e.g., whose accuracy or resolution is improved compared to oculometric parameters prior to the deconvolution process194).

In some embodiments, in determining the PSF, the blind deconvolution process considers various factors. For example, the OPS114may input (a) the video stream125or the preprocessed video stream151(e.g., which is a time-series image sequence) with high temporal resolution (e.g., equal to or faster than 30 ms per frame); (b) stimulus data such as spatio-temporal information on a stimulus presented on the display of the client device106, and optical properties information on the of the stimulus, including spectral properties (e.g., color) and intensity (e.g., brightness); (c) environment data such as lighting in the environment (e.g., a room) where the user is located, which can be measured using information obtained from the camera120, (d) device data such as orientation information of the client device106, information from one or more sensors associated with the client device106such as acceleration sensors. Such factors help in efficient calculation of the PSF, minimizes the noise uncertainty, leading to better accuracy in the overall deconvolution.

In some embodiments, the deconvolution process194is integrated with an MLE process to further improve the accuracy of the oculometric parameters. For example, the MLE process may be used to improve the accuracy of the eyelid data155. As described above, the eyelid data155includes coordinates of an array of points that describe a shape (e.g., polygon) of the eyelids. The MLE process performs a parabolic curve fitting operation on the eyelid data155with a constraint that the eyelids are parabolic in shape to obtain a more accurate representation of the eyelids, as the adjusted eyelid data164. In some embodiments, the adjusted eyelid data164may include coordinates of a collection of points that describe a shape (e.g., parabolic) of an eyelid. Such adjusted eyelid data164may be obtained for both eyelids of the eye and for both eyes. In some embodiments, the OPS114may use the adjusted eyelid data164information to predict the precise oculometric parameter values, such as the pupil radius center, even when some are physically challenging (e.g., when the upper eyelid covers the pupil center, or when only some of the iris is exposed to the camera.

In some embodiments, the MLE process may also be used for obtaining or improving pupil related data, such as pupil radius168. In some embodiments, by performing the MLE-based curve fitting operation the accuracy of the pupil related data may be further improved. In some embodiments, in the MLE process, the OPS114may consider different pupil centers, construct a pupil shape for each of the candidate pupil centers, and assign a confidence score for each of the shapes. Such a method may be repeated for different pupil centers, and the shape having a score satisfying a score criterion (e.g., the best score) is selected. Once the shape is selected, the corresponding center is selected as the adjusted pupil center166and the pupil radius168may be determined based on the selected shape and the adjusted pupil center166. In some embodiments, the score for each iteration may reflect a log-likelihood of the examined parameter set (e.g., pupil radius and center), based on the known physical constraints and assumptions. In some embodiments, the physical constraints and assumptions may improve the efficiency of the MLE process by introducing better estimated initial conditions before iterating towards convergence. For example, some assumptions may include that: the pupil is circular (which when projected on a 2D plane is represented as ellipse/oval, pupil center is expected at the centroid of the circle, degree of similarity of orientation between left and right eye, stimulus-based assumption on the expected gaze point, and brightness of a display of the client device on which a video stimulus is shown to the user. In some embodiments, examples of physical constraints may include: pupil dilation dependence in overall brightness (light conditions), limited range of face orientation as users look at the display, blinking may initiate uncertainty due to readjustment time, or physical possible range of motion (both face and eyeball). Accordingly, the OPS114may obtain adjusted oculometric parameters, such as the adjusted eyelid data164, adjusted pupil center166and the pupil radius168from the deconvolution process194. In some embodiments, the above process193of processing iris data and the deconvolution process194may output the following oculometric parameters—adjusted eyelid data164, adjusted pupil center166and the pupil radius168, the iris rotation158, iris translation160and iris radius162(also referred to as “a first set of oculometric parameters”) in a first resolution. For example, the first resolution may be at a pixel level or other lower resolution.

In some embodiments, the OPS114may further improve the resolution of the first set of oculometric parameters. For example, the OPS114may improve the resolution from the first resolution to a second resolution (e.g., 0.1 mm, sub-pixel level or some other resolution that is greater than the first resolution). In some embodiments, the OPS114may input the first set of oculometric parameters to an improve resolution process195to obtain a second set of oculometric parameters (e.g., oculometric parameters170-178) at the second resolution. In some embodiments, the improve resolution process195may obtain the second set of oculometric parameters via a prediction model (e.g., based on the first set of oculometric parameters). As an example, the OPS114may input the first set of oculometric parameters obtained at the first resolution to the prediction model, which then outputs the second set of oculometric parameters at the second resolution. In some embodiments, the system100may train or configure a prediction model to facilitate the generation of the second set of oculometric parameters. In some embodiments, system100may obtain input data such as (a) a video stream having a video of the face of a user (e.g., video stream125or preprocessed video stream151), (b) the first set of oculometric parameters (obtained at a first resolution as described above), (c) environment data such as lighting in the environment (e.g., a room) where the user is located, which can be measured using information obtained from the camera120, (d) device data such as a display size, display resolution, display brightness, or display contrast associated with a display of the client device106, model and manufacturer information of the camera120, or (e) user information such as demographic, clinical history, optometric data, etc.

The system100may provide such input data to a prediction model to generate predictions (e.g., the second set of oculometric parameters such as iris rotation170, iris translation172, iris radius174, pupil center176, and pupil radius178at a second resolution). System100may provide reference feedback to the prediction model and the prediction model may update one or more portions of the prediction model based on the predictions and the reference feedback. As an example, where the prediction model generates predictions based on the above input data, a second set of oculometric parameters associated with such input data may be provided as reference feedback to the prediction model. As an example, a particular set of oculometric parameters obtained at the second resolution may be verified as an appropriate set of oculometric parameters (e.g., via user confirmation of the set of oculometric parameters, via one or more subsequent actions demonstrating such goal, etc.). The foregoing user input information may be provided as input to the prediction model to cause the prediction model to generate predictions of the second set of oculometric parameters, and the verified set of oculometric parameters may be provided as reference feedback to the prediction model to update the prediction model. In this way, for example, the prediction model may be trained or configured to generate more accurate predictions. In some embodiments, the reference feedback having the oculometric parameters at the second resolution may be obtained, determined or derived from information obtained using any of several eye tracking devices that produce oculometric parameters at a high resolution (e.g., the second resolution). For example, some tracking devices produce oculometric parameters such as gaze origin, gaze point and pupil diameter at the second resolution. The OPS114may derive the second set of oculometric parameters such as the iris rotation, iris translation, iris radius, pupil center, and pupil radius from the oculometric parameters generated using the eye tracking device and provide the derived second set of oculometric parameters as reference feedback to train the prediction model. Such reference feedback may be obtained for several videos and provided as training dataset to train the prediction model.

In some embodiments, the foregoing operations for updating the prediction model may be performed with a training dataset with respect to one or more users (e.g., a training dataset associated with a given user to specifically train or configure the prediction model for the given user, a training dataset associated with a given cluster, demographic, or other group to specifically train or configure the prediction model for the given group, a training dataset associated with any given set of users, or other training dataset). As such, in some embodiments, subsequent to the updating of the prediction model, system100may use the prediction model to facilitate generation of a first set of oculometric parameters.

In some embodiments, the OPS114may also extract additional oculometric parameters such as a pupil visible fraction, a pupil coverage asymmetry, iris visible fraction, or iris coverage asymmetry. In some embodiments, the pupil visible fraction is calculated as the ratio between the pupil area not covered by the eyelids and the pupil iris area. The pupil coverage asymmetry may be defined as the mean of the pupil upper eyelid covered fraction and the pupil lower eyelid covered fraction when the upper eyelid covered fraction is represented in a positive value, and the lower eyelid covered fraction is represented in a negative value, normalized by the iris total covered area. The values of this parameter may vary between −1 and 1 and project the asymmetry between the eyelid coverage of the upper and the lower eyelids (e.g., “−1” may denote that all covered area is covered by the lower eyelid, “1” may denote that all covered area is covered by the upper eyelid, and “0” may denote that the upper and lower eyelids cover equal areas).

The OPS114may extract the additional oculometric parameters based on the second set of oculometric parameters170-178. For example, the OPS114may perform geometrical projections and calculations using the second set of oculometric parameters170-178to determine the additional oculometric parameters.

In some embodiments, the OPS114may also extract, generate or derive the eye movement parameters (e.g., pupillary response parameters and gaze parameters) that are indicative of eye movements responsive to a non-standard stimulus (e.g., a video or image displayed on the client device106). The eye movement parameters may be derived using the second set of oculometric parameters (e.g., obtained as described above). In some embodiments, the pupillary response parameter is indicative of a response of the eye to a particular stimulus, and the gaze parameters are indicative of where the eyes are focused or looking. The OPS114may obtain different types of gaze parameters, such as fixation, saccade, pursuit, or another gaze parameter. In some embodiments, fixation is defined as the distribution of eye movements while inspecting a specific area of the stimulus. In some embodiments, saccade is defined as the distribution of eye movements between inspection areas. In some embodiments, pursuit is defined as the distribution of eye movements while following movement. These eye movement parameters may be used in generating various digital biomarkers.

In some embodiments, the marker subsystem116may produce digital biomarkers (e.g., parameters) that may act as an indicator of one or more neurological disorders. The marker subsystem116may produce the digital biomarkers based on based on the oculometric parameters or the eye movement parameters (e.g., discussed above). In some embodiments, these digital biomarkers are tightly correlated (and can therefore serve as proxies) for disease progression or acuity. The digital biomarkers generated by the marker subsystem116may be objective, sensitive, accurate (0.1 mm or less), or correlated with clinical progression of diseases, and may be obtained remotely, outside of lab settings and even in a distributed manner.

Note that the oculometric parameters, the eye movement parameters, or the digital biomarkers may be obtained in real-time (e.g., based on a real-time video stream obtained from the camera120) or offline (e.g., using a video stored in the database132). Further, the oculometric parameters (e.g., second set of oculometric parameters170-178or other oculometric parameters) are obtained as time-series data and for one or more eyes of the user. That is, the oculometric parameters170-178may be extracted continuously at a specified time interval (e.g., every millisecond, every few milliseconds, or other temporal resolution).

In some embodiments, the prediction models described above may include one or more neural networks or other machine learning models. As an example, neural networks may be based on a large collection of neural units (or artificial neurons). Neural networks may loosely mimic the manner in which a biological brain works (e.g., via large clusters of biological neurons connected by axons). Each neural unit of a neural network may be connected with many other neural units of the neural network. Such connections can be enforcing or inhibitory in their effect on the activation state of connected neural units. In some embodiments, each individual neural unit may have a summation function which combines the values of all its inputs together. In some embodiments, each connection (or the neural unit itself) may have a threshold function such that the signal must surpass the threshold before it propagates to other neural units. These neural network systems may be self-learning and trained, rather than explicitly programmed, and can perform significantly better in certain areas of problem solving, as compared to traditional computer programs. In some embodiments, neural networks may include multiple layers (e.g., where a signal path traverses from front layers to back layers). In some embodiments, back propagation techniques may be utilized by the neural networks, where forward stimulation is used to reset weights on the “front” neural units. In some embodiments, stimulation and inhibition for neural networks may be more free-flowing, with connections interacting in a more chaotic and complex fashion.

A neural network may be trained (i.e., whose parameters are determined) using a set of training data (e.g., ground truths). The training data may include a set of training samples. Each sample may be a pair comprising an input object (typically an image, a measurement, a tensor or vector which may be called a feature tensor or vector) and a desired output value (also called the supervisory signal). A training algorithm analyzes the training data and adjusts the behavior of the neural network by adjusting the parameters (e.g., weights of one or more layers) of the neural network based on the training data. For example, given a set of N training samples of the form {(x1, y1), (x2, y2), . . . (xN, yN)} such that xiis the feature tensor/vector of the i-th example and y, is its supervisory signal, a training algorithm seeks a neural network g: X→Y, where X is the input space and Y is the output space. A feature tensor/vector is an n-dimensional tensor/vector of numerical features that represent some object (e.g., a complex electric field image). The tensor/vector space associated with these vectors is often called the feature or latent space. After training, the neural network may be used for making predictions using new samples.

As an example, with respect toFIG.2, machine learning model202may take inputs204and provide outputs206. In one use case, outputs206may be fed back to machine learning model202as input to train machine learning model202(e.g., alone or in conjunction with user indications of the accuracy of outputs206, labels associated with the inputs, or with other reference feedback information). In another use case, machine learning model202may update its configurations (e.g., weights, biases, or other parameters) based on its assessment of its prediction (e.g., outputs206) and reference feedback information (e.g., user indication of accuracy, reference labels, or other information). In another use case, where machine learning model202is a neural network, connection weights may be adjusted to reconcile differences between the neural network's prediction and the reference feedback. In a further use case, one or more neurons (or nodes) of the neural network may require that their respective errors are sent backward through the neural network to them to facilitate the update process (e.g., backpropagation of error). Updates to the connection weights may, for example, be reflective of the magnitude of error propagated backward after a forward pass has been completed. In this way, for example, the machine learning model202may be trained to generate better predictions.

FIG.4is a flow diagram of a process400for obtaining oculometric parameters at a high resolution, consistent with various embodiments. In some embodiments, the process400may be executed in the system100ofFIG.1A.

In an operation402, a video stream is obtained from a client device associated with a user. The video stream may include a video of a face of the user. For example, the video stream125is obtained from the client device106. In some embodiments, the video stream may be obtained in real-time from the client device, e.g., capture by a camera associated with the client device such as the camera120of the client device106, or may be a recorded video that is obtained from a data storage device, e.g., database132, data storage or a memory on the client device, or other data storage devices.

In an operation404, the video stream is input to a first prediction model to obtain a first set of oculometric parameters. In some embodiments, the video stream may pre-processed prior to being input to the first prediction model. In some embodiments, preprocessing the video stream may include performing an ETL process on the video stream to adjust color and brightness (e.g., white balance) in the video stream, reduce noise from the video stream, improve resolution of the video stream by implementing one or more multi-frame super-resolution techniques, or other such video processing. For example, the video stream input to the first prediction model may be the video stream125or the preprocessed video stream151.

In an operation406, the first set of oculometric parameters are obtained from the first prediction model. The oculometric parameters correspond to various characteristics of an eye. For example, the first set of oculometric parameters may include iris data, eyelid data and pupil center. The iris data may include coordinates of several points that describe a center and a shape (e.g., an ellipse) of the iris. The eyelid data may include coordinates of an array of points that describe a shape (e.g., a polygon) representing the eyelids. The pupil center may include a coordinate of a point representing the center of the pupil. In some embodiments, the first set of oculometric parameters are obtained at a first resolution. For example, the first resolution may be at a pixel level or other lower resolution. In some embodiments, the iris data is further processed to obtain iris rotation, iris translation and iris radius from the iris data. For example, the iris data153is processed to obtain the iris rotation158, iris translation160and iris radius162. The iris rotation may be representative of a rotation of the iris and may be retrieved as the iris normal vector (e.g., perpendicular to the iris circle plain). The iris translation may be a center of the iris and may be represented using coordinates that identify a location of the iris center. The iris radius may be a radius of the iris and may be represented as a scalar with distance units. In some embodiments, an MLE-based curve fitting method may be performed to estimate the iris related parameters. The iris data is provided as input to the MLE-based curve fitting method (e.g., oval fit), which generates the above iris related parameters.

In an operation408, the video stream is deconvolved to adjust (e.g., improve accuracy or resolution) the first set of oculometric parameters. The video stream input to the deconvolution process may be the video stream125or the preprocessed video stream151. In some embodiments, the video stream is deconvolved by using a blind deconvolution process integrated with an MLE process, the additional details of which are described above at least with reference toFIGS.1A and1Band below at least with reference toFIG.5. In some embodiments, applying blind deconvolution algorithm may help in decreasing or removing the blurring in the images/video, or any other noise from the video stream to improve the accuracy of the oculometric parameters extracted. The deconvolution process may be repeated one or more times to obtain the adjusted oculometric parameters (e.g., whose accuracy or resolution is improved compared to the oculometric parameters prior to the deconvolution process). The adjusted parameters may include adjusted eyelid data, adjusted pupil center and pupil radius.

In an operation410, the video stream and the first set of parameters are input to a second prediction model to obtain the oculometric parameters at a high resolution. For example, the first set of oculometric parameters, including the iris rotation158, iris translation160, the iris radius162, the adjusted eyelid data164, adjusted pupil center166and pupil radius168, etc. are input to the second prediction model. In some embodiments, the second prediction model is trained to predict the set of oculometric parameters at a high resolution. For example, the second prediction model is trained with several training datasets in which each dataset includes a video of the face of the user, device data of the client device, environment data of an environment in which the user is located and user information of the user as input data, and the corresponding set of oculometric parameters at a high resolution as the ground truth. In some embodiments, the set of oculometric parameters at the high resolution is obtained using one or more eye tracking devices that are configured to produce the oculometric parameters at a high resolution.

In an operation412, a second set of oculometric parameters are obtained at a second resolution from the second prediction model. For example, the second set of oculometric parameters including iris rotation170, iris translation172, iris radius174, pupil center176, and pupil radius178are obtained at a second resolution (e.g., 0.1 mm, sub-pixel level or some other resolution greater than the first resolution).

Optionally, in an operation414, additional set of oculometric parameters may be obtained. For example, additional oculometric parameters such as a pupil visible fraction, a pupil coverage asymmetry, iris visible fraction, or iris coverage asymmetry may be obtained based on the second set of oculometric parameters. The OPS114may obtain the addition oculometric parameters by performing geometrical projections and calculations using the second set of oculometric parameters. In some embodiments, the pupil visible fraction is calculated as the ratio between the pupil area not covered by the eyelids and the pupil iris area. The pupil coverage asymmetry may be defined as the mean of the pupil upper eyelid covered fraction and the pupil lower eyelid covered fraction when the upper eyelid covered fraction is represented in a positive value, and the lower eyelid covered fraction is represented in a negative value, normalized by the iris total covered area. The values of this parameter may vary between −1 and 1 and project the asymmetry between the eyelid coverage of the upper and the lower eyelids (e.g., “−1” may denote that all covered area is covered by the lower eyelid, “1” may denote that all covered area is covered by the upper eyelid, and “0” may denote that the upper and lower eyelids cover equal areas.

Operations402-414may be performed by a subsystem that is the same as or similar to OPS114, in accordance with one or more embodiments.

FIG.5is a flow diagram of a process500for deconvolving a video stream to obtain adjusted oculometric parameters, consistent with various embodiments. In some embodiments, the process500may be performed as part of operation408of process400.

In an operation502, input data such as the video stream, stimulus data, environment data, device data or other data is obtained to perform deconvolution of the video stream. The video stream input to the deconvolution process may be the video stream125or the preprocessed video stream151. The stimulus data may include spatio-temporal information on a stimulus presented on the display of the client device106, or optical properties information on the of the stimulus, including spectral properties (e.g., color) and intensity (e.g., brightness). The environment data may include information such as lighting in the environment (e.g., a room) in which the user is located, which can be measured using information obtained from the camera120. The device data may include information such as orientation information of the client device106, information from one or more sensors associated with the client device106such as acceleration sensors. Further, the input data may also include eyelid data (e.g., eyelid data155) and the pupil center (e.g., pupil center157).

In an operation504, a point spread function of the blind deconvolution algorithm is determined based on the input data.

In an operation506, the video stream is deconvolved based on the point spread function. The deconvolution process removes or reduces any blurring, or any other noise introduced into the video stream due to user environment related factors, client device related factors, camera related factors, video stimulus related factors etc. to improve the video stream and therefore, improve the accuracy of the oculometric parameters retrieved.

In an operation508, the adjusted oculometric parameters are derived from the deconvolved video stream as described above. For example, the adjusted oculometric parameters may include adjusted eyelid data164, adjusted pupil center166or pupil radius168. In some embodiments, the deconvolution process is integrated with an MLE process to further improve the accuracy of the oculometric parameters. For example, the MLE process may be used to improve the accuracy of the eyelid data155. The MLE process may be used to perform a parabolic curve fitting operation on the eyelid data155to obtain a more accurate representation of the eyelids, as adjusted eyelid data164. In another example, the MLE process may be used to obtain or improve the accuracy of pupil related data, such as pupil radius168. The MLE process may be used to perform a curve fitting operation (e.g., oval fit) to improve a shape of the pupil. For example, the MLE process may consider different pupil centers, and may construct a pupil shape for each of the candidate pupil centers and assign a confidence score for each of the shapes. Such a method may be repeated for different pupil centers, and the shape having a score satisfying a score criterion (e.g., the best score) is selected. Once the shape is selected, the corresponding center may be selected as the adjusted pupil center166and the pupil radius168may be determined based on the selected shape and the adjusted pupil center166.

In some embodiments, the process500may be repeated one or more times (e.g., until the accuracy or resolution of the oculometric parameters satisfy a criterion (e.g., exceed a threshold value) or the PSF satisfies a criterion) to improve the accuracy or resolution of the adjusted oculometric parameters.

In some embodiments, the various computers and subsystems illustrated inFIG.1Amay include one or more computing devices that are programmed to perform the functions described herein. The computing devices may include one or more electronic storages (e.g., prediction database(s)132, which may include training data database(s)134to store training data, model database(s)136to store prediction models, etc., or other electronic storages), one or more physical processors programmed with one or more computer program instructions, and/or other components. The computing devices may include communication lines or ports to enable the exchange of information within a network (e.g., network150) or other computing platforms via wired or wireless techniques (e.g., Ethernet, fiber optics, coaxial cable, Wi-Fi, Bluetooth, near field communication, or other technologies). The computing devices may include a plurality of hardware, software, and/or firmware components operating together. For example, the computing devices may be implemented by a cloud of computing platforms operating together as the computing devices.

The processors may be programmed to provide information processing capabilities in the computing devices. As such, the processors may include one or more of a digital processor, an analog processor, a digital circuit designed to process information, an analog circuit designed to process information, a state machine, and/or other mechanisms for electronically processing information. In some embodiments, the processors may include a plurality of processing units. These processing units may be physically located within the same device, or the processors may represent processing functionality of a plurality of devices operating in coordination. The processors may be programmed to execute computer program instructions to perform functions described herein of subsystems112-116or other subsystems. The processors may be programmed to execute computer program instructions by software; hardware; firmware; some combination of software, hardware, or firmware; and/or other mechanisms for configuring processing capabilities on the processors.

It should be appreciated that the description of the functionality provided by the different subsystems112-116described herein is for illustrative purposes, and is not intended to be limiting, as any of subsystems112-116may provide more or less functionality than is described. For example, one or more of subsystems112-116may be eliminated, and some or all of its functionality may be provided by other ones of subsystems112-116. As another example, additional subsystems may be programmed to perform some or all of the functionality attributed herein to one of subsystems112-116.

FIG.6is a block diagram of a computer system as may be used to implement features of the disclosed embodiments. The computer system600may be used to implement any of the entities, subsystems, components or services depicted in the examples of the foregoing figures (and any other components described in this specification). The computer system600may include one or more central processing units (“processors”)605, memory610, input/output devices625(e.g., keyboard and pointing devices, display devices), storage devices620(e.g., disk drives), and network adapters630(e.g., network interfaces) that are connected to an interconnect615. The interconnect615is illustrated as an abstraction that represents any one or more separate physical buses, point to point connections, or both connected by appropriate bridges, adapters, or controllers. The interconnect615, therefore, may include, for example, a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or industry standard architecture (ISA) bus, a small computer system interface (SCSI) bus, a universal serial bus (USB), IIC (I2C) bus, or an Institute of Electrical and Electronics Components (IEEE) standard 1394 bus, also called “Firewire”.

The memory610and storage devices620are computer-readable storage media that may store instructions that implement at least portions of the described embodiments. In addition, the data structures and message structures may be stored or transmitted via a data transmission medium, such as a signal on a communications link. Various communications links may be used, such as the Internet, a local area network, a wide area network, or a point-to-point dial-up connection. Thus, computer readable media can include computer-readable storage media (e.g., “non-transitory” media) and computer-readable transmission media. The storage media of the electronic storages may include one or both of (i) system storage that is provided integrally (e.g., substantially non-removable) with servers or client devices or (ii) removable storage that is removably connectable to the servers or client devices via, for example, a port (e.g., a USB port, a firewire port, etc.) or a drive (e.g., a disk drive, etc.). The electronic storages may include one or more of optically readable storage media (e.g., optical disks, etc.), magnetically readable storage media (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.), electrical charge-based storage media (e.g., EEPROM, RAM, etc.), solid-state storage media (e.g., flash drive, etc.), and/or other electronically readable storage media. The electronic storages may include one or more virtual storage resources (e.g., cloud storage, a virtual private network, and/or other virtual storage resources). The electronic storage may store software algorithms, information determined by the processors, information obtained from servers, information obtained from client devices, or other information that enables the functionality as described herein.

The instructions stored in memory610can be implemented as software and/or firmware to program the processor(s)605to carry out actions described above. The processors may be programmed to execute computer program instructions by software; hardware; firmware; some combination of software, hardware, or firmware; and/or other mechanisms for configuring processing capabilities on the processors. In some embodiments, such software or firmware may be initially provided to the computer system600by downloading it from a remote system through the computer system600(e.g., via network adapter630).

REMARKS

Those skilled in the art will appreciate that the logic illustrated in each of the flow diagrams discussed above, may be altered in various ways. For example, the order of the logic may be rearranged, substeps may be performed in parallel, illustrated logic may be omitted; other logic may be included, etc.