Patent ID: 12211416

The features of the present disclosure will become more apparent from the detailed description set forth below when takin in conjunction with the drawings, in which like reference characters identify corresponding elements throughout. In the drawings, like reference numbers generally indicate identical, functionally similar, and/or structurally similar elements. Additionally, generally, the left-most digit(s) of a reference number identifies the drawings in which the reference number first appears. Unless otherwise indicated, the drawings provided throughout the disclosure should not be interpreted as to-scale drawings.

DETAILED DESCRIPTION

Aspects of the present disclosure relate to a system for estimating a time delay from a display device (e.g., a monitor output) to an image sensor. In particular, the present disclosure relates to estimating a time delay between when a modulation is presented on the display device and a reception of the modulation by the image sensor.

This specification discloses one or more embodiments that incorporate the features of the present disclosure. The disclosed embodiment(s) are provided as examples. The scope of the present disclosure is not limited to the disclosed embodiment(s). Claimed features are defined by the claims appended hereto.

The embodiment(s) described, and references in the specification to “one embodiment,” “an embodiment,” “an example embodiment,” etc., indicate that the embodiment(s) described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is understood that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.

Spatially relative terms, such as “beneath,” “below,” “lower,” “above,” “on,” “upper” and the like, may be used herein for ease of description to describe one element or feature's relationship to another element(s) or feature(s) as illustrated in the figures. The spatially relative terms are intended to encompass different orientations of the device in use or operation in addition to the orientation depicted in the figures. The apparatus may be otherwise oriented (rotated 90 degrees or at other orientations) and the spatially relative descriptors used herein may likewise be interpreted accordingly.

The term “about,” “approximately,” or the like may be used herein to indicate a value of a quantity that may vary or be found to be within a range of values, based on a particular technology. Based on the particular technology, the terms may indicate a value of a given quantity that is within, for example, 1-20% of the value (e.g., ±1%, ±5% ±10%, ±15%, or ±20% of the value).

Embodiments of the disclosure may be implemented in hardware, firmware, software, or any combination thereof. Embodiments of the disclosure may also be implemented as instructions stored on a machine-readable medium, which may be read and executed by one or more processors. A machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read only memory (ROM); random access memory (RAM); magnetic disk storage media; optical storage media; flash memory devices; electrical, optical, acoustical or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and others. Further, firmware, software, routines, and/or instructions may be described herein as performing certain actions. However, it should be appreciated that such descriptions are merely for convenience and that such actions in fact result from computing devices, processors, controllers, or other devices executing the firmware, software, routines, instructions, etc. In the context of computer storage media, the term “non-transitory” may be used herein to describe all forms of computer readable media, with the sole exception being a transitory, propagating signal.

As noted in the Background section above, neurological disorders may be determined using minute eye movements. Typically, these eye movements are measured in well-controlled lab settings (e.g., no movements, controlled ambient light, or other such parameters) using dedicated devices (e.g., infrared eye trackers, pupilometers, or other such devices). However, setting up and maintaining such a controlled test environment may be extremely costly, and may require a significant amount of time and effort. Furthermore, because there exist only a limited number of such well-controlled lab settings, it may be difficult to schedule appointments and/or travel thereto.

It would be desirable to make eye movement measurements available at low cost by, for example, using the ubiquitous cameras included in smartphones, tablets, laptop computers, desktop computers and the like, to observe the behavior of eyes in response to a visual stimulus. Using the eye movement measurements, digital markers indicative of a neurological condition or a mental health condition may be determined.

Time delays (e.g., lags) between when the visual stimulus is presented to a user (e.g., a subject ongoing testing or monitoring) on a display device and when a reaction of the user is detected by an image sensor may affect the accuracy of the eye movement measurements. For example, a saccadic latency of user may be determined using the eye movement measurements obtained while a gaze response stimulus is shown on the display device. The saccadic latency is measured in a range from about 150 to about 250 milliseconds. In some aspects, a time delay from the display device to a computing system (e.g., an application software) that receives and analyzes images of a face of the user to determine the eye movement measurements may be about 50 to about 100 milliseconds or more with variation of +/− many tens of milliseconds. Additional delays such as network delays or server delays may also affect the accuracy of the eye movement measurements. In addition, a computer processor used for determining the digital markers or the eye movement measurements may have unpredictable delays (e.g., due to other resource-intensive applications or high demand for available memory). These delays cannot be pre-estimated. Thus, it is desired to adjust the eye movement measurements to account for such delays in order to improve the accuracy of the measurements.

Multiple factors may contribute to the time delay. In some aspects, the image sensor and the display device are not synchronized. This may cause a relatively slow frequency drift that contributes to variations in a cumulative processing time. In addition, frames may be occasionally dropped at either the display device or the image sensor. The variations may be due to different types of image sensors and/or display devices used for continuous monitoring of progression of neurological disorders. For example, image sensors may include a wide-range of cameras used in smartphones, tablets, laptops, desktop computers, etc. Further, due to different types of image devices used, the software measuring eye movement may not control an exposure time of the image sensor. In addition, the exposure level of each image frame may not be readily available.

In some aspects, illumination of an environment of the user may vary. For example, illumination in an indoor environment may vary from a softly lit room to a brightly lit room. Indoor light level may vary from about 200 lux to about 2000 lux. Furthermore, illuminance in an indoor environment may change rapidly. For example, lights may be turned on or off, curtains may be opened or closed which cause fast changes in strong light coming through a window, etc.

Embodiments described herein address some or all of the foregoing issues by determining a time delay between when a pattern is sent to a display device and when the effect of that pattern is detected in a video of the user. Embodiments described herein may modulate a brightness level and/or color of the display device and then detect an effect of the modulation in one or more frames of the face of the user. In some aspects, the modulation of the brightness level may affect the illumination of the face of the user. In turn, a brightness measurement in one or more image frames of the user is changed due to the change in the illumination. The brightness measurement may correspond to the brightness of the pixels associated with the face of the user. In some aspects, the brightness measurement may correspond to the brightness of all the pixels of the image frame when the background behind the subject is close, has sufficient reflectance, and is stable. In the framework of communications theory, this modulation may be considered to be a transmission of information through a communications channel to be received and interpreted by the programmed application software. Specifically, the software is configured to detect the earliest moment that a designated change in the brightness modulation arrives at the image sensor. The modulation of the display is selected large enough for good detection and frequent enough to track changing cumulative processing delay, but in one embodiment, is not visible to the user so as to annoy or distract the user.

FIG.1is a block diagram of a system100for estimating a time delay from a display device108to an image sensor106, according to some embodiments. System100may include a computing system102, image sensor106, and display device108.

Display device108may be a device that is capable of rendering images generated or acquired by computing system102such that a user104may visually perceive them. Display device108may include a display screen integrated with computing system102(e.g., an integrated display of smartphone, a tablet computer, or a laptop computer) or a monitor separated from but communicatively coupled to computing system102(e.g., a monitor connected to a desktop computer via a wired connection) or a projector system (e.g., a projection screen and a projector comprising a light source). Display device108may also comprise display panels of a standalone or tethered extended reality headset. In one example, display device108may be a color display monitor with a display rate of 60 frames per second (FPS). In some aspects, display device108may display a plurality of frames generated by computing system102. In some aspects, the plurality of frames may be received by display device108via a network.

The network may be a telecommunications network, such as a wired or wireless network. The network can span and represent a variety of networks and network topologies. For example, the network can include wireless communication, wired communication, optical communication, ultrasonic communication, or a combination thereof. For example, satellite communication, cellular communication, Bluetooth, Infrared Data Association standard (IrDA), wireless fidelity (WiFi), and worldwide interoperability for microwave access (WiMAX) are examples of wireless communication that may be included in the network. Cable, Ethernet, digital subscriber line (DSL), fiber optic lines, fiber to the home (FTTH), and plain old telephone service (POTS) are examples of wired communication that may be included in the network. Further, the network can traverse a number of topologies and distances. For example, the network can include a direct connection, personal area network (PAN), local area network (LAN), metropolitan area network (MAN), wide area network (WAN), or a combination thereof.

In some aspects, image sensor106may be an optical device that is capable of capturing and storing images and videos. Image sensor106may comprise, for example, a digital camera that captures images and videos via an electronic image sensor. Image sensor106may be integrated with computing system102(e.g., an integrated camera of a smartphone, a tablet computer, or a laptop computer) or part of a device that is separate from but communicatively coupled to computing system102(e.g., a USB camera or webcam connected to a desktop computer via a wired connection). In one example, image sensor106may be a video camera. The video camera may operate at a rate of 60 FPS. In some aspects, image sensor106may transmit the captured images or videos via the network.

As described previously herein, computing system102may be a mobile device, a laptop computer, or a desktop computer. In some aspects, computing system102may operate on one or more servers and/or databases. The servers may be a variety of centralized or decentralized computing devices. For example, a server may be grid-computing resources, a virtualized computing resource, peer-to-peer distributed computing devices, or a combination thereof. The servers may be centralized in a single room, distributed across different rooms, distributed across different geographic locations, or embedded within the network. In some embodiments, computing system102may be implemented using computer system700described with reference toFIG.7. Computing system102may provide a cluster computing platform or a cloud computing platform to perform eye movement measurements based on image frames received or acquired from image sensor106. Computing system102may determine a time delay and adjust the eye movement measurements based on the time delay.

Computing system102may generate and send a plurality of frames to display device108. Computing system102may store a plurality of transmit timestamps. Each transmit timestamp may correspond to when a respective frame is transmitted. In some aspects, computing system102may also transmit display brightness data corresponding to the plurality of frames.

Image sensor106may record a video of user104while the plurality of frames are displayed on display device108. Computing system102may acquire the video from image sensor106. Image sensor106may include a plurality of image frames. Computing system102may also determine and store a plurality of receive timestamps. The plurality of receive timestamps may correspond to a time when each image frame of the video is received by computing system102.

User104may be a person interacting with computing system102. In one embodiment, user104may be the person or the subject undergoing oculometric testing or monitoring. The testing may include determining an oculomotor ability of user104. The oculometric may include a test to determine the saccadic latency of the user. These are only examples and other types of oculometric tests may be applied to user104. In some aspects, user104may be a person interacting with a virtual reality system (e.g., playing a virtual reality game) where a reaction of the user to a stimulus is determined and used to control one or more parameters.

As discussed above, there may be a large variation in cumulative processing times in generating the frames to display device108and in receiving the image frames from image sensor106. This large variation results from variation in multiple stages of software and hardware processing. For example, computing system102may send frames (display frames) to display device108. As discussed above, computing system102may record a transmit timestamp for each frame. After receiving the frames, a driver (e.g., monitor driver) of display device108may control display device108to show the frames. Image sensor106may integrate light reflected from a face of user104. Image sensor106may transfer the captured images to computing system102. In some aspects, image sensor106may transmit pixel values via an image sensor interface (e.g., mobile industry processor interface (MIPI)) to a USB controller. The USB controller may be associated with computing system102. The USB controller may transmit pixel values through a USB interface to a memory of computing system102. The pixels may be transmitted in a densely packed format via direct memory access (DMA) to the memory of computing system102. Computing system102may process the pixel values. For example, computing system102may reformat the pixels. In some aspects, the pixel values may be stored in a first in first out (FIFO) buffer in a main memory of computing system102. Computing system102may read the pixel values for the image frames from the FIFO buffer and associate a corresponding receive timestamp for each frame.

In some aspects, computing system102may include a time delay determination module110and a synchronization module112. Time delay determination module110and synchronization module112may be implemented by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, programmable logic, microcode), software (e.g., instructions executed by one or more processors of computing system102), or a combination thereof.

In order to determine the time delay, time delay determination module110may change the brightness level in the plurality of frames. Simultaneously to changing the brightness level in the frames, image sensor106may capture a face of user104. Time delay determination module110may analyze the image frames received from image sensor106to determine a time delay between when the brightness level is changed and when the change in the brightness level is reported by image sensor106.

In order to determine when the brightness level is reported by image sensor106, time delay determination module110may determine a brightness measurement for pixels that correspond to the face of user104. In order to identify the pixels that correspond to the face of user104, time delay determination module110may implement one or more face recognition or detection techniques. Face recognition and detection techniques may include traditional face detection (e.g., elastic graph matching, singular value decomposition (SVD), Viola-Jones), artificial neural network (e.g., deep convolutional neural network), or three dimensional (3D) face recognition techniques.

In some aspects, a driver of display device108may control the brightness based on the brightness data received with or included with the plurality of frames. For example in a projection system, a driver of a light source (e.g., lamp) can control the brightness of an illumination light emitted from the light source or the brightness of a projection image by adjusting the lamp power. In some aspects, brightness data may include signals to control drivers of pixels or subpixels (e.g., red, green, and blue) of display device108.

In some aspects, the time delay determined by time delay determination module110may include a network delay and/or a server delay. This provides the advantage of improving accuracy and reducing computing inefficiencies (e.g., the network and/or server delays are not calculated separately). The network delay may represent delays from receiving and transmitting data to a cloud server when computing system102is implemented as a server in a cloud computing network that receives image frames from image sensor106and transmit frames to display device108. The delays may include a connection establishment delay, a round trip network delay, a data transfer time, and the like. The server delay may include a connection time, a window exhaustion metric, and the like.

As described previously herein, computing system102may be used to determine an oculomotor ability of user104. Computing system102may transmit to display device108one or more frames corresponding to a visual stimulus. For example, user104may be told to gaze directly at a fixation point on the display as the display is instantaneously switched from the first fixation point to the second (referred to as a gaze response stimulus or a saccade test) as described in relation toFIGS.2A and2B. Image sensor106may capture an image of the face of the user104while the plurality of frames are displayed on display device108. Further, computing system102may extract an eye region from the image of the face of user104. Computing system102may analyze the eye region to determine one or more eye movement measurements or other oculometric parameters. Computing system102may determine a saccadic latency based on the eye movement measurements. The saccadic latency is the time from the presentation of the second point of fixation to the start of the saccade. Based on the saccadic latency and/or other oculometric parameters, computing system102may determine one or more digital markers that may be indicative of a neurological condition or a mental health condition of user104.

Synchronization module112may modify one or more oculometric parameters to account for the time delay. The time delay may be received from the time delay determination module110. For example, one or more oculometric parameters or other eye tracking data may be stored as a time series. A time associated with each parameter may be shifted by the time delay (i.e., x milliseconds) to compensate for the lag or time delay.

FIG.2Aillustrates a gaze response stimulus displayed on display device108at the start of the gaze tracking test such as a saccade test, according to some embodiments.FIG.2Billustrates the gaze response stimulus displayed on display device108at the end of the gaze tracking test, according to some embodiments. A saccade is a rapid movement of the eye between two fixation points. The saccade test measures the ability of a subject to move the eye (or eyes) from one fixation point to another in a single, quick movement. In some aspects, the gaze response test may comprise displaying a first image on display device108. The first image may include a target202(e.g., a dot) on a background204. The background can be a solid background (a solid uniform color). Target202may have a display attribute different from background204, for example, a different color or intensity. Target202may be displayed at a first position. In a second image displayed on display device108, target202can be at a second position as shown inFIG.2B. Target202may move along a vertical direction and/or a horizontal direction. Target202may be moved from left to right, from right to left, from up to down, from down to up, or in a diagonal direction. In some aspects, target202may move in a circle. Target202may move at different speeds in a smooth movement or abruptly. The movement direction, speed, and other attributes of the gaze tracking test may be selected based on the desired oculometric parameters. As described above, a brightness level of the background may be modified. For example, the background may have a solid gray color. The brightness level of the gray background may be changed from a white background (100% brightness) to a black background (0% brightness).

FIG.3Ais a schematic that shows a modulation of a brightness level in a plurality of frames, according to some embodiments. Graph306shows the brightness level of display device108. As shown by graph306, the brightness level may be modulated with respect to time (e.g., changed for each displayed frame). The brightness level may be changed around a nominal value N. The change in the brightness level in one or more frames of the plurality of frames is referred to herein as the modulation. The nominal value N may represent a solid background (e.g., gray level) without modulation. In some aspects, a first limit308may represent a brightness level equals to twice the nominal value (e.g., may be referred to as +100%). A second limit310may correspond to a pure black background (may be referred to as −100%). In some aspects, the brightness level may be changed to a first brightness level (e.g., an upper limit) between the first limit308and the second limit310. For example, the brightness level may be changed to +50% (as shown by graph306) of the nominal value for a first period (e.g., for 3 frames). Then, the brightness level is changed to a second brightness level below the nominal value. For example, the brightness level may be changed to −50% of the nominal value for a second period (e.g., for 3 frames).

A modulation percentage may be based on a noise level in the received image frames. For example, if the noise level in the image frames is low then the modulation percentage may be decreased (e.g., the brightness level may be modulated between +/−20% of the nominal value). In some aspects, if the noise level in the image frames is high then the modulation percentage is increased (e.g., the brightness level may be modulated between +/−70% of the nominal value).

In some aspects, the modulation may be for a first set number of frames at the first brightness level followed by a second set number of frame at the second brightness level. In other aspects, the modulation may be for a first set of frames at the second brightness level followed by a second set of frames at the first brightness limit. In some aspects, the first set of frames and the second set of frames may be consecutive. In some aspects, the first set of frames and the second set of frames may include the same number of frames. Thus, the average of the modulation is 0%. In some aspects, the first set of frames may include three frames. In some aspects, the second set of frames may include three frames.

For a display device operating at a rate of 60 FPS, each frame duration may be 16.7 milliseconds. Thus, 3 frames or about 16.7×3=50.1 milliseconds at the first brightness level are followed by three frames or about 50.1 milliseconds at the second brightness level.

In some aspects, the number of frames in each set may be selected such that the modulation is not perceived by user104. In addition, to minimize perception or distraction additional techniques may be implemented as described further below.

FIG.3Bis a schematic that shows brightness measurements for a plurality of image frames, according to some embodiments. The brightness measurements may be sampled at a time TCin each image frame resulting from the transmission shown inFIG.3A. As described above a brightness measurement corresponding to the image frames may change due to the change in the brightness level in one or more frames of the plurality of frames presented at display device108. In some aspects, the brightness measurement may correspond to an average of pixel values of the image frame. In some aspects, the average of pixel values of the pixels corresponding to the face of the user is determined. As shown by graph302, the brightness measurement changes around a received nominal brightness value318. The received nominal brightness value318may be determined based on the brightness measurements corresponding to a plurality of image frames. For example, an average of brightness measurements for a particular number of frames may be determined (e.g., 60 frames). The received nominal brightness value318may represent the brightness measurement when the displayed frame has a brightness level equal to the nominal value N. For convenience of calculation, the vertical scales inFIG.5may be chosen so that the current received nominal brightness value318is aligned with zero. Note that the horizontal axis (time) inFIG.3Bcorresponds to the horizontal axis inFIG.3A.

The brightness measurements may depend on a position of user104(e.g., distance from display device108), reflectance pattern of the user's face, and settings of image sensor106(e.g., exposure). Exposure setting of image sensor106may be controlled to keep the brightness scale reasonably stable. Each brightness value312in graph302is the calculated face brightness (mean brightness of the face pixels) minus the received nominal value318inFIG.3B.

The received nominal value318may be affected by a plurality of factors (e.g., lighting level in the environment of the user). Thus, computing system102may update the value of received nominal value318(e.g., continuously or periodically). Computing system102may calculate the mean of multiple samples of face brightness. For example, a FIFO queue of 60 samples may be collected in the one second before a first modulation pulse is detected. The received nominal value318may correspond to the mean of these 60 samples. For every calculation of mean to set the received nominal value318, a corresponding standard deviation S is calculated. An upper bound (UB)314of noise and a lower bound (LB) of noise may be determined as a function of the standard deviation. In some aspects, UB314is set to three times the standard deviation S. LB316may be set to minus three times the standard deviation. In some aspects, the measurement of the received nominal value318may be updated by pushing each subsequent face brightness measurement that is less than UB314and greater than LB316into the FIFO queue and recalculating. Each brightness value312that is greater than or equal to UB corresponds to the highest brightness indicated in graph306. Each brightness value312that is less than or equal to LB corresponds to the lowest brightness indicated in graph306. Using the brightness measurements corresponding to the image frames, the time delay may be determined as further described below.

In some aspects, image sensor106may employ a rolling shutter mechanism. Exposure of each row of pixels of image sensor106is slightly delayed from the exposure of the preceding row of pixels. Determination of the time delay using image frames acquired from an image sensor that employ a rolling shutter mechanism is discussed below.

FIG.4Ais a schematic that illustrates rows in a rolling shutter image sensor operating at full exposure time, according to some embodiments.FIG.4Bis a schematic that illustrates rows in the rolling shutter image sensor operating at half exposure time, according to some embodiments.FIG.4Cis a schematic that illustrates rows in a rolling shutter image sensor operating at short exposure time, according to some embodiments.

FIGS.4A-4Cshow a first image frame406aand a second image frame406b. As shown inFIGS.4A-4C, the readout proceeds from a row to another in a sequential manner from top to bottom with no overlap in the readout time of different rows. Each row408is slightly delayed with respect to a preceding row.

As discussed above, some of the pixels are associated with the face of user104. In the examples shown inFIGS.4A-4C, the middle rows (e.g., 25% to 75%) may correspond to the face of user104. InFIGS.4A-4C, rows corresponding to the face of the user are shown in a shaded pattern. Each row410shows a reset time408a, an exposure time408b, and a readout time408c.

FIGS.4A-4Cshow rows associated with the face of user104in a horizontal direction. However, it is understood that the pixels may also be in a vertical direction due to the orientation of the image sensor (e.g., a 90 degrees change in the orientation of image sensor106).

As discussed above, a receive timestamp may be associated with each image frame. The receive timestamp is the time at which the readout of the corresponding frame is completed. A first receive timestamp RT1 may correspond to the time at which the readout of first image frame406ais completed. A second receive timestamp RT2 corresponds to the time at which the readout of second image frame406bis completed.

A mean brightness value of the face pixels (face brightness) is determined for each image frame. The sampling of face brightness is done during the time indicted by the shaded portion of row408inFIGS.4A-4C. Each face brightness measurement may be located in time by the time coordinate of a centroid of the shaded area that produced it. The centroid is shown as label C inFIGS.4A-4C.

The time coordinates of the centroids precede the corresponding receive timestamps by an offset. The offset may depend on: (1) a percentage of a frame period from the end of the first row readout to the end of the last row readout, (2) a proportion of exposure, and (3) a vertical position of the face (a horizontal position if the sensor rows are striped vertically across the subject's face). In some aspects, the face of the user may be assumed to be centered in the image frame. Variations in the offset due to a position of the face may be neglected. In other aspects, a position of the face may be determined using face recognition techniques. An adjusted offset for each of the image frame may be determined based on the determined position of the face.

In some aspects, a time coordinate (TC) of the centroid may be determined as follows:

TC=TRX-(0.5×O×FP)-(0.5×E×FP)=TRX-(0.5×FP)×(O+E)
where TRXis the receive timestamp, O is the overlap (i.e., the percentage of a frame period from an end of the first row readout to an end of the last row readout), E is an exposure value, and FP is a frame period (e.g., 16.7 milliseconds when image sensor106is operating at 60 FPS). The overlap O and the exposure E may be dependent on a type and/or a mode of operation of image sensor106. In some aspects, the overlap O and/or the exposure E may not be readily known. For example, the value of overlap O in a particular camera sensor design depends on the speed of the shared readout circuitry. The largest overlap can approach 100% when the readout of the last row is completed just before it is switched to read out the first row of the next frame. The smallest overlap may correspond to a faster readout circuitry. Cost and radio frequency (RF) emission considerations push chip designers not to make circuitry any faster than it has to be. For image sensors that are commonly used (e.g., in laptops, smartphones), an overlap equals to 0.9 may be used. However, the overlap may range from about 0.8 to about 1.0.

Exposure values E may range from about 0.1 to about 1.0. Since a good exposure control with typical ambient illumination is near the middle of the range, the exposure E may be assumed to be equal to 0.55. Thus, the selected value of TCmay be expressed as: TC=TRX−(0.5×FP)×(0.9+0.55)=TRX−0.725×FP. The earliest value of TCis TC=TRX−(0.5×FP)×(1.0+1.0)=TRX−1.0×FP. The latest value of TCis TC=TRX−(0.5×FP)×(0.8+0.1)=TRX−0.45×FP.

In some aspects, image sensor106may employ a global shutter mechanism in which all the pixels of image sensor106are exposed at once. The time coordinate (TC) of the centroid may also be determined using the technique described above by using an overlap equal to zero.

Referring back toFIG.3B, the received nominal value318may be used to detect fast changes in indoor illuminance. If the received nominal value changes too much and/or too fast, an ongoing measurement of eye behavior may be aborted and restarted.

The time from commanding display device108to change a brightness level (TTX) to the earliest detection of the brightness change at image sensor106is determined. In particular, the earliest time at which a large down transition in the middle of each up/down pulse pair in the modulation begins to affect image sensor106, TEDis detected. Thus, the time delay between a monitor display pattern sent by the software and the appearance of the effect of that pattern in the video of the subject received by the software is TD=TED−TTX.

The face brightness samples of an up/down pulse pair by looking for two consecutive samples above UB314is determined. All the succeeding samples until a sample below LB316is followed by a sample above LB316are added to this sequence of two samples. The last sample above LB316is not used. The rest of the typically six samples in the sequence are fitted to a cubic polynomial.

In some aspects, TEDis determined by fitting the face brightness samples of each up/down pulse pair to a cubic polynomial as illustrated inFIG.5(e.g., by regression analysis using the method of least-squares).

FIG.5is a schematic500that shows fitted brightness measurements, according to some embodiments. Graph504shows the brightness measurements. Graph502shows the fitted data. The horizontal axis is in milliseconds relative to TCof the first sample. The vertical axis may correspond to the vertical axis ofFIG.3Bbut with a specific example of scale. The time coordinate of the central zero crossing of this cubic polynomial is TED. From the timestamp TTXof the first frame of the transmitted down pulse, the time delay TD=TED−TTXis calculated. This is done for every up/down pulse pair.

In some aspects, the techniques for determining the time delay TEDdescribed herein have the advantage of being robust against errors. For example, the time delay may be determined even when one or more frames are occasionally dropped (e.g., a frame is not displayed by display device108and/or an image frame is not transmitted by image sensor106). This is because four samples are sufficient to specify a cubic polynomial. A useful value for TEDis obtained even when 1 or 2 frames are dropped. Using a higher number of frames (e.g., 5 or 6) provides the advantage of more resistance to variations caused by noise in the image frames.

By way of further illustration, the following is one example algorithm by which time delay determination module110may determine a time delay in accordance with some embodiments.

1. Use continuous face detection. For each camera frame, if there is face detection andsegmentation with high confidence, then mark FACE = true, else mark FACE = false.2. Start test sequence with background set to TX Nominal and set ACTIVE = true. At endof test sequence, set ACTIVE = false.3. Initialize TD= 0.4. Setup empty TX FIFO queue to save TTXof every large down transition.5. If ACTIVE and FACE:a. Set up 60-element FIFO queue for computation of RX Nominal, collect 60 facebrightness samples into the queue, and compute initial values of RX Nominal, UB, andLB.6. While ACTIVE and FACE:a. Set the brightness of the gray monitor background for 3 frame times to +50%from TX Nominal.b. Set the brightness of the gray monitor background for next 3 frame times to −50% from TX Nominal. Push TTXof the first of these frames into TX FIFO.c. Set the brightness of the gray monitor background for next 3 frame times to TXNominal.7. While ACTIVE and FACE:a. Collect brightness value, B. [B is mean brightness of the face pixels minus RXNominal]b. If LB < B < UB:i. Push mean brightness of the face pixels into RX Nominal FIFO queue andrecompute RX Nominal, UB, and LB.c. If B >= UB:i. If next face brightness sample is >= UB:a. Save this and preceding sample with corresponding TC's in asequential list.b. Collect each succeeding B sample with its corresponding TCandadd it to the list until (latest sample in list <= LB and next sample > LB) or(length of list > 6)c. If (latest sample in list <= LB and next sample > LB):i. Use sequential list to compute TED.ii. Update TD= TED− TTX, where TTXis pulled from TXFIFO.

As noted in step 1 of the algorithm, time delay determination module110continuously applies a face detection algorithm to each frame of the video acquired by image sensor106to find a face with each frame. If time delay determination module110determines that a face is detected with a high confidence, it will set the variable FACE to true for that frame, otherwise it will set the variable FACE to false for that frame. Any of a wide variety of well-known techniques for performing face detection with respect to an image may be used to implement the foregoing aspects of the example algorithm.

As noted in step 2 of the algorithm, time delay determination module110starts a test sequence (e.g., presenting user104with a plurality of frames that include a visual stimulus) with a brightness of the background set to a TX Nominal (corresponding to N inFIG.3A) at which point the variable ACTIVE is set to true. At the end of the test sequence, time delay determination module110may set the variable ACTIVE to false.

As noted in step 3 of the algorithm, time delay determination module110may initialize variable TD. For example, time delay determination module110may set the variable TDto zero.

As noted in step 4, time delay determination module110sets up an empty TX FIFO queue. Time delay determination module110utilizes TX FIFO to store time TTXthat corresponds to the time of every large down transition.

Step 5 of the algorithm describes a process to determine the initial values of RX Nominal (corresponding to318inFIG.3B), UB, and LB. For example, as shown in the algorithm if variables ACTIVE and FACE are true, time delay determination module110may set up a 60-element FIFO queue. The FIFO queue may be used to collect brightness measurements. Time delay determination module110determines 60 brightness measurements and stores the measurement in the 60-element FIFO queue. Time delay determination module110determines the initial values of RX Nominal, UB, and LB. For example, time delay determination module110determines a mean of the 60 brightness measurements and sets the value of RX Nominal to the mean. In addition, time delay determination module110determines the values of UB and LB as a function of RX Nominal.

In this embodiment, steps 6 and 7 operate in parallel.

Step 6 of the algorithm describes a process that controls the brightness level of the display device. For example, as shown in the algorithm, while ACTIVE and FACE are true, a brightness level of a background of display device108is set to +50% from TX Nominal for a particular duration. In this example case, the duration is 3 frames. As further noted in step 6, time delay determination module110sets the brightness level of a background of display device108to −50% from Tx Nominal for the particular duration (e.g., 3 frames). Time delay determination module110pushes a time TTXcorresponding to the first frame of −50% brightness into TX FIFO. Then, time delay determination module110sets the brightness level of the background to TX Nominal for the particular duration (e.g., 3 frames).

Step 7 of the algorithm describes a process of collecting the brightness measurements and determining the time delay based on the collected brightness measurements. As noted in step 7 of the algorithm, while ACTIVE and FACE are true, time delay determination module110may collect a brightness value B. In this example, the brightness value is the mean of the face pixels minus the value of RX Nominal. If the brightness value B is greater than LB and less than UB, the mean brightness of the face pixels is pushed into the 60-element FIFO queue. Then, time delay determination module110computes new values for RX Nominal, UB, and LB.

As noted in step 7 of the algorithm, if the brightness value B is greater than or equal to UB, then time delay determination module110monitors the brightness value B of the subsequent frame. If the brightness value B of the subsequent frame is greater than or equal to UB, the brightness values B and the corresponding Tcfor both frames are stored in a sequential list. As noted in step 7, time delay determination module110collects each succeeding brightness value B corresponding to subsequent frames with their corresponding Tc. The collected brightness values and the corresponding Tcare stored to the sequential list until one or more criteria are satisfied. For example, time delay determination module110may stop collecting brightness values B when the brightness value of the last sample is less or equal to LB and the brightness value of the subsequent sample is greater than LB. Time delay determination module110may stop collecting brightness values B if a length of the sequential list exceeds a threshold. In this example, the threshold is six. If the brightness value of the last sample is less or equal to LB and the brightness value of the subsequent sample is greater than LB, then time delay determination module110determines TEDusing the sequential list. Then, the time delay is determined or updated. The time delay TDis equal to TEDminus TTX. Time delay determination module110pulls TTXfrom TX FIFO.

In some embodiments, one or more of the steps may be executed in parallel.

FIG.6is an example method for estimating a delay from a monitor output to a sensor, in accordance with an embodiment of the present disclosure. Method600may be performed as a series of steps by a computing unit such as a processor. For example, method600may be implemented by computing system102and/or computer system700ofFIG.7. It is to be appreciated that not all steps may be needed to perform the disclosure provided herein. Further, some of the steps may be performed simultaneously, or in a different order than shown inFIG.6, as will be understood by one of ordinary skill in the art.

Method600shall be described with reference toFIG.1, however, method600is not limited to that example embodiment.

At602, computing system102may present a plurality of frames on a display device.

At604, computing system102may modulate a brightness level in one or more frames of the plurality of frames.

At606, computing system102may receive a plurality of image frames of a face of the user. The plurality of image frames is obtained from a video stream recorded by an image sensor during the presentation of the plurality of frames on the display device.

At608, computing system102may process the plurality of image frames to determine a time delay between the modulating the brightness level is performed in the one or more frames of the plurality of frames and when the modulating the brightness level in the one or more frames of the plurality of frames is reported by the image sensor.

In some aspects, computing system102may identify a set of pixels in an image frame of the plurality of image frames. The set of pixels may correspond to the face of the user. Computing system102may determine a brightness measurement for the set of pixels. In some aspects, the brightness measurement corresponds to a mean of brightness values of each pixel of the set of pixels. Each brightness measurement is determined at a respective brightness measurement time.

In some aspects, computing system102may identify a first set of image frames from the plurality of image frames. The first set of image frames is identified such as a brightness measurement corresponding to each image frame included in the first set exceeds an upper threshold. The upper threshold is greater than a nominal brightness level. Computing system102may identify a second set of image frames from the plurality of image frames. The second set of image frames may be identified such that a brightness measurement correspond to each image frame in the second set is less than a lower threshold. The lower threshold is below the nominal brightness level. Image frames of the first set and the second set are consecutive frames. The first set of image frames corresponds to a first set of frames in which the brightness level is changed to a first brightness level. The second set of image frames corresponds to a second set of frames in which the brightness level is changed to a second brightness level lower than the lower threshold.

In some aspects, computing system102may determine a transition detection time based on at least respective brightness measurement times for frames of the first set of image frames and the second set of image frames. Computing system102may also determine a time delay as a function of the transition detection time and a modulation time. The modulation time corresponds to a time at which the modulating the brightness level is performed in the one or more frames of the plurality of frames. In some aspects, the time delay is a difference between the transition detection time and the modulation time. The transition detection time corresponds to a time at which a change in a brightness level from the first brightness level to the second brightness level is detected by the image sensor. In some aspects, the brightness measurement time is determined based on at least one of a receive timestamp of the received frame, an overlap of frame period, an exposure value of the image sensor, or a duration of the image frame.

In some aspects, computing system102may adjust a parameter based on the time delay. The parameter is associated with a response of the user to a stimulus presented in the plurality of frames. For example, the parameter may be an oculometric parameter. The oculometric parameter may be determined from eye data obtained while the stimulus is presented on the display device. For example, a time component of an oculometric parameter may be shifted to compensate for the time delay (e.g., t−x, where x is the determined time delay). Thus, the accuracy of the oculometric parameter is improved. In addition, one or more digital markers indicative of a neurological condition or a mental health condition of the user may be obtained from the oculometric parameters.

In some aspects, display device108may include an array of individually controllable pixels associated with a plurality of colors. For example, display device108can include a triad (red, green, blue) or other combination of color components. Each color component for each pixel is typically made up of multiple sub-pixels. As used herein the term “pixel” is used to describe the triad of sub-pixels. In some aspects, display device108may include multiple light emitting diodes (LEDs). Each LED is a light source that can be either a white LED or on of a variety of colors such as red, green, or blue. In some aspects, a brightness of pixels corresponding to a first color of the plurality of colors may be controlled individually. In some aspects, light source drivers coupled to a LED or to a group of LEDs can control brightness signaling for the LED or the group of LEDs. The brightness may be controlled via analog current or voltage techniques, or via a digital pulse width modulation (PWM).

In some aspects, the red component of pixels may be changed (i.e., modulated as described previously herein) without changing the green or blue component. For example, the brightness of the pixels associated with the red component is modulated between −50% to +50%.

In some aspects, modulating the brightness of the red component of display pixels provides advantages. Red light reflects much better from dark skin tan compared to green or blue light as described in “Quantitative Analysis of Skin using Diffuse Reflectance for Non-invasive Pigments Detection” by Li et al. This provides the advantage of increasing a signal-to-noise ratio for detecting the brightness modulation using the image frames (i.e., based on reflection from the face of the user). In addition, modulating the brightness the red pixel component is less noticeable to the user compared to grey modulation. This is due to the color sensitivity of the human eye. Thus, a larger amplitude of modulation may be used without disturbing the user, and the signal-to-noise ratio is increased.

FIG.7shows a computer system700, according to some embodiments. Various embodiments and components therein can be implemented, for example, using computer system700or any other well-known computer systems. For example, the method steps ofFIG.6may be implemented via computer system700.

In some aspects, computer system700may comprise one or more processors (also called central processing units, or CPUs), such as a processor704. Processor704may be connected to a communication infrastructure or bus706.

In some aspects, one or more processors704may each be a graphics processing unit (GPU). In an embodiment, a GPU is a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

In some aspects, computer system700may further comprise user input/output device(s)703, such as monitors, keyboards, pointing devices, etc., that communicate with communication infrastructure706through user input/output interface(s)702. Computer system700may further comprise a main or primary memory708, such as random access memory (RAM). Main memory708may comprise one or more levels of cache. Main memory708has stored therein control logic (e.g., computer software) and/or data.

In some aspects, computer system700may further comprise one or more secondary storage devices or memory710. Secondary memory710may comprise, for example, a hard disk drive712and/or a removable storage device or drive714. Removable storage drive714may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive. Removable storage drive714may interact with a removable storage unit718. Removable storage unit718may comprise a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit718may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive714reads from and/or writes to removable storage unit718in a well-known manner.

In some aspects, secondary memory710may comprise other means, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system700. Such means, instrumentalities or other approaches may comprise, for example, a removable storage unit722and an interface720. Examples of the removable storage unit722and the interface720may comprise a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

In some aspects, computer system700may further comprise a communication or network interface724. Communication interface724enables computer system700to communicate and interact with any combination of remote devices, remote networks, remote entities, etc. (individually and collectively referenced by reference number728). For example, communication interface724may allow computer system700to communicate with remote devices728over communications path726, which may be wired and/or wireless, and which may comprise any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system700via communications path726.

In some aspects, a non-transitory, tangible apparatus or article of manufacture comprising a non-transitory, tangible computer useable or readable medium having control logic (software) stored thereon is also referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system700, main memory708, secondary memory710, and removable storage units718and722, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system700), causes such data processing devices to operate as described herein.

Based on the teachings contained in this disclosure, it will be apparent to those skilled in the relevant art(s) how to make and use embodiments of this disclosure using data processing devices, computer systems and/or computer architectures other than that shown inFIG.7. In particular, embodiments may operate with software, hardware, and/or operating system implementations other than those described herein.

It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present disclosure is to be interpreted by those skilled in relevant art(s) in light of the teachings herein.

It is to be appreciated that the Detailed Description section, and not the Summary and Abstract sections, is intended to be used to interpret the claims. The Summary and Abstract sections may set forth one or more but not all exemplary embodiments of the present disclosure as contemplated by the inventor(s), and thus, are not intended to limit the present disclosure and the appended claims in any way.

The present disclosure has been described above with the aid of functional building blocks illustrating the implementation of specified functions and relationships thereof. The boundaries of these functional building blocks have been arbitrarily defined herein for the convenience of the description. Alternate boundaries can be defined so long as the specified functions and relationships thereof are appropriately performed.

While specific embodiments of the disclosure have been described above, it will be appreciated that embodiments of the present disclosure may be practiced otherwise than as described. The descriptions are intended to be illustrative, not limiting. Thus, it will be apparent to one skilled in the art that modifications may be made to the disclosure as described without departing from the scope of the claims set out below.

The foregoing description of the specific embodiments will so fully reveal the general nature of the present disclosure that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present disclosure. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein.

The breadth and scope of the protected subject matter should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims and their equivalents.