Two-eye tracking based on measurements from a pair of electronic contact lenses

A system includes a pair of electronic contact lenses that obtain respective motion sensor measurements in response to eye movements. A tracking module derives estimated orientations for both eyes based on the sensor measurements and a set of correlations and constraints that describe human eye movement. The model describes the limited number of ways that an individual eye can move and relationships between relative movement of the left and right eye. The tracking module performs filtering based on the measurements and the eye model to suppress noise and generate orientation estimates for both eyes.

BACKGROUND

1. Technical Field

This disclosure relates generally to eye tracking using measurements from a pair of electronic contact lenses.

2. Description of Related Art

An electronic contact lens may include various integrated electronic components such as projectors, imaging devices, sensors, and batteries. These electronic contact lenses can be utilized for various virtual reality or augmented reality applications in which images are projected by the electronic lenses onto the user's retinas to replace or augment the user's view of the external environment. Integrated sensors in such electronic contact lenses may furthermore measure motion data associated with eye movements. If estimated accurately, this motion data is useful to track estimated eye position in order to determine where to position projected images or to perform other functions.

DETAILED DESCRIPTION

A system includes a pair of electronic contact lenses that obtain respective motion sensor measurements in response to eye movements. A tracking module derives estimated orientations for both eyes based on the sensor measurements and a set of correlations and constraints that describe human eye movement. This model accounts for the principles that an individual eye can move in only a limited number of ways and that left and right eye motions are not independent. The tracking module is therefore able to provide more accurate tracking than single eye models which ignore the correlations and constraints governing the relative motions of the left and right eyes. Furthermore, by combining multiple sensor measurements according to a two-eye model, the tracking module naturally filters out noise in the sensor measurements.

One approach to the orientation estimation model applies a filter (such as a Kalman filter) to sensor measurements from both electronic contact lenses to jointly estimate orientations of the left and right eyes. The filter is designed to output the estimates based on both the sensor measurements and the modeled correlations and constraints governing relative eye motion.

In another approach, the system applies separate filters to the respective sensor measurements from each electronic contact lens to independently estimate orientations of each eye using a single eye model. The estimates are then adjusted based on the set of constraints and correlations governing two-eye movement.

Various actions can be initiated based on orientation tracking of the left and right eyes. For example, virtual objects projected by the electronic contact lenses can be positioned and rotated to compensate for detected eye movements so that a stable image is perceived by the user. Furthermore, alignment of augmented reality images projected by respective femtoimagers may be rotated to compensate for estimated relative roll between the left and right eyes. In another example, an optical power of a variable focus of the electronic contact lenses may be configured based on estimated vergence between the left and right eyes. Eye gestures may furthermore be detected based on the respective estimated orientations to perform various functions associated with an augmented reality environment such as selecting virtual menu items or controlling an externally linked device.

FIG.1Ashows a user wearing a pair of electronic contact lenses110.FIG.1Bshows a magnified view of one of the electronic contact lenses110, andFIG.1Cshows a cross sectional view of the electronic contact lens110. The electronic contact lens110is worn on the surface of the user's eye. The following examples use a scleral contact lens in which the contact lens is supported by the sclera of the user's eye, but the contact lens does not have to be scleral.

As shown inFIG.1B, the electronic contact lens110contains a femtoprojector120and a femtoimager130. The femtoprojector120is a small projector that projects images inward onto the user's retina. It is located in a central region of the contact lens110, so that light from the femtoprojector120propagates through the user's pupil to the retina. The femtoprojector120typically includes an electronics backplane (e.g., driver circuitry), a frontplane of light emitting elements (e.g., an LED array) and projection optics. The frontplane produces an image (referred to as the source image), which is optically projected by the projection optics through the various eye structures and onto the retina105, as shown inFIG.1C.

The femtoimager130is a small imager that is outward facing and captures images of the external environment. In this example, it is located outside the central region of the contact lens110so that it does not block light from entering the user's eye. The femtoimager130typically includes imaging optics, a sensor array and sensor circuitry. The imaging optics images a portion of the external environment onto the sensor array, which captures the image. The sensor array may be an array of photosensors. In some embodiments, the sensor array operates in a visible wavelength band (i.e., ˜390 nm to 770 nm). Alternatively or additionally, the sensor array operates in a non-visible wavelength band, such as an infrared (IR) band (i.e.,˜750 nm to 10 μm) or an ultraviolet band (i.e., <390 nm). For example, the sensor array may be a thermal infrared sensor.

The lead line from reference numeral110inFIG.1Bpoints to the edge of the contact lens. The femtoprojector120and femtoimager130typically are not larger than 2 mm wide. They may fit within a 2 mm×2 mm×2 mm volume.

The electronic contact lens110also includes other electronic components150, which may be mounted on a flexible bus140located in a peripheral zone. Electronic components150in the lens110may include microprocessors/controllers, inertial sensors (such as accelerometers and gyroscopes), magnetometers, radio transceivers, power circuitry, antennas, batteries and elements for receiving electrical power inductively for battery charging (e.g., coils). Sensed data from the inertial sensors and magnetometer may be combined to estimate parameters such as position, velocity, acceleration, orientation, angular velocity, angular acceleration or other motion parameters. For clarity, connections between the femtoprojector120, femtoimager130and electronics components150are not shown inFIG.1B. The flexible bus140may optionally be cut out, for example on the temporal (as opposed to nasal) side of the electronic contact lens110as shown inFIG.1B. The electronic contact lens110may include cosmetic elements, for example covering the electronic components150son the flexible bus140. The cosmetic elements may be surfaces colored to resemble the iris and/or sclera of the user's eye.

FIG.1Cshows a cross sectional view of the electronic contact lens mounted on the user's eye. For completeness,FIG.1Cshows some of the structure of the eye100, including the cornea101, pupil102, iris103, lens104, retina105and sclera106. The electronic contact lens110preferably has a thickness that is less than two mm. The contact lens110maintains eye health by permitting oxygen to reach the cornea101.

The femtoimager130is outward-facing, so that it “looks” away from the eye100and captures images of the surrounding environment. The femtoimager130is characterized by a line of sight132and a field of view134, as shown inFIG.1C. The line of sight132indicates the direction in which the femtoimager130is oriented, and the field of view134is a measure of how much of a scene the femtoimager130captures. If the femtoimager130is located on the periphery of the electronic contact lens110, the contact lens surface will be sloped and light rays will be bent by refraction at this interface. Thus, the direction of the line of sight132in air will not be the same as the direction within the contact lens material. Similarly, the angular field of view134in air (i.e., the external environment) will not be the same as the angular field of view in the contact lens material. The terms line of sight132and field of view134refer to these quantities as measured in the external environment (i.e., in air).

The femtoprojector120projects an image onto the user's retina105. This is the retinal image125shown inFIG.1C. This optical projection from femtoprojector120to retina105is also characterized by an optical axis, as indicated by the dashed line within the eye inFIG.1C, and by some angular extent, as indicated by the solid lines within the eye inFIG.1C. However, the femtoprojector120typically will not be described by these quantities as measured internally within the eye100. Rather, it will be described by the equivalent quantities as measured in the external environment. The retinal image125appears as a virtual image in the external environment. The virtual image125has a center, which defines the line of projection122for the femtoprojector120. The virtual image125will also have some spatial extent, which defines the “span of eccentricity”124for the femtoprojector120. As with the femtoimager line of sight132and field of view134, the terms line of projection122and span of eccentricity124for the femtoprojector120refer to these quantities as measured in the external environment.

The femtoimager130and femtoprojector120both move together with the eye100because the electronic contact lens110is physically mounted to the eye100. Thus, images captured by the femtoimager130naturally have a line of sight132corresponding to the user's gaze direction and virtual images projected by the femtoprojector120naturally move together with the eye100.

FIG.2shows a block diagram of an augmented reality system200that performs various functions based on orientation tracking of a pair of electronic contact lenses110worn on the left and right eyes. The augmented reality system200includes the electronic contact lenses110, as described above, an accessory device212, a network214, a server216and an external imager218. The accessory device212is a computing device that is communicatively coupled to one or both of the electronic contact lenses110(e.g., via a wireless interface) and performs computing or storage functions that support operation of the electronic contact lenses110. The accessory device212may be embodied as an electronic wearable device (e.g., necklace, headband, waistband, etc.), smartphone, smart-watch or another device. The accessory device212may also be connected to a server216via a network214. The server212provides access to various information relevant to operation of the electronic contact lenses110and may furthermore provide additional computing or storage functions that support operation of the electronic contact lenses110. The accessory device212may also optionally be coupled to an external imager218. The external imager218, if present, captures images of the external environment and may be used to supplement images captured by the femtoimager130of the electronic contact lenses110. The external imager218may capture images having a wider field of view, higher resolution or other improved image characteristics relative to the images captured by the femtoimager130.

A lens control module220interfaces with the electronic contact lenses110to perform orientation tracking and to initiate various actions in response to orientation sensing. Various components of the lens control module220may be implemented in whole or in part on one or more of the electronic contact lenses110, on the accessory device212, on the server216or a combination thereof. In some implementations, certain time-sensitive functions of the lens control module220may be implemented directly on the electronic contact lenses110for low latency while other more computationally intensive functions may be offloaded to the accessory device212or to the server216to enable the electronic contact lenses110to operate with relatively light computational and storage requirements. For example, in one implementation, the electronic contact lenses110transfers images captured by the femtoimager130to the accessory device212for performing image processing tasks. The accessory device212may perform these functions directly or may offload the functions in whole or in part to the server216. Alternatively, the electronic contact lens110may perform some lightweight initial processing on the images prior to offloading them to the accessory device212. For example, one or both of the electronic contact lenses110may compress images or extract features from the images and send the compressed images or features to the accessory device212for processing instead of transferring the raw images. The task of generating virtual images for displaying on the electronic contact lenses110can furthermore be performed in whole or in part on the accessory device212or the server216before providing the virtual images to the electronic contact lenses110for display. Additionally, the accessory device212may configure various aspects of the electronic contact lens110that affect its operation. For example, the accessory device212may configure parameters of motion sensors in the electronic contact lenses110.

The lens control module220includes a tracking module230, a communication module228and an image generator226. The image generator226generates virtual images for display by the femtoprojector120. The virtual images may be text, graphics or control elements that are projected by the femtoprojector120onto the user's eye100. The virtual images may be oriented based on in part on the tracked orientations of the eyes so as to provide the user with the perception of a stable image.

The communication module228facilitates communication with external systems to acquire information for displaying by the electronic contact lenses110, to perform external processing relating to orientation tracking, or to control external systems based on interactions using the electronic contact lenses110.

The tracking module230obtains sensor measurements from the motion sensors of the electronic contact lenses110and generates orientation estimates for both eyes. The measurements from the sensors are often subject to noise, thus making individual measurements prone to inaccuracies. However, by capturing a sequence of measurements over time from both electronic contact lenses, and by applying an eye model that describes correlations and constraints associated with the expected motions of the eyes, the tracking module230can generate accurate orientation estimates even in the presence of noise. The orientation estimates can be used to stabilize projected images, to detect eye gestures, to adjust a control function of the electronic contact lenses110, or initiate other actions as described in further detail below.

FIG.3is a diagram illustrating possible motions of a human eye. The eye can generally rotate with three degrees of freedom with its orientation described by pitch302, yaw304, and roll306. In this context, pitch302is a measure of vertical eye motion about a horizontal axis, yaw304is a measure of horizontal eye motion about a vertical axis, and roll306is a measure of torsional eye motion about the line of sight. In normal human eye movement, the eye will only assume a limited number of orientations. In other words, many arbitrary combinations of pitch302, yaw304, and roll306are not actually possible. For example, in normal human eye movement, roll306is constrained as a fixed function of pitch302and yaw304such that the roll can be computed if the pitch302and yaw304are known. Furthermore, with some exceptions, the eye will only assume orientations that can be reached from a “primary position”308(a position looking straight ahead when the head is fixed) by a single rotation about an axis in a plane orthogonal to the line of sight when the eye. InFIG.3, the position in the center308represents the primary position and the page represents the plane orthogonal to the line of sight. The remaining illustrated orientations are all achievable because they can be reached from the primary position308by a rotation about an axes (illustrated by the solid lines) within the plane of the page. This principle is known as Listing's Law.

More complex constraints on eye movement also exist, some of which describe the various exceptions to Listing's Law. For example, Listing's Half-Angle rule describes how the eye moves when the eye starts a rotation from an eccentric eye position that is not the primary position308. A binocular extension to Listing's Law describes how roll306of the eye changes when the eye converges on a near object compared to an object at optical infinity. Additional rules describe constraints on eye movements in other situations such as during vestibulo-ocular reflex motion.

Besides the single eye constraints described above, there are also correlations and constraints that describe the possible orientations and motions of the left and right eyes relative to each other. For example, the pitch302of the left eye will generally match the pitch302of the right eye within some limited deviation range. The difference in yaw304of the left and right eye, referred to as vergence, is constrained to a limited range and directly relates to the focal distance of an object. The relative roll306of the left and right eyes are also related with the difference in roll generally being described by a function of pitch and yaw (within some limited deviation). The tracking module230described herein utilizes these constraints and correlations that describe how an individual eye can move and how a pair of eyes can move relative to each other to generate orientation estimates for both eyes in the presence of potentially noisy sensor measurements.

FIG.4illustrates a first example of a technique for estimating orientations of left and right eyes based on sensor measurements from respective electronic contact lenses110. In this example, each electronic contact lens110includes a set of sensors410such as an accelerometer402, a magnetometer404, a gyroscope406, and a femtoimager408. These sensors410capture respective measurements associated with movement of the left eye and the right eye. For example, the accelerometer402measures acceleration along three axes, the gyroscope406measures angular rate of change along the three axes, and the magnetometer404measures magnetic field strength along the three axes. The femtoimagers408can be used to generate measurements relating to rotation and translation. For example, sequential comparisons of images may be analyzed to detect features that may be tracked to derive relevant motion data. Separate right eye and left eyes filters filter the respective left and right eye sensor measurements to generate respective left and right eye state vectors420estimating the yaw422, pitch424, and roll426of each eye. In this technique, the orientations of each eye are estimated independently of each other to generate independently estimated state vectors420.

The filters may be implemented as Kalman filters or other types of recursive Bayesian estimators. The filters provide the best estimate of each eye's orientation given all available information from the sensor measurements. In an example implementation, left and right eye filters update state vectors representing respective eye orientations at each time step. Each filter generates an updated state vector based on the current state vector, the filter parameters (representing the eye model that predicts how each eye's orientation changes over time), and sensor measurements captured during the current time step.

FIG.5illustrates an example of a process for generating estimated orientations of both left and right eyes based on independent estimated orientations of each eye. In this process, a set of right eye sensors502(such as those illustrated inFIG.4) obtain measurements associated with motion of the right eye and a set of left eye sensors504(such as those illustrated inFIG.4) obtain measurements associated with motion of the left eye. A right-eye orientation estimate508is performed based on the right eye sensor measurements from the right eye sensors502and a one-eye model506that describes single eye motion. A left-eye orientation estimate510is similarly performed based on the left eye sensor measurements from the left eye sensors504and the one-eye model506. Here, the one-eye model may comprise filter parameters of a filter (e.g., a Kalman filter) that are applied to a sequence of measurements to generate the respective orientation estimates508,510as described above. The two estimates508,510may then be further refined by enforcing the various constraints and correlations512of relative eye motion, such as Listing's Law and its extensions described above to generate a two-eye estimate514. In other words, the constraints and correlations512are applied after the individual eye orientation estimates are obtained from the filtering step. Here, the constraints and correlations512describe the permissible relative motion of the eyes based on a two-eye motion model. The constraints and correlations512may be implemented as parameters of a two-eye filter (e.g., another Kalman filter or other type of filter) that generates the two-eye orientation estimate based on the individual single eye estimates. For example, the tracking module230may adjust an initially estimated pitch of one eye to be within a predefined noise range of the estimated pitch of the second eye. Furthermore, the tracking module230may adjust a yaw of one eye to be within a predefined range of a yaw of the second eye (which may be based on a combination of a predefined noise range and a predefined vergence range). In another example, the tracking module230may adjust a roll of one eye to be within a predefined noise range of a roll of the second eye, which may in turn be adjusted to be within a predefined noise range of a function of the pitch and yaw.

FIG.6illustrates another technique for estimating orientations of the left and right eyes. In this example, the sensor measurements from the left and right eye sensors610(e.g., accelerometer602, magnetometer604, gyroscope606, femtoimager608) are input to a single filter that models both the individual constraints on motion of each eye and constraints and correlations relating to the relative motion of the eyes. In this example, the filter produces a single state vector620describing the orientations of both eyes. For example, the state vector620includes a yaw622of one eye (which could be either the left or right eye), a vergence624representing the difference in yaw between the eyes, a roll628of one eye (which could be either the left or right eye), a difference in roll630between the eyes, and a pitch626(which is estimated to be the same for both eyes). The filter takes sensor measurement from both eyes as inputs and applies parameters based on a two-eye motion model to apply the constraints and correlations as part of the filter operation when generating the predicted eye orientations.

In an alternative embodiment, the left and right electronic contact lenses110may each include disparate sets of sensors. For example, in one implementation, one of the lenses110may include only a subset of the sensors included in the other lens110. In another implementation, the electronic contact lenses110include different non-overlapping sets of sensors. For example, one electronic contact lens110may include only an accelerometer and gyroscope, while the other electronic contact lens110includes a magnetometer and femtoimager. In yet another implementation, sensors in different electronic contact lenses110may provide measurements relating to different axes of motion. For example, one electronic contact lens110may include an accelerometer that measures acceleration along two axes while the other lens110includes an accelerometer that measures acceleration along a third axis. Here, the distribution of sensors may enable measurement of yaw independently in both electronic contact lenses110(since yaw may differ between the eyes), while enabling pitch measurement in only one electronic contact lens110(since pitch is generally the same in both eyes). Regardless of the distribution of sensors, the filter may operate to estimate the orientations of both eyes using the available sensor measurements as inputs together with parameters derived from the two-eye model.

In further examples, a more limited set of measurements may be taken depending on the desired elements of the output state vector. For example, for some applications (as described below), an estimate of vergence is the only desired element, and it is not necessary to estimate other elements of the eye orientations. In this case, a more limited set of measurements is sufficient to enable accurate vergence estimation. For example, in an embodiment, electronic contact lenses110include only accelerometers and magnetometers without other sensors for the purpose of vergence estimation. In another example, the filter may be applied to a set of difference values between the first and second state sensor measurements from each eye instead of directly to the measurements.

FIG.7illustrates an example of a process for generating estimated orientations of both left and right eyes using a two-eye model. In this process, a set of right eye sensors702(such as those illustrated inFIG.6) obtain measurements associated with motion of the right eye and a set of left eye sensors704(such as those illustrated inFIG.6) obtain measurements associated with motion of the left eye. A set of constraints and correlations706provide parameters to a two-eye model708that describes how the eyes may move individually and relative to each other. A two-eye orientation estimate710is performed based on the right eye sensor measurements from the right eye sensors702, the left eye sensor measurements from the left eye sensors704, and the two-eye model708to generate estimated orientations710for both eyes. The two-eye orientation estimate710may be performed by applying a filter (e.g., a Kalman filter) to the respective right and left eye measurements in which the two-eye model708comprises a set of filter parameters that models the motion of a set of eyes based on the constraints and correlations706. In this example, a single filter may be applied that generates the estimated output orientations of both eyes in a single operation. The filter furthermore applies the constraints and correlations706as part of the filtering operation so that the estimates for each eye are each dependent on both the left and right eye measurements.

Accurate detection of eye orientation has various applications relating to augmented reality displays using the electronic contact lenses110. For example, by accurately tracking eye orientations, projected images may be shifted to compensate for changes in orientation such that the image appears fixed to the user in a world frame. This technique thus stabilizes the images even the presence of small eye movements.

FIG.8illustrates an example technique for adjusting display of augmented reality images projected by a set of electronic contact lenses110based on estimated eye orientations. InFIG.8, the desired image802represents the intended orientation of the object as it should be perceived by the user. If both eyes are perfectly centered in the eye socket, the relative roll between the eyes should be zero or negligible and identical images can be displayed by femtoimagers in both electronic contact lenses110that are consistent with the desired orientation. However, because the relative roll of the eyes changes as a function of pitch and yaw, projecting identical images on both eyes when the eyes are not centered in the socket will result in perceived misalignment due to the relative roll. To compensate for the relative roll, the images may be rotated804in an amount corresponding to the detected roll of each eye using the orientation estimation techniques described above. The resulting images806that are rotated in the lens reference frame will appear aligned in the correct orientations from the user's perspective. For this purpose, the relative roll of the eyes can be estimated much more accurately and with better noise rejection using the above-described techniques relative conventional techniques that rely only on directly measuring roll.

The electronic contact lens system may also trigger an action in response to detecting a difference in relative roll between the eyes. As described above, the relative roll is zero or negligible when the eyes are centered in their sockets. Thus, detecting a relative roll indicates that the user's gaze has deviated from the centered position. The electronic contact lens system can recognize eye gestures based on detected patterns in the relative roll and trigger actions in response. A specific eye gesture may be detected when the relative roll goes from approximately zero to a threshold value, indicative of the user changing from a straight-ahead gaze to an off-center gaze (e.g., up and to the left). The detected movements may be filtered to distinguish between deliberate and inadvertent movements, for example, by only triggering a detection when the difference in relative roll is present for at least a threshold time period. In another example, a gesture may be detected when the eyes move to an off-center gaze position for a threshold time period and then return to the center position. This gesture may be used to trigger various actions such as activating or deactivating a contact lens visual user interface.

The above-described technique provides one example of indirectly estimating head motion based only on eye tracking (i.e., without directly sensing head movement). Other kinds of head motion can also be inferred from detecting eye motions that are known only to occur during certain kinds of head movements.

The orientation estimates described above can be similarly applied to adapt other aspects of projected images based on the estimated orientations. For example, a relative translation can be applied to images projected on the left and right eye to compensate for detected relative yaw (vergence) between the eyes. Furthermore, because the relative yaw of the eyes relates to the focal distance, images may be projected to appear at a particular apparent depth based on the detected vergence.

Accurate orientation estimates can also enable solutions for presbyopia. For example, distances to objects on which a user is fixating can be estimated based on the estimated vergence between the eyes. The distance estimate can then be used to adjust an optical power of a variable focus of the electronic contact lenses110. The electronic contact lenses110can therefore supplement or replace the eye's natural ability to accommodate these changes for individuals with varying degrees of presbyopia.

Although the detailed description contains many specifics, these should not be construed as limiting the scope of the invention but merely as illustrating different examples. It should be appreciated that the scope of the disclosure includes other embodiments not discussed in detail above. Various other modifications, changes and variations which will be apparent to those skilled in the art may be made in the arrangement, operation and details of the method and apparatus disclosed herein without departing from the spirit and scope as defined in the appended claims. Therefore, the scope of the invention should be determined by the appended claims and their legal equivalents.