Object selection based on eye tracking in wearable device

A wearable computing device includes a frame, a camera mounted on the frame so as to capture images of an environment outside of the wearable computing device, a display device mounted on the frame so as to display the images captured by the camera, and at least one eye gaze tracking device mounted on the frame so as to track a gaze directed at the images displayed by the display device. In response to the detection of a fixation of the gaze on the display of images, the system may identify a pixel area corresponding to a fixation point of the fixation gaze on the display of images. The system may identify an object in the ambient environment corresponding to the identified pixel area, and set the identified object as a selected object for user interaction.

TECHNICAL FIELD

This description generally relates to object selection in a wearable device, and in particular, to world object selection in a wearable device based on eye tracking information.

BACKGROUND

Wearable computing devices can include, for example, head mounted display devices including smart glasses and the like, arm/wrist/hand worn devices including smart watches, smart bracelets/bands, smart rings, smart pendants and the like, headphones/ear buds, and other such devices. Users can interact with content made available by wearable computing devices, can capture and/or process and/or access information using wearable computing devices, and can rely on wearable devices to interact with content made available by the wearable device.

SUMMARY

In a first general aspect, a computer-implemented method includes capturing, by a camera mounted on a frame of a wearable computing device, images of an ambient environment outside of the wearable computing device; triggering display, by a display device mounted on the frame of the wearable computing device, of the images captured by the camera; tracking, by at least one gaze tracking device mounted on the frame of the wearable computing device, a user gaze directed at the display of the images by the display device; identifying a pixel area, of a plurality of pixel areas of the display of images of the display device, corresponding to the detected user gaze; and matching the identified pixel area with a corresponding object in the ambient environment.

In some implementations, the wearable computing device is a head mounted display device including the frame, with the camera, the display device and the at least one gaze tracking device all mounted to the frame, and the display device projecting the images of the ambient environment captured by the camera onto a lens of the head mounted display device.

In some implementations, capturing images of the ambient environment includes capturing portions of the ambient environment within a field of view of the camera as the wearable computing device changes at least one of position or orientation in the ambient environment. In some implementations, tracking the user gaze includes tracking, by at least one eye gaze tracking device, eye gaze of at least one eye of a user of the wearable computing device.

In some implementations, tracking user gaze directed at the display of the images and identifying the pixel area corresponding to the detected user gaze includes detecting a fixation gaze including fixation of the user gaze directed at the display of the images; identifying a fixation point of the detected fixation gaze at an intersection of the fixation gaze with the display of the images; and identifying the pixel area at the intersection of the fixation gaze with the display of the images. In some implementations, detecting the fixation of the user gaze includes detecting movement of the user gaze of less than or equal to a previously set threshold of movement of the user gaze. In some implementations, detecting movement of the user gaze includes detecting at least one of a magnitude or a frequency associated with the detected movement. In some implementations, identifying the pixel area includes identifying a pixel within the display of the images at which the fixation gaze intersects the display of the images.

In some implementations, matching the identified pixel area with the corresponding object in the ambient environment includes analyzing an area surrounding the identified pixel area; identifying features within the area surrounding the identified pixel area based on the analyzing; recognizing at least one of the identified features; and matching the pixel area to the corresponding object based on the at least one identified feature. In some implementations, recognizing at least one of the identified features includes at least one of recognizing at least one of the identified features based on a search of an external database of previously stored images; or recognizing at least one of the identified features based on known features of the ambient environment.

In some implementations, the tracking of the user gaze, the identifying of the pixel area, and the matching of the identified pixel area to the corresponding object in the ambient environment includes initiating, by a processor of the wearable computing device, a select mode of operation of the wearable computing device in response to at least one of a detected user input or detection of a stationary condition of the wearable computing device for greater than or equal to a previously set threshold of time; and performing the tracking, the identifying and the matching only during operation of the wearable computing device in the select mode.

In another general aspect, a computing system includes a wearable computing device, including a frame; a camera coupled to the frame so as to capture images of an ambient environment outside of the wearable computing device; a display device coupled to the frame; at least one gaze tracking device coupled to the frame; at least one processing device; and a memory storing instructions. The instructions, when executed, cause the system to perform operations capturing, by the camera, images of the environment outside of the wearable computing device; displaying, by the display device, the images captured by the camera; tracking, by the at least one gaze tracking device, a gaze of a user of the wearable computing device directed at the display of the images by the display device; identifying a pixel area, of a plurality of pixel areas of the display of images of the display device, corresponding to the detected user gaze; and matching the identified pixel area with a corresponding object in the ambient environment.

In some implementations, the camera is configured to capture images of portions of the ambient environment within a field of view of the camera as the wearable computing device changes at least one of position or orientation in the ambient environment, such that the instructions cause the system to perform operations including detecting a fixation gaze including fixation of an eye gaze of the user directed at the display of the images; identifying a fixation point of the detected fixation gaze at an intersection of the fixation gaze with the display of the images; and identifying the pixel area at the intersection of the fixation gaze with the display of the images. In some implementations, the instructions cause the system to perform operations including analyzing an area surrounding the identified pixel area; identifying features within the area surrounding the identified pixel area based on the analyzing; recognizing at least one of the identified features; and matching the pixel area to the corresponding object based on the at least one identified feature. In some implementations, the instructions cause the system to perform operations including searching an external database of previously stored images to identify at least one of the identified features and match the identified at least one feature to an object in the ambient environment; or recognizing at least one of the identified features based on known features of the ambient environment.

In some implementations, the instructions also cause the system to perform operations including detecting a condition for entering a select mode of operation of the wearable computing device; initiating operation of the wearable computing device in the select mode in response to the detected condition; and performing the tracking, the identifying and the matching only during operation of the wearable computing device in the select mode.

In another general aspect, a non-transitory, machine-readable medium has instructions stored thereon that, when executed by a processor, cause a computing device to perform operations including capturing, by a camera mounted on a frame of a wearable computing device, images of objects within a field of view of the camera in an ambient environment outside of the wearable computing device; triggering display, by a display device mounted on the frame of the wearable computing device, of the images captured by the camera; tracking, by at least one gaze tracking device mounted on the frame of the wearable computing device, a user gaze directed at the display of the images by the display device; identifying a pixel area, of a plurality of pixel areas of the display of images of the display device, corresponding to the detected user gaze; and matching the identified pixel area with a corresponding object in the ambient environment.

In some implementations, the instructions also cause the computing device to perform operations including detecting a condition for entering a select mode of operation of the wearable computing device; initiating operation of the wearable computing device in the select mode in response to the detected condition; and performing the tracking, the identifying and the matching only during operation of the wearable computing device in the select mode. In some implementations, the instructions also cause the computing device to perform operations including tracking, by at the least one gaze tracking device, eye gaze of at least one eye of a user of the wearable computing device; detecting a fixation gaze including fixation of the user eye gaze directed at the display of the images; identifying a fixation point of the detected fixation gaze at an intersection of the fixation gaze with the display of the images; and identifying the pixel area at the intersection of the fixation gaze with the display of the images.

In some implementations, the instructions also cause the computing device to perform operations including analyzing an area surrounding the identified pixel area; identifying features within the area surrounding the identified pixel area based on the analyzing; recognizing at least one of the identified features; and matching the pixel area to the corresponding object based on the at least one identified feature. In some implementations, the instructions also cause the computing device to perform operations including setting the corresponding object as a selected object; and providing for interaction with the selected object through the wearable computing device.

Implementations of the described techniques may include hardware, a method or process, or computer software on a computer-readable medium. The details of one or more implementations are set forth in the accompanying drawings and the description below. Other features will be apparent from the description and drawings, and from the claims.

DETAILED DESCRIPTION

This document describes examples related to real world object identification and selection for electronic wearable devices. The object identification and selection may include the use of sensors of the wearable device that allow the wearable device to capture and assess an ambient environment surrounding the wearable device. The object identification and selection may also include the use of sensors of the wearable device that provide for user gaze tracking, for example, eye gaze tracking. Correlation between a detected gaze, for example, eye gaze of the user and a representation of the real world view of the ambient environment, captured by a camera of the wearable device and presented to the user by a display device of the wearable device, may allow the user gaze to be isolated to a specific portion of the representation of the real world view, or to a specific pixel area of the display device. This may in turn allow the object on which the user gaze is focused to be identified for selection.

Systems and methods, in accordance with implementations described herein, provide an advantage of quickly and accurately identifying an object of interest to a user based on a detected gaze, rather than relying on other forms of user interaction and/or input for object selection. Other forms of interaction and/or input for object identification and/or selection may rely on the detection of, for example gestural input commands, touch input commands, audible input commands and the like. The use of these types of input methods for interaction may be disruptive in some environments. Further, identification of objects for which the input is intended and/or a particular action to be taken may be somewhat inaccurate when using these types of input methods. Systems and methods, in accordance with implementations described herein, may solve these technical challenges, or technical problems. Systems and methods, in accordance with implementations described herein, may provide for user interaction with objects made available to the user via a mobile computing device through the use of a gaze input mode, allowing the user to interact with content, select and interact with objects and the like in a less disruptive, relatively discreet manner. Information collected by an eye gaze tracking device may be correlated with information related to the display of content to the user, to isolate an object, and in particular a pixel area, at which the user gaze is directed. Isolation of the user gaze to a particular pixel area, or pixel, may yield improved accuracy in identification of the object on which the user gaze is focused, and/or improved accuracy in the identification of the user input command associated with the detected user gaze.

FIG.1Aillustrates a system in which a user can use one or more wearable devices100to access one or more external resources1100via network1200.FIG.1Aprovides a third person view of a user in an ambient environment1000, including various examples of mobile computing devices100.FIG.1Ashows example mobile computing devices100including a first head worn wearable computing device100A, a second head worn wearable computing device100B, a wrist worn computing device100C, and a handheld computing device100D. In some examples, the first head worn computing device100A can include various components including for example a display, a camera that can capture images of the ambient environment, audio input/output capability, user input capability and the like, such as in a head mounted display device, in the form of smart glasses, or a headset. In some examples, the second head worn computing device100B can be an ear worn computing device such as headphones, or earbuds, and can include audio input/output capability, a camera that can capture images of the ambient environment, user input capability and the like. In some examples, the wrist worn computing device100C can include a display, a camera that can capture images of the ambient environment, audio input/output capability, user input capability and the like, such as in a smart watch, or wristband. In some examples, the handheld computing device100D can include a display, a camera that can capture images of the ambient environment, audio input/output capability, user input capability, and the like, such as in a smartphone.FIG.1Aillustrates just some examples of mobile computing devices. The principles to be described herein may be applied to other types of mobile computing devices, including wearable computing devices, not specifically shown inFIG.1A. In some examples, mobile computing devices100, including wearable devices such as the example devices shown inFIG.1A, can communicate with each other and/or with the external resources1100, to exchange information, to receive and transmit input and/or output, and the like.

FIG.1Bis a front view, andFIG.1Cis a rear view, of the example of the wearable computing device100A shown inFIG.1A, in the form of a head mounted display (HMD) device. In some implementations, the example HMD may take the form of a pair of smart glasses, or augmented reality glasses, or an augmented reality and/or virtual reality headset or goggles. Hereinafter, systems and methods in accordance with implementations described herein will be described with respect to the wearable computing device100in the form of an HMD, simply for ease of discussion and illustration. The principles to be described herein can be applied to other types of mobile computing devices and/or combinations of mobile computing devices working together, including the example mobile computing devices described with respect toFIG.1A, that are capable of image capture and display, and capable of gaze tracking for object identification and selection.

As shown inFIG.1B, the example HMD100includes a frame102. In the example shown inFIGS.1B and1C, the frame102includes a rim portion surrounding glass portion(s), or lenses, and arm portions coupled to the rim portion. A display device104coupled in a portion of the frame102. In the example shown inFIGS.1B and1C, the display device104is coupled in the arm portion of the frame102. The HMD100can also include an audio output device106, an illumination device108, a sensing system110, a control system112, at least one processor114, and an outward facing image sensor116, or camera116. In some implementations, the display device104may include a see-through near-eye display. For example, the display device104may be configured to project light from a display source onto a portion of teleprompter glass functioning as a beamsplitter seated at an angle (e.g., 30-45 degrees). The beamsplitter may allow for reflection and transmission values that allow the light from the display source to be partially reflected while the remaining light is transmitted through. Such an optic design may allow a user to see both physical items in the world next to digital images (e.g., user interface elements, virtual content, etc.) generated by the display device104. In some implementations, waveguide optics may be used to depict content on the display device104of the HMD100.

The audio output device106may include one or more speakers coupled to the frame102. The sensing system110may include various sensing devices and the control system112may include various control system devices to facilitate operation of the HMD100including, for example, one or more processors114operably coupled to the components of the control system112. In some implementations, the control system112may include a communication module providing for communication and exchange of information between the HMD100and other external devices.

In some implementations, the camera116may be an outward facing camera116, or a world facing camera116that is capable of capturing still and/or moving images of external objects in the ambient environment within a field of view of the camera116. In some implementations, the camera116may be a depth camera that can collect data related to distances of the external objects from the camera116. In some implementations, the illumination device108may selectively operate, for example, with the camera116, for detection of objects in the field of view of the camera116.

As shown inFIG.1C, in some implementations, the HMD100includes a gaze tracking device120including, for example, one or more sensors125for example, one or more image sensors125. The gaze tracking device120may detect and track eye gaze direction and movement. Images captured by the sensor(s)125may be processed to detect and track gaze direction and movement, and to detect gaze fixation. In some implementations, the detected gaze may be processed as a user input to be translated into a corresponding interaction with external objects in the ambient environment1000that are visible to the user through the HMD100and/or interaction with objects displayed to the user by the display device104.

FIGS.2A-2Hillustrate an example system and method of real world object identification and/or selection, in accordance with implementations described herein. An example of real world object identification and/or selection by the user will be described hereinafter with respect to the use of one of the mobile computing devices100described above, particular the first wearable computing device in the form of the head mounted display device, or smart glasses, simply for ease of discussion and illustration. The principles to be described herein may be applied to other mobile computing devices, and/or may be applicable to multiple paired devices that can communicate with each other for the exchange of information, the detection and transmission of inputs and commands, and the like.

FIGS.2A and2Bare third person views of the user in the ambient environment1000. As illustrated in the example shown inFIGS.2A and2B, in some implementations, the user may view objects in the ambient environment1000through the wearable computing device100. The ambient environment1000may be viewable by the user with and without wearing the wearable computing device100. In some implementations, portions of the ambient environment1000captured within the field of view of the camera116/objects within the captured portion of the ambient environment1000may be displayed to the user by the display device104. In the example shown inFIG.2A, a portion of the ambient environment1000including a workspace200is captured within the field of view, or live viewfinder, of the camera116of the wearable computing device100. The example workspace200in the ambient environment1000includes a laptop computer220, a lamp230, a printer240and a book250on a desk210. As shown inFIG.2B, the display device104may display an image1000A, or a capture area1000A, to the user. The display of the image1000A, or capture area1000A may appear to be some set distance from the user, outside of the wearable computing device100. The display of the image1000A, or capture area1000A, may depict content captured within the live viewfinder, or within the field of view of the camera116of the wearable computing device100. The content depicted within the image1000A, or capture area1000A may be representative of the corresponding portion of the ambient environment1000. The content depicted within the image1000A, or capture area1000A displayed to the user may change as the user moves in the ambient environment1000and/or as the user changes head orientation (thus re-orienting the camera116of the wearable computing device100), and the like. In some implementations, the content depicted within the image1000A, or capture area1000A may at some point become fixed, even in the event of movement of the wearable computing device100, which would change the portion of the ambient environment1000captured within the field of view of the camera116. In some implementations, the content depicted within the image1000A, or capture area1000A may be fixed, for example in response to a user input or other detected condition, even as the user changes position in the ambient environment1000and/or head orientation. In some implementations, the fixing of the image1000A, or capture area1000A displayed by the display device104may facilitate user interaction with the displayed content, without the need to remain still (e.g., maintain a stationary position) to focus on a particular object included in the image1000A, or capture area1000A.

In the example shown inFIGS.2B,2C and2D, the image1000A, or capture area1000A depicts a representation220A of the laptop computer220, a representation230A of the lamp230, a representation240A of the printer240and a representation250A of the book250on a representation210A of the desk210, corresponding to the laptop computer220, the lamp230, the printer240and the book250on the desk210in the workspace200in the ambient environment1000.FIGS.2C and2Dillustrate a user gaze G directed at an object in the workspace200. In particular, in the examples shown inFIGS.2C and2D, the user gaze G is directed at the representation of the real world object included in the image1000A, or capture area1000A displayed to the user by the display device104of the wearable computing device100.FIG.2Cis a third person view illustrating the image1000A, or capture area1000A as viewed by the user wearing the wearable computing device100.FIG.2Dis a third person view of the image1000A, or capture area1000A as viewed from a position facing the user.

As shown inFIGS.2C and2D, a gaze G, for example, an eye gaze G of the user may be detected by the gaze tracking device120. In some implementations, the wearable computing device100includes a single gaze tracking device120including one or more sensors125such as, for example, image sensors and/or other types of sensors configured to collect data for eye gaze tracking. In some implementations, the single gaze tracking device120tracks eye gaze, for example, tracks pupil movement, for one eye of the user. In some implementations, the single gaze tracking device120tracks eye gaze, for example, pupil movement, for both eyes of the user. In some implementations, the wearable computing device100includes multiple gaze tracking devices120such as, for example, a first gaze tracking device120including one or more image sensors125to track movement, for example pupil movement, of a first eye of the user, and a second gaze tracking device120including one or more image sensors125to track movement, for example pupil movement, of a second eye of the user.

In the example shown inFIGS.2C and2D, the detected gaze G is scanning the representation200A of the workspace200in the image1000A, or capture area1000A representative of the portion of the ambient environment1000captured within the live viewfinder, or field of view of the camera116of the wearable computing device100. In some implementations, the system may operate to track user gaze, to detect fixation of the gaze on an object, and to perform an operation to identify the object on which the gaze G is focused.

In some implementations, the wearable computing device100may operate in a select mode, in which the wearable computing device100operates to detect user focus on an object for selection. In some implementations, operation of the wearable computing device100in the select mode may be triggered in response to a detected condition for entering the select mode. For example, operation of the wearable computing device100in the select mode may be triggered in response to detected movement of the wearable computing device100within a previously set range, indicative that the user has essentially stopped moving. In some implementations, movement of the wearable computing device100may be detected by, for example, one or more sensing devices of the wearable computing device100, such as, for example, an inertial measurement unit of the wearable computing device100and the like. In some implementations, operation of the wearable computing device100in the select mode may be triggered in response to a user input. In some examples, the user input may include, for example, an audible command detected by the wearable computing device100and/or an external device paired with the wearable computing device100. In some examples, the user input may include a touch input or a gesture input detected by the wearable computing device100and/or an external device paired with the wearable computing device100. In some examples, other inputs detected at the wearable computing device100and/or an external device paired with the wearable computing device100may trigger operation of the wearable computing device100in the select mode.

In some implementations, operation in the select mode may cause the wearable computing device100to initiate operation of the gaze tracking device(s)120. In some implementations, operation in the select mode may cause the wearable computing device100, for example, a processor of the wearable computing device100, to initiate an operation in which eye gaze data collected by the gaze tracking device(s)120may be analyzed to identify an object of focus based on the detected eye gaze data. In some implementations, the eye gaze may be directed at, or focused on a portion of the image1000A or capture area1000A captured by the camera116and viewable by the user on the display device104of the wearable computing device100. The eye gaze data may be analyzed to match the detected gaze to a pixel area290A, or to a pixel290A, of the image1000A. Isolation, or matching of the detected gaze to a particular pixel are290A, or to a particular pixel290A, of the displayed image1000A, or capture area1000A may facilitate identification of the object of focus of the detected gaze, and may facilitate user selection and interaction with the object of focus.

A live image feed of objects in the ambient environment1000captured within the field of view of the camera116may be displayed to the user by the display device104. As the user moves in the ambient environment1000, as the head of the user shifts or changes position and/or orientation, and the like, live content displayed in the image1000A, or capture area1000A may change as the objects in the ambient environment1000captured within the field of view of the camera116change. As the object in the ambient environment1000that is of interest to the user is captured within the field of view of the camera116and displayed to the user in the image1000A, or capture area1000A by the display device104, the user gaze G may begin to settle on the object of interest in the image1000A, or capture area1000A, as shown inFIG.2E.

In some situations, the field of view of the camera116may be offset from the field of view of the user. In this example, the field of view of the user as viewed through the lenses of the wearable computing device100is offset from the field of view of the camera116). In this example arrangement, the view captured by the camera116is different than the user view of the ambient environment1000through the lenses of the wearable computing device100. For example, in an arrangement in which the camera116is oriented essentially straight out from the wearable computing device100/the head of the user, and the display device104is oriented so as to be offset by some angular distance from a nominal gaze of the user, the ambient environment1000visible through the lenses of the wearable computing device100and the content captured by the camera116and depicted in the image1000A, or capture area1000A by the display device104would not match. In some situations, this may allow the user to clearly separate the content provided by the display device104from the user view of the ambient environment1000through the wearable computing device100, and thus may facilitate user interaction. In some situations, the display device104may be essentially aligned with the nominal gaze of the user. In this example, the image1000A, or capture area1000A displayed by the display device104may essentially coincide with, or match, the ambient environment1000as viewed by the user, specifically at some fixed point in the ambient environment1000.

User movement or positioning outside of, or away from that fixed point may produce misalignment between the image1000A, or capture area and the user view of the ambient environment1000through the lenses of the wearable computing device100. In some situations, this deterioration in the alignment or match between the image1000A, or capture area1000A and the user view of the ambient environment1000may cause the user to move to align, or re-align the image1000A, or capture area1000A with the user view of the ambient environment1000. In some implementations, the system may prompt the user to move the image1000A, or capture area1000A into alignment with the user view of the ambient environment1000. In some situations, this may allow the user to more clearly discern object(s) of interest, may facilitate selection and interaction, and the like. In some implementations, this may facilitate system calibration. In some implementations, given the calibration and the location of the fixed point at which the displayed image1000A, or capture area11000A is aligned with the user view of the ambient environment1000, the system can then verify (based on data collected by the gaze tracking device(s)120) that the user is looking at the displayed image1000A, or capture area from the fixed point, and calculate a vergence to an object within the displayed image1000A, or capture area1000A to estimate a corresponding distance. In a situation in which the user aligns the displayed image1000A, or capture area1000A with the user view of the ambient environment1000(for example, in response to a system prompt to carry out the alignment), the system may detect alignment based on relative lack of movement, fixation of the user gaze, and the like. In response to the detected alignment, the pixel area290A, or pixel290A of the display of the image1000A, or capture area1000A associated with the detected gaze produces a first ray extending from the camera116through an imaging plane of the camera116directed to a point corresponding to the pixel290A, and a second ray extending from the eye of the through the pixel290A of the display of the image1000A, or capture area1000A displayed by the display device104. Factory calibration of components of the gaze tracking device(s)120, positioning of the display device104, and positioning of the camera116allow for the first and second rays to be referenced to a common coordinate system for example, the coordinate system associated with the camera116) and for the angle between the first and second rays to be computed. A known position of the eye of the user by the gaze tracking device(s)120and a transform from the field of view of the gaze tracking device120to the field of view of the camera116allows the angle and the distance from the camera116to the eye of the user to be computed, yielding one side and two angles of a triangle formed by the camera116, the eye of the user, and the object of interest. Casting rays along these two angles to their intersection may provide an estimate of distance

In some implementations, the gaze tracking device(s)120may continue to track the eye gaze of the user as the user views and scans the image1000A, or capture area1000A for the representation of the object of interest. As the user scans the image1000A, or capture area1000A, the gaze tracking device(s)120may detect corresponding smooth pursuit, or smooth eye gaze motion. As the user locates the object of interest within the image1000A, or capture area1000A, eye motion may begin to settle (for example, at or near the object of interest), and the gaze tracking device(s)120may begin to detect fixation of the gaze, in which the gaze may transition from the smooth pursuit motion (during scanning) to a relatively fixed condition, or a fixation gaze. In the fixation gaze, the user gaze may be focused on the representation of the object of interest depicted in the image1000A, or capture area1000A provided by the display device104. In some implementations, the fixation gaze may be detected based on detected eye gaze movement of less than or equal to a previously set threshold amount of movement. In some implementations, the set threshold amount of movement may include a magnitude and/or a frequency and/or a rate of movement. In some implementations, fixation gaze may be characterized by detection of involuntary microsaccadic eye movement centered around a target, Fixation may be characterized by less than approximately 1 degree of gaze angle at a frequency of approximately 1 to 2 Hertz, or a speed of less than approximately 100 degrees per second. In contrast, smooth pursuit eye gaze, or free motion eye gaze, may be characterized by saccades of greater than approximately 1 degree and/or frequencies above approximately 2 Hertz.

In response to detection of gaze fixation, the system may correlate or match the detected fixation gaze to a corresponding portion of the image1000A, or capture area1000A, for example a particular pixel area, or pixel. For example, the system may match an intersection of the detected fixation gaze with a corresponding portion, or pixel area, or pixel, of the image1000A, or capture area1000A provided by the display device104. In some implementations, the system matches the detected fixation gaze with the pixel area, or pixel, based on a known relationship between the world facing camera116, the corresponding content depicted in the image1000A, or capture area1000A, and the orientation of the gaze tracking device(s)120.

Based on, for example, the known orientation of the world facing camera116, the display device104, and the gaze tracking device(s)120tracking eye gaze, the pixel area290A, or pixel290A associated with the detected the fixation of the gaze G, the system may initiate a process to identify the object of interest associated with the fixation of the user's gaze G. In some implementations, the identification of the object corresponding to the identified pixel area290A, or pixel290A may include an image search based on features that are recognized or detected in a portion of the image1000A, or capture area1000A surrounding the identified pixel area290A, or pixel290A. In some implementations, features recognized or detected in the portion of the image1000A, or capture area1000A may be matched to known objects in the ambient environment1000to identify the object associated with the fixation gaze. In some implementations, the system may access an external source, such as an external database, to search for and recognize or identify the object associated with the fixation gaze.

Matching of the detected fixation gaze G to the pixel290A, or pixel area290A will now be described. In some implementations, light may be generated by the display device104and directed toward the eye(s) of the user wearing the wearable computing device100. Reflections of this light off the cornea may produce vectors normal to the center of the cornea, detectable by the gaze tracking device(s)120, with two such normal vectors intersecting at an estimated 3D center of the cornea. The pupil may be located and a calibrated or estimated cornea radius may be used to form a sphere based on the estimated 3D center of the cornea. The index of refraction of the cornea and aqueous humor at a known depth of the surface of the cornea to a plane of the pupil may provide a 3D location of the pupil. A ray from the 3D center of the cornea center through the 3D center of the pupil may define the optical axis of the eye. This may then be adjusted for a calibrated visual axis offset for the user. The visual axis offset may extend toward the nose, for example, less than approximately 5 degrees towards the nose. The intersection of that ray and the image1000A, or capture area1000A provided by the display device104may represent a single pixel290A at which the detected gaze is directed. Multiple such observations may define a pixel area290A. The detection of the intersection of the ray with the image1000A, or capture area1000A may be facilitated by calibration of the position of the display device104with respect to the camera116.

In some implementations, a best-fit 2D ellipse for the pupil may be used to compute horizontal and vertical rotations that would translate as a circular projection on the data, for example, image data of the pupil captured by the gaze tracking device(s)120. The horizontal and vertical rotations may form the basis for an estimate of an inverse of an angle of the gaze.

In some implementations, the data collected by the gaze tracking device(s) such as, for example, image data, may be fed into neural networks, employ numerous deep layers, and regress to X and Y coordinates associated with the detected gaze. These X and Y coordinates may then be used to determine the intersection of the detected gaze with the display of the image1000A, or capture area1000A by the display device104.

Identification of the pixel290A, or the pixel area290A, of interest may provide for the eventual identification of an object of interest. For example, edge detection may be used to segment the area surrounding the pixel290A, or pixel area290A, of interest from surrounding areas in the image1000A, or capture area1000A, to define an image patch. The image patch may be processed, for example, by an external resource or by the wearable computing device100, to assign one or more semantic labels identifying the object depicted in the image patch.

In some implementations, identification of the object depicted in the image patch may include supplemental information related to the identified object of interest. In some implementations, the object of interest may be known to correspond to a particular system action or function which may be implemented in response to identification of the object of interest. In some implementations, in which the object of interest includes a visual code, such as, for example a quick response (QR) code, specialized action or functionality or information may be returned in response to the identification of the visual code in the image patch.

Identification of the pixel290A, or pixel area290A of the image1000A, or capture area1000A may be translated into a ray that extends outward from the camera116into the ambient environment1000. Segmentation of the area surrounding the pixel290A, or pixel area290A, may provide a boundary which may at least in part define an angular distance from top to bottom, left to right associated with the segmented area. In some implementations, an estimate of the size of the object of interest (based on the angular size and the distance to the object) or the distance to the object of interest (based on the angular size and known dimensions of the object of interest) may facilitate identification of the object of interest. In some implementations, identification of objects in proximity to the object of interest may also facilitate the determination of scale and distance associated with the object of interest.

In some implementations, the system may interpret the detected fixation of the gaze G on the representation240A of the printer240as a user input, for example, a user input selecting the printer240. In a wearable computing device100configured for operation in the select mode as described above, the system may interpret the detected fixation of the gaze G on the representation240A of the printer240as a user input selecting the printer240when the wearable computing device100is operating in the select mode. In some implementations, in response to a detected selection of an object such as the detected selection of the printer240as described above, user interaction with the selected object may be enabled.

In some implementations, once operating in the select mode, content of the image1000A, or capture area1000A may remain static, or fixed, so that the objects depicted in the image1000A, or capture area1000A remain displayed by the display device104regardless of subsequent user movement, changes in orientation of the world facing camera116, changes in objects within the field of view of the world facing camera116and the like. This may allow a user to initiate and/or complete an interaction with the selected object, with that object still displayed to the user by the display device104. This may improve user interaction with the object and avoid discomfort and distraction during user interaction with the object.

In some implementations, operation in the select mode, control of the content depicted in the image1000A, or capture area1000A during operation in the select mode, release of operation in the select mode and the like may be set in accordance with user preferences.

In some implementations, an indication of the detected selection may be implemented prior to providing for user interaction with the selected object. For example, in response to detected selection, an appearance of the representation of the selected object as depicted in the image1000A, or capture area1000A may be altered to reflect the detected selection. For example, the representation of the object may be highlighted or bolded, as the representation240A of the printer240shown inFIG.2E, to indicate selection of the printer240based on the detected fixation gaze. In some implementations, a verification of the detected selection may be implemented prior to providing for user interaction with the selected object. For example, a fixation gaze that is sustained for greater than a previously set threshold amount of time, a user input or gesture including, for example an eye gesture such as a blink, and the like may provide verification of the selection. In some implementations, a user input such as a touch input, an audible input and the like received by the wearable computing device100may provide verification of the selection. In some implementations, a user input providing verification of the selection may be received at an eternal device that is in communication with the wearable computing device100. In some implementations, a user input such as a head and/or eye gaze movement away from the representation of the selected object, an input detected at the wearable computing device100, an input received via an external device in communication with the wearable computing device100, or other such user input may release selection of the object.

As noted above, in response to a detected selection of an object such as the detected selection of the printer240as described above (and in some implementations, in response to user verification of selection of the object), user interaction with the selected object may be enabled. For example, in response to detected selection of the printer240as described above, the user may be provided with access to additional information related to the printer240, access to control of the printer, and the like. In some implementations, access to additional information and control related to the selected object (the printer240in this example) may be provided in the form of a menu280A displayed to the user by the display device104of the wearable computing device100, as shown inFIG.2F. In the example shown inFIG.2F, the example menu280A includes various items that may be selected by the user to access and control functionality of the printer240, to access additional information related to the printer240and the like. In some implementations, selection of one of the items from the menu280A may be triggered in response to a user input. User inputs may include, for example, a gaze input as described above, a voice input, a gesture input, a touch input at the wearable computing device100and/or an external device in communication with the wearable computing device100, and the like.

In some implementations, in response to the detection of the fixation of the user gaze G as described above with respect toFIG.2E, a zooming action may be implemented, as shown inFIG.2G. The zooming action may allow the user to view a zoomed-in version of the image1000A, or capture area1000A including the portion of the image1000A or capture area1000A on which the user's gaze G is fixed or focused. This zooming action may allow the user to further focus, or more precisely focus his or her gaze at a desired object, and/or may allow the user to gain additional information not visible without the zooming action. For example, a brand and an operating status of the printer240(based on the representation240A of the printer240visible to the user in the image1000A, or capture area1000A), which were not visible to the user inFIGS.2E and2F, are visible to the user in the zoomed-in view shown inFIG.2G. Thus, the zooming action may provide the user with additional information related to the object of focus, and may obviate or eliminate the need for the system to identify the object and/or to obtain additional information related to the object. In some implementations, the zooming action may allow the user to further discriminate the area of focus on the image1000A, or capture area1000A, to more accurately fix the gaze in the desired area/on the desired object/portion of the object.

In some implementations, the ability to zoom in this manner may allow a gaze tracking device having relatively lower accuracy to still facilitate the identification of an object of interest in this manner. For example, by zooming in on the image1000A, or capture area1000A (corresponding to that portion of the ambient environment1000), the object of interest may fill a relatively large portion, or substantially all of a quadrant area, thus allowing a relatively low resolution/low fidelity gaze tracking device to disambiguate the object of interest. In some implementations, zooming may be done in response to a user input at the wearable computing device100and/or at an external computing device that is paired to the wearable computing device100, including for example, touch inputs, audible inputs and the like. In some implementations, the system may perform a panning action so that the fixation gaze, or a fixated area of interest is at a central portion of the displayed image1000A, or capture area1000A to facilitate localized zooming and retain context. For example, if the user fixation gaze is directed to the lower right corner of the displayed image1000A, or capture area1000A, the system may correlate the detected fixation gaze with an area of interest at the lower right corner of the displayed image1000A, or capture area1000A, and perform the panning action so that the area of user interest is centered on the display of the image1000A, or capture area1000A, as shown inFIG.2H. In some implementations, the panning action may be performed at a rate such that the user can visually follow and maintain the gaze on the object of interest.

As noted above, in a system and method in accordance with implementations described herein, user gaze may be tracked to detect a fixation gaze relative to an image, or a capture area depicting content captured within a field of view of a world facing camera of a wearable computing device. A fixation point of the fixation gaze may be correlated to a pixel area, or a pixel of the image, or capture area. The pixel area, or pixel corresponding to the fixation point may in turn be used to identify the object of focus of the fixation gaze.

In detecting the fixation gaze, the gaze tracking device(s) may track eye movement, including natural, unaffected eye movement and deliberate eye movement, substantially continuously. Continued sampling of eye gaze direction may generate an eye gaze pattern, and may allow the system to estimate a center of the eye gaze pattern as the pattern closes in on a particular pixel in the image, or capture area captured by the world facing camera and displayed by the display device. This analysis may isolate a pixel area, or a pixel of the image captured by the world facing camera at which the user's eye gaze is fixated. Correlation of the tracked eye gaze direction and/or fixation gaze with the image, or capture area depicting objects in the field of view of the world facing camera of the wearable device may allow the system to determine an object of focus, regardless of a position of the wearable device on the head of the user. This may eliminate complexities and error generated due to differences in fit for different users of the wearable device, movement or slippage of the wearable device and the like. This may also eliminate the need to account for depth which is typically encountered when correlating eye tracking directly to a real world view of the ambient environment, rather than to the image, or capture area provided by the display device. The substantially fixed relationship (for example, on the frame of the wearable computing device) of the world facing camera, the gaze tracking device(s), and the display device, may eliminate the need for depth data, and may eliminate numerous variables due to fit, orientation, user habits and the like, which otherwise add cost, complexity and error into the determination of an object of focus of the user gaze. Calibration of the world facing camera with the display device to establish a known relationship between the capture area per pixel of the camera and the optical axis associated with the display device may further facilitate the accurate correlation of a detected fixation gaze with a particular pixel area, or pixel.

FIG.3is a block diagram of an example system, according to implementations described throughout this disclosure. As shown inFIG.3, in some implementations, the system includes one or more computing devices including at least one wearable computing device300(such as, for example, the wearable computing devices100A-100D described above, and/or other wearable computing devices). The wearable computing device300can communicate selectively via a network306to access external resources302such as, for example, server computer systems, processors, databases, memory storage, and the like. The wearable computing device300can operate under the control of a control system370. The at least one wearable computing device300can communicate with an external computing device304(another wearable computing device, another mobile computing device and the like) either directly (via wired and/or wireless communication), or via the network306. In some implementations, the wearable computing device300includes a communication module380to facilitate external communication. In some implementations, the wearable computing device300includes a sensing system320including various sensing system components including, for example one or more gaze tracking sensors322including, for example image sensors, one or more position/orientation sensor(s)324including for example, an inertial measurement unit, accelerometer, gyroscope, magnetometer and the like, and one or more audio sensors326that can detect audio input. The wearable computing device300can include more, or fewer, sensing devices and/or combinations of sensing devices.

In some implementations, the wearable computing device300may include an output system340including, for example, one or more display devices that can display still and/or moving image content and one or more audio output devices that can output audio content. In some implementations, the wearable computing device300may include one or more camera(s)360. The camera(s)360may be, for example, outward facing, or world facing cameras that can capture still and/or moving images of an environment outside of the wearable computing device300. The still and/or moving images may be displayed by the display device of the output system340and/or transmitted externally via the communication module380and the network306, and/or stored in a memory330of the wearable computing device300. The wearable computing device300may include one or more processors350, which may be formed in a substrate configured to execute one or more machine executable instructions or pieces of software, firmware, or a combination thereof. The processor(s)350can be semiconductor-based that include semiconductor material that can perform digital logic. The memory330may include any type of storage device that stores information in a format that can be read and/or executed by the processor(s)350. The memory330may store applications and modules that, when executed by the processor(s)350, perform certain operations. In some examples, the applications and modules may be stored in an external storage device and loaded into the memory330.

AlthoughFIG.3illustrates a single wearable computing device300, the implementations described herein may encompass any number of such systems (e.g., more than two). Similarly, the devices depicted inFIGS.1A-1C and2A-2Gmay include (or have access to) any or all of the elements of the system shown inFIG.3.

FIG.4is a flowchart of an example process400to identify an object for selection based on a detected fixation gaze directed to a display of a wearable computing device, and correlation of the detected fixation gaze to an object for interaction, in accordance with implementations described herein. In general, process400utilizes the systems and algorithms described herein to enable a wearable computing device, such as the wearable computing devices described herein, to identify an object for selection by a user based on a detected fixation gaze on an image of an environment outside of the wearable computing device. The process400may utilize a computing system with at least one processing device and memory storing instructions that when executed cause the processing device to perform the plurality of operations and computer implemented steps described in the claims.

The process400includes operating a wearable computing device including a world facing camera and a display device including a display that is visible to the user wearing the wearable computing device (block410). The world facing camera can capture still and/or moving images of an environment outside of the wearable computing device, and the display device can display the images captured by the camera. In this example, the wearable computing device may be the wearable computing device100and/or the wearable computing device300described above with respect toFIGS.1A through3. In response to detection of operation in a select mode (block430), one or more gaze tracking devices of the wearable computing device may be operated (block440). In the select mode, the gaze tracking device(s) may operate to track eye gaze of the user relative to the images, displayed on the display device, of the ambient environment within the field of view of the camera. In the select mode, the gaze tracking device(s) may track eye gaze to detect gaze fixation relative to the image of the ambient environment displayed by the display device (block450). Gaze fixation may be detected when, for example, detected eye gaze motion is fixated on a portion of the displayed image, and thus eye gaze motion is minimal, and/or is below a set threshold of movement. For example, gaze fixation may be detected when detected eye gaze movement is less than a set number of degrees from an optical axis. In response to detection of gaze fixation (block450), a gaze fixation point may be mapped to a pixel area, or a pixel, of the image of the ambient environment displayed by the display device (block460). The mapped pixel area, or pixel, may be used to identify an object that is the focus of the user gaze, and to set the identified object for selection (block470).

FIG.5shows an example of a computer device500and a mobile computer device550, which may be used with the techniques described here. Computing device500is intended to represent various forms of digital computers, such as laptops, desktops, tablets, workstations, personal digital assistants, smart devices, appliances, electronic sensor-based devices, televisions, servers, blade servers, mainframes, and other appropriate computing devices. Computing device550is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The memory504stores information within the computing device500. In one implementation, the memory504is a volatile memory unit or units. In another implementation, the memory504is a non-volatile memory unit or units. The memory504may also be another form of computer-readable medium, such as a magnetic or optical disk. In general, the computer-readable medium may be a non-transitory computer-readable medium.

The processor552can execute instructions within the computing device550, including instructions stored in the memory564. The processor may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor may provide, for example, for coordination of the other components of the device550, such as control of user interfaces, applications run by device550, and wireless communication by device550.

The computing device550may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone583. It may also be implemented as part of a smart phone581, personal digital assistant, or other similar mobile device.

In some embodiments, the computing devices depicted inFIG.5can include sensors that interface with an AR/VR headset (AR glasses/eyeglasses/VR headset/AR headset/HMD device590). For example, one or more sensors included on computing device550or other computing device depicted inFIG.5, can provide input to AR/VR headset590or in general, provide input to an AR/VR space. The sensors can include, but are not limited to, a touchscreen, accelerometers, gyroscopes, pressure sensors, biometric sensors, temperature sensors, humidity sensors, and ambient light sensors. Computing device550can use the sensors to determine an absolute position and/or a detected rotation of the computing device in the AR/VR space that can then be used as input to the AR/VR space. For example, computing device550may be incorporated into the AR/VR space as a virtual object, such as a controller, a laser pointer, a keyboard, a weapon, etc. Positioning of the computing device/virtual object by the user when incorporated into the AR/VR space can allow the user to position the computing device to view the virtual object in certain manners in the AR/VR space.

In some embodiments, one or more input devices included on, or connected to, the computing device550can be used as input to the AR/VR space. The input devices can include, but are not limited to, a touchscreen, a keyboard, one or more buttons, a trackpad, a touchpad, a pointing device, a mouse, a trackball, a joystick, a camera, a microphone, earphones or buds with input functionality, a gaming controller, or other connectable input device. A user interacting with an input device included on the computing device550when the computing device is incorporated into the AR/VR space can cause a particular action to occur in the AR/VR space.

In some embodiments, one or more output devices included on the computing device550can provide output and/or feedback to a user of the AR/VR headset590in the AR/VR space. The output and feedback can be visual, tactical, or audio. The output and/or feedback can include, but is not limited to, rendering the AR/VR space or the virtual environment, vibrations, turning on and off or blinking and/or flashing of one or more lights or strobes, sounding an alarm, playing a chime, playing a song, and playing of an audio file. The output devices can include, but are not limited to, vibration motors, vibration coils, piezoelectric devices, electrostatic devices, light emitting diodes (LEDs), strobes, and speakers.

In some embodiments, computing device550can be placed within AR/VR headset590to create an AR/VR system. AR/VR headset590can include one or more positioning elements that allow for the placement of computing device550, such as smart phone581, in the appropriate position within AR/VR headset590. In such embodiments, the display of smart phone581can render stereoscopic images representing the AR/VR space or virtual environment.

In some embodiments, the computing device550may appear as another object in a computer-generated, 3D environment. Interactions by the user with the computing device550(e.g., rotating, shaking, touching a touchscreen, swiping a finger across a touch screen) can be interpreted as interactions with the object in the AR/VR space. As just one example, computing device can be a laser pointer. In such an example, computing device550appears as a virtual laser pointer in the computer-generated, 3D environment. As the user manipulates computing device550, the user in the AR/VR space sees movement of the laser pointer. The user receives feedback from interactions with the computing device550in the AR/VR environment on the computing device550or on the AR/VR headset590.

In some embodiments, a computing device550may include a touchscreen. For example, a user can interact with the touchscreen in a particular manner that can mimic what happens on the touchscreen with what happens in the AR/VR space. For example, a user may use a pinching-type motion to zoom content displayed on the touchscreen. This pinching-type motion on the touchscreen can cause information provided in the AR/VR space to be zoomed. In another example, the computing device may be rendered as a virtual book in a computer-generated, 3D environment. In the AR/VR space, the pages of the book can be displayed in the AR/VR space and the swiping of a finger of the user across the touchscreen can be interpreted as turning/flipping a page of the virtual book. As each page is turned/flipped, in addition to seeing the page contents change, the user may be provided with audio feedback, such as the sound of the turning of a page in a book.

In some embodiments, one or more input devices in addition to the computing device (e.g., a mouse, a keyboard) can be rendered in a computer-generated, 3D environment. The rendered input devices (e.g., the rendered mouse, the rendered keyboard) can be used as rendered in the AR/VR space to control objects in the AR/VR space.

The computer system (e.g., computing device) may be configured to wirelessly communicate with a network server over a network via a communication link established with the network server using any known wireless communications technologies and protocols including radio frequency (RF), microwave frequency (MWF), and/or infrared frequency (IRF) wireless communications technologies and protocols adapted for communication over the network.

In accordance with aspects of the disclosure, implementations of various techniques described herein may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. Implementations may be implemented as a computer program product (e.g., a computer program tangibly embodied in an information carrier, a machine-readable storage device, a computer-readable medium, a tangible computer-readable medium), for processing by, or to control the operation of, data processing apparatus (e.g., a programmable processor, a computer, or multiple computers). In some implementations, a tangible computer-readable storage medium may be configured to store instructions that when executed cause a processor to perform a process. A computer program, such as the computer program(s) described above, may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may be deployed to be processed on one computer or on multiple computers at one site or distributed across multiple sites and interconnected by a communication network.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments, however, may be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

It will be understood that although the terms “first,” “second,” etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Thus, a “first” element could be termed a “second” element without departing from the teachings of the present embodiments.