TAGGING USING EYE GAZE DETECTION

Various embodiments relating to tagging human subjects in images are provided. In one embodiment, an image including a human subject is presented on a display screen. A dwell location of a tagging user's gaze on the display screen is received. The human subject in the image is recognized as being located at the dwell location. An identification of the human subject is received, and the image is tagged with the identification.

DETAILED DESCRIPTION

The present disclosure relates to tagging images with metadata, such as identification of human subjects depicted in images. More particularly, the present disclosure relates to tagging human subjects in images using eye gaze tracking based selection. In one example, the present disclosure provides mechanisms that enable receiving a dwell location of a tagging user's gaze on an image presented on a display screen, recognizing that a human subject in the image is located at the dwell location, receiving an identification of the human subject, and tagging the image with the identification. Typically, humans are attuned to recognizing patterns, such as faces of other humans. Accordingly, a user may select a human subject in an image by looking at the human subject quite faster than selecting the human subject in the image with a mouse or touch input.

Furthermore, in some embodiments, the present disclosure provides mechanisms to receive a name of the human subject recognized in the image from a voice recognition system that listens for the name being spoken by the tagging user. The recognized name may be mapped to the image to tag the human subject. By using voice recognition to tag a name of a recognized human subject to an image, a tagging user may avoid having to type the name on a keyboard. Accordingly, a large volume of images may be tagged in a timelier and less labor intensive manner relative to a tagging approach that uses a mouse and keyboard.

FIG. 1shows a computing system100according to an embodiment of the present disclosure. The computing system100may include a user input device102, a computing device104, and a display device106.

The user input device102may include an eye tracking camera108configured to detect a direction of gaze or location of focus of one or more eyes110of a user112(e.g., a tagging user). The eye tracking camera108may be configured to determine a user's gaze in any suitable manner. For example, in the depicted embodiment, the user input device102may include one or more glint sources114, such as infrared light sources, configured to cause a glint of light to reflect from each eye110of the user112. The eye tracking camera108may be configured to capture an image of each eye110of the user112including the glint. Changes in the glints from the user's eyes as determined from image data gathered via the eye tracking camera may be used to determine a direction of gaze. Further, a location116at which gaze lines projected from the user's eyes intersect a display screen118of the display device106may be used to determine an object at which the user is gazing (e.g., a displayed object at a particular location).

Furthermore, the user input device102may include a microphone120(or other suitable audio detection device) configured to detect the user's voice. More particularly, the microphone120may be configured to detect the user's speech, such as a voice command. It is to be understood that the microphone may detect the user's speech in any suitable manner.

The user input device102may be employed to enable the user112to interact with the computing system100via gestures of the eye, as well as via verbal commands. It is to be understood that the eye tracking camera108and the microphone120are shown for the purpose of example and are not intended to be limiting in any manner, as any other suitable sensors and/or combination of sensors may be utilized.

The computing device104may be in communication with the user input device102and the display device106. The computing device104may be configured to receive and interpret inputs from the sensors of the user input device102. For example, the computing device104may be configured to track the user's gaze on the display screen118of the display device106based on eye images received from the eye tracking camera108. More particularly, the computing device104may be configured to detect user selection of one or more objects displayed on the display screen (e.g., a human subject in an image) based on establishing a dwell location. The computing device104may be configured to process voice commands received from the user input device102to recognize a particular word or phrase (e.g., a name of a selected human subject). The computing device104may be configured to perform actions or commands on selected objects based on the processed information received from the user input device (e.g., tagging a human subject in an image with a name).

It will be appreciated that the depicted devices in the computing system are described for the purpose of example, and thus are not meant to be limiting. Further, the physical configuration of the computing system and its various sensors and subcomponents may take a variety of different forms without departing from the scope of the present disclosure. For example, the user input device, the computing device, and the display device may be integrated into a single device, such as a mobile computing device.

FIG. 2schematically shows a block diagram of a computer architecture200according to an embodiment of the present disclosure. The computer architecture200may enable tagging of a human subject in an image presented on a display screen using gaze detection of a tagging user to select the human subject, and voice recognition to recognize a name of the selected human subject to be tagged. For example, the computer architecture may be implemented in the computing system100ofFIG. 1.

In one example, the eye tracking camera108may provide eye images of the tagging user's eyes to an eye tracking service202. The eye tracking service202may be configured to interpret the eye images to determine the tagging user's eye gaze on a display screen. More particularly, the eye tracking service202may be configured to determine whether the tagging user's gaze is focused on a location of the display screen for greater than a threshold duration (e.g., 100 microseconds). If the user's gaze is focused on the location for greater than the threshold duration, then the eye tracking service202may be configured to generate a dwell location signal that is sent to a client application204.

The client application204may be configured to receive the dwell location signal from the eye tracking service202. The dwell location signal may include display screen coordinates of the dwell location. The client application204may be configured to determine whether a human subject in an image presented on the display screen is located at the dwell location. If a human subject is recognized as being located at the dwell location, the client application204may be configured to provide visual feedback to the tagging user that the human subject is recognized or selected. For example, the client application204may be configured to display a user interface on the display screen that facilitates provision or selection of a name to tag the image of the human subject. For example, the client application204may be configured to prompt a user to provide a name for the human subject and command a voice recognition service206to listen for a name being spoken by the tagging user via the microphone120.

It is to be understood that the client application204may be any suitable application that is configured to associate metadata with an image (i.e., tagging). In one example, the client application may be a photograph editing application. As another example, the client application may be a social networking application.

The microphone120may be configured to detect a voice command from the tagging user, and send the voice command to the voice recognition service206for processing. The voice recognition service206may be configured to recognize a name from the voice command, and send the name as identification of the human subject to the client application204. The client application204may be configured to tag the image with the identification.

In some embodiments, identification for tagging of the human subject may be provided without voice recognition. For example, identification may be provided merely through gaze detection. In one example, the client application204may be configured to display a set of previously recognized names on the display screen responsive to a human subject being recognized as being positioned at a dwell location. The client application204may be configured to receive a different dwell location of the tagging user's gaze on the display screen, recognize that a name from the set of previously recognized names is located at the different dwell location, and select the name as the identification of the human subject in the image.

It is to be understood that the set of previously recognized names may be populated in any suitable manner. For example, the set of previously recognized names may be populated by previous tagging operations, social networking relationships of the tagging user, closest guesses based on facial recognition, etc.

In some embodiments, the client application204may be configured to determine whether the name received from the voice recognition service206(or via another user input) has been previously recognized by comparing the name to the set of previously recognized names. If the name has not been previously recognized, then the client application204may be configured to add the name to the set of previously recognized names. For example, the set of previously recognized names may be used to speed up name recognition processing by the voice recognition service, among other operations. In one example, mapping of names to human subjects may be made more accurate by having a smaller list of possible choices (e.g., the set of previously recognized names).

In some embodiments, the client application204may be configured to display different images that potentially include the recognized human subject on the display screen in order to perform additional tagging operations. For example, the client application204may be configured to identify a facial pattern of the recognized human subject, run a facial pattern recognition algorithm on a plurality of images to search for the facial pattern of the recognized human subject, and display different images that potentially include the facial pattern of the recognized human subject on the display screen. Furthermore, the client application204may be configured to prompt the tagging user to confirm whether a human subject in a different image is the recognized human subject. If a confirmation that the recognized human subject is in the different image is received (e.g., via a vocal confirmation from the tagging user detected by the microphone120or a gaze dwelling on a confirmation button for a threshold duration), then the client application204may be configured to tag the different image with the name of the human subject. The client application204may be configured to repeat the process for all images that potentially include the recognized human subject. In this way, the plurality of images may be tagged in a quicker and less labor intensive manner than a tagging approach that uses a mouse and keyboard.

It is to be understood that, in some embodiments, the eye tracking service202and the voice recognition service206may be implemented as background services that may be continuously operating to provide the dwell location and recognized name to a plurality of different client applications (e.g., via one or more application programming interfaces (APIs)). In some embodiments, the eye tracking service202and the voice recognition service206may be incorporated into the client application204.

FIGS. 3-5show various examples of visual feedback that may be provided to a tagging user to indicate that a human subject in an image is recognized as being positioned at a dwell location of the tagging user's gaze. For example, the visual feedback may be provided in a graphical user interface that may be generated by the client application204shown inFIG. 2.

FIG. 3shows an image300including three human subjects. The middle human subject is recognized as being positioned at a dwell location by visual feedback302in the form a box surrounding a head of the human subject. The box highlights the selection of the middle human subject by the tagging user's gaze.

FIG. 4shows the same image300as shown inFIG. 3. However, in this example, the visual feedback304includes graying out the image surrounding the head of the human subject that is recognized as being positioned at the dwell location.

FIG. 5shows the same image300as shown inFIG. 3. However, in this example, the visual feedback306includes enlarging the head of the human subject that is recognized as being positioned at the dwell location relative to the remainder of the image. It is to be understood that any suitable visual feedback may be provided to a tagging user to indicate selection of a human subject in an image based on the tagging user's gaze.

FIG. 6schematically shows a tagging interface600for tagging a human subject in an image. For example, the tagging interface may be generated by the client application204shown inFIG. 2. The image602includes the human subject604recognized as being positioned at a dwell location of a tagging user's gaze via visual feedback606in the form of a box surrounding a head of the human subject604. In response to the human subject604being recognized, a tag prompt608may be displayed in the tagging interface600that prompts the tagging user to provide or select an identification of the human subject.

In some embodiments, in response to the tag prompt608being displayed, the voice recognition service may be signaled to listen for a name being spoken by the tagging user via the microphone. If the voice recognition service detects a name, then the image may be tagged with the name.

In some embodiments, a set of previously recognized names610may be displayed in the tagging interface600to aid a user in providing or selecting an identification of the human subject604. In some embodiments, a name612of the set of previously recognized names610may be selected as the identification of the human subject when the name612is recognized as being positioned at a dwell location of the tagging user's gaze on the display screen (e.g., the user's gaze may remain at the location of the name for greater than a first threshold duration). In other words, after the tagging user is prompted to provide an identification of the human subject, the tagging user merely looks at the name long enough to establish a dwell location signal in order to select the name.

In some embodiments, visual feedback may be provided in response to recognizing that the name612is located at the dwell location of the user's gaze. For example, the visual feedback may include highlighting the name, displaying a box around the name, displaying a cursor or other indicator pointing at the name, bolding the name, or otherwise modifying the name, etc. Once the visual feedback has been provided, the name may be selected as identification of the human subject in response to the gaze remaining on the name for a second threshold duration. The second threshold duration may start after the first threshold duration has concluded. For example, the second threshold duration may begin when the visual feedback that the name is recognized is provided.

The above described approach allows for the recognition of a human subject in an image and tagging of the image with the identification of the human subject to be done with only gaze detection and without any speaking or use of a mouse and/or keyboard. Moreover, the approach may be employed to tag a plurality of images only using gaze detection.

It is to be understood that, in some cases, the set of previously recognized names610need not include all previously recognized names, but may be a subset with only the closest guesses based on facial recognition or the like. In other cases, the set of previously recognized names may include all names that have been previously recognized. Furthermore, it is to be understood that the set of previously recognized names610may be displayed regardless of whether the tagging user provides an identification of the human subject via voice command or by gazing at a name in the set of previously recognized names.

Furthermore, in some embodiments, if a new name614is received as the identification of the human subject that is not included in the set of previously recognized names610, the new name614may be added to the set of previously recognized names for future image tagging operations.

In some embodiments, when an image is tagged with an identification of a human subject, the identification may be associated with the entire image. In some embodiments, when an image is tagged with an identification of a human subject, the identification may be associated with a portion of the image that includes the human subject. For example, in the illustrated embodiment, the identification of the human subject604may be associated with the portion of the image contained by the visual feedback606(or the portion of the image occupied by the human subject). Accordingly, an image including a plurality of human subjects may be tagged with different identifications for each of the plurality of human subjects, and the different identifications may be associated with different portions of the image.

FIG. 7schematically shows a tagging interface700for tagging a recognized human subject in different images. For example, the tagging interface700may be displayed on the display screen once a human subject has been recognized in an image, such as in tagging interface600shown inFIG. 6, and a facial pattern recognition algorithm has been run to identify images that potentially include the facial pattern of the human subject. The tagging interface700may include an instance of the recognized human subject702(e.g., extracted from the original image). The tagging interface700may include a plurality of images704that potentially include the recognized human subject702. A confirmation prompt706may be displayed in the tagging interface700to prompt the tagging user to confirm whether a human subject in each of the plurality of images matches the recognized human subject702.

In some embodiments, the tagging user may provide confirmation by establishing a dwell location on an image and providing a vocal confirmation, such as by saying “YES.” If a vocal confirmation is received, then the image may be tagged with the identification of the recognized human subject. On the other hand, the tagging user may say “NO” if the image does not include the recognized human subject. Alternatively or additionally, the tagging user may provide a name of the person in the image, and the image may be tagged with the name.

In some embodiments, the tagging user may provide confirmation by establishing a dwell location on a confirmation indicator (e.g., “YES”)708of an image. If a visual confirmation is received, then the image may be tagged with the identification of the recognized human subject. On the other hand, the tagging user may establish a dwell location on a denial indicator (e.g., “NO”)710if the image does not include the recognized human subject. Each image may have corresponding confirmation and denial indicators so that the plurality of images may be visually tagged in a quick manner.

FIG. 8shows a method800for tagging a human subject in an image presented on a display screen according to an embodiment of the present disclosure. For example, the method800may be performed by the computing system100shown inFIG. 1, and more particularly, the computer architecture200shown inFIG. 2

At802, the method800may include receiving a dwell location of a tagging user's gaze on a display screen.

At804, the method800may include recognizing that a human subject in an image displayed on the display screen is located at the dwell location.

At806, the method800may include providing visual feedback that the human subject is recognized as being at the dwell location.

At808, the method800may include receiving an identification of the human subject. For example, the identification may include a name of the human subject. However, it is to be understood that the identification may include any suitable description or characterization.

At810, the method800may include tagging the image with the identification. In some embodiments, the identification may be associated with the entire image. In some embodiments, the identification may be associated with a portion of the image that just corresponds to the human subject.

At812, the method800may include displaying a different image that potentially includes the human subject on the display screen.

At814, the method800may include determining whether confirmation that the different image includes the human subject is received. If a confirmation that the human subject is in the different image is received, then the method800moves to816. Otherwise, the method800returns to other operations.

At816, the method800may include tagging the different image with the identification.

At818, the method800may include determining whether there are any more images that potentially include the human subject to be confirmed and/or tagged with the identification. If there are more images that potentially include the human subject to be confirmed, then the method800returns to812. Otherwise, the method800returns to other operations.

FIG. 9shows a method900for establishing a dwell location of a tagging user's gaze according to an embodiment of the present disclosure. For example, the method900may be performed by the computing system100shown inFIG. 1, and more particularly, the computer architecture200shown inFIG. 2. For example, the method900may be performed to provide a dwell location for step802of the method800shown inFIG. 8.

At902, the method900may include tracking a tagging user's gaze on a display screen. For example, the tagging user's gaze may be tracked by the eye tracking camera108shown inFIGS. 1 and 2.

At904, the method900may include determining whether the tagging user's gaze remains at a location on the display screen for greater than a first threshold duration (e.g., 100 microseconds). If it is determined that the tagging user's gaze remains at the location on the display screen for greater than the first threshold duration, then the method900moves to906. Otherwise, the method900returns to904.

At906, the method900may include establishing the dwell location at the location on the display screen where the tagging user's gaze remained for greater than the first threshold duration. In one example, the dwell location may be established by the eye tracking service202and sent to the client application204.

FIG. 10shows a method1000for recognizing identification of a human subject according to an embodiment of the present disclosure. For example, the method1000may be performed by the computing system100shown inFIG. 1, and more particularly, the computer architecture200shown inFIG. 2. For example, the method1000may be performed to provide an identification of a human subject for step808of the method800shown inFIG. 8, among other method steps.

At1002, the method1000may include determining whether a name of a human subject is received from a voice recognition system that listens for a name being spoken. If a name is received from the voice recognition system, then the method1000moves to1004. Otherwise, the method1000returns to other operations.

At1004, the method1000may include determining whether the name received as identification of the human subject is a new name or a previously recognized name. If a new name that is not included in set of previously recognized names is received, then the method1000moves to1006. Otherwise, the method1000returns to other operations.

At1006, the method1000may include adding the new name to a set of previously recognized names.

The above described method may be performed using a voice recognition system to receive a name as identification of a human subject recognized via detection of a tagging user's gaze.

FIG. 11shows a method1100for recognizing identification of a human subject according to another embodiment of the present disclosure. For example, the method1100may be performed by the computing system100shown inFIG. 1, and more particularly, the computer architecture200shown inFIG. 2. For example, the method1100may be performed to provide an identification of a human subject for step808of the method800shown inFIG. 8, among other method steps.

At1102, the method1100may include displaying a set of previously recognized names on the display screen. In some embodiments, the set of previously recognized names may be displayed on the display screen in response to the human subject being recognized as being located at a dwell location of the tagging user's gaze.

At1104, the method1100may include receiving a dwell location of the tagging user's gaze on the display screen.

At1106, the method1100may include recognizing that a name from the set of previously recognized names is located at the dwell location. For example, the user's gaze may remain at the location of the name on the display screen for greater than a first threshold duration (e.g., 100 microseconds).

At1108, the method1100may include providing visual feedback that the name is recognized as being at the dwell location. For example, a cursor or other indicator may point to the name or the name may be bolded, highlighted, or otherwise modified to indicate the visual feedback.

At1110, the method1100may include determining whether the user's gaze remain at the dwell location for greater than a second threshold duration (e.g., 100 microseconds. The second threshold duration may begin once the first threshold duration has concluded, such as when the visual feedback that the name is recognized is provided. The second threshold duration may be employed to aid the user in making an accurate selection. If the user's gaze remains at the dwell location for greater than the second threshold duration, then the method1100moves to1112. Otherwise, the method1100returns to other operations.

At1112, the method1100may include selecting the name as the identification in response to recognizing that the name is located at the dwell location.

The above described method may be performed to select a name as an identification of a human subject only using gaze detection. It is to be understood that such an approach may be performed while the user is silent and still (e.g., no mouth, head, or hand motion).

The above described methods may be performed to tag images in a manner that is quicker and less labor intensive then a tagging approach that uses a keyboard and mouse. It is to be understood that the methods may be performed at any suitable time. For example, the methods may be performed while taking a photograph, or just after taking a photograph, such tagging may be performed using a camera or mobile device. As another example, the tagging methods may be performed as a post processing operation, such as on a desktop or tablet computer. Moreover, it is to be understood that such methods may be incorporated into any suitable application including image management software, social networking applications, web browsers, etc.

While the tagging approach has been discussed in the particular context of recognizing a human subject and providing a name as identification of the human subject, it is to be understood that such concepts are broadly applicable to recognizing any suitable object and providing any suitable identification of that object.

FIG. 12schematically shows a non-limiting embodiment of a computing system1200that can enact one or more of the methods and processes described above. Computing system1200is shown in simplified form. Computing system1200may take the form of one or more personal computers, server computers, tablet computers, home-entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smart phone), and/or other computing devices. For example, computing system1200may be representative of any or all of the computing devices in the computing system100shown inFIG. 1. Further, the computing system1200may be configured to implement the computer architecture200shown inFIG. 2.

Computing system1200includes a logic machine1202and a storage machine1204. Computing system1200may optionally include a display subsystem1206, input subsystem1208, communication subsystem1210, and/or other components not shown inFIG. 12.

Storage machine1204includes one or more physical devices configured to hold instructions executable by the logic machine to implement the methods and processes described herein. When such methods and processes are implemented, the state of storage machine1204may be transformed—e.g., to hold different data.

Aspects of logic machine1202and storage machine1204may be integrated together into one or more hardware-logic components. Such hardware-logic components may include field-programmable gate arrays (FPGAs), program- and application-specific integrated circuits (PASIC/ASICs), program- and application-specific standard products (PSSP/ASSPs), system-on-a-chip (SOC), and complex programmable logic devices (CPLDs), for example.

When included, display subsystem1206may be used to present a visual representation of data held by storage machine1204. This visual representation may take the form of a graphical user interface (GUI). As the herein described methods and processes change the data held by the storage machine, and thus transform the state of the storage machine, the state of display subsystem1206may likewise be transformed to visually represent changes in the underlying data. Display subsystem1206may include one or more display devices utilizing virtually any type of technology. Such display devices may be combined with logic machine1202and/or storage machine1204in a shared enclosure, or such display devices may be peripheral display devices.