GAZE TRACKING FOR USER INTERFACE

Gaze tracking to track gaze of a user within a user view screen including: combining the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen; dividing the user view screen into a plurality of gaze zones when gaze zone estimation is turned on; and combining the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user.

BACKGROUND

Field

The present disclosure relates to gaze tracking, and more specifically, to gaze tracking using eye movements as user inputs.

Background

Sometimes human eyes may move or twitch in different directions without any specific purpose or reason. Thus, detailed gaze or eye tracking is generally not used as inputs in games or other application programs since it may generate high fidelity noise.

SUMMARY

The present disclosure provides for gaze tracking to track gaze of a user within a user view screen.

In one implementation, a method for gaze tracking to track gaze of a user within a user view screen is disclosed. The method includes combining the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen. The method also includes dividing the user view screen into a plurality of gaze zones when gaze zone estimation is turned on. The method further includes combining the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user.

In another implementation, a game device is disclosed. The game device includes a sensor and a processor. The sensor tracks gaze movements of a user. The processor is operable (a) to obtain information about the gaze movements from the sensor and (b) to execute an application program to display a gaze icon based on the gaze information and a notification icon within a screen image of the application program on a display. The notification icon is displayed at a predetermined position of the screen image.

In yet another implementation, a system for gaze tracking to track gaze of a user within a user view screen includes a gaze determiner, a processor, and a user interface. The processor combines the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen. The gaze determiner directs the processor to divide the user view screen into a plurality of gaze zones, and to combine the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user, when the gaze determiner determines that gaze zone estimation is turned on.

In a further implementation, a non-transitory computer-readable storage medium storing a computer program to track gaze of a user within a user view screen is disclosed. The computer program includes executable instructions that cause a computer to combine the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen. The computer program also includes executable instructions that cause a computer to divide the user view screen into a plurality of gaze zones when gaze zone estimation is turned on. The computer program further includes executable instructions that cause a computer to combine the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user.

Other features and advantages should be apparent from the present description which illustrates, by way of example, aspects of the disclosure.

DETAILED DESCRIPTION

As described above, human eyes may move or twitch in different directions without any specific purpose or reason. Thus, detailed gaze or eye tracking is generally not used as inputs in games or interactive programs since it may generate high fidelity noise. However, in certain situations, gaze tracking can be used in coordination or conjunction with other methods and systems to enable it to be used with user interface.

Certain implementations of the present disclosure provide methods and systems for gaze tracking for user interface. In one implementation, the gaze tracking is combined with head movement tracking to provide fine tuning. In another implementation, gaze zone estimation is used to mitigate high fidelity noise introduced in detailed eye tracking. Thus, in the gaze zone estimation, one or more gaze zones may be expanded or highlighted in overlay picture(s) or picture-in-picture (PIP) mode. In a further implementation, the gaze tracking is used to provide notification to the user such that the user is minimally disrupted while, for example, playing a game or experiencing other application programs.

After reading the below descriptions, it will become apparent how to implement the disclosure in various implementations and applications. Although various implementations of the present disclosure will be described herein, it is understood that these implementations are presented by way of example only, and not limitation. As such, the detailed description of various implementations should not be construed to limit the scope or breadth of the present disclosure.

FIG.1is a flow diagram of a process100for gaze tracking for user interface in accordance with one implementation of the present disclosure. In one implementation, the process100enables gaze tracking for user interface by combining the gaze tracking with other methods and systems including head movement tracking and gaze zone estimation. In one implementation, the process100is programmed into a hardware processor to perform gaze tracking within a combined system.

In the illustrated implementation ofFIG.1, a determination is made, at step110, whether head movement tracking is turned on. If it determined, at step110, that the head movement tracking is turned on, the gaze tracking is combined with the head movement tracking, at step120, to provide fine tuning. For example, the head movement tracking provides a rough estimate of the direction in which the user's head is facing, while the gaze tracking provides more detailed estimation of the direction of the user's head.

A determination is made, at step130, whether gaze zone estimation is on. In one implementation, the gaze zone estimation divides the user view screen into zones (e.g., a preset size of a rectangular box), at step140, and uses the gaze tracking to estimate the direction of the user's gaze to be within a selected zone. If it determined, at step130, that the gaze zone estimation is on, the gaze tracking is combined with the gaze zone estimation, at step150, to select a gaze zone. In one implementation, one or more gaze zones (including the selected zone) may be expanded or highlighted in overlay picture(s) or picture-in-picture (PIP) mode to show the detail view of the objects or persons within the user's gaze.

A determination is then made, at step160, whether a notification is received. In one implementation, the notification includes communication to the user while the user is playing or experiencing a game or other interactive application program. If it determined, at step160, that the notification is received, a notice is provided on the user view screen, at step170, indicating that the communication is received. In one implementation, the communication includes message(s) from other users playing or experiencing the same game or interactive application program with the user. In another implementation, the communication includes message(s) from the game system. In a further implementation, the notice is provided on the user view screen as a bubble icon. In one implementation, when the user gazes in the direction of the icon, the details of the message are displayed on the user view screen. The above-described notification method enables providing notification to a user playing, for example, an interactive game, with minimal disruption.

In a particular implementation, a method for gaze tracking to track gaze of a user within a user view screen is disclosed. The method includes: combining the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen; dividing the user view screen into a plurality of gaze zones when gaze zone estimation is turned on; and combining the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user.

In one implementation, the method further includes displaying an icon on the user view screen when the gaze tracking is turned on. In one implementation, each gaze zone of the plurality of gaze zones is a preset size of a rectangular box. In one implementation, one or more gaze zones around the selected zone is expanded or highlighted in an overlay picture or picture-in-picture mode to show a detail view of objects or persons within the gaze of the user. In one implementation, the method further includes displaying a notification icon on the user view screen when a notification is received. In one implementation, the notification includes a message from a game system. In one implementation, the notification includes a message from another user in the game system. In one implementation, the method further includes displaying the message on the user view screen when it is determined that the gaze of the user has moved to the notification icon.

FIG.2is a block diagram of a gaze tracking system200for gaze tracking for user interface in accordance with one implementation of the present disclosure. In the illustrated implementation ofFIG.3, the system200includes a gaze determiner220, a processor230, and a user interface240coupled to the processor230and a game system210. In one implementation, the blocks220,230,240of the system200are configured entirely with hardware including one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. In one implementation, the system200combines gaze tracking with other methods including at least one of: head movement tracking, gaze zone estimation, and message notification.

In the illustrated implementation ofFIG.2, the gaze determiner220determines whether head movement tracking is turned on. If it is determined that the head movement tracking is turned on, the gaze determiner220directs the processor230to combine the gaze tracking with the head movement tracking to provide fine tuning. For example, the head movement tracking provides a rough estimate of the direction in which the user's head is facing, while the gaze tracking provides more detailed estimation of the direction of the user's head. In one implementation, the head movement tracking is performed by accelerometer(s) and/or gyroscope(s) disposed within the headset. In another implementation, the gaze tracking is performed by camera(s) disposed within the headset.

The gaze determiner220then determines whether gaze zone estimation is on. In one implementation, the gaze zone estimation divides the user view screen into zones (e.g., a preset size of a rectangular box) and uses the gaze tracking to estimate the direction of the user's gaze to be within a selected zone. If it determined that the gaze zone estimation is on, the gaze determiner220directs the processor230to combine the gaze tracking with the gaze zone estimation to select a gaze zone. In one implementation, one or more gaze zones (including the selected zone) may be expanded or highlighted in overlay picture(s) or picture-in-picture (PIP) mode to show the detail view of the objects or persons within the user's gaze.

The gaze determiner220also determines whether a notification is received. In one implementation, the notification includes communication to the user while the user is playing or experiencing a game or interactive application program. If it determined that the notification is received, the gaze determiner220directs the processor230to display a notice on the user view screen indicating that the communication is received. In one implementation, the communication includes message(s) from other users playing or experiencing the same game or interactive application program with the user. In another implementation, the communication includes message(s) from the game system. In a further implementation, the notice is provided on the user view screen as a bubble icon. In one implementation, when the user gazes in the direction of the icon, the details of the message are displayed on the user view screen. Thus, the gaze determiner220and the processor230provide gaze tracking for the user interface240. The above-described notification method provides notification to the user playing, for example, an interactive game, with minimal disruption.

In one implementation, the game system210includes a gaming console providing interactive games or other application programs. In another implementation, the game system210includes an online interactive game or other application program. In yet another implementation, the game system210includes a virtual reality and/or augmented reality interactive game or application program.

In one implementation, the gaze tracking features are used by game developer(s)/central commander(s) to customize the game or experience. For example, in shooting games, the user's gaze may be used to switch weapons or functions of the weapons in real-time.

FIG.3Ais one example of the user view screen300mentioned above in accordance with one implementation of the present disclosure. The user view screen300ofFIG.3Aincludes two icons310,320. Icon310may appear at the beginning of an interactive game, for example, to indicate to the user that the gaze tracking mode is turned on. Icon320appears on the screen300when a notification message is received.

As shown in the example ofFIG.3A, the gaze icon310is displayed on the view screen300to indicate to the user that the gaze tracking mode is turned on. In one implementation, the gaze icon310is an eye-shaped icon. In another implementation, the gaze icon310is a cursor.

In one implementation, a message may be received while the user is playing a game. Thus, when the message is received, a notification icon320may be displayed in a predetermined area of the view screen300. The message which triggers the notification icon320may be related to the game currently being played (e.g., a notice from a game provider), or it may be unrelated to the game (e.g., a text from a friend). When the message is received and the notification icon320is displayed, the gaze icon310may be moved by the user directing the gaze toward the notification icon320.

FIG.3Bshows the user moving the gaze icon310toward the notification icon320by moving the gaze312toward the notification icon320.

FIG.3Cshows the gaze icon310being moved into a decision area322around the notification icon320. Thus, when the gaze icon310is moved into the decision area322, additional information about the message may be displayed.

FIG.3Dshows a view screen350including a message panel330which replaces the notification icon320with details of a received message332, when the gaze icon310is moved into the decision area322as illustrated inFIG.3C. Thus, the implementations ofFIGS.3A through3Denable the user to briefly check the contents of the message332corresponding to the notification icon320, for example, without interrupting the game and without operating a gaming controller used by the user for the game. When the user wishes to perform any additional actions related to the message332, the user can perform the actions using the gaming controller. For example, the user can check the full text of the message332by launching a message application or by replying to the message332. In this example, the game application may then be suspended to perform the additional actions.

In one implementation, when the gaze icon310stays within the decision area322for a predetermined period of time, additional actions related to the message332may be performed. In another implementation, after the gaze icon310moves into the decision area322, the additional actions may be performed by a user's input such as gaming controller or voice command. In one implementation, the decision area322is same or different size as the gaze icon310. In another implementation, the notification icon320changes in size when the gaze icon310moves into the decision area322. In a further implementation, the notification icon320is highlighted to help the user easily understand the status of the message332.

In one implementation, once the notification icon320is selected (e.g., by moving the gaze icon310into the decision area322), a message332is displayed within the message panel330so that the user can check the message. The size of the message panel330may be larger than the notification icon320to accommodate the large size of the message332. In one implementation, the notification icon320and the additional information (e.g., the message panel330and the message332) disappears from the view screen300after a predetermined period of time (after it appears on the screen). Accordingly, the above-described implementations enable gaze operation for the user to check messages without disturbing the game screen or gaming control operations currently being played.

In a particular implementation, a game device is disclosed. The game device includes: a sensor to track gaze movements of a user; a processor operable to: obtain information about the gaze movements from the sensor; execute an application program to display a gaze icon based on the gaze information and a notification icon within a screen image of the application program on a display, wherein the notification icon is displayed at a predetermined position of the screen image.

In one implementation, the application program is a game application. In one implementation, the notification icon receives notification to display from application programs different from the game application. In one implementation, the game device is a head-mounted display (HMD).

In another particular implementation, a system for gaze tracking to track gaze of a user within a user view screen is disclosed. The system includes: a processor to combine the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen; a gaze determiner to direct the processor to divide the user view screen into a plurality of gaze zones, and to combine the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user, when the gaze determiner determines that gaze zone estimation is turned on.

In one implementation, the head movement tracking is performed by at least one accelerometer and at least one gyroscope disposed within a headset communicating with a game system. In one implementation, the gaze tracking is performed by at least one camera disposed within the headset. In one implementation, the system further includes a user interface coupled to the processor, the user interface to display an icon on the user view screen when the gaze tracking is turned on. In one implementation, the user interface also displays a notification icon on the user view screen when a notification is received. In one implementation, the notification includes a message from a game system. In one implementation, the notification includes a message from another user in a game system. In one implementation, the game system comprises a gaming console providing interactive games.

FIG.4Ais a representation of a computer system400and a user402in accordance with an implementation of the present disclosure. The user402uses the computer system400to implement a gaze tracking application490for gaze tracking for user interface with respect to the process100ofFIG.1and the system200ofFIG.2.

The computer system400stores and executes the gaze tracking application490ofFIG.4B. In addition, the computer system400may be in communication with a software program404. Software program404may include the software code for the gaze tracking application490. Software program404may be loaded on an external medium such as a CD, DVD, or a storage drive, as will be explained further below.

Furthermore, the computer system400may be connected to a network480. The network480can be connected in various different architectures, for example, client-server architecture, a Peer-to-Peer network architecture, or other type of architectures. For example, network480can be in communication with a server485that coordinates engines and data used within the gaze tracking application490. Also, the network can be different types of networks. For example, the network480can be the Internet, a Local Area Network or any variations of Local Area Network, a Wide Area Network, a Metropolitan Area Network, an Intranet or Extranet, or a wireless network.

FIG.4Bis a functional block diagram illustrating the computer system400hosting the gaze tracking application490in accordance with an implementation of the present disclosure. A controller410is a programmable processor and controls the operation of the computer system400and its components. The controller410loads instructions (e.g., in the form of a computer program) from the memory420or an embedded controller memory (not shown) and executes these instructions to control the system, such as to provide the data processing. In its execution, the controller410provides the gaze tracking application490with a software system. Alternatively, this service can be implemented as separate hardware components in the controller410or the computer system400.

Memory420stores data temporarily for use by the other components of the computer system400. In one implementation, memory420is implemented as RAM. In one implementation, memory420also includes long-term or permanent memory, such as flash memory and/or ROM.

Storage430stores data either temporarily or for long periods of time for use by the other components of the computer system400. For example, storage430stores data used by the gaze tracking application490. In one implementation, storage430is a hard disk drive.

The media device440receives removable media and reads and/or writes data to the inserted media. In one implementation, for example, the media device440is an optical disc drive.

The user interface450includes components for accepting user input from the user of the computer system400and presenting information to the user402. In one implementation, the user interface450includes a keyboard, a mouse, audio speakers, and a display. In another implementation, the user interface450also includes a headset worn by the user and used to collect eye movements as user inputs. The controller410uses input from the user402to adjust the operation of the computer system400.

The I/O interface460includes one or more I/O ports to connect to corresponding I/O devices, such as external storage or supplemental devices (e.g., a printer or a PDA). In one implementation, the ports of the I/O interface460include ports such as: USB ports, PCMCIA ports, serial ports, and/or parallel ports. In another implementation, the I/O interface460includes a wireless interface for communication with external devices wirelessly.

The network interface470includes a wired and/or wireless network connection, such as an RJ-45 or “Wi-Fi” interface (including, but not limited to 802.11) supporting an Ethernet connection.

The computer system400includes additional hardware and software typical of computer systems (e.g., power, cooling, operating system), though these components are not specifically shown inFIG.4Bfor simplicity. In other implementations, different configurations of the computer system can be used (e.g., different bus or storage configurations or a multi-processor configuration).

In a particular implementation, a non-transitory computer-readable storage medium storing a computer program to track gaze of a user within a user view screen is disclosed. The computer program includes executable instructions that cause a computer to: combine the gaze tracking with head movement tracking, wherein head movements of the head movement tracking provide rough estimate of a direction of the gaze of the user, while eye movements of the gaze tracking provide fine tuning of the direction of the gaze of the user within the user view screen; divide the user view screen into a plurality of gaze zones when gaze zone estimation is turned on; and combine the gaze tracking with the gaze zone estimation to select a gaze zone from the plurality of gaze zones as the direction of the gaze of the user.

In one implementation, the computer program further includes executable instructions that cause a computer to: display a notification icon on the user view screen when a notification is received; and display a message of the notification on the user view screen when it is determined that the gaze of the user has moved to the notification icon.

The description herein of the disclosed implementations is provided to enable any person skilled in the art to make or use the present disclosure. Numerous modifications to these implementations would be readily apparent to those skilled in the art, and the principles defined herein can be applied to other implementations without departing from the spirit or scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the implementations shown herein but is to be accorded the widest scope consistent with the principal and novel features disclosed herein. Accordingly, additional variations and implementations are also possible.

All features of each of the above-discussed examples are not necessarily required in a particular implementation of the present disclosure. Further, it is to be understood that the description and drawings presented herein are representative of the subject matter which is broadly contemplated by the present disclosure. It is further understood that the scope of the present disclosure fully encompasses other implementations that may become obvious to those skilled in the art and that the scope of the present disclosure is accordingly limited by nothing other than the appended claims.