Patent Description:
Graphical items may be used on a display to show data and information to a viewer. These graphical items may include text, images, and video. Graphical items in the area of computing are well known and have been in use for many years. Recently, showing three dimensional (3D) graphical items on a display has increased in importance in areas such as gaming, modeling and movies.

When displaying graphical items, a system such as a computer uses a processor in combination with memory to display the items on a screen or other display device. Methods for displaying graphical items vary, but typically they rely on a computer interpreting commands to control a graphics processing device that provides graphical items for display. The graphics processing device typically contains custom hardware for this purpose including a processor and memory. In some computer systems the graphics processing device is fully integrated, and in others it is provided as a separate component known as a graphics card.

Graphics processing devices have limits on their processing power, usually quantified in terms of the amount of graphical items that can be displayed on a screen at any given time. This is typically limited by the capabilities of the hardware embodying the graphics processing device, including processors, memory, and communication channels. Additionally, the amount of graphical items able to be displayed on a screen at a given point can be limited by communication limits between the graphics processing device and computer.

In many scenarios that require graphical items be displayed on a screen, a user only focuses on a portion of the screen, and therefore only a portion of the graphical items, an any given time. Meanwhile, other graphical items continue to be displayed on the remaining portions of the screen, which the user is not focused on. This wastes valuable graphics processing device resources to produce graphical items that cannot be fully appreciated by the user because the visual acuity of a human drops dramatically outside those images immediately focused on. <CIT> describes various techniques for displaying virtual and augmented reality content via a head-mounted display, HMD. The HMD may monitor the user's eye movement to identify a focal point of the user's gaze, and then increase the resolution in an area surrounding the focal point, decrease the resolution elsewhere, or both. <CIT> relates to determining an image resource allocation for displaying content within a display area. User movement, including eye movement, may be monitored and processed to determine and/or adjust the image resource allocation for content displayed within the display area.

The present invention is described in conjunction with the appended figures:.

In the appended figures, similar components and/or features may have the same numerical reference label. Further, various components of the same type may be distinguished by following the reference label by a letter that distinguishes among the similar components and/or features. If only the first numerical reference label is used in the specification, the description is applicable to any one of the similar components and/or features having the same first numerical reference label irrespective of the letter suffix.

The ensuing description provides exemplary embodiments only, and is not intended to limit the scope, applicability or configuration of the disclosure. Rather, the ensuing description of the exemplary embodiments will provide those skilled in the art with an enabling description for implementing one or more exemplary embodiments. It being understood that various changes may be made in the function and arrangement of elements without departing from the scope of the invention as set forth in the appended claims.

Specific details are given in the following description to provide a thorough understanding of the embodiments. However, it will be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. For example, circuits, systems, networks, processes, and other elements in the invention may be shown as components in block diagram form in order not to obscure the embodiments in unnecessary detail.

In some embodiments of the invention, and with reference to <FIG>, a system <NUM> for presenting graphics on a display device <NUM> is provided. System <NUM> may include an eye tracking device <NUM> and a graphics processing device <NUM>. In some embodiments, the system may also include a processor/computer <NUM> which communicates with, and controls, graphics processing device <NUM>. In some embodiments, any function of graphics processing device <NUM> may be performed, in whole or in part, by processor/computer <NUM>. Merely by way of example, eye tracking device <NUM> may be provided integral to, or in addition to, a personal computer <NUM> having graphics processing device <NUM> and a central processing unit (in some configurations, graphics processing device <NUM> and the central processing unit are integrated). In other embodiments, eye tracking device <NUM> may be provided integral to, or in addition to, a gaming console <NUM> or other device having graphics processing device <NUM> and a central processing unit. Examples of gaming consoles include those produced and available from Microsoft™, Nintendo™, or Sony™. In other embodiments, the eye tracking device <NUM> may be provided integral to, or in addition to, a wearable headset such as a Virtual Reality (VR) or Augmented Reality (AR) or the like. Examples of wearable headsets include those produced and available under the names Oculus Rift™, HTC Vive™, Sony PlaystationVR™ and Fove™. Thus, embodiments of the invention may be applied to the presentation of graphics in any number of possible devices and applications, including video display, video games, video production and editing, video communications, computer aided drafting and design, etc..

Eye tracking device <NUM> may be for determining at least one of a gaze point of a user on display device <NUM>, or a change in the gaze point of the user on display device <NUM>. Eye tracking devices and methods, sometimes referred to as gaze detection systems and methods, include, for example, products produced and available from Tobii Technology AB, and which operate by using infrared illumination and an image sensor to detect reflection from the eye of a user. An example of such a gaze detection system is described in <CIT>. Other alternative gaze detection systems may also be employed by the invention, regardless of the technology behind the gaze detection system. Eye tracking device <NUM> may employ its own processor or the processor of another device (i.e., processor/computer <NUM>) to interpret and process data received. When an eye tracking device is referred to herein, both possible methods of processing data are referred to.

Graphics processing device <NUM> employed by various embodiments of the invention may be for causing an image to be displayed on display device <NUM>. Graphics processing device <NUM> may modify what image is displayed on display device <NUM> based at least in part on the gaze point of the user on display device <NUM>, or a change in the gaze point of the user on display device <NUM>, as determined by eye tracking device <NUM>. While in some embodiments a separate non-included or non-integrated display device will be controlled by the graphics processing device <NUM>, other embodiments may include or integrate display device <NUM>.

The way in which the image displayed on display device <NUM> may be modified by graphics processing device <NUM> may vary depending on the embodiment, but regardless, the way in which the image is displayed may be intended to increase the image quality of portions of the image on which a user's gaze, or focused gaze, is directed, relative to those portions of the image to which the user's gaze, or focused gaze, is not directed. In this manner, the use of available resources of graphics processing device <NUM>, and/or other system resources, are maximized to deliver image quality where it matters most on display device <NUM>. To demonstrate, <FIG> illustrates a display device <NUM> showing a user's gaze point <NUM> and an area <NUM> around user's gaze point <NUM> in which embodiments of the invention may increase the quality of the image relative to the remaining area <NUM> of the display device <NUM>. Thus, in various embodiments of the invention, the quality of the image produced across display device <NUM> may be increased in area <NUM> relative to remaining area <NUM>.

When "modification" of an image presented on display device <NUM> is discussed herein, it shall be understood that what is intended is that a subsequent image displayed on display device <NUM>, is different than a prior image displayed on display device <NUM>. Thus, graphics processing device <NUM> and display device <NUM>, or other device(s) discussed herein, "modify" an image by causing a first image to be displayed and then a second image to be displayed which is different than the first image. Any other change of an image discussed herein, for example, increasing or decreasing of image quality, shall also be understood to mean that a subsequent image is different than a prior image. Note that a change or modification of an image may include changing or modifying only a portion of the image. Thus, some portions of a prior image may be the same as a subsequent image, while other portions may be different. In other situations, the entirety of a prior image may be different than a subsequent image. It shall be understood that the modification of an area or an entirety of an image does not necessarily mean every finite portion of the area or entirety are changed (for example, each pixel), but rather that the area or entirety may be changed in some potentially consistent, predefined, or ordered manner (for example, the quality of the image is changed).

Increasing the quality of the image may include increasing the quality of any one or more of the below non-exclusive list of graphical characteristics, and/or modifying the content of the graphics, in addition to other possible characteristics known in the art:.

Other parameters beyond the quality of specific graphic content which could be modified include the content of the graphics presented itself. For instance, if normally a collection of objects would be displayed in the periphery, fewer objects might be displayed in the periphery under foveated rendering conditions discussed herein. In some applications, this may reduce processing requirements if objects and or activity occurring in the periphery would not be sufficiently cognizable to a user under foveated rendering conditions. In these embodiments, content of a certain kind could be removed from peripheral presentation. For example, objects having certain virtual characteristics (i.e., characters in a video game, objects in a video game, an environment in a video game, species of objects in other software applications, etc.), certain geometric characteristics (i.e., shape, size, etc.), certain virtual geographic characteristics (i.e., virtual location), or any other specific characteristics might be displayed or not displayed in peripheral rendering under foveated rendering conditions.

The size and shape of the area of the image which may be modified to appear in greater quality can vary depending on the embodiment. Merely by way of example, the shape of the area may be circular, oval, square, rectangular, or polygonal. In some embodiments, the quality of the image within the area may be uniformly increased. In other embodiments, the increase in quality of the image may be greatest at the center of the area (i.e., proximate to the gaze point), and decrease towards the edges of the area (i.e., distal to the gaze point), perhaps to match the quality of the image surrounding the area. To demonstrate, <FIG> shows how image quality may decrease in a linear or non-liner continuous manner from the center of a gaze area outward, while <FIG> shows how image quality may decrease in a stepped manner from the center of a gaze area outward.

In some embodiments, modifying the image displayed on display device <NUM> may occur in response to the detection of a change in the gaze point. This may occur in a number of fashions, some of which are described below.

In some embodiments, an entirety of the image may be modified during the period of change in the gaze point of the user, and once the change in the gaze point of the user ceases, either the area around end gaze point of the user or a remainder of the image (portions of the image not around the end gaze point) may be modified. Merely by way of example, in one embodiment, the quality of the entire image may be increased during movement of the user's gaze (sometimes referred to as a saccade), but the increase in quality may only be sustained in an area around the user's end gaze point once the saccade is complete (i.e., the quality of the remainder of the image may be decreased upon completion of the saccade). In a different embodiment, the quality of the entire image may be decreased during a saccade, but the decrease in quality may only be sustained areas besides around the user's end gaze point once the saccade is complete (i.e., the quality of the area of the image around the user's end gaze point may be increased upon completion of the saccade).

Additionally, the use of other system resources, including for example processor/computer <NUM> and related resources, may also be modified during a user's saccade. For example, non-graphical operations may be supplemented by the resources of processor/computer <NUM> and graphics processing device <NUM>, during a saccade. More specifically, during a saccade, non-graphical calculations necessary for other system operations may proceed at greater speed or efficiency because additional resources associated with processor/computer <NUM> and graphics processing device <NUM> are made available for such operations.

In some embodiments, modifying the image displayed on display device <NUM> may include modifying a portion of the image in an area around an anticipated gaze point of the user, potentially by increasing the quality thereof. The anticipated gaze point may be determined based on the change in the gaze point of the user. To determine the anticipated gaze point of a user, eye tracking device <NUM> and/or another processor (i.e., the computer or game consoler's processor), may determine a rate of the change in the gaze point of the user on display device <NUM>, and determine the anticipated gaze point based at least in part on this rate of the change.

The rate of change of the gaze point of the user, also referred to as the velocity or speed of a saccade by the user is directly dependent on the total change in the gaze point of the user (often referred to as the amplitude of the saccade). Thus, as the intended amplitude of a user's saccade increases, so does the speed of the saccade. While the saccade of a human user can be as fast as <NUM>°/second in humans, for saccades of less than or about <NUM>°, the velocity of a saccade is generally linearly and directly dependent on the amplitude of the saccade. For example, a <NUM>° amplitude is associated with a velocity of <NUM>°/second and a <NUM>° amplitude is associated with a velocity of <NUM>°/second. For saccades of greater than <NUM>°, the peak velocity starts to plateau toward the maximum velocity attainable by the eye (<NUM>°/second). In response to an unexpected stimulus, a saccade normally takes about <NUM> milliseconds (ms) to be initiated and then lasts from about <NUM> to about <NUM>. Based on these relationships between saccade speed and amplitude, embodiments of the invention may determine anticipated gaze points based on saccade velocity. Other predetermined models of mathematical relationships between saccade speed and amplitude may also be employed by various embodiments of the invention to determine an anticipated gaze point.

In some embodiments, the portion of the image modified around the anticipated gaze point may also include the portion of the image around the original gaze point (i.e., the gaze point from which the user's saccade started). While the shape of the portion of the image modified may be any of those shapes described above, in some embodiments it may be a triangle or a trapezoidal shape having a progressively greater width perpendicular to a direction of the saccade as shown in <FIG>.

In <FIG>, display device <NUM> is shown, and an initial user gaze point <NUM> is shown thereon. Prior to any change in initial gaze point <NUM>, embodiments of the invention may provide increased graphics quality in area <NUM>. When a user saccade, represented by arrow <NUM>, is detected by eye tracking device <NUM>, the size and shape of area <NUM> may change to accommodate both initial gaze point <NUM> and anticipated gaze point <NUM>. The changed area <NUM>, while being triangular and/or trapezoidal in this embodiment, may be shaped and sized differently in other embodiments. Merely by way of example, an entire side of display device <NUM> from the initial gaze point to the edges of the display in the direction of the saccade may also be included in changed area <NUM> to account for more possibilities of where the user's gaze point may end. In other embodiments, a circular, oval, or square changed area <NUM> may be provided. In yet other embodiments, changed area <NUM> may include separate and distinct areas around the initial gaze point <NUM> and anticipated gaze point <NUM>.

In some embodiments, the size or shape of the area around the gaze point for which an image is modified (or which remains unmodified from a heightened quality in various embodiments), is dynamic. This may occur based at least in part on any number of factors, including the current location of the gaze point relative to the image or display device. Merely by way of example, if a user moves their gaze point to a certain portion of the screen, a predefined portion of the screen may be modified via increased quality therein (for example, a corner portion of the display having a map of a virtual area in a video game). In some embodiments, if enough user saccades having one or more predefined characteristics are detected in predefined amount of time, the entirety of the display may be modified to be rendered in greater quality.

The performance of a computing system may be directly influenced by the consumption of resources the system has at its disposal. These resources include, but are not limited to, processing power, memory size, memory access speed, and computer bus speed. The display of information such as images and other graphical items may directly require the use of such resources. The higher the quality of this information, as has been previously described, the greater the amount of resources required, or the greater level of strain on existing resources. The present invention seeks to decrease the consumption of these resources by allocating graphical processing and execution resources first and primarily to areas of display device <NUM> that can be readily perceived in high definition by a user, as the areas in which high definition information is actually displayed. Other areas of the display device, which will not be, or cannot easily be, perceived in high definition by a user may be allocated a lesser or remaining amount of resources. Due to latency between the output of information to display device <NUM> and the speed at which a user can move their eyes and perceive information, it may be desirable to provide a system in which the user does not perceive that there is any change to the quality or definition of information being displayed on the display device <NUM>.

In some embodiments, the gaze point information may be determined by, or based on information from, eye tracking device <NUM>, and may be filtered to minimize the change in areas around the gaze point for which an image may be modified. Referring to <FIG> the embodiment includes display device <NUM> comprising an area <NUM> containing a sub-area <NUM>. An objective of this and other embodiments may be to maintain high definition and/or other improved graphical rendering qualities in sub-area <NUM> and/or area <NUM> around any determined gaze points <NUM>. An image may thus be modified in area <NUM> such that it contains greater quality graphics or other modified parameters as previously described. If it is determined that a gaze point <NUM> remains within sub-area <NUM>, the quality of the graphics or the like in area <NUM> may be modified such that the graphical quality of the images in area <NUM> are displayed at a higher quality or the like than images outside of area <NUM> on display device <NUM>. If it is determined that the gaze point is located outside of sub-area <NUM>, as shown in <FIG>, a new area 800A is defined containing a new sub-area 810A. New area 800A may then be modified to contain higher quality graphics or other parameters. The invention then repeats, in that if it is determined that the gaze point remains within new sub-area 810A, area 800A remains constant, however if the gaze point is detected outside of new sub-area 810A, area 800A is redefined.

In these or other embodiments, filtering of gaze information may performed to ensure that relocation of area <NUM> is necessary. For example, the system may determine a gaze point <NUM> is located outside the sub-area <NUM> however it may perform no action (such as relocating the area <NUM>) until a predetermined number of gaze points <NUM> are located outside the sub-area (for example <NUM>, <NUM>, <NUM>, <NUM>). Alternatively, the system could temporarily enlarge area <NUM> until it is certain the gaze point <NUM> is located within a certain area. Additionally, predefined time periods may be established to determine if gaze points <NUM> have moved outside of sub-area <NUM> for at least those time periods prior to enlarging or changing are <NUM>.

As described herein, when the amount of noise in a gaze point determination is low, and gaze points <NUM> are located close together, then area <NUM> and/or sub-area <NUM> may be smaller, and closely correlated to the actual size of the field of gaze points <NUM>. Conversely, if the amount of noise in a gaze point determination is high, and gaze points <NUM> are dispersed and/or not located close together, then area <NUM> and sub-area <NUM> may be larger. Noise may include, merely by way of example, (<NUM>) errors or differences between (a) the calculated/determined gaze point location and (b) the actual gaze point location, as well as (<NUM>) drift of the user's actual gaze point even when the user's attention is actually directed toward a particular point on the display device <NUM>. Thus, when noise in a set of gaze point determinations is low, there may be high precision in the gaze point determination, and when noise in a set of gaze point determinations is high, there may be low precision in the gaze point determination.

In some embodiments, a number of different factors can affect the amount of noise in the gaze point <NUM> determinations. Consequently, the amount of noise in gaze point <NUM> determinations can affect the size of area <NUM> and/or sub-area <NUM>. Noise in gaze point <NUM> determinations can be software and/or hardware based, resulting from inaccuracies in determining the actual gaze point of the user consistently, and/or can be user-based, resulting from drift of the user's gaze from an actual point of interest that the user is consciously focusing on. In either or both cases, multiple and/or continuous gaze point <NUM> determinations will result in a pattern of gaze points <NUM> as shown in <FIG> and <FIG>.

Rather than attempting to determine which of the determined gaze points <NUM> is the actual gaze point <NUM> of the user, instead, as described above, area <NUM> and/or sub-area <NUM> may be sized to contain all determined gaze points, or a large sub-total thereof (i.e., perhaps excluding certain extreme outliers using statistical and/or error checking routines such as a standard deviation method, Z-score, and/or other methods known in the art). Because certain conditions or "secondary factors" may be known before even a first gaze point <NUM> is determined, or after only a minimal number of gaze points <NUM> have been determined, it may be possible to at least initially, if not continually, size area <NUM> and/or area <NUM> based on a number of secondary factors, thereby allowing for rendering of area <NUM> and sub-area <NUM> immediately around the first (or first few) determined gaze point(s) <NUM>, without the need to obtain additional gaze point <NUM> data.

Some secondary factors or conditions which may inform an initial or continuing determination as to the size of area <NUM> and/or sub-area <NUM> may include (<NUM>) which user is viewing the display, (<NUM>) environmental factors, (<NUM>) content of display device <NUM>, and (<NUM>) where on display device <NUM> the user is gazing.

If the gaze determination system <NUM> is informed of which user is using system <NUM>, then that user may have a previously determined amount of noise in their gaze data. Different users may have inherently different and/or distinct noise levels due to the particular characteristics of how their gaze wanders on display device <NUM>. From prior usage of system <NUM>, it may thus be known what size of area <NUM> and/or area <NUM> will be sufficient for areas of increased graphical quality, without requiring analysis of initial, or additional, gaze point <NUM> determinations.

Characteristics from the user's body, face, and/or eyes, as detected by the eye tracking device imaging sensors or other sensors, may also inform system <NUM> of likely characteristics of the user's gaze, and therefore also be used to anticipate the necessary size of area <NUM> and/or sub-area <NUM>. For example, slumped shoulders and/or squinted eyes may indicate the user is tired (or in some other state), and therefore inform system <NUM> that area <NUM> and sub-area <NUM> should be adjusted accordingly (i.e., made smaller or larger).

Furthermore, the pairing of the user with system <NUM> may also be relevant to the determination. This meaning that the same user operating two different gaze determination systems may have different noise levels in their gaze point <NUM> determinations. Thus, USER-A using SYSTEM-A may be associated with a first amount of gaze point <NUM> noise, while the same USER-A using SYSTEM-B may be associated with a second amount of gaze point <NUM> noise, with each amount of gaze point <NUM> noise corresponding to a differently sized area <NUM> and/or sub-area <NUM>. Additionally, the shape of area <NUM> and/or sub-area <NUM> may also correspond to different users and/or different user/system pairs.

Environmental factors may also play a part in an initial or continuing determination of the necessary size for area <NUM> and/or sub-area <NUM>. Merely by way of example, if system <NUM> determines that the environment around the user is bright, dim, humid, dry, polluted (particulate matter in the air), windy, subject to extreme or unusual temperatures, or oscillating between various states (e.g., there are flashing lights in the area), system <NUM> may adjust the size of area <NUM> and sub-area <NUM> based upon such information. Thus system <NUM> may be able to anticipate a certain amount of noise in gaze point <NUM> determination under such conditions, and anticipatorily adjust the size of area <NUM> and/or sub-area <NUM>. Other environmental characteristics such as time of day may also affect anticipated noise. Perhaps because users are more likely to be tired during morning or evening hours, and have more or less drift associated with their gaze.

The content displayed on display device <NUM> may also may also inform system <NUM> as to the likely noise in gaze point <NUM> determination. Merely by way of example, brighter or darker images displayed on display device <NUM> may be known to cause more or less noise in gaze determination. In other example, text may make more or less noise likely for a given user. Other examples include the nature of the graphical content. Merely by way of example, dynamic fast moving images may produce more or less noise than static or slow moving images. System <NUM> may be able to take this into account to anticipate what size area <NUM> and sub-area <NUM> are necessary.

Finally, where a user is gazing on the display device <NUM> may also affect likely noise levels. Merely by way of example, when gaze point <NUM> of the user is near the boundaries of display device <NUM>, noise may be amplified compared to when the user's gaze point <NUM> is near the center of display device <NUM>. Noise may also vary less or more greatly depending on whether gaze point <NUM> is to the left or right of center versus being above or below center of the display.

Thus, in one embodiment, as shown in <FIG>, a method of one embodiment of the invention may include, at block <NUM>, displaying graphics on a display device. At block <NUM>, and continually thereafter, one or more gaze points of the user may be determined. At block <NUM>, secondary factors as discussed above may be determined. At block <NUM>, the graphics displayed may be modified based at least on the determined gaze point(s) and the determined secondary factor(s).

In some embodiments, another method of determining the size of area <NUM> and/or sub-area-<NUM> may be to employ predefined regions on display device <NUM> which would dictate the size and/or shape of area <NUM> and/or sub-area <NUM>. Thus, display device <NUM> may be divided into a plurality of predefined regions within the display area. For example, as shown in <FIG>, display device <NUM> may be divided into sixteen predefined regions <NUM>. Predefined regions <NUM> may be fixed by hardware and/or software, or may be manually set by the user before the usage. Different software programs may also be able to predefine regions <NUM> based on the particular needs of the software program.

If all of the user's gaze points <NUM> are located within a particular predefined region <NUM>, then the particular predefined region <NUM> is rendered in a higher quality than compared to the other predefined regions <NUM> in which no gaze position is included. In another example, if the gaze positions are located in more than one predefined region <NUM>, then all predefined regions <NUM> in which gaze positions are located are rendered in a higher quality than compared to the other predefined regions <NUM> in which no gaze position is included. In other related embodiments, variations in the above examples may be present. Merely by way of example, the predefined regions <NUM> may be divided from just a portion of display device <NUM> instead of the entire display area of display device <NUM>.

Such methods of using predefined regions <NUM> for foveated rendering may have particular merit in some virtual and/or augmented reality headsets that are not equipped with especially accurate and/or powerful eye tracking systems. Low quality eye tracking in such embodiments may exist in some part due to a low maximum frame rate of the eye tracking camera employed. In such situation, the system may only provide a rough determination of gaze positions, which may result in the low precision eye tracking data (i.e., the determined gaze positions are widely spread). In order to provide foveated rendering for such low quality eye tracking situations, we predefined regions <NUM> may be employed.

In an alternative embodiment shown in <FIG>, the method may determine gaze positions <NUM> first, and then a high quality region <NUM> (size and shape) is determined based on at least a portion (for example, majority) of the gaze positions. A plurality of surrounding regions <NUM>, surrounding initial high quality region <NUM> are then determined based on the determined size and shape of the high quality region <NUM>. The display quality of those surrounding regions are lower than the determined high quality region <NUM>. In a special case, each of the surrounding regions <NUM> may have differing display quality, depending on various rendering conditions (e.g. displayed object characteristics, displayed object positioning, distance to the high quality region <NUM>, etc.).

In another possible embodiment, if a sub-area <NUM> represents a certain zone where the gaze point <NUM> of the user is located, and to which a first degree of increased rendering quality is applied, surrounding zones to may be represented by area <NUM> to which a second degree of increased rendering quality is applied (the second degree being less significant than the first degree).

In some embodiments, the size of area <NUM> may be based on an eccentricity angle of about <NUM>-<NUM> degrees from the currently determined gaze point. In exemplary embodiments, the angle may be about <NUM> degrees. Although the concept of an eccentricity angle would be well understood by a person of skill in the art, for demonstration purposes, its use in the present embodiment will now be described with reference to <FIG>.

The eccentricity angle θ represents a person's fovea vision. In some embodiments, it may be preferable that area <NUM> is larger than the area labeled "object" in <FIG>. This means that there may be a high probability that while a person's gaze point <NUM> remains within the sub-area <NUM>, that person will not be able to perceive information outside the area <NUM> at high quality. The size of the area labeled "object" on display device <NUM> is primarily dependent on the physical size of display device <NUM> and the distance between the eye(s) and display device <NUM>.

The size of area <NUM> may be modified to affect performance, as the greater the size of area <NUM>, the more system resources are required to render graphics at higher quality. By way of example, the area <NUM> may be of such size so as to fit within the display <NUM> to <NUM> times. This size may be optionally linked directly to the size of display device <NUM>, or to the eccentricity angle, such that the size may scale efficiently. In a further embodiment, gaze point <NUM> may be determined after adjusting information obtained by eye tracking device <NUM> to remove noise. Different individuals may have different levels of noise in the gaze information obtained by eye tracking device <NUM> (for example, due to wandering of their eye(s) about the gaze point). If an individual has a low level of noise, area <NUM> may be smaller, and thus performance of the system on which the present embodiment is being practiced may be increased.

In another embodiment of the invention, a non-transitory computer readable medium having instructions thereon for presenting graphics on display device <NUM> is provided. The instructions may be executable by one or more processors to at least display an image on display device <NUM>. The instructions may also be executable to receive information from eye tracking device <NUM> indicative of at least one of a gaze point of a user on display device <NUM>, or a change in the gaze point of the user on display device <NUM>. The instructions may further be executable to cause graphics processing device <NUM> to modify the image displayed on display device <NUM> based at least in part on the gaze point of the user on display device <NUM>, or the change in the gaze point of the user on display device <NUM>. Thus, a non-transitory computer readable medium able to implement any of the features described herein in relation to other embodiments is also provided.

In another embodiment of the invention, a method <NUM> for presenting graphics on display device <NUM> is provided as shown in <FIG>. At step <NUM>, method <NUM> may include displaying an image on display device <NUM>. At step <NUM>, method <NUM> may also include receiving information from eye tracking device <NUM> indicative of at least one of a gaze point of a user on display device <NUM>, or a change in the gaze point of the user on display device <NUM>. At step <NUM>, method <NUM> may further include causing graphics processing device <NUM> to modify the image displayed on display device <NUM> based at least in part on the gaze point of the user on display device <NUM>, or the change in the gaze point of the user on display device <NUM>. Step <NUM> may include, at step <NUM>, increasing the quality of the image in an area around the gaze point of the user, relative to outside the area. Step <NUM> may also include, at step <NUM>, decreasing the quality of the image outside an area around the gaze point of the user, relative to inside the area. Thus, a method to implement any of the features described herein in relation to other embodiments is also provided.

In some embodiments, the systems and methods described herein may be toggled on and off by a user, possibly to account for multiple additional viewers of display device <NUM> being present. In other embodiments, the systems and methods described herein may automatically toggle on when only one user is viewing display device <NUM> (as detected by eye tracking device <NUM>), and off when more than one user is viewing display device <NUM> (as detected by eye tracking device <NUM>). Additionally, in some embodiments, the systems and methods described herein may allow for reduction in rendering quality of an entire display device <NUM> when no viewers are detected, thereby saving system resources and power consumption when display device <NUM> is not the primary focus of any viewer.

In other embodiments, the systems and methods described herein may allow for modifying multiple portions of an image on display device <NUM> to account for multiple viewers as detected by eye tracking device <NUM>. For example, if two different users are focused on different portions of display device <NUM>, the two different areas of the image focused on may be rendered in higher quality to provide enhanced viewing for each viewer.

In yet other embodiments, data associated with an image may inform the systems and methods described herein to allow prediction of which areas of an image may likely be focused on next by the user. This data may supplement data provided by eye tracking device <NUM> to allow for quicker and more fluid adjustment of the quality of the image in areas likely to be focused on by a user. For example, during viewing of a sporting event, a picture-in-picture of an interview with a coach or player may be presented in a corner of the image. Metadata associated with the image feed may inform the systems and methods described herein of the likely importance, and hence viewer interest and likely focus, in the sub-portion of the image.

<FIG> is a block diagram illustrating an exemplary computer system <NUM> in which embodiments of the present invention may be implemented. This example illustrates a computer system <NUM> such as may be used, in whole, in part, or with various modifications, to provide the functions of eye tracking device <NUM>, graphics processing device <NUM>, the game console, the processor/computer <NUM>, and/or other components of the invention such as those discussed above. For example, various functions of eye tracking device <NUM> and associated processors may be controlled by the computer system <NUM>, including, merely by way of example, tracking a user's gaze point, determining an anticipated gaze point, controlling graphics processing device <NUM>, etc..

The computer system <NUM> is shown comprising hardware elements that may be electrically coupled via a bus <NUM>. The hardware elements may include one or more central processing units <NUM>, one or more input devices <NUM> (e.g., a mouse, a keyboard, etc.), and one or more output devices <NUM> (e.g., a display device, a printer, etc.). The computer system <NUM> may also include one or more storage device <NUM>. By way of example, storage device(s) <NUM> may be disk drives, optical storage devices, solid-state storage device such as a random access memory ("RAM") and/or a read-only memory ("ROM"), which can be programmable, flash-updateable and/or the like.

The computer system <NUM> may additionally include a computer-readable storage media reader <NUM>, a communications system <NUM> (e.g., a modem, a network card (wireless or wired), an infra-red communication device, Bluetooth™ device, cellular communication device, etc.), and working memory <NUM>, which may include RAM and ROM devices as described above. In some embodiments, the computer system <NUM> may also include a processing acceleration unit <NUM>, which can include a digital signal processor, a special-purpose processor and/or the like.

The computer-readable storage media reader <NUM> can further be connected to a computer-readable storage medium, together (and, optionally, in combination with storage device(s) <NUM>) comprehensively representing remote, local, fixed, and/or removable storage devices plus storage media for temporarily and/or more permanently containing computer-readable information. The communications system <NUM> may permit data to be exchanged with a network, system, computer and/or other component described above.

The computer system <NUM> may also comprise software elements, shown as being currently located within a working memory <NUM>, including an operating system <NUM> and/or other code <NUM>. It should be appreciated that alternate embodiments of a computer system <NUM> may have numerous variations from that described above. For example, customized hardware might also be used and/or particular elements might be implemented in hardware, software (including portable software, such as applets), or both. Furthermore, connection to other computing devices such as network input/output and data acquisition devices may also occur.

Software of computer system <NUM> may include code <NUM> for implementing any or all of the function of the various elements of the architecture as described herein. For example, software, stored on and/or executed by a computer system such as system <NUM>, can provide the functions of eye tracking device <NUM>, graphics processing device <NUM>, the game console, the processor/computer, and/or other components of the invention such as those discussed above. Methods implementable by software on some of these components have been discussed above in more detail.

Claim 1:
A system (<NUM>) comprising:
an eye tracking device (<NUM>) for determining an initial gaze point of a user on a display device (<NUM>); and
one or more processors (<NUM>) for:
causing a first image to be displayed on a display device;
determining a velocity and an amplitude of a saccade of the user based on a change in the initial gaze point;
determining an anticipated gaze point based on a relationship between the saccade velocity and amplitude;
modifying the first image to generate a second image based at least in part on the anticipated gaze point,
wherein a part of the first image is modified by increasing an image quality of an area of the first image comprising the anticipated gaze point and the initial gaze point.