Patent Description:
Aspects of the present disclosure are related to video compression. In particular the present disclosure relates to differential coding qualities within a video image.

Graphical display devices having a wide field of view (FOV) have been developed. Such devices include head mounted display (HMD) devices. In an HMD device, a small display device is worn on a user's head. The display device has a display optic in front of one eye (monocular HMD) or each eye (binocular HMD). An HMD device typically includes sensors that can sense the orientation of the device and change the scene shown by the display optics as the user's head moves. Conventionally, most video displayed on wide FOV displays are compressed using a single compression algorithm where all parts of the screen have the same resolution.

However, displaying video for virtual reality (VR) programs, which is often performed in conjunction with HMD devices, requires a higher frame rate than conventional flat screen displays to prevent a user from experiencing motion sickness. Displaying video in very high resolution and high frame rate is very costly in terms of bandwidth. HMD for VR has optical systems to show video in wide FOV for immersive experiences. While the screen area around a primary gaze point (sometimes called the foveal region) requires high resolution, the areas outside the primary gaze point are observed only by the peripheral vision and can therefore be displayed at a lower resolution. Such video is sometimes referred to as foveated video.

It is within this context that the present disclosure arises. Previously proposed arrangements are disclosed by <CIT>, <CIT>, <CIT>, <CIT>, <CIT>, <CIT>, <CIT>, and <CIT>.

The present disclosure is defined by the appended claims.

The disadvantages associated with the prior art are overcome by aspects of the present disclosure relating to encoding and transmission of foveated video. description of the specific embodiments.

The following detailed description contains many specific details for the purposes of illustration, Accordingly, the illustrative implementations of the present disclosure described below are set forth without any loss of generality to, and without imposing limitations upon, the claimed invention.

Eye gaze tracking has been used in a wide range of applications, including medical research, automobile technology, computer entertainment and video game programs, control input devices, augmented reality glasses, and more. There are a number of techniques for eye tracking, also known as gaze tracking. Techniques for eye gaze tracking and selective rendering compression are described in co-pending application <CIT>. Some of these techniques determine a user's gaze direction from the orientation of the pupils of the user's eyes. Some known eye gaze tracking techniques involve illuminating the eyes by emitting light from one or more light sources and detecting reflections of the emitted light off of the corneas with a sensor. Typically, this is accomplished using invisible light sources in the infrared range and capturing image data (e.g., images or video) of the illuminated eyes with an infrared sensitive camera. Image processing algorithms are then used to analyze the image data to determine eye gaze direction.

Generally, eye tracking image analysis takes advantage of characteristics distinctive to how light is reflected off of the eyes to determine eye gaze direction from the image. For example, the image may be analyzed to identify eye location based on corneal reflections in the image data, and the image may be further analyzed to determine gaze direction based on a relative location of the pupils in the image.

Two common gaze tracking techniques for determining eye gaze direction based on pupil location are known as Bright Pupil tracking and Dark Pupil tracking. Bright Pupil tracking involves illumination of the eyes with a light source that is substantially in line with the optical axis of the camera, causing the emitted light to be reflected off of the retina and back to the camera through the pupil. The pupil presents in the image as an identifiable bright spot at the location of the pupil, similar to the red eye effect which occurs in images during conventional flash photography. In this method of gaze tracking, the bright reflection from pupil itself helps the system locate the pupil if contrast between pupil and iris is not enough.

Dark Pupil tracking involves illumination with a light source that is substantially off line from the optical axis of the camera, causing light directed through the pupil to be reflected away from the optical axis of the camera, resulting in an identifiable dark spot in the image at the location of the pupil. In alternative Dark Pupil tracking systems, an infrared light source and cameras directed at eyes can look at corneal reflections. Such camera based systems track the location of the pupil and corneal reflections which provides parallax due to different depths of reflections gives additional accuracy.

<FIG> depicts an example of a dark pupil gaze tracking system <NUM> that may be used in the context of the present disclosure. The gaze tracking system tracks the orientation of a user's eye E relative to a display screen <NUM> on which visible images are presented. While a display screen is used in the example system of <FIG>, certain alternative embodiments may utilize an image projection system capable of projecting images directly into the eyes of a user. In these embodiments, the user's eye E would be tracked relative to the images projected into the user's eyes. In the example of <FIG>, the eye E gathers light from the screen <NUM> through a variable iris I and a lens L projects an image on the retina R. The opening in the iris is known as the pupil. Muscles control rotation of the eye E in response to nerve impulses from the brain. Upper and lower eyelid muscles ULM, LLM respectively control upper and lower eyelids UL,LL in response to other nerve impulses.

Light sensitive cells on the retina R generate electrical impulses that are sent to the user's brain (not shown) via the optic nerve ON. The visual cortex of the brain interprets the impulses. Not all portions of the retina R are equally sensitive to light. Specifically, light-sensitive cells are concentrated in an area known as the fovea.

The illustrated image tracking system includes one or more infrared light sources <NUM>, e.g., light emitting diodes (LEDs) that direct non-visible light (e.g., infrared light) toward the eye E. Part of the non-visible light reflects from the cornea C of the eye and part reflects from the iris. The reflected non-visible light is directed toward a suitable sensor <NUM> (e.g., an infrared camera) by a wavelength-selective mirror <NUM>. The mirror transmits visible light from the screen <NUM> but reflects the non-visible light reflected from the eye.

The sensor <NUM> is preferably an image sensor, e.g., a digital camera that can produce an image of the eye E which may be analyzed to determine a gaze direction GD from the relative position of the pupil. This image may be produced with a local processor <NUM> or via the transmission of the obtained gaze tracking data to a remote computing device <NUM>. The local processor <NUM> may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The image tracking data may be transmitted between the sensor <NUM> and the remote computing device <NUM> via a wired connection (not shown), or wirelessly between a wireless transceiver <NUM> included in the eye tracking device <NUM> and a second wireless transceiver <NUM> included in the remote computing device <NUM>. The wireless transceivers may be configured to implement a local area network (LAN) or personal area network (PAN), via a suitable network protocol, e.g., Bluetooth, for a PAN.

The gaze tracking system <NUM> may also include an upper sensor <NUM> and lower sensor <NUM> that are configured to be placed, for example, respectively above and below the eye E. Sensors <NUM> and <NUM> may be independent components, or may alternatively be part of a component <NUM> worn on the user's head that may include, but is not limited to, any combination of the sensor <NUM>, local processor <NUM>, or inertial sensor <NUM> described below. In the example system shown in <FIG>, sensors <NUM> and <NUM> are capable of collecting data regarding the electrical impulses of the nervous system and/or the movement and/or vibration of the muscular system from those areas surrounding the eye E. This data may include for example, electrophysiological and/or vibrational information of the muscles and/or nerves surrounding the eye E as monitored by the upper sensor <NUM> and lower sensor <NUM>. The electrophysiological information collected by sensors <NUM> and <NUM> may include, for example, electroencephalography (EEG), electromyography (EMG), or evoked potential information collected as a result of nerve function in the area(s) surrounding the eye E. Sensors <NUM> and <NUM> may also be capable of collecting, for example, mechanomyogram or surface electromyogram information as a result of detecting the muscular vibrations or twitches of the muscles surrounding the eye E. The sensors <NUM> may also be capable of collecting information related to a motion sickness response, including for example, heart rate data, electrocardiography (ECG) or Galvanic skin response data. The data collected by sensors <NUM> and <NUM> may be delivered with the image tracking data to the local processor <NUM> and/or the remote computing device <NUM> as described above.

The gaze tracking system <NUM> may also be capable of tracking a user's head. Head tracking may be performed by an inertial sensor <NUM> capable producing signals in response to the position, motion, orientation or change in orientation of the user's head. This data may be sent to the local processor <NUM> and/or transmitted to the remote computing device <NUM>. The inertial sensor <NUM> may be an independent component, or may alternatively be part of a component <NUM> worn on the user's head that may include, but is not limited to, any combination of the sensor <NUM>, local processor <NUM>, or sensors <NUM> and <NUM> described above. In alternative embodiments, head tracking may be performed via the tracking of light sources on the component <NUM>. The gaze tracking system <NUM> may also include one or more memory units <NUM> (e.g., random access memory (RAM), dynamic random access memory (DRAM), Read only memory (ROM) and the like.

The local processor <NUM> may be configured to receive encoded data from the network connection <NUM>. The local processor <NUM> may be operatively coupled to the one or more memory units <NUM> and configured to execute one or more programs stored on the memory units <NUM>. The execution of such programs may cause the system to decode video streams from the remote computing device <NUM> and generate foveated video for display on the display <NUM>. By way of example and not by way of limitation the programs may include a Blender / Conversion Space Configuration program <NUM>, a Scaler program <NUM> and a decoder program.

The remote computing device <NUM> may be configured to operate in coordination with the eye tracking device <NUM> and the display screen <NUM>, in order to perform eye gaze tracking and determine lighting conditions in accordance with aspects of the present disclosure. The computing device <NUM> may include one or more processor units <NUM>, which may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The computing device <NUM> may also include one or more memory units <NUM> (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), and the like).

The processor unit <NUM> may execute one or more programs, portions of which may be stored in the memory <NUM>, and the processor <NUM> may be operatively coupled to the memory <NUM>, e.g., by accessing the memory via a data bus <NUM>. The programs may be configured to perform eye gaze tracking and determine lighting conditions for the system <NUM>. By way of example, and not by way of limitation, the programs may include gaze tracking programs <NUM>, the execution of which may cause the system <NUM> to track a user's gaze, e.g., as discussed above, Conversion space configuration programs (CSC) <NUM> that convert the video frame stream to a form that can be presented by a display device, Encoder programs <NUM>, and video stream scaler programs <NUM> the execution of which encodes a stream of low resolution video frames and selected high resolution sections of the video frames to be sent to the display. The encoder programs <NUM> may use a viewer motion sickness response to adjust that size of the region of interest (ROI) to be presented on the display at high resolution to reduce the chance of motion sickness occurring in the viewer, e.g., by reducing the size of the ROI.

By way of example, and not by way of limitation, the gaze tracking programs <NUM> may include processor executable instructions which cause the system <NUM> to determine one or more gaze tracking parameters of the system <NUM> from eye tracking data gathered with the image sensor <NUM> and eye movement data gathered from the upper and lower sensors <NUM> and <NUM>, respectively, while light is emitted from the lighting source <NUM>. The gaze tracking programs <NUM> may also include instructions which analyze images gathered with the image sensor <NUM> in order to detect a presence of a change in lighting conditions.

As seen in <FIG>, the image <NUM> showing a user's head H may be analyzed to determine a gaze direction GD from the relative position of the pupil. For example, image analysis may determine a <NUM>-dimensional offset of the pupil P from a center of the eye E in the image. The location of the pupil relative to the center may be converted to a gaze direction relative to the screen <NUM>, by a straightforward geometric computation of a three-dimensional vector based on the known size and shape of the eyeball. The determined gaze direction GD is capable of showing the rotation and acceleration of the eye E as it moves relative to the screen <NUM>.

As also seen in <FIG>, the image may also include reflections <NUM> and <NUM> of the non-visible light from the cornea C and the lens L, respectively. Since the cornea and lens are at different depths, the parallax and refractive index between the reflections may be used to provide additional accuracy in determining the gaze direction GD. An example of this type of eye tracking system is a dual Purkinje tracker, wherein the corneal reflection is the first Purkinje Image and the lens reflection is the 4th Purkinje Image. There may also be reflections <NUM> from a user's eyeglasses <NUM>, if these are worn a user.

Current HMD panels refresh at a constant rate of <NUM> or <NUM> Hertz (Hz) depending on the manufacturer. The high refresh rate increases power consumption of the panel and bandwidth requirements of the transmission medium to send frame updates.

<FIG> depicts an example system <NUM> to further illustrate various aspects of the present disclosure. The system <NUM> may include a computing device <NUM> which is coupled to an eye tracking display system <NUM>. The eye tracking display device <NUM> comprises, a local processor <NUM>, local memory <NUM>, well known support circuits <NUM>, a network interface <NUM>, an eye tracking device <NUM> and a display device <NUM> in order to perform eye gaze tracking and/or calibration for eye tracking in accordance with aspects of the present disclosure. The display device <NUM> may be in the form of a cathode ray tube (CRT), flat panel screen, touch screen, or other device that displays text, numerals, graphical symbols, or other visual objects. The local processor <NUM> may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The eye tracking display system <NUM> may also include one or more memory units <NUM> (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), and the like).

The local processor unit <NUM> may execute one or more programs, portions of which may be stored in the memory <NUM>, and the processor <NUM> may be operatively coupled to the memory <NUM> e.g., by accessing the memory via a data bus <NUM>. The programs may be configured to create foveated video for the eye tracking display system <NUM>. By way of example, and not by way of limitation, the programs may include a video stream blending program and CSC <NUM>, a video stream scaling program <NUM>, and decoder programs <NUM>. By way of example, and not by way of limitation, blender/CSC <NUM> may include processor executable instructions which cause the system <NUM> combine a video stream received from the Scaler program <NUM> and a subsection stream received from the decoder program <NUM> to create a foveated video for display on the display device <NUM>. The scaler <NUM> may contain instruction that when executed cause the local processor to up-scale a video stream, received from the decoder <NUM>. The decoder program <NUM> may contain instruction when executed by the local processor that causes the system to receive and decode encoded video stream data from the network interface <NUM>. The decoder programs alternately may be implemented as discrete logical units (not shown) communicatively coupled to the local processor by e.g. the main bus <NUM>. According to aspects of the present disclosure, eye tracking display device <NUM> may be an embedded system, mobile phone, personal computer, tablet computer, portable game device, workstation, game console, head mounted display device and the like. Moreover the computing device <NUM> may also be an embedded system, mobile phone, personal computer, tablet computer, portable game device, workstation, game console, and the like.

The eye tracking display device <NUM> may be coupled to the computing device <NUM>, and may include a dynamic lighting source <NUM> similar to light sources <NUM> of <FIG>. By way of example, and not by way of limitation, the lighting source <NUM> may be an invisible lighting source in the form of one or more infrared LEDs, which may be configured to illuminate a user's eyes in order to gather eye tracking data with the sensor <NUM>. The sensor <NUM> of the eye tracking device may be a detector which is sensitive to light emitted from the light source <NUM>. For example, the sensor <NUM> may be a camera sensitive to the light source such as an infrared camera, and the camera <NUM> may be positioned relative to the eye tracking device and the lighting source so that it may capture images of an area illuminated by the lighting source <NUM>.

The computing device <NUM> may be configured to operate in coordination with the eye tracking display system <NUM>, in order to perform eye gaze tracking and determine lighting conditions in accordance with aspects of the present disclosure. The computing device <NUM> may include one or more processor units <NUM>, which may be configured according to well-known architectures, such as, e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, and the like. The computing device <NUM> may also include one or more memory units <NUM> (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), and the like).

The processor unit <NUM> may execute one or more programs, portions of which may be stored in the memory <NUM>, and the processor <NUM> may be operatively coupled to the memory <NUM>, e.g., by accessing the memory via a data bus <NUM>. The programs may be configured to perform eye gaze tracking and determine lighting conditions for the system <NUM>. By way of example, and not by way of limitation, the programs may include gaze tracking programs <NUM>, execution of which may cause the system <NUM> to track a user's gaze, By way of example, and not by way of limitation, the gaze tracking programs <NUM> may include processor executable instructions which cause the system <NUM> to determine one or more gaze tracking parameters of the system <NUM> from eye tracking data gathered with the camera <NUM> while light is emitted from the dynamic lighting source <NUM>. The gaze tracking programs <NUM> may also include instructions which analyze images gathered with the camera <NUM>, e.g., as described above with respect to <FIG>. The encoder programs alternately may be implemented as discrete logical units (not shown) communicatively coupled to the local processor by e.g. the main bus <NUM>.

In some implementations, the gaze tracking program <NUM> may analyze gaze tracking information to predict periods in which the user's visual percept is obscured, e.g., during blinks, or inactive, e.g., during saccades. Predicting the onset of such periods can be used to reduce unnecessary rendering computations, power consumption and network bandwidth usage. Examples of such techniques are described in commonly-assigned <CIT>.

The computing device <NUM> and the eye tracking display device <NUM> may also include well-known support circuits <NUM><NUM>, such as input/output (I/O) circuits <NUM><NUM>, power supplies (P/S) <NUM><NUM>, a clock (CLK) <NUM><NUM>, and cache <NUM><NUM>, which may communicate with other components of the system, e.g., via the bus <NUM><NUM>, respectively. The computing device <NUM> may include a network interface <NUM> to facilitate communication with similarly configured network interface <NUM>on the eye tracking display device <NUM>. The processor units <NUM><NUM> and network interfaces <NUM><NUM> may be configured to implement a local area network (LAN) or personal area network (PAN), via a suitable network protocol, e.g., Bluetooth, for a PAN. The computing device <NUM> may optionally include a mass storage device <NUM> such as a disk drive, CD-ROM drive, tape drive, flash memory, or the like, and the mass storage device <NUM> may store programs and/or data. The computing device <NUM> may also include a user interface <NUM> to facilitate interaction between the system <NUM> and a user. The user interface <NUM> may include a keyboard, mouse, light pen, game control pad, touch interface, or other device.

The system <NUM> may also include a controller (not pictured) which interfaces with the eye tracking display device <NUM> in order to interact with programs executed by the processor unit <NUM>. The system <NUM> may also execute one or more general computer applications (not pictured), such as a video game or video stream, which may incorporate aspects of eye gaze tracking as sensed by the tracking device <NUM> and processed by the tracking programs <NUM>, CSC <NUM>, Scaler <NUM> that convert the video frame data to a form that can be presented by a display device, and video stream encoder <NUM>.

The computing device <NUM> may include a network interface <NUM>, configured to enable the use of Wi-Fi, an Ethernet port, or other communication methods. The network interface <NUM> may incorporate suitable hardware, software, firmware or some combination thereof to facilitate communication via a telecommunications network. The network interface <NUM> may be configured to implement wired or wireless communication over local area networks and wide area networks such as the Internet. The network interface <NUM> may also include the aforementioned wireless transceiver that facilitates wireless communication with the eye tracking device <NUM> and display device <NUM>. The computing device <NUM> may send and receive data and/or requests for files via one or more data packets <NUM> over a network.

As an aspect of the current disclosure foveated video can reduce the transmission bandwidth requirements and computational load of high resolution video processing, while still preserving essential details in regions of interest in the image presented by the display. Foveated video reduces computation by performing low loss compression or no compression on high resolution video frame subsections in the regions of interest (ROI) of the displayed image where the fovea is focused and performing downscaling and high compression on the video frame outside this region. To utilize foveated video, an image display device, such as a head-mounted display (HMD) would use eye gaze tracking technology to determine where the user is focusing on the screen.

The foveated video may be configured to display high resolution images in the ROI of the viewer as determined by the gaze tracking system. <FIG> depicts a foveated video image <NUM> according to an aspect of the current disclosure. In the image <NUM> an ROI <NUM> may be an area determined by eye gaze tracking to be directly in front of the center of gaze <NUM> (sometimes called the fovea) of the user. Within <NUM> degrees of the gaze center the viewer's visual acuity is highest. Thus it is advantageous to display this portion of the image <NUM> at high resolution. As an aspect of the current disclosure the ROI <NUM> may be a region of the screen selected to be within <NUM> degrees from the center a viewer's gaze <NUM>. The ROI may be any shape, including but not limited to two overlapping circular regions or two adjacent squares. Generally the ROI will consist of two regions next to each other but other shapes will work including but without limitation a single rectangular region or a single ovoid region or two non-adjacent circular regions.

According to aspects of the present disclosure video data corresponding to the image <NUM> may be encoded so that sections <NUM> of the image within the ROI <NUM> are displayed at high resolution and sections <NUM> outside of the ROI <NUM> are displayed at a reduced resolution. Using reduced resolution for the sections <NUM> outside the ROI <NUM> of the viewer allows for reduced bandwidth usage during transmission of the video stream and reduced latency while encoding the stream without a loss in perceived video quality by the viewer. In some implementations there may be a transition area at the edge of the ROI <NUM> where the low and high resolution sections are blended to create a gradual transition between high and low resolution.

The ROI <NUM> may change location from frame to frame within the video stream. A viewer's attention to an area within a video image may change as the video progresses. According to an aspect of the current disclosure as seen in <FIG> a gaze tracking system may detect a movement in the primary gaze points <NUM>, <NUM> for a viewer's left and right eyes and in response the system moves the ROI <NUM>, <NUM> to correspond to the new gaze point along the gaze point movement vector. High resolution Video stream sections <NUM> within the new ROI <NUM> and <NUM> may be fetched in response to the change in the location of the primary gaze points. Alternatively the probability of a gaze point and region of interest being located in a certain area of the video stream may be predicted by algorithm as known in the art such as those described in "<NPL>. Based on the probable gaze point information the system may fetch several high resolution areas and send them to the display where the final image displayed is determined by the actual gaze point determined by the gaze tracker. It should be noted that latency between the display of high resolution areas after movement of the ROI is not as important because generally quick eye movement imitates a saccade event and motion blurs image sections between a first ROI and the next.

In an unclaimed aspect of the present disclosure the gaze tracking system may predict the probability of movement of the ROI from one section of the video stream to another section of the video. Such a prediction may be based upon the movement velocity vector of the gaze point of the viewer. The system may also use probable gaze point information as discussed above to further refine prediction of the ROI location.

Certain aspects of the present disclosure related to video encoding may be better appreciated in light of the following discussion. By way of example, and not by way of limitation, as shown in <FIG>, a single digital picture <NUM> (e.g., a digital video frame) may be broken down into one or more sections. As used herein, the term "section" can refer to a group of one or more luma or chroma samples within the picture <NUM>. A section can range from a single luma or chroma sample within the picture, up to the whole picture. Non-limiting examples of sections include slices (e.g., macroblock rows) <NUM>, macroblocks <NUM>, sub-macroblocks <NUM>, blocks <NUM> and individual pixels <NUM>. As illustrated in <FIG>, each slice <NUM> contains one or more rows of macroblocks <NUM> or portions of one or more such rows. The number of macroblocks in a row depends on the size of the macroblocks and the size and resolution of the picture <NUM>. For example, if each macroblock contains sixteen by sixteen chroma or luma samples then the number of macroblocks in each row may be determined by dividing the width of the picture <NUM> (in chroma or luma samples) by sixteen. Each macroblock <NUM> may be broken down into a number of sub-macroblocks <NUM>. Each sub-macroblock <NUM> may be broken down into a number of blocks <NUM> and each block may contain a number of chroma or luma samples <NUM>. By way of example, and without limitation of the invention, in a common video coding scheme, each macroblock <NUM> may be broken down into four sub-macroblocks <NUM>. Each sub-macroblock may be broken down into four blocks <NUM> and each block may contain a four by four arrangement of sixteen chroma or luma samples <NUM>. Some codecs, such as H. <NUM>, allow a given picture to be broken down into two or more sections of different sizes for encoding. In particular, the H. <NUM> standard introduces the "tile" concept of partitioning a picture. Tiles are independently decodable regions of a picture that are encoded with some shared header information. Tiles can additionally be used for the purpose of spatial random access to local regions of video pictures. A typical tile configuration of a picture consists of segmenting the picture into rectangular regions with approximately equal numbers of coding units (CUs) in each tile. A coding unit is analogous to a macroblock (MB) in the H. <NUM> standard. However, the size of the CU can be set by the encoder, and can be larger than a macroblock. The size of the CU can be flexible and adaptive to video content for best partitioning of the picture.

It is noted that each picture may be either a frame or a field. A frame refers to a complete image. A field is a portion of an image used for to facilitate displaying the image on certain types of display devices. Generally, the chroma or luma samples in an image are arranged in rows. To facilitate display an image may sometimes be split by putting alternate rows of pixels into two different fields. The rows of chroma or luma samples in the two fields can then be interlaced to form the complete image. For some display devices, such as cathode ray tube (CRT) displays, the two fields may simply be displayed one after the other in rapid succession. The afterglow of the phosphors or other light emitting elements used to illuminate the pixels in the display combined with the persistence of vision results in the two fields being perceived as a continuous image. For certain display devices, such as liquid crystal displays, it may be necessary to interlace the two fields into a single picture before being displayed. Streaming data representing encoded images typically includes information indicating whether the image is a field or a frame. Such information may be included in a header to the image.

Modern video coder/decoders (codecs), such as MPEG2, MPEG4 and H. <NUM> generally encode video frames as one of three basic types known as Intra-Frames, Predictive Frames and Bipredicitve Frames, which are typically referred to as I-frames, P-frames and B-frames respectively.

An I-frame is a picture coded without reference to any picture except itself. I-frames are used for random access and are used as references for the decoding of other P-frames or B-frames. I-frames may be generated by an encoder to create random access points (to allow a decoder to start decoding properly from scratch at a given picture location). I-frames may be generated when differentiating image details prohibit generation of effective P or B frames. Because an I-frame contains a complete picture, I-frames typically require more bits to encode than P-frames or B-frames. Video frames are often encoded as I-frames when a scene change is detected in the input video.

P-frames require the prior decoding of some other picture(s) in order to be decoded. P-frames typically require fewer bits for encoding than I-frames. A P-frame contains encoded information regarding differences relative to a previous I-frame in decoding order. A P-frame typically references the preceding I-frame in a Group of Pictures (GoP). P-frames may contain both image data and motion vector displacements and combinations of the two. In some standard codecs (such as MPEG-<NUM>), P-frames use only one previously-decoded picture as a reference during decoding, and require that picture to also precede the P-frame in display order. <NUM>, P-frames can use multiple previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction.

B-frames require the prior decoding of either an I-frame or a P-frame in order to be decoded. Like P-frames, B-frames may contain both image data and motion vector displacements and/or combinations of the two. B-frames may include some prediction modes that form a prediction of a motion region (e.g., a segment of a frame such as a macroblock or a smaller area) by averaging the predictions obtained using two different previously-decoded reference regions. In some codecs (such as MPEG-<NUM>), B-frames are never used as references for the prediction of other pictures. As a result, a lower quality encoding (resulting in the use of fewer bits than would otherwise be used) can be used for such B pictures because the loss of detail will not harm the prediction quality for subsequent pictures. In other codecs, such as H. <NUM>, B-frames may or may not be used as references for the decoding of other pictures (at the discretion of the encoder). Some codecs (such as MPEG-<NUM>), use exactly two previously-decoded pictures as references during decoding, and require one of those pictures to precede the B-frame picture in display order and the other one to follow it. In other codecs, such as H. <NUM>, a B-frame can use one, two, or more than two previously-decoded pictures as references during decoding, and can have any arbitrary display-order relationship relative to the picture(s) used for its prediction. B-frames typically require fewer bits for encoding than either I-frames or P-frames.

As used herein, the terms I-frame, B-frame and P-frame may be applied to any streaming data units that have similar properties to I-frames, B-frames and P-frames, e.g., as described above with respect to the context of streaming video.

There are typically two prediction modes for P-frames and B-frames generally referred to as inter prediction and intra prediction. Inter prediction is based on sections of a different frame than the one being encoded. Intra prediction is based on sections within the frame being encoded. Many codecs include a prediction mode decision process that compares the bitrate usage for both modes before making a final determination of which mode to use to encode a particular frame or section within a frame. The coding of I-frames does not require motion search or prediction mode decision.

For encoding digital video pictures, an encoder receives a plurality of digital images and encodes each image. Encoding of the digital picture may proceed on a section-by-section basis. The encoding process for each section may optionally involve padding, image compression and motion compensation. As used herein, image compression refers to the application of data compression to digital images. The objective of the image compression is to reduce redundancy of the image data for a give image in order to be able to store or transmit the data for that image in an efficient form of compressed data.

Entropy encoding is a coding scheme that assigns codes to signals so as to match code lengths with the probabilities of the signals. Typically, entropy encoders are used to compress data by replacing symbols represented by equal-length codes with symbols represented by codes proportional to the negative logarithm of the probability.

CABAC is a form of entropy encoding used in the H. <NUM>/MPEG-<NUM> AVC and High Efficiency Video Coding (HEVC) standards. CABAC is notable for providing much better compression than most other entropy encoding algorithms used in video encoding, and it is one of the key elements that provide the H. <NUM>/AVC encoding scheme with better compression capability than its predecessors. However, it is noted that CABAC uses arithmetic coding which may requires a larger amount of processing to decode.

Context-adaptive variable-length coding (CAVLC) is a form of entropy coding used in H. <NUM>/MPEG-<NUM> AVC video encoding. <NUM>/MPEG-<NUM> AVC, it is used to encode residual, zig-zag order, blocks of transform coefficients. It is an alternative to CABAC. CAVLC uses a table look-up method and thus requires considerably less processing for decoding than CABAC, although it does not compress the data quite as effectively. Since CABAC tends to offer better compression efficiency (about <NUM>% more compression than CAVLC), CABAC is favored by many video encoders in generating encoded bitstreams.

According to aspects of the current disclosure the system may be able to reduce the chance that motion sickness will occur in the viewer through a reduction in the size of the high resolution ROI. The system may detect that a viewer is experience motion related sickness through means known in the art including but not limited to, heart rate, body temperature, galvanic skin response or Electro cardiogram (ECG). More information on detecting motion sickness can be found at "<NPL>In an unclaimed aspect of the current disclosure the system detects a potential for motion sickness to occur through monitoring the data rate of the video stream. The data rate of the video stream is related to the amount of displayed data on the screen therefore a high data rate is indicative of an increased likelihood of motion sickness as there will be a large amount of newly displayed information.

According to additional aspects of the current disclosure the system may also employ viewer feedback to reduce motion sickness. In one unclaimed aspect of the system a tactile response from the viewer is employed to initiate the motion sickness reduction scheme. The system may also collect the response from viewers and use statistical analysis of the responses to generate a model for potential motion sickness within the video stream. In another unclaimed aspect the system may use a user defined motion sickness threshold which when a variable for example; video stream data rate, or motion vectors within the video, exceeds the threshold the motion sickness reduction scheme is initiated.

Upon detection of a motion sickness related response the system may narrow the displayed area of the ROI to reduce the amount of high resolution information displayed to the viewer. The system may blend the a larger amount high resolution section of the displayed area with the low resolution video stream to generate a smaller high resolution section with a large but gradual degradation in quality around the edges of the ROI.

A sharp degradation in image resolution from the ROI to the rest of the image can be jarring to the viewer. As such according to aspects of the current disclosure the edges of the high resolution area may be blended with the low resolution area to create a quality degradation gradient between the areas. Such a degradation gradient may be a linear or non-linear transition from high resolution to low resolution. By way of example, a non-linear transition may be a sigmoidal ("S"-shaped) function transition between high and low resolution with respect to distance from the region of interest. In general, the integral of any smooth, positive, "bump-shaped" function will be sigmoidal. Examples of sigmoid functions include, but are not limited to, the logistic function , the generalized logistic function, sigmoid functions include the ordinary arctangent, the hyperbolic tangent, the Gudermannian function, and the error function <MAT>, the complementary error function (<NUM>-erf(x)), and algebraic functions like <MAT>.

The logistic function has the form <MAT>, where:.

Additionally the system may use video streams of multiple resolutions around the edges of the ROI to generate a smoother degradation in quality between the high resolution area and the rest of the display.

In other aspects of the current disclosure the areas surrounding the ROI may be dimmed to save power and create a theater like effect. The dimming function may be a function selected by the viewer or embedded in the video stream.

<FIG> illustrates a system <NUM> according to aspects of the present disclosure. The illustrated system <NUM> is comprised of two computing groups, an encoding group <NUM> and a decoding group <NUM>. The encoding group <NUM> is coupled to the decoding group <NUM> via a network <NUM> which may, for example, be a wireless network, wired local area network connection (LAN), USB connection, serial connection or other device connection wires known in the art. The encoding group <NUM> comprises; a Color Space Conversion (CSC) <NUM>, a down scaler <NUM>, a Region of Interest (ROI) I-Frame encoder <NUM> and a scaled image AVC/HEC encoder <NUM>. The decoder group comprises an ROI I-Frame decoder <NUM>, a AVC/HEV decoder <NUM>, an up-scaler <NUM>, and a video stream blender/CSC <NUM>. Additionally the decoder group is coupled to the display device. The gaze tracking unit may be coupled to the display device <NUM> and is configured to transmit gaze tracking information <NUM> to the ROI I-frame encoder <NUM> in the encoder group <NUM>.

The Encoding group may receive an unencoded video data stream from a scanout unit <NUM>. The unencoded video data stream may include video data from a video game or an encoded data-stream which must be decoded before being re-encoded for use by the currently disclosed system (not shown). The initial unmodified video stream is then sent to the CSC <NUM> which converts the video data stream to a form that is suitable for encoding. For example the color space for the image may be converted from RGB, which is more typical for displays, to YUV color space, which is more typical for encoders. The CSC <NUM> then passes the initial video stream to the ROI I-Frame Encoder <NUM> and the Downscaler <NUM>.

<FIG> shows the ROI encoder side of the encoder group. The ROI I-frame encoder <NUM> receives gaze tracking information <NUM> received from the gaze tracker. According to an aspect of the current disclosure the ROI I-Frame encoder <NUM> may use the gaze-tracking information <NUM> to determine an ROI within the initial video stream. The ROI I-Frame encoder only encodes the subsections of the initial video stream images which are located in the ROI.

The shape of encoded subsections corresponding to the ROI may be coded as a square alternatively the shape may be a circle, oval, rectangle or other regular or irregular shapes. There may be two ROI sections encoded by the ROI I-Frame encoder corresponding to the two eyes of the viewer. The gaze tracking system may detect the presence or absence of an eye and the ROI I-frame encoder may adjust the number of encoded sections to account for the absence of an eye. Alternatively the ROI I-Frame encoder may have logic configured to display a single wide section in the ROI for both eyes.

According to aspects of the present disclosure the sections of the image that make up the ROI may be larger than the actual ROI of the viewer to account for any minor variations in vision radius and sudden eye movement. The sections of the image that make up the ROI may be of any suitable size, for example <NUM> macro blocks wide by <NUM> macro blocks long. The size of the ROI sections may be reduced during a Zoom-in action or increased in a Zoom out action to reduce motion sickness. As discussed in prior sections the encoded subsections may also be reduced in size to reduce motion sickness during display of the video stream.

The ROI I-frame encoder <NUM> only encodes the subsections of the images in the video stream in the ROI as Intra frames (I-Frames) in order to reduce latency and frame level delay. By encoding the ROI as an I-frame, certain time-consuming operations, e.g., motion search and encoding mode decision are omitted, thereby reducing latency. Because the ROI is relatively small compared to the size of the frame, fewer bits are required for an I-frame encoded ROI than for a normal I-Frame that encompasses an entire image. The ROI I-Frame encoder may use any low loss encoder known in the art. An example of a low loss encoder suitable for this purpose without limitation is the X. <NUM> lossless encoder. Sections of the video image outside the ROI need not be encoded or transmitted by the ROI I-frame encoder. The low loss encoded subsections of the video image hereinafter referred to as low compression sections, may have their locations within the original video frame written to a frame header of the low compression sections so that a decoder can subsequently reconstruct the ROI in the correct location within the frame. Once the low compression subsections have been processed they are sent through the network to the decoder side.

<FIG> depicts the high compression side of the encoder group. The down-scaler <NUM> may receive the initial video frame from the CSC <NUM>. The down-scaler <NUM> may apply a size reduction algorithm to the video frame to reduce the data rate required to transmit the video stream and processing power required for encoding. Any scaling algorithm may be used for this process as would be known to one of ordinary skill in the art. Examples of suitable algorithms without limitation would be, a bilinear interpolation, cubic interpolation or Lanczos resampling.

The AVC/HEV encoder <NUM> may then receive the downscaled video frame from the downscaler <NUM>. The downscaled video stream is then encoded using a lossy encoder to reduce encoding time and bandwidth requirement. Suitable encoders for the present applicant may be, without limitation; H. <NUM>/AVC, or H. <NUM>/HEVC. Once encoded the video frame hereinafter the "high compression video frame" is sent to the decoder side through the network.

<FIG> shows the I-Frame decoder side of the decoder group. The I-frame decoder <NUM> receives the low compression section data from the I-Frame encoder over the network. The low compression section data is then decoded by the I-frame decoder <NUM> and the decoded sections are sent to the Blender/CSC <NUM>. Known low loss decoders would be suitable for the currently disclosed application including but not limited to, FFmpeg.

Frame 6B depicts the high compression side of the decoder group. According to aspects of the present disclosure the AVC/HEVC decoder <NUM> receives the High compression video over the network from the encoder group. The high compression video stream is decoded using a suitable decoder and sent to the up-scaler <NUM>.

The up-scaler <NUM> receives the downscaled video stream from the AVC/HEVC decoder <NUM>. The downscaled video stream is then up-scaled using an appropriate upscaling algorithm to match the algorithm applied in the encoder group. The up-scaled video stream is then sent to blender/CSC <NUM>.

The Blender/CSC <NUM> receives the video stream from the up-scaler <NUM> and the subsection stream from the I-frame decoder <NUM>. According to aspects of the current disclosure the Blender/CSC <NUM> may read the Frame header of the subsection stream to determine the location within the video stream to place the subsections. Alternatively the Blender/CSC may be synchronized with the ROI I-Frame encoder to use the gaze tracking data <NUM> to determine the location of placement for the subsection stream (not shown). According to additional aspects of the current disclosure the Blender/CSC <NUM> may receive the location for the subsection stream over other channels such as a separate wired or wireless transmission (not shown).

The subsection stream may be blended with the video stream at the Blender/CSC <NUM>. Blending may be accomplished by applying a mask to the subsection stream and the video stream. The mask will set weights for pixel values in images of the subsection stream and the video stream. The mask may define a linear resolution degradation gradient between the high resolution subsection stream and the low resolution video stream. According to alternate aspects of the current disclosure the mask may define a non-linear resolution degradation gradient. The Blender/CSC <NUM> will then combine the subsection stream and the video stream by superimposing the subsection stream on top of the video stream at the proper location and applying the set pixel values on the images so that the high resolution image and low resolution images are blended according to pixel value weight.

The mask applied to the image may determine the shape of the displayed ROI subsections. As such the size and shape of the displayed ROI subsections may be varied at the Blender/CSC <NUM>. The size and shape of the ROI may be fixed or varied in logic. The Blender/CSC <NUM> may take an ROI shape from the ROI I-Frame encoder and further refine the edges of the ROI. The mask may also be used to define pixel values for the video stream to implement peripheral dimming.

The Blender/CSC <NUM> may also take a subsection stream with more than one potential ROI subsection and determine the correct ROI of the viewer from gaze tracking data Accordingly the Blender/CSC <NUM> will apply a mask to the potential ROI subsection which hides them from view while not hiding the correct ROI subsection.

In an alternate embodiment a single mask is applied to the ROI subsection stream at Blender/CSC <NUM> and the ROI subsection stream is then superimposed upon the video stream with mixing according to the pixel weights applied to the ROI subsection stream. This mixing scheme may also be applied at the ROI I-Frame encoder. In such an embodiment the mask pixel values are applied at the ROI I-Frame encoder and then at the Blender/CSC <NUM> the ROI subsection stream is blended with the video stream according to those mask pixel values.

According to additional aspects of the current disclosure the Blender/CSC <NUM> may implement the motion sickness reduction scheme as discussed above by varying the mask size or shape based on the motion sickness response. The Blender/CSC <NUM> may also vary the resolution gradient as a part of the motion sickness reduction scheme.

<FIG> depicts a masked foveated video according to aspects of the current disclosure. The foveated display may have one or more high resolution subsections <NUM> around the central gaze point or ROI of the viewer <NUM>. The system may apply a mask around the edges of the high resolution <NUM> to achieve a resolution degradation gradient between the high resolution subsections<NUM> and the low resolution background <NUM>. The system may use gaze tracking data to predict a movement vector <NUM> for the ROI. In other embodiments the area of masked sections <NUM> may be enlarged in response to a motion sickness response. Alternatively the encoded high resolution area <NUM> may be reduced due to a motion sickness response.

<FIG> depicts a block diagram of the motion sickness reduction scheme. The motion reduction scheme <NUM> may run either on the encoder side <NUM> or the decoder side <NUM> of the foveated display system of <FIG>. The viewer of a foveated video may a set threshold level of movement above which the viewer experiences motion sickness <NUM>. Alternatively the threshold level <NUM> may be determined by an average motion sickness response from viewers or from an empirically determined physiologic response due to motion sickness. The system continuously measures variables within the video stream or sensor data to determine whether a response has exceeded the set threshold level <NUM>. Monitoring may either be at the eye tracking display device <NUM> or the computing device <NUM> shown in <FIG>. According to additional aspects of the present disclosure the threshold may be manually triggered by the viewer. If the device determines that a motions sickness threshold has been exceeded the system will notify <NUM> either the ROI I-Frame encoder <NUM> or the Blender/CSC <NUM> to reduce the displayed high resolution area <NUM>.

Although examples are described herein with respect to head mounted display (HMD) applications, aspects of the present disclosure are not limited to such implementations. HMD implementations represent a relatively straightforward implementation because the relative locations of the user's eyes and the display screen remain more or less fixed. In principle, however, the disclosed system and method may be adapted to any image display system that can work with gaze tracking. The gaze tracking system may be modified to track the location and orientation of the user's head and eye(s) relative to the display screen in implementations where these are not fixed.

<FIG> illustrate examples of the use of facial orientation and eye gaze direction in conjunction with aspects of the present disclosure. As seen in <FIG> a face <NUM> of a user may appear in an image <NUM>A obtained with a camera trained on the user. Such cameras are common features of devices such as laptop computers, smart phones, and tablet computers. Image analysis software may identify reference points on the face <NUM>. The software may characterize certain of these reference points, e.g., located at the corners of the mouth <NUM>M, the bridge of the nose <NUM>N, the part in the hair <NUM>H, and at the tops of the eyebrows <NUM>E, as being substantially fixed relative to the face <NUM>. The software may also identify the pupils <NUM> and corners <NUM> of the user's eyes as reference points and determine the location of the pupils relative to the corners of the eyes. In some implementations, the centers of the user's eyes can be estimated from the locations of the pupils <NUM> and corners <NUM> of eyes. Then, the centers of eyes can be estimated and the locations of pupils can be compared with the estimated locations of the centers. In some implementations, face symmetry properties can be used.

The software can determine the user's facial characteristics, e.g., head tilt angle and eye gaze angle from analysis of the relative locations of the reference points and pupils <NUM>. For example, the software may initialize the reference points <NUM>E, <NUM>H, <NUM>M, <NUM>N, <NUM> by having the user look straight at the camera and register the locations of the reference points and pupils <NUM> as initial values. The software can then initialize the head tilt and eye gaze angles to zero for these initial values. Subsequently, whenever the user looks straight ahead at the camera, as in <FIG> and the corresponding top view shown in <FIG>, the reference points <NUM>E, <NUM>H, <NUM>M, <NUM>N, <NUM> and pupils <NUM> should be at or near their initial values.

By way of example and not by way of limitation, the pose of a user's head may be estimated using five reference points, the outside corners <NUM> of each of the eyes, the outside corners <NUM>M of the mouth, and the tip of the nose (not shown). A facial symmetry axis may be found by connecting a line between a midpoint of the eyes (e.g., halfway between the eyes' outside corners <NUM>) and a midpoint of the mouth (e.g., halfway between the mouth's outside corners <NUM>M). A facial direction can be determined under weak-perspective geometry from a 3D angle of the nose. Alternatively, the same five points can be used to determine the head pose from the normal to the plane, which can be found from planar skew-symmetry and a coarse estimate of the nose position. Further details of estimation of head pose can be found, e.g., in "<NPL>. Other examples of head pose estimation that can be used are described in "<NPL>. Additional examples of head pose estimation that can be used are described in "<NPL>. Further examples of head pose estimation are described in "<NPL>.

When the user tilts his head, the relative distances between the reference points in the image may change depending upon the tilt angle. For example, if the user pivots his head to the right or left, about a vertical axis Z the horizontal distance x<NUM> between the corners <NUM> of the eyes may decrease, as shown in the image <NUM>C depicted in <FIG>. Other reference points may also work, or be easier to detect, depending on the particular head pose estimation algorithm being used. The amount change in the distance can be correlated to an angle of pivot θH as shown in the corresponding top view in FIG. It is noted that if the pivot is purely about the Z axis the vertical distance Yi between, say, the reference point at the bridge of the nose <NUM>N and the reference points at the corners of the mouth <NUM>M, would not be expected to change significantly. However, it would be reasonably expected for this distance y<NUM> to change if the user were to tilt his head upwards or downwards. It is further noted that the software may take the head pivot angle θH into account when determining the locations of the pupils <NUM> relative to the corners <NUM> of the eyes for gaze direction estimation. Alternatively the software may take the locations of the pupils <NUM> relative to the corners <NUM> of the eyes into account when determining head pivot angle θH. Such an implementation might be advantageous if gaze prediction is easier, e.g., with an infrared light source on a hand-held device, the pupils could be located relatively easily. In the example, shown in <FIG>, the user's eye gaze angle θE is more or less aligned with the user's head tilt angle. However, because of the pivoting of the user's head and the three-dimensional nature of the shape of the eyeballs, the positions of the pupils <NUM> will appear slightly shifted in the image <NUM>D compared to their positions in the initial image <NUM>A.

In some situations, the user may be facing the camera, but the user's eye gaze is directed elsewhere, e.g., as shown in <FIG> and the corresponding top view in <FIG>. In this example, the user's head is tilt angle θH is zero but the eye gaze angle θE is not. Instead, the user's eyeballs are rotated counterclockwise, as seen in <FIG>. Consequently, the reference points <NUM>E, <NUM>H, <NUM>M, <NUM>N, <NUM> are arranged as in <FIG>, but the pupils <NUM> are shifted to the left in the image <NUM>E.

It is noted that the user's head may pivot in one direction and the user's eyeballs may pivot in another direction. For example, as illustrated in <FIG> and FIG. 9I, the user <NUM> may pivot his head clockwise and rotate his eyeballs counterclockwise. Consequently, the reference points <NUM>E, <NUM>H, <NUM>M, <NUM>N, <NUM> are shifted as in <FIG>, but the pupils <NUM> are shifted to the right in the image <NUM>G shown in <FIG>. The gaze tracking system <NUM>, as described in <FIG>, may take this configuration or any of the configurations described above into account in determining the gaze direction GD of the user's eye E.

As may be seen from the foregoing discussion it is possible to track certain user facial orientation characteristics using just a camera. However, many alternative forms of facial orientation characteristic tracking setups could also be used. <FIG> illustrate examples of five facial orientation characteristic tracking systems that, among other possible systems, can be implemented according to embodiments of the present invention.

In <FIG>, the user <NUM> is facing a camera <NUM> and infrared light sensor <NUM>, which are mounted on top of a visual display <NUM>. To track the user's head tilt angle, the camera <NUM> may be configured to perform object segmentation (i.e., track user's separate body parts) and then estimate the user's head tilt angle from the information obtained. The camera <NUM> and infrared light sensor <NUM> are coupled to a processor <NUM> running software <NUM>, which may be configured as described above. By way of example, and not by way of limitation, object segmentation may be accomplished using a motion model to describe how the image of a target might change in accordance to different possible movements of the object. It is noted that some implementations may use more than one camera, for example, some implementations may use two cameras. One camera can provide a zoomed-out image of the field of view to locate the user, and a second camera can zoom-in and focus on the user's face to provide a close-up image for better head and gaze direction estimation.

A user's eye gaze direction may also be acquired using this setup. By way of example, and not by way of limitation, infrared light may be initially directed towards the user's eyes from the infrared light sensor <NUM> and the reflection captured by the camera <NUM>. The information extracted from the reflected infrared light will allow a processor coupled to the camera <NUM> to determine an amount of eye rotation for the user. Video based eye trackers typically use the corneal reflection and the center of the pupil as features to track over time. Thus, <FIG> illustrates a facial orientation characteristic tracking setup that is configured to track both the user's head tilt angle and eye gaze direction in accordance with an embodiment of the present invention. It is noted that, for the purposes of example, it has been assumed that the user is straight across from the display and camera. However, embodiments of the invention can be implemented even if the user is not straight across from the display <NUM> and/or camera <NUM>. For example, the user <NUM> can be +<NUM>° or -<NUM>° to the right/left of display. As long as the user <NUM> is within field of view of the camera <NUM>, the head angle θH and eye gaze θE can be estimated. Then, a normalized angle can be computed as a function of the location of user <NUM> with respect to the display <NUM> and/or camera <NUM> (e.g. body angle θB as shown in <FIG>), the head angle θH and eye gaze θE. By way of example and not by way of limitation, if the user <NUM>, is located such that the body angle θB is +<NUM>° and if the head is turned at an angle θH of -<NUM>°, the user <NUM> is fixing the deviation of the body from the display <NUM> by turning his head, and this is almost as good as having the person looking straight at the display. Specifically, if, e.g., the user's gaze angle θE is zero (i.e., the user's pupils are centered), the normalized angle (e.g., θB+θH+ θE) is zero.

<FIG> provides another facial orientation characteristic tracking setup. In <FIG>, the user <NUM> is facing a camera <NUM> mounted on top of a visual display <NUM>. The user <NUM> is simultaneously wearing a pair of glasses <NUM> (e.g., a pair of 3D shutter glasses) with a pair of spaced-apart infrared (IR) light sources <NUM> (e.g., one IR LED on each lens of the glasses <NUM>). The camera <NUM> may be configured to capture the infrared light emanating from the light sources <NUM>, and then triangulate user's head tilt angle from the information obtained. Because the position of the light sources <NUM> will not vary significantly with respect to its position on the user's face, this setup will provide a relatively accurate estimation of the user's head tilt angle.

The glasses <NUM> may additionally include a camera <NUM> which can provide images to the processor <NUM> that can be used in conjunction with the software <NUM> to find the location of the visual display <NUM> or to estimate the size of the visual display <NUM>. By way of example, and not by way of limitation, the visual display be of a known type having known vertical and horizontal screen dimensions. A test image of a known size relative to the screen may be displayed. Images of the test image may be obtained by the camera and analyzed to determine the orientation and dimensions of the test image in the images obtained by the camera <NUM>. Gathering this information allows the system to normalize the user's facial orientation characteristic data so that calculation of those characteristics is independent of both the absolute locations of the display <NUM> and the user <NUM>. Moreover, the addition of the camera will allow the system to more accurately estimate visible range. Thus, <FIG> illustrates an alternative setup for determining a user's head tilt angle according to an aspect of the present disclosure. In some unclaimed implementations, separate cameras may be mounted to each lens of the glasses <NUM> facing toward the user's eyes to facilitate gaze tracking by obtaining images of the eyes showing the relative location of the pupil with respect to the centers or corners of the eyes, e.g., as discussed above. The relatively fixed position of the glasses <NUM> relative to the user's eyes facilitates tracking the user's eye gaze angle θE independent of tracking of the user's head orientation θH.

<FIG> provides a third facial orientation characteristic tracking setup. In <FIG>, the user <NUM> is facing a camera <NUM> mounted on top of a visual display <NUM>. The user is also holding a controller <NUM> with one or more cameras <NUM> (e.g., one on each side) configured to facilitate interaction between the user <NUM> and the contents on the visual display <NUM>. Images from the camera <NUM> may be analyzed to determine the location of the visual display <NUM> or to estimate the size of the visual display <NUM>, e.g., using a displayed test image as in the above example. Gathering this information allows the system to normalize the user's facial orientation characteristic data so that calculation of those characteristics is independent of both the absolute locations of the display <NUM> and the user <NUM>. Moreover, the addition of the cameras <NUM> to the controller <NUM> allows the system to more accurately estimate visible range.

It is important to note that the setup in <FIG> may be further combined with the setup in <FIG> (not shown in <FIG>) in order to track the user's eye gaze direction in addition to tracking the user's head tilt angle while making the system independent of display size and location. Because the user's eyes are unobstructed in this setup, his eye gaze direction may be obtained through the infrared light reflection and capturing process discussed above.

<FIG> provides yet another alternative facial orientation characteristic tracking setup. In <FIG>, the user <NUM> is facing a camera <NUM> mounted on top of a visual display <NUM>. The user <NUM> is also wearing a headset <NUM> with infrared light sources <NUM> (e.g., one on each eyepiece) and a microphone <NUM>, the headset <NUM> being configured to facilitate interaction between the user <NUM> and the contents on the visual display <NUM>. Much like the setup in <FIG>, the camera <NUM> may capture images of the infrared light emanating from the light sources <NUM> on the headset <NUM>, and the user's head tilt angle may be triangulated from analysis the images obtained. Because the position of the headset <NUM> tends not to vary significantly with respect to its position on the user's face, this setup can provide a relatively accurate estimation of the user's head tilt angle.

In addition to tracking the user's head tilt angle using the infrared light sensors <NUM>, the position of the user's head with respect to a specified target may also be tracked by a separate microphone array <NUM> that is not part of the headset <NUM>. The microphone array <NUM> may be configured to facilitate determination of a magnitude and orientation of the user's speech, e.g., using suitably configured software <NUM> running on the processor <NUM>. Examples of such methods are described e.g., in commonly assigned <CIT>, commonly assigned <CIT>, and commonly-assigned <CIT>.

A detailed explanation of directional tracking of a user's speech using thermographic information may be found in <CIT> entitled "BLOW TRACKING USER INTERFACE SYSTEM AND METHOD", (Attorney Docket No. SCEA10042US00-I).

By way of example, and not by way of limitation, the orientation of the user's speech can be determined using a thermal imaging camera to detect vibration patterns in the air around the user's mouth that correspond to the sounds of the user's voice during speech. A time evolution of the vibration patterns can be analyzed to determine a vector corresponding to a generalized direction of the user's speech.

Using both the position of the microphone array <NUM> with respect to the camera <NUM> and the direction of the user's speech with respect to the microphone array <NUM>, the position of the user's head with respect to a specified target (e.g., display) may be calculated. To achieve greater accuracy in establishing a user's head tilt angle, the infrared reflection and directional tracking methods for determining head tilt angle may be combined. Alternative embodiments may additionally include an inertial sensor <NUM>, as described with respect to <FIG> above.

The headset <NUM> may additionally include a camera <NUM> configured to obtain images of the visual display <NUM> that may be analyzed to find the location of the display and/or to estimate the size of the visual display <NUM>. Gathering this information allows the system to normalize the user's facial orientation characteristic data so that calculation of those characteristics is independent of both the absolute locations of the display <NUM> and the user <NUM>. Moreover, the addition of the camera will allow the system to more accurately estimate visible range. In some embodiments, one or more cameras <NUM> may be mounted to the headset <NUM> facing toward the user's eyes to facilitate gaze tracking by obtaining images of the eyes showing the relative location of the pupil with respect to the centers or corners of the eyes, e.g., as discussed above. The relatively fixed position of the headset <NUM> (and therefore, the camera(s) <NUM>) relative to the user's eyes facilitates tracking the user's eye gaze angle θE independent of tracking of the user's head orientation θH.

It is important to note that the setup in <FIG> may be combined with the setup in <FIG> (not shown in <FIG>) in order to track the user's eye gaze direction in addition to tracking the user's head tilt angle. Because the user's eyes are unobstructed in this setup, his eye gaze direction may be obtained through infrared light reflection and capturing process discussed above.

Embodiments of the present invention can also be implemented in hand-held devices, such as cell phones, tablet computers, personal digital assistants, portable internet devices, or portable game devices, among other examples. <FIG> illustrates one possible example of determining eye gaze direction in the context of a hand-held device <NUM>. The device <NUM> generally includes a processor <NUM> which can be programmed with suitable software, e.g., as described above. The device <NUM> may include a display screen <NUM> and camera <NUM> coupled to the processor <NUM>. One or more microphones <NUM> and control switches <NUM> may also be optionally coupled the processor <NUM>. The microphone <NUM> may be part of a microphone array. The control switches <NUM> can be of any type normally used with the particular type of hand-held device. For example, if the device <NUM> is a cell phone, the control switches <NUM> may include a numeric keypad or alpha-numeric keypad, touch screen, or touch pad, as commonly used in such devices. Alternatively, if the device <NUM> is a portable game unit, the control switches <NUM> may include digital or analog joysticks, digital control switches, triggers, and the like. In some embodiments, the display screen <NUM> may be a touch screen interface and the functions of the control switches <NUM> may be implemented by the touch screen in conjunction with suitable software, hardware or firmware. The camera <NUM> may be configured to face the user <NUM> when the user looks at the display screen <NUM>. The processor <NUM> may be programmed with software to implement head pose tracking and/or eye-gaze tracking. The processor may be further configured to utilize head pose tracking and/or eye-gaze tracking information, e.g., as discussed above.

It is noted that the display screen <NUM>, microphone(s) <NUM>, camera <NUM>, control switches <NUM> and processor <NUM> may be mounted to a case that can be easily held in a user's hand or hands. In some embodiments, the device <NUM> may operate in conjunction with a pair of specialized glasses, which may have features in common with the glasses <NUM> shown in <FIG> and described hereinabove. Such glasses may communicate with the processor through a wireless or wired connection, e.g., a personal area network connection, such as a Bluetooth network connection. In some embodiments, the device <NUM> may be used in conjunction with a headset, which can have features in common with the headset <NUM> shown in <FIG> and described hereinabove. Such a headset may communicate with the processor through a wireless or wired connection, e.g., a personal area network connection, such as a Bluetooth network connection. The device <NUM> may include suitable antenna and transceiver to facilitate wireless network connection.

It is noted that the examples depicted in <FIG> are only a few examples of many setups that could be used to track a user's facial orientation characteristics in accordance with aspects of the present disclosure. Similarly, various body and other facial orientation characteristics in addition to the head tilt angle and eye gaze direction described above may be tracked to facilitate the adjustment of regions of interest rendered on a display.

Aspects of the present disclosure also include a non-transitory computer-readable medium having computer executable instructions embodied therein that, when executed, implement video processing in accordance with the above-mentioned aspects, e.g., as described above with respect to <FIG>, <FIG>, <FIG>, <FIG>, FIGs. 6A-6C, and <FIG>. Examples of suitable computer-readable media include, but are not limited to RAM, ROM, hard disk, flash drive, SDRAM, CD-ROM, Blu-Ray, magnetic tape, and floppy disk.

Claim 1:
A method comprising:
a) obtaining gaze tracking information (<NUM>) representing one or more regions of interest (<NUM>) of a video frame (<NUM>);
b) compressing one or more sections of the video frame corresponding to the one or more regions of interest with a first encoder (<NUM>) to generate low-compression section data, and encoding the low-compression section data as intra frames;
c) downscaling the video frame with a size reduction algorithm and compressing a resulting downscaled video frame with a second encoder (<NUM>) to generate high-compression frame data, wherein the compressing with the second encoder comprises lossy encoding including deciding on intra or inter prediction modes, wherein the low-compression section data is characterized by a lesser degree of compression than the high-compression frame data;
d) transmitting the high compression frame data and the low compression section data.