Foveated domain storage and processing

An electronic device, method, and computer readable medium for foveated storage and processing are provided. The electronic device includes a memory, and a processor coupled to the memory. The processor performs head tracking and eye tracking; generates a foveated image from an original image based on the head tracking and the eye tracking; and stores the foveated image using one of: a tile-based method or a frame-based method.

TECHNICAL FIELD

This disclosure relates generally to systems for image processing. More specifically, this disclosure relates to systems and methods for foveated domain storage and processing.

BACKGROUND

Rendering in augmented reality (AR) or virtual reality (VR) application present challenging requirements for the need to provide a good user interface (UX) experience. A typical field of view is 60 pixels horizontally (H) by 60 pixels vertically (V). The required resolution is 3600×3600 (60 pixels/degree assuming 20/20 vision accuracy is rendering) per eye. At a 60 Hz rate, the pixel rate requirement for a left eye and a right eye in RGB888 images are around 3600×3600×2×60×3=4,666 Mpixels/sec. In AR/VR applications, both pixel computation and DRAM bandwidth are proportional to these pixel rates.

SUMMARY

In one embodiment, an electronic device provides for foveated storage and processing. The electronic device includes a memory, and a processor coupled to the memory. The processor perform head tracking and eye tracking; generates a foveated image from an original image based on the head tracking and the eye tracking; and stores the foveated image using one of: a tile-based method or a frame-based method.

In a second embodiment, a method provides for foveated storage and processing. The method includes performing head tracking and eye tracking; generating a foveated image from an original image based on the head tracking and the eye tracking; and storing the foveated image using one of: a tile-based method or a frame-based method.

In a third embodiment, a non-transitory medium embodying a computer program provides for foveated storage and processing. The program code, when executed by at least one processor, causes a processor to perform head tracking and eye tracking; generate a foveated image from an original image based on the head tracking and the eye tracking; and store the foveated image using one of: a tile-based method or a frame-based method.

DETAILED DESCRIPTION

FIGS. 1 through 13, discussed below, and the various embodiments of the present disclosure are described with reference to the accompanying drawings. However, it should be appreciated that the present disclosure is not limited to the embodiments and all changes and/or equivalents or replacements thereto also belong to the scope of the present disclosure. The same or similar reference denotations may be used to refer to the same or similar elements throughout the specification and the drawings.

As used herein, the terms “have,” “may have,” “include,” or “may include” a feature (e.g., a number, function, operation, or a component such as a part) indicate the existence of the feature and do not exclude the existence of other features.

As used herein, the terms “A or B,” “at least one of A and/or B,” or “one or more of A and/or B” may include all possible combinations of A and B. For example, “A or B,” “at least one of A and B,” “at least one of A or B” may indicate all of (1) including at least one A, (2) including at least one B, or (3) including at least one A and at least one B.

As used herein, the terms “first” and “second” may modify various components regardless of importance and do not limit the components. These terms are only used to distinguish one component from another. For example, a first user device and a second user device may indicate different user devices from each other regardless of the order or importance of the devices. For example, a first component may be denoted a second component, and vice versa without departing from the scope of the present disclosure.

As used herein, the terms “configured (or set) to” may be interchangeably used with the terms “suitable for,” “having the capacity to,” “designed to,” “adapted to,” “made to,” or “capable of” depending on circumstances. The term “configured (or set) to” does not essentially mean “specifically designed in hardware to.” Rather, the term “configured to” may mean that a device can perform an operation together with another device or parts.

For example, the term “processor configured (or set) to perform A, B, and C” may mean a generic-purpose processor (e.g., a CPU or application processor) that may perform the operations by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) for performing the operations.

The terms as used herein are provided merely to describe some embodiments thereof, but not to limit the scope of other embodiments of the present disclosure. It is to be understood that the singular forms “a,” “'an,” and “the” include plural references unless the context clearly dictates otherwise. All terms including technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of the present disclosure belong. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein. In some cases, the terms defined herein may be interpreted to exclude embodiments of the present disclosure.

For example, examples of the electronic device according to embodiments of the present disclosure may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop computer, a netbook computer, a workstation, a PDA (personal digital assistant), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device (e.g., smart glasses, a head-mounted device (HMD), electronic clothes, an electronic bracelet, an electronic necklace, an electronic appcessory, an electronic tattoo, a smart mirror, or a smart watch).

According to embodiments of the present disclosure, the electronic device may be a smart home appliance. Examples of the smart home appliance may include at least one of a television, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washer, a drier, an air cleaner, a set-top box, a home automation control panel, a security control panel, a TV box (e.g., Samsung HomeSync™, APPLE TV™, or GOOGLE TV™), a gaming console (XBOX™, PLAYSTATION™), an electronic dictionary, an electronic key, a camcorder, or an electronic picture frame.

According to an embodiment of the present disclosure, examples of the electronic device may include at least one of various medical devices (e.g., diverse portable medical measuring devices (a blood sugar measuring device, a heartbeat measuring device, or a body temperature measuring device), a magnetic resource angiography (MRA) device, a magnetic resource imaging (MRI) device, a computed tomography (CT) device, an imaging device, or an ultrasonic device), a navigation device, a global positioning system (GPS) receiver, an event data recorder (EDR), a flight data recorder (FDR), an automotive infotainment device, an sailing electronic device (e.g., a sailing navigation device or a gyro compass), avionics, security devices, vehicular head units, industrial or home robots, automatic teller's machines (ATMs), point of sales (POS) devices, or Internet of Things devices (e.g., a bulb, various sensors, an electric or gas meter, a sprinkler, a fire alarm, a thermostat, a street light, a toaster, fitness equipment, a hot water tank, a heater, or a boiler).

According to certain embodiments of the disclosure, the electronic device can be at least one of a part of a piece of furniture or building/structure, an electronic board, an electronic signature receiving device, a projector, or various measurement devices (e.g., devices for measuring water, electricity, gas, or electromagnetic waves).

According to embodiments of the present disclosure, the electronic device is one or a combination of the above-listed devices. According to embodiments of the present disclosure, the electronic device is a flexible electronic device. The electronic device disclosed herein is not limited to the above-listed devices, and can include new electronic devices depending on the development of technology.

Hereinafter, electronic devices are described with reference to the accompanying drawings, according to various embodiments of the present disclosure. As used herein, the term “user” may denote a human or another device (e.g., an artificial intelligent electronic device) using the electronic device.

Instead of implementing image processing algorithms in full resolution, implement the algorithms in a foveated domain that requires less computation resources and DRAM bandwidth, improving in both processing throughput and power consumption. Using a foveated domain requires DRAM storage formats that are implementation friendly and can allow native image processing. Several use cases for foveated domain are disclosed based on eye tracking and head tracking inputs from sensor processing algorithms.

This application presents new foveated storage formats to reduce memory bandwidth and storage by using a separate format and tile format. A method is presented to compute algorithms like image warp in a foveated storage format. A method is presented to compensate for gaze direction change in a foveated storage format. A method is presented to compensate for head position change in a foveated storage format. A method is presented to compensate for both gaze direction change and head location change in a foveated storage format. A method is presented to compensate for head position change when an output is in full resolution and an input is in foveated format for streaming to a display. A method is presented to compensate for head position change when an input is in full resolution and an output is in foveated format. A method is presented to compensate for color aberration and/or lens distortion when an output is in full resolution and an input is in foveated format for streaming to a display. A method is presented to compensate for head position and color aberration and/or lens distortion when both an input and an output are in full resolution and intermediate image is in foveated format.

FIG. 1illustrates an example network configuration100according to various embodiments of the present disclosure. The embodiment of the network configuration100shown inFIG. 1is for illustration only. Other embodiments of the network configuration100could be used without departing from the scope of this disclosure.

According to an embodiment of the present disclosure, an electronic device101is included in a network environment100. The electronic device101may include at least one of a bus110, a processor120, a memory130, an input/output (IO) interface150, a display160, a communication interface170, or sensors180. In some embodiments, the electronic device101may exclude at least one of the components or may add another component.

The bus110includes a circuit for connecting the components120to170with one another and transferring communications (e.g., control messages and/or data) between the components.

The processor120includes one or more of a central processing unit (CPU), an application processor (AP), or a communication processor (CP). The processor120is able to perform control on at least one of the other components of the electronic device101, and/or perform an operation or data processing relating to communication.

For example, the processor120can receive a plurality of frames captured by the camera during a capture event. The processor120can identify a salient region in each of the plurality of frames. The processor120can determine a reference frame from the plurality of frames based on the identified salient regions. The processor120can fuse non-reference frames with the determined reference frame into a completed frame. The processor120can operate the display to display the completed frame.

The memory130can include a volatile and/or non-volatile memory. For example, the memory130can store commands or data related to at least one other component of the electronic device101. According to an embodiment of the present disclosure, the memory130can store software and/or a program140. The program140includes, e.g., a kernel141, middleware143, an application programming interface (API)145, and/or an application program (or “application”)147. At least a portion of the kernel141, middleware143, or API145may be denoted an operating system (OS).

For example, the kernel141can control or manage system resources (e.g., the bus110, processor120, or a memory130) used to perform operations or functions implemented in other programs (e.g., the middleware143, API145, or application program147). The kernel141provides an interface that allows the middleware143, the API145, or the application147to access the individual components of the electronic device101to control or manage the system resources.

The middleware143can function as a relay to allow the API145or the application147to communicate data with the kernel141, for example. A plurality of applications147can be provided. The middleware143is able to control work requests received from the applications147, e.g., by allocating the priority of using the system resources of the electronic device101(e.g., the bus110, the processor120, or the memory130) to at least one of the plurality of applications134.

The API145is an interface allowing the application147to control functions provided from the kernel141or the middleware143. For example, the API145includes at least one interface or function (e.g., a command) for filing control, window control, image processing or text control.

The IO interface150serve as an interface that can, e.g., transfer commands or data input from a user or other external devices to other component(s) of the electronic device101. Further, the IO interface150can output commands or data received from other component(s) of the electronic device101to the user or the other external device.

The display160includes, e.g., a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or a microelectromechanical systems (MEMS) display, or an electronic paper display. The display160is able to display, e.g., various contents (e.g., text, images, videos, icons, or symbols) to the user. The display160can include a touchscreen and may receive, e.g., a touch, gesture, proximity or hovering input using an electronic pen or a body portion of the user.

For example, the communication interface170is able to set up communication between the electronic device101and an external electronic device (e.g., a first electronic device102, a second electronic device104, or a server106). For example, the communication interface170can be connected with the network162or164through wireless or wired communication to communicate with the external electronic device. The communication interface170can be a wired or wireless transceiver or any other component for transmitting and receiving signals, such as video feeds or video streams.

Electronic device101further includes one or more sensors180that can meter a physical quantity or detect an activation state of the electronic device101and convert metered or detected information into an electrical signal. For example, sensor180may include one or more buttons for touch input, a camera, a gesture sensor, a gyroscope or gyro sensor, an air pressure sensor, a magnetic sensor or magnetometer, an acceleration sensor or accelerometer, a grip sensor, a proximity sensor, a color sensor (e.g., a red green blue (RGB) sensor), a bio-physical sensor, a temperature sensor, a humidity sensor, an illumination sensor, an ultraviolet (UV) sensor, an electromyography (EMG) sensor, an electroencephalogram (EEG) sensor, an electrocardiogram (ECG) sensor, an IR sensor, an ultrasound sensor, an iris sensor, a fingerprint sensor, etc. The sensor(s)180can further include a control circuit for controlling at least one of the sensors included therein. Any of these sensor(s)180may be located within the electronic device101. A camera sensor180can capture a plurality of frames for a single image to be combined by the processor120.

The first external electronic device102or the second external electronic device104may be a wearable device or an electronic device101-mountable wearable device (e.g., a head mounted display (HMD)). When the electronic device101is mounted in a HMD (e.g., the electronic device102), the electronic device101is able to detect the mounting in the HMD and operate in a virtual reality mode. When the electronic device101is mounted in the electronic device102(e.g., the HMD), the electronic device101can communicate with the electronic device102through the communication interface170. The electronic device101can be directly connected with the electronic device102to communicate with the electronic device102without involving with a separate network.

The wireless communication is able to use at least one of, e.g., long term evolution (LTE), long term evolution-advanced (LTE-A), 5th generation wireless system (5G), mm-wave or 60 GHz wireless communication, Wireless USB, code division multiple access (CDMA), wideband code division multiple access (WCDMA), universal mobile telecommunication system (UMTS), wireless broadband (WiBro), or global system for mobile communication (GSM), as a cellular communication protocol. The wired connection can include at least one of universal serial bus (USB), high definition multimedia interface (HDMI), recommended standard 232 (RS-232), or plain old telephone service (POTS).

The network162includes at least one of communication networks, e.g., a computer network (e.g., local area network (LAN) or wide area network (WAN)), Internet, or a telephone network.

The first and second external electronic devices102and104and server106each can be a device of the same or a different type from the electronic device101. According to certain embodiments of the present disclosure, the server106includes a group of one or more servers. According to certain embodiments of the present disclosure, all or some of operations executed on the electronic device101can be executed on another or multiple other electronic devices (e.g., the electronic devices102and104or server106). According to certain embodiments of the present disclosure, when the electronic device101should perform some function or service automatically or at a request, the electronic device101, instead of executing the function or service on its own or additionally, can request another device (e.g., electronic devices102and104or server106) to perform at least some functions associated therewith. The other electronic device (e.g., electronic devices102and104or server106) is able to execute the requested functions or additional functions and transfer a result of the execution to the electronic device101. The electronic device101can provide a requested function or service by processing the received result as it is or additionally. To that end, a cloud computing, distributed computing, or client-server computing technique may be used, for example.

AlthoughFIG. 1shows that the electronic device101includes the communication interface170to communicate with the external electronic device104or server106via the network162, the electronic device101may be independently operated without a separate communication function, according to an embodiment of the present disclosure.

The server106can support to drive the electronic device101by performing at least one of operations (or functions) implemented on the electronic device101. For example, the server106can include a processing module or processor that may support the processor120implemented in the electronic device101.

FIG. 2illustrates an example of foveated rendering200according to embodiments of the present disclosure. The embodiment of the foveated rendering200shown inFIG. 2is for illustration only. Other embodiments of the foveated rendering200could be used without departing from the scope of this disclosure.

Foveated rendering200can reduce the amount of GPU workload for rendering AR or VR images. The minimum angle of resolution (MAR)205is the angular distance210from the central viewing or gaze direction215at the viewpoint220. The MAR is determined by the following equation:

where slope is equal to 0.22-0.34 degrees and pixels per degree is PPD=2/MAR. The MAR represents the outer perimeter of the capability of the eye to discern full resolution details. As the angle increases outside the MAR, the amount of resolution that an eye can perceive is reduced. More details regarding MAR are found in Guenter et. al., “Foveated 3D Graphics”, ACM SIGGRAPH ASIA 2012, which is hereby incorporated by reference.

FIG. 3illustrates an exemplary graph300that presents no foveation305, quantized foveation310, and ideal foveation315according to embodiments of the present disclosure. The embodiment of the exemplary graph300shown inFIG. 3is for illustration only. Other embodiments of the exemplary graph300could be used without departing from the scope of this disclosure.

The exemplary graph300illustrates the pixel resolution (ppd) versus angular distance from the center viewing or gaze direction. With no foveation305, the pixel resolution is 60 ppd across the viewing spectrum, which is useful when there is movement of the gaze. This pixel resolution requires significant processing power, especially considering the wasted resolution outside the MAR area. The ideal foveation315provides the optimal in processing ability, but may experience difficultly with movements. Quantized foveation310provides a mixed benefit of the processing less as the angle increase, but has steps that require less pixels to be processed with movements of the gaze.

FIG. 4illustrates an AR/VR system400according to embodiments of the present disclosure. The embodiment of the AR/VR system400shown inFIG. 4is for illustration only. Other embodiments of the AR/VR system400could be used without departing from the scope of this disclosure.

The AR/VR system400manages the AR/VR experience. The AR/VR system400includes head tracking sensors405, eye tracking sensors410, rendering GPU415, a memory420, and a display processing unit (DPU)430. The AR/VR system not only requires a high bandwidth, but also suffers from higher latency from head tracking to the DPU430.

The head tracking sensors405can include outward facing optic sensors, gyrometer, etc. The head tracking sensor405monitors the vertical motion, horizontal motion, or rotation of the head. The head tracking sensor405outputs the head position at each time interval.

The eye tracking sensor410can include inward facing optic sensor focus on the eyes or any other sensor to monitor the movement of the eye. The eye tracking sensor410outputs the eye position at a time interval. The time interval for the eye tracking sensor410can be the same or different as the time interval for the head tracking sensor405.

The rending GPU415handles the rendering workload and incorporate updates to the eye position and the head position. The rendering GPU415receives the head position from the head tracking sensors405and the eye position from the eye tracking sensor410. The rendering GPU415can accommodate any difference in time intervals between the head tracking and the eye tracking.

The memory420stores the output of the rendering GPU415. The DPU430reads the image or video files in the memory420and outputs the resulting video or image to a display.

FIG. 5illustrates an AR/VR system500with reduced latency via time-warp according to embodiments of the present disclosure. The embodiment of the AR/VR system500shown inFIG. 5is for illustration only. Other embodiments of the AR/VR system500could be used without departing from the scope of this disclosure.

The AR/VR system500provides for a reduced latency via time-warping of an AR or VR experience. The AR/VR system500includes head tracking sensors505, eye tracking sensors510, rendering GPU515, memory520, post-rending correction GPU525, and DPU530.

The head tracking sensors505, eye tracking sensors510, GPU515, memory520and DPU530, perform similarly to the respective components inFIG. 4. The head tracking sensors505and the eye tracking sensors510also output a head position and eye position to the post-rendering correction GPU525at one increment past the time that the head position is output to the GPU. The post-rendering correction GPU525performs post rendering correction that incorporates the latest head and eye position to optimize latency from the head position and eye position to the display. The post rendering correction requires high bandwidth and computation resources due to a high pixel rate.

FIG. 6illustrates a foveated system600with reduced DRAM BW/Power according to embodiments of the present disclosure. The embodiment of the foveated system600shown inFIG. 6is for illustration only. Other embodiments of the foveated system600could be used without departing from the scope of this disclosure.

The AR/VR system600provides for a reduced DRAM bandwidth and power of an AR or VR experience. The AR/VR system600includes head tracking sensors605, eye tracking sensors610, rendering GPU615, foveated image buffer520, post-rending correction GPU525, and DPU530.

The head tracking sensors605, eye tracking sensors610, GPU615, post-rendering correction GPU625and DPU630, perform similarly to the respective components inFIG. 5. The eye tracking sensors510also output a head position and eye position to the DPU530at one increment past the time that the head position is output to the GPU. Foveated rendering can be used for the rending GPU615, post-rendering correction GPU625and the DPU630to reduce the memory bandwidth and pixel computation requirements.

Foveated rendering can include two formats of storage of foveated images in the foveated image buffer can be used including a tile-based method and a frame-based method. The tile-based method is where each tile is stored as a scaled tile. The tile-based method retains a random access and same memory layout at a frame level. A two-dimensional (2D) tile format where a resolution for a given tile is scaled as tiles are traversed from a fovea center outward. A tile of size (W, H) is addressed by its location (x,y). Now a scale factor (Sx,Sy) are defined in the following equations:
Sx=f(∥x−cx,y−cy∥)
Sy=g(∥x−cx,y−cy∥)

where ∥x−cx, y−cy∥ is a 2D distance (could be L1 or L2 norm) from an eye center (cx, cy). In this format, the location of a 2D tile does not change, but a number of pixels stored inside are scaled by (Sx,Sy) saving bandwidth while allowing random access of the pixels.

The frame-based method is where different resolutions within an image are specified by line and partition boundaries. The frame-based method optimizes memory storage and is suitable for faster scan access. The frame-based method can include a 2D separable format where Sx and Sy scale factors are defined by the following equations:
Sx=f(|x−cx|)

where |x−cx| is a horizontal distance from the eye center (cx,cy).
Sy=g(|y−cy|)

where |y−cy| is a vertical distance from the eye center (cx,cy).

Foveated rendering can include conversion between a foveated image format and a full resolution format. Foveated rendering can also include direct processing of algorithms in a foveated domain avoiding the conversion. Foveated rendering can also include a post-processing block for any blocking artifacts due to foveation, e.g. multi-resolution boundaries.

FIG. 7illustrates conversions700of pixel addresses both to and from a foveated format according to embodiments of the present disclosure. The embodiment of the conversion of conversions700shown inFIG. 7is for illustration only. Other embodiments of the conversion of conversions700could be used without departing from the scope of this disclosure.

Recomputing foveated pixel address710in the foveated format from or to the pixel address715in an original non-compressed format allows use of the pixels in either format for reading and writing. The conversion700of the pixel address715to a foveated pixel address710uses a foveation function720. Conversion700from the pixel address710in full resolution to the foveated pixel address710in the foveated domain is defined by the following equations:
(xf,yf)=fov(x,y,cx,cy)
(cxf,cyf)=fov(cx,cy,cx,cy)

where (cxf,cyf) is the new fovea center. The fov( ) function provides conversion from full resolution to fovea resolution. (x′,y′)=fov(x,y,cx,cy) defines maps full resolution (x,y) pixel to foveated (x′,y′) pixels. This is used to access (x,y) pixel location in foveated image. This function is defined in this psuedo-code (fov_scl is fovea function as described in fovea format).
function[xf]=fov(x,c,W,p)

% given, fovea center c in full resolution image, full resolution image width Wand full resolution location x, parameter p indicating fovea region boundaries

% find location xf in foveated image

The conversion700of the foveated pixel address715to the pixel address715uses an inverse foveation function725. Conversion700of the foveated pixel address710from the foveated domain to the pixel address715in full resolution is defined by the following equations:
(x,y)=ifov(xf,yf,cxf,cyf)
(cxf,cyf)=ifov(cxf,cyf,cxf,cyf)

The ifov( ) function provides conversion from fovea resolution to full resolution. (x′,y′)=fov(x,y,cx,cy) defines maps full resolution (x,y) pixel to foveated (x′,y′) pixels. This is used to access (x,y) pixel location in foveated image. This function is defined in this psuedo-code (ifov_scl is inverse fovea function).
function[x]=ifov(xf,cf,Wf,pf)

% Given, fovea center cf in fovea resolution image, fovea resolution image width Wf and fovea resolution location xf parameter pf indicating fovea region boundaries

% find location x in full resolution image

for idx=1:1:Wifovsum(idx+1)=ifovsum(idx)+ifov_scl(idx,pf);

FIG. 8illustrates a foveated warp function800according to embodiments of the present disclosure. The embodiment of the foveated warp function800shown inFIG. 8is for illustration only. Other embodiments of the foveated warp function800could be used without departing from the scope of this disclosure.

The foveated warp function800includes inputs (xoutf,youtf)805, an inverse foveation function810, a warp function815, foveation function820, foveated input image825and foveated output image830. The inputs805, the foveated input image825and the foveated output image830are all in the foveated domain. Because the foveated domain reduces the amount of storage of different areas of an image, a warping in the foveated domain would not be consistent across the stored values. Thus, the image would need to be converted back to the original domain in order for consistent warping.

The foveated warp function800provides an alternative to conversion to a full resolution, only requiring pixel coordinates in the foveated domain. The inverse foveation function810and foveation warp function820are previously described to the respective function inFIG. 7. The warp function815in the foveated domain is defined by the following equations:
(xinf,yinf)=fov(warp(ifov(xoutf,youtf,cxf,cyf)),cx,cy)

where (cxf,cyf) is the fovea center for the foveated image and (cx,cy) is the center for the full image. All pixels coordinates are in the foveated domain and no intermediate conversion to full resolution is required, saving both computation resources and memory access bandwidth.

FIG. 9illustrates original images900, foveated image905with a foveation focus910and restored images915according to the embodiments of the present disclosure. The embodiment of the original image900, a foveated image905and a restored image915shown inFIG. 9is for illustration only. Other embodiments of the original image900, a foveated image905and a restored image915could be used without departing from the scope of this disclosure.

FIGS. 10A, 10B, and 10C illustrate foveated format examples1000,1005,1010according to embodiments of the present disclosure. The embodiment of the foveated format examples1000,1005,1010shown inFIG. 10is for illustration only. Other embodiments of the foveated format examples1000,1005,1010could be used without departing from the scope of this disclosure.

In the foveated format examples1000,1005, and1010, the tiles are scaled at different levels based on a horizontal distance and a vertical distance from the gaze direction. In example1000, the gaze direction is at the center of the gaze direction tile1015. First side tiles1020are the tiles on the edges of the gaze direction tile1015. In example1000, the first side tiles1020are scaled down by a factor of 2×1. First corner tiles1025are the tiles that share two edges with different first side tiles1020and share a corner with the gaze direction tile1015. The first corner tiles can be scaled by a factor of 2×2. The total compression of the foveated image example1000would be about 0.57 of the full image. The fovea function is defined by as Sx=f(|x−cx|), Sy=f(|y−cy|), where Sx and Sy are scale factors for pixel locations. The function f( ) is defined by the following equation:

In example1005, the gaze direction tile1015, first side tiles1020, and first corner tiles1025can be scaled by the same factors as example1000. A second side tile1030are tiles that share an edge of the first side tile1020opposite to the edge shared with the gaze direction tile1015. The second side tile1030can be scaled by a factor of 4×1. The intermediate tiles1035share an edge with the second side tile1030and an edge with the first corner tile1025. The intermediate tiles1035can be scaled by a factor of 4×2. The second corner tiles1040share two edges with different intermediate tiles1035and share a corner with a first corner tile1025. The second corner tiles1040can be scaled down by a factor of 4×4. The total compression of the foveated image example1005would be about 0.36 of the full image. The fovea function is defined by as Sx=f(|x−cx|), Sy=f(|y−cy|), where Sx and Sy are scale factors for pixel locations. The function f( ) is defined by the following equation:

In example1010, scaling for each tile is the same as example1005. The difference is that the tiles are all the same size, except for the tiles on the outer edges of the image. The total compression of the foveated image example1005would be about 0.32 of the full image. The fovea function is defined by as Sx=f(|x−cx|), Sy=f(|y−cy|), where Sx and Sy are scale factors for pixel locations. The function f( ) is defined by the same equation from example1005.

FIG. 11illustrates a foveated tile storage format1100according to embodiments of the present disclosure. The embodiment of the foveated tile storage format1100shown inFIG. 11is for illustration only. Other embodiments of the foveated tile storage format1100could be used without departing from the scope of this disclosure.

The image1105is split into a plurality of tiles1110. The image1105has an image width1115and an image height1120. The image width1115is equal to the tile width times the number of tiles1110per row. The image height1120is equal to the tile height times the number of tiles1110per column. In tile storage format, a first tile location (tx,ty) is found from the global coordinates (x,y). Then within the tile1110, the local pixel location is found based on a scale factor sx,sy. The tile location does not change from the foveated domain to the full resolution domain. The foveated tile1125includes used locations1130and1135. The foveated tile1125is the number of pixels equal to the product of tile width and the tile height over the product of the scale width times the scale height.

The fov( ) function for the tile storage format is defined by the following equations:

where the foveated coordinates can be accessed as a 1D array of size (x/sx,y/sy) to be more efficient in accessing within the allocated space of wx*wy for the tile.

Similarly, the ifov( ) function is defined by the following equations:
[x,y]=[tx,ty]+ifov(xf−tx,yf−ty,sx,sy)
fov(x,y,sx,sy)=(x*sx,y*sy)

where the foveated coordinates can be accessed as a 1D array of size (x/sx,y/sy) to be more efficient in accessing within the allocated space of wx*wy for the tile.

FIGS. 12A-12Hillustrate example foveated functions1200,1201,1202,1203,1204,1205,1206,1207according to embodiments of the present disclosure. The embodiment of the foveated functions1200,1201,1202,1203,1204,1205,1206,1207shown inFIGS. 12A-12Hare for illustration only. Other embodiments of the foveated functions1200,1201,1202,1203,1204,1205,1206,1207could be used without departing from the scope of this disclosure.

Warp function1200is based on reverse mapping and is defined by (xin,yin)=warp (xout,yout). Where warp(x,y) can refer to a formula-based transformation (e.g., affine, perspective, radial, etc.) or a mesh-based look-up table or RAM.

Warp function1201in a foveated domain is defined as (xinf,yinf)=fov(warp(ifov(xoutf,yout,cxf,cyf)),cx,cy). Where (cxf,cyf) is a fovea center for the foveated image and (cx,cy) is a center for the full image. All pixel coordinates are in a foveated domain and no intermediate conversion to a full resolution is required, saving both computation resources and memory access bandwidth.

Function1202illustrates a foveated gaze direction change. If a WARP function needs to accommodate for a change in gaze direction (i.e. center (cx,cy) of central region), the accommodation can be performed by using different address calculations for the input and output sides. The foveated gaze direction change function1202is defined by the following equation:
(xinf,yinf)=fov((ifov(xout,youtf,cxoutf,cyoutf)),cxin,cyin)

where (cxin,cyin) and (cxoutf,cyoutf) are center points in the input image and the output image.

Function1203illustrates a foveated head position change. If both a gaze direction and a head position changes, then the WARP function is defined by an affine matrix. The foveated head position change function1203is defined by the following equation:
(xin,yin)=fov((warp_affine(ifov(xoutf,youtf,cxf,cyf)),cx,cy)

where (cx,cy) are the center points for the input image and the output image.

For a warp function for both a foveated gaze direction and a head position change, the WARP function is defined by an affine matrix. The foveated gaze direction and head position change WARP function1203is defined by the following equation:
(xin,yin)=fov((warp_affine(ifov(xoutf,youtf,cxoutf,cyoutf)),cxin,cyin)

where (cxin,cyin) and (cxoutf,cyout) are the center points for the input image and the output image.

Function1204illustrates a foveated head position change with a full resolution output. If a head position changes while the output is streamed to a display in full resolution format while the input is in foveated format, then the WARP function is defined by an affine matrix. The foveated head position change with a full resolution output function1204is defined by the following equation:
(xin,yin)=fov((warp_affine(xoutf,youtf),cx,cy)

where (cxf,cyf) is the center point for the input image.

Function1205illustrates a foveated head position change with a full resolution input. If a head position changes while the output is streamed to a display in foveated resolution format while the input is in full resolution format, then the WARP function is defined by an affine matrix. The foveated head position change with a full resolution input function1205is defined by the following equation:
(xin,yin)=(warp_affine(ifov(xoutf,youtf,cxoutf,cyoutf))

where (cxoutf,cyoutf) is the center point for the input foveated image.

Function1206illustrates a foveated lens distortion correct. If color aberration and/or lens distortion correction is required while the output is streamed to a display in full resolution format while the input is in foveated format, then the WARP function is defined by an either a LUT-based mesh or radial distortion function. The foveated lens distortion correct function1206is defined by the following equation:
(xin,yin)=(map(ifov(xoutf,youtf,cxf,cyf))

where (cxf,cyf) is the center point for the input foveated image.

Function1206illustrates a foveated head position correction and lens distortion correct. If head position change and/or color aberration and/or lens distortion correction is required while both the input image and the output image are in full resolution format, then an intermediate storage can be in a foveated format. The warp affine for head rotation of the function1206is defined by the following equation:
(xin,yin)=(warp_affine(ifov(xintf,yintf,cxf,cyf))

where (cxf,cyf) is the center point for the intermediate foveated image. The lens distortion correction of the function1206is defined by the following equation:
(xintf,yintf)=fov(map(xout,yout),cx,cy)

It is possible that an input and an output for fovea parameters are different. Sometimes the input is allowed to have wider fovea to allow a shift in the fovea center from input to output or because of change in orientation. In this situation, parameter p is passed inside fov( ) and ifov( ) functions as shown in psuedo-codes. For example, initial rendering from GPU could be done with wider fovea and then final post-rendering correction can be done with more compact fovea.

It is possible to use foveated rendering based on lighting parameters instead of gaze direction. For example, the resolution on a certain portion of the image could be based on lighting in that portion of the image. This can be done by splitting an image based on lighting conditions. A tile-based strategy should work the best where the tile is defined by tile lighting conditions to derive required resolution in that tile.

It is possible for post rendering correction block to apply delta correction only based on some movements in certain object. In that case only the moving portion of the image needs to be updated. This can be done by estimating motion map across the image and then warping based on that motion map. This warping operation can be done in natively foveated domain instead of full resolution similar to time-warp.

FIG. 13illustrates an exemplar flow diagram1300for foveated domain storage and processing according to the various embodiments of the present disclosure. While the flow chart depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process depicted inFIG. 13can be performed by the electronic device101inFIG. 1.

In operation1305, the electronic device101performs head tracking and eye tracking. The electronic device uses the position and angle of the head to determine the image frame and focus of the eyes to determine a gaze direction.

In operation1310, the electronic device101generates a foveated image from an original image based on the head tracking and the eye tracking. The foveated image uses a step reduction in resolution based on the angular distance from the gaze direction.

In operation1315, the electronic device101stores the foveated image using one of a tile-based method or a frame-based method. The tile-based method includes dividing the original image into a plurality of tiles and scaling each of the plurality of tiles based on a distance of each tile from a gaze direction, where the foveated image is further generated based on the plurality of scaled tiles. The frame-based method includes applying a scale factor for each pixel based on a horizontal distance and a vertical distance from the gaze direction.

In operation1320, the electronic device101performs a function on the foveated image when a condition occurs. The function that is performed can be any of functions1200-1207.

AlthoughFIG. 13illustrates an example process, various changes could be made toFIG. 13. For example, while shown as a series of steps, various steps in each figure could overlap, occur in parallel, occur in a different order, or occur multiple times.