Point-of-view image warp systems and methods

An electronic device may include an electronic display to display an image based on processed image data. The electronic device may also include image processing circuitry to generate the processed image data. The image processing circuitry may receive input image data corresponding to an image in a first perspective and warp the input image data from the first perspective to a second perspective, generating warped image data. Additionally, the image processing circuitry may determine one or more occluded regions in the second perspective and determine fill-data corresponding to the occluded regions. The processed image data may be generated by combining the warped image data and the fill-data.

BACKGROUND

The present disclosure relates generally to image processing and, more particularly, to image warp operations correcting for a change in point-of-view.

Electronic devices often use one or more electronic displays to present visual information such as text, still images, and/or video by displaying one or more images. For example, such electronic devices may include computers, mobile phones, portable media devices, tablets, televisions, virtual-reality headsets, and vehicle dashboards, among many others. To display an image, an electronic display may control light emission of its display pixels based at least in part on corresponding image data.

An image may be represented by image data defining areas with particular resolutions or distributions of pixel values. However, in some instances, it may be desirable to change the resolutions or distributions of the pixel values to account for different display scenarios. For example, image data may be warped to account for environmental surroundings, display characteristics, a viewer's point-of-view (POV) and other factors that may distort the perceived image to a viewer. Thus, before being displayed, the image data may be processed to warp the image using the desired changes to the amount or distribution of pixel values such that the perceived image is not distorted. However, at least in some instances, when warping for changes in POV, objects may be revealed for which no image data exists, which may lead to image artifacts.

SUMMARY

In some scenarios, an image may appear distorted when perceived by a viewer due to differences between the perspective with which the image was captured or rendered and the viewer's point-of-view (POV)/perspective. Furthermore, a viewer's POV (e.g., as determined based on location and/or eye-tracking) relative to the display may alter how the viewer perceives the image. For example, in augmented, virtual, or mixed reality devices, an image may be captured, previously or in real time, and displayed with or without augmentation to a viewer as if the image was reality. However, the image may have been captured with a POV different from that which the viewer would normally experience. For example, objects may appear shorter, taller, wider, smaller, or otherwise out of perspective relative to itself or other objects if the image is left uncorrected. Thus, before being displayed, the image data may be processed to warp the image such that the perceived image has reduced or no distortion.

A POV warp may assist in altering the image data such that, when displayed, the image appears from a perspective consistent with that of the viewer. Such a POV warp may also shift objects' relative positions and generate one or more occluded regions in the image. For example, a foreground object may shift in a POV warp such that a portion of the object or background is visible after the warp that was not previously a part of the input image. As such, in some embodiments, occluded regions may be filled-in by blending, generating new pixel values, or additional image captures (e.g., from additional cameras. Additionally or alternatively, in some embodiments, filling-in the occluded regions may utilize machine learning (e.g., deep learning) to estimate the missing pixel values.

Additionally, in some embodiments, the correction for the POV warp may utilize a pixel grid to map pixel locations in the input image data to the warped locations corrected for the viewer's POV. During generation of the pixel grid, occlusions may be identified as invalid regions having no input image data associated therewith. As the occluded regions will be filled-in with fill-data, the invalid regions may be ignored when warping and fetching image data to increase operational efficiency.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

Electronic devices often use electronic displays to present visual information. Such electronic devices may include computers, mobile phones, portable media devices, tablets, televisions, virtual-reality headsets, and vehicle dashboards, among many others. An electronic display may take the form of a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a plasma display, or the like.

To display an image, an electronic display controls the luminance (and, as a consequence, the color) of its display pixels based on corresponding image data received at a particular resolution. For example, an image data source may provide image data as a stream of pixel data, in which data for each pixel indicates a target luminance (e.g., brightness and/or color) of one or more display pixels located at corresponding pixel positions. In some embodiments, image data may indicate luminance per color component, for example, via red component image data, blue component image data, and green component image data, collectively referred to as RGB image data (e.g., RGB, sRGB). Additionally or alternatively, image data may be indicated by a luma channel and one or more chrominance channels (e.g., YCbCr, YUV, etc.), grayscale (e.g., gray level), or other color basis. It should be appreciated that a luma channel, as disclosed herein, may encompass linear, non-linear, and/or gamma-corrected luma values.

In some scenarios, the image to be displayed may, if unaltered, appear distorted when perceived by a viewer. Distortions could be due to environmental effects, properties of the display, the viewer's point-of-view (POV) perspective, image processing warps such as shifts and scaling, and/or other distorting factors. For example, the display may include a screen, opaque or transparent, with curved edges and/or lensing effects that may distort an image if displayed without correction. Furthermore, a viewer's POV (e.g., as determined based on location and/or eye-tracking) relative to the display may alter how the viewer perceives the image. For example, in augmented reality (AR), virtual reality (VR), or mixed reality (MR) devices, an image may be captured, previously or in real time, and displayed with or without augmentation to a viewer as if the image was reality. However, the image may have been captured with a POV different from that which the viewer would normally experience if the image were reality. For example, objects may appear shorter, taller, wider, smaller, or otherwise out of perspective relative to itself or other objects. Thus, before being displayed, the image data may be processed to warp the image such that the perceived image has reduced or no distortion.

A POV warp may cause one or more occluded regions in the image. For example, a foreground object may shift in a POV warp such that a portion of the object or background is visible after the warp that was not previously a part of the input image. As such, in some embodiments, occluded regions may be filled-in by blending, generating new pixel values, additional image captures (e.g., from additional cameras), etc. Additionally or alternatively, in some embodiments, filling-in the occluded regions may utilize machine learning (e.g., deep learning) to estimate the missing pixel values.

In some embodiments, the correction for some types of warps, for example the POV warp may utilize a pixel grid to map pixel locations in the input image data to the warped locations corrected for the viewer's POV. However, warps that use a coarser grid may result in image artifacts, while finer grid warping may be taxing on processing bandwidth and/or take additional time. To avoid image artifacts and to increase efficiency, a hierarchical grid may have a variable grid partition size when performing the warp operation. In some embodiments, the hierarchical grid may have a quad-tree grid structure determined by reiteratively splitting grid partitions into fourths depending on image statistics associated with a corresponding portion of the image data. For example, the hierarchical grid may be determined by starting with 64×64 pixel partitions. Depending on the image statistics (e.g., homogeny, occlusions, perceived depth, edges, or other image features that may warrant finer warp calculations), that particular 64×64 partition of the grid may remain a full partition or the partition may be split into four partitions of 32×32 pixels. Each new partition may be maintained or further split depending on the image statistics until a desired granularity is reached. Furthermore, some partitions may be designated as invalid by being out of an active area of the display or as including an occlusion. As discussed herein invalid regions may be ignored for warp purposes to reduce bandwidth and filled-in separately, if necessary (e.g., in the active area).

With the coordinate mapping to the input image data for each partition, hierarchical interpolation may be used to obtain coordinates for each grid point. For example, coordinates corresponding to a 32×32 pixel partition may be interpolated from the 64×64 partition coordinates. Subsequently, coordinates corresponding to a 16×16 pixel partition may be interpolated from the 32×32 partition coordinates and so on until a coordinate mapping is determined for each grid point. Additionally, at each stage of interpolation, if the coordinates for a partition size is available (e.g., the grid was split into smaller than 64×64 partitions), coordinates for the corresponding pixels may be already calculated as part of the grid and may be directly used instead of interpolated.

By reducing warp calculations in partitions where artifacts are less likely and performing finer warp operations in certain regions based on the image statistics, processing time and/or bandwidth usage may be decreased while image quality is maintained. Moreover, the reduced processing time may provide for real-time or enhanced feedback to a user's change in position/POV. As should be appreciated, the grid sizing is given as an example and is non-limiting. Further, the disclosed techniques regarding the pixel grid may be used for any of multiple different image processing operations including but not limited to warps such as the POV warps, geometric distortion warps, temporal warps, etc. Indeed, other image processing operations may utilize a hierarchical grid structure for increased efficiency and/or reduced processing time. Accordingly, to improve image quality and/or increase efficiency, the present disclosure provides techniques for a general hierarchical grid and for enhanced POV warping.

One embodiment of an electronic device10that utilizes the hierarchical grid is shown inFIG.1. As will be described in more detail below, the electronic device10may be any suitable electronic device, such as a handheld electronic device, a tablet electronic device, a notebook computer, and the like. Thus, it should be noted thatFIG.1is merely one example of a particular implementation and is intended to illustrate the types of components that may be present in the electronic device10.

The electronic device10may include one or more electronic displays12, input devices14, input/output (I/O) ports16, a processor core complex18having one or more processors or processor cores, local memory20, a main memory storage device22, a network interface24, a power source26, and image processing circuitry28. The various components described inFIG.1may include hardware elements (e.g., circuitry), software elements (e.g., a tangible, non-transitory computer-readable medium storing instructions), or a combination of both hardware and software elements. As should be appreciated, the various components may be combined into fewer components or separated into additional components. For example, the local memory20and the main memory storage device22may be included in a single component. Additionally, the image processing circuitry28(e.g., a graphics processing unit, a display image processing pipeline, etc.) may be included in the processor core complex18.

The processor core complex18may be operably coupled with local memory20and the main memory storage device22. The local memory20and/or the main memory storage device22may include tangible, non-transitory, computer-readable media that store instructions executable by the processor core complex18and/or data to be processed by the processor core complex18. For example, the local memory20may include random access memory (RAM) and the main memory storage device22may include read only memory (ROM), rewritable non-volatile memory such as flash memory, hard drives, optical discs, and/or the like.

The processor core complex18may execute instructions stored in local memory20and/or the main memory storage device22to perform operations, such as generating source image data. As such, the processor core complex18may include one or more general purpose microprocessors, one or more application specific processors (ASICs), one or more field programmable logic arrays (FPGAs), or any combination thereof.

The network interface24may connect the electronic device10to a personal area network (PAN), such as a Bluetooth network, a local area network (LAN), such as an 802.11x Wi-Fi network, and/or a wide area network (WAN), such as a 4G, LTE, or 5G cellular network. In this manner, the network interface24may enable the electronic device10to transmit image data to a network and/or receive image data from the network.

The power source26may provide electrical power to operate the processor core complex18and/or other components in the electronic device10. Thus, the power source26may include any suitable source of energy, such as a rechargeable lithium polymer (Li-poly) battery and/or an alternating current (AC) power converter.

The I/O ports16may enable the electronic device10to interface with various other electronic devices. The input devices14may enable a user to interact with the electronic device10. For example, the input devices14may include buttons, keyboards, mice, trackpads, and the like. Additionally or alternatively, the electronic display12may include touch sensing components that enable user inputs to the electronic device10by detecting occurrence and/or position of an object touching its screen (e.g., surface of the electronic display12).

The electronic display12may display a graphical user interface (GUI) of an operating system, an application interface, text, a still image, or video content. To facilitate displaying images, the electronic display12may include a display panel with one or more display pixels. Additionally, each display pixel may include one or more sub-pixels, which each control the luminance of a color component (e.g., red, green, or blue). As used herein, a display pixel may refer to a collection of sub-pixels (e.g., red, green, and blue subpixels) or may refer to a single sub-pixel.

As described above, the electronic display12may display an image by controlling the luminance of the sub-pixels based at least in part on corresponding image data. In some embodiments, the image data may be received from another electronic device, for example, via the network interface24and/or the I/O ports16. Additionally or alternatively, the image data may be generated by the processor core complex18and/or the image processing circuitry28. Moreover, in some embodiments, the electronic device10may include multiple electronic displays12and/or may perform image processing (e.g., via the image processing circuitry28) for one or more external electronic displays12, such as connected via the network interface24and/or the I/O ports16.

The electronic device10may be any suitable electronic device. To help illustrate, one example of a suitable electronic device10, specifically a handheld device10A, is shown inFIG.2. In some embodiments, the handheld device10A may be a portable phone, a media player, a personal data organizer, a handheld game platform, and/or the like. For example, the handheld device10A may be a smart phone, such as an iPhone® model available from Apple Inc.

The handheld device10A may include an enclosure30(e.g., housing) to, for example, protect interior components from physical damage and/or shield them from electromagnetic interference. Additionally, the enclosure30may surround, at least partially, the electronic display12. In the depicted embodiment, the electronic display12is displaying a graphical user interface (GUI)32having an array of icons34. By way of example, when an icon34is selected either by an input device14or a touch-sensing component of the electronic display12, an application program may launch.

Furthermore, input devices14may be provided through openings in the enclosure30. As described above, the input devices14may enable a user to interact with the handheld device10A. For example, the input devices14may enable the user to activate or deactivate the handheld device10A, navigate a user interface to a home screen, navigate a user interface to a user-configurable application screen, activate a voice-recognition feature, provide volume control, and/or toggle between vibrate and ring modes. Moreover, the I/O ports16may also open through the enclosure30. Additionally, the electronic device may include one or more cameras36to capture pictures or video. In some embodiments, a camera36may be used in conjunction with a virtual reality or augmented reality visualization on the electronic display12.

Another example of a suitable electronic device10, specifically a tablet device10B, is shown inFIG.3. For illustrative purposes, the tablet device10B may be an iPad® model available from Apple Inc. A further example of a suitable electronic device10, specifically a computer10C, is shown inFIG.4. For illustrative purposes, the computer10C may be a MacBook® or iMac® model available from Apple Inc. Another example of a suitable electronic device10, specifically a watch10D, is shown inFIG.5. For illustrative purposes, the watch10D may be an Apple Watch® model available from Apple Inc. As depicted, the tablet device10B, the computer10C, and the watch10D each also includes an electronic display12, input devices14, I/O ports16, and an enclosure30.

As described above, the electronic display12may display images based at least in part on image data. Before being used to display a corresponding image on the electronic display12, the image data may be processed, for example, via the image processing circuitry28. In general, the image processing circuitry28may process the image data for display on one or more electronic displays12. For example, the image processing circuitry28may include a display pipeline, memory-to-memory scaler and rotator (MSR) circuitry, warp compensation circuitry, or additional hardware or software means for processing image data. The image data may be processed by the image processing circuitry28to reduce or eliminate image artifacts, compensate for one or more different software or hardware related effects, and/or format the image data for display on one or more electronic displays12. As should be appreciated, the present techniques may be implemented in standalone circuitry, software, and/or firmware, and may be considered a part of, separate from, and/or parallel with a display pipeline or MSR circuitry.

To help illustrate, a portion of the electronic device10, including image processing circuitry28, is shown inFIG.6. In some embodiments, the image processing circuitry28may be implemented by circuitry in the electronic device10, circuitry in the electronic display12, or a combination thereof. For example, the image processing circuitry28may be included in the processor core complex18, a timing controller (TCON) in the electronic display12, or any combination thereof. As should be appreciated, although image processing is discussed herein as being performed via a number of image data processing blocks, embodiments may include hardware or software components to carry out the techniques discussed herein.

The electronic device10may also include an image data source38, a display panel40, and/or a controller42in communication with the image processing circuitry28. In some embodiments, the display panel40of the electronic display12may be a liquid crystal display (LCD), a light emitting diode (LED) display, an organic LED (OLED) display, or any other suitable type of display panel40. In some embodiments, the controller42may control operation of the image processing circuitry28, the image data source38, and/or the display panel40. To facilitate controlling operation, the controller42may include a controller processor44and/or controller memory46. In some embodiments, the controller processor44may be included in the processor core complex18, the image processing circuitry28, a timing controller in the electronic display12, a separate processing module, or any combination thereof and execute instructions stored in the controller memory46. Additionally, in some embodiments, the controller memory46may be included in the local memory20, the main memory storage device22, a separate tangible, non-transitory, computer-readable medium, or any combination thereof.

The image processing circuitry28may receive source image data48corresponding to a desired image to be displayed on the electronic display12from the image data source38. The source image data48may indicate target characteristics (e.g., pixel data) corresponding to the desired image using any suitable source format, such as an 8-bit fixed point αRGB format, a 10-bit fixed point αRGB format, a signed 16-bit floating point αRGB format, an 8-bit fixed point YCbCr format, a 10-bit fixed point YCbCr format, a 12-bit fixed point YCbCr format, and/or the like. In some embodiments, the image data source38may be included in the processor core complex18, the image processing circuitry28, or a combination thereof. Furthermore, the source image data48may reside in a linear color space, a gamma-corrected color space, or any other suitable color space. As used herein, pixels or pixel data may refer to a grouping of sub-pixels (e.g., individual color component pixels such as red, green, and blue) or the sub-pixels themselves.

As described above, the image processing circuitry28may operate to process source image data48received from the image data source38. The data source38may include captured images from cameras36, images stored in memory, graphics generated by the processor core complex18, or a combination thereof. The image processing circuitry28may include one or more sets of image data processing blocks50(e.g., circuitry, modules, or processing stages) such as the warp processing block52. As should be appreciated, multiple other processing blocks54may also be incorporated into the image processing circuitry28, such as a color management block, a dither block, a rotate block, etc. Furthermore, in some embodiments, multiple warp processing blocks52may be used to provide separate warp operations for different applications of the image processing circuitry28. For example, different warp processing blocks52may be used for image data from different image data sources38(e.g., captured images, graphically generated images, etc.). The image data processing blocks50may receive and process source image data48and output display image data56in a format (e.g., digital format and/or resolution) interpretable by the display panel40. Further, the functions (e.g., operations) performed by the image processing circuitry28may be divided between various image data processing blocks50, and while the term “block” is used herein, there may or may not be a logical separation between the image data processing blocks50.

The warp processing block52may facilitate improving perceived image quality by changing the amount (e.g., resolution) or distribution (e.g., shape, relative size, perspective, etc.) of pixel values to account for different display scenarios during one or more warp operations. To help illustrate,FIG.7is a block diagram58of a warp processing block52including a point-of-view (POV) warp sub-block60and a hierarchical grid interpolation sub-block62. In general, the warp processing block52may receive input image data64and generate processed image data66by applying one or more warps to the input image data64. As should be appreciated, the warp processing block52may include other warp sub-blocks for geometric warps, lens correction warps, temporal warps, etc. Moreover, the POV warp sub-block60may be combined (e.g., stacked) with one or more other warps in a combined warp operation.

The POV warp sub-block60may provide compensation for a viewer's POV (e.g., as determined based on location, eye relief, and/or focus) relative to the display as compared to the capture of the image to be displayed, for example via one or more cameras36. Moreover, the viewer's POV may be monitored via any suitable method such as eye-tracking. Additionally, a height of the viewer may be determined based on a user input or a relative height difference between the electronic device10and an estimated position of the viewer with respect to the electronic device10.

As non-limiting examples, augmented, virtual, or mixed reality devices, may capture images, previously or in real time, and display them with or without augmentation to a viewer as if the image was physically in view of the viewer. However, the image may have been captured with a perspective different from that which the viewer would normally experience, which may alter how the viewer perceives the image. For example, objects may appear shorter, taller, wider, smaller, or otherwise out of perspective relative to themselves or other objects. Thus, in some embodiments, the POV warp sub-block60may warp the image data before it is displayed such that the perceived image has reduced or no distortion.

In some scenarios, a POV warp68from a source frame70(a.k.a. a source space) to a POV warped frame72(a.k.a. a warped space) may cause one or more occluded regions74in the image76, as shown inFIG.8. For example, a foreground object78may shift in a POV warp68such that a portion of the foreground object or the background80is visible after the POV warp68that was not previously a part of the image76. As such, after the POV warp68there may be portions of the POV warped frame72that do not have mappings to the input image data64in the source frame70.

In some embodiments, the warp processing block52may generate image data to fill-in an occluded region74in the POV warped frame72. For example, the occluded region74may be filled-in by blending pixel values surrounding the occluded region74or by using image data from additional image captures such as images captured at a different time (e.g., before or after the original capture) from one or more different angles and/or by an additional image capture device (e.g., camera36) having a different perspective. For example, in some embodiments, cameras36in stereo provide perspectives that may be combined to reduce or eliminate each other's occluded regions74. In such a case, input image data64from the second camera36may undergo a POV warp68to the POV warped frame72, and the portion corresponding to the occluded region74may be used as fill-data. Additionally or alternatively, in some embodiments, the warp processing block52may generating new pixel values to estimate and fill-in the pixel values of the occluded region74by utilizing painting algorithms and/or machine learning (e.g., deep learning).

Additionally, in some embodiments, sampling or resampling may occur with a window82that overlaps with or abuts edges of an occluded region74. In some scenarios, filtering across occluded region74boundaries may generate artifacts84. As such, in some embodiments, filtering may be reduced or eliminated for areas about the occluded region74.

FIG.9is a flowchart of an example process86performed by the warp processing block52. In general, the warp processing block52may receive input image data64and/or image statistics indicative thereof (process block88). The warp processing block52may also apply a POV warp68to a pixel grid to generate a mapping of pixel locations in the source frame70to pixel locations in the POV warped frame72(process block90). The input image data64may be mapped, according to the warped pixel grid coordinate values to the POV warped frame72(process block92). Additionally, fill-data for occluded regions74may be determined (process block94). The fill-data and the warped image data in the POV warped frame72may be joined (e.g., blended) to generate the processed image data66(process block96), and the processed image data66may then be output (process block98) to the other processing blocks54and/or the display panel40.

In some embodiments, to apply different types of warps, for example the POV warp68and/or the warp processing block52may utilize a pixel grid to map the pixel locations in the source frame70to locations in the POV warped frame72corrected for the viewer's POV. Performing the warp on a pixel grid may increase efficiency by creating a mapping of pixel coordinates without fetching or processing all of the input image data. For example, after the POV warp68or other warp operation certain image data may be superfluous, and reducing or eliminating fetching and/or processing of unused data may increase efficiency. In general, the pixel grid may include grid points that define mappings to coordinates of the input image data64in the source frame70(i.e., source space). The mappings may be generated based on the POV warp68, which may be accomplished in software or hardware. After determining the mappings, portions of the input image data64, corresponding to the coordinate mappings of the grid points, may be fetched to resolve the pixel grid, generating the image76in the POV warped frame72.

In some scenarios, a warp using a coarser grid may result in one or more image artifacts such as blurring. On the other hand, while a warp using a finer grid may provide increased clarity and reduced image artifacts, finer grids may be taxing on processing bandwidth and/or take additional time to render. As such, in some embodiments, the warp processing block52may utilize a hierarchical grid100, as inFIG.10. The hierarchical grid100may have variable size grid partitions102A,102B,102C,102D (cumulatively102) when performing a warp operation. For example, each warped grid partition102may correlate a single pixel coordinate in the source frame70to a single pixel the warped frame (e.g., POV warped frame72), such as a corner or center grid point of the grid partition102. Depending on the granularity of the grid partition102, the mapped coordinates to the source frame70for a number of other grid points (e.g., larger number for coarser/larger grid partitions102and smaller number for finer/smaller grid partitions102) within the grid partition102may be interpolated (e.g., by the hierarchical grid interpolation sub-block62).

In some embodiments, the hierarchical grid100may have a quad-tree grid structure determined by reiteratively splitting or merging grid partitions102by fourths depending on image statistics associated with a corresponding portion of the input image data64. For example, the hierarchical grid100may be determined via a top-down split104starting with a uniform grid106of relatively larger grid partitions102. The coarse uniform grid106may undergo zero, one, or more intermediate split partition stages108before arriving at the hierarchical grid100. Additionally or alternatively, a hierarchical grid100may be generated by bottom-up merging110. Bottom-up merging110may begin with a uniform grid112of relatively smaller grid partitions102that may undergo zero, one, or more intermediate merged partition stages114before arriving at the hierarchical grid100. Moreover, the top-down split104and/or bottom-up merge110may be accomplished iteratively or in a single pass. Furthermore, a base partition116may be set depending on a desired maximum granularity that the coarse uniform grid106may start with during a top-down split104. Similarly, in some embodiments, the bottom-up merge110may not merge to include grid partitions102larger than the base partition116. Although discussed herein as having a quad-tree grid structure, the hierarchical grid100may use any suitable structure depending on implementation. For example, the structure of the hierarchical grid may bifurcate in halves, fourths, eighths, or any suitable fraction at each hierarchical tier/level.

To help further illustrate,FIG.13is a top-down split104of a 64×64 base partition116in accordance withFIG.10. As should be appreciated, the size of the base partition116(e.g., 64×64), the smallest partition102(e.g., 4×4 grid partition102D), and/or the step sizes of the splits or merger may vary depending on implementation such as bandwidth and/or image quality desires. In general, depending on the image statistics of corresponding input image data64(e.g., homogeny, occlusions, active area location, perceived depth, edges, and/or other image features that may warrant finer warp calculations), a particular base partition116may remain or be split into four 32×32 grid partitions102A. Each new grid partition102may be maintained or further split depending on the image statistics until no more splits are desired or a smallest partition102is achieved. For example, if a grid partition102is generally homogenous and has no occlusions regions74, the grid partition102may be maintained, whereas if a grid partition102has occlusions, is significantly non-homogenous (e.g., relative to a set or calculated parameter), or includes edges or changes in perceived depth greater than a set or calculated threshold amount, the grid partition102may be split to increase granularity in such areas of interest. In the depicted example, one 32×32 grid partition102A is split into four 16×16 grid partitions102B, and one of the 16×16 grid partitions102B is split into four 8×8 grid partitions102C.

In addition to modulating the granularity based on the image statistics for areas of interest relative to the input image data64, the hierarchical grid100may also be partitioned based on an active area of the display12. For example, the input image data64may include portions of the image76that lie outside the displayed portion (e.g., active area) of the image76. In some embodiments, grid partitions102of the hierarchical grid100that are outside of the active area and/or correspond to occluded regions74may be designated as invalid region118or partition. In some embodiments, invalid region118may be ignored (e.g., corresponding input image data64not fetched) for warp purposes to reduce bandwidth usage and increase speed and efficiency.

When a warp (e.g., the POV warp68, a geometric warp, a temporal warp, etc.) is applied to the hierarchical grid100by the POV warp sub-block60or the hierarchical grid interpolation sub-block62, at least one characteristic grid point for each grid partition102is mapped to a pixel coordinate in the source frame70of the input image data64. The mapping may be calculated in hardware or software based on POV parameters corresponding to the viewer's eye's location, eye relief, eye focus and/or other POV calculations relative to the display12, image capturing mechanism (e.g., camera36), or object of interest. As should be appreciated, more than one characteristic grid point may be mapped per grid partition102depending on implementation. Moreover, the characteristic grid point may be aligned with an edge, corner, or middle of the grid partition102for simplistic reference. For example, the characteristic grid point may be the top left corner grid point of each grid partition102and mapped to coordinates (e.g., “x” and “y” coordinates relative to a pixel grid of the input image data64) in the source frame70. Furthermore, while at the characteristic grid point mapping for each grid partition102is known (e.g., calculated), the additional grid points of the grid partitions102may be unknown from the partition mapping. As such, the hierarchical grid interpolation sub-block62may hierarchically interpolate between the known characteristic grid point coordinate mappings to determine the coordinates for the remaining grid point mappings.

FIG.14is a schematic diagram120of the hierarchical interpolation beginning with a base partition116(e.g., 64×64 grid partition). In the case where the base partition116was not split into smaller grid partitions102, the known x and y coordinates mappings for the characteristic grid points of multiple surrounding base partitions116may be used to interpolate midpoints between the known coordinate mappings of the characteristic grid points. Because of the construction of the hierarchical grid100, such midpoints may be aligned with the characteristic grid points of the next tiered 32×32 grid partition102A. As such, the characteristic grid points for the 32×32 grid partitions102A and the base partition116may be determined. Further, the characteristic grid points of the base partition116and the 32×32 grid partitions102A may be interpolated to determine the characteristic grid points for the 16×16 grid partitions102B and so on until the characteristic grid points for the smallest grid partitions (e.g., 4×4 grid partitions102D) are determined. However, when the base partition116has already been split into smaller grid partitions102, at least some of the characteristic grid points for the smaller grid partitions may have already been calculated and, therefore, be available for completing the warped grid without interpolation. Indeed, at each tier of the hierarchical interpolation, known values may be input from their respective grid partitions (e.g.,102A-102D) and unknown values may be interpolated. In some embodiments, iterative interpolation may utilize previously interpolated coordinate mappings to interpolate additional coordinate mappings.

Furthermore, interpolation may continue until the coordinate mappings are interpolated for a 1×1 grid partition size, completing the mapping for each pixel position in the warped grid to the input image data64of the source frame70. As should be appreciated, any suitable method for interpolation such as bilinear or bicurvature interpolation may be used to interpolate the coordinate mappings. In some embodiments, a more accurate interpolation (e.g., bicurvature interpolation) may be used while determining higher tier interpolations such as the 32×32 grid point interpolations and 16×16 grid point interpolations while lower tier interpolations such as the 1×1 grid interpolations may use more efficient interpolation methods such as bilinear interpolation. As should be appreciated, although stated herein as using “x” and “y” coordinates, any suitable coordinate system may be used to map the input image data64to warped image data such as the POV warped frame72. Furthermore, in some embodiments, the “x” and “y” coordinates interpolations may be treated separately and to allow for parallel processing of the interpolations.

To help further illustrate the hierarchical interpolation,FIG.15is a schematic diagram of a hierarchical grid100having multiple base partitions116. As discussed above, the mappings for the characteristic grid points122for base partitions116may be calculated based on the warp from the source frame70to the destination frame (e.g., POV warped frame72or other warped frame). In the case where the base partition116was not split into smaller grid partitions102, the known x and y coordinates mappings for the characteristic grid points122of the base partitions116may be used to interpolate grid points half way between them. Because of the construction of the hierarchical grid100, such interpolated grid points may be aligned with the characteristic grid points124of the next tiered grid partition102, such as the 32×32 grid partition102A. As such, the characteristic grid points124for the 32×32 grid partitions102A and the base partition116may be known after the first level of interpolation. Additionally, if one or more of the base partitions116were split into 32×32 grid partitions102A or smaller grid partitions102B-102D, some of the characteristic grid points124for the 32×32 grid partitions102A may already be known from the warp calculation. In some embodiments, interpolations for grid points previously calculated may be skipped or performed and ignored in favor of the warp calculated grid points.

Using the characteristic grid points122of the base partitions116and the characteristic grid points124for the 32×32 grid partitions102A, the characteristic grid points126for the 16×16 grid partitions102B may be interpolated. As with the 32×32 grid partitions102A, if one or more base partitions116were split into 16×16 grid partitions102B or smaller grid partitions102C,102D, some of the characteristic grid points126for the 16×16 grid partitions102B may already be known from the warp calculation. The iterated interpolations may continue at each tier of the hierarchical interpolation until mappings for each grid point are determined, either from the warp operation or by interpolation.

Additionally, while depicted as the upper-left-most grid point, as should be appreciated, any relative position within the base partition116and grid partitions102may be used as characteristic grid points122,124,126. Furthermore, as mentioned above, although the base partition116is exampled by a 64×64 grid partition, any suitable size base partition116may be utilized with iterated interpolations occurring until each grid point is resolved.

FIG.16is a flowchart of an example process128for using a hierarchical grid100in mapping a warp operation from a source frame70to a warped frame such as the POV warped frame72. The image processing circuitry28, such as warp processing block52, may receive input image data64and/or image statistics based on the input image data64(process block130). In some embodiments, the warp processing block52may determine the image statistics on-the-fly and/or receive the image statistics from another processing block54. The image processing circuitry28may also determine invalid regions118(process block132) and the grid partitions102of the hierarchical grid100(process block134). For example, occluded regions74and/or regions outside an active area of the display12may be deemed invalid, and, in some embodiments, no input image data64corresponding to the invalid regions118may be fetched to save bandwidth. Moreover, the grid partitions102may be determined based on the image statistics which may include whether a region is invalid and/or if edge effects, changes in perceivable depth, or homogeneity, or other image features exist in the region. Further, the hierarchical grid100may be warped according to the grid partitions102(process block136). Warping the hierarchical grid may include determining mappings for characteristic grid points of the grid partitions102to the source frame70(process block138) and determining hierarchical interpolations (process block140). The image processing circuitry28may then map the input image data64according to the warped grid to generate the warped image data (process block142), and the warped image data may be output (process block144). As should be appreciated, applying the mappings of the input image data64from the source frame70to a destination frame (e.g., a warped frame) may include applying the mappings of the warped grid to the input image data64or fetching the values of the corresponding input image data64mapped in the warped grid and assembling them in place of the warped grid. In some embodiments, occluded regions74or other invalid regions118, if present, may be filled-in by blending surrounding pixel values, by using image data from additional image captures such as images captured at a different time or an additional camera36having a different perspective, and/or by generating new image data based on characteristics of the image76. As such, processed image data66may be output for additional image processing or as display image data56.

By reducing warp calculations in partitions where artifacts are less likely and performing finer warp operations in certain regions based on the image statistics, processing time and/or bandwidth usage may be decreased while image quality is maintained. Accordingly, the present techniques for a general hierarchical grid and for enhanced POV warping improve image quality and/or increase efficiency provide for real-time or enhanced feedback to a user's change in position/POV.

Although the above referenced flowcharts are shown in a given order, in certain embodiments, process/decision blocks may be reordered, altered, deleted, and/or occur simultaneously. Additionally, the referenced flowcharts are given as illustrative tools and further decision and process blocks may also be added depending on implementation.