CHROMATIC ABERRATION CORRECTION FOR FOVEATED DISPLAY

An electronic device uses a chromatic aberrations correction (CAC) circuit to correct chromatic aberration on a display panel. An input image is warped based on a first color channel only geometric distortions associated with displaying the input image on the display panel. The CAC circuit applies spatial resampling to other color channels to compensate chromatic aberration based on distortion offsets between the other color channels and the first color channel

BACKGROUND

The present disclosure relates generally to displayed image processing and, more particularly, to chromatic aberration correction (CAC) in a foveated electronic display.

Electronic displays are found in numerous electronic devices, such as mobile phones, computers, televisions, automobile dashboards, and augmented reality or virtual reality glasses, to name just a few. Electronic displays control the amount of light emitted from their display pixels based on corresponding image data to produce images. Processing circuitry of the electronic device may generate or retrieve the image data that may be used to program the display pixels of the electronic display to display an image. In some scenarios, the image to be displayed may appear distorted when perceived by a viewer due to environmental effects, properties of the display, the viewer's point-of-view (POV), image processing alterations such as shifts and scaling, and/or other distorting factors. For example, if the electronic display includes a screen or a filter with curved edges and/or lensing effects, distortion such as lateral chromatic aberration may occur.

SUMMARY

Before being displayed, the image data may be processed to warp the image using the desired changes to the amount (e.g., resolution) or distribution (e.g., shape, relative size, perspective) of pixel values such that the perceived image is not distorted. This disclosure provides systems and methods for using a chromatic aberration correction (CAC) block to perform correction to compensate for lateral chromatic aberrations. Chromatic aberrations are color-dependent distortions; therefore, the distortions may be different for different color channels (e.g., red channel, green channel, blue channel). The image processing circuitry may include a CAC block, and the input signals to the CAC block may be pre-warped in frontend warp pipes based on a geometric distortion applied to a single first color channel (e.g., green channel). The CAC block may apply spatial resampling to other color channels (e.g., red and blue channels) to compensate for chromatic aberration based on distortion offsets (e.g., vertical offsets, horizontal offsets) between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel). Multiple warped color channels of image data may be blended into a single data path. The corrected image data for all color channels may be combined together and output for display.

DETAILED DESCRIPTION OF SPECIFIC EMBODIMENTS

When introducing elements of various embodiments of the present disclosure, the articles “a,” “an,” and “the” are intended to mean that there are one or more of the elements. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. Additionally, it should be understood that references to “one embodiment”, “an embodiment”, or “some embodiments” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features. Use of the term “approximately” or “near” should be understood to mean including close to a target (e.g., design, value, amount), such as within a margin of any suitable or contemplatable error (e.g., within 0.1% of a target, within 1% of a target, within 5% of a target, within 10% of a target, within 25% of a target, and so on). Furthermore, the phrase A “based on” B is intended to mean that A is at least partially based on B. Moreover, the term “or” is intended to be inclusive (e.g., logical OR) and not exclusive (e.g., logical XOR). In other words, the phrase A “or” B is intended to mean A, B, or both A and B.

Electronic devices often use one or more electronic displays to present visual information such as text, still images, and/or video by displaying one or more images. To display an image, an electronic display may control light emission of its display pixels based at least in part on corresponding image data. The image data may be processed to account for distortions due to different display scenarios and/or input image characteristics before the image data being displayed. Image processing circuitry may warp one or more sets of input image data to account for input distortions (e.g., camera lens distortion), output distortions (e.g., lensing effects associated with the physical shape of the display panel and/or glass cover thereof, lensing effects associated with the light emission mechanism of the display (e.g., liquid crystal displays (LCDs), digital micromirror devices (DMD), organic light-emitting diodes (OLEDs), micro-light-emitting diodes (micro-LEDs)), processing distortions (e.g., a POV change, shifts, scaling, foveation related resolution changes, etc.) and/or to achieve a common image space for blending. For example, lateral chromatic aberration may occur on some curved electronic displays and/or curved glass covers or filters on the electronic displays, and the lateral chromatic aberration may be color dependent.

This disclosure provides image processing systems and methods to correct color dependent distortions by using a chromatic aberration correction (CAC) block. The image processing circuitry may utilize the chromatic aberration correction (CAC) block to perform correction to compensate for lateral chromatic aberrations. In particular, the input signals to the CAC block may be pre-warped in frontend warp pipes based on a first color channel (e.g., green channel) only geometric distortion. The CAC block may apply spatial resampling to other color channels (e.g., red and blue channels) to compensate for chromatic aberration based on distortion differences between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel). Multiple warped color channels of image data may be blended into a single data path. The corrected image data for all color channels may be combined together and output for display.

For example, the image processing circuitry may utilize configuration data associated with the desired warp effects for the first color channel (e.g., green channel) to generate a mapping from the input image space to the warped image space. Based on the configuration data, a mapping may be determined correlating the output pixel values of warped image data to pixel values of the input image data. Distortion differences, or distortion offsets (e.g., vertical offsets, horizontal offsets), between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel) in the output image space may be converted to the input image space using a reverse of the mapping. The CAC may utilize the offsets between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel) in the input image space to apply spatial resampling to other color channels (e.g., red and blue channels) of the input signals. As should be appreciated, the output image space may be associated with the physical pixel locations of the display panel (e.g., the display image space) or any desired image space. Moreover, the image processing circuitry may fetch the input image data (e.g., from memory) and, utilizing the mapping, generate an output pixel value based on the input image data. Furthermore, in some embodiments, the output pixel value may be interpolated from a set of multiple input pixel values selected based on the mapping. The image data for all corrected color channels in the warped image space may be combined together and output for display.

Moreover, certain electronic displays, known as “foveated” displays, display images at higher resolution where a viewer is looking and at lower resolution in the peripheral vision of the viewer. The image data for foveated displays thus may have some pixels that are grouped together to display the same image data. This is referred to as “grouped space,” whereas the electronic display itself has numerous individual pixels that may be considered to have an “ungrouped space.” For foveated displays, both the input and output of the CAC block are in the grouped space, and pixels may be directly resampled from the grouped space to the grouped space before being displayed in the ungrouped space.

Furthermore, the warped image data may include static and/or dynamic aspects to account for warp characteristics that do not change (e.g., display geometry) and things that do (e.g., POV changes, shifts, scaling, foveation related resolution changes). For example, when operating in multiple resolutions, such as for a foveated display that displays multiple different resolutions of an image at different locations on the electronic display depending on a viewer's gaze or focal point on the display, viewer's POV may change and content displayed in different locations as well as sizes, resolutions, or/and positions of the different locations may also change. In other words, which input pixels map to which output pixel positions on the display panel (e.g., as achieved by warping the input image data) may change based on parameters, algorithms, mappings, etc. that are captured in the configuration data. For instance, the dynamic part of the distortions, e.g., the warp characteristics that change (e.g., POV changes, shifts, scaling, foveation related resolution changes), may be decoupled from the rest of the distortions and be processed separately. For example, the image data may be pre-warped in frontend warp pipes based on a geometric distortion correction for a first color channel (e.g., green channel). The CAC block may apply spatial resampling to other color channels (e.g., red and blue channels) in relation to the first color channel (e.g., green color channel) to compensate the dynamic part of the chromatic aberration based on distortion differences between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel) in the ungrouped space (e.g., display panel space). The compensated image data may be converted to the grouped space by pixel grouping. The dynamic chromatic aberration correction may be applied for per-frame updates.

The image processing circuitry may include separate warp hardware (e.g., for parallel processing) and/or perform separate warp operations using the same hardware for different sets of input image data. For example, the image processing circuitry (e.g., a warp block) may utilize configuration data associated with the desired warp effects to generate a mapping from the input image data to the warped image data. The configuration data may include or define mappings, algorithms, and/or parameters indicative of the warp to be accomplished for a set of input image data. The image processing circuitry may fetch the input image data (e.g., from memory) and, utilizing the mapping, generate an output pixel value based on the input image data. Furthermore, in some embodiments, the output pixel value may be interpolated from a set of multiple input pixel values selected based on the mapping.

With the preceding in mind and to help illustrate, an electronic device10including an electronic display12is shown inFIG.1. As is described in more detail below, the electronic device10may be any suitable electronic device, such as a computer, a mobile phone, a portable media device, a tablet, a television, a virtual-reality headset, a wearable device such as a watch, a vehicle dashboard, or the like. Thus, it should be noted thatFIG.1is merely one example of a particular implementation and is intended to illustrate the types of components that may be present in an electronic device10.

The electronic device10includes the electronic display12, image processing circuitry11, one or more input devices14, one or more input/output (I/O) ports16, a processor core complex18having one or more processing circuitry(s) or processing circuitry cores, local memory20, a main memory storage device22, a network interface24, a power source26(e.g., power supply), and eye tracker28. The various components described inFIG.1may include hardware elements (e.g., circuitry), software elements (e.g., a tangible, non-transitory computer-readable medium storing executable instructions), or a combination of both hardware and software elements. It should be noted that the various depicted components may be combined into fewer components or separated into additional components. For example, the local memory20and the main memory storage device22may be included in a single component.

The processor core complex18is operably coupled with local memory20and the main memory storage device22. Thus, the processor core complex18may execute instructions stored in local memory20or the main memory storage device22to perform operations, such as generating or transmitting image data to display on the electronic display12. As such, the processor core complex18may include one or more general purpose microprocessors, one or more application specific integrated circuits (ASICs), one or more field programmable logic arrays (FPGAs), or any combination thereof.

In addition to program instructions, the local memory20or the main memory storage device22may store data to be processed by the processor core complex18. Thus, the local memory20and/or the main memory storage device22may include one or more tangible, non-transitory, computer-readable media. For example, the local memory20may include random access memory (RAM) and the main memory storage device22may include read-only memory (ROM), rewritable non-volatile memory such as flash memory, hard drives, optical discs, or the like.

The network interface24may communicate data with another electronic device or a network. For example, the network interface24(e.g., a radio frequency system) may enable the electronic device10to communicatively couple to a personal area network (PAN), such as a Bluetooth network, a local area network (LAN), such as an 802.11× Wi-Fi network, or a wide area network (WAN), such as a 4G, Long-Term Evolution (LTE), or 5G cellular network. The power source26may provide electrical power to one or more components in the electronic device10, such as the processor core complex18or the electronic display12. Thus, the power source26may include any suitable source of energy, such as a rechargeable lithium polymer (Li-poly) battery or an alternating current (AC) power converter. The I/O ports16may enable the electronic device10to interface with other electronic devices. For example, when a portable storage device is connected, the I/O port16may enable the processor core complex18to communicate data with the portable storage device.

The input devices14may enable user interaction with the electronic device10, for example, by receiving user inputs via a button, a keyboard, a mouse, a trackpad, a touch sensing, or the like. The input device14may include touch-sensing components (e.g., touch control circuitry, touch sensing circuitry) in the electronic display12. The touch sensing components may receive user inputs by detecting occurrence or position of an object touching the surface of the electronic display12.

In addition to enabling user inputs, the electronic display12may be a display panel with one or more display pixels. For example, the electronic display12may include a self-emissive pixel array having an array of one or more of self-emissive pixels or liquid crystal pixels. The electronic display12may include any suitable circuitry (e.g., display driver circuitry) to drive the self-emissive pixels, including for example row driver and/or column drivers (e.g., display drivers). Each of the self-emissive pixels may include any suitable light emitting element, such as an LED (e.g., an OLED or a micro-LED). However, any other suitable type of pixel, including non-self-emissive pixels (e.g., liquid crystal as used in liquid crystal displays (LCDs), digital micromirror devices (DMD) used in DMD displays) may also be used. The electronic display12may control light emission from the display pixels to present visual representations of information, such as a graphical user interface (GUI) of an operating system, an application interface, a still image, or video content, by displaying frames of image data. To display images, the electronic display12may include display pixels implemented on the display panel. The display pixels may represent sub-pixels that each control a luminance value of one color component (e.g., red, green, or blue for an RGB pixel arrangement or red, green, blue, or white for an RGBW arrangement).

The electronic display12may display an image by controlling pulse emission (e.g., light emission) from its display pixels based on pixel or image data associated with corresponding image pixels (e.g., points) in the image. Before being used to display a corresponding image on the electronic display12, the image data may be processed via the image processing circuitry11. The image processing circuitry11may process the image data for display on one or more electronic displays12. For example, the image processing circuitry11may include a display pipeline, memory-to-memory scaler and rotator (MSR) circuitry, warp compensation circuitry, or additional hardware or software means for processing image data. The image data may be processed by the image processing circuitry11to reduce or eliminate image artifacts, compensate for one or more different software or hardware related effects, and/or format the image data for display on one or more electronic displays12. As should be appreciated, the present techniques may be implemented in standalone circuitry, software, and/or firmware, and may be considered a part of, separate from, and/or parallel with a display pipeline or MSR circuitry. The image processing circuitry11may be implemented in the electronic device10, in the electronic display12, or a combination thereof. For example, the image processing circuitry11may be included in the processor core complex18, a timing controller (TCON) in the electronic display12, or any combination thereof.

In some embodiments, pixel or image data may be generated by an image source (e.g., image data, digital code), such as the processor core complex18, a graphics processing unit (GPU), or an image sensor. Additionally, in some embodiments, image data may be received from another electronic device10, for example, via the network interface24and/or an I/O port16. Similarly, the electronic display12may display an image frame of content based on pixel or image data generated by the processor core complex18, or the electronic display12may display frames based on pixel or image data received via the network interface24, an input device, or an I/O port16.

The eye tracker28may measure positions and movement of one or both eyes of someone viewing the electronic display12of the electronic device10. For instance, the eye tracker28may include a camera that can record the movement of a viewer's eyes as the viewer looks at the electronic display12. However, several different practices may be employed to track a viewer's eye movements. For example, different types of infrared/near infrared eye tracking techniques such as bright-pupil tracking and dark-pupil tracking may be used. In both of these types of eye tracking, infrared or near infrared light is reflected off of one or both of the eyes of the viewer to create corneal reflections. A vector between the center of the pupil of the eye and the corneal reflections may be used to determine a point on the electronic display12at which the viewer is looking. The processor core complex18may use the gaze angle(s) of the eyes of the viewer when generating image data for display on the electronic display12.

The electronic device10may be any suitable electronic device. To help illustrate, an example of the electronic device10, a handheld device10A, is shown inFIG.2. The handheld device10A may be a portable phone, a media player, a personal data organizer, a handheld game platform, or the like. For illustrative purposes, the handheld device10A may be a smart phone, such as any IPHONE® model available from Apple Inc.

The handheld device10A includes an enclosure30(e.g., housing). The enclosure30may protect interior components from physical damage or shield them from electromagnetic interference, such as by surrounding the electronic display12. The electronic display12may display a graphical user interface (GUI)32having an array of icons. When an icon34is selected either by an input device14or a touch-sensing component of the electronic display12, an application program may launch. The handheld device10A includes one or more cameras36for capturing images.

The input devices14may be accessed through openings in the enclosure30. The input devices14may enable a user to interact with the handheld device10A. For example, the input devices14may enable the user to activate or deactivate the handheld device10A, navigate a user interface to a home screen, navigate a user interface to a user-configurable application screen, activate a voice-recognition feature, provide volume control, or toggle between vibrate and ring modes.

Another example of a suitable electronic device10, specifically a tablet device10B, is shown inFIG.3. The tablet device10B may be any IPAD® model available from Apple Inc. A further example of a suitable electronic device10, specifically a computer10C, is shown inFIG.4. For illustrative purposes, the computer10C may be any MACBOOK® or IMAC® model available from Apple Inc. Another example of a suitable electronic device10, specifically a wearable electronic device10D, is shown inFIG.5. For illustrative purposes, the wearable electronic device10D may be any APPLE WATCH® model available from Apple Inc. As depicted, the tablet device10B, the computer10C, and the watch10D each also includes an electronic display12, input devices14, I/O ports16, and an enclosure30. The electronic display12may display a GUI32. Here, the GUI32shows a visualization of a clock. When the visualization is selected either by the input device14or a touch-sensing component of the electronic display12, an application program may launch, such as to transition the GUI32to presenting the icons34discussed inFIGS.2and3.

Turning toFIG.6, a computer10E may represent another embodiment of the electronic device10ofFIG.1. The computer10E may be any computer, such as a desktop computer, a server, or a notebook computer, but may also be a standalone media player or video gaming machine. By way of example, the computer10E may be an iMac®, a MacBook®, or other similar device by Apple Inc. of Cupertino, California. It should be noted that the computer10E may also represent a personal computer (PC) by another manufacturer. A similar enclosure30may be provided to protect and enclose internal components of the computer10E, such as the electronic display12. In certain embodiments, a user of the computer10E may interact with the computer10E using various peripheral input structures14, such as the keyboard14A or mouse14B (e.g., input structures14), which may connect to the computer10E.

To help illustrate, a portion of the electronic device10, including image processing circuitry11, is shown inFIG.7. As should be appreciated, although image processing is discussed herein as being performed via a number of image data processing blocks, embodiments may include hardware or software components to carry out the techniques discussed herein. The electronic device10may include an image data source38, a display panel40, and/or a controller42in communication with the image processing circuitry11. In some embodiments, the display panel40of the electronic display12may be a reflective technology display, a liquid crystal display (LCD), or any other suitable type of display panel40. In some embodiments, the controller42may control operation of the image processing circuitry11, the image data source38, and/or the display panel40. To facilitate controlling operation, the controller42may include a controller processor44and/or controller memory46. In some embodiments, the controller processor44may be included in the processor core complex18, the image processing circuitry11, a timing controller in the electronic display12, a separate processing module, or any combination thereof and execute instructions stored in the controller memory46. Additionally, in some embodiments, the controller memory46may be included in the local memory20, the main memory storage device22, a separate tangible, non-transitory, computer-readable medium, or any combination thereof.

The image processing circuitry11may receive source image data48corresponding to a desired image to be displayed on the electronic display12from the image data source38. The source image data48may indicate target characteristics (e.g., pixel data) corresponding to the desired image using any suitable source format, such as an RGB format, an αRGB format, a YCbCr format, and/or the like. Moreover, the source image data may be fixed or floating point and be of any suitable bit-depth. Furthermore, the source image data48may reside in a linear color space, a gamma-corrected color space, or any other suitable color space. As used herein, pixels or pixel data may refer to a grouping of sub-pixels (e.g., individual color component pixels such as red, green, and blue) or the sub-pixels themselves.

As described above, the image processing circuitry11may operate to process source image data48received from the image data source38. The image data source38may include captured images (e.g., from one or more cameras36), images stored in memory, graphics generated by the processor core complex18, or a combination thereof. Additionally, the image processing circuitry11may include one or more sets of image data processing blocks50(e.g., circuitry, modules, or processing stages) such as a warp block52and a chromatic aberration correction (CAC) block53. As should be appreciated, multiple other processing blocks54may also be incorporated into the image processing circuitry11, such as a white point compensation (WPC) block, a color lookup table (CLUT) block, an optical crosstalk compensation (OXTC) block, a burn-in compensation (BIC), a pixel contrast control (PCC) block, a sub-pixel uniformity compensation (SPUC) block, a color management block, a dither block, a blend block, a scaling/rotation block, etc. before and/or after the warp block52, or before and/or after the CAC block53. A pipeline may be used for preparing the source image data48to be displayed on the display panel40, and the pipeline may use one or more processing blocks in the image data processing blocks50. For example, the pipeline may include a frontend, which may include several processing blocks, such as the warp block52, the blend block, and the color management block, etc. The pipeline may also include a backend, which may include several processing blocks, such as the WPC block, the CLUT block, the CAC block53, the OXTC block, the BIC block, the PCC block, the SPUC block, and the dither block, etc. The image data processing blocks50may receive and process source image data48and output display image data56in a format (e.g., digital format, image space, and/or resolution) interpretable by the display panel40. Further, the functions (e.g., operations) performed by the image processing circuitry11may be divided between various image data processing blocks50, and, while the term “block” is used herein, there may or may not be a logical or physical separation between the image data processing blocks50.

FIG.8is a schematic diagram of the warp block52ofFIG.7. In some scenarios, an image to be displayed may, if unaltered, appear distorted when perceived by a viewer due to environmental effects, properties of the electronic display12, the viewer's perspective (e.g., POV), image processing alterations such as shifts and scaling, and/or other distorting factors. As such, in some embodiments, the warp block52may remap input image data60such that the generated warped image data62accounts for such distortions when displayed (e.g., on the display panel40), as illustrated inFIGS.9&10. For color-dependent distortions, the input image data60may include image data from a first color channel (e.g., green channel) of the source image data48, and the warp block52may generate the warped image data62for the first color channel. Accordingly, the warped image data62for the first color channel may be generated in the frontend of the pipeline, and the image data from the other channels (e.g., red channel, blue channel) may be compensated in the CAC block53, which may be at the backend of the pipeline. The image data from the other channels (e.g., red channel, blue channel) may be compensated for the chromatic aberration based on the offsets (e.g., vertical offsets, horizontal offsets) between the corresponding channel and the first color channel, as illustrated inFIGS.11-12.

As should be appreciated, the input image data60may include any suitable image data desired to be transformed (e.g., warped). For example, the input image data60may include graphics image data64(e.g., a stored or generated digital image), captured image data66(e.g., a video image taken by a camera36), and/or other image data68such as matting image data generated to represent alpha values for an image blending process, image data received via the network interface24or the I/O ports16), etc. As such, the warp block52may generate the warped image data62(e.g., warped graphics image data70, warped captured image data72, warped other image data74, etc.) to change the amount (e.g., resolution) or distribution such as (e.g., shape, relative size, perspective, etc.) of pixel values of the input image data60to account for different display scenarios and/or input image characteristics.

For example, the warped image data62may account for curved edges and/or lensing effects (e.g., of a cover glass) associated with the display panel40and/or for a viewer's POV relative to the display panel40or relative to an image capturing device (e.g., the camera36). Furthermore, the electronic display12may be a foveated display such that different portions of the display panel40are displayed at different resolutions (e.g., depending on a viewer's gaze), and the warp block52may consider the resolution at the different portions of the display panel40when determining the mapping between the input image data60and the warped image data62. Additionally, the warp block52may also take into account distortions associated with the input image data60and/or the image data source38. For example, captured image data66may be warped to account for lensing effects (e.g., camera lens distortion) associated with capturing the image and/or to account for a difference between the POV of a user and the POV of the camera36. As should be appreciated, captured image data66is given as an example set of input image data60that may be warped for distortions associated with the image data source38and any set of input image data60may be warped for distortions associated with the respective image data source38and/or to obtain a common image space. Moreover, multiple warp operations (e.g., accounting for multiple distortion effects) may be accomplished via a single warp (e.g., a single mapping accounting for multiple distortions) or consecutively warped. As such, before being displayed, input image data60may be warped to change to the amount or distribution of pixel values such that the perceived image has limited or no distortion.

Furthermore, in some embodiments, the warp block52may warp multiple different sets of input image data60(e.g., graphics image data64, captured image data66, other image data68, etc.) simultaneously (e.g., in parallel) or sequentially for use separately or together. For example, an image may be generated by blending multiple sets of input image data60from one or more image data sources38. However, in some scenarios, image data to be blended may be warped to a common image space prior to blending, which may be accomplished by the warp block52. Image blending may be utilized (e.g., for virtual reality, mixed reality, and/or augmented reality) to incorporate multiple sets of warped image data62into a single image frame. For example, a generated object (e.g., warped graphics image data70) may be incorporated into a captured image of a real-life surrounding (e.g., warped captured image data72) and/or a portion of the captured image may be utilized as a separate blended layer for a foreground (e.g., based on warped matting image data) such that the generated object is between the portion in the foreground and a background portion of a captured image. Additionally or alternatively, a portion of a captured image (e.g., warped captured image data72) may be incorporated into a virtual surrounding (e.g., warped graphics image data70). As such, the input image data60of one or more image data sources38may be blended together to form a single output image after being warped to a common image space via the warp block52.

As discussed above, the warp block52of the image processing circuitry11may warp one or more sets of input image data60to account for input distortions (e.g., camera lens distortion), output distortions (e.g., lensing effects associated with the shape of the display panel and/or glass cover thereof), processing distortions (e.g., a POV change, shifts, scaling, etc.) and/or to achieve a common image space for blending. Moreover, the image processing circuitry may include separate warp hardware (e.g., for parallel processing) and/or perform separate warp operations using the same hardware for different sets of input image data. For example, in some embodiments, the warp block52may include a graphics warp sub-block76, a captured warp sub-block78, and/or another warp sub-block80. As should be appreciated, the sub-blocks described herein are given as examples, and any suitable warping sub-block may utilize the features discussed herein to warp any suitable set of input image data60and generate warped image data62.

In some embodiments, the warp block52may utilize configuration data82associated with the desired warp effects to generate a mapping from the input image data60to the warped image data62. The configuration data82may include mappings, algorithms, and/or parameters indicative of the warp to be accomplished for a set of input image data60. Furthermore, the configuration data82may include static and/or dynamic aspects and may include different parameters/mappings for different sets of input image data60. For example, the configuration data82may include a static mapping between a generated graphics image space (e.g., graphics image data64) to a display image space (e.g., warped graphics image data70) accounting for distortions associated with the electronic display12that do not change. Moreover, the configuration data82may include a static mapping between a camera image space (e.g., captured image data66) to a display image space (e.g., warped captured image data72) accounting for camera lens distortions that do not change and distortions associated with the electronic display12that do not change. As should be appreciated, captured image data66from a camera36is given as an example set of input image data60, and such data may or may not be processed or partially processed prior to the warp block52of the image processing circuitry11.

Dynamic aspects of the configuration data may provide for different mappings according to the scenario at the time of warping (e.g., for the image frame being processed). For example, in a foveated display, the output resolution at different portions of the display panel may change depending on a focal point of the user's gaze (e.g., determined by eye-tracking), which may alter the mapping. In other words, which input pixels of the input image data60map to which output pixel positions for the display panel40(e.g., as characterized by warping the warped image data62) may change based on parameters, algorithms, mappings, etc. that are captured in the configuration data82. As should be appreciated, the configuration data82may include any suitable information (e.g., parameters, tags, flags, algorithms, mappings, etc.) that characterize the warp to be achieved for a particular set of input image data60. For instance, the dynamic part of the distortions, i.e., the warp characteristics that change (e.g., POV changes, shifts, scaling, foveation related resolution changes, etc.), may be decoupled from the rest of the distortions and be processed separately. For example, the image data may be pre-warped in the warp block52based on the first color channel (e.g., green channel) only geometric distortion. The CAC block53may apply spatial resampling to other color channels (e.g., red and blue channels) to compensate the dynamic part of the chromatic aberration based on distortion differences between the other color channels (e.g., red and blue channels) and the first color channel (e.g., green channel) in the ungrouped space (e.g., display panel space). The compensated image data may be converted to the grouped space by pixel grouping. The dynamic chromatic aberration correction may be applied for per-frame update. For foveated display, both the input and output of the CAC block53are in the grouped space, and pixels are directly resampled from the grouped space to the grouped space.

Based on the configuration data82, mapping data84may be generated (e.g., via a mapping and interpolation sub-block86) correlating the output pixel values of the warped image data62to pixel values of the input image data60. As should be appreciated, the output image space may be associated with the physical pixel locations of the display panel40(e.g., the display image space) or any desired image space. Moreover, the warp block52(e.g., the graphics warp sub-block76, the captured warp sub-block78, the other warp sub-block80, etc.) may perform fetches88of the input image data60from the relevant image data source38(e.g., memory20, a graphics generator of the processor core complex18, other processing blocks54, a network interface24, a camera36, etc.). Utilizing the mapping data84, the warp block52may generate warped image data62based on the input image data60.

FIG.9shows a warped input image100(e.g., A′B′C′D′) in an input image space102. The warped input image100may be warped (e.g., by the warp block52) to account for one or more physical or digital effects associated with displaying the image data. The warped input image100, when displayed (e.g., on the display panel40or in a viewer's eye), corresponds to an output image104(e.g., ABCD) in an output image space106(e.g., on the display panel40or in a viewer's eye). For example, the output image104may correspond to the input image without being distorted due to the one or more physical or digital effects. Accordingly, to display the image104in the output image space106, the corresponding input image (not shown) may be warped to the warped input image100by a mapping108. For instance, without the mapping108, an image in the input image space102may be distorted when displayed in the output image space106(e.g., on the display panel40or in a viewer's eyes). The warped image100generated by the mapping108may account for the distortions. The mapping108may include a vertical mapping stage for mapping Y coordinate and a horizontal mapping stage for mapping X coordinate. In a separable warp architecture, the vertical mapping stage may be separated from the horizontal mapping stage. For example, the input pixels in the input image space102may be mapped to intermediate locations in the vertical mapping stage of the mapping108. And in the horizontal mapping stage, the intermediate locations may be mapped to locations corresponding to output pixels in the output image space106. For example, the input image may include three input pixels,110,112, and114, which, when displayed in the output image space106, correspond to three output pixels116,118, and120, respectively. The three input pixels110,112, and114may be warped to the warped input image100by the mapping108. For example, the three input pixels110,112, and114may be mapped to three intermediate locations122,124, and126on a line E′F′ in the warped input image100in the vertical mapping stage of the mapping108. The intermediate locations122,124, and126are obtained by mapping the input pixels110,112, and114to the line E′F′ along corresponding vertical lines128,130, and132, respectively, in the vertical mapping stage of the mapping108. The line E′F′ may correspond to a horizontal line EF in the output image104, as illustrated inFIG.9. Accordingly, the line E′F′, when displayed in the output image space106, corresponds to the line EF. The intermediate locations122,124, and126may be mapped along the line E′F′ to locations corresponding to the output pixels116,118, and120, respectively, in the horizontal mapping stage of the mapping108, as illustrated inFIG.10.

FIG.10illustrates the horizontal mapping stage of the mapping108. As described above, the intermediate locations122,124, and126are obtained by mapping the input pixels110,112, and114to the line E′F′ along corresponding vertical lines128,130, and132, respectively, in the vertical mapping stage of the mapping108. In the horizontal mapping stage of the mapping108, the intermediate locations122,124, and126may be mapped to locations134,136, and138, respectively, along the line E′F′. The locations134,136, and138, when displayed in the output image space106, correspond to the output pixels116,118, and120, respectively. Accordingly, the mapping108maps the input pixels110,112, and114to the locations134,136, and138on the line E′F′, which correspond to the output pixels116,118, and120on the line EF in the output image104, respectively. The mapping108with separable warp architecture, as described inFIGS.9&10, may be used in the warp block52and/or the CAC block53. In addition, the reverse mapping of the mapping108may be used to convert pixels from the warped input image100to the input image. Accordingly, the reverse mapping of the mapping108may be used to map relative locations of pixels in the output image space106(e.g., on the display panel40or in a viewer's eye) to the input image. For example, a distance between the output pixels116and118on the line EF in the output image space106may correspond to a distance between the locations134and136on the line E′F′ in the input image space102. Since the locations134and136are obtained from the input pixels110and112, respectively, via the mapping108, a reverse mapping of the mapping108may convert the locations134and136to the input pixels110and112, respectively.

Due to chromatic aberrations, distortions may be color dependent. Thus, different color channels of the same input image may correspond to different output images in the output image space106, which means the output images of different color channels of the input image may have distortion offsets in the output image space106. Thus, the mapping108may be different for different color channels. For lateral chromatic aberrations, the output images for different color channels may be corrected by using the distortion offsets. As mentioned previously, for color dependent distortions, the input image data60may include image data from a first color channel (e.g., green channel) of the source image data48, and the warp block52may generate the warped image data62for the first color channel, for example, by using the method describe above inFIGS.9&10. In some embodiments, the warped image data62may be in a grouped space (e.g., foveated display). The warped image data62for the first color channel may be input into the CAC block53and used to compensate image data from the other channels (e.g., red channel, blue channel). The image data from the other channels (e.g., red, blue) may be compensated for the chromatic aberrations based on the relative distortion offsets (e.g., vertical offsets, horizontal offsets) between the corresponding channel and the first color channel, as illustrated inFIG.11.

FIG.11is a schematic diagram of the CAC block53ofFIG.7. The input image data200may include image data from multiple color channels (e.g., red (R), green (G), blue (B)), and the image data from the first color channel (e.g., green (G)) are pre-warped (e.g., in the warp block52) to compensate geometric distortions caused by one or more physical or digital effects associated with displaying the image data. The input image data200may be in a grouped space (e.g., the warped image space of the warp block52) for foveated display. A programmable selector202may be used to select whether a bit-shift function may be applied to the input image data200based on a bit-shift enable signal204. The bit-shift function may be used to accommodate the bit-depth requirement (e.g., for the CLUT block) in the downstream of the CAC block53. A pixel line buffers206may be used to store the input image data200, and the image data of the first color channel (e.g., green (G)), which are pre-warped, may be delayed in delay buffers to match outputs from other color channels (e.g., red (R), blue (B)). Thus, the CAC block53may only process chromatic aberration corrections for the other color channels (e.g., red, blue) and leave the first color channel (e.g., green) unchanged. The delayed image data of the first color channel (e.g., green (G))207may be output from the CAC block53when image data of other color channels (e.g., red (R), blue (B)) are processed and output from the CAC block53. The corrected image data for all color channels may be combined together for display.

The input image data of the other color channels (e.g., red (R), blue (B))208may be input into a vertical pixel interpolation block210. Corresponding grouped pixel positions212of the input image data208in the grouped space may be input into a grouped space-to-panel space (G2P) block214to convert from the grouped space to the output image space106(e.g., display panel space). Accordingly, the output of the G2P block214includes corresponding output pixel positions215of the input image data208in the output image space106, which may be input into offset grid buffers216. Each grid point in the offset grid buffers216may have four components corresponding to the distortion offsets of the other color channels (e.g., red (R), blue (B)) from the first color channel (e.g., green (G)) along the oX and oY directions in the output image space106due to the chromatic aberrations. For example, the four components may include R_dx and R_dy corresponding to offsets along the oX and oY directions for red (R) channel, and B_dx and B_dy corresponding to offsets along the oX and oY directions for blue (B) channel. The output pixel positions215of the other color channels (e.g., red, blue) may be adjusted by using the corresponding offsets at each grid point. The output of the offset grid buffers216may include adjusted pixel positions217for the other color channels (e.g., red and blue) in the output image space106, which are compensated for the chromatic aberrations. The adjusted pixel positions217may be input into a vertical offset interpolator218to perform vertical grid interpolation in the output image space106to map to the display pixels in the output image space106. For example, a linear interpolation may be used in the vertical offset interpolator218. The output from the vertical offset interpolator218may include output pixels219with vertically corrected pixel positions in the output image space106. The output pixels219may be input into a panel space-to-grouped space (P2G) block220to convert to the grouped space. The output of the P2G block220may include vertically corrected pixel positions221in the grouped space, and may be input into the vertical pixel interpolation block210to obtain the corresponding vertical pixel values211for the vertically corrected pixel positions221, as described inFIG.12. The corresponding vertical pixel values211may be input into a horizontal pixel interpolation block226and used for horizontal pixel interpolation.

In addition, the adjusted pixel positions217may be input into a horizontal offset interpolator222to perform horizontal grid interpolation in the output image space106to map to the display pixels in the output image space106. For example, a linear interpolation may be used in the horizontal offset interpolator222. The output from the horizontal offset interpolator222may include output pixels223with horizontally corrected pixel positions in the output image space106. The output pixels223may be input into a P2G block224to convert to the grouped space. The output of the P2G block224may include horizontally corrected pixel positions225in the grouped space. The horizontally corrected pixel positions225may be input into the horizontal pixel interpolation block226and used with the corresponding vertical pixel values211to obtain the chromatic aberrations corrected input image data228for the other channel (e.g., red, blue). The method described above may be performed on all other color channels (e.g., red, blue), and the corrected input image data228for all other color channels and the delayed image data of the first channel207may be output from the CAC block53and combined together for display.

FIG.12shows a process of vertical pixel interpolation for a color channel (e.g., red, blue) used in the block210ofFIG.11. An input fetching window250(e.g., 4×7 pixels) may be used when fetching a group of pixels (e.g., 4×7 pixels) from the pixel line buffers206to the vertical pixel interpolation block210. As mentioned above, the pixel line buffers206store the input image data200in the grouped space with the image data of the first color channel (e.g., green (G)) pre-warped to correct geometry distortions. Accordingly, the input fetching window250may include a set of input image pixels in the grouped space for one color channel (e.g., red, blue). The corrected input image data228in the grouped space may include pixels in the one color channel (e.g., red, blue) corresponding to locations252,254,256, and258along a line G′H′ in the grouped space. The vertically corrected pixel positions221from the P2G block220may include corresponding vertically corrected locations260,262,264, and266for the locations252,254,256, and258in the grouped space. Corresponding vertically corrected positions260,262,264, and266for the locations252,254,256, and258in the grouped space may be at corresponding intersections of the line G′H′ with the vertical pixel lines of the input image pixels in the input fetching window250. For example, the corrected position260for the location252is at the intersection of the line G′H′ with a vertical pixel line270in the input fetching window250; the corrected position262for the location254is at the intersection of the line G′H′ with a vertical pixel line272in the input fetching window250; the corrected position264for the location256is at the intersection of the line G′H′ with a vertical pixel line274in the input fetching window250; the corrected position266for the location258is at the intersection of the line G′H′ with a vertical pixel line276in the input fetching window250. Corresponding pixel values for the corrected vertical positions260,262,264, and266in the grouped space may be obtained by using vertical pixel interpolations in the block210. For example, the corrected position260may be inside of a pixel group278, and its pixel value may be determined by the pixels in the pixel group278; the corrected position262may be inside of a pixel group280, and its pixel value may be determined by the pixels in the pixel group280; the corrected position264may be inside of a pixel group282, and its pixel value may be determined by the pixels in the pixel group282; and the corrected position266may be inside of a pixel group284, and its pixel value may be determined by the pixels in the pixel group284. The number of pixels inside each pixel group may vary (e.g., 2, 3, or 4 pixels) and may be different or the same for different pixel groups, which may be associated with corresponding locations of the pixel groups. The corresponding pixel values and the vertically corrected positions260,262,264, and266may be input into the horizontal pixel interpolation226. Similar as the horizontal mapping stage of the mapping108illustrated inFIG.10, the horizontal pixel interpolation block226may use the horizontally corrected pixel positions225to obtain the locations252,254,256, and258.

FIG.13shows a method300for applying the chromatic aberrations correction to a color channel of the input image data. At block310, a configuration data is used to characterize the warp to be achieved between an input image space and an output image space for input image data in a first color channel (e.g., green). At block320, the first color channel (e.g., green) of input image data may be warped (e.g., in the warp block52) into a grouped space based on the configuration data. At block330, the warped first color channel (e.g., green) of the input image data and a second color channel (e.g., red, blue) of the input image data in the grouped space may be input into a CAC block (e.g., the CAC block53). The chromatic aberration position offsets between the first color channel (e.g., green) and the second color channel in the output image space may be obtained at block340. The position offsets may be stored in the CAC block. For example, the position offsets may be stored in the offset grid buffers216. At block350, the offset interpolator may perform grid interpolation (e.g., vertically, horizontally) to obtain corrected pixel positions in the output image space. The pixel positions in the output image space may be converted into the grouped space. At block360, the CAC block may perform vertical pixel interpolation on the second color channel of the input image data in the grouped space and generate vertical output data. At block370, the CAC block may perform horizontal pixel interpolation on the vertical output data of the second color channel. At block380, the CAC block may output the corrected image data for the second color channel. To protect the privacy of the image content, on-the-fly calculations may be used. For example, in the offset interpolation at block350, slope may be computed on-the-fly (e.g., with Newton-Raphson method), and on-the-fly conversion may be used from ungrouped space to grouped space, and vice versa. The CAC function may be disabled, for example, for power saving.

FIG.14shows an input image400in a grouped space (e.g., foveated display) and a chromatic aberrations corrected image450after the CAC block53for the input image400. The input image400, when displayed (e.g., on a display panel or in a viewer's eye) without correction, may have distortions due to different display scenarios and/or input image characteristics, and the distortions may be color dependent. The corrected image450may be processed to account for the distortions as well as the chromatic aberrations. Accordingly, the corrected image450, when displayed (e.g., on a display panel or in a viewer's eye), may have no or reduced distortions.