Patent Description:
Wide angle optics (lenses) may be used to focus on the display and enable a wide field of view. Cheaper, lighter and lower quality lenses may introduce spatial and chromatic distortions of the image such as radial distortions and chromatic aberrations. The radial distortions created by these lenses typically include pin-cushion distortions. As a result, the images to be displayed may be pre-warped to cancel the distortions. Pre-warping may typically be performed in a post-processing step using a pixel shader. However, the discrete resampling utilized by pixel shaders may lead to a loss in image quality. Further, some graphics architectures may not support the use of pixel shaders.

<NPL>, describes a pixel shader for pre-warping of images.

<CIT> provides a display device including a display unit that displays an image thereon, an eyepiece optical unit that projects a display image of the display unit on the eyes of a user, a correction information retaining unit that retains correction information created in advance according to a state of the user, and a distortion correction unit that corrects distortion of the display image based on correction information according to a current state of the user.

Advantageous embodiments are subject to the dependent claims.

The various novel aspects of the embodiments of the present disclosure will become evident to a person of ordinary skill in the art given the following enabling specification and appended claims, and by referencing the following drawings, in which:.

1A shows a block diagram of an example of a system <NUM> in accordance with an embodiment of the present disclosure. The system <NUM> a head mounted display (HMD) <NUM> having one or more optical lens(es) <NUM> for viewing and/or capturing images. The HMD <NUM> may be worn by a user to provide an immersive viewing experience that may be particularly suited for three-dimensional (3D), gaming (e.g., video, alternative reality, and augmented reality), and other similar applications. While a standard HMD <NUM> is shown, it should be noted that this illustration is for discussion purposes only. The lens <NUM> may be a wide angle lens that is particularly suited for 3D, gaming, and similar applications. The lens <NUM>, however, depending on the optical quality may introduce distortions including radial and chromatic distortions within the images viewed via the lens <NUM>.

In at least some embodiments, the system <NUM> may also optionally include a HD audio-video (AV) source <NUM> (e.g., a Blu-ray disc, digital versatile disc/DVD, or streaming video device), and a High Definition (HD) display <NUM> (e.g., an HDMI compatible television/TV, HDTV or Smart TV). The AV source <NUM> may enable images to be reproduced and viewed via the HMD <NUM>. In some embodiments, the AV source <NUM> enables images to be displayed on the HD display <NUM> via, for example, an HDMI input such that the images can be viewed via the HMD <NUM>. In some embodiments, the AV source <NUM> and/or HD display <NUM> may be in direct communication with HMD <NUM>.

The illustrated system <NUM> includes a distortion compensation system <NUM> in communication with lens <NUM>. The distortion compensation system <NUM> receives an input image <NUM> from an input source such as, for example, the AV source <NUM> and/or lens <NUM>. The illustrated distortion compensation system <NUM> includes a system processor <NUM> and logic module <NUM> to perform processes to reduce distortions within the received input images. In some embodiments, the distortion compensation system <NUM> may be completely or partially incorporated within the HMD <NUM>. In at least some embodiments, the logic module <NUM> includes an image mapping module <NUM>, radial aberration compensation (RAC) module <NUM>, chromatic aberration compensation (CAC) module <NUM>, blending module <NUM>, and memory <NUM> having a frame buffer <NUM>.

In at least one embodiment, the distortion compensation system <NUM> receives input images captured from an image source such as, for example, the AV source <NUM>, and causes the logic module <NUM> to perform processes to map a received image onto one or more of a plurality of distortion meshes, to compensate for (e.g., correct), radial distortions and chromatic aberrations within the received images, to blend the corrected images, and output a blended/corrected image to the HMD <NUM> such that a corrected image having reduced distortions can be viewed by a user. The logic module <NUM> may include image mapping technology, radial and chromatic distortion technology, and image blending technology, which may be implemented via, for example, a set of logic instructions, configurable logic or fixed functionality hardware logic, suitable to perform the radial and chromatic distortion compensation discussed herein.

In an aspect of the invention, the image mapping module <NUM> receives one or more input images and maps the received images onto a plurality of distortion meshes. The RAC module <NUM> of system <NUM> corrects or compensates for radial aberrations within the received images. The RAC module <NUM> may utilize, for example, bi-cubic texture interpolation, to generate a better approximation towards the original image signal in order to maintain sharpness of the received image. The bi-cubic interpolation may include a barrel-shaped distortion designed to cancel or correct the radial distortions and output the radially corrected images. The illustrated CAC module <NUM> corrects or compensates for chromatic aberrations within the received images. For example, the CAC module <NUM> may independently correct for chromatic aberrations for each of the different color channels (e.g., red color channel, green color channel, and blue color channel). The CAC module <NUM> may also minimize chromatic aberration by considering three color channels (i.e., red, green, blue) instead of the entire visible light spectrum. The CAC module <NUM> may apply lens specific parameters depending on the color channel to control the degree of the chromatic aberration. The CAC module <NUM>, in performing this correction, separately renders individual corrected images (e.g., chromatically corrected images), for each color channel. The blending module <NUM> then blends the resulting individual corrected images to yield a blended corrected (i.e., composite) image. The blended corrected image is output to the HMD <NUM> such that the blended corrected image contains reduced radial distortions and chromatic aberrations when viewed via lens <NUM>. The outputs of the RAC module <NUM>, CAC module <NUM>, and blending module <NUM> may be stored in memory <NUM>. The memory <NUM> may include video compatible storage such as the frame buffer <NUM> to store the individually rendered corrected images and blended corrected images.

Chromatic aberration, sometimes referred to as "fringing", "color fringing", "purple fringing", etc., may be a problem common to optical lenses, particularly low-quality and wide angled lenses. Chromatic aberration typically occurs when a lens is unable to focus all wavelengths of color at the same focal plane and/or when wavelengths of color are focused at different points in the focal plane. Chromatic aberration tends to be caused by dispersion within the lens, where the various colors of light travel at different speeds while passing through the lens. This effect causes the image to look blurred or include colored edges (e.g., red, green, blue, yellow, purple, magenta) around objects, particularly in high-contrast situations. A "perfect" lens may focus all wavelengths of light into a single focal point, having the best focus with the "circle of least confusion".

<FIG> illustrate examples of lenses and the associated chromatic aberration according to an embodiment of the present disclosure. <FIG> illustrates a "perfect" lens <NUM> having no chromatic aberration, wherein the lens <NUM> has an optical axis <NUM> and a best focus plane <NUM>. The illustrated lens <NUM> is of high quality and produces no chromatic aberration, distortion or dispersion of light passing through the lens. Therefore, the lens <NUM> allows the different rays of light R, G, B passing through the lens to be focused on the same focus point <NUM>. <FIG> illustrates, on the other hand, a typical, lower-quality lens <NUM> that produces a lateral chromatic aberration. Lateral chromatic aberration, sometimes referred to as "transverse" chromatic aberration, occurs when different wavelengths (i.e., colors) of light R, G, B pass at an angle through the lens <NUM> with respect to an optical axis <NUM> and focus at different points along the same focal plane <NUM>. Lateral chromatic aberration does not appear in the center of the image and tends to only be visible towards the edges or corners of the image in high contrast areas. Blue and purple fringing commonly appears in some fisheye, wide-angle and low-quality lenses.

<FIG> illustrate examples of distortion meshes at various phases of an embodiment of the present disclosure. <FIG> illustrates a pincushion effect or pincushion distortion <NUM> created by spatial (i.e., radial) distortions within a lens. Spatial distortions in optical systems may result from both the shape and material quality of the lens. Pincushion distortions may be canceled or corrected by pre-warping the image presented on the display panel with a corresponding barrel-shaped distortion. <FIG> illustrates a barrel-shaped distortion <NUM> that may be applied to cancel or correct the pincushion distortion <NUM> (<FIG> illustrates a barrel-shaped distortion <NUM> including a triangulation grid <NUM> applied to an image <NUM>. Triangulation grid <NUM> includes a grid or network of triangles that allow for more accurate correction of distortions within an image such as image <NUM>. The barrel-shaped distortion <NUM> and triangulation grid <NUM> may approximate certain behavior and can be applied to an image by resampling the image or by mapping the image onto a distortion mesh in which the vertices have been displaced.

<FIG> is a flowchart of an example of a method of distortion compensation according to an embodiment. The method <NUM> may be implemented in executable software as a set of logic instructions stored in a machine- or computer-readable storage medium of a memory such as random access memory (RAM), read-only memory (ROM), programmable ROM (PROM), firmware, flash memory, etc., in configurable logic such as, for example programmable logic arrays (PLAs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), in fixed functionality logic hardware using circuit technology such as, for example, application-specific integrated circuits (ASIC), complementary metal oxide semiconductor (CMOS) or transistor-transistor logic (TTL) technology, or any combination thereof. For example, computer program code to carry out operations shown in method <NUM> may be written in any combination of one or more programming languages including an object-oriented programming language such as Java, Smalltack, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

Illustrated processing block <NUM> provides for mapping a received image onto one or more of a plurality of distortion meshes. The distortion meshes are generated to compensate for radial and chromatic distortions within the received image. Illustrated processing block <NUM> provides for compensating for radial distortions within the received image and outputting a radially corrected image. Illustrated block <NUM> provides for compensating for chromatic distortions within the received image and outputting a chromatically corrected image. The distortion meshes may also apply lens specific parameters in order to control the degree of the distortion. The distortion meshes may be individually generated for each color channel (i.e., red, green, blue) and the image may be independently corrected for each color channel. Illustrated processing block <NUM> provides for adaptively blending the plurality of independent distortion meshes into a composite corrected image to reduce distortion errors within the received image. Adaptive blending may refer to the process of adding the individual distortion meshes together to produce a composite corrected image. The composite corrected image may be created by adaptively blending, for example, by layering and combining, the corrected images of the individual distortion meshes to yield a single image. Illustrated processing block <NUM> provides for optionally outputting the composite corrected image to a display having a lens to view the corrected image such that the corrected image (at post processing) has reduced distortions (compared to pre-processing) when viewed via the lens.

Spatial distortions in optical systems may result from the shape, quality and material of the lens. This distortion (and its correction) may be described with respect to a Taylor series, as shown below with respect to Equation <NUM> below. A relatively small number of terms are sufficient to capture a barrel-shaped distortion such as shown in <FIG> and 2C. In at least one embodiment, barrel-shaped distortions (including displacement from the optical axis) appropriate and suitable for correcting or compensating for radial distortions, such as pincushion distortions, in accordance with the present disclosure may be calculated by Equation <NUM>, as follows: <MAT> where,.

The lens specific parameters relate to the shape, optical quality and materials of the lens and are typically available from the lens manufacturers.

Color fringing artifacts occur when light of different wavelengths refracts differently through a lens. This chromatic aberration can be corrected by separately resampling or mapping the red, green, and blue color channels of an image. In at least one embodiment, resampling and/or mapping appropriate and suitable for correcting or compensating for chromatic aberrations in accordance with the present disclosure may be calculated by Equation <NUM>, as follows: <MAT> where,.

The lens specific parameters relate to the shape, optical quality and materials of the lens and are typically available from the lens manufacturers. rGnew provides a base or default measurement because, as shown in <FIG>, the green light rays are between the red and blue regarding color fringing through chromatic aberrations. The red color, rRnew, and blue color, rBnew, may be scaled with the lens specific parameters and the squared radius based on the green color, rGnew, to provide the new distance from the lens distortion center, rRGBnew.

In at least one embodiment, spatial and chromatic distortion correction may be performed in image space or object space based on Equations <NUM> and <NUM> in accordance with the present disclosure. The individually rendered images may be adaptively blended in accordance with the present disclosure by Equation <NUM>, as follows: <MAT> where,.

When performing the blending process, the blending module <NUM> of system <NUM> (<FIG>) may allow for the individually rendered corrected images to be left in the frame buffer <NUM> of memory <NUM> (<FIG>). Thus, the new individually rendered corrected images may be added on top of the corrected images stored in the frame buffer <NUM> (<FIG>). Assuming a 3x <NUM>-bit layout of the frame buffer indicates that the color values may be between <NUM> and <NUM> for each channel.

An example of a representation of a single pixel during rendering for adaptive blending in accordance with the present disclosure is provided, as follows:.

The adaptive blending process, discussed above, allows the system to map a received image onto a plurality of distortion meshes, and use the one or more of the plurality of distortion meshes to compensate for radial and chromatic distortions within the received image. The distortion meshes are then adaptively blended to add the individually rendered distortion meshes into a composite corrected image having reduced distortion errors (both radial and chromatic). The composite corrected image may be output to a display having a wide-angle or fisheye lens to view the composite corrected image such that the composite corrected image is substantially free of radial distortions and chromatic aberrations when viewed via the lens.

<FIG> shows a system <NUM>. The system <NUM> may be part of a platform having computing functionality (e.g., video game console, desktop computer, laptop, tablet computer, convertible tablet, personal digital assistant/PDA), communications functionality (e.g., wireless smart phone), imaging functionality, media playing functionality (e.g., smart television/TV), wearable functionality (e.g., clothing, eyewear, headwear, jewelry) or any combination thereof (e.g., mobile Internet device/MID). In the illustrated example, the system <NUM> includes a battery <NUM> to supply power to the system <NUM> and a processor <NUM> having an integrated memory controller (IMC) <NUM>, which may communicate with system memory <NUM>. The system memory <NUM> may include, for example, dynamic random access memory (DRAM) configured as one or more memory modules such as, for example, dual inline memory modules (DIMMs), small outline DIMMs (SODIMMs), etc..

The illustrated system <NUM> also includes a input output (IO) module <NUM>, sometimes referred to as a Southbridge of a chipset, that functions as a host device and may communicate with, for example, a display <NUM> (e.g., HD display, organic light emitting diode/OLED display, liquid crystal display/LCD, etc.), a peripheral device <NUM> (e.g., an AV player, Blu-ray player, DVD player, camera), one or more lenses <NUM> (e.g., optical lenses) of a head mounted display (HMD), and mass storage <NUM> (e.g., hard disk drive/HDD, optical disk, flash memory, etc.). The processor <NUM> may execute one or more distortion correction processes (not shown).

The illustrated processor <NUM> may also execute logic <NUM> that is configured to receive one or more images from the peripheral device <NUM>, map the received images onto a plurality of distortion meshes, use the distortion meshes to compensate for chromatic aberrations and radial distortions within the images, adaptively blend the distortion meshes into a corrected image to reduce distortions within the image, and output the corrected image to the display <NUM> and/or lens <NUM>. The corrected image reduces distortions within the image when viewed through the lens <NUM>. Thus, the illustrated logic <NUM> may function similarly to the logic module (<FIG>), already discussed.

One or more aspects of at least one embodiment may be implemented by representative instructions stored on a non-transitory machine-readable storage medium which represents various logic within the processor, which when read by a machine causes the machine to fabricate logic to perform the techniques described herein. Such representations, known as "IP cores" may be stored on a tangible, non-transitory, machine readable storage medium and supplied to various customers or manufacturing facilities to load into the fabrication machines that actually make the logic or processor.

Embodiments are applicable for use with all types of semiconductor integrated circuit ("IC") chips. Examples of these IC chips include but are not limited to processors, controllers, chipset components, programmable logic arrays (PLAs), memory chips, network chips, and the like. In addition, in some of the drawings, signal conductor lines are represented with lines. Some may be different, to indicate more constituent signal paths, have a number label, to indicate a number of constituent signal paths, and/or have arrows at one or more ends, to indicate primary information flow direction. This, however, should not be construed in a limiting manner. Rather, such added detail may be used in connection with one or more exemplary embodiments to facilitate easier understanding of a circuit. Any represented signal lines, whether or not having additional information, may actually comprise one or more signals that may travel in multiple directions and may be implemented with any suitable type of signal scheme, e.g., digital or analog lines implemented with differential pairs, optical fiber lines, and/or single-ended lines.

Example sizes/models/values/ranges may have been given, although embodiments are not limited to the same. As manufacturing techniques (e.g., photolithography) mature over time, it is expected that devices of smaller size could be manufactured. In addition, well-known power/ground connections to IC chips and other components may or may not be shown within the figures, for simplicity of illustration and discussion, and so as not to obscure certain aspects of the embodiments. Further, arrangements may be shown in block diagram form in order to avoid obscuring embodiments, and also in view of the fact that specifics with respect to implementation of such block diagram arrangements are highly dependent upon the platform within which the embodiment is to be implemented, i.e., such specifics should be well within purview of one skilled in the art. Where specific details (e.g., circuits) are set forth in order to describe example embodiments, it should be apparent to one skilled in the art that embodiments can be practiced without, or with variation of, these specific details. The description is thus to be regarded as illustrative instead of limiting.

Some embodiments may be implemented, for example, using a machine or tangible computer-readable storage medium or article which may store an instruction or a set of instructions that, if executed by a machine, may cause the machine to perform a method and/or operations in accordance with the embodiments. Such a machine may include, for example, any suitable processing platform, computing platform, computing device, processing device, computing system, processing system, computer, processor, or the like, and may be implemented using any suitable combination of hardware and/or software. The machine-readable storage medium or article may include, for example, any suitable type of memory unit, memory device, memory article, memory medium, storage device, storage article, storage medium and/or storage unit, for example, memory, removable or non-removable media, erasable or non-erasable media, writeable or re-writeable media, digital or analog media, hard disk, floppy disk, Compact Disk Read Only Memory (CD-ROM), Compact Disk Recordable (CD-R), Compact Disk Rewriteable (CD-RW), optical disk, magnetic media, magneto-optical media, removable memory cards or disks, various types of Digital Versatile Disk (DVD), a tape, a cassette, or the like. The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, and the like, implemented using any suitable high-level, low-level, object-oriented, visual, compiled and/or interpreted programming language.

Claim 1:
A head mounted display system (<NUM>) comprising:
a lens (<NUM>, <NUM>, <NUM>);
a display (<NUM>, <NUM>) configured to present an image (<NUM>, <NUM>) to be viewed via the lens (<NUM>, <NUM>, <NUM>); and
a processor (<NUM>, <NUM>) configured to:
receive an image (<NUM>, <NUM>) to be presented on the display (<NUM>, <NUM>);
apply distortion meshes to the received image (<NUM>, <NUM>) to compensate for distortion to be caused by the lens (<NUM>, <NUM>, <NUM>) when the image is viewed via the lens (<NUM>, <NUM>, <NUM>), wherein the distortion meshes are configured to compensate radial aberration and chromatic aberration by the lens (<NUM>, <NUM>, <NUM>),
generate a composite corrected image (<NUM>, <NUM>) by blending the distortion meshes applied to the received image (<NUM>, <NUM>), and
present the composite corrected image (<NUM>, <NUM>) to the display (<NUM>, <NUM>).