Patent Description:
The present invention relates generally to high dynamic range digital images. The invention relates specifically to methods and apparatus for encoding and decoding high dynamic range images, whether still or moving pictures, and to data structures containing digital high dynamic range images.

Human vision is capable of appreciating contrast ratios of up to <NUM>:<NUM>,<NUM>. That is, a person can take in a scene in which some parts of the scene are <NUM>,<NUM> times brighter than other parts of the scene and see details in both the brightest and darkest parts of the scene. Further, human vision can adapt its sensitivity to brighter or darker scenes over a further <NUM> orders of magnitude.

Most conventional digital image formats (so-called <NUM>-bit formats) use up to <NUM> bits to store color and luminance information for each pixel in an image. For example, each of a red, green and blue (RGB) value for a pixel may be stored in one byte (<NUM> bits). Such formats are capable of representing brightness variations over only about two orders of magnitude (each byte can store one of <NUM> possible values). There exist a number of standard formats for representing digital images (which include both still and video images). These include JPEG (Joint Photographic Experts Group), MPEG (Motion Picture Experts Group), AVI (Audio Video Interleave), TIFF (Tagged Image File Format), BMP (Bit Map), PNG (Portable Network Graphics), GIF (Graphical Interchange Format), and others. Such formats may be called "output referred standards" because they do not attempt to preserve image information beyond what can be reproduced by electronic displays of the types most commonly available. Until recently, displays such as computer displays, televisions, digital motion picture projectors and the like have been incapable of accurately reproducing images having contrast ratios better than <NUM>:<NUM> or so.

Display technologies being developed by the assignee, and others, are able to reproduce images having high dynamic range (HDR). Such displays can reproduce images which more faithfully represent real-world scenes than conventional displays. There is a need for formats for storing HDR images for reproduction on these displays and other HDR displays that will become available in the future.

A number of formats have been proposed for storing HDR images as digital data. These formats all have various disadvantages. A number of these formats yield prohibitively large image files that can be viewed only through the use of specialized software. Some manufacturers of digital cameras provide proprietary RAW formats. These formats tend to be camera-specific and to be excessive in terms of data storage requirements. Non-patent literature <NPL>), discloses the use of the JPEG file format for transmitting HDR image data.

There is a need for a convenient framework for storing, exchanging, and reproducing high dynamic range images. There is a particular need for such a framework which is backwards compatible with existing image viewer technology.

Example possible embodiments, which relate to HDR encoding, decoding, and data structures, are described herein. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

According to one embodiment of the present invention, an HDR data structure is configured to be readable by legacy image viewers. The legacy image viewers can read a tone map information and ignore HDR information, such as a ratio data (as explained later). In some embodiments, the data structure comprises a JFIF file and the tone map information comprises a JPEG image. In some embodiments, the data structure comprises a MPEG file and the tone map information comprises a frame of a MPEG video.

Another aspect of the invention provides a data structure for representing a high dynamic range image having an initial dynamic range. The data structure comprises a tone map portion and a high dynamic range information portion. The tone map portion contains tone map information representing the image and has a dynamic range less than the initial dynamic range. The high dynamic range information portion contains information describing ratios of (luminance) values in the tone map portion to luminance values of the high dynamic range image.

One aspect of this invention provides methods for encoding high dynamic range image data. The methods involve obtaining, or otherwise generating, a tone map information corresponding to the high dynamic range image data. The tone map information has a dynamic range lower than that of the high dynamic range image data. The method computes ratio data comprising ratios of values in the high dynamic range image data and corresponding values in the tone map information. The ratio data (or derivative information therefrom) and the tone map information can be stored and transmitted for decoding.

Another aspect of this invention provides methods for decoding a codestream to reconstruct a high dynamic range image. The methods involve receiving, or otherwise accessing, a tone map information and corresponding ratio data (or derivative information therefrom). The method computes a high dynamic range image utilizing the values in the tone map information and corresponding ratio data.

Ratio data, as referred to in this application in its entirety, can be computed, without limitation, (i) as mathematical division of numerator and denominator values, including without limitation, further mathematical operations - such as logarithm of the ratio, or (ii) alternatively, as subtraction of two logarithmic values, including without limitation, further mathematical operations. Typically, ratio data describes luminance, but can be used for chroma channels (e.g., Cr, Cb) as well. For the sake of clarity, ratio data is sometimes described herein as residual data or included with residual data.

<FIG> illustrates an exemplary decoding process according to an embodiment of the present invention. The process begins with a legacy decoder block which reconstructs the base image. This image is then optionally chroma upsampled, followed by an inverse decorrelation block. The output of this transformation is a low-dynamic range, backward compatible image with eight bits per sample in, for example, an RGB-type color space.

The low-dynamic range components are further mapped by the base mapping and color space conversion block to a floating point image which is called a precursor image. The precursor image is optionally converted to HDR color space and luminance can be calculated. The noise level maybe used to avoid division by zero and to reduce the compression artifacts which can be amplified in following blocks.

The residual decoder path uses the residual data that is embedded in the codestream in the APP11 markers. This data is reconstructed and then optionally upsampled. It is then processed by a residual mapping and inverse decorrelation block. This block maps the residual data to floating point domain which is optionally inversely decorrelated. This mapping can use the luminance computed by the base mapping and color space conversion block. The mapped residual data and the precursor image are processed by the HDR reconstruction block to produce a reconstructed HDR Image.

<FIG> illustrates exemplary decoding process according to another embodiment of the present invention. The decoding process relies on a layered approach by decomposing an HDR image into a base layer and an HDR residual ratio layer. The base layer is a tone mapped image tone mapped from the original floating point HDR with either a local or global tonemapper. This codestream will be the backwards compatible with, accessible by, legacy decoders. The residual ratio layer contains HDR quantized log luminance ratio and the chrominance residual difference, this data is put together and represented as a single residual ratio image.

Since the residual data is hidden in the APP11 markers, legacy decoders can skip over this residual image and only access the base image codes stream, and thus this decoding process is backwards compatible. However, decoders implementing the present invention can combine the two layers to reconstruct an HDR image.

In <FIG>, the upper path which comprise of blocks B1, B2, and B3 can be the standard flow of a legacy decoder and outputs a backwards compatible lower dynamic range (LDR) image in typically sRGB space. This base image data is then mapped into linear HDR space and processed by the color space conversion operation in block B4. This block converts the LDR image into the color space of the original HDR image, and it also maps the image to floating point value and called linear pre RGB2, it can also be referred to as "LP_RGB2. " A noise floor value specified in the parameter codestream is added to the luminance component of the LP_RGB2 to avoid divide by <NUM> and to avoid amplifying any noise that could occur due to operations downstream from this block B4 for small values.

In <FIG>, the lower path starting from B5 begins with the residual data of the high-dynamic range image, and is represented by the ISO/IEC <NUM>-<NUM> codestream format (which shows desired formats). This codestream is embedded in the APP11 marker as a residual data segment described below. After being decoded by the decoder the chroma upsampling step is performed by B6 to bring all components to full resolution, e.g., <NUM>:<NUM>:<NUM>.

The residual ratio data is then separated by B7 into floating point linear ratio luminance values and a linear residual color difference value. The incoming residual luminance values are inverse quantized according to parameters in the codestream. A specific embodiment, this is either provided by an explicit lookup table in the parameter segment in the codestream. If this table is not present, then using the min and max, referred to as ln1, ln0 in the parameters segment, and an inverse log map is calculated. Similarly, the incoming chroma residual sample values are inverse quantized according to the minimum and maximum parameters stored in the parameter segment of the codestream as cb0, cb1 and cr0, cr1, if present.

The chroma values are then processed by B8, the YCbCr to RGB2 block and will convert the linear dequantized YCbCr to a linear residue RGB2 in the HDR color space, alternatively referred to as "LR_RGB2. " Finally, blocks B9 and B10 constructs an HDR image by first adding the linear pre RGB2 to the linear residue RGB2 in B9 and then multiplying the result by the linear ratio luminance in B10.

As shown in <FIG>, the APP11 marker segment is broken into a parameter data segment and a data segment. The parameter segment has two or more (e.g., <NUM>) types of segments, such as a parameter ASCII type segment, residual segment, and a parameter binary type segment. This structure for the APP11 marker segment can be used in connection with any embodiment of the invention described herein, including without limitation, the exemplary embodiments reflected in <FIG>.

A parameter data segment (PDS) carries parameters encoded, as ASCII or binary text, as payload data. The last parameter in the segment is a checksum of the base layer codestream. In a specific embodiment, the ckb (ASCII) or chksum (binary, <NUM> bits) parameter is a checksum of the base layer codestream computed by summing all bytes in the base layer codestream. The checksum includes the first SOF (e.g., start of frame) marker after the last APP11 marker segment and includes all following bytes up to and including the EOI (e.g., end of frame) marker. It can be used by the decoder to detect an edit of the base layer, which may result in undesirable artifacts when the high dynamic range (HDR) image is decoded. In a specific embodiment, the checksum is position (or order) dependent, such as a Fletcher's checksum (e.g., Fletcher-<NUM>, Fletcher -<NUM>, Flectcher-<NUM>). See <NPL> for additional information.

In an alternative embodiment, the PDS can indicate the use of a more complex hash algorithm than checksum. A more complex hash algorithm reduces the possibilities of hash collisions, e.g., undetectable alterations in the data when different input data results in same hash value. According, a hash value generated for the original base layer should probabilistically be unlikely to match if the base layer is altered. Exemplary hash functions can be, or implemented by:.

In yet other alternative embodiments, fingerprinting or media watermarking techniques can be signaled by the PDS and verified during decoding or image reproduction/rendering.

The checksum, hash function or the other described alternatives for base layer edit detection can be used in connection with any embodiment of the invention described herein, including without limitation, the exemplary embodiments reflected in <FIG>. Additionally, based on the teaching herein, a checksum, hash function or alternative can be used for edit detection of the residual ratio layer too.

Another parameter within the PDS or elsewhere can be an encryption parameter, such as an encryption key. This information can be used to decrypt the ratio residue layer, for example, on a per segment basis of the codestream. A segment can be an independently decodable sequence of entropy encoded bytes of compressed image data. In other words, according to an embodiment of the present invention, a different encryption parameter can be provided and used for each segment. The encryption parameter and associated processing can be used in connection with any embodiment of the invention described herein, including without limitation, the exemplary embodiments reflected in <FIG>.

A degamma lookup table (LUT) described above (as block B4 in <FIG>) is a <NUM> entry table loaded by a default Rec. <NUM> table (ITU-R Recommendation BT. <NUM>, available at http://www. int/rec/R-REC-BT. <NUM>-<NUM>-<NUM>-I/en) which is typically an inverse linear and power function of <NUM>. If it is in an alternate color space, such as Adobe RGB by Adobe Systems, Inc. , the look up table can be sent in header information. Additionally, the degamma LUT can include an inverse tone mapping function/curve, such as for reverse histogram equalization or inverse Reinhard tone mapper. In some cases, the degamma LUT with inverse tone mapping can reduce memory used for the residual ratio layer. For additional information on the Reinhard tone mapper, see http://www. edu/~reinhard/cdrom/tonemap. pdf ("Photographic Tone Reproduction for Digital Images").

The APP11 marker segment can include binary parameter data, as shown as "Type <NUM>" in <FIG>. The Type <NUM> segment and its associated processing can be used in connection with any embodiment of the invention described herein, including without limitation, the exemplary embodiments reflected in <FIG>.

In an embodiment of the present invention, the span and extent of segments for residual ratio image need to be coincident with a base layer image. For example, a residual ratio image can be partitioned into a plurality of segments, contiguous and non-contiguous. A set of these segments of the residual ratio image need not correspond to a complete image, but can define one or more portions of an image. This functionality allows HDR reconstruction from a portion of the base layer image, but not the entire base layer image. For example, an encryption parameter can be provided for one segment (e.g., left half image, top half image) for HDR reconstruction, while residual ratio information for another segment (e.g., right half image, bottom half image) remains encrypted for limited base layer reproduction.

Each segment of the residual ratio image can be specified by coordinate references (e.g., x and y coordinates for one of the four corners if rectangular segment) and its length and width. If segment is a different geometric shape, then it can be defined by a center position and a radius/diameter or the like. <FIG> illustrate exemplary segments of the residual ratio image, which can used in connection with any embodiment of the present invention, including without limitation, the exemplary embodiments reflected in <FIG>.

According to one embodiment, the techniques as described herein are performed by computer system <NUM> in response to processor <NUM> executing one or more sequences of one or more instructions contained in main memory <NUM>.

Claim 1:
A decoder apparatus, including a processor, which implements a method for decoding a high dynamic range - HDR - image, the method comprising:
receiving base layer data, wherein the base layer data includes a tone-mapped low dynamic range version of the HDR image;
receiving an encoder alteration parameter indicative of an alteration of the base layer data after encoding, wherein the encoding alteration parameter is a checksum of the encoded base layer data;
receiving residual ratio data in one or more APP11 marker segments of the HDR image, wherein the residual ratio data includes information describing ratios of luminance values of the HDR image and corresponding luminance values of the base layer data;
computing a decoder alteration parameter for the base layer data, wherein the decoder alteration parameter is a checksum of the received base layer data; and
comparing the encoder alteration parameter to the decoder alteration parameter to determine whether the received base layer data has been altered after encoding.