Patent Description:
The present invention relates generally to images. More particularly, an embodiment of the present invention relates to image prediction for high dynamic range (HDR) imaging in mobile applications.

As used herein, the term 'dynamic range' (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest grays (blacks) to brightest whites (highlights). In this sense, DR relates to a 'scene-referred' intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a 'display-referred' intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g., interchangeably.

As used herein, the term high dynamic range (HDR) relates to a DR breadth that spans the <NUM>-<NUM> orders of magnitude of the human visual system (HVS). In practice, the DR over which a human may simultaneously perceive an extensive breadth in intensity range may be somewhat truncated, in relation to HDR.

In practice, images comprise one or more color components (e.g., luma Y and chroma Cb and Cr) wherein each color component is represented by a precision of n-bits per pixel (e.g., n=<NUM>). Using linear or gamma luminance coding, images where n ≤ <NUM> (e.g., color <NUM>-bit JPEG images) are considered images of standard dynamic range, while images where n > <NUM> may be considered images of enhanced or high dynamic range. HDR images may also be stored and distributed using high-precision (e.g., <NUM>-bit) floating-point formats, such as the OpenEXR file format developed by Industrial Light and Magic.

Most consumer desktop displays currently support luminance of <NUM> to <NUM> cd/m<NUM> or nits. Most consumer HDTVs range from <NUM> to <NUM> nits with new models reaching <NUM> nits (cd/m<NUM>). Such conventional displays thus typify a lower dynamic range (LDR), also referred to as a standard dynamic range (SDR), in relation to HDR. As the availability of HDR content grows due to advances in both capture equipment (e.g., cameras) and HDR displays (e.g., the PRM-<NUM> professional reference monitor from Dolby Laboratories), HDR content may be color graded and displayed on HDR displays that support higher dynamic ranges (e.g., from <NUM>,<NUM> nits to <NUM>,<NUM> nits or more).

As used herein, the term "reshaping" or "remapping" denotes a process of sample-to-sample or codeword-to-codeword mapping of a digital image from its original bit depth and original codewords distribution or representation (e.g., gamma, PQ, or HLG and the like) to an image of the same or different bit depth and a different codewords distribution or representation. Reshaping allows for improved compressibility or improved image quality at a fixed bit rate. For example, without limitation, forward reshaping may be applied to <NUM>-bit or <NUM>-bit PQ-coded HDR video to improve coding efficiency in a <NUM>-bit video coding architecture. In a receiver, after decompressing the received signal (which may or may not be reshaped), the receiver may apply an inverse (or backward) reshaping function to restore the signal to its original codeword distribution and/or to achieve a higher dynamic range.

In HDR coding, image prediction (or reshaping) allows an HDR image to be reconstructed using a baseline standard dynamic range (SDR) image and a set of prediction coefficients representing a backward reshaping function. Legacy devices may simply decode the SDR image; however, HDR displays may reconstruct the HDR image by applying the backward reshaping function to the SDR image. In video coding, such image prediction may be used to improve coding efficiency while maintaining backwards compatibility.

Reconstructing HDR video on mobile devices adds additional challenges because most mobile devices lack special hardware needed to accelerate compute-intensive operations while supporting a desired level of expected quality in HDR imaging. As appreciated by the inventors here, improved techniques for efficient image prediction in mobile applications are desired.

<CIT> discloses coding syntaxes in compliance with same or different VDR specifications signaled by upstream coding devices such as VDR encoders to downstream coding devices such as VDR decoders in a common vehicle in the form of RPU data units. VDR coding operations and operational parameters may be specified as sequence level, frame level, or partition level syntax elements in a coding syntax. Syntax elements in a coding syntax may be coded directly in one or more current RPU data units under a current RPU ID, predicted from other partitions/segments/ranges previously sent with the same current RPU ID, or predicted from other frame level or sequence level syntax elements previously sent with a previous RPU ID. A downstream device may perform decoding operations on multi-layered input image data based on received coding syntaxes to construct VDR images.

<CIT> discloses a tensor-product B-Spline (TBP) predictor. A set of TPB basis functions is determined. A set of selected TPB prediction parameters to be used with the set of TPB basis functions for generating predicted image data in mapped images from source image data in source images of a source color grade is generated. The set of selected TPB prediction parameters is generated by minimizing differences between the predicted image data in the mapped images and reference image data in reference images of a reference color grade. The reference images correspond to the source images and depict same visual content as depicted by the source images. The set of selected TPB prediction parameters is encoded in a video signal as a part of image metadata along with the source image data in the source images. The mapped images are caused to be reconstructed and rendered with a recipient device of the video signal.

The scope of protection of the present invention is as defined in the appended claims. Any embodiments not falling within such scope should be construed as examples which do not describe parts of the invention.

The features and effects of the application will become apparent from the following detailed description when read in conjunction with the accompanying drawings in which like reference numerals refer to similar elements and in which:.

Image prediction techniques for the efficient coding of images in mobile applications are described herein. Well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

<FIG> illustrates an example single-layer decoder architecture using image prediction, which may be implemented with one or more computing processors in a downstream video decoder. <FIG> illustrates an example HDR encoder architecture, which may also be implemented with one or more computing processors in one or more upstream video encoders.

Under this framework, given reference HDR content (<NUM>), corresponding SDR content (<NUM>) (that is, content that represents the same images as the HDR content, but color-graded and represented in standard dynamic range) is encoded and transmitted in a single layer of a coded video signal (<NUM>) by an upstream encoding device that implements the encoder-side codec architecture. The SDR content (<NUM>) is received and decoded, in the single layer of the video signal, by a downstream decoding device. Prediction metadata (e.g., forward or backward reshaping parameters) (<NUM>) is also encoded and transmitted in the video signal with the SDR content so that HDR display devices can reconstruct HDR content based on the SDR content (<NUM>) and the received metadata (<NUM>).

In <FIG>, in an embodiment, given input HDR data (<NUM>), SDR data (<NUM>) may be generated from the HDR data via tone-mapping, forward reshaping, manually (during color-grading), or via a combination of techniques known in the art. In another embodiment, given reference SDR data (<NUM>), the HDR data (<NUM>) may be generated from the SDR data via inverse tone-mapping, backward reshaping, manually (during color-grading), or via a combination of techniques known in the art. A compression block <NUM> (e.g., an encoder implemented according to any known video coding algorithms, like AVC, HEVC, AV1, and the like) compresses/encodes the SDR images (<NUM>) in a single layer <NUM> of a coded bitstream.

The metadata (<NUM>), as generated by unit <NUM>, may be multiplexed as part of the video signal <NUM>, for example, as supplemental enhancement information (SEI) messaging or as part of proprietary messaging. Thus, the metadata (<NUM>) can be generated or pre-generated on the encoder side to take advantage of powerful computing resources and offline encoding flows (including but not limited to content adaptive multiple passes, look ahead operations, inverse luma mapping, inverse chroma mapping, CDF-based histogram approximation and/or transfer, etc.) available on the encoder side.

The encoder architecture of <FIG> can be used to avoid directly encoding the input HDR images (<NUM>) into coded/compressed HDR images in the video signal; instead, the metadata (<NUM>) in the video signal can be used to enable downstream decoding devices to reconstruct the SDR images (<NUM>) (which are encoded in the video signal) into reconstructed HDR images (<NUM>) that are identical to or closely/optimally approximate the reference HDR images (<NUM>).

In some embodiments, as illustrated in <FIG>, the video bitstream (<NUM>) with the compressed SDR images and the metadata (<NUM>) with prediction parameters generated by the encoder are received as input on the decoder side of the codec framework. A decompression block <NUM> decompresses/decodes compressed video data in the single layer (<NUM>) of the video signal into the decoded SDR images (<NUM>). Decompression <NUM> typically corresponds to the inverse of compression <NUM>. The decoded SDR images (<NUM>) may be the same as the SDR images (<NUM>), subject to quantization errors in the compression block (<NUM>) and in the decompression block (<NUM>), which may have been optimized for SDR display devices. The decoded SDR images (<NUM>) may be outputted in an output SDR video signal (e.g., over an HDMI interface, over a video link, etc.) to be rendered on an SDR display device.

In addition, prediction block <NUM> (which also may be referred to as a "composer") applies the metadata (<NUM>) from the input bitstream to the decompressed data (<NUM>) to generate reconstructed HDR images (<NUM>). In some embodiments, the reconstructed images represent production-quality or near-production-quality HDR images that are identical to or closely/optimally approximating the reference HDR images (<NUM>). The reconstructed images (<NUM>) may be outputted in an output HDR video signal (e.g., over an HDMI interface, over a video link, etc.) to be rendered on an HDR display device.

In some embodiments, display management operations specific to the HDR display device may be performed on the reconstructed images (<NUM>) as a part of HDR image rendering operations that render the backward reshaped images (<NUM>) on the HDR display device.

While <FIG> depict example embodiments of single-layer HDR encoding and decoding systems. The methods and systems described herein are also applicable to multi-layer and scalable systems where HDR content is transmitted using a base layer and one or more enhancement layer bitstreams, and in a decoder, information from the enhancement layers is added back to the base layer.

Because mobile devices (e.g., mobile phones, tablets, and the like) are limited in terms of computing power, storage, and battery life, the playback of HDR content on mobile devices adds additional challenges. For example, unlike HDR-enabled TVs or set-top boxes, most mobile devices do not have dedicated hardware accelerators to reconstruct HDR content. HDR playback on mobile devices relies mostly on software running on general purpose processors. However, without hardware support, the power consumption and potential picture frame dropping become serious issues for playback and perceived visual quality. To alleviate those impacts, instead of adjusting the SDR-to-HDR mapping in every frame, a more "static," (e.g., scene-based), SDR to HDR mapping is preferred since the mapping can be done using a "static" 3D look-up table (LUT) to avoid re-computing mapping LUTs for every video frame. However, existing legacy schemes (e.g., Ref. [<NUM>]) which rely on luma prediction using <NUM>-piece polynomials and chroma prediction using multiple-color channel, multiple regression (MMR) predictors (Ref. [<NUM>]) cannot provide a satisfactory static mapping to cover a wide color gamut (WCG), especially close to ITU-R Rec. Tensor-Product B-Spline (TPB)-based prediction can achieve this WCG goal; however, support for TPB metadata may not be possible with existing legacy metadata protocols for HDR imaging (e.g., the current Profiles of Dolby Vision). Given backward-compatibility requirements, example embodiments describe a mobile-optimized profile in which the encoded bitstreams contain two different prediction methods using a single base layer: a) a proposed new TPB prediction method, and b) a legacy, composer-based, method. Before providing a detailed description of the proposed profile, the next sections provide a quick summary of TPB-based prediction and a novel embodiment for fast TPB-based decoding.

Tensor-Product B-Splines (TPB) may be used to perform predictions in an HDR composer pipeline. B-splines or basis splines are functions that can approximate a given one-dimensional curve using polynomials with continuity constraints at knot points. TPB multiplies multiple B-spline functions together to estimate higher dimensional curves. The accuracy in predicting complex curves while maintaining smooth connectivity at knot points makes TPB a powerful tool for forward and backward reshaping in high-dynamic range processing (Ref. [<NUM>]).

As depicted in <FIG>, during encoding, HDR pixels are mapped to base layer (BL) pixels (<NUM>) in a forward reshaping process. After subsequent compression and decompression, these base layer pixels are mapped back to HDR pixels during backward reshaping. Metadata containing backward reshaping coefficients is transmitted along with the compressed base layer to facilitate backward reshaping at the decoder side. Although TPB-based decoding produces far superior results compared to the existing techniques, the computational complexity is much higher. Later on, embodiments for a fast TPB decoding process, that achieves 100x speed up as compared to regular TPB decoding, will be described. Specifically, embodiments for computing the values of 3D-LUT (<NUM>-dimensional lookup table) entries used in the decoding process will be described.

A Full-HD frame contains approximately <NUM> pixels and an Ultra-HD frame has around <NUM> pixels. Predicting each of these pixels by applying backward reshaping coefficients directly is not a practical solution for time-sensitive applications. A faster way is to build a 3D-LUT that stores the backward reshaped values at sampling locations in the input pixel value range. Interpolation techniques are then used to find backward reshaped values for pixels not present in the 3D-LUT. For example use-cases, but with no limitation, these 3D-LUTs are now generally 17x17x17 or 33x33x33 in size; however, depending on the desired accuracy of reconstruction and/or available processing power, one may select smaller or larger LUTs as well. That amounts to ~<NUM> or ~ <NUM> pixel entries. A 3D-LUT significantly reduces time complexity for any backward reshaping process (TPB-based or MMR-based) by reducing the total number of predictions from <NUM> or <NUM> to <NUM> or <NUM> for each frame. The proposed method further reduces the time complexity for constructing a TPB-based 3D-LUT by taking into account the properties of TPB functions. Experimental results show that the time required to construct a 3D-LUT with the fast method is one-hundredth of the time to build a 3D-LUT with a traditional method.

From Ref. [<NUM>], a function f: [a, b] → R is called a polynomial spline of degree [ >=<NUM> with knots a = k<NUM> <. < kT-<NUM> = b, if it fulfills the following conditions:.

B-Spline basis functions are constructed from piecewise polynomials that are fused smoothly at the knots to achieve the desired smoothness constraints. A B-Spline basis function consists of (l+<NUM>) polynomial pieces of degree l, which are joined in an (l-<NUM>) times continuously differentiable way. Using the complete basis, the function f(z) can be represented through a linear combination of D = T + l - <NUM> basis functions, where T is the number of knots, i.e. <MAT>.

Note that a B-Spline is only positive on an interval based on l + <NUM> knots. The <NUM>-th order of a B-Spline is defined as <MAT> The higher order of a B-Spline is defined recursively as <MAT>.

One needs <NUM> outer knots outside [a, b] (as exterior knots) in addition to the interior knots k<NUM>, k<NUM>,. , kT-<NUM>. For example, for T = <NUM> knots and using <NUM>-d order B-Spline, one will have D = <NUM> basis functions and <NUM> coefficients {mt} to determine, such that, a 1D curve can be approximated. The following subsection demonstrates a cross-channel TPB coefficients estimation process for luma and chroma backward mapping.

Suppose there are P pixels in one picture. Let the triplets ( <MAT>) and ( <MAT>) represent the normalized (e.g., in [<NUM>, <NUM>]) Y, Co and C<NUM> values for the i-th pixel in the j-th SDR (base layer) and HDR frames respectively. Take the prediction for luma channel as an example. For a single channel B-Spline predictor, one would like to predict <MAT> from <MAT>. With a set of selected knot points, one constructs the required Dy = T + l - <NUM> basis functions, <MAT>. The prediction <MAT> using coefficients <MAT> is as follows: <MAT>.

Tensor-Product B-Splines (TPB) can use information from luma and both the chroma channels ( <MAT>) to better predict the luma. One needs three independent basis functions in each dimension. In this case, one has <MAT>, and <MAT> basis functions along Y, C<NUM> and C<NUM> dimension. Denote ty, tc<NUM>, and tc<NUM> as the knot point indices in each of the Y, C<NUM>, and C<NUM> dimensions. Given a set of knot points in three dimensions, one has three individual basis function sets: <MAT>, and <MAT>. The tensor-product B-Spline basis function for predicting the luma channel can be constructed by multiplying all three channels together as shown below. <MAT> The cross-channel prediction can be performed as <MAT>.

One can vectorize the 3D index (i.e., ty, tc<NUM>, and tc<NUM>) to become a 1D index (i.e., t) to simplify the expression. For the basis function, then: <MAT> Let <MAT>, then the prediction can be expressed as: <MAT> With all P pixels in one frame, one can construct the design matrix <MAT> The prediction coefficients can be expressed as a vector <MAT>.

The prediction can be expressed as <MAT> where <MAT>.

The optimal solution of <MAT> can be obtained by minimizing a least squared error criterion: <MAT> In the above equation, the vector <MAT> represents the ground truth HDR luma values.

Similarly, one can build the tensor-product B-Spline predictor for the two chroma channels. Assuming one has <MAT>, and <MAT> (where c can be c0 or c1) basis functions along Y, C<NUM> and C<NUM> dimension. Given a set of knot points in three dimensions, one has three individual basis function sets <MAT>, and <MAT>. The tensor-product basis function for chroma channel is as follows. <MAT> One can vectorize the 3D index (i.e., ty, tc<NUM>, and tc<NUM>) to become a 1D index (i.e., t) to simplify the equation. Let <MAT>, <MAT> With all P pixels in one frame, one can construct the matrix form <MAT> The prediction coefficients can be expressed as a vector <MAT> The prediction can be expressed as <MAT> The optimal solution of <MAT> can be obtained via a least squared error minimization, as: <MAT> It is worthy to note that if all three channels have the same tensor-product basis functions, then the S matrix will be identical, i.e. <MAT> At the decoder side, one just needs to compute <MAT>. There is no need to compute it three times for each channel. Then the predicted value for each channel will be simply the multiplication of the S matrix with corresponding prediction coefficients.

The location of knot points can be uniformly or non-uniformly distributed. Uniform distribution simplifies the decoding process and reduces the amount of metadata to be sent to the decoders. Instead of the locations of all the knot points, only the total number of knot points is required. Assuming uniformly-distributed knot points, in an embodiment, the TPB metadata consists of:.

These parameters may not be directly related to tensor-product B-spline representation, but they do assist in TPB-based prediction. A more detailed description is presented later.

An example of TPB parameters present in the metadata is depicted in Table <NUM>.

This metadata is transmitted along with the compressed base layer to the decoders. The remaining of the TPB parameters, such as, knot points and the knot-points distance h, are derived from the above parameters. Assuming a uniform distribution of knot points for the equations, for simplicity, one can drop the superscripts and subscripts for the symbols. Then, for T knot points, the distance between the knot points, h, is calculated as: <MAT>.

Knot points {kt} may form a 1D array that can be derived from the TPB parameters in the metadata as follows. <MAT> In the equation above, t∈[K(l - <NUM>), <NUM> × l + T - <NUM>] and K = {<NUM>,<NUM>,<NUM>}. These derived TPB parameters and the original parameters in the metadata are used together in the decoding process.

In an embodiment, as explained earlier, it is worthwhile to build a 3D-LUT to store the mapping from the SDR (base layer) to the HDR domain for sample pixels. <MAT> Let us represent the TPB-based 3D-LUT that maps input SDR pixel ( <MAT>) to output HDR pixel ( <MAT>) by <MAT>. Here i is the pixel index in the 3D-LUT and j is the frame index. Let the size of <MAT> be represented by L<NUM> × L<NUM> × L<NUM>. In an example, without limitation, a 3D-LUT can be <NUM> × <NUM> × <NUM> or <NUM> × <NUM> × <NUM> in size, i.e., L<NUM> = L<NUM> = L<NUM> = <NUM> or L<NUM> = L<NUM> = L<NUM> = <NUM>.

Metadata received from the composite bitstream is parsed to get the TPB parameters, coefficients and the auxiliary data. Then a 3D-LUT can be constructed using two methods:.

Both methods are described in the following subsections. Henceforth it is assumed that the TPB knot points are uniformly distributed. TPB parameters are tabulated below, where every dimension in each channel has its corresponding knot point array {kt}, knot point distance h , and TPB order l.

When the knot points are uniformly distributed, including the exterior knot points, the B-Spline basis function can be expressed by a linear combination of truncated polynomials. An l-th order truncated polynomial is defined as <MAT> In the following, {kt} denotes knot points and h denotes the distance between two consecutive knot points for a uniform distribution. For simplicity, the core equations are defined without the channel-specific or dimension-specific details. The channel and dimension specific notations will be reintroduced when explaining the luma- or chroma-specific TPB prediction equations.

The <NUM>-st order of the t-th B-Spline basis function can be expressed as <MAT> <MAT> is only positive between [kt-<NUM> kt+<NUM>] or <NUM> interval. It will be <NUM> outside this range.

The <NUM>-nd order of the t-th B-Spline basis function can be expressed as <MAT> <MAT> is only positive between [kt-<NUM> kt+<NUM>] or <NUM> interval. It will be <NUM> outside this range.

The <NUM>-rd order of the t-th B-Spline basis function can be expressed as <MAT> <MAT> is only positive between [kt-<NUM> kt+<NUM>] or <NUM> interval. It will be <NUM> outside this range.

Having the truncated polynomial expression avoids the need to apply the recursive B-Spline basis function at the decoder side. Note that a recursive B-Spline expression is not friendly for hardware implementation. The recursive expression takes more time and memory space to compute. The truncated polynomial expression can save the computation (by trading the flexibility to have "non-uniform" knot points). The overall uniform knot point TPB basis function for single channel Bt is described as follows.

The above single channel basis function can be extended to cross-channel TPB basis function <MAT> by taking the product of the B-Spline functions in the three dimensions. Let us take the luma channel prediction as an example. Using <MAT> and <MAT> as knot points for dimensions Y, C0 and C1 for luma channel predictor, the distance between every two consecutive knot points can be expressed as hy,y, hy,c<NUM> and hy,c<NUM> respectively. The orders of TPB in these three dimensions are ly,y, ly,c<NUM> and ly,c<NUM> respectively. Then the following equation gives the cross channel TPB basis function for luma channel predictor TPB,y <MAT>.

The t-th TPB basis function above is formed by multiplying the p-th, q-th, and r-th B-Spline functions, one in each of the three dimensions Y, C0 and C1. Please note that each of the three inputs <MAT> are clipped using their respective low clipping values, i.e., <MAT> and high clipping values, namely <MAT>, received from the metadata as shown here. <MAT> <MAT> <MAT>.

The clip3(w, wmin, wmax) function clips the value of w to keep it within the range [wmin, wmax]. In other words, if w < wmin then w = wmin. Else if w > wmax, then w = wmax. Otherwise, the value of w is not changed. Given the clipped SDR pixel ( <MAT>), one predicts the HDR values <MAT> using the luma channel TPB coefficients. <MAT> Denote computing a B-Spline function value, such as <MAT> for a given <MAT> and a given knot point <MAT>, as one B-Spline operation.

<MAT> So, <NUM> B-Spline operations are required to evaluate each 3D-LUT entry.

For chroma channels, let us represent <MAT> and <MAT> as knot points for dimensions Y, C0 and C1 for a chroma c = c0, c1 channel predictor. The distance between every two consecutive knot points can be expressed as hc,y, hc,c<NUM> and hc,c<NUM> respectively. The order of TPB in these three dimensions are lc,y, lc,c<NUM> and lc,c<NUM> respectively. Then the following equation gives the cross channel TPB basis function for chroma channel predictor <MAT>. <MAT> One can then predict the HDR values <MAT> and <MAT> using the chroma channel TPB coefficients. <MAT> Similarly to the luma channel predictor, it takes around <NUM> B-Spline operations to compute all the HDR values in a <NUM> × <NUM> × <NUM> LUT for one chroma channel, given the same settings as those used for the luma channel predictor.

After all the three predicted HDR values are obtained, after clipping to the valid signal range, they are saved in the 3D-LUT <MAT>. For each entry in <MAT> the corresponding HDR pixel ( <MAT>) is predicted using the three cross-channel TPB predictors. The entire process of calculating the truncated polynomials, B-Spline functions, TPB basis function and TPB prediction is repeated for every single entry in the 3D-LUT. This makes the process too slow for time sensitive applications; however, many of the computations are repeated over and over for each entry of the 3D-LUT. These repetitive operations can be avoided by saving their values in arrays and reusing these stored values instead of calculating them every time. This gives significant savings in compute time as explained in the next subsection.

To speed up the process of predicting the HDR pixels from the base layer pixels for the 3D-LUT entries, in an embodiment, a faster method uses a small amount of extra memory to store intermediate results that are frequently used in the prediction process. Without the loss of generality, as an example, let us consider the luma channel predictor. The TPB basis function for the cross-channel luma predictor is given by: <MAT> If one focuses on the constituent B-Spline functions:.

Note: Functions <MAT>, and <MAT>, are dimension-specific variations of the generalized function Bt(x, kt, h, l) described in equation (<NUM>), where, for simplicity subscripts and superscripts were dropped. The subscript t in equation (<NUM>), is replaced with subscripts p, q, or r for each of the first, second, and third dimensions under consideration.

Now the TPB basis function equation for luma prediction can be represented in the form of arrays as: <MAT> One can predict the HDR values <MAT> using the luma channel TPB coefficients. <MAT> Even though Dy additions are required, they only access the values pre-computed in the arrays. Accessing arrays is much faster than calculating the corresponding B-Spline function. If one applies the same settings for knot points and order as used earlier, i.e..

If TPB predictions are computed using the fast method, then it takes <NUM> B-Spline operations to compute all the luma HDR values in the 3D-LUT. On the contrary, the regular method needs <NUM> B-Spline operations to predict all the HDR luma values in the 3DLUT.

The following equation gives the cross-channel TPB basis function for chroma channel predictor <MAT>. <MAT> The B-Spline functions can be replaced with the array entries here as well. The details are skipped here, as they are similar to the derivations in the luma channel predictor.

For the chroma channels the number of B-Spline operations is <NUM> to predict all the HDR chroma values for a <NUM> × <NUM> × <NUM>3DLUT with the above settings. One can reduce the number of B-Spline operations further, if the TPB parameters in metadata for two or more channels are the same. For the discussion to follow, let the collection of TPB parameters present in the metadata for a channel c = y, c<NUM>, c<NUM> be denoted by, <MAT> The number of knot points T and orders l are the only TPB parameters present in the metadata. Note that TPB coefficients are not included in the definition of TPB parameters. If the parameter collections for two channels are compared elementwise and all the corresponding values are equal, then the TPB parameters for these two channels are considered to be the same. Otherwise, the TPB parameters for the two channels are different. This gives rise to three common scenarios:.

These scenarios can be identified from the basis flags <MAT>, and <MAT> in the metadata. An example flow chart of these three scenarios is depicted in <FIG>. Henceforth, without loss of generality, it is assumed that the clipping values for all the three channels are different in these three scenarios. That is: <MAT> <MAT>.

This is the default, worst case, scenario, where the TPB parameters for all the channels are different from each other. Mathematically, it can be expressed as Γy ≠ Γc<NUM> ≠ Γc<NUM> or more elaborately:<MAT><MAT><MAT> This scenario can also be directly deduced from the metadata syntax, for brevity, to also be referred to as reference-processing-unit (RPU) syntax, as well, when basis flags are <MAT> <MAT>. There will be three different 2D arrays for each channel to store the B-Spline functions. Table <NUM> shows the 2D arrays in this scenario and <FIG> depicts an example illustration.

To estimate the number of B-Spline operations, let us assume that Γy = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}, Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>} and Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}. Even though these example values resemble the case that all channels have the same TPB parameters, using these values will help us to do a fair comparison between scenarios. There are <NUM> B-Spline operations for each channel. The total number of B-Spline operations in this scenario will be <NUM> × <NUM> = <NUM>,<NUM> for constructing a complete 3D-LUT of size <NUM> × <NUM> × <NUM>.

This scenario is quite common as the chroma channels usually have the same TPB parameters and the luma channel has different parameters, i.e., Γy ≠ Γc<NUM> = Γc<NUM>. The corresponding equations are shown below. <MAT><MAT><MAT>.

This second scenario can also be indicated through the RPU syntax with basis flags <MAT>. Humans are more sensitive to luma changes than chroma variations. Therefore, luma is predicted by a more complex TPB basis function and chroma channels are predicted by simpler functions. The luma channel has different 2D arrays, but the chroma channels can share the same set of 2D arrays. Table <NUM> shows the 2D arrays used in this scenario and <FIG> depicts an example illustration.

When the basis flags are <MAT>, the second scenario can still be used; however, in practice, this scenario is rather rare and unusual. To estimate the number of B-Spline operations, let us assume that Γy = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}, Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>} and Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}. There are <NUM> B-Spline operations for each channel. The total number of B-Spline operations in this scenario is <NUM> × <NUM> = <NUM> for constructing a complete 3DLUT of size <NUM> × <NUM> × <NUM>.

In this scenario, the TPB parameter collections for the three channels are all the same, i.e., Γy = Γc<NUM> = Γc<NUM>, as shown in these equations. <MAT><MAT><MAT> This reduces the amount of metadata transmitted and the computations are faster at the decoder side. This third scenario can also be inferred from the RPU syntax when basis flags are <MAT>. All the three channels share the same set of 2D arrays. Table <NUM> shows the 2D arrays and <FIG> provides an example illustration.

To estimate the number of B-Spline operations, let us assume that Γy = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}, Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>} and Γc<NUM> = {<NUM>,<NUM>,<NUM>,<NUM>,<NUM>,<NUM>}. There are <NUM> B-Spline operations for each channel. The total number of B-Spline operations in this scenario is <NUM>,<NUM> for constructing a complete 3DLUT of size <NUM> × <NUM> × <NUM>.

In all the above scenarios, the sampling locations can be either uniformly distributed or non-uniformly distributed along the Y, C0 and C1 axes. Similarly, the sampling locations along Y, C0 and C1 axes can be the same (e.g., <MAT>) or different (e.g., <MAT>). The algorithm will still work as usual. Thus, variations in the sampling locations have no impact on the execution time of the above three scenarios.

There is a possible fourth scenario, where (i) TPB parameters are the same for all the channels (ii) sampling locations are also same for all the axes and (iii) low and high clipping values are same for all the channels too. Then only one 2D array should suffice. But as the clipping values are generally different, this scenario is very rare.

<FIG> summarizes the process flow for the three scenarios described earlier. For scenario <NUM>, nine temporary arrays are generated (e.g., see Table <NUM>). Each color component uses a separate set of three of these tables. For scenario <NUM>, six temporary arrays are generated (e.g., see Table <NUM>). Luma uses three of these tables, and both chroma components use the other three. For scenario <NUM>, three temporary arrays are generated (e.g., see Table <NUM>), and the same tables are used to generate the 3D-LUT to predict all color components. Given these temporary tables, a decoder builds a 3D-LUT (see equation (<NUM>)), which can be used for SDR-to-HDR prediction as depicted in equations (<NUM>) and (<NUM>).

As described earlier, while TPB-based mapping offers improved video quality compared to the traditional prediction (e.g., via piece-wise linear prediction for luma or MMR prediction for chroma), HDR content needs to be played back by both legacy devices (say, HDR TVs on the market) and new devices that support TPB-related metadata.

In an embodiment, an extended mapping index in combination with an existing parameter that denotes the bit-depth of an enhancement-layer stream (e.g., as used in legacy Blu-Ray HDR content) is used to point to the usage of TPB prediction parameters. When the bitstream arrives at a new mobile device which supports the new RPU syntax and TPB prediction, the device will use TPB to build a static or dynamic 3D-LUT to reconstruct the HDR. When the bitstream arrives at a legacy device (such as a Dolby Vision TV), the device will use the (static or dynamic) composer to construct HDR images. This way, streams can be backward-compatible but the user experience can be optimized for users with the latest decoders, e.g., mobile users. <FIG> depicts an example process of such backward compatibility.

As depicted in <FIG>, in an example embodiment, metadata for HDR content targeting both legacy devices and the new mobile-optimized profile (MOP) includes metadata within two syntax structures, denoted here as RPU (type <NUM>), for legacy content, and RPU (type <NUM>), for additional TPB-related content. Legacy devices (<NUM>) will ignore the RPU (type <NUM>) content and will reconstruct the HDR content using traditional HDR decoding (<NUM>). Mobile devices (<NUM>) that support RPU (type <NUM>) metadata will use all available metadata to generate a TPB-based 3D-LUT and perform TPB-based decoding (<NUM>), with improved quality.

In an embodiment, TPB prediction may be signaled by using a combination of two syntax elements: EL_bit_depth_minus8 and ext_mapping_idc, with example descriptions as follows:
EL_bit_depth_minus8 is used to derive the bit depth of enhancement layer signal, namely EL_bit_depth, and to derive the extended base layer inverse mapping method indicator, namely ext_mapping_idc. EL_bit_depth = (EL_bit_depth_minus8 & 0xFF) + <NUM>. ext_mapping_idc = (EL_bit_depth_minus8 & 0xFF00) >> <NUM>. The value of ((EL_bit_depth_minus8 & 0xFFFF0000) >> <NUM>) shall be equal to zero. The value of EL_bit_depth_minus8 shall be in the range of <NUM> to <NUM>, inclusive. EL_bit_depth_minus8 is not present when rpu_type is equal to <NUM> and rpu_format&0x700 is not equal to <NUM>. The lower <NUM> bits of ext_mapping_idc is an extended base layer inverse mapping method indicator as defined in Table <NUM>. Other bits of ext_mapping_idc are reserved for future use, for example to identify newer predictors, beyond the capabilities of the TPB predictor described herein. Note - If ext_mapping_idc & 0x1F is equal to <NUM>, decoder shall take BL signal as-is for reconstructed HDR signal. If the decoder doesn't recognize the ext_mapping_idc field, alternatively, it shall fall back to use the base layer mapping method as indicated by mapping_idc to reconstruct HDR signal. Note - If ext_mapping_idc & 0x1F is larger than <NUM> and smaller than <NUM> or equal to <NUM>, decoder shall fall back to use the base layer mapping method as indicated by mapping_idc to reconstruct HDR signal. Note - If ext_mapping_idc is larger than <NUM> and smaller than <NUM>, base layer inverse mapping shall be done using TPB prediction method, if the decoder supports the current TPB prediction type as indicated by ext_mapping_idc. If the decoder doesn't support the current TPB prediction type or doesn't recognize the ext_mapping_idc field at all, alternatively, it shall fall back to use the base layer mapping method as indicated by mapping_idc to reconstruct HDR signal. Note - If ext_mapping_idc is equal to <NUM>, there shall be a companion RPU with rpu_type = <NUM> to be transmitted immediately after an RPU with rpu_type = <NUM> for the current picture. If the decoder recognizes RPU with rpu_type = <NUM>, RPU parser shall extract the custom TPB prediction coefficients and use them in TPB prediction process.

To ensure backward compatibility, the mobile optimized profile uses the EL_bit_depth_minus8 field in the existing RPU syntax to signal the <NUM>-bit TPB ext_mapping_idc to the decoders. In EL_bit_depth_minus8, an unsigned <NUM>-bit field encapsulates the ext_mapping_idc and EL_bit_depth_minus8 values in different bit locations as shown in Table <NUM>.

Thus, for backward compatibility, the LSB <NUM>-bits carry the EL_bit_depth_minus8 value. Legacy devices discard the most significant <NUM>-bits and only parse the lower <NUM>-bits to fetch the value. The next <NUM>-bits from <NUM> to <NUM> contain the information that can be interpreted by the devices supporting the latest RPU syntax. The semantics of ext_mapping_idc are shown in Table <NUM>.

If ext_mapping_idc is larger than <NUM> and smaller than <NUM>, the TPB prediction method is used for base layer inverse mapping, when the decoder supports the current TPB prediction type as indicated by ext_mapping_idc. If the decoder doesn't support the current TPB prediction type or doesn't recognize the ext_mapping_idc field at all, it falls back to use the base layer mapping method as indicated by the mapping_idc syntax variable to reconstruct HDR signal. Without the need to go into details, in an embodiment, the mapping_idc syntax variable is a <NUM>-bit field that specifies a variety of legacy mapping options.

If ext_mapping_idc is equal to <NUM>, then there will be a companion RPU structure with rpu_type = <NUM> to be transmitted immediately after an RPU with rpu_type = <NUM> for the current picture. If the decoder recognizes RPU metadata with rpu_type = <NUM>, the RPU parser shall extract the custom TPB prediction coefficients and use them in the TPB prediction process. This process is summarized in <FIG>.

As depicted in <FIG>, given a video elementary stream (<NUM>), a decoder will parse metadata (<NUM>, <NUM>) and a base layer (BL) bitstream (<NUM>). Looking at the EL_bit_depth_minus8 parameter in the metadata, a legacy device will just extract the <NUM> least significant bits to read the bit-depth of enhancement-layer data (if any). On the other hand, newer devices will read the least <NUM> bits and can extract both the bit-depth of the enhancement-layer data (if any) and the value of a new syntax parameter with information related to TPB prediction (e.g., ext_mapping_idc).

Section "TPB Cross-color Channel Predictor" describes already the key RPU (type <NUM>) metadata, which includes TPB Parameters, TPB Coefficients, and Auxiliary data. In an embodiment, Table <NUM> describes an example of the RPU (type <NUM>) payload syntax.

The semantics of RPU data payload for rpu_type equal to <NUM> are defined below. tpb_coef_data_type[ y ][ x ] specifies the length of non-zero TPB coefficients used for the partition with vertical coordinate y and horizontal coordinate x. The data types are defined as below.

While example embodiments are provided for mobile applications, a persons skilled in the art would appreciate that the proposed methods and systems are applicable to non-mobile devices as well, such as set-top boxes, television sets, computers, gaming consoles, and other devices for the play-pack of video content.

Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control or execute instructions relating to image prediction techniques, such as those described herein. The computer and/or IC may compute, any of a variety of parameters or values that relate to the generation of image prediction techniques as described herein. The image and video dynamic range extension embodiments may be implemented in hardware, software, firmware and various combinations thereof.

Claim 1:
A method for reconstructing high-dynamic range images and video, the method comprising:
receiving an input image encoded in an input dynamic range;
receiving metadata for the input image to generate a mapping function to map the input image from the input dynamic range to an output image in an output dynamic range;
parsing a first metadata parameter to identify whether a second metadata parameter is present;
if the second metadata parameter is present, then
generating the mapping function using a non-backward compatible predictor, wherein the non-backward compatible predictor comprises a tensor-product B-spline (TPB) predictor, wherein generating the mapping function using a tensor-product B-spline (TPB) predictor comprises:
accessing TPB metadata within the metadata of the input image, wherein the TPB metadata comprise TPB-related prediction parameters, prediction coefficients, and auxiliary data; and
generating based on the TPB metadata a three dimensional look up table (3D-LUT) to map selected triplets of pixel values in the input dynamic range to corresponding output triplets of pixel values in the output dynamic range,
wherein generating the 3D-LUT involves, for each 3D-LUT index, generating respective basis functions by using the prediction parameters and computing a respective output pixel value by using the prediction coefficients based on the basis functions;
wherein generating the 3D-LUT involves first storing frequently used values of the basis functions in one or more 2D arrays for later computing output pixel values by using the prediction coefficients;
wherein the prediction parameters comprise:
a number of knot points (T) in each of three dimensions of a luma channel and two chroma channels; and
polynomial order (l) values in each of the three dimensions of the luma channel and the two chroma channels;
wherein the prediction coefficients comprise:
prediction coefficients for the luma channel and the two chroma channels; and
wherein the auxiliary data comprise:
a luma-channel basis flag and two chroma-channel basis flags for determining correlation among the prediction coefficients for the luma channel and the two chroma channels;
else generating the mapping function using a backward-compatible predictor supported by legacy devices; and
applying the mapping function to the input image to generate the output image,
wherein the input dynamic range comprises a standard dynamic range and the output dynamic range comprises a high dynamic range.