VIDEO COMPRESSION WITH COLOR SPACE SCALABILITY

An image decoder includes a base layer to decode at least a portion of an encoded video stream into a first image having a first image format. The image decoder can generate a color space prediction by scaling a color space of the first image from the first image format into a color space corresponding to a second image format. The image decoder includes an enhancement layer to decode the encoded video stream to generate a second image in the second image format based, at least in part, on the color space prediction.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENT

FIG. 1is a block diagram example of a video coding system100. The video coding system100can include a video encoder300to receive video streams, such as an Ultra High Definition Television (UHDTV) video stream102, standardized as BT.2020, and a BT.709 video stream104, and to generate an encoded video stream112based on the video streams. The video encoder300can transmit the encoded video stream112to a video decoder500. The video decoder500can decode the encoded video stream112to generate a decoded UHDTV video stream122and/or a decoded BT.709 video stream124.

The UHDTV video stream102can have a different resolution, different quantization bit-depth, and represent different color gamut compared to the BT.709 video stream104. For example, a UHDTV or BT.2020 video standard has a format recommendation that can support a 4k (3840×2160 pixels) or an 8k (7680×4320 pixels) resolution and a 10 or 12 bit quantization bit-depth. The BT.709 video standard has a format recommendation that can support a 2k (1920×1080 pixels) resolution and an 8 or 10 bit quantization bit-depth. The UHDTV format recommendation also can support a wider color gamut than the BT.709 format recommendation. Embodiments of the color gamut difference between the UHDTV video standard and the BT.709 video standard will be shown and described below in greater detail with reference toFIG. 2.

The video encoder300can include an enhancement layer encoder302and a base layer encoder304. The base layer encoder304can implement video encoding for High Definition (HD) content, for example, with a codec implementing a Moving Picture Experts Group (MPEG)-2 standard, or the like. The enhancement layer encoder302can implement video encoding for UHDTV content. In some embodiments, the enhancement layer encoder302can encode an UHDTV video frame by generating a prediction of at least a portion of the UHDTV image frame using a motion compensation prediction, an intra-frame prediction, and a scaled color prediction from a BT.709 image frame encoded in the base layer encoder302. The video encoder300can utilize the prediction to generate a prediction residue, for example, a difference between the prediction and the UHDTV image frame, and encode the prediction residue in the encoded video stream112.

In some embodiments, when the video encoder300utilizes a scaled color prediction from the BT.709 image frame, the video encoder300can transmit color prediction parameters114to the video decoder500. The color prediction parameters114can include parameters utilized by the video encoder300to generate the scaled color prediction. For example, the video encoder300can generate the scaled color prediction through an independent color channel prediction or an affine matrix-based color prediction, each having different parameters, such as a gain parameter per channel or a gain parameter and an offset parameter per channel. The color prediction parameters114can include parameters corresponding to the independent color channel prediction or the affine matrix-based color prediction utilized by the video encoder300. In some embodiments, the encoder300can include the color prediction parameters114in a normative portion of the encoded video stream112, for example, in a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), or another lower level section of the normative portion of the encoded video stream112. In some embodiments, the video encoder300can utilize default color prediction parameters114, which may be preset in the video decoder500, alleviating the video encoder300from having to transmit color prediction parameters114to the video decoder500. Embodiments of video encoder300will be described below in greater detail.

The video decoder500can include an enhancement layer decoder502and a base layer decoder504. The base layer decoder504can implement video decoding for High Definition (HD) content, for example, with a codec implementing a Moving Picture Experts Group (MPEG)-2 standard, or the like, and decode the encoded video stream112to generate a decoded BT.709 video stream124. The enhancement layer decoder502can implement video decoding for UHDTV content and decode the encoded video stream112to generate a decoded UHDTV video stream122.

In some embodiments, the enhancement layer decoder502can decode at least a portion of the encoded video stream112into the prediction residue of the UHDTV video frame. The enhancement layer decoder502can generate a same or a similar prediction of the UHDTV image frame that was generated by the video encoder300during the encoding process, and then combine the prediction with the prediction residue to generate the decoded UHDTV video stream122. The enhancement layer decoder502can generate the prediction of the UHDTV image frame through motion compensation prediction, intra-frame prediction, or scaled color prediction from a BT.709 image frame decoded in the base layer decoder504. Embodiments of video encoder400will be described below in greater detail.

AlthoughFIG. 1shows color prediction-based video coding of an UHDTV video stream and a BT.709 video stream with video encoder300and video decoder500, in some embodiments, any video streams representing different color gamuts can be encoded or decoded with color prediction-based video coding.

FIG. 2is an example graph200illustrating color gamuts supported in a BT.709 video standard and in a UHDTV video standard. Referring toFIG. 2, the graph200shows a two-dimensional representation of color gamuts in an International Commission on Illumination (CIE)1931chrominance xy diagram format. The graph200includes a standard observer color gamut210to represent a range of colors viewable by a standard human observer as determined by the CIE in1931. The graph200includes a UHDTV color gamut220to represent a range of colors supported the UHDTV video standard. The graph200includes a BT.709 color gamut230to represent a range of colors supported the BT.709 video standard, which is narrower than the UHDTV color gamut220. The graph also includes a point that represents the color white240, which is included in the standard observer color gamut210, the UHDTV color gamut220, and the BT.709 color gamut230.

FIGS. 3A and 3Band3C are block diagram examples of the video encoder300shown inFIG. 1. Referring toFIG. 3A, the video encoder300can include an enhancement layer encoder302and a base layer encoder304. The base layer encoder304can include a video input362to receive a BT.709 video stream104having HD image frames. The base layer encoder304can include an encoding prediction loop364to encode the BT.709 video stream104received from the video input362, and store the reconstructed frames of the BT.709 video stream in a reference buffer368. The reference buffer368can provide the reconstructed BT.709 image frames back to the encoding prediction loop364for use in encoding other portions of the same frame or other frames of the BT.709 video stream104. The reference buffer368can store the image frames encoded by the encoding prediction loop364. The base layer encoder304can include entropy encoding function366to perform entropy encoding operations on the encoded-version of the BT.709 video stream from the encoding prediction loop364and provide an entropy encoded stream to an output interface380.

The enhancement layer encoder302can include a video input310to receive a UHDTV video stream102having UHDTV image frames. The enhancement layer encoder302can generate a prediction of the UHDTV image frames and utilize the prediction to generate a prediction residue, for example, a difference between the prediction and the UHDTV image frames determined with a combination function315. In some embodiments, the combination function315can include weighting, such as linear weighting, to generate the prediction residue from the prediction of the UHDTV image frames. The enhancement layer encoder302can transform and quantize the prediction residue with a transform and quantize function320. An entropy encoding function330can encode the output of the transform and quantize function320, and provide an entropy encoded stream to the output interface380. The output interface380can multiplex the entropy encoded streams from the entropy encoding functions366and330to generate the encoded video stream112.

The enhancement layer encoder302can include a color space predictor400, a motion compensation prediction function354, and an intra predictor356, each of which can generate a prediction of the UHDTV image frames. The enhancement layer encoder302can include a prediction selection function350to select a prediction generated by the color space predictor400, the motion compensation prediction function354, and/or the intra predictor356to provide to the combination function315.

In some embodiments, the motion compensation prediction function354and the intra predictor356can generate their respective predictions based on UHDTV image frames having previously been encoded and decoded by the enhancement layer encoder302. For example, after a prediction residue has been transformed and quantized, the transform and quantize function320can provide the transformed and quantized prediction residue to a scaling and inverse transform function322, the result of which can be combined in a combination function325with the prediction utilized to generate the prediction residue and generate a decoded UHDTV image frame. The combination function325can provide the decoded UHDTV image frame to a deblocking function351, and the deblocking function351can store the decoded UHDTV image frame in a reference buffer340, which holds the decoded UHDTV image frame for use by the motion compensation prediction function354and the intra predictor356. In some embodiments, the deblocking function351can filter the decoded UHDTV image frame, for example, to smooth sharp edges in the image between macroblocks corresponding to the decoded UHDTV image frame.

The motion compensation prediction function354can receive one or more decoded UHDTV image frames from the reference buffer340. The motion compensation prediction function354can generate a prediction of a current UHDTV image frame based on image motion between the one or more decoded UHDTV image frames from the reference buffer340and the UHDTV image frame.

The intra predictor356can receive a first portion of a current UHDTV image frame from the reference buffer340. The intra predictor356can generate a prediction corresponding to a first portion of a current UHDTV image frame based on at least a second portion of the current UHDTV image frame having previously been encoded and decoded by the enhancement layer encoder302.

The color space predictor400can generate a prediction of the UHDTV image frames based on BT.709 image frames having previously been encoded by the base layer encoder304. In some embodiments, the reference buffer368in the base layer encoder304can provide the reconstructed BT.709 image frame to a resolution upscaling function370, which can scale the resolution of the reconstructed BT.709 image frame to a resolution that corresponds to the UHDTV video stream102. The resolution upscaling function370can provide an upscaled resolution version of the reconstructed BT.709 image frame to the color space predictor400. The color space predictor can generate a prediction of the UHDTV image frame based on the upscaled resolution version of the reconstructed BT.709 image frame. In some embodiments, the color space predictor400can scale a YUV color space of the upscaled resolution version of the reconstructed BT.709 image frame to correspond to the YUV representation supported by the UHDTV video stream102. In some embodiments, the upscaling and color prediction are done jointly. The reference buffer368in the base layer encoder304can provide reconstructed BT.709 images frames to the joint upscaler color predictor. The joint upscaler color predictor375generates an upscaled and color prediction of the UHDTV image frame. The combined upscaler and color prediction functions enable reduced complexity as well as avoiding loss of precision resulting from limited bit-depth between the separate upscaler and the color prediction modules.

There are several ways for the color space predictor400to scale the color space supported by BT.709 video coding standard to a color space supported by the UHDTV video stream102, such as independent channel prediction and affine mixed channel prediction. Independent channel prediction can include converting each portion of the YUV color space for the BT.709 image frame separately into the prediction of the UHDTV image frame. The Y portion or luminance can be scaled according to Equation 1:

The U portion or one of the chrominance portions can be scaled according to Equation 2:

The V portion or one of the chrominance portions can be scaled according to Equation 3:

The gain parameters g1, g2, and g3and the offset parameters o1, o2, and o3can be based on differences in the color space supported by the BT.709 video coding standard and the UHDTV video standard, and may vary depending on the content of the respective BT.709 image frame and UHDTV image frame. The enhancement layer encoder304can output the gain parameters g1, g2, and g3and the offset parameters o1, o2, and o3utilized by the color space predictor400to generate the prediction of the UHDTV image frame to the video decoder500as the color prediction parameters114, for example, via the output interface380.

In some embodiments, the independent channel prediction can include gain parameters g1, g2, and g3, and zero parameters. The Y portion or luminance can be scaled according to Equation 4:

The U portion or one of the chrominance portions can be scaled according to Equation 5:

The V portion or one of the chrominance portions can be scaled according to Equation 6:

The gain parameters g1, g2, and g3can be based on differences in the color space supported by the BT.709 video coding standard and the UHDTV video standard, and may vary depending on the content of the respective BT.709 image frame and UHDTV image frame. The enhancement layer encoder304can output the gain parameters g1, g2, and g3utilized by the color space predictor400to generate the prediction of the UHDTV image frame to the video decoder500as the color prediction parameters114, for example, via the output interface380. Since the video decoder500can be pre-loaded with the zero parameters, the video encoder300can generate and transmit fewer color prediction parameters114, for example, three instead of six, to the video decoder500.

In some embodiments, the zero parameters used in Equations 4-6 can be defined based on the bit-depth of the relevant color space and color channel. For example, in Table 1, the zero parameters can be defined as follows:

The affine mixed channel prediction can include converting the YUV color space for a BT.709 image frame by mixing the YUV channels of the BT.709 image frame to generate a prediction of the UHDTV image frame, for example, through a matrix multiplication function. In some embodiments, the color space of the BT.709 can be scaled according to Equation 7:

In some embodiments, the color space of the BT.709 can be scaled according to Equation 8:

The matrix parameters m11, m12, m13, m22, and m33and the offset parameters o1, o2, and o3can be based on the difference in color space supported by the BT.709 video coding standard and the UHDTV video standard, and may vary depending on the content of the respective BT.709 image frame and UHDTV image frame. The enhancement layer encoder304can output the matrix and offset parameters utilized by the color space predictor400to generate the prediction of the UHDTV image frame to the video decoder500as the color prediction parameters114, for example, via the output interface380.

By replacing the matrix parameters m21, m23, m31, and m32with zero, the luminance channel Y of the UHDTV image frame prediction can be mixed with the color channels U and V of the BT.709 image frame, but the color channels U and V of the UHDTV image frame prediction may not be mixed with the luminance channel Y of the BT.709 image frame. The selective channel mixing can allow for a more accurate prediction of the luminance channel UHDTV image frame prediction, while reducing a number of prediction parameters114to transmit to the video decoder500.

In some embodiments, the color space of the BT.709 can be scaled according to Equation 9:

The matrix parameters m11, m12, m13, m22, m23, m32, and m33and the offset parameters o1, o2, and o3can be based on the difference in color space supported by the BT.709 video standard and the UHDTV video standard, and may vary depending on the content of the respective BT.709 image frame and UHDTV image frame. The enhancement layer encoder304can output the matrix and offset parameters utilized by the color space predictor400to generate the prediction of the UHDTV image frame to the video decoder500as the color prediction parameters114, for example, via the output interface380.

By replacing the matrix parameters m21and m31with zero, the luminance channel Y of the UHDTV image frame prediction can be mixed with the color channels U and V of the BT.709 image frame. The U and V color channels of the UHDTV image frame prediction can be mixed with the U and V color channels of the BT.709 image frame, but not the luminance channel Y of the BT.709 image frame. The selective channel mixing can allow for a more accurate prediction of the luminance channel UHDTV image frame prediction, while reducing a number of prediction parameters114to transmit to the video decoder500.

The color space predictor400can generate the scaled color space predictions for the prediction selection function350on a per sequence (inter-frame), a per frame, or a per slice (intra-frame) basis, and the video encoder300can transmit the prediction parameter114corresponding to the scaled color space predictions on a per sequence (inter-frame), a per frame, or a per slice (intra-frame) basis. In some embodiments, the granularity for generating the scaled color space predictions can be preset or fixed in the color space predictor400or dynamically adjustable by the video encoder300based on encoding function or the content of the UHDTV image frames.

The video encoder300can transmit the color prediction parameters114in a normative portion of the encoded video stream112, for example, in a Sequence Parameter Set (SPS), a Picture Parameter Set (PPS), or another lower level section of the normative portion of the encoded video stream112. In some embodiments, the color prediction parameters114can be inserted into the encoded video stream112with a syntax that allows the video decoder500to identify that the color prediction parameters114are present in the encoded video stream112, to identify a precision or size of the parameters, such as a number of bits utilized to represent each parameter, and identify a type of color space prediction the color space predictor400of the video encoder300utilized to generate the color space prediction.

In some embodiments, the normative portion of the encoded video stream112can include a flag (use_color_space_prediction), for example, one or more bits, which can annunciate an inclusion of color space parameters114in the encoded video stream112. The normative portion of the encoded video stream112can include a size parameter (color_predictor_num_fraction_bits_minus—1), for example, one or more bits, which can identify a number of bits or precision utilized to represent each parameter. The normative portion of the encoded video stream112can include a predictor type parameter (color_predictor_idc), for example, one or more bits, which can identify a type of color space prediction utilized by the video encoder300to generate the color space prediction. The types of color space prediction can include independent channel prediction, affine prediction, their various implementations, or the like. The color prediction parameters114can include gain parameters, offset parameters, and/or matrix parameters depending on the type of prediction utilized by the video encoder300.

Referring toFIG. 3B, a video encoder301can be similar to video encoder300shown and described above inFIG. 3Awith the following differences. The video encoder301can switch the color space predictor400with the resolution upscaling function370. The color space predictor400can generate a prediction of the UHDTV image frames based on BT.709 image frames having previously been encoded by the base layer encoder304.

In some embodiments, the reference buffer368in the base layer encoder304can provide the encoded BT.709 image frame to the color space predictor400. The color space predictor can scale a YUV color space of the encoded BT.709 image frame to correspond to the YUV representation supported by the UHDTV video format. The color space predictor400can provide the color space prediction to a resolution upscaling function370, which can scale the resolution of the color space prediction of the encoded BT.709 image frame to a resolution that corresponds to the UHDTV video format. The resolution upscaling function370can provide a resolution upscaled color space prediction to the prediction selection function350.

FIG. 4is a block diagram example of the color space predictor400shown inFIG. 3A. Referring toFIG. 4, the color space predictor400can include a color space prediction control device410to receive a reconstructed BT.709 video frame402, for example, from a base layer encoder304via a resolution upscaling function370, and select a prediction type and timing for a generation for a color space prediction406. In some embodiments, the color space prediction control device410can pass the reconstructed BT.709 video frame402to at least one of an independent channel prediction function420, an affine prediction function430, or a cross-color prediction function440. Each of the prediction functions420,430, and440can generate a color space prediction of a UHDTV image frame (or portion thereof) from the reconstructed BT.709 video frame402, for example, by scaling the color space of a BT.709 image frame to a color space of the UHDTV image frame.

The independent color channel prediction function420can scale YUV components of the encoded BT.709 video stream402separately, for example, as shown above in Equations 1-6. The affine prediction function430can scale YUV components of the reconstructed BT.709 video frame402with a matrix multiplication, for example, as shown above in Equation 7. The cross-color prediction function440can scale YUV components of the encoded BT.709 video stream402with a modified matrix multiplication that can eliminate mixing of a Y component from the encoded BT.709 video stream402when generating the U and V components of the UHDTV image frame, for example, as shown above in Equations 8 or 9.

In some embodiments, the color space predictor400can include a selection device450to select an output from the independent color channel prediction function420, the affine prediction function430, and the cross-color prediction function440. The selection device450also can output the color prediction parameters114utilized to generate the color space prediction406. The color prediction control device410can control the timing of the generation of the color space prediction406and the type of operation performed to generate the color space prediction406, for example, by controlling the timing and output of the selection device450. In some embodiments, the color prediction control device410can control the timing of the generation of the color space prediction406and the type of operation performed to generate the color space prediction406by selectively providing the encoded BT.709 video stream402to at least one of the independent color channel prediction function420, the affine prediction function430, and the cross-color prediction function440.

FIGS. 5A and 5Band5C are block diagram examples of the video decoder500shown inFIG. 1. Referring toFIG. 5A, the video decoder can include an interface510to receive the encoded video stream112, for example, from a video encoder300. The interface510can demultiplex the encoded video stream112and provide encoded UHDTV image data to an enhancement layer decoder502of the video decoder500and provide encoded BT.709 image data to a base layer decoder504of the video decoder500. The base layer decoder504can include an entropy decoding function552and a decoding prediction loop554to decode encoded BT.709 image data received from the interface510, and store the decoded BT.709 video stream124in a reference buffer556. The reference buffer556can provide the decoded BT.709 video stream124back to the decoding prediction loop554for use in decoding other portions of the same frame or other frames of the encoded BT.709 image data. The base layer decoder504can output the decoded BT.709 video stream124. In some embodiments, the output from the decoding prediction loop554and input to the reference buffer556may be residual frame data rather than the reconstructed frame data.

The enhancement layer decoder502can include an entropy decoding function522, a inverse quantization function524, an inverse transform function526, and a combination function528to decode the encoded UHDTV image data received from the interface510. A deblocking function541can filter the decoded UHDTV image frame, for example, to smooth sharp edges in the image between macroblocks corresponding to the decoded UHDTV image frame, and store the decoded UHDTV video stream122in a reference buffer530. In some embodiments, the encoded UHDTV image data can correspond to a prediction residue, for example, a difference between a prediction and a UHDTV image frame as determined by the video encoder300. The enhancement layer decoder502can generate a prediction of the UHDTV image frame, and the combination function528can add the prediction of the of the UHDTV image frame to encoded UHDTV image data having undergone entropy decoding, inverse quantization, and an inverse transform to generate the decoded UHDTV video stream122. In some embodiments, the combination function528can include weighting, such as linear weighting, to generate the decoded UHDTV video stream122.

The enhancement layer decoder502can include a color space predictor600, a motion compensation prediction function542, and an intra predictor544, each of which can generate the prediction of the UHDTV image frame. The enhancement layer decoder502can include a prediction selection function540to select a prediction generated by the color space predictor600, the motion compensation prediction function542, and/or the intra predictor544to provide to the combination function528.

In some embodiments, the motion compensation prediction function542and the intra predictor544can generate their respective predictions based on UHDTV image frames having previously been decoded by the enhancement layer decoder502and stored in the reference buffer530. The motion compensation prediction function542can receive one or more decoded UHDTV image frames from the reference buffer530. The motion compensation prediction function542can generate a prediction of a current UHDTV image frame based on image motion between the one or more decoded UHDTV image frames from the reference buffer530and the UHDTV image frame.

The intra predictor544can receive a first portion of a current UHDTV image frame from the reference buffer530. The intra predictor544can generate a prediction corresponding to a first portion of a current UHDTV image frame based on at least a second portion of the current UHDTV image frame having previously been decoded by the enhancement layer decoder502.

The color space predictor600can generate a prediction of the UHDTV image frames based on BT.709 image frames decoded by the base layer decoder504. In some embodiments, the reference buffer556in the base layer decoder504can provide a portion of the decoded BT.709 video stream124to a resolution upscaling function570, which can scale the resolution of the encoded BT.709 image frame to a resolution that corresponds to the UHDTV video format. The resolution upscaling function570can provide an upscaled resolution version of the encoded BT.709 image frame to the color space predictor600. The color space predictor can generate a prediction of the UHDTV image frame based on the upscaled resolution version of the encoded BT.709 image frame. In some embodiments, the color space predictor600can scale a YUV color space of the upscaled resolution version of the encoded BT.709 image frame to correspond to the YUV representation supported by the UHDTV video format.

In some embodiments, the upscaling and color prediction are done jointly. The reference buffer556in the base layer decoder504can provide reconstructed BT.709 images frames to the joint upscaler color predictor575. The joint upscaler color predictor generates an upscaled and color prediction of the UHDTV image frame. The combined upscaler and color prediction functions enable reduced complexity as well as avoiding loss of precision resulting from limited bit-depth between the separate upscaler and the color prediction modules. An example of the combination of upscaling and color prediction may be defined by a sample set of equations. Conventional upsampling implemented by separable filter calculations followed by an independent color prediction. Example calculations are shown below in three steps by equations 10, 11 and 12.

The input samples xi,jare filtered in one direction by taps akto give intermediates yi,j. An offset, o1, is added and the result is right shifted by the value s1as in Equation 10:

The intermediate samples yi,jare then filtered by taps bkto give samples zi,jand a second offset, o2, is added and the result is right shifted by a second value, s2as in Equation 11:

The results of the upsampling process zi,jare then processed by the color predition to generate prediction samples pi,j. A gain is applied then an offset, o3, is added before a final shift by s3. The color prediction process described in Equation 12:

The complexity may be reduced by combining the color prediction calculation with the second separable filter calculation. The filter taps bkof Equation 11 are combined with the gain of Equation 12 to produce new taps ck=gain·bkthe shift values of Equations 11 and Equation 12 are combined to give a new shift value s4=s2+s3. The offset of Equation 12 is modified to o4=o3<<s2. The individual calculations of Equation 11 and Equation 12 are defined in a single result Equation 13:

The combined calculation of Equation 13 has the advantage compared to Equations 11 and Equation 12 of reducing computation by using a single shift rather than two separate shifts and reducing the number of multiplies by premultiplying the filter taps by the gain value.

In some embodiments, it may be desirable to implement the separable filter calculations with equal taps so that ak=bkin Equation 10 and Equation 11. Direct application of the combined upscaling and color prediction removes this equality of taps since the values bkare replaced with the combined values ckAn alternate embodiment will maintain this equality of taps. The gain is represented as a the square of a value r shifted by a factor e in the form gain=(r·r)<<e. Where the value r is represented with m bits.

The results of Equations 10 and Equation 13 may be replaced by the pair of Equation 14 and Equation 15:

The offsets and shifts used in Equation 15 and Equation 16 are derived from the values in Equations 10 and Equation 13 and the representation of the gain value as shown in Equation 16:

The filter calculations in Equation 15 and Equation 16 use equal tap values r·ak. The use of the exponent factor e allows large gain values to be approximated with small values of r by increasing the value of e.

The color space predictor600can operate similarly to the color space predictor400in the video encoder300, by scaling the color space supported by BT.709 video coding standard to a color space supported by the UHDTV video format, for example, with independent channel prediction, affine mixed channel prediction, or cross-color channel prediction. The color space predictor600, however, can select a type of color space prediction to generate based, at least in part, on the color prediction parameters114received from the video encoder300. The color prediction parameters114can explicitly identify a particular a type of color space prediction, or can implicitly identify the type of color space prediction, for example, by a quantity and/or arrangement of the color prediction parameters114.

As discussed above, in some embodiments, the normative portion of the encoded video stream112can include a flag (use_color_space_prediction), for example, one or more bits, which can annunciate an inclusion of color space parameters114in the encoded video stream112. The normative portion of the encoded video stream112can include a size parameter (color_predictor_num_fraction_bits_minus—1), for example, one or more bits, which can identify a number of bits or precision utilized to represent each parameter. The normative portion of the encoded video stream112can include a predictor type parameter (color_predictor_idc), for example, one or more bits, which can identify a type of color space prediction utilized by the video encoder300to generate the color space prediction. The types of color space prediction can include independent channel prediction, affine prediction, their various implementations, or the like. The color prediction parameters114can include gain parameters, offset parameters, and/or matrix parameters depending on the type of prediction utilized by the video encoder300.

The color space predictor600identify whether the video encoder300utilize color space prediction in generating then encoded video stream112based on the flag (use_color_space_prediction). When color prediction parameters114are present in the encoded video stream112, the color space predictor600can parse the color prediction parameters114to identify a type of color space prediction utilized by the video encoded based on the predictor type parameter (color_predictor_idc), and a size or precision of the parameters (color_predictor_num_fraction_bits_minus—1), and locate the color space parameters to utilize to generate a color space prediction.

For example, the video decoder500can determine whether the color prediction parameters114are present in the encoded video stream112and parse the color prediction parameters114based on the following example code in Table 2:

The example code in Table 2 can allow the video decoder500to identify whether color prediction parameters114are present in the encoded video stream112based on the use_color_space prediction flag. The video decoder500can identify the precision or size of the color space parameters based on the size parameter (color_predictor_num_fraction_bits_minus—1), and can identify a type of color space prediction utilized by the video encoder300based on the type parameter (color_predictor_idc). The example code in Table 2 can allow the video decoder500to parse the color space parameters from the encoded video stream112based on the identified size of the color space parameters and the identified type color space prediction utilized by the video encoder300, which can identify the number, semantics, and location of the color space parameters. Although the example code in Table 2 shows the affine prediction including 9 matrix parameters and 3 offset parameters, in some embodiments, the color prediction parameters114can include fewer matrix and/or offset parameters, for example, when the matrix parameters are zero, and the example code can be modified to parse the color prediction parameters114accordingly.

An alternate method for signaling the color prediction parameters is described here. The structure of the Picture Parameter Set (PPS) of HEVC is shown in the table below:

Additional fields to carry color prediction data are added when the pps_extension_flag is set.

In extension data signal the following:

A flag to use color prediction on the current picture

Indicator of color prediction model used to signal gain and offset values.

For each model the following values are signaled or derived: number_gain_fraction_bits, gain[ ] and offset[ ] values for each color component.

Bit Increment (BI) model: the number of fraction bits is zero, the gain values are equal and based on the difference in bit-depth between base and enhancement layer i.e. 1<<(bit_depth_EL-bit-depth_BL), all offset values are zero.

Fixed Gain Offset model: an index is signaled indicating the use of a set of parameters signaled previously for instance out of band or through a predefined table of parameter values. This index indicates a previously define set of values including: number of fraction bits, gain and offset values for all components. These values are not signaled but reference to a predefined set. If only a single set of parameters is predefined, an index is not sent and this set is used when the Fixed Gain Offset model is used.

Picture Adaptive Gain Offset Offset model: parameter values are signaled in the bitstream through the following fields. Number of fraction bits is signaled as an integer in a predefined range i.e. 0-5. For each channel gain and offset values are signaled as integers. An optional method is to signal the difference between the Fixed Gain Offset model and the parameter values of the Dynamic Gain Offset model.

Each layer will may have independently specified color space for instance using the HEVC Video Usability Information (VUI) with colour_description_present_flag indicating the presence of colour information. As an example, separate VUI fields can be specified for each layer through different Sequence Parameter Sets.

The color space predictor600can generate color space predictions for the prediction selection function540on a per sequence (inter-frame), a per frame, or a per slice (intra-frame) basis. In some embodiments, the color space predictor600can generate the color space predictions with a fixed or preset timing or dynamically in response to a reception of the color prediction parameters114from the video encoder300.

Referring toFIG. 5B, a video decoder501can be similar to video decoder500shown and described above inFIG. 5Awith the following differences. The video decoder501can switch the color space predictor600with the resolution upscaling function570. The color space predictor600can generate a prediction of the UHDTV image frames based on portions of the decoded BT.709 video stream124from the base layer decoder504.

In some embodiments, the reference buffer556in the base layer decoder504can provide the portions of the decoded BT.709 video stream124to the color space predictor600. The color space predictor600can scale a YUV color space of the portions of the decoded BT.709 video stream124to correspond to the YUV representation supported by the UHDTV video standard. The color space predictor600can provide the color space prediction to a resolution upscaling function570, which can scale the resolution of the color space prediction to a resolution that corresponds to the UHDTV video standard. The resolution upscaling function570can provide a resolution upscaled color space prediction to the prediction selection function540.

FIG. 6is a block diagram example of a color space predictor600shown inFIG. 5A. Referring toFIG. 6, the color space predictor600can include a color space prediction control device610to receive the decoded BT.709 video stream122, for example, from a base layer decoder504via a resolution upscaling function570, and select a prediction type and timing for a generation for a color space prediction606. The color space predictor600can select a type of color space prediction to generate based, at least in part, on the color prediction parameters114received from the video encoder300. The color prediction parameters114can explicitly identify a particular a type of color space prediction, or can implicitly identify the type of color space prediction, for example, by a quantity and/or arrangement of the color prediction parameters114. In some embodiments, the color space prediction control device610can pass the decoded BT.709 video stream122and color prediction parameters114to at least one of an independent channel prediction function620, an affine prediction function630, or a cross-color prediction function640. Each of the prediction functions620,630, and640can generate a color space prediction of a UHDTV image frame (or portion thereof) from the decoded BT.709 video stream122, for example, by scaling the color space of a BT.709 image frame to a color space of the UHDTV image frame based on the color space parameters114.

The independent color channel prediction function620can scale YUV components of the decoded BT.709 video stream122separately, for example, as shown above in Equations 1-6. The affine prediction function630can scale YUV components of the decoded BT.709 video stream122with a matrix multiplication, for example, as shown above in Equation 7. The cross-color prediction function640can scale YUV components of the decoded BT.709 video stream122with a modified matrix multiplication that can eliminate mixing of a Y component from the decoded BT.709 video stream122when generating the U and V components of the UHDTV image frame, for example, as shown above in Equations 8 or 9.

In some embodiments, the color space predictor600can include a selection device650to select an output from the independent color channel prediction function620, the affine prediction function630, and the cross-color prediction function640. The color prediction control device610can control the timing of the generation of the color space prediction606and the type of operation performed to generate the color space prediction606, for example, by controlling the timing and output of the selection device650. In some embodiments, the color prediction control device610can control the timing of the generation of the color space prediction606and the type of operation performed to generate the color space prediction606by selectively providing the decoded BT.709 video stream122to at least one of the independent color channel prediction function620, the affine prediction function630, and the cross-color prediction function640.

FIG. 7is an example operational flowchart for color space prediction in the video encoder300. Referring toFIG. 7, at a first block710, the video encoder300can encode a first image having a first image format. In some embodiments, the first image format can correspond to a BT.709 video standard and the video encoder300can include a base layer to encode BT.709 image frames.

At a block720, the video encoder300can scale a color space of the first image from the first image format into a color space corresponding to a second image format. In some embodiments, the video encoder300can scale the color space between the BT.709 video standard and an Ultra High Definition Television (UHDTV) video standard corresponding to the second image format.

There are several ways for the video encoder300to scale the color space supported by BT.709 video coding standard to a color space supported by the UHDTV video format, such as independent channel prediction and affine mixed channel prediction. For example, the independent color channel prediction can scale YUV components of encoded BT.709 image frames separately, for example, as shown above in Equations 1-6. The affine mixed channel prediction can scale YUV components of the encoded BT.709 image frames with a matrix multiplication, for example, as shown above in Equations 7-9.

In some embodiments, the video encoder300can scale a resolution of the first image from the first image format into a resolution corresponding to the second image format. For example, the UHDTV video standard can support a 4k (3840×2160 pixels) or an 8k (7680×4320 pixels) resolution and a 10 or 12 bit quantization bit-depth. The BT.709 video standard can support a 2k (1920×1080 pixels) resolution and an 8 or 10 bit quantization bit-depth. The video encoder300can scale the encoded first image from a resolution corresponding to the BT.709 video standard into a resolution corresponding to the UHDTV video standard.

At a block730, the video encoder300can generate a color space prediction based, at least in part, on the scaled color space of the first image. The color space prediction can be a prediction of a UHDTV image frame (or portion thereof) from a color space of a corresponding encoded BT.709 image frame. In some embodiments, the video encoder300can generate the color space prediction based, at least in part, on the scaled resolution of the first image.

At a block740, the video encoder300can encode a second image having the second image format based, at least in part, on the color space prediction. The video encoder300can output the encoded second image and color prediction parameters utilized to scale the color space of the first image to a video decoder.

FIG. 8is an example operational flowchart for color space prediction in the video decoder500. Referring toFIG. 8, at a first block810, the video decoder500can decode an encoded video stream to generate a first image having a first image format. In some embodiments, the first image format can correspond to a BT.709 video standard and the video decoder500can include a base layer to decode BT.709 image frames.

At a block820, the video decoder500can scale a color space of the first image corresponding to the first image format into a color space corresponding to a second image format. In some embodiments, the video decoder500can scale the color space between the BT.709 video standard and an Ultra High Definition Television (UHDTV) video standard corresponding to the second image format.

There are several ways for the video decoder500to scale the color space supported by BT.709 video coding standard to a color space supported by the UHDTV video standard, such as independent channel prediction and affine mixed channel prediction. For example, the independent color channel prediction can scale YUV components of the encoded BT.709 image frames separately, for example, as shown above in Equations 1-6. The affine mixed channel prediction can scale YUV components of the encoded BT.709 image frames with a matrix multiplication, for example, as shown above in Equations 7-9.

The video decoder500can select a type of color space scaling to perform, such as independent channel prediction or one of the varieties of affine mixed channel prediction based on channel prediction parameters the video decoder500receives from the video encoder300. In some embodiments, the video decoder500can perform a default or preset color space scaling of the decoded BT.709 image frames.

In some embodiments, the video decoder500can scale a resolution of the first image from the first image format into a resolution corresponding to the second image format. For example, the UHDTV video standard can support a 4k (3840×2160 pixels) or an 8k (7680×4320 pixels) resolution and a 10 or 12 bit quantization bit-depth. The BT.709 video standard can support a 2k (1920×1080 pixels) resolution and an 8 or 10 bit quantization bit-depth. The video decoder500can scale the decoded first image from a resolution corresponding to the BT.709 video standard into a resolution corresponding to the UHDTV video standard.

At a block830, the video decoder500can generate a color space prediction based, at least in part, on the scaled color space of the first image. The color space prediction can be a prediction of a UHDTV image frame (or portion thereof) from a color space of a corresponding decoded BT.709 image frame. In some embodiments, the video decoder500can generate the color space prediction based, at least in part, on the scaled resolution of the first image.

At a block840, the video decoder500can decode the encoded video stream into a second image having the second image format based, at least in part, on the color space prediction. In some embodiments, the video decoder500can utilize the color space prediction to combine with a portion of the encoded video stream corresponding to a prediction residue from the video encoder300. The combination of the color space prediction and the decoded prediction residue can correspond to a decoded UHDTV image frame or portion thereof.

FIG. 9is another example operational flowchart for color space prediction in the video decoder500. Referring toFIG. 9, at a first block910, the video decoder500can decode at least a portion of an encoded video stream to generate a first residual frame having a first format. The first residual frame can be a frame of data corresponding to a difference between two image frames. In some embodiments, the first format can correspond to a BT.709 video standard and the video decoder500can include a base layer to decode BT.709 image frames.

At a block920, the video decoder500can scale a color space of the first residual frame corresponding to the first format into a color space corresponding to a second format. In some embodiments, the video decoder500can scale the color space between the BT.709 video standard and an Ultra High Definition Television (UHDTV) video standard corresponding to the second format.

There are several ways for the video decoder500to scale the color space supported by BT.709 video coding standard to a color space supported by the UHDTV video standard, such as independent channel prediction and affine mixed channel prediction. For example, the independent color channel prediction can scale YUV components of the encoded BT.709 image frames separately, for example, as shown above in Equations 1-6. The affine mixed channel prediction can scale YUV components of the encoded BT.709 image frames with a matrix multiplication, for example, as shown above in Equations 7-9.

The video decoder500can select a type of color space scaling to perform, such as independent channel prediction or one of the varieties of affine mixed channel prediction based on channel prediction parameters the video decoder500receives from the video encoder300. In some embodiments, the video decoder500can perform a default or preset color space scaling of the decoded BT.709 image frames.

In some embodiments, the video decoder500can scale a resolution of the first residual frame from the first format into a resolution corresponding to the second format. For example, the UHDTV video standard can support a 4k (3840×2160 pixels) or an 8k (7680×4320 pixels) resolution and a 10 or 12 bit quantization bit-depth. The BT.709 video standard can support a 2k (1920×1080 pixels) resolution and an 8 or 10 bit quantization bit-depth. The video decoder500can scale the decoded first residual frame from a resolution corresponding to the BT.709 video standard into a resolution corresponding to the UHDTV video standard.

At a block930, the video decoder500can generate a color space prediction based, at least in part, on the scaled color space of the first residual frame. The color space prediction can be a prediction of a UHDTV image frame (or portion thereof) from a color space of a corresponding decoded BT.709 image frame. In some embodiments, the video decoder500can generate the color space prediction based, at least in part, on the scaled resolution of the first residual frame.

At a block940, the video decoder500can decode the encoded video stream into a second image having the second format based, at least in part, on the color space prediction. In some embodiments, the video decoder500can utilize the color space prediction to combine with a portion of the encoded video stream corresponding to a prediction residue from the video encoder300. The combination of the color space prediction and the decoded prediction residue can correspond to a decoded UHDTV image frame or portion thereof.

The system and apparatus described above may use dedicated processor systems, micro controllers, programmable logic devices, microprocessors, or any combination thereof, to perform some or all of the operations described herein. Some of the operations described above may be implemented in software and other operations may be implemented in hardware. Any of the operations, processes, and/or methods described herein may be performed by an apparatus, a device, and/or a system substantially similar to those as described herein and with reference to the illustrated figures.

The processing device may execute instructions or “code” stored in memory. The memory may store data as well. The processing device may include, but may not be limited to, an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, or the like. The processing device may be part of an integrated control system or system manager, or may be provided as a portable electronic device configured to interface with a networked system either locally or remotely via wireless transmission.

The processor memory may be integrated together with the processing device, for example RAM or FLASH memory disposed within an integrated circuit microprocessor or the like. In other examples, the memory may comprise an independent device, such as an external disk drive, a storage array, a portable FLASH key fob, or the like. The memory and processing device may be operatively coupled together, or in communication with each other, for example by an I/O port, a network connection, or the like, and the processing device may read a file stored on the memory. Associated memory may be “read only” by design (ROM) by virtue of permission settings, or not. Other examples of memory may include, but may not be limited to, WORM, EPROM, EEPROM, FLASH, or the like, which may be implemented in solid state semiconductor devices. Other memories may comprise moving parts, such as a known rotating disk drive. All such memories may be “machine-readable” and may be readable by a processing device.

Operating instructions or commands may be implemented or embodied in tangible forms of stored computer software (also known as “computer program” or “code”). Programs, or code, may be stored in a digital memory and may be read by the processing device. “Computer-readable storage medium” (or alternatively, “machine-readable storage medium”) may include all of the foregoing types of memory, as well as new technologies of the future, as long as the memory may be capable of storing digital information in the nature of a computer program or other data, at least temporarily, and as long at the stored information may be “read” by an appropriate processing device. The term “computer-readable” may not be limited to the historical usage of “computer” to imply a complete mainframe, mini-computer, desktop or even laptop computer. Rather, “computer-readable” may comprise storage medium that may be readable by a processor, a processing device, or any computing system. Such media may be any available media that may be locally and/or remotely accessible by a computer or a processor, and may include volatile and non-volatile media, and removable and non-removable media, or any combination thereof.

A program stored in a computer-readable storage medium may comprise a computer program product. For example, a storage medium may be used as a convenient means to store or transport a computer program. For the sake of convenience, the operations may be described as various interconnected or coupled functional blocks or diagrams. However, there may be cases where these functional blocks or diagrams may be equivalently aggregated into a single logic device, program or operation with unclear boundaries.

One of skill in the art will recognize that the concepts taught herein can be tailored to a particular application in many other ways. In particular, those skilled in the art will recognize that the illustrated examples are but one of many alternative implementations that will become apparent upon reading this disclosure.

Although the specification may refer to “an”, “one”, “another”, or “some” example(s) in several locations, this does not necessarily mean that each such reference is to the same example(s), or that the feature only applies to a single example.