Techniques for advanced chroma processing

Image and video processing techniques are disclosed for processing components of a color space individually by determining limits for each component based on the relationship between each component in a color space. These limits may then be used to clip each component such that the component values are within the determined range for that component. In this manner, more efficient processing of images and/or video may be achieved.

BACKGROUND

The present disclosure relates to image and video processing and, in particular, to improved image and video processing by using the characteristics of a color space.

Many modern electronic devices support exchange of video between them. In many applications, a first device captures video locally and codes it for transmission to a second device. The second device may decode the coded video and display it locally. The first device may perform pre-processing operations upon the source video to condition it for coding and/or transmission. Typical pre-processing operations include color space conversion, resizing of video, frame rate conversion of video and/or video filtering operations among others. Several coding protocols have been defined to support video coding and decoding operations. They include, for example, the MPEG-2, MPEG-4, H.263, H.264 and/or HEVC coding protocols.

The majority of digital image and video applications employ a Y′CbCr color space encoding for the representation of images, given its ability to better take into account human visual perception than other color spaces such as RGB or XYZ. Y′CbCr, also called YCbCr, Y′UV, or YUV, comprises of a luma (Y′) component, and two chroma/color components (Cb and Cr) for each pixel. The prime (′) symbol indicates the application of a transfer function on the original linear light R, G, and B signals, allowing for perceptual quantization. However, a number of other color spaces may also be used.

The components of a color space share certain relationships with each other. For example, in a YCrCb color space, the Y component may be calculated given the R, G and B components of an input signal as long as general characteristics of the video signal are known. For example, general characteristics of the video signal may include the color primaries being used, e.g. whether the signal uses BT.2020, P3D65, or BT.709 color primaries, the chroma sampling (phase) location, the color difference processes (or as are commonly called in some specifications the matrix coefficients), and the transfer functions applied to the original linear light signals, among others. Similarly, Cr and Cb components may be calculated using the Y and R, G, and B values along with the general characteristics of the video signal. Given the relationship of Cb or Cr with Y, as well as of Y with B and R respectively, limits on the value of Cb and Cr are impacted by the value of Y. If Y is known, then Cb and/or Cr are only allowed to be within a particular range of values.

However, existing algorithms tend to process each component independently, without taking into account the limitations in value that may be imposed upon certain components by previously calculated components. For example, during an encoding operation (assuming input data is provided in an N-bit integer representation, e.g. N=8) values commonly are clipped independently to stay within the valid limits of the N-bit representation, i.e. from a value of 0 up to (2^N)−1. Further, if an encoder determines that image data corresponds to a particular limited representation, such as the standard/limited representation used for TV applications, additional clipping within that range may be performed. However, existing algorithms perform additional clipping only when the signal is fully converted back to an RGB representation. This may result in accumulation of out of range sample values, and thus introduce artifacts in the final image representation.

The inventors perceive a need for a chroma processing scheme that can operate on each component in a color space individually, and operate on subsequent components with respect to previously ascertained components, taking into account the relationships between the components to allow for more efficient processing.

DETAILED DESCRIPTION

Embodiments of the present disclosure provide techniques for processing individual components of a color space with respect to previously processed components of the color space based on the relationships between each component. Thus, processing of a luma component may allow a range of values for each of a pair of chroma components to be determined. A processing operation may be performed upon a first chroma component to obtain a set of values for the first component. Subsequently, values from the set of values for the first component that are outside the determined range of values for the first component may be clipped. A range of limits for a second color component may be determined based on the clipped set of values for the first component and/or the processed luma component. A processing operation may be performed upon the second chroma component to obtain a set of values for the second component. Subsequently, values from the set of values for the second component that are outside the determined range of values for the second component may be clipped. Clipping each component in this manner may prevent the propagation of an unclipped signal through each processing stage which could result in large errors and the presence of artifacts in the image signal. In addition, the number of bits required to represent each component may be reduced as the bit depth required to represent each component may be modified based on the determined range of values for each component.

The principles of the present disclosure find application in applications that perform exchange of coded image and video data. Examples of such applications include but are not limited to video/image encoding, format conversion (i.e. conversion from a 4:2:0 or 4:2:2 YCbCr representation to a 4:4:4 representation), image scaling, spatio-temporal filtering such as denoising, and frame rate conversion among others. One such application is illustrated inFIG. 1, where a system100includes a pair of terminals110,120provided in communication by a network130. The terminals110,120may support either unidirectional or bidirectional exchange of coded video data. For bidirectional video exchange, each of the terminals110,120may capture video data at a local location and code the video data for transmission to the other terminal via the network130. Each terminal110,120may receive the coded video data of the other terminal from the network130, decode the coded data and display the recovered video data locally. For unidirectional exchange of video, only one of the terminals (say, terminal110) would capture video locally and code it for transmission to the other terminal120. The second terminal120would decode the coded video and display it locally.

InFIG. 1, the terminals110,120are illustrated as smart phones but the principles of the present disclosure are not so limited. Embodiments of the present disclosure find application with personal computers (both desktop and laptop computers), tablet computers, computer servers, media players and/or dedicated video conferencing equipment. The network130represents any number of networks that convey coded video data between the terminals110,120, including for example wire-line and/or wireless communication networks. The communication network130may exchange data in circuit-switched and/or packet-switched channels. Representative networks include telecommunications networks, local area networks, wide area networks and/or the Internet. For the purposes of the present discussion, the architecture and topology of the network130are immaterial to the operation of the present disclosure unless explained herein.

FIG. 2illustrates a simplified functional block diagram of functional units within terminals210,250to support video exchange, according to an embodiment of the present disclosure. The terminal210, for example, may include a video source215, a pre-processor220, a video coder225and a transmitter230, all operating under control of a controller235. The video source215may provide source video data representing the video content to be coded and transmitted to the second terminal250. The pre-processor220may perform signal conditioning operations to tailor the source video data for coding by the video coder225. The video coder225may code the source video data according to predictive coding techniques to reduce the video's bandwidth. The transmitter230may format coded video data from the video coder225for transmission to the second terminal250via a channel240.

The video source215may be represented by a camera system that may include an image sensor and related circuitry to generate video data representing locally-captured image content. In other embodiments, however, the video source215may be represented by a storage device that stores video data authored from other sources (for example, computer graphics or the like). Alternatively, the source video may include synthetically-generated image content authored by an application or other processes that executes on the terminal210. Source video may be output from the video source in a first format (such as RGB).

The pre-processor220may perform operations upon the source video to condition it for coding and/or transmission. Typical pre-processing operations include color space conversion, resizing of video, frame rate conversion of video and/or video filtering operations. Color space conversion may convert the source video from its first format to a second format on which the video coder225operates (such as Y′CrCb). Resizing operations may enlarge or reduce the size of video frames in the source video. Frame rate conversions may increase or decrease the frame rate of the source video. Filtering operations may include a variety of processing operations to either increase or decrease spatial and/or temporal smoothness of content in the source video. While the present discussion focuses on color space control operations of the pre-processor220, in practice, such operations may be performed cooperatively with other pre-processing operations as may be desired to achieve given application needs. Moreover, the types of pre-processing operations applied to source video may vary dynamically based on operating conditions of the terminal210, including availability of processing resources at the terminal210, conditions of a channel240and any operating conditions of a far end terminal250that are reported to the terminal210. The pre-processor220may provide pre-processed video to the video coder225.

The video coder225may code the video data presented to it by the pre-processor220. The video coder225may exploit spatial and/or temporal redundancies in the input video by coding the input video according to motion-compensated prediction. Such processes typically involve coding content of a new frame with reference to content of previously-coded reference frames. When a prediction is made between content of a new frame and a previously-coded reference frame, the video coder225may provide prediction references (which may be explicit or implicit) in the coder's output, which allows a decoder to invert the prediction operations during decoding. Prediction often operates on spatial areas of an input frame (called, “pixel blocks” herein), on a pixel-block-by-pixel-block basis. Such prediction references often include motion vectors, which identify spatial translations of image content between a frame being coded and a reference frame. The video coder225also may include a local decoder (not shown) to decode and store coded video data of reference frames for use in subsequent prediction operations. Video coding operations may occur according to predetermined coding protocols such as the MPEG-2, MPEG-4, H.263, H.264 and/or HEVC coding protocols.

The transmitter230may format coded video data for transmission in a channel240. The channel240itself may operate according to predetermined communication protocols. The transmitter230may format the coded video data according to the protocols that are appropriate for the channel240and may output the coded video data to the channel240.

The second terminal250may include a receiver255, video decoder260, post-processor265and video sink270. The receiver255may receive data from the channel240and parse the channel data into various data stream(s), including a stream of coded video. The video decoder260may decode the coded video data, inverting coding processes performed by the video coder225, to recover video data therefrom. The post-processor265may perform signal conditioning operations on recovered video data from the video coder to improve video quality prior to rendering. The video sink270may consume video data output by the post-processor265, typically, by displaying it on a display of the terminal250.

As noted, the video decoder260may decode the video data by inverting coding processes applied by the video coder225. The video decoder260may interpret coded video to identify prediction references therein, then apply them during coding operations. Thus, when the video coder225codes a given pixel block of a new frame using a reference frame as a source of prediction, the video decoder260may use the prediction reference data to identify the same content of the reference frame that video coder225used and perform its own prediction operation. The video decoder260may output recovered video to the pre-processor. The video decoder260also may store video of recovered reference frames for use in subsequent prediction operations. As with the video coder225, video decoding operations may occur according predetermined coding protocols such as the MPEG-2, MPEG-4, H.263, H.264 and/or HEVC coding protocols.

The post-processor265may perform additional video processing to condition the recovered video data for rendering, commonly at a display device. Typical post-processing operations may include color space conversions, filtering, resizing and/or frame rate conversions. Color space conversion may involve converting recovered video data from a format used by the video decoder260(such as Y′CrCb) to an output format appropriate for the video sink270; the output format may but need not be the same format as the video supplied by the video source215. Common filtering operations include deblocking filters, edge detection filters, ringing filters and the like. Resizing operations may enlarge or reduce the size of video frames in the recovered video. Frame rate conversions may increase or decrease the frame rate of the recovered video. The post-processor265may output recovered video sequence to video sink270.

The video sink270may consume recovered video data output by the post-processor265. Most often, the video sink270may be represented by a display that renders the recovered video data for view on the terminal250. Video sinks270may include storage devices that store recovered video data later use by the terminal250, such as by later viewing or later transmission to other devices (not shown).

The functional blocks illustrated inFIG. 2support video coding and decoding in one direction only. For bidirectional communication, terminal250may include functional blocks (not shown) for video coding of locally-captured video and terminal210may include functional blocks (also not shown) for decoding of coded video data received from the terminal250via the network240. Although it is envisioned that the principles of the present disclosure may be applied at coders resident at either terminal210,250, the functional blocks provided for the terminal250to terminal210coding direction are omitted fromFIG. 2merely to simplify the present discussion.

FIG. 3illustrates a functional block diagram of a video coder300according to an embodiment of the present disclosure. The video coder300may include a Y′ coder310, a Cr coder320, a Cb coder330, a Min/Max Estimator340and clipping units342,344. The video coder300may accept input video data as component signals Y′, Cr, and Cb, which are input to their respective coders310-330. The coders310-330may output coded video data to the transmitter, which may be assembled into a coded video stream that conforms to syntax requirements of the governing coding standard.

The Min/Max Estimator340may estimate whether to clip data output from one or both of the Cr and Cb coders320,330based on the coded Y′ data output from the Y′ coder310. In an embodiment, Min/Max Estimator340may determine the maximum and minimum values of one or both of Cr and Cb based on the coded Y′ data output from the Y′ coder. The Min/Max Estimator340may then determine whether to clip data output from one or both of the Cr and Cb coders based on a comparison of such data with the determined maximum and minimum values for each respectively. For example, for each component, the Min/Max Estimator340may determine that all values outside the range established by the maximum and minimum values for that component must be clipped. In an embodiment, the Min/Max Estimator340may determine that only those values that are beyond a threshold distance of the range established by the maximum and minimum values for that component must be clipped. In an embodiment, the Min/Max Estimator340may determine the maximum and minimum values for one of the color components (for example, Cr) based on the coded Y′ data. The Min/Max estimator340may determine which values of the coded Cr must be clipped based on the maximum and minimum values and send a corresponding clip signal to clip unit342. The Min/max estimator340may then consider the coded video data for the Cr component that is within the determined Cr maximum and minimum values in addition to coded Y′ data when determining whether to clip coded video data of the other color component (Cb, in the above example). The Min/Max Estimator340may output clip control signals to the clip units342,344, which modify coded Cr and/or Cb data as directed.

In an embodiment, Min/Max Estimator340may adjust the determined maximum/minimum values for one or more components to compensate for effects that can interfere with the encoder output such as quantization noise or other sources of interference. Min/Max Estimator340may allow an additional offset to expand the allowed range of values for one or more particular components beyond the determined maximum and minimum values for those components. The offset may be based on the amount and/or level of effects affecting the encoder output.

In an embodiment, clip units342and344may adjust the number of bits required to represent their respective components based on the determined maximum and minimum values of one or both of Cr and Cb. Instead of allocating a fixed number of bits per component value, Clip units342and344may adjust the number of bits based on the determined maximum and minimum values for one or both of Cr and Cb. For example, if the value of the Y′ component is calculated as zero, then Clip units342and344may adjust the bit representation such that no additional information is be signaled, since in this case Cb and Cr values are implied to be zero as well. Similarly, a decoder, after decoding a Y′ value of 0, may determine that no additional Cb and Cr data is present in the bit stream. In an embodiment, given a value for Y′, an encoder and/or decoder may compute the bitdepth required to cover all possible Cb values, whereas similarly given both Y′ and Cb an encoder and/or decoder may adjust the number of bits allocated for Cr.

The majority of digital image and video applications employ a Y′CbCr color space encoding for the representation of images, given its ability to accommodate human visual perception better than other color spaces such as RGB or XYZ. Y′CbCr, also called YCbCr, Y′UV, or YUV, comprises of a luma (Y′) component, and two chroma/color components (Cb and Cr). In its most common form, the Y′ component may be an approximation of the luminance/brightness information of the image (Y) of the CIE 1931 XYZ color space.

Therefore, in an embodiment, the video coder300may further include a format conversion unit305. The format conversion unit305may convert images received in an RGB format, for example, into a Y′CbCr format. The format conversion unit305may compute Y′ as:
Y′=wYRR′+wYGG′+wYBB′
where the prime (′) symbol indicates the application of a transfer function on the original linear light R, G, and B signals. The format converter305may apply the transfer function onto the original linear light signals to enable the use of perceptual quantization. This may allow for a more efficient conversion from a floating point to a fixed precision representation for the image signal. The wYR, wYG, and wYBweights may correspond to the constant weights used for the conversion of the linear light R, G, and B quantities back to the CIE 1931 Y component. The format converter305may specify the weights according to the color gamut limitations of the current RGB color space compared to the overall XYZ color space. Also wYR+wYG+wYB=1. In general, conversion from RGB to XYZ is performed using a 3×3 matrix conversion of the form:

[XYZ]=[aXRaXGaXBaYRaYGaYBaZRaZGaZB]⁡[RGB]
with the a weights relating to the gamut limitations of the RGB color space in question. Common RGB color spaces currently in use include the ITU-R BT.601, BT.709 (sRGB), BT.2020, Adobe RGB, scRGB, CIE RGB, P3DCI, and P3D65 color spaces among others.

Given a particular RGB color space, the format converter305may calculate the Cb and Cr components in the YCbCr as:

In an embodiment, the format converter305may compute values of alpha and beta in such a way so as to guarantee that Cb and Cr are always within the range of [−0.5, 0.5]. Cb may be equal to −0.5 when B′ is equal to 0 and G′=R′=1, and Cb may be equal to 0.5 B′ when is equal to 1, and G′=R′=0. Similarly, Cr may be equal to −0.5 when R′ is equal to 0 and G′=B′=1, and Cr may be equal to 0.5 when R′ is equal to 1, and G′=B′=0.

The format converter305may output each component of the Y′CrCb color space to a respective coder. Each coder may perform a processing operation upon its respective component as described herein and output the coded component to the min/max estimator340as well as to the respective clipping unit.

Given the relationship of Cb or Cr with Y′ as well as of Y′ with B′ and R′ respectively, the limits on each component may be impacted by the value of Y′. Therefore, if the min/max340estimator determines Y′, then it may derive a range of values for Cb and/or Cr. More specifically, if Y′=Y′C, then the following applies:

If Y′C≦wYR+wYG=1−wYBthen the min/max estimator340may compute the minimum value for Cb as

-YC′alpha=-YC′2*(1-wYB),
i.e. when B′=0 and Y′Cconsists of only R′ and G′. Otherwise, the minimum would occur when G′=R′=1, in which case

B′=YC′-(wYR+wYG)wYB=YC′-(1-wYB)wYB.
In this case, the min/max estimator340may calculate the minimum value for Cb as

YC′-(1-wYB)-wYB*YC′2*(1-wYB)*wYB.
In an embodiment, these two equations for minimum Cb may combine as:

min⁢⁢Cb=max⁡(0,YC′-(1-wYB))-wYB*YC′2*(1-wYB)*wYB.
The Min/Max Estimator340may apply this equation in its derivation of minCb.

If, on the other hand, Y′C≧wYBthen the min/max estimator340may compute the maximum value for Cb as

1-YC′alpha=1-YC′2*(1-wYB),
i.e. when B′=1 regardless of what the values for R′ and G′ may be. Otherwise, the maximum would occur when G′=R′=0, in which case

B′=YC′wYB.
In this scenario, the min/max estimator340may compute the maximum Cb value as

YC′-wYB*YC′2*(1-wYB)*wYB.
In an embodiment, these the two equations for maximum Cb may combine as:

The Min/Max Estimator340may apply this equation in its derivation of maxCb.

In an embodiment, given Y′=Y′C, the min/max estimator may compute the minimum and maximum values for Cr as:

In an embodiment, the min/max estimator340may derive two components (for example, Y′=Y′Cand Cb=CbC). In this case, the min/max estimator340may determine the relationship between R′ and the known values for Y′ and Cb as:

Given the above relationship between R′, the known values for Y′ and Cb, the min/max estimator340may also derive the values for B/B′.

-YC′beta=-YC′2*(1-wYR),
i.e. when R′=0 and Y′Cconsists of only B′ and G′. Otherwise, the minimum may occur when G′=1, in which case

R′=YC′-(wYG+wYB*B′)wYR=YC′-(wYG+wYB*(2*(1-wYB)*CbC+YC′))wYR.
In this case, the min/max estimator340may calculate the minimum Cr as

YC′-(wYG+wYB*(2*(1-wYB)*CbC+YC′))-wYR*YC′2*(1-wYR)*wYR.
In an embodiment, these two equations may combine as:

In an embodiment, if Y′C≧wYR+wYB*B′ then the min/max estimator340may calculate the maximum value for Cr as

1-YC′beta=1-YC′2*(1-wYR),
i.e. when R′=1 regardless of what the value of G′ is. Otherwise, the maximum may occur when G′=0, in which case

R′=YC′-wYB*B′wYR.
In this scenario, the min/max estimator340may calculate the maximum Cr value as

YC′-wYB*B′-wYR*YC′2*(1-wYR)*wYR.
In an embodiment, these two equations may combine as:

max⁢⁢Cr=min⁡(wYR,YC′-wYB*B′)-wYR*YC′2*(1-wYR)*wYR=min⁡(wYR,YC′-wYB*(2*(1-wYB)*CbC+YC′))-wYR*YC′2*(1-wYR)*wYR
The Min/Max Estimator340may apply this equation in its derivation of maxCr. In an extreme case, if

Similarly for Cb, given Y′ and Cr, the min/max estimator340may calculate the limits of Cb as:

min⁢⁢Cb=max⁡(0,YC′-(wYG+wYR*(2*(1-wYR)*CrC+YC′)))-wYB*YC′2*(1-wYB)*wYBmax⁢⁢Cb=min⁡(wYB,YC′-wYR*(2*(1-wYR)*CrC+YC′))-wYB*YC′2*(1-wYB)*wYBand,if⁢⁢Cr=YC′2*wYR⁢⁢then⁢⁢Cb=0.
Additionally, if Y=0, then both Cb and Cr components may be equal to 0.

FIG. 4illustrates an encoder system400according to an embodiment of the present disclosure. The encoder system400illustrates basic coding operations that may be performed by any of the color-specific coders310-330ofFIG. 3on input component data that has been parsed into predetermined units, called “pixel blocks” herein. The encoder system400may include a subtractor410, a transform unit415, a quantizer420, an entropy coder425, a dequantizer430, an inverse transform unit435, an adder440, a de-blocking unit445, a sample adaptive offset (SAO) filter450, a decoder picture buffer (DPB)455, a motion compensation/intra prediction unit460, a mode selector465, an intra-mode decision unit470, and a motion estimator475.

The subtractor410may receive an input color component pixel block at a first input and a predicted pixel block at a second input. The subtractor410may generate data representing a difference between an input pixel block and the predicted pixel block. The transform unit415may convert the difference to an array of transform coefficients, as by a discrete cosine transform (DCT) process or wavelet transform, for example. The quantizer420may quantize the transform coefficients obtained from the transform unit415by a quantization parameter QP (not shown). The entropy coder425may code the quantized coefficient data by run-value coding, run-length coding, arithmetic coding or the like, and may generate coded video data, which is outputted from the encoder system400. The output signal may then undergo further processing for transmission over a network, fixed media, etc.

The encoder system400may include a prediction loop that includes the dequantizer430, inverse transform unit435, adder440, deblock filter445and SAO filter450. The inverse transform unit435may reverse the quantization performed by the quantizer420. The inverse transform unit435may apply an inverse transform on the de-quantized data, which inverts operations of the transform unit415.

The adder440may be coupled to the inverse transform unit435and may receive, as an input, the inverse transformed data generated by the inverse transform unit435. The adder440may also receive an input from the mode selector465, which will be described in further detail below. The adder440may output resultant data to the deblocking filter445, which may reduce/remove artifacts of block encoding. The SAO filter450may be coupled to the deblocking unit445for further filtering. The filtered data may then be output from the coder400.

In an embodiment, the coder400may further include a clip unit480. The clip unit480may perform clipping of certain pixels from the filtered data whose color components are outside a valid range after all steps of decoding are completed. For example, the clip unit480may operate upon completion of prediction (intra/inter), residual reconstruction, deblocking and SAO filtering. The clip unit480may clip pixels whose color components exceed their respective valid range. Clipping at this stage may enforce the valid range imposed by the Y′, Cb, and Cr component relationships, as discussed hereinabove.

For reference pictures, pictures that may serve as prediction references for later-received frames, the decoder400may assemble reconstructed frames from the SAO filter450and store the reconstructed frames in a decoded picture buffer (DPB)455. The DPB455may store the reconstructed frames for use in decoding later-received coded data.

The motion estimator475may receive input data and search among frames previously stored in the DPB455for content in the reconstructed frames that match the input data. The motion estimator475may output to the mode selector465data identifying a best-matching pixel blocks from the stored reconstructed frames. Thus, the motion estimator475, for each desired reference, may derive motion information that would result in an inter prediction hypothesis for the current block to be coded.

The intra-mode decision unit470may receive the input signal and determine, from reconstructed data representing decoded portions of the frame currently being coded (e.g., previously-coded pixel blocks in the same frames) a prediction reference that can be developed by intra coding. The intra-mode decision unit470may determine prediction references for one or more intra-coding modes. Thus, the intra-mode decision unit470may estimate the “best” intra coding mode for the current block to be coded. In an embodiment, each mode may include an associated rate distortion cost, and intra-mode decision unit may estimate the “best” mode by determining which mode has the least rate distortion cost. The intra-mode decision unit470may output results of its prediction search to the mode selector465.

In an embodiment, the coder400may derive color component limits as discussed herein during estimation of a prediction mode. The intra-mode decision unit470may monitor the number of out of range values generated by each mode. The intra-mode decision unit470may eliminate modes having a considerable number of out of range values. In an embodiment, the intra-mode decision unit470may compare the number of out of range values generated by each mode to a threshold, if the number of out of range values for a particular mode exceeds the threshold, then that mode is eliminated. Alternatively, the intra-mode decision unit470may prioritize different prediction modes over others based on the relative numbers of out of range values that they would generate.

The mode selector465may select a coding mode to be applied to the input pixel block on indications furnished by the motion estimator475and inter-mode decision unit470. For example, the mode selector465may select from a variety of mode/prediction type, block size, reference modes, or even perform slice/frame level coding decisions including: use of intra, or single or multi-hypothesis (commonly bi-predictive) inter prediction; the size of the prediction blocks; whether a slice/picture shall be coded in intra (I) mode without using any other picture in the sequence as a source of prediction; whether a slice/picture shall be coded in single list predictive (P) mode using only one reference per block when performing inter predictions, in combination with intra prediction; and whether a slice/picture shall be coded in a bi-predictive (B) or multi-hypothesis mode, which allows, apart from single list inter and intra prediction the use of bi-predictive and multi-hypothesis inter prediction. Typically, coding modes are selected to minimize coding distortion and maximized coding efficiency. At times, however, a mode selector465may apply a less optimal coding mode to support other coding needs. For example, an intra-coding mode may be applied to inject an I frame into a coded bit stream (to support scrubbing play modes and the like) even if P coding may be more efficient.

The motion compensation/intra prediction unit460may receive input from the mode selector465and the decoded data from the DPB455. Based on received information, the motion compensation/intra prediction unit460may generate a reference block for the current input that is to be coded. The reference block may then be subtracted from the input component pixel block by the subtractor410.

In an embodiment, the coder400may further include another clip unit485. The clip unit485may provide clipping of certain pixels that are outside a valid range if additional clipping is required at other stages of the reconstruction process. For example, the clip unit485may provide clipping before the deblocking445or during the interpolation process for the subpixel motion estimation475and the compensation460. The clip unit485may clip pixels having one or more color components that exceed the valid range for that component. Clipping at this stage may enforce the valid range imposed by the Y′, Cb, and Cr component relationships, as discussed hereinabove.

In an embodiment, when performing intra prediction, the intra-mode decision unit470may predict the luma components of the data first. In this case, the coder400may perform clipping on subsequently predicted chroma component values, in the same manner as described hereinabove.

The coder400may represent a single component coder (i.e. it is meant to perform a processing operation on one component of a color space). In an embodiment, the coder400may output its coded component data to the respective coder for each component in a color space (not shown inFIG. 4). In addition, the coder400may receive coded component data from the respective coder for each component in the color space (not shown inFIG. 4). In this way, the coder400may be able to calculate limit values for its respective component and perform one or more processing operations upon its respective component as described herein. In an embodiment, the coder400may receive the determined range of values for each component from a min/max estimator (such as the min/max estimator340shown inFIG. 3).

In an embodiment, the coder400may include an analyzer490to develop metadata providing contextual information for the clipping process. The metadata may be reported to a decoder (FIG. 2) for use in recovery of coded video data. The metadata may include, for example, information representing the source image signal that is input to the coder (FIG. 2). For example, such metadata may describe the color primaries of the source video, transfer functions that operated on the source video, and/or matrix coefficients used during coding. In an embodiment, certain additional parameters may be known beforehand by being coded into the system, in which case metadata may convey information representing departures from these pre-coded parameters. For example, in a case where a channel is capable of handling data in particular color primaries/volume, such as BT.2020, P3D65, or BT.709, metadata parameters may identify circumstances where the source content effectively only covers a more limited color volume, such as BT.709, P3D65 or possibly even less. Metadata parameters may signal, for example, the effective colour primaries of the signal. Signaling could be done per region, image, or for a sequence of images. Primaries could be signaling using, for example, the three color primary system (e.g. red, blue, green, and optionally white, since commonly that would coincide with the white of the original color space, with their corresponding points in either the xy (CIE 1931) or u′v′ (CIE 1976) colorspaces) or even a more comprehensive color system that may also include secondary colors, e.g. cyan, magenta, and yellow. In another example, brightness characteristics of the signal representation may go up to a particular number X (e.g. 10000), however the signal may never exceed a value Y (e.g. 500). Information also about the original signals limits in terms of hue or saturation, or other properties, such as percentage of data that correspond within particular color volume ranges, could also be provided. This information can be used to further assist in the clipping and processing process of not only the chroma components but also of luma given the more comprehensive information and knowledge about the limitations of the original signal.

Coding protocols typically define a syntax for exchange of video data among different terminals. Many coding protocols allocate bandwidth to permit terminals to exchange data that does not have predetermined meaning under the protocol. For example, the MPEG-4 AVC/H.264 and MPEG-H part 2/HEVC/H.265 coding protocols define Supplemental Enhancement Information (“SEI”) and video usability information (“VUI”) messages for such purposes. Additional parameters and other metadata may be communicated in such messages. For coding protocols that do not reserve bandwidth for such purposes, communication of additional parameters and other metadata may occur between terminals in a channel that falls outside the syntax occupied by coded video data (e.g., a separate communication channel between the terminals).

In an embodiment, the encoder400may use additional parameters to adjust the various equations/processes described herein to achieve more efficient processing. Additionally, the encoder400may transmit the additional parameters as described above to a decoder (e.g. the decoder900, described more fully herein below) to enable the decoder to overcome errors generated during the decoding process, such as induced distortion from de-quantization.

Although the foregoing discussion has presented the analyzer490as resident in a coder400to develop data representing performance of clipping units therein, the principles of the present disclosure permit analyzers to capture data at different levels of performance within a system. For example, an analyzer (not shown) may find application in a system such as shown inFIG. 3and develop metadata representing performance of clip units342and344therein.

FIG. 5illustrates a method500according to an embodiment of the present disclosure. The method500may examine Y component data, which may be at full resolution, and then compute the maxCb and minCb limits as described above (box510). Then it processes Cb component data, which typically is provided at ¼ resolution of the Y component data. In one implementation, the Cb and Cr data may be at type 0 locations, where the chroma values are located in a co-sited manner with Y′ samples horizontally, but in the middle vertically. Further discussion of type 0 may be found in the AVC specification, Annex E (ISO/IEC 14496-10-MPEG-4 Part 10, Advanced Video Coding).

For the Cb component, the method may perform a vertical interpolation using a vertically applied filter (box515). The method500may compare the outcome of the interpolation versus the maxCb and minCb limits defined by the Y component and clip accordingly (box520). The method500may apply horizontal interpolation using the already-clipped results obtained in box520(box525). Interpolated Cb values may be clipped given the Y limits for those positions (box530).

The method500may perform similar operations on Cr component data, also. The method500may examine Y component data and compute the maxCr and minCr limits as described above (box540). For the Cr component, the method may perform a vertical interpolation using a vertically applied filter (box545). The method500may compare the outcome of the interpolation versus the maxCr and minCr limits defined by the Y component and clip accordingly (box550). The method500may apply horizontal interpolation using the already-clipped results obtained in box550(box555). Interpolated Cr values may be clipped given the Y limits for those positions (box560).

The operations illustrated in boxes510-530and in boxes540-560may be performed in parallel with each other using Y component data to set the max/min levels of both Cb and Cr component data. Alternatively, they may be performed sequentially. In such a case, the max/min levels of one component may be set using both the Y component data and the processed (and possibly clipped) color component data. For example, if boxes510-530are performed prior to performance of boxes540-560, Cr values may be calculated from both the Y component data and the processed Cb component data. As can be seen, sequential performance of the operations enables method500to perform clipping at every processing step to avoid propagation of an unclipped signal, and thus the possibility of more severe errors in any subsequent steps.

The method500ofFIG. 5finds application with processing operations that involve spatial processing of image data in multiple directions, for example interpolation, upconversion, scaling, downconversion and downsampling. WhereasFIG. 5illustrates interpolation as the spatial processing (in boxes515,525,545and555), in other embodiments of the present invention, upsampling or other processing may be performed in those boxes.

In an embodiment, the processing may be an adaptive downconversion scheme where a processing system may develop knowledge of the upconversion method that will be used (i.e. if downconverting from 4:4:4 to 4:2:0 and the upconversion scheme likely uses a particular set of upsampling filters). In such a case, the method500may apply the clipping range during the decision process to provide a more accurate estimate of the distortion that could be introduced given a particular downconversion decision. This may improve performance during the downconversion decision process.

FIG. 6illustrates a method600according to another embodiment. The method600ofFIG. 6may find application in processing operations that involve a single iteration of processing. For example, the method600may be employed for systems that perform denoising, frame rate conversion, debanding, inpainting, deringing, deblocking, dithering, and other such processing methods. When such a process is performed in connection with the method600ofFIG. 6, it is deemed the “calling process” for discussion purposes.

The method600may begin by performing a calling process on Y component data (box610). Thereafter, the method600may compute max and min limits for a first color component based on the processed Y data (box620). The method600may clip first color component values that exceed the max and min limits computed in box620(box630). Thereafter, the method600may perform the calling process on the first color component data obtained in box640, which may or may not have been clipped as determined by the limits (box640).

The method600may compute max and min limits for a second color component based on the processed Y data and the processed first color component data (box650). The method600may clip second color component values that exceed the max and min limits computed in box650(box660). Thereafter, the method600may perform the calling process on the second color component data obtained in box660, which may or may not have been clipped as determined by the limits computed in box650(box670)

FIG. 7illustrates another method700according to an embodiment of the present disclosure. As with the method500ofFIG. 5, the method700finds application with processing operations that involve spatial processing of image data in multiple directions, for example interpolation, upconversion, and scaling. WhereasFIG. 7illustrates interpolation as the spatial processing to be performed (referenced in boxes715and720), in other embodiments of the present invention, upsampling or other processing may be performed in those boxes.

The method700may begin by calculating maxCb, minCb, maxCr, and minCr values from the Y component data (box710). The method700then may perform a first iteration of clip processing (boxes715-755), using vertical interpolation. Specifically, the method700may interpolate Cb and Cr data in the selected interpolation direction (box715). Thereafter, the method700may determine whether any interpolated component (Cb and/or Cr) requires clipping (box720).

If neither component requires clipping, then the method may identify a component, either Cb or Cr, that is “furthest” away from its associated limits. The method700may “fix” the identified component, indicating that no clipping of the fixed component is required (box725). For example, if the distance of the Cb component to the maxCb and minCb limits is less than the distance of the Cr component to the maxCr and minCr limits, the Cb component is designated as “fixed.” The method700may revise the max/min limits of the other color component (Cr in the example above) (box730). The method700may determine whether the other component (Cr) requires clipping (box735) and, if so, performs clipping on that color component (box740).

In some cases, the Cr component will be designated as fixed during performance of box725and the maxCb and minCb values will be revised during performance of box730. Thus, the designation of “fixed” and “other” color components will be determined by the values of the Cb and Cr data obtained from interpolation.

At box720, it may be determined that only one of the interpolated color components requires clipping (say, component Cb). In this case, the other color component (Cr) may be designated as “fixed” (box745). Thereafter, the method700may advance to box730to revise the max/min limits of the component that was identified as requiring clipping. In boxes735and740, if the component (Cb) still requires clipping in view of the revised max/min limits, it will be clipped.

At box720, it may be determined that both interpolated color components require clipping. In this case, the method700may identify the component value (say, Cr) that is closest to its max or min limit and may clip that value (box750). Thereafter, the method700may revise the max/min limits for the other component (Cb in the foregoing example). The method may advance to box735. In boxes735and740, if the component (Cb) still requires clipping in view of the revised max/min limits, it will be clipped.

Following completion of box740or, if no clipping was required in box735, operation of the method700may return to box715for a second iteration. In the second iteration, horizontal interpolation may be performed. Operation of boxes720-755may repeat as before. Upon completion of the second iteration of boxes720-755, the method700may terminate (step not shown).

If desired, the method700may perform the interpolation horizontally in the first iteration and vertically in the second iteration. It may be convenient to perform the interpolation first vertically for type 0 chroma sample locations because the method700immediately generates the co-sited values for chroma vertically. If the method700performs the interpolation horizontally first, the values would not be co-sited with the luma components and computation of the limits becomes slightly more complex. In that case, the method700may interpolate luma values at co-sited locations of chroma and compute the limits given those interpolated, and temporary, luma values.

FIG. 8illustrates a method800according to another embodiment. The method800ofFIG. 8may find application in processing operations that involve a single iteration of processing. For example, the method800may be employed for systems that perform denoising, frame rate conversion, debanding, inpainting, deringing, deblocking, dithering, and other such processing methods. When such a process is performed in connection with the method800ofFIG. 8, it is deemed the “calling process” for discussion purposes.

The method800may begin by performing the calling process on Y component data (box810). The method800may calculate maxCb, minCb, maxCr, and minCr values from the Y component data (box815). The method800may perform the calling process on the Cb and Cr component data (box820). Thereafter, the method800may determine whether any processed color component (Cb and/or Cr) requires clipping (box825).

If neither processed component requires clipping, then the method may identify a component, either Cb or Cr, that is “furthest” away from its associated limits. The method800may “fix” the identified component, indicating that no clipping of the fixed component is required (box830). For example, if the distance of the Cb component to the maxCb and minCb limits is less than the distance of the Cr component to the maxCr and minCr limits, the Cb component is designated as “fixed.” The method800may revise the max/min limits of the other color component (Cr in the example above) (box835). The method800may determine whether the other component (Cr) requires clipping (box840) and, if so, performs clipping on that color component (box845).

In some cases, the Cr component will be designated as fixed during performance of box830and the maxCb and minCb values will be revised during performance of box835. Thus, the designation of “fixed” and “other” color components will be determined by the values of the Cb and Cr data obtained from processing.

At box825, it may be determined that only one of the processed color components requires clipping (say, component Cb). In this case, the other color component (Cr) may be designated as “fixed” (box850). Thereafter, the method800may advance to box835to revise the max/min limits of the color component that was identified as requiring clipping. In boxes840and845, if the color component (Cb) still requires clipping in view of the revised max/min limits, it will be clipped.

At box825, it may be determined that both processed color components require clipping. In this case, the method800may identify the color component value (say, Cr) that is closest to its max or min limit and may clip that value (box855). Thereafter, the method800may revise the max/min limits for the other component (Cb in the foregoing example) (box860). The method may advance to box840. In boxes840and845, if the color component (Cb) still requires clipping in view of the revised max/min limits, it will be clipped.

Following completion of box845or, if no clipping was required in box840, the method800may terminate.

FIG. 9illustrates a decoder system900in accordance with an embodiment of the present disclosure. The decoder system900may include an entropy decoder910, a de-quantizer915, an inverse transform unit920, an adder925, a de-blocking unit930, a sample adaptive offset (SAO) filter935, a post-processor940, a decoder picture buffer (DPB)945and a motion compensator950. The decoder system900may perform decoding operations according to the same coding protocol applied by the coding system400and may comply with MPEG-4, H.263, H.264 and/or HEVC.

The decoder system900may generate recovered video data from the coded video data. If an encoder (e.g encoder400) coded an element of a source video sequence with reference to a given element of reference picture data, the decoder system900may decode coded data of the source video element with reference to the same reference picture data. The motion compensator950may retrieve previously-decoded video data from the DPB945and supply it to the adder925as part of prediction operations.

The entropy decoder910may perform decoding of the encoded bitstream to obtain quantized transform coefficients. The dequantizer915and inverse transform920may then perform inverse quantization and inverse transformation on the quantized transform coefficients respectively to derive a decoded representation of the residual signal. The adder925may add the residual signal to predicted frame data transmitted by an encoder (e.g. the encoder400) and the resulting signal may be fed to one or more loop filters to smooth out artifacts induced by block-wise processing and quantization.

The SAO filter935may modify the decoded samples by conditionally adding an offset value to each sample after application of the deblocking filter930. The SAO filter935may introduce nonlinear amplitude mapping within the inter picture prediction loop after the deblocking filter to better reconstruct the original signal amplitudes. In an embodiment, the SAO filter935may reconstruct the original signal amplitudes on a region basis, based on a filtering type selected for each coding tree block (“CTB”) by the syntax of the metadata message.

The post processor940may perform other signal conditioning operations on recovered video obtained from the SAO filter935. Post-processing may include filtering operations or other video alterations to improve the recovered video for consumption by other components (not shown) within the decoder terminal (FIG. 2). In an embodiment, as part of its operation, the post-processor940may select processing operations based on metadata reported to it by an encoder in a Supplemental Enhancement Information (SEI) message or the like.

By way of example, a post processor940may receive information in an SEI message that, although the recovered video content is represented in the BT.2020 signal representation, the encoder operated on source content that were contained within a smaller color volume, such as BT.709 or P3D65. In response, the post processor940may select a different set of filters, such as filters that enforce clipping of all reconstructed values within the specified smaller color volume, than in a different use case where the encoder operated on video content that utilized the full range of the BT.2020 signal representation. In another example, given this additional information about the signal, a post-processor of limited precision (e.g. 10 or 12-bits) may map the video data for processing given their actual range, with effectively higher precision than it would have been possible without this information. Other examples may include the use of different or simpler tone mappers when remapping the content to a system or display of lower capabilities than the intended format representation. Knowledge of additional information, such as min/max hue or saturation in the signal, could result in filters that operate while trying to preserve the limits of such signals instead. Selection of these different filters would be based on an inference, coded into the post-processor, that encoder clipping proceeded differently for the reduced-range source video than for source video that spanned a larger range of available representations.

In another example, a post processor940may receive information in an SEI message that, although brightness of recovered video is represented by data that can go up to a particular number X (e.g. 10,000), the brightness of the source content signal never goes more than Y (e.g. 500). In response, the post processor940may select a different set of filters than in a different use case where the encoder operated on video content that utilized the full range of the brightness signal representation. For example, the filters would be able to clip the data given not the full range of the brightness signal representation but given the additional limits specified by the SEI or other metadata mechanism, expand precision of by scaling the data according to their dynamic range and the capabilities of the post-processor, and allow the use of filtering mechanism intended for applications of a lower dynamic range but not suitable for high dynamic range applications above a certain threshold, among others. Again, selection of these different filters would be based on an inference, coded into the post-processor, that encoder clipping proceeded differently for the reduced-range source brightness than for source brightness that spanned a larger range of available representations.

The above strategies are not restricted to any particular color space or primaries, or even chroma data with a specific chroma sampling type (chroma sampling location). The process finds application with data in other color spaces such as Y′u′v′, Y′u″v″, YDzDx, IPT, YCbCr using any color primaries such as BT.709, BT.2020, P3D65 etc., both constant and non-constant luminance representations, sampling types 0, 1, 2, etc. The two chroma samples also need not be co-sited, i.e. do not need to have been sampled with the same phase.

As it has been discussed earlier, the above techniques could apply to any processing scheme that could be used on the image or video data, apart from chroma upconversion or scaling. This may include denoising such as spatio-temporal filtering, frame rate conversion, debanding, inpainting, deringing, deblocking, dithering, and many other processing methods that are commonly applied on image and video data.

The foregoing discussion has described operation of the embodiments of the present disclosure in the context of terminals that embody encoders and/or decoders. Commonly, these components are provided as electronic devices. They can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors. Alternatively, they can be embodied in computer programs that execute on personal computers, notebook computers, tablet computers, smartphones or computer servers. Such computer programs typically are stored in physical storage media such as electronic-, magnetic- and/or optically-based storage devices, where they are read to a processor under control of an operating system and executed. Similarly, decoders can be embodied in integrated circuits, such as application specific integrated circuits, field programmable gate arrays and/or digital signal processors, or they can be embodied in computer programs that are stored by and executed on personal computers, notebook computers, tablet computers, smartphones or computer servers. Decoders commonly are packaged in consumer electronics devices, such as gaming systems, DVD players, portable media players and the like; and they also can be packaged in consumer software applications such as video games, browser-based media players and the like. And, of course, these components may be provided as hybrid systems that distribute functionality across dedicated hardware components and programmed general-purpose processors, as desired.