Patent Description:
The present invention relates generally to images. More particularly, an embodiment of the present invention relates to communicating and processing source color volume information.

Recommendation ITU-T H. <NUM> [<NUM>] (also known as HEVC) for "coding of moving video," in Annex D, "Supplemental enhancement information" (SEI), and Annex E, "Video usability information" (VUI), describes the syntax for providing supplemental SEI and VUI information in a coded bitstream to enable a decoder to better map the decoded samples into a display.

In parallel with the MPEG/ITU standardization processes, the society of motion picture and television engineers (SMPTE) has also defined a number of Recommendations related to communicating metadata related to the color volume information for both the source video and a targeted display. For example, the SMPTE ST <NUM> suite of documents (e.g., [<NUM>] and [<NUM>]) define metadata for use in color volume transforms of video content. These metadata may vary scene-by-scene or frame-by-frame. For example, such metadata may assist a decoder to present high-dynamic range (HDR) and wide color gamut (WCG) data on a display having a smaller color volume than that of the mastering display used for mastering the source images.

As used herein, the term "metadata" relates to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder to render a decoded image. Such metadata may include, but are not limited to, color space or gamut information, prediction parameters, reference display parameters, and auxiliary signal parameters, as those described herein.

While Annexes D and E of H. <NUM> support a number of color volume related metadata, they do not carry all the required metadata for the most efficient display management of HDR content. In July of <NUM>, in the joint collaborative team on video coding (JCT-VC) meeting in Geneva, three proposals [<NUM>-<NUM>] were submitted on how to describe content color volume information using SEI or VUI messaging. Some of these proposals were influenced by SMPTE ST. <NUM> [<NUM>], but they were considerably different in scope.

In [<NUM>], a content-SEI message is proposed to signal content color gamut in 2D, which describes the actual color distribution of the video content. In VUI, the variable colour_primaries is used to indicate the container color gamut instead of true source color gamut [<NUM>]. In [<NUM>], multiple primary expressions and spatial regions are proposed to be associated with the identified source characteristics. In [<NUM>], a content color volume SEI message is proposed to indicate the color volume occupied by the content. It uses an (x, y, Y) description of the color coordinates and has slices of luminance Y with associated polygons for each slice. These proposals have multiple shortcomings such as: provide information of little use to most display manufacturers, may add significant overhead, and may require too much computational overhead to generate. To improve existing coding and decoding schemes, as appreciated by the inventors here, improved techniques for generating and communicating source color volume information are required.

An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:.

Techniques for communicating source color volume information using SEI messaging are described herein. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention.

Example embodiments described herein relate to techniques for communicating source color volume information using SEI messaging. In a decoder, a processor for extracting SEI messaging receives a source color volume identification messaging variable identifying the presence of source color volume information in an input bitstream. The processor receives a first messaging variable as part of the source color volume information. If the first messaging variable matches a first predetermined value, then for one or more color primaries, it generates x and y chromaticity coordinates for the one or more color primaries based on the source color volume information in the input bitstream. It generates a minimum, a maximum, and an average luminance value based on the source color volume information in the input bitstream. The processor receives a second messaging variable as part of the source color volume information, and if the second messaging variable matches a second predetermined value, then for one or more color primaries, it generates x and y chromaticity coordinates for the one or more color primaries corresponding to the minimum, maximum, and average luminance values based on the source color volume information.

<FIG> depicts an example process of a video delivery pipeline (<NUM>) showing various stages from video capture to video content display. A sequence of video frames (<NUM>) is captured or generated using image generation block (<NUM>). Video frames (<NUM>) may be digitally captured (e.g. by a digital camera) or generated by a computer (e.g. using computer animation) to provide video data (<NUM>). Alternatively, video frames (<NUM>) may be captured on film by a film camera. The film, after appropriate editing (not shown), is converted to a digital format to provide video data (<NUM>).

The video data (<NUM>) is then provided to a processor at block (<NUM>) for post-production editing. Post-production editing (<NUM>) may include adjusting or modifying colors or brightness in particular areas of an image to enhance the image quality or achieve a particular appearance for the image in accordance with the video creator's creative intent. This is sometimes called "color timing" or "color grading. " Other editing (e.g. scene selection and sequencing, image cropping, addition of computer-generated visual special effects, etc.) may be performed at block (<NUM>) to yield a final version (<NUM>) of the production for distribution. During post-production editing (<NUM>), video images are viewed on a reference display (<NUM>) (also to be referred to as the "targeted display" since the studio optimizes the video for).

In some embodiments, before video coding (<NUM>), video content may be analyzed to extract source-color-volume metadata (<NUM>), for example as defined in SMPTE ST <NUM>-<NUM> [<NUM>], or as will be defined later in this invention. Such metadata may also define the characteristics of the targeted display (e.g., reference display (<NUM>)) and color remapping information so that a downstream receiver can render the decoded data in the best possible way.

Following post-production (<NUM>) and source-color volume analysis (<NUM>), video data of the final production (<NUM>) and associated metadata (<NUM>) may be delivered in an appropriate color format (e.g., <NUM>-bit YCbCr in <NUM>:<NUM>:<NUM>, ICtCp, and the like) to encoding block (<NUM>) for delivering downstream to decoding and playback devices such as television sets, set-top boxes, movie theaters, and the like. In some embodiments, coding block (<NUM>) may include audio and video encoders, such as those defined by ATSC, DVB, DVD, Blu-Ray, and other delivery formats, to generate coded bit stream (<NUM>). Coded bitstream (<NUM>) may be represented by a single layer video coded bitstream or by a multi-layer bitstream. For example, in a multi-layer bitstream, signal (<NUM>) may include a base layer (say, an SDR layer or a <NUM>-bit HDR (HDR10) layer) and an enhancement layer, which when combined with the base layer yields a HDR bitstream with higher dynamic range than the base layer alone (e.g., a <NUM>-bit HDR signal). Signal (<NUM>), the output bitstream from the encoder (<NUM>) may also include metadata (<NUM>) and additional coding-related metadata, such as prediction parameters and other data to assist a decoder to better reconstruct an HDR signal.

In a receiver, the coded bit stream (<NUM>) is decoded by decoding unit (<NUM>) to generate a decoded signal (<NUM>) and associated metadata (<NUM>). The receiver (or target) display (<NUM>) may have completely different characteristics than the reference (or targeted) display (<NUM>). For example, without limitation, the reference display (<NUM>) may be a <NUM>,<NUM> nits display while the receiver display may be a <NUM> nits display. In that case, a display management module (<NUM>) may be used to map the dynamic range of decoded signal (<NUM>) to the characteristics of the receiver display (<NUM>) by generating display-mapped signal (<NUM>). As used herein, the term "display management" denotes the processing (e.g., tone and gamut mapping) required to map an input video signal of a first dynamic range (e.g., <NUM> nits) to a display of a second dynamic range (e.g., <NUM> nits). Display management unit (<NUM>) may take into consideration metadata (<NUM>) to improve the quality of the output video on display (<NUM>). For example, as shown in [<NUM>], information about the luminance range of the targeted (or reference) display (e.g., <NUM>) and the source data may be used on a receiver to better map the dynamic range of the video content into the receiver display (e.g., <NUM>).

<FIG> depicts an example of the "largest" possible color volume of a pre-defined container format (e.g., BT. <NUM>) (also to be referred as the "container color volume"). Such a volume can be constructed by two-dimensional (2D) color-gamut primaries, the white point chromaticity (e.g., D65), a maximum luminance value (e.g., Lmax = <NUM>,<NUM> nits), and a minimum luminance value (e.g., <NUM> nits). Such a plot indicates the largest possible color volume boundary for all the colors within the source video content.

In practice, as depicted by the darker "cloud" (<NUM>) in <FIG> or the darker regions (<NUM>) in <FIG>, the source-color volume of the source content (e.g., <NUM>) for a particular frame, or even within a whole scene, may be significantly smaller than the largest possible color volume (<NUM>). Because the actual color volume (<NUM>) has very irregular shape, transmitting such source color volume information for each frame or the whole scene requires lots of information. For example, in an embodiment, one may signal the color gamut information for multiple luminance values (say, at <NUM>, <NUM>, <NUM>, and the like). The question then becomes: how many and which are the most important luminance values? One also needs to take into consideration not only the required overhead of such information on the coded bit stream, but also the complexity of generating such content on an encoder and/or reconstructing color volume information on the decoder.

While communicating minimum and maximum luminance values in the source content is important, as appreciated by the inventors, communicating the average luminance (or mid-point luminance) is also valuable to a receiver. These three values together can help generate a reasonable tone curve for display mapping. In this disclosure, it is proposed to signal the following metadata to describe source color volume: a) the largest 2D color gamut the source occupied (e.g., the source color volume); b) the maximum, minimum and average luminance of the source; and c) optionally, the sliced (2D) color gamut for those three luminance values (e.g., see <FIG>). It is assumed that the white points of the container primary and the source content primary should be the same, so there is no reason to retransmit such information. This information may be updated as needed, e.g., on a per-frame or a per-scene basis. <FIG> depict examples of 2D slices of the source color volume (<NUM>) and the container color volume (<NUM>) at specific luminance (Y) values. In <FIG>, the 2D slice is at Y = <NUM> nits, and in <FIG>, the 2D slice is at Y = <NUM> nits. The chromaticity (rgb) triangles, surrounding source color volume (<NUM>) and within the container RGB space, are provided for illustration purposes only. An encoder may select to define and communicate to a receiver smaller or bigger such areas.

Table <NUM> depicts an example of source-color volume SEI messaging according to an embodiment that follows the nomenclature and syntax of the H. <NUM> specification. The description of color primaries follows the definition of CIE <NUM> (x,y) color chromaticity coordinates for color primaries as defined in ISO <NUM>-<NUM> (see also ISO <NUM>-<NUM> and CIE <NUM>), and uses red, green, and blue color primaries. Other types of color primaries, such four, five, or six, or other polygon-based color primary presentation can also be used. For the largest actual color gamut within the source content, in an embodiment, without limitation, the syntax is similar to the definition of the colour_primaries parameter (or variable) defined in Section E. <NUM>, for Table E. <NUM>, of the H. <NUM> specification. It is believed that current source content can reach the P3 color space, but it will take some time to reach BT. <NUM>/<NUM> color ("DCI-P3" is defined in SMPTE EG <NUM>-<NUM> and SMPTE RP <NUM>-<NUM>). Therefore, in those cases where the source color gamut is smaller or equal to P3, or equal to BT. <NUM>/<NUM> color primaries, Table E. <NUM> can be used; however, for sources whose color gamut is larger than P3 but smaller than BT. <NUM>/<NUM>, explicit signalling of color gamut might be required. Luminance values are specified using their absolute value in nits (cd/m<NUM>). Alternatively, to save bits, luminance values may also be encoded using a non-linear representation, e.g., as values encoded according to the inverse EOTF of SMPTE ST <NUM>. The color gamut information corresponding to the max, min, and average (mid) luminance values is made optional, allowing applications to reduce the metadata overhead as desired.

Notes: in a preferred embodiment, <NUM>) the source color volume metadata should describe the color volume of the source in its original form, before any luma or chroma preprocessing. For example, it should describe source color volume before any chroma subsampling process (e.g., from <NUM>:<NUM>:<NUM> to <NUM>:<NUM>:<NUM>) or bit depth conversion process (e.g., from <NUM> b to <NUM> b), because chroma subsampling or bit depth conversion will modify the color volume information. <NUM>) The source color gamut is typically different from the container color primaries, which is indicated in Annex E (e.g., Table E. <NUM>) of H. <NUM>) The source color volume is typically different from the mastering display color volume, which may be indicated by the mastering display color volume SEI messages.

In an example embodiment, parameters (or variables) and coding semantics in Table <NUM> may be described as follows:.

Table <NUM> provides what is believed to be the minimal information for a useful representation of source color volume. In an another embodiment, one may decide to define additional details, like multiple primary expressions [<NUM>] or the description of the color primaries of more than three slices of luminance (Y), with associated polygons for each slice.

<FIG> depicts an example process for extracting color volume information for a video source using SEI messaging according to an embodiment. First (<NUM>), a decoder may detect whether a first SEI messaging variable indicating an identifying number (ID) of source color volume information (e.g., source_colour_volume_id) is present. Then, given the presence of such a variable, the decoder may check (step <NUM>) whether its value is within a permissible range. If it is an illegal value, then the process terminates (step <NUM>). If it is a legal value, then in step (<NUM>), as shown also in Table <NUM>, the decoder can read additional flags related to the persistence of the first variable across the bit stream (e.g., see the syntax elements for source_colour_volume_cancel_flag and source_colour_volume_persistence_flag). In step (<NUM>), via a second SEI messaging parameter (e.g., source_colour_primaries), a decoder may check whether the metadata define explicitly the color volume that source data content truly occupies. If it is true (e.g., source _colour_primaries = <NUM>) then, in step (<NUM>), the (x, y) color chromaticity coordinates for each color primary (e.g., red, green, and blue) are read, otherwise, in step (<NUM>), the decoder extracts the minimum, maximum, and average luminance values. Optionally, SEI messaging may also define the (x, y) color chromaticity coordinates corresponding to the color primaries of the min, mid, and max luminance values defined earlier. In an embodiment, this may be indicated by a third parameter (e.g., luminance_colour_primaries_info_present_flag = <NUM>). If no such information is present (step <NUM>), then the process terminates (<NUM>), otherwise, (in step <NUM>), the decoder extracts the (x, y) color chromaticity coordinates for the color primaries for each of the min, mid, and max luminance values.

After extracting the source color volume information, a decoder may use the source color volume data during its display management process (e.g., <NUM>). In an example, display management may include two steps: tone mapping and gamut mapping. The min, mid, and max luminance value can be used to generate a tone mapping curve as described in [<NUM>-<NUM>]. The maximal RGB color gamut and the sliced RGB gamut can be used to perform gamut mapping.

In some embodiments it may be beneficial to define an active region as part of the metadata related to the source color volume. For example, when video is encoded in a letterbox format, encoders and decoders should not include the black letterbox areas when computing luma and chroma characteristics of each video frame (e.g., min, max, and average luminance). Experimental results have shown that taking into consideration the "framing" or "matting" (e.g., pillarboxing, windowboxing, and letterboxing) of frames in a video sequence can improve significantly overall output picture quality. Though letter box detection can be implemented by a decoder, thus reducing the signaling overhead to define the active picture region, in an embodiment, such signaling may be explicitly signaled to support decoders with low computational complexity. Table <NUM> depicts an example of source-color volume SEI messaging with active region signaling according to an embodiment.

Table <NUM> is a superset of Table <NUM> and considers two different semantics of defining an active region.

The active region is defined with horizontal picture coordinates from active_region_left_offset + SubWidthC * conf_win_left_offset to CtbSizeY * PicWidthInCtbsY - SubWidthC * conf_win_right_offset - active_region_right_ offset - <NUM> and vertical picture coordinates from active_region_top_offset + SubHeightC * conf_win_top_offset to CtbSizeY * PicHeightInCtbsY - SubHeightC * conf_win_bottom_offset - active_region_bottom_offset - <NUM> , inclusive.

The value of ( active_region_left_offset + active_region_right_offset ) shall be less than CtbSizeY * PicWidthInCtbsY - SubWidthC * (conf_win_right_offset + conf_win_left_offset) , and the value of ( active_region_top_offset + active_region_bottom_offset ) shall be less than CtbSizeY * PicHeightInCtbsY - SubHeightC * (conf_win_bottom_offset + conf_win_top_offset ).

Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control, or execute instructions related to communicating source color volume information using SEI messaging, such as those described herein. The computer and/or IC may compute any of a variety of parameters or values that relate to the processes described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to communicating source color volume information using SEI messaging as described above by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.

Claim 1:
A method for extracting, by a processor of a decoder, source color volume information of an input bitstream from source color volume metadata, the method comprising:
receiving (<NUM>), by the processor, the source color volume metadata that includes a first messaging variable (source_colour_volume_id) identifying the presence of source color volume information in the source color volume metadata;
receiving (<NUM>), by the processor, a second messaging variable (source_colour_primaries) as part of the source color volume information in the source color volume metadata;
if the second messaging variable matches a first predetermined value, extracting (<NUM>), by the processor, first x and y chromaticity coordinates for at least one color primary from the source color volume information in the source color volume metadata; and
extracting (<NUM>), by the processor, a minimum, a maximum, and an average luminance value from the source color volume information in the source color volume metadata.