Patent Publication Number: US-10332481-B2

Title: Adaptive display management using 3D look-up table interpolation

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of priority to U.S. Provisional Patent Application Ser. No. 62/249,622, filed on Nov. 2, 2015, which is hereby incorporated herein by reference in its entirety. 
    
    
     TECHNOLOGY 
     The present invention relates generally to images. More particularly, an embodiment of the present invention relates to adaptive display management using 3D look-up table interpolation. 
     BACKGROUND 
     As used herein, the terms “display management” or “display mapping” denote the processing (e.g., tone and gamut mapping) required to map an input video signal of a first dynamic range (e.g., 1000 nits) to a display of a second dynamic range (e.g., 500 nits). Examples of display management processes can be found in WIPO Publication Ser. No. WO2014/130343 (to be referred to as the &#39;343 publication), “Display Management for High Dynamic Range Video,” which is incorporated herein by reference in its entirety. 
     As used herein, the term ‘dynamic range’ (DR) may relate to a capability of the human visual system (HVS) to perceive a range of intensity (e.g., luminance, luma) in an image, e.g., from darkest blacks (darks) to brightest whites (highlights). In this sense, DR relates to a ‘scene-referred’ intensity. DR may also relate to the ability of a display device to adequately or approximately render an intensity range of a particular breadth. In this sense, DR relates to a ‘display-referred’ intensity. Unless a particular sense is explicitly specified to have particular significance at any point in the description herein, it should be inferred that the term may be used in either sense, e.g. interchangeably. 
     A reference electro-optical transfer function (EOTF) for a given display characterizes the relationship between color values (e.g., luminance) of an input video signal to output screen color values (e.g., screen luminance) produced by the display. For example, ITU Rec. ITU-R BT. 1886, “Reference electro-optical transfer function for flat panel displays used in HDTV studio production,” (03/2011), which is incorporated herein by reference in its entity, defines the reference EOTF for flat panel displays based on measured characteristics of the Cathode Ray Tube (CRT). Given a video stream, any ancillary information is typically embedded in the bit stream as metadata. As used herein, the term “metadata” relates to any auxiliary information that is transmitted as part of the coded bitstream and assists a decoder to render a decoded image. Such metadata may include, but are not limited to, color space or gamut information, reference display parameters, and auxiliary signal parameters, as those described herein. 
     Most consumer HDTVs range from 300 to 500 nits with new models reaching 1000 nits (cd/m 2 ). As the availability of HDR content grows due to advances in both capture equipment (e.g., cameras) and displays (e.g., the PRM-4200 professional reference monitor from Dolby Laboratories), HDR content may be color graded and displayed on displays that support higher dynamic ranges (e.g., from 1,000 nits to 5,000 nits or more). Such displays may be defined using alternative EOTFs that support high luminance capability (e.g., 0 to 10,000 nits). An example of such an EOTF is defined in SMPTE ST 2084:2014 “High Dynamic Range EOTF of Mastering Reference Displays,” which is incorporated herein by reference in its entirety. In general, without limitation, the methods of the present disclosure relate to any dynamic range higher than SDR. As appreciated by the inventors here, improved techniques for the display of high-dynamic range images are desired. 
     The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which: 
         FIG. 1  depicts an example process for backlight control and display management according to an embodiment of this invention; 
         FIG. 2  depicts an example process for display management using a 3D look-up table for color gamut mapping according to an embodiment of this invention; 
         FIG. 3  depicts an example process for color gamut processing using 3D LUT interpolation according to an embodiment of this invention; 
         FIG. 4  depicts examples of ST 2084 (PQ) to BT 1886 (gamma) mappings according to an embodiment of this invention; and 
         FIG. 5  depicts examples of 3D LUT interpolation scalers computed according to embodiments of this invention. 
     
    
    
     DESCRIPTION OF EXAMPLE EMBODIMENTS 
     Techniques for backlight control and display management or mapping of high dynamic range (HDR) images are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail, in order to avoid unnecessarily occluding, obscuring, or obfuscating the present invention. 
     Overview 
     Example embodiments described herein relate to adaptive display management of HDR images using 3D look-up table (LUT) interpolation. In an embodiment, two or more look-up tables (LUTs) related to display management (say, for color gamut mapping) are precomputed for a set of distinct maximum brightness values for a display. Given a target maximum brightness value which does not match any of the values in the set, a new output look-up table is determined based on interpolating values from two of the pre-computed LUTs; one LUT pre-computed for a first maximum display brightness larger than the target maximum brightness value, and one LUT pre-computed for a second maximum display brightness lower than the target brightness value. An interpolation scale is computed based at least on the target maximum brightness value and the first maximum display brightness. 
     In an embodiment, the process of converting the output of a 3D color-gamut mapping LUT from a first color representation (say, RGB in ST 2084) to a second color representation (say, RGB in BT 1886) may be simplified by a) pre-computing a set of ST 2084 (PQ) to BT 1886 (gamma) tables for a small set of possible maximum brightness values for the target display and b) by interpolating values from these tables to perform color conversion given the target brightness value of the target display. 
     In one embodiment, the interpolation scale is computed based on a linear interpolation of the target brightness between the first maximum display brightness and the second maximum display brightness in the first color representation (say, RGB ST 2084). 
     In another embodiment, the interpolation scale is computed based on a linear interpolation of the target brightness between the first maximum display brightness and the second maximum display brightness in the second color representation (say, BT 1866). 
     Example Display Control and Display Management 
       FIG. 1  depicts an example process ( 100 ) for display control and display management according to an embodiment. Input signal ( 102 ) is to be displayed on display ( 120 ). Input signal may represent a single image frame, a collection of images, or a video signal. Image signal ( 102 ) represents a desired image on some source display typically defined by a signal EOTF, such as ITU-R BT. 1886 or SMPTE ST 2084, which describes the relationship between color values (e.g., luminance) of the input video signal to output screen color values (e.g., screen luminance) produced by the target display ( 120 ). The display may be a movie projector, a television set, a monitor, and the like, or may be part of another device, such as a tablet or a smart phone. 
     Process ( 100 ) may be part of the functionality of a receiver or media player connected to a display (e.g., a cinema projector, a television set, a set-top box, a tablet, a smart-phone, a gaming console, and the like), where content is consumed, or it may be part of a content-creation system, where, for example, input ( 102 ) is mapped from one color grade and dynamic range to a target dynamic range suitable for a target family of displays (e.g., televisions with standard or high dynamic range, movie theater projectors, and the like). 
     In some embodiments, input signal ( 102 ) may also include metadata ( 104 ). These can be signal metadata, characterizing properties of the signal itself, or source metadata, characterizing properties of the environment used to color grade and process the input signal (e.g., source display properties, ambient light, coding metadata, and the like). 
     In some embodiments (e.g., during content creation), process ( 100 ) may also generate metadata which are embedded into the generated tone-mapped output signal. A target display ( 120 ) may have a different EOTF than the source display. A receiver needs to account for the EOTF differences between the source and target displays to accurate display the input image. Display management ( 115 ) is the process that maps the input image into the target display ( 120 ) by taking into account the two EOTFs as well as the fact that the source and target displays may have different capabilities (e.g., in terms of dynamic range.) 
     In some embodiments, the dynamic range of the input ( 102 ) may be lower than the dynamic range of the display ( 120 ). For example, an input with maximum brightness of 100 nits in a Rec. 709 format may need to be color graded and displayed on a display with maximum brightness of 1,000 nits. In other embodiments, the dynamic range of input ( 102 ) may be the same or higher than the dynamic range of the display. For example, input ( 102 ) may be color graded at a maximum brightness of 5,000 nits while the target display ( 120 ) may have a maximum brightness of 1,500 nits. 
     In an embodiment, unless specified already by the source metadata ( 104 ), for each input frame in signal ( 102 ) the image analysis ( 105 ) block may compute its minimum (min), maximum (max), and median (mid) (or average gray) luminance value. These values may be computed for the whole frame or part of a frame. Given min, mid, and max luminance source data ( 107  or  104 ), image processing block ( 110 ) may compute the display parameters (e.g., the level of backlight) that allow for the best possible environment for displaying the input video on the target display. 
     In an embodiment, display ( 120 ) is controlled by display controller ( 130 ). Display controller ( 130 ) provides display-related data ( 134 ) to the display mapping process ( 115 ) (such as: minimum and maximum brightness of the display, color gamut information, and the like) and control data ( 132 ) for the display, such as control signals to modulate the backlight or other parameters of the display for either global or local dimming. 
     Displays using global or local backlight modulation techniques adjust the backlight based on information from input frames of the image content and/or information received by local ambient light sensors. For example, for relatively dark images, the display controller ( 130 ) may dim the backlight of the display to enhance the blacks. Similarly, for relatively bright images, the display controller may increase the backlight of the display to enhance the highlights of the image. 
     As described in WO2014/130343, and depicted in  FIG. 2 , given an input ( 112 ), the display characteristics of a target display ( 120 ), and metadata ( 104 ), the display management process ( 115 ) may be sub-divided into two main steps:
         a) Step ( 205 )—Determining the optimum color volume mapping (CVM) for the target display   b) Step ( 210 )—Determining the optimum color gamut mapping (CGM) for the target display       

     As used herein, the term “color volume space” denotes the 3D volume of colors that can be represented in a video signal and/or can be represented in display. Thus, a color volume space characterizes both luminance and color/chroma characteristics. For example, a first color volume “A” may be characterized by: 400 nits of peak brightness, 0.4 nits of minimum brightness, and Rec. 709 color primaries. Similarly, a second color volume “B” may be characterized by: 4,000 nits of peak brightness, 0.1 nits of minimum brightness, and Rec. 709 primaries. 
     During color volume mapping ( 205 ), display management operates on the input signal to adjust its intensity (luminance) and chroma to match the characteristics of a target display. This step may result in colors outside of the target display gamut. During color gamut mapping ( 210 ), a 3D color gamut look-up table (LUT) may be computed and applied to adjust the color gamut. In some embodiment, an optional color transformation step ( 215 ) may also be used to translate the output of CGM ( 212 ) (say, RGB) to a color representation suitable for display or additional processing (say, YCbCr). 
     In some embodiment, color volume mapping may be performed in the IPT-PQ color space. The term “PQ” as used herein refers to perceptual quantization. The human visual system responds to increasing light levels in a very non-linear way. A human&#39;s ability to see a stimulus is affected by the luminance of that stimulus, the size of the stimulus, the spatial frequency or frequencies making up the stimulus, and the luminance level that the eyes have adapted to at the particular moment one is viewing the stimulus. In a preferred embodiment, a perceptual quantizer function maps linear input gray levels to output gray levels that better match the contrast sensitivity thresholds in the human visual system. An example of a PQ mapping function is described in the SMPTE ST 2084 specification, where given a fixed stimulus size, for every luminance level (i.e., the stimulus level), a minimum visible contrast step at that luminance level is selected according to the most sensitive adaptation level and the most sensitive spatial frequency (according to HVS models). Compared to the traditional gamma curve, which represents the response curve of a physical cathode ray tube (CRT) device and coincidently may have a very rough similarity to the way the human visual system responds, a PQ curve, imitates the true visual response of the human visual system using a relatively simple functional model. 
     The IPT-PQ color space was first introduced in the &#39;343 publication and combines a PQ mapping with the IPT color space as described in “ Development and testing of a color space  ( ipt )  with improved hue uniformity ,” Proc. 6 th  Color Imaging Conference: Color Science, Systems, and Applications, IS&amp;T, Scottsdale, Ariz., November 1998, pp. 8-13, by F. Ebner and M. D. Fairchild, which is incorporated herein by reference in its entirety. IPT is like the YCbCr or CIE-Lab color spaces; however, it has been shown in some scientific studies to better mimic human visual processing than these spaces. 
     The display management process ( 115 ), with a single 3D LUT ( 210 ), works well when both the color volume of the target display and color encoding are fixed; however, for some use cases both of these may change dynamically. For example, many devices, such as TVs or tablets, may support dynamic backlight technology, where the intensity of the backlight may change on a per frame or per scene basis. Changing the backlight affects both the available color volume as well as the color encoding, which in turn, requires the 3D LUT in CGM ( 210 ) to be updated. However, updating the 3D LUT is computationally intensive, which limits the number of updates that can be done in real-time, resulting in poor viewing experience. As appreciated by the inventors, it would be beneficial to allow for dynamic color-gamut mapping (or backlight control) without having to re-compute the 3D LUT. 
     3D LUT Interpolation 
     Without loss of generality, given an input in a first color representation (say, in the IPT color space using ST 2084, or IPT-PQ), a 3D LUT for CGM generates output values in a second color representation (say, in RGB-PQ) assuming a given set of color primaries (say, Rec. 709). In a preferred embodiment, without limitation, the output color space is in RGB instead of say, YCbCr, since in most applications the PQ encoding after display management may change to some other, more commonly used, encoding (say, gamma encoding as defined by BT 1886), which is only possible in the RGB domain. 
       FIG. 3  depicts an example process for color gamut processing using 3D LUT interpolation according to an embodiment. Given a display that can be adjusted to display at a variety of possible maximum brightness values (e.g., by adjusting the backlight), given a video input (e.g.,  102 ), a display management process (e.g.,  100 ) computes first the desired maximum brightness of the target display, to be denoted as TMax or TMaxPQ. As used herein, the term “PQ” at the end of variable name, say TMaxPQ, denotes that the variable&#39;s original value (say TMax=400 nits) is mapped to a value in the (0,1) range according to the ST 2084 EOTF. Examples of optimum adjustment of the target display brightness are described in U.S. Provisional application Ser. No. 62/193,678, “Backlight control and display mapping for high dynamic range images,” filed on Jul. 17, 2015, (also filed, on May 11, 2016, as PCT/US2016/031920) which is incorporated herein by reference in its entirety. 
     In an embodiment, let LUT(i), i=1, 2, . . . , N, (N≥2), denote a set of pre-computed 3D CGM LUTs, each one targeting a specific color volume for a maximum target display luminance level, to be denoted as LUTMax(i) (e.g., for N=4, LUTMax(i)={100, 200, 300, and 400} nits). Let LUTMaxPQ(i), denote the maximum target brightness for LUT(i) in the PQ domain. In step ( 310 ), given TMaxPQ, two LUTs (say LUT(A) and LUT(B) are determined to generate the output LUT (LUTOut). For example, in an embodiment, the two LUTs may be selected so that
 
LUTMaxPQ( A )&gt;TMaxPQ&gt;LUTMaxPQ( B ).  (1)
 
     Let alpha denote an interpolation scale to be used to interpolate LUTOut based on LUT(A) and LUT(B), then, in an embodiment, in step ( 315 ), a linear interpolation scale may be generated as:
 
alpha =(LUTMaxPQ( A )−TMaxPQ)/(LUTMaxPQ( A )−LUTMaxPQ( B )).  (2)
 
     Given alpha, in step ( 320 ), values v of the output LUT may be computed as
 
LUTOut( v )=alpha*LUT( B )( v )+(1−alpha)*LUT( A )( v ),  (3)
 
where v denotes an input vector (say, IPT values). In a preferred embodiment, the interpolation points for all LUT(i)s may be identical to simplify computations.
 
     Since, as discussed earlier, in a preferred embodiment, the output of LUTOut is in RGB-PQ, its output may need to be adapted according to the expected input for the target display. For example, if the target display expects YCbCr in BT 1886, then step ( 325 ) may include the following steps:
         Convert RGB-PQ to linear RGB, using an inverse transformation defined by ST 2084   Convert linear RGB to RGB-BT1886 using the BT1866 specification   Convert RGB-BT1886 to YCbCr-BT1886 using an RGB to YCbCr transformation       

     These steps can be applied directly to the output of the interpolated LUTOut table; however, they may require too many computations to be effectively supported by the target device. A more computationally-efficient approach may include the following steps: 
     Offline (Pre-computed) 
     
         
         
           
             a) Select K, (K≥2), possible maximum luminance levels for the target display. In an embodiment, it is preferred for these values to be evenly spaced in the PQ domain (say, 100, 160, 250, and 400 nits) 
             b) For each of these luminance levels (say, LPQ(i), i=1 to K), compute PQ to linear and linear to BT 1866 values, to generate a PQ to BT 1886 mapping (e.g., PQtoBT1886L(i)). An example of such mapping for four maximum brightness values (e.g., K=4, L(i)={100, 160, 250, and 400} nits) is depicted in  FIG. 4 .
 
In Real-time (On a Per-frame or a Per-scene Basis)
 
           
         
       
    
     Using TMaxPQ (the maximum brightness level of the target display), compute the target RGB-BT1886 values by interpolation:
 
PQtoBT1886Out( v )=beta*PQtoBT1886 L ( B )( v )+(1-beta)*PQtoBT1886 L ( A )( v ),   (4)
 
where v denotes the R, G, or B pixel value at the output of equation (3) in RGB-PQ domain, and as before, the PQtoBT1886(A) and PQtoBT1886(B) LUTs may be selected so that in nits
 
 L ( A )&gt;TMax&gt; L ( B ),  (5a)
 
or in the PQ domain
 
LPQ( A )&gt;TMaxPQ&gt;LPQ( B ).  (5b)
 
     Using linear interpolation, as before, the interpolation scale beta in equation (4) may be expressed as: 
             beta   =           LPQ   ⁡     (   A   )       -   TMaxPQ         LPQ   ⁡     (   A   )       -     LPQ   ⁡     (   B   )           .           
Note that in some embodiments, the number of 3D LUTs (e.g., N) used to determine the interpolated CGM LUTOut in step ( 310 ) may be different than the number of LUTs (e.g., K) used to do the color conversion in step ( 325 ).
 
     In some embodiments, due to the non-linear relationship between the target device maximum brightness levels, the performance of the interpolation method may be improved significantly by precomputing additional tables of interpolation parameters (alpha). In an embodiment, such tables may be computed as follows: 
     Offline 
     
         
         
           
             a) Generate a list of M, M≥2, maximum target display luminance levels denoted as LalphaPQ(i). For example, for M=16, let Lalpha(i)={100, 110, 121, 133, 146, 160, 176, 193, 212, 232, 254, 279, 305, 334, 366, 400} in nits. 
             b) Translate these luminance values from PQ to linear and from linear to BT 1886 to generate LalphaBT L)j) )(i) values. Note that, as depicted in  FIG. 4 , the PQ to BT mapping depends on the maximum luminance of the target display (L(j), j=1 to K), hence a separate table needs to be generated for each potential maximum brightness display value (e,g., for K=4, L(j)={100, 160, 250, and 400} nits). 
             c) Compute the interpolation values for each of these M and K values: alpha L(j) (i)=(1-LalphaBT L(j) (i))/(LalphaBT L(j) )(i+1)−LalphaBT L(j) (i)), for i=1 to M, and j=1 to K. 
           
         
       
    
     Given these alpha L(j) (i) values, additional alpha values for LalphaPQ values not in the input set (LalphaPQ(i)), can be computed by simple linear interpolation. 
       FIG. 5  shows example alpha values computed by both the default method of equation (2) (straight dotted lines) and the new method that relies on a PQ to BT 1866 mapping (curved solid lines), for maximum luminance values (L(j)) at 100, 160, and 254 nits. Hence, given an input TMaxPQ value, an upper boundary brightness value (L 1 ) and a lower boundary brightness value (L 2 ), in an embodiment, one can compute the preferred interpolation scale as follows: 
     In Real-time 
     
         
         
           
             a) As in step ( 310 ), determine the two CGM 3D LUTs to be used for interpolation; say, LUT(A) and LUT(B), where L 1 =LUTMaxPQ(A)&gt;TMaxPQ&gt;LUTMaxPQ(B)=L 2   
             b) Compute alpha based on the pre-computed alpha L(j) (i) values; for example, for 
           
         
       
    
     
       
         
           
             
               s 
               = 
               
                 
                   
                     LUTMaxPQ 
                     ⁡ 
                     
                       ( 
                       A 
                       ) 
                     
                   
                   - 
                   TMax 
                 
                 
                   
                     LUTMaxPQ 
                     ⁡ 
                     
                       ( 
                       A 
                       ) 
                     
                   
                   - 
                   
                     LUTMaxPQ 
                     ⁡ 
                     
                       ( 
                       B 
                       ) 
                     
                   
                 
               
             
             , 
           
         
       
         
         
           
             
               
                 alpha=s*alpha L2 (TMaxPQ)+(1−s)*alpha L1 (TMaxPQ) 
               
             
             c) Using the computed alpha, use equation (3) to generate LUTOut 
             d) Use the PQtoBT1886 LUTs to convert the output of LUTOut to RGB-BT1866 values 
             e) Optionally, convert the RGB-BT1866 values to YCbCr or any other desired color format 
           
         
       
    
     Example Computer System Implementation 
     Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and/or apparatus that includes one or more of such systems, devices or components. The computer and/or IC may perform, control, or execute instructions relating to backlight control and display mapping processes, such as those described herein. The computer and/or IC may compute any of a variety of parameters or values that relate to backlight control and display mapping processes described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof. 
     Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder or the like may implement methods related to backlight control and display mapping processes as described above by executing software instructions in a program memory accessible to the processors. The invention may also be provided in the form of a program product. The program product may comprise any non-transitory medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted. 
     Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a “means”) should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionally equivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention. 
     Equivalents, Extensions, Alternatives and Miscellaneous 
     Example embodiments that relate to efficient backlight control and display mapping processes are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.