Abstract:
In some embodiments, color and contrast enhancement video processing may be done in one shot instead of adjusting one of color and contrast enhancement, then the other, and then going back to the first one to readjust because of the second adjustment. In some embodiments, global lightness adjustment, local contrast enhancement, and saturation enhancement may be done at the same time and in parallel. Lightness adjustment improves visibility of details for generally dark or generally light images without changing intended lighting conditions in the original shot, and is used to enhance the range of color/saturation enhancement. Local contrast enhancement done in parallel improves visual definition of objects and textures and thus local contrast and perceived sharpness.

Description:
BACKGROUND 
       [0001]    This relates generally to processing video in a video processing pipeline. 
         [0002]    A video processing pipeline is used to process video streams or still images from a video source, such as a television tuner, a set top box, or a media player such as an optical disk drive. A video processing pipeline may be used to improve received audio and video. The video processing pipeline may also enhance color and contrast of the received video. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0003]      FIG. 1  is a depiction of a video processing pipeline in accordance with one embodiment; 
           [0004]      FIG. 2  is a schematic depiction of a color and contrast enhancement unit in accordance with one embodiment; 
           [0005]      FIG. 3  is a cumulative distribution function graph for the image shown in  FIG. 7 ; 
           [0006]      FIG. 4  is an intensity lookup table for the image shown in  FIG. 7 ; 
           [0007]      FIG. 5  is a cumulative distribution function graph for the image shown in  FIG. 8 ; 
           [0008]      FIG. 6  is an intensity lookup table for the image shown in  FIG. 8 ; 
           [0009]      FIG. 7  is a test image used in connection with  FIGS. 3 and 4 ; 
           [0010]      FIG. 8  is a test image used in connection with  FIGS. 5 and 6 ; 
           [0011]      FIG. 9  is a flow chart for saturation enhancement according to one embodiment; and 
           [0012]      FIG. 10  is a graph of a saturation lookup table with a saturation boost of −0.15, according to one embodiment. 
       
    
    
     DETAILED DESCRIPTION 
       [0013]    Referring to  FIG. 1 , a video processing pipeline  100  may be used in a variety of applications, including a personal computer, a set top box, a television receiver, or a media player. The video source  102  may, for example, be a tuner, a DVD disk player, or a set top box, to mention a few examples.  FIG. 1  shows typical components of a video pipeline, not necessarily proper sequencing, which depends on content, requirements, and the specific algorithms that are used. 
         [0014]    The video source  102  provides interlaced video, in one embodiment, to a noise reduction and filtering unit  104 . The block  104  performs noise reduction filtering on the digital signal provided from the video source  102 . The de-interlacing module  106  converts interlaced video to progressive scan video in one embodiment. 
         [0015]    The color and contrast enhancement  108  is responsible for enhancing both the color and the contrast. In some embodiments, the color and contrast enhancements may be done in “one shot,” meaning that it is not necessary to adjust a first one of color or contrast and then to adjust the second one of color or contrast and then to go back and adjust the first one again to account for the changes in the second one. The position of color/contrast enhancement varies, and is known to interact with sharpness enhancement (perception of sharpness is affected by contrast). 
         [0016]    A sharpness enhancement unit  110  may implement picture sharpness enhancement. A scaling unit  112  may resize the image for display on the display  114 . 
         [0017]    Referring to  FIG. 2 , the color and contrast enhancement unit  108  may enhance the color and contrast of received video in one shot. In one embodiment, the unit  108  may first convert to a suitable color space, as indicated in block  12 . For video streams, the input color space is usually YCbCr, and for stills, RGB color space is common. 
         [0018]    The color space to implement the color enhancement algorithm may be uniform in terms of perceived hue so that enhanced colors do not change in hues. In some embodiments, lightness prediction of chromatic colors does not have substantial hue or lightness modulation dependency. The color space may have invertible functional mapping between XYZ (or some other fundamental color description) and the color space dimensions. Also, the color space may be an opponent (has a neutral axis along one dimension) three-dimensional space. 
         [0019]    Transformation to and from a linear color space may not be computationally expensive in some cases. One color space that is suitable is the IPT color space. “I” in IPT is loosely related to intensity, “P” is the red-green dimension, and the yellow-blue dimension is indicated by the letter “T.” The IPT color space linearly models constant perceived hue. 
         [0020]    The input image is unlikely to be in the IPT color space. Therefore, color space conversion may be necessary. Standard RGB (or sRGB) is converted to CIEXYZ for the 1931 2° standard observer with an illuminant of D65. Equation 1 shows the conversion from XYZ to IPT (sRGB is gamma corrected RGB; thus gamma correction is built-into this algorithm) 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       [ 
                       
                         
                           
                             X 
                           
                         
                         
                           
                             Y 
                           
                         
                         
                           
                             Z 
                           
                         
                       
                       ] 
                     
                     = 
                     
                       
                         
                           
                             [ 
                             
                               
                                 
                                   0.4124 
                                 
                                 
                                   0.3576 
                                 
                                 
                                   0.1805 
                                 
                               
                               
                                 
                                   0.2126 
                                 
                                 
                                   0.7152 
                                 
                                 
                                   0.0722 
                                 
                               
                               
                                 
                                   0.0193 
                                 
                                 
                                   0.1192 
                                 
                                 
                                   0.9505 
                                 
                               
                             
                             ] 
                           
                            
                           
                             [ 
                             
                               
                                 
                                   R 
                                 
                               
                               
                                 
                                   G 
                                 
                               
                               
                                 
                                   B 
                                 
                               
                             
                             ] 
                           
                         
                          
                         
                           
 
                         
                         [ 
                         
                           
                             
                               L 
                             
                           
                           
                             
                               M 
                             
                           
                           
                             
                               S 
                             
                           
                         
                         ] 
                       
                       = 
                       
                         
                           [ 
                           
                             
                               
                                 0.4002 
                               
                               
                                 0.7075 
                               
                               
                                 
                                   - 
                                   0.0807 
                                 
                               
                             
                             
                               
                                 
                                   - 
                                   0.2280 
                                 
                               
                               
                                 1.1500 
                               
                               
                                 0.0612 
                               
                             
                             
                               
                                 0 
                               
                               
                                 0 
                               
                               
                                 0.9184 
                               
                             
                           
                           ] 
                         
                          
                         
                           [ 
                           
                             
                               
                                 
                                   X 
                                   
                                     D 
                                      
                                     
                                         
                                     
                                      
                                     65 
                                   
                                 
                               
                             
                             
                               
                                 
                                   Y 
                                   
                                     D 
                                      
                                     
                                         
                                     
                                      
                                     65 
                                   
                                 
                               
                             
                             
                               
                                 
                                   Z 
                                   
                                     D 
                                      
                                     
                                         
                                     
                                      
                                     65 
                                   
                                 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         L 
                         ′ 
                       
                       = 
                       
                         L 
                         0.43 
                       
                     
                     ; 
                     
                       L 
                       ≥ 
                       0 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         L 
                         ′ 
                       
                       = 
                       
                         - 
                         
                           
                             ( 
                             
                               - 
                               L 
                             
                             ) 
                           
                           0.43 
                         
                       
                     
                     ; 
                     
                       L 
                       &lt; 
                       0 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         M 
                         ′ 
                       
                       = 
                       
                         M 
                         0.43 
                       
                     
                     ; 
                     
                       M 
                       ≥ 
                       0 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         M 
                         ′ 
                       
                       = 
                       
                         - 
                         
                           
                             ( 
                             
                               - 
                               M 
                             
                             ) 
                           
                           0.43 
                         
                       
                     
                     ; 
                     
                       M 
                       &lt; 
                       0 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         S 
                         ′ 
                       
                       = 
                       
                         S 
                         0.43 
                       
                     
                     ; 
                     
                       S 
                       ≥ 
                       0 
                     
                   
                    
                   
                     
 
                   
                    
                   
                     
                       
                         S 
                         ′ 
                       
                       = 
                       
                         - 
                         
                           
                             ( 
                             
                               - 
                               S 
                             
                             ) 
                           
                           0.43 
                         
                       
                     
                     ; 
                     
                       
                         S 
                         &lt; 
                         
                           0 
                            
                           
                             
 
                           
                           [ 
                           
                             
                               
                                 I 
                               
                             
                             
                               
                                 P 
                               
                             
                             
                               
                                 T 
                               
                             
                           
                           ] 
                         
                       
                       = 
                       
                         
                           [ 
                           
                             
                               
                                 0.400 
                               
                               
                                 0.4000 
                               
                               
                                 0.2000 
                               
                             
                             
                               
                                 4.4550 
                               
                               
                                 
                                   - 
                                   4.8510 
                                 
                               
                               
                                 0.3960 
                               
                             
                             
                               
                                 0.8056 
                               
                               
                                 0.3572 
                               
                               
                                 
                                   - 
                                   1.1628 
                                 
                               
                             
                           
                           ] 
                         
                          
                         
                           [ 
                           
                             
                               
                                 
                                   L 
                                   ′ 
                                 
                               
                             
                             
                               
                                 
                                   M 
                                   ′ 
                                 
                               
                             
                             
                               
                                 
                                   S 
                                   ′ 
                                 
                               
                             
                           
                           ] 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
         [0000]    If the input image is in YCbCr space, an additional step to convert to RGB may be used. 
         [0021]    The color data in the IPT space, in accordance with one embodiment, may then be subjected to parallel local contrast enhancement, as indicated in block  16 , global lightness adjustment or contrast enhancement, as indicated in block  14 , and chroma or saturation enhancement, as indicated in block  18 . The results of the parallel, one shot enhancements and adjustments may then be combined in  20 ; the enhanced value for I comes from the ΔI from local contrast enhancement, the new I after lightness correction, and the P and T components come from the chroma enhancement block. The resulting IPT output may then be converted back to its original color space, as indicated in block  22 . 
         [0022]    A color and contrast enhancement unit  108  may be both automatic and independent of image content or color in some embodiments. The unit  108  may also integrate color and contrast enhancement functionalities. The unit  108  may improve the overall saturation while maintaining the original hue in some embodiments. Advantageously, the unit  108  leaves the achromatic colors unaltered. The unit  108  advantageously does not increase the saturation of colors that are already saturated enough, and, thus, reduces perceived loss in sharpness and undesirable color enhancement. Also, the unit  108  may not produce out-of-gamut colors or color artifacts (e.g. blotchiness) in some embodiments. Finally, the unit  108  may be fast enough for the video processing chain operating at a regular frame rate in some embodiments. 
         [0023]    The unit  108  may be implemented in software, hardware, or firmware. A software implementation may include computer executable instructions stored on a computer readable medium. A computer readable medium may be a semiconductor, magnetic, or optical storage. 
         [0024]    In a typical image enhancement operation, increasing the saturation may lead to out-of-gamut or impermissible colors. As the saturation of a color increases, the lightness automatically goes down. Conversely, reducing the lightness gives a perception of increased saturation. Thus, in a typical image with normal exposure, the overall lightness may be reduced before increasing the saturation. 
         [0025]    However, if an image is mostly dark to begin with, either due to under-saturation or due to high dynamic range of the image (which means a part of the image is quite bright at the same time), the lightness may be increased before any perceived color enhancement can be attained. In this case, saturation enhancement may be higher than the normal to counteract the effect of a perceived loss of saturation due to an increase in the lightness. Thus, a determination is made as to whether an image needs to be darkened or lightened, and to what extent. This may be achieved by the global lightness adjustment  14 . 
         [0026]    The global lightness adjustment  14  may be done using a lookup table (LUT). The pixel intensity curve is a straight line with unity slope below a lower threshold (for example, 0.1) and above upper threshold (for example, 0.9). Thus pixel intensities less than the lower threshold and more than the upper threshold are not altered in one embodiment. These thresholds are parameterized, and can be used as user controls. 
         [0027]    Between these thresholds, the intensities are adjusted nonlinearly, for example, according to equation 2: 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         
                           
                             I 
                             
                               OUT 
                                
                               
                                 - 
                               
                                
                               GC 
                             
                           
                           = 
                           
                             I 
                             i 
                           
                         
                         , 
                       
                     
                     
                       
                         
                           I 
                           i 
                         
                         ≤ 
                         
                           I 
                           LowTH 
                         
                       
                     
                   
                   
                     
                       
                         
                           
                             I 
                             
                               OUT 
                                
                               
                                 - 
                               
                                
                               GC 
                             
                           
                           = 
                           
                             
                               I 
                               i 
                             
                             - 
                             
                               β 
                                
                               
                                   
                               
                                
                               
                                 sin 
                                  
                                 
                                   ( 
                                   
                                     π 
                                     * 
                                     
                                       I 
                                       i 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                         
                         , 
                       
                     
                     
                       
                         
                           I 
                           LowTH 
                         
                         ≤ 
                         
                           I 
                           i 
                         
                         ≤ 
                         
                           I 
                           HighTH 
                         
                       
                     
                   
                   
                     
                       
                         
                           
                             I 
                             
                               OUT 
                                
                               
                                 - 
                               
                                
                               GC 
                             
                           
                           = 
                           
                             I 
                             i 
                           
                         
                         , 
                       
                     
                     
                       
                         
                           I 
                           i 
                         
                         ≥ 
                         
                           I 
                           HighTH 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where I i  and I OUT-GC  are the input and output intensities, and the beta parameter (β) determines the amount of adjustment and depends on the overall image characteristics as described below. A positive beta value darkens the image and a negative beta value lightens the image. The sine function is used to provide a unimodal, symmetric enhancement curve. Other similar mathematical functions can be used, and even sigmoid curves can be applied to implement functions equivalent to histogram equalization which could be built into the block. However, one objective of the lightness adjustment block is to provide the option to increase lightness in a way that synergistically makes saturation enhancement more effective (e.g. lightening a color gives more room for increased saturation, darkening a color improves colorfulness without much saturation enhancement). 
         [0028]    The image intensity corresponding to a cumulative distribution function (CDF) of 0.5 is used to determine whether the image should be darkened or lightened in one embodiment. The 0.5 point of a CDF indicates the lightness below which 50% of the pixels are found. A cumulative distribution function plots the same intensity information normally depicted by histograms. However, the difference between a cumulative distribution function and a histogram is that the intensity values are summed from left to right in the cumulative distribution function. Typically, the cumulative distribution function displays relative frequency in terms of percentiles on the vertical axis and intensity or other units on the horizontal axis. 
         [0029]    If the image intensity for the 0.5 CDF value lies below a pre-defined low CDF threshold (0.1, for example, meaning that 50% of the pixels are at or below 0.1% of the max lightness/intensity), the image may be lightened to the maximum level (β=−0.15, for example). If this intensity lies above a pre-defined high CDF threshold (0.9, for example), the image may be darkened to the maximum level (β=0.15, for example). When the intensity lies in between, β may be computed by linear interpolation. Once β has been calculated, β is used in the calculation of global lightness adjustment formula (2). β is also used in the chroma enhancement block  18 . 
         [0030]    The calculation of β involves computing empirical CDF, finding intensity I at CDF values of 0.2, 0.5, and 0.8 (L 20 , L 50 , L 80 ), and applying the following computation method. If 50% of pixel intensity values (L 50 ) lies below a low threshold (paramLowThresCDF 50 ), then the image is lightened to the maximum and beta is set equal to paramBetaMax. Otherwise, if 50% of pixel intensities are greater than a high threshold (paramHighThresCDF 50 ), then the image is darkened to the maximum and beta is set equal paramBetaMax. If % beta lies in between −betaMax and betaMax then beta is computed as a linear interpolation between −betaMax and betaMax {((2*abs(paramBetaMax)*(L 50 −paramLowThresCDF 50 ))/(paramHighThresCDF 50 −paramLowThresCDF 50 ))}−paramBetaMax;. 
         [0031]    Two additional conditions may be checked to ensure darkening or lightening does not cause some part of the image to be clipped. The first additional condition makes sure that no lightness correction is applied when beta is less than 0 and L 80  is greater than or equal paramThresCDF 80 ; i.e. if β&lt;0 &amp;&amp;(L 80 &gt;=paramLowThresCDF 80 ) then β=0. The second condition makes sure that no lightness corrections is applied when beta is greater than 0 and L 20  is less or equal than paramThresCDF 20 ; i.e. if β&gt;0 &amp;&amp; (L 20 &lt;=paramThresCDF 20 ) then β=0. 
         [0032]    Typically, paramLowThresCDF 50  is 0.1 with a typical range 0.1 to 0.3, paramBetaMax is 0.15 with a typical range 0.1 to 0.3, and paramHighThresCDF 50  is 0.9 with a typical range 0.6 to 0.9. If top 20% pixel intensity in the CDF lies above paramThresCDF 80  (which is 0.8), then the image should not be lightened. If beta is greater than 0 and L 20  is less than or equal paramThresCDF 20  (which is 0.2), i.e. if bottom 20% pixel intensity in the CDF lies below paramThresCDF 20 , then the image should not be darkened. 
         [0033]    In  FIG. 3 , the CDF of an image that needs the maximum amount of lightening is depicted. For this image, the intensity corresponding to a CDF of 0.5 is only 0.107, which means 50% of the image pixels have intensities below 0.107. A plot for the intensity lookup table for β=−0.15 for that image is shown in  FIG. 4 . 
         [0034]    In a properly exposed image, shown in  FIG. 5 , the intensity corresponding to a CDF of 0.5 is 0.56. By linear interpolation, the value of β was found to be 0.02. This leads to slight darkening of the image, as apparent from the intensity lookup table in  FIG. 6 . 
         [0035]    The local contrast enhancement  16  is based on amplification of the difference between each pixel intensity and the local mean calculated on a properly selected region of support (e.g. 5×5 for standard definition images, the size and mean calculation can be made adaptive to size and resolution). 
         [0036]    A 5×5 low-pass averaging filter may be used to get a filtered version of the original intensity image. An exponent is applied to the difference of the original and the filtered intensity data while preserving the sign (sgn) of the difference. The operation is depicted by equation 3, where I i  is the input intensity and the parameter φ determines the amount of local enhancement: 
         [0000]    
       
         
           
             
               
                 
                   
                     I 
                     
                       OUT 
                        
                       
                         - 
                       
                        
                       LC 
                     
                   
                   = 
                   
                     
                       sgn 
                        
                       
                         ( 
                         
                           
                             I 
                             i 
                           
                           - 
                           
                             I 
                             F 
                           
                         
                         ) 
                       
                     
                     * 
                     
                       
                          
                         
                           
                             I 
                             i 
                           
                           - 
                           
                             I 
                             F 
                           
                         
                          
                       
                       
                         ( 
                         
                           1 
                           + 
                           
                             φ 
                             100 
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
         [0000]    where I OUT-LC  is the output intensity and I F  is the low-pass filtered value. The operation preserves the sign of the difference, and φ is the local gain adjustable to ˜15% maximum. Local gain φ can be also a function of the specific lightness or each pixel. One embodiment may use a fixed value and a clipping protection mechanism to avoid generating improper values of I, as was done for lightness adjustment, so values near the end of the range are not modified. 
         [0037]    Finally, the output intensity resulting from global lightness adjustment and local contrast enhancement is obtained from summing the two effects in combiner  20  ( FIG. 2 ), as shown in equation 4: 
         [0000]        I   OUT   =I   OUT-GC   +I   OUT-LC   (4) 
         [0000]    where I OUT-LC  is a ΔI value added to the output of I OUT-GC  (output of the global lightness adjustment). 
         [0038]    Saturation enhancement  18  may be achieved by multiplying P and T by a saturation enhancement factor (for example, between 1.0 and 1.4). The saturation factor is applied to a range (for example, between 0.1 and 0.9) with a smooth rolloff at the ends so it will not produce banding artifacts (sudden saturation jumps). 
         [0039]    If the chroma (saturation) of a pixel is below the minimum (min) threshold (achromatic) or above the maximum (max) threshold (already saturated), the saturation is left untouched. Otherwise, the saturation values are increased nonlinearly (using a lookup LUT similar to that of lightness adjustment). The saturaton increase has a peak value of 20% (a maximum of up to 40% can be used for highly lightened images), with smooth roll-off at 0.1 and 0.9 saturation (so they stay unmodified and no illegal values will be generated)-implemented using a LUT as explained below. The maximum peak saturation of 40% is applied if the lightness gain beta is greater or equal to 0.75 of the max beta value paramBetaMax used in the global lightness adjustment (which is 0.15, as explained before). In IPT space, chroma is square root of P 2 +T 2 . 
         [0040]    The output of the block  34  is the image which is saturation enhanced depending on the value of beta. 
         [0041]    Depending on the value of beta, saturation uses a maximum boost, or moderate boost value: 
         [0000]    if ((m_Beta&lt;0) &amp;&amp; (abs(m_Beta)&gt;=0.75 *m _BetaMax)) then compute LUT with maximum boost. Otherwise, compute LUT with normal boost. The saturation LUT implements a function of the form Boost*sin(π*I i ). A 100 point LUT may be used, in some embodiments, for which the bottom and top portions are linear, and set using two threshold values (e.g. 0.1 and 0.9). The curve for a boost of −0.15 is shown in  FIG. 10 . 
         [0042]    In  FIG. 9 , P and T planes of the IPT are received, as indicated in block  24 . The chroma value is calculated as √{square root over (P 2 +T 2 )} (block  26 ), and the enhanced value is taken from the LUT. If the chroma is greater than an upper threshold or lower than a lower threshold (diamond  28 ), then the chroma is unchanged (block  30 ). Otherwise, the enhancement factor is the original chroma and the LUT value. Then, the new P and T values are computed using the enhancement factor to save an inverse computation of P and T from the enhanced chroma value, as indicated in block  32 . 
         [0043]    Similar to the global lightness adjustment, saturation enhancement may be achieved through a lookup table. The use of a lookup table avoids hard thresholds and the problems of artifacts associated with them. If the chroma of a pixel is below the low threshold (for example, 0.1), the color is likely to be achromatic and is left unaltered. Also, if the chroma of a pixel is above the high threshold (for example, 0.9), the color is already quite saturated and so is left unchanged. Colors in between these thresholds are increased in a nonlinear fashion, leading to a lookup table very similar to that shown in  FIGS. 4 and 6 . 
         [0044]    If an image is significantly lightened, the perceived saturation reduces. Actual saturation enhancement in this case needs to be higher to counteract this effect. In one embodiment, an increase in the saturation was 20%, for significantly lightened images the increase was 40%. If the parameter β in equation 2 is negative, with the absolute value equal to or more than 75% of the maximum permissible β (e.g. 0.15), the image is assumed to be significantly lightened, and thus higher saturation enhancement factor may be used. 
         [0045]    In some embodiments, an integrated approach for color and contrast enhancement can be achieved, in parallel, and in one shot. 
         [0046]    The video processing techniques described herein may be implemented in various hardware architectures. For example, video processing functionality may be integrated within a chipset. Alternatively, a discrete video processor may be used. As still another embodiment, the video processing functions may be implemented by a general purpose processor, including a multicore processor. 
         [0047]    References throughout this specification to “one embodiment” or “an embodiment” mean that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one implementation encompassed within the present invention. Thus, appearances of the phrase “one embodiment” or “in an embodiment” are not necessarily referring to the same embodiment. Furthermore, the particular features, structures, or characteristics may be instituted in other suitable forms other than the particular embodiment illustrated and all such forms may be encompassed within the claims of the present application. 
         [0048]    While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.