Abstract:
The present invention relates to method for image visual effect improvement of video encoding and decoding. The method at an encoder comprises enhancement process of image boundary information, adaptive brightness transformation process, and adaptive enhancement process of chrominance information. The method at a decoder comprises adaptive enhancement of boundary information based on the block statistical characteristic, adaptive brightness transformation process, and adaptive enhancement process of the chrominance information. Based on characteristics of the image source at the encoder and the decoder, the present invention uses brightness space processing technology to eliminate blocking artifacts, enhance high frequency information and improve brightness information distribution, so as to improve brightness information of the image in one aspect, and uses the chroma space processing technology to enhance color effect of the image in the other aspect, and finally, the objective of improving visual effect of the image is achieved.

Description:
FIELD OF THE INVENTION 
       [0001]    The present invention relates to video encoding and decoding methods, more particularly, to a method for image visual effect improvement of video encoding and decoding. 
       BACKGROUND OF THE INVENTION 
       [0002]    Based on human vision system, color can be described by brightness, hue and saturation. Usually, hue and saturation are generally referred to as chroma, which is used to represent the category and depth of color. In the video encoding process, for different frames, regions people cared about are dynamically changed, which requires that the algorithm is able to adjust transformation function according to the change of the video sequences, so that brightness distribution of the image can be improved according to demand in various scenes. The visual quality of the image can be improved by a constant transformation function of brightness with the parameters obtained by considerable statistical experiments. However, if the same approach used in the ordinary scenes is carried out in some specific scenes (such as a wholly dark scene), visual quality of the image will be decreased. 
         [0003]    For color information of an object, people always hope that, the more colorful the better. Considering the requirement of visual comfort, the bigger the transform intensity is, the more color of the image with insufficient chroma information is improved. Skin color of human beings is between yellow and red. If the same model is used for the whole region, taking relatively large adjusting values, uncomfortable feeling to skin color will be generated, and taking relatively small adjusting values, the requirement of enhancing color information of objects in other color gamut will be restricted. If the algorithm is dependent on the detection of skin color regions, firstly, computational complexity is increased, and secondly there isn&#39;t a detection algorithm for skin color regions with 100% accuracy, thirdly many problems such as balance transition brought in by incorrect judgment of discrete point field will occur. Although people are more sensitive to luminance than to chrominance, preprocessing should be employed to enhance the color of the image, since chroma information carried by the image sequence (such as image captured by a camera) processed by the video encoder is insufficient at some time. Most conventional color processing methods are based on RGB or HSV color model, while a separate representation mode of luminance and chrominance, i.e., YUV, is used in video encoding. Although transformation between different models can be realized through color space transformation technology, computational complexity bought in by transformation and invert transformation is also considerable. 
         [0004]    Image quality will be decreased in varying degrees after encoding. Problems, such as blocking artifacts brought in by block-based encoding and decoding strategy, attenuation and losing of high frequency information and so on, are present in the image sequence after decoding. In order to eliminate blocking artifacts without losing of boundary high frequency information, and take characteristics of block-based encoding and decoding strategy into account that the blocking artifacts always present at the boundary between blocks, a method for block-based boundary adaptive enhancement is employed. 
       SUMMARY OF THE INVENTION 
       [0005]    In order to improve visual effect of video sequences at an encoder, the present invention provides a method for image visual effect improvement of video encoding and decoding, wherein a boundary information enhancement technology is used to increase the amount of high frequency information contained in the image, and adaptive enhancement technologies for luminance and chrominance respectively are provided for improving the brightness information distribution of the image and enhancing chroma information of the image. 
         [0006]    The method according to present invention comprises the following steps at the encoder: 
         [0007]    S 11 : extracting image boundary information and enhancing a boundary information operation, the step further comprising:
       S 111 : Extracting boundary information h(x,y) of an image f(x,y),       
 
         [0000]        h ( x,y )=γ( f ( x,y ))
       S 112 : Obtaining a boundary enhanced image g(x,y) by transforming the extracted boundary information h(x,y) and the image f(x,y),       
 
         [0000]        g ( x,y )=φ( f ( x,y ), h ( x,y ))
 
         [0010]    wherein f (x,y) is a brightness value of the original image at the encoder, γ(x) is a boundary information extracting function, φ(f(x,y),h(x,y)) is a transformation function selected according to characteristics of the original image and the boundary information; 
         [0011]    S 12 : Adaptive luminance transforming to improve luminance distribution: 
         [0000]        g ′( x,y )=ψ( f ( x,y ),α( k )| k= 1, 2 , . . . , K ),
 
         [0012]    wherein g′(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is a transformation function, wherein α(k) is a set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters; 
         [0013]    S 13 : Adaptively enhancing the chrominance information, which is performed in the UV color space, 
         [0000]      ( u ′( x,y ),ν′( x,y ))= w* φ( u ( x,y ),ν( x,y ),α u ,α ν ,β u ,β ν )
 
         [0014]    wherein φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α u  and α ν , a chroma adjusting step is determined by β u  and β ν . 
         [0015]    Image quality will be decreased in varying degrees after encoding. Problems, such as blocking artifacts brought in by block-based encoding and decoding strategy, attenuation and losing of high frequency information and so on, are present in the image sequence after decoding. In consideration of a need for improving visual effect of the image at the decoder, a method for image visual effect improvement of video encoding and decoding is provided in the present invention. 
         [0016]    The method according to present invention comprises the following steps at the decoder: 
         [0017]    S 21 : selecting a processing mode 
         [0000]    
       
         
           
             
               t_ 
                
               0 
             
             = 
             
               
                 ∑ 
                 
                   x 
                   , 
                   
                     y 
                     ∈ 
                     
                       0 
                     
                   
                 
               
                
               
                   
               
                
               
                 
                   
                     ℏ 
                     
                       
                           
                       
                        
                       0 
                     
                   
                    
                   
                     ( 
                     
                       f 
                        
                       
                         ( 
                         
                           x 
                           , 
                           y 
                         
                         ) 
                       
                     
                     ) 
                   
                 
                  
               
             
           
         
       
     
         [0000]    according to block statistical characteristic,
       if (t — 0&gt;Thres — 1) mod e=Flat region processing method   else mod e=Complex region processing method
 
then implementing operations for eliminating blocking artifacts and enhancing boundary information based on the determined processing mode;
       
 
         [0020]    wherein f(x,y) is a original image value at the decoder, t — 0 is a statistical variable name of a statistical region          , Å j (j=0, 1, 2) is a statistical characteristic function,            j  is a statistical region corresponding to Å j , and Thres — 1 is a threshold for determining whether the current processing region is a flat region or a complex region; 
         [0021]    S 22 : Adaptively transforming the brightness and improving brightness distribution of the image: 
         [0000]        g ( x,y )=ψ( f ( x,y ),α( k )| k= 1, 2 , . . . , K ),
 
         [0022]    wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K), is a transformation function, wherein α(k) is a set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters; 
         [0023]    S 23 : Adaptively enhancing the chroma information, wherein the chroma information adaptive enhancement is performed in a UV chroma space, 
         [0000]      ( u ′( x,y ),ν′( x,y ))= w* φ( u ( x,y ),ν( x,y ),α u ,α ν ,β u ,β ν )
 
         [0024]    wherein φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α u  and α ν , a chroma adjusting step is determined by β u  and β ν . 
         [0025]    Through the above mentioned method, adaptive adjustment can be applied to eliminate the blocking artifacts, and enhance the image luminance and chrominance information, in such a way the object of improving the objective effect and subjective effect of the coded and decoded images can be achieved. When the adaptive boundary information enhancement technology according to present invention is employed at the decoder, the effect of separation method in enhancing boundary information and eliminating blocking artifacts can be maintained while the processing speed is improved, and the objective effect and subjective effect of the image also can be improved remarkably. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0026]    The following drawings illustrate preferred, but not exclusive embodiments of the inventions: 
           [0027]      FIG. 1  is a flow diagram of a method at the encoder according to an embodiment of the present invention; 
           [0028]      FIG. 2  is a flow diagram of boundary information enhancement shown in  FIG. 1 ; 
           [0029]      FIG. 3  is a flow diagram of adaptive brightness transformation shown in  FIG. 1 ; 
           [0030]      FIG. 4  is a flow diagram of adaptive enhancement of chroma information shown in  FIG. 1 ; 
           [0031]      FIG. 5  is a flow diagram of a method at the decoder according to an embodiment of the present invention; 
           [0032]      FIG. 6  is a contrast diagram showing an original decoded image at the decoder and two images processed by the conventional separation method and the adaptive boundary enhancement respectively using test source in WMV format; 
           [0033]      FIG. 7  is a contrast diagram showing effects of an original decoded image and a processed image by the present invention using test source in WMV format. 
       
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
       [0034]    Referring  FIG. 1 ,  FIG. 1  is a flow diagram of the method for image visual effect improvement of video encoding and decoding at the encoder according to an embodiment of the present invention. The method comprises the following steps at the encoder: 
         [0035]    1. The present step implements boundary information enhancement process, and further comprises the following steps:
       a1: Extracting boundary information h(x,y) of an image f(x,y),       
 
         [0000]        h ( x,y )=γ( f ( x,y ))
       b1: Obtaining a boundary enhanced image g(x,y) by transforming the extracted boundary information h(x,y) and the image f(x,y),       
 
         [0000]        g ( x,y )=φ( f ( x,y ), h ( x,y ))
 
         [0038]    wherein: f (x,y) is a brightness value of the original image at the encoder, φ(f(x,y),h(x,y)) is a transformation function selected according to characteristics of the original image and its boundary information, γ(x) is a boundary information extracting function, wherein different methods for extraction can be employed according to different applications requirements. With respect to derivative method, for example, a first order derivative, a second order derivative and so on can be employed, such as gradient module extracting method: 
         [0000]    
       
         
           
             
                
               
                 grad 
                  
                 
                   ( 
                   
                     f 
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   ) 
                 
               
                
             
             = 
             
               
                 max 
                 l 
               
                
               
                 ( 
                 
                   
                     ∂ 
                     
                       f 
                        
                       
                         ( 
                         
                           x 
                           , 
                           y 
                         
                         ) 
                       
                     
                   
                   
                     ∂ 
                     l 
                   
                 
                 ) 
               
             
           
         
       
     
         [0039]    2. In video encoding process, for different frames, regions people cared about are dynamically changed, which requires that the algorithm is able to adjust transformation function according to the change of the video sequences, so that brightness distribution of the image can be improved according to demand in various scenes. 
         [0040]    The visual quality of the image can be improved by a constant transformation function of brightness with the parameters obtained by considerable statistical experiments. However, if the same approach used in the ordinary scenes is carried out in some specific scenes (such as a wholly dark scene), visual quality of the image will be decreased. 
         [0041]    The present step implements adaptive brightness transformation and improvement of image brightness distribution. The principle of adaptive brightness transformation is that, the set of parameters of the transformation function is adaptively updated according to a statistical characteristic of brightness value of the image before being transformed, so that the transformation function is adjusted dynamically along with different image characteristics, and thus the processing method is optimized: 
         [0000]        g ( x,y )=ψ( f ( x,y ),α( k )| k= 1, 2 , . . . , K ),
 
         [0042]    wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters. 
         [0043]    As shown in  FIG. 3 , step  2  further comprises the following steps: 
         [0044]    a2: Given a characteristic space of the current frame image is 
         [0000]    
       
         
           
             
               ξ 
               = 
               
                 
                   ⋃ 
                   
                     k 
                     = 
                     1 
                   
                   M 
                 
                  
                 
                   ξ 
                   k 
                 
               
             
             , 
           
         
       
     
         [0000]    ξ i ∩ξ j =φ,i≠j, and a whole statistical characteristic of the image is obtained by statistic of the brightness information, 
         [0045]    for(k=0; k&lt;M; k++) if(f(x,y)εξ k ) Calculating a statistical characteristic φ k (f(x,y)) of ξ k . 
         [0046]    Finally, the statistical characteristic of the current frame image is obtained: 
         [0000]      {φ k ( f ( x,y ))| k= 1, 2 , . . . , M};  
 
         [0047]    wherein ξ k  and φ k (f(x,y)) are image characteristic subspace, the statistical characteristic of ξ k  respectively; 
         [0048]    b2: The threshold is adjusted according to visual characteristic together with the regional statistical characteristic, and the image is divided into different regions 
         [0000]    
       
         
           
             Ω 
             = 
             
               
                 ⋃ 
                 
                   k 
                   = 
                   1 
                 
                 N 
               
                
               
                 
                   Ω 
                   k 
                 
                 . 
               
             
           
         
       
     
         [0000]    The statistical characteristic threshold P H  is adjusted to be P H ′ according to a statistical relationship between the global area and the regions, 
         [0000]        P   H ′=ratio*η( P   H , Φ 1 , Φ 2 , . . . , Φ N )
 
         [0049]    wherein Φ k  is a statistical characteristic of Ω k , 
         [0000]      Φ k ={φ 1 (Ω k ), φ 2 (Ω k ), . . . , φ M (Ω k )}
 
         [0050]    P H′  P H ′ are the threshold obtained through the whole statistical information and the adjusted threshold respectively; 
         [0051]    c2: Parameter values of the transformation function is obtained based on the statistical characteristic; 
         [0000]      α( k )=Å k ( P   H ′) k= 1, 2 , . . . K  
 
         [0052]    Å k (x) is a adjusting function of the parameter α(k) of the transformation function ψ(x,α(k)|k=1, 2, . . . , K); 
         [0053]    d2: By using brightness transformation function ψ(f(x,y),α(k)|k=1, 2, . . . , K), brightness transformation is implemented and distribution of the image brightness information is improved; 
         [0054]    wherein f(x,y) is the brightness value of the original image at the decoder, g(x,y) is the adjusted brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), K is the number of the parameters. 
         [0055]    3. Chroma information is adaptively enhanced, wherein the chroma information adaptive enhancement is performed in a UV chroma space, 
         [0000]      ( u ′( x,y ),ν′( x,y ))= w* φ( u ( x,y ),ν( x,y ),α u ,α ν ,β u ,β ν )
 
         [0056]    wherein φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α u  and α ν , a chroma adjusting step is determined by β u  and β ν . 
         [0057]    As shown in  FIG. 4 , following steps are further comprised: 
         [0058]    a3: Saturation information of the UV space κ is obtained through a statistic of the UV characteristic of the current image frame; 
         [0059]    b3: Adjusting parameters are calculated with the color saturation information; 
         [0000]      α u =γ u (κ)β u =γ u (κ)
 
         [0000]      α ν =γ ν (κ)β ν =γ ν (κ)
 
         [0060]    c3: By statistical experiments in the UV space model, empirical value range of skin color distribution is obtained, and the weight function w=η(θ) is determined, wherein θ is the empirical value range of skin color, θε[θ 1 ,θ 2 ]. 
         [0061]    w=η(θ), η(θ) is a continuous function having only one minimum value, and w min =η((θ 1 +θ 2 )/2). 
         [0062]    d3: Chroma transformation is implemented using the chroma transformation function (u′(x,y),ν′(x,y))=w*φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ), and the chroma information of the image is enhanced; 
         [0063]    wherein φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ) is the transformation function, w is the weight function, the UV chroma deviation position is determined by α u  and α ν , and the chroma adjusting step is determined by β u  and β ν . 
         [0064]    Although transformation between different models can be realized through color space transformation technologies, computational complexity bought in by the transformation and invert transformation is also considerable. Considering the document format processed by the encoder, format conversation time should be reduced. The present invention implements a color information process directly in the UV chroma space. 
         [0065]    Referring  FIG. 5 ,  FIG. 5  is a flow diagram of the method for image visual effect improvement of video encoding and decoding at the decoder according to an embodiment of the present invention. The method comprises the following steps at the decoder: 
         [0066]    10. Selecting a processing mode 
         [0000]    
       
         
           
             
               t_ 
                
               0 
             
             = 
             
               
                 ∑ 
                 
                   x 
                   , 
                   
                     y 
                     ∈ 
                     
                       0 
                     
                   
                 
               
                
               
                 
                   ℏ 
                   0 
                 
                  
                 
                   ( 
                   
                     f 
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                       
                       ) 
                     
                   
                   ) 
                 
               
             
           
         
       
     
         [0000]    according to block statistical characteristic
       if(t — 0&gt;Thres — 1) mod e=Flat region processing method   else mod e=Complex region processing method       
 
         [0069]    Then, operations for eliminating blocking artifacts and enhancing boundary information are implemented based on the determined processing mode. 
         [0070]    For processing of flat regions: 
         [0000]        t   — 1 j =Å 1 ( f ( x,y )| f ( x,y )ε           j )
 
         [0000]      if( t   — 1 j   &gt;Thres   — 2)weighted lowpass filtering 
         [0000]      else protecting the boundary information 
         [0071]    wherein t — 1j is a statistical characteristic variable name of the j-th flat region            j , Thres — 2 is a threshold for the currently processed flat region processed by the selected different processing methods. 
         [0072]    For processing of complex regions: 
         [0000]        t   — 2 j =Å 2 ( f ( x,y )| f ( x,y )ε           j )
 
         [0000]      if( t   — 2 j   &gt;Thres   — 3) 
         [0000]        f ( x   M   ,y   N |( x   m   ,y   n )εblock j   , m= 1, 2 , . . . , M, n= 1, 2 , . . . , N )−=κ;
 
         [0000]        f ( x   1   ,y   1 |( x   m   ,y   n )εblock j+1   , m= 1, 2 , . . . , M, n= 1, 2 , . . . , N )+=κ;
 
         [0000]      else enhancing the boundary information. 
         [0073]    wherein f(x,y) is a original image value at the decoder, t — 0 is a statistical variable name of a statistical region            0 , Å j  (j=0, 1, 2) is a statistical characteristic function,            j  is a statistical region corresponding to Å j , and Thres — 1 is a threshold for determining whether the current processing region is a flat region or a complex region; t — 2j is a statistical characteristic variable name of the j-th complex region            j , Thres — 3 is a threshold for the currently processed complex region processed by the selected different processing methods. 
         [0074]    The following table shows comparative experiment between the adaptive boundary information enhancement process of the present step and the separation method process in the prior art. The experiment uses images with a source size of 320×240, and same decoders are used. Comparison of objective effects is as follows: 
         [0000]    
       
         
               
               
             
               
               
               
               
             
               
               
             
               
               
               
               
             
           
               
                   
                   
               
               
                   
                 processing method 
               
             
          
           
               
                   
                   
                 Adaptive 
                 Increase of 
               
               
                   
                 Separation 
                 boundary information 
                 processing 
               
               
                 Test 
                 method 
                 enhancement method 
                 speed 
               
             
          
           
               
                 sequence 
                 Speed (fps) 
               
               
                   
               
             
          
           
               
                 Test1_WMV 
                 187 
                 280 
                 49.7% 
               
               
                 Test3_FLV 
                 181 
                 253 
                 39.8% 
               
               
                 Test4_FLV 
                 182 
                 249 
                 36.8% 
               
               
                 Test2_FLV 
                 188 
                 252 
                 34.0% 
               
               
                   
               
             
          
         
       
     
         [0075]    It can be seen from the above table that, processing speed can be significantly improved by the adaptive boundary information enhancement method in accordance with the present invention. 
         [0076]      FIG. 6  is a contrast diagram showing an original decoded image at the decoder and two images processed by the conventional separation method and the adaptive boundary enhancement respectively using test source in WMV format. Through comparison of the objective effect, the objective quality of the image is significantly improved by the image processing method of adaptive boundary enhancement according to the present invention. 
         [0077]    20. Brightness of the image is adaptively transformed, and the brightness distribution of the image is improved. The principle of adaptive brightness transformation is that, the set of parameters of the transformation function is adaptively updated according to a statistical characteristic of brightness value of the image before being transformed, so that the transformation function is adjusted dynamically along with different image characteristics, and thus the processing method is optimized: 
         [0000]        g ( x,y )=ψ( f ( x,y ),α( k )| k= 1, 2 , . . . , K ),
 
         [0078]    wherein f(x,y) is a brightness value of the original image at the decoder, g(x,y) is a transformed brightness value, ψ(x,α(k)|k=1, 2, . . . , K) is the transformation function, wherein α(k) is the set of parameters of the transformation function ψ(x,α(k)|k=1, 2, . . . , K), and K is the number of the parameters. 
         [0079]    30. Chroma information is adaptively enhanced, wherein the chroma information adaptive enhancement is performed in a UV chroma space, 
         [0000]      ( u ′( x,y ),ν′( x,y ))= w* φ( u ( x,y ),ν( x,y ),α u ,α ν ,β u ,β ν )
 
         [0080]    wherein φ(u(x,y),ν(x,y),α u ,α ν ,β u ,β ν ) is a transformation function, w is a weight function, and a UV chroma deviation position is determined by α u  and α ν , a chroma adjusting step is determined by β u  and β ν . 
         [0081]      FIG. 7  is a contrast diagram showing effects of an original decoded image and a processed image by the present invention using test source in WMV format. After implementing the step  10 , step  20  and step  30  of the above-mentioned method, blocking artifacts are eliminated, luminance and chrominance information of the image are enhanced by adaptive adjustment, so that the object of improving objective effect and subjective effect of the decoded image is achieved. 
         [0082]    The foregoing description of the exemplary embodiments of the invention has been presented only for the purposes of illustration and description and is not intended to be exhaustive or to limit the invention to the precise forms disclosed. Many modifications and variations are possible in light of the above teaching without departing from the protection scope of the present invention.