Abstract:
A method includes selecting a target pixel and comparing a value of the target pixel with a respective value of each of a plurality of pixels located in an area that includes the target pixel. Further, for each pixel of the plurality of pixels that has a value different by at least a threshold amount from the value of the target pixel, the value of such pixel is replaced by the value of the target pixel. A filter function is applied to a set of pixels which includes the value of the target pixel and current values, after the selective replacement step, of the plurality of pixels.

Description:
BACKGROUND 
       [0001]    It is known to perform noise reduction processing on video signals to improve perceived image quality by mitigating the effects of noise that typically is present in the video signals. Sources of noise include compression encoding/decoding, which may result in various types of artifacts such as block noise, ringing noise, mosquito noise and transform noise. Other sources of noise may include video capture processing, analog-to-digital conversion, and signal transmission. 
         [0002]    Noise reduction processing may be performed after video signal decompression (decoding) for purposes of image quality improvement, and may also be performed prior to transmission or recording, to avoid wasting bandwidth on transmission/recording of artifacts as well as for quality considerations. 
         [0003]    Typical kinds of noise reduction processing involve low pass filtering of the video signal. Although generally worthwhile to mitigate artifacts, such filtering may also blur the image to some extent, so that noise reduction processing may entail trading off one type of distortion for another. Computational complexity may also be a drawback in noise reduction processing. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0004]      FIG. 1  is a block diagram of an apparatus which generates a video bitstream from captured images in accordance with some embodiments. 
           [0005]      FIG. 2  is a block diagram of an apparatus which generates a video display from a video bitstream in accordance with some embodiments. 
           [0006]      FIG. 3  is a functional block diagram that illustrates some operations of a noise reduction block that is part of the apparatus of  FIG. 1  or  FIG. 2 . 
           [0007]      FIG. 4  is a flow chart that illustrates a process performed by the noise reduction block. 
           [0008]      FIG. 5  schematically illustrates aspects of the process of  FIG. 4 . 
           [0009]      FIG. 6  is a functional block diagram that illustrates some operations of the noise reduction block in some other embodiments. 
           [0010]      FIGS. 7A and 7B  together form a flow chart that illustrates a process that may be performed in connection with the functional arrangement of  FIG. 6 . 
           [0011]      FIG. 8  graphically illustrates an aspect of the process of  FIGS. 7A and 7B . 
           [0012]      FIG. 9  is a functional block diagram that illustrates aspects of noise reduction filtering performed according to still other embodiments in the apparatus of  FIG. 2 . 
           [0013]      FIGS. 10A and 10B  together form a flow chart that illustrates a process that may be performed in connection with the functional arrangement of  FIG. 9 . 
       
    
    
     DETAILED DESCRIPTION 
       [0014]      FIG. 1  is a block diagram of an apparatus  100  which generates a video bitstream from captured images in accordance with some embodiments. 
         [0015]    The apparatus  100  includes a video signal capture device  102  such as a digital video camera. The apparatus  100  further includes a pre-processing block  104  which is coupled to the video signal capture device  102 . The pre-processing block  104  performs one or more kinds of pre-processing on the captured video signal provided by the video signal capture device  102 . For example, the pre-processing block  104  may perform one or more different kinds of noise reduction processing as in one of the embodiments described below. 
         [0016]    The apparatus  100  also includes a compression encoding block  106  which is coupled to the pre-processing block  104 . The compression encoding block  106  may apply compression encoding in accordance with conventional practices to the pre-processed video signal provided by the pre-processing block  104 . 
         [0017]    In addition, the apparatus  100  may include a transmitter  108  coupled to the compression encoding block  106  to transmit the compression encoded video signal in the form of a video signal bitstream over a communication channel which is not separately shown. 
         [0018]    Details of embodiments of the pre-processing block  104  will be discussed below. 
         [0019]      FIG. 2  is a block diagram of an apparatus  200  which generates a video display from a video bitstream in accordance with some embodiments. 
         [0020]    The apparatus  200  includes a source  202  of a video signal bitstream. For example, the video signal source  202  may receive a video signal via a communication channel (which is not separately shown) or may reproduce a video signal from a storage medium such as a DVD or a hard disk drive. For example, the video signal source may include a video tuner, a satellite earth station, or a DVD player. It will be assumed that the video signal bitstream represents a video signal that has been compression encoded, e.g., in accordance with one of the MPEG compression standards. The video signal source  202  may operate in accordance with conventional practices. 
         [0021]    The apparatus  200  also includes a video decoder  204  which is coupled to the video signal source to de-compress the video signal bitstream supplied by the video signal source  202 . The video decoder  204  may operate in accordance with conventional principles, and may tend to produce artifacts in the output video image, subject to amelioration via embodiments to be described below. 
         [0022]    The apparatus  200  further includes a post-processing block  206  which is coupled to the video decoder  204 . The post-processing block  206  performs one or more kinds of post processing on the decompressed video signal output from the video decoder  204 . For example, the post-processing block  206  may perform one or more different kinds of noise reduction processing as in one or more of the embodiments described below. 
         [0023]    In addition, the apparatus  200  includes a display device  208 , such as a conventional television set or a computer display monitor. The display device  208  displays the video signal that is output from the post-processing block  206 . 
         [0024]      FIG. 3  is a functional block diagram that illustrates aspects of a noise reduction process performed in the pre-processing block  104  or in the post-processing block  206  in accordance with some embodiments. 
         [0025]    Block  302  represents the pre-processing block  104  or the post-processing block  206  (or a noise reduction sub-block) receiving an input video signal (e.g., directly or indirectly from the video capturing device  102 , or directly or indirectly from the video decoder  204 ) that is to be subjected to noise reduction processing. Block  304  represents circuitry (specially designed or programmed) provided in accordance with some embodiments to apply a filtering process that may be referred to as a “content adaptive substitute filtering” process. As will be described below in more detail, prospective pixel values to be used for filtering are (as indicated by block  306 ) compared with target pixel values and are replaced in some cases to support improved low-pass filtering (indicated by block  308 ). Output of the resulting low-pass filtered video signal is indicated at block  310 . 
         [0026]      FIG. 4  is a flow chart that illustrates an embodiment of a content adaptive substitute filtering process that may be performed in block  304  ( FIG. 3 ). Referring now to  FIG. 4 , block  402  represents block  304  receiving the input video signal (e.g., after video capture or after de-compression of a previously compression-encoded video signal). Block  404  represents selection of a target pixel x from the input video signal. The target pixel x is the pixel for which a filtered output value is now to be calculated. In a particular example of content adaptive substitute filtering, the filter output value for each target pixel is determined on the basis of pixel values in a filter support neighborhood area that is a three-pixel-by-three-pixel square centered on the target pixel x. This area may be referred to as NH(x) or, since the total area corresponds to 9 pixels, as NH9(x). (In some embodiments, a filter support area other than three-pixels-by-three-pixels may be used. For example, filter support areas of four-pixels-by-four-pixels, five-pixels-by-five-pixels, six-pixels-by-six-pixels or seven-pixels-by-seven-pixels may be used in alternative embodiments.) 
         [0027]    At  406  in  FIG. 4 , the value of one of the neighboring pixels y in NH9(x) is compared with the value of pixel x. At  408  it is determined whether the absolute value of the difference between the two pixel values is at least as great as a threshold amount TH(x). (In the particular embodiment now being discussed, TH(x) may be a constant that is used in connection with all target pixels, but in other embodiments, as described below, TH(x) may vary from target pixel to target pixel. In some embodiments, the constant used for TH(x) may be programmable by a user or designer of the apparatus  100  or  200 .) 
         [0028]    If at  408  it is determined that the absolute value of the difference between the two pixel values is at least as great as TH(x), then, at  410 , the value of neighboring pixel y is replaced with the value of the target pixel x for the purposes of applying a filtering function to calculate an output filtered pixel value for the target pixel. If at  408  it is determined that the absolute value of the difference between the two pixel values is less than TH(x), then the value of the neighboring pixel is not replaced. Following  408  in the latter case, or following  410 , as the case may be, is a decision block  412 , at which it is determined whether there remain any neighboring pixels in NH9(x), which have not been compared with the target pixel. If such is the case, then the process of  FIG. 4  loops back to  406  for consideration of another neighboring pixel in NH9(x). If such is not the case, the process proceeds to apply a low-pass filter characteristic to generate a filtered output value for the target pixel, as indicated at  414  and as will be discussed further below. 
         [0029]      FIG. 5  schematically illustrates an example of how the loop  406 - 412  may operate in connection with a particular target pixel x. The original condition of NH(x) is indicated at  502 , with the target pixel x being represented by the solid black pixel at the center of the array shown at  502 . The hollow circles in the array  502  represent neighboring pixels for which the values differ from the value of pixel x (if at all) by less than TH(x). The shaded circles in the array  502  represent neighboring pixels for which the values differ from the value of pixel x by at least TH(x). As indicated schematically at  504 , the values of the latter neighboring pixels are replaced by the value of the target pixel x to form a substitute filter support array SNH(x). The set of pixel values represented by SNH(x) includes the value of the target pixel and “current values” of the neighboring pixels, where the current value of a neighboring pixel is the original value of the pixel if the value has not undergone replacement and is the value of the target pixel if replacement has occurred. By operation of this compare and replace process, outlier pixel values are removed from the filter support area so that more satisfactory filter results may be obtained. At the same time, the outlier values are each replaced with the target pixel value, so that the number of input values for the filtering function is the same for each target pixel, thereby reducing complexity of hardware implementation. This is in contrast to the known “sigma filter”, in which outlier values are simply discarded, leading to varying numbers of input values and complex implementation. 
         [0030]    Referring again to  412  in  FIG. 4 , in some embodiments the filter function to be applied to generate the output filtered value for the target pixel x may be defined by the following weight matrix: 
         [0000]    
       
         
           
             
               [ 
               
                 
                   
                     1 
                   
                   
                     2 
                   
                   
                     1 
                   
                 
                 
                   
                     2 
                   
                   
                     4 
                   
                   
                     2 
                   
                 
                 
                   
                     1 
                   
                   
                     2 
                   
                   
                     1 
                   
                 
               
               ] 
             
               
           
         
       
     
         [0031]    To reduce the complexity of the calculations, this two-dimensional matrix may be decomposed into two one dimensional matrices: 
         [0000]    
       
         
           
             
               [ 
               
                 
                   
                     1 
                   
                 
                 
                   
                     2 
                   
                 
                 
                   
                     1 
                   
                 
               
               ] 
             
             * 
             
               [ 
               
                 1 
                  
                 
                     
                 
                  
                 2 
                  
                 
                     
                 
                  
                 1 
               
               ] 
             
           
         
       
     
         [0032]    (Filter characteristics other than that indicated above may be used in other embodiments. For example, filter characteristics suitable for other sizes of filter support may be used.) 
         [0033]    In some embodiments, the output filtered value for the target pixel x may be calculated as follows: 
         [0000]    
       
         
           
             
               
                 ( 
                 
                   1 
                   / 
                   16 
                 
                 ) 
               
               * 
               
                 ( 
                 
                   
                     ( 
                     
                       
                         
                           ∑ 
                           
                             w 
                              
                             
                               ( 
                               y 
                               ) 
                             
                           
                         
                         
                           y 
                           ∈ 
                           
                             NH 
                              
                             
                                 
                             
                              
                             9 
                              
                             
                               ( 
                               x 
                               ) 
                             
                           
                         
                       
                       * 
                       
                         { 
                         y 
                         } 
                       
                     
                     ) 
                   
                   + 
                   8 
                 
                 ) 
               
             
             , 
           
         
       
     
         [0034]    where w(y) is the value of the weighting matrix at the position of the pixel y∈NH9(x), and {y} is the value of that pixel. It will be noted that 16 is the summation over the weighting matrix and 8 is one-half of that summation, the latter term being applied for purposes of rounding. 
         [0035]    In other embodiments, other filter functions may be applied. Also, in other embodiments, other sizes and/or shapes of filter support areas may be employed rather than the above-described three-pixel-by-three-pixel filter support area. 
         [0036]    Once the filter function has been applied to SNH(x) and the resulting output filtered value for the target pixel has been calculated, the output filtered value may be stored, as indicated at  416  in  FIG. 4 . It is next determined, at decision block  418 , whether output filtered values have been obtained for all of the target pixels in the image that is being processed. If not, the process of  FIG. 4  loops back to  404 . Otherwise the process is complete. It will be appreciated that the content adaptive substitute filter process illustrated in  FIG. 4  may be applied to each image represented by the input video signal. 
         [0037]    The content adaptive substitute filtering arrangement of  FIGS. 3-5  may provide improved noise reduction filtering with a low degree of implementation complexity. 
         [0038]      FIG. 6  is a functional block diagram that illustrates aspects of a noise reduction process performed in the pre-processing block  104  or in the post-processing block  206  in accordance with some other embodiments. In these other embodiments, operation of the content adaptive substitute filtering may be modified to incorporate adaptation of the substitution threshold amount TH(x) by taking into consideration the presence of edge conditions at the target pixel x. 
         [0039]    Referring to  FIG. 6 , the functional blocks that were discussed above in connection with  FIG. 3  are also present in  FIG. 6 , except that blocks  304 ′ and  306 ′ in  FIG. 6  reflect capability to modify in the value of TH(x) from target pixel to target pixel. In addition, the arrangement of  FIG. 6  includes a functional block  602  which performs edge detection processing at each target pixel. 
         [0040]      FIGS. 7A and 7B  together form a flow chart that illustrates an embodiment of an edge-sensitive content adaptive substitute filtering process that may be performed by blocks  602  and  304 ′ of  FIG. 6 . The process of  FIGS. 7A and 7B  may begin with the same blocks  402  and  404  discussed above in connection with  FIG. 4 . In addition, the process of  FIGS. 7A and 7B  includes a block  702  at which an edge metric EM(x) is calculated with respect to the target pixel x. In some embodiments, the so-called Sobel edge detector may be employed, using the following matrices: 
         [0000]    
       
         
           
             E_h 
             = 
             
               
                 
                   [ 
                   
                     
                       
                         
                           - 
                           1 
                         
                       
                       
                         
                           - 
                           2 
                         
                       
                       
                         
                           - 
                           1 
                         
                       
                     
                     
                       
                         0 
                       
                       
                         0 
                       
                       
                         0 
                       
                     
                     
                       
                         1 
                       
                       
                         2 
                       
                       
                         1 
                       
                     
                   
                   ] 
                 
                  
                 
                     
                 
                  
                 and 
                  
                 
                     
                 
                  
                 E_v 
               
               = 
               
                 [ 
                 
                   
                     
                       
                         - 
                         1 
                       
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                   
                     
                       
                         - 
                         2 
                       
                     
                     
                       0 
                     
                     
                       2 
                     
                   
                   
                     
                       
                         - 
                         1 
                       
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                 
                 ] 
               
             
           
         
       
     
         [0041]    The edge metric value may be calculated as follows as the convolution of the edge detection weighting matrices with the 3×3 neighborhood NH9(x) of the target pixel: 
         [0000]        EM ( x )=| NH 9( x )* E   —   h|+|NH 9( x )* E   —   v|   
         [0042]    At  704  an edge-related substitution threshold ETH(x) is calculated to be used (in place of the above described threshold TH(x)) in a content adaptive substitute filtering process to be applied to the target pixel. ETH(x) may be a function of the calculated edge metric EM(x) for the target pixel; i.e.: 
         [0000]        ETH ( x )= f ( EM ( x ))= C*EM ( x )+ Th   — 0, 
         [0043]    where C and Th — 0 may be pre-determined (and/or programmable) constants, and ETH(x) is constrained to be non-negative, Th — 0 is greater than zero and C is less than zero.  FIG. 8  graphically illustrates ETH(x). It will be appreciated that with increasing edge strength at the target pixel, the substitution threshold is reduced, so that the amount of filtering at the target pixel may be reduced. If the edge metric EM(x) reaches or exceeds a saturation point EM_s, the substitution threshold is reduced to zero and no filtering occurs at the target pixel. Thus strong edges are preserved unfiltered and blurring due to noise reduction filtering may be reduced. 
         [0044]    Other edge detection processes besides the above-described Sobel edge detector may be employed in place of the Sobel edge detector in some embodiments. For example, the Canny edge detector may be used. 
         [0045]    The balance of the process blocks in the process of  FIGS. 7A and 7B  may be the same as the blocks described above in connection with  FIG. 4 . In regard to the process of  FIGS. 7A and 7B  and other processes described herein, the illustrations and descriptions of the processes are not intended to imply a fixed order of process stages. Rather, the process stages may be performed in any order that is practicable. For example, instead of calculating the edge metric and edge-sensitive substitution threshold for each target pixel as it is selected, one or both of edge detection and threshold calculation may be performed for all target pixels in an image before substituting any neighboring pixel values and performing filtering with respect to any target pixel in the image. 
         [0046]      FIG. 9  is a functional block diagram that illustrates aspects of a noise reduction process performed in the post-processing block  206  in accordance with still other embodiments. In the embodiments now to be described in connection with  FIGS. 9 ,  10 A and  10 B, determination of the substitution threshold reflects both edge conditions at the target pixel and a degree of quantization previously applied during compression encoding to a now-decoded (decompressed) video signal that is to be filtered for noise reduction purposes. 
         [0047]      FIG. 9  shows blocks  202  and  204  discussed above in connection with  FIG. 2 , blocks  302  and  310  discussed in connection with  FIG. 3  and block  602  discussed in connection with  FIG. 6 . In addition, the content adaptive substitute block (now labeled  304 ″), and in particular its constituent compare and replace block (now labeled  306 ″) are also present, with the compare and replace block  306 ″ having been modified to receive from the video decoder block  204  a signal that indicates a degree of quantization employed in the compression-encoding of the video signal decoded by block  204 . The compare and replace block  306 ″ generates substitution threshold values based both on edge conditions detected by block  602  and on the signal from the video decoder  204  to indicate the degree of quantization. 
         [0048]      FIGS. 10A and 10B  together form a flow chart that illustrates an embodiment of a decoder-assisted edge-sensitive adaptive substitute filtering process that may be performed by blocks  602  and  304 ″ of  FIG. 9 . The process of  FIGS. 10A and 10B  may begin with the same block  402  as discussed in connection with  FIG. 4 . At  1002  in  FIG. 10A , the compare and replace block  306 ″ receives from the video decoder  204  a quantization parameter signal QP(x) which increases in proportion to the degree of quantization (coarseness of quantization) that was applied during compression encoding of the video signal decoded by the video decoder  204 . 
         [0049]    The process of  FIGS. 10A and 10B  further includes block  404  discussed above in connection with  FIG. 4 , and block  702  discussed above in connection with  FIG. 7A . At  1004  a quantization-adaptive edge-related substitution threshold ETH(x) is calculated to be used in a content adaptive substitute filtering process to be applied to the target pixel. In some embodiments, ETH(x) may be calculated according to the following formula: 
         [0000]        ETH ( x )= C*EM ( x )+ K*QP ( x ), 
         [0050]    where K is a pre-determined (and/or programmable) constant. Both K and QP(x) are greater than zero in this example; C and EM(x) are as described in connection with  702  and  704  of  FIG. 7A . With this arrangement, the degree of filtering that may be performed by block  304 ″ may increase with the degree of quantization applied to the video signal during compression encoding. This may be desirable, since an increased amount of coding noise may be present where more quantization was applied. 
         [0051]    The balance of the process blocks in the process of  FIGS. 10A and 10B  may be the same as the blocks described above in connection with  FIG. 4 . 
         [0052]    In some embodiments, edge detection may be omitted or may not be present and/or C may be set to zero, so that the substitution threshold is quantization-adaptive but not edge-related. 
         [0053]    The edge detection block and/or the content adaptive substitute filtering block, or other blocks herein, may be implemented as application-specific logic circuitry or by one or more programmable processors controlled by software instructions stored in a memory or memories coupled to the processor or processors. 
         [0054]    The several embodiments described herein are solely for the purpose of illustration. The various features described herein need not all be used together, and any one or more of those features may be incorporated in a single embodiment. Therefore, persons skilled in the art will recognize from this description that other embodiments may be practiced with various modifications and alterations.