Abstract:
A noise filter method and apparatus for producing at least one of a video or an image with reduced noise. The noise filter method includes performing noise estimation on a frame of at least one of an image or video and applying a low pass filter on the noise level according to the noise estimation, performing spatial filtration on the frame, performing motion detection on a spatially filtered frame, determining motion-to-blending factor conversion and, accordingly, performing frame blending, and outputting a frame with reduced noise.

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
       [0001]    This application claims benefit of U.S. provisional patent application Ser. No. 61/013,682, filed Dec. 14, 2007, which is herein incorporated by reference. 
     
    
     BACKGROUND OF THE INVENTION 
       [0002]    1. Field of the Invention 
         [0003]    Embodiments of the present invention generally relate to a method and apparatus for a noise filter for reducing noise in a noisy image or video. 
         [0004]    2. Description of the Related Art 
         [0005]    Video and image noise reduction is an important part of video and image processing in both input side and display side of digital consumer electronics. For example, videos captured by digital camcorders, cameras, and video cellular phones under low-light and high ISO gain contain significant amount of noise. Analog video inputs from TV cable and DVD/VCR are also contaminated by transmission noise. The noise not only degrades the video quality, but also hurts the video coding efficiency because the encoder has to spend extra bits to encode the noise. 
         [0006]    Therefore, there is a need for a method and/or apparatus for an improved noise filter that reduces noise in a noisy image or video. 
       SUMMARY OF THE INVENTION 
       [0007]    Embodiments of the present invention relate to a noise filter method and apparatus for producing at least one of a video or an image with reduced noise. The noise filter method includes performing noise estimation on a frame of at least one of an image or video and applying a low pass filter on the noise level according to the noise estimation, performing spatial filtration on the frame, performing motion detection on a spatially filtered frame, determining motion-to-blending factor conversion and, accordingly, performing frame blending, and outputting a frame with reduced noise 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0008]    So that the manner in which the above recited features of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to embodiments, some of which are illustrated in the appended drawings. It is to be noted, however, that the appended drawings illustrate only typical embodiments of this invention and are therefore not to be considered limiting of its scope, for the invention may admit to other equally effective embodiments. 
           [0009]      FIG. 1  an embodiment of a block diagram of a noise filter utilizing both spatial filtration and temporal filtration; 
           [0010]      FIG. 2  is an embodiment of a blending factor controlled by the motion value 
           [0011]      FIG. 3  is an embodiment of an offset α 0  is controlled by the total noise level N total ; 
           [0012]      FIG. 4  is a flow diagram depicting an embodiment of a method for filtering noise utilizing both spatial filtration and temporal filtration; and 
           [0013]      FIG. 5  is a flow diagram depicting an embodiment for generating a spatially filtered frame; and 
           [0014]      FIG. 6  is a flow diagram depicting an embodiment for noise estimation. 
       
    
    
     DETAILED DESCRIPTION 
       [0015]    For the purposes of this application, a computer readable medium is any medium that may be accessed by a computer for reading, writing, executing, and the like of data and/or computer instructions. 
         [0016]    Described herein is a noise filter for video or images that utilizes both spatial filtration and temporal filtration to effectively reduce the noise in noisy videos or images. The filter is adaptive to motion and noise level to achieve constantly good results for moving scenes and videos with changing noise level. The noise filter improves both visual quality and coding efficiency significantly. Even though this application describes the spatial filtration first, the noise estimation may be performed before or simultaneously. 
         [0017]      FIG. 1  an embodiment of a block diagram of a noise filter  100  utilizing both spatial filtration and temporal filtration. The noise filter includes a noise level estimation  102 , a spatial filter  104 , a motion detection  106 , a buffer  108 . 
         [0018]    I(x,y,n) is the input frame  110  and I s (x,y,n) is the output frame  114  of the spatial filter  104 . 
         [0000]        I   s ( x,y,n )= F   s ( I ( x,y,n )). 
         [0019]    The spatial filter F s , of the spatial filter  104  may be applied block-by-block or line-by-line. The spatial filter F s , involves three steps, which are discussed below. Note that the steps described may occur in different order. 
         [0020]    First is the creation of a hierarchical representation. Hence, an h×v-level (horizontally h-level, vertically v-level) hierarchical representation is created of each frame by successive high-pass and low-pass filtration. The representation is a set of coefficient arrays in every level. 
         [0021]    For k-th level, the high-pass filter and low-pass filter are: 
         [0000]        f   L =[1(2 k−1 −1)zeros 1 ], f   H =[1(2 k−1 −1)zeros−1]. 
         [0000]    Without loss of generality, we assume h≧v. Let I 1 =I. Starting from level  1 , for the levels 1≦k≦v, apply the filters in the following way:
       Filter I k  vertically by f L  to create vL k .   Filter I k  vertically by f H  to create vH k .   Filter vL k  horizontally by f L  to create I k+1 .   Filter vL k  horizontally by f H  to create vLhH k .   Filter vH k  horizontally by f L  to create vHhL k .   Filter vH k  horizontally by f H  to create vHhH k .
 
For the levels v&lt;k≦h, apply the filters in the following way:
   Filter I k  horizontally by f L  to create I k+1 .   Filter I k  horizontally by f H  to create hH k .       
 
         [0030]    For different system complexity constraints, we can choose different h and v to create spatial filter F s , of the spatial filter  104  with different size. For example, if h and v are both 3, the size of F s , is 15×15. If h=3 and v=2, the size of F s , is 15×7. If h=2 and v=1, the size of F s , is 7×3. 
         [0031]    Second is the modification of the hierarchical representation. In this step, certain coefficient arrays in k-th level of the hierarchical representation are modified. For levels 1≦k≦v, vLhH k , vHhL k , vHhH k  need to be modified. For levels v&lt;k≦h, hH k  need to be modified. For each of these coefficient arrays that need to be modified, we modify all the elements in them by using the following mapping function: 
         [0000]    
       
         
           
             
               
                 
                   
                     y 
                     = 
                       
                      
                     
                       
                         x 
                          
                         
                             
                         
                          
                         for 
                          
                         
                             
                         
                          
                         x 
                       
                       ≥ 
                       
                         T 
                         k 
                       
                     
                   
                   , 
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       0 
                        
                       
                           
                       
                        
                       for 
                        
                       
                           
                       
                        
                       x 
                     
                     &lt; 
                     
                       
                         T 
                         k 
                       
                       . 
                     
                   
                 
               
             
           
         
       
     
         [0000]    T k  is the threshold of k-th level which is a scaled version of the noise level N f  which will be determined by the noise estimation part. 
         [0000]        T   k   =T   0k   N   f . 
         [0000]    T 0k  is an input strength parameter of the k-th level of the spatial noise filter. Larger T 0k  produces smoother results. Smaller T 0k  keeps more details. The spatial noise filter for frame n can use T k (n−1) if T k (n) may not available before finishing processing frame n. 
         [0032]    Third is the creation of a spatially filtered frame  114 , in which modified hierarchical representation is used to create the spatially filtered frame  114 . For k-th level, the high-pass filter and low-pass filter are: 
         [0000]        f   L =[1(2 k−1 −1)zeros 1 ],f   H =[−1(2 k−1 −1)zeros 1]. 
         [0033]    Starting from level h, for the levels v&lt;k≦h, the filters are applied in the following way:
       Filter I k+1  horizontally by f L  to create hLhL k .   Filter hH k  horizontally by f H  to create hHhH k .   I k =(hLhL k +hHhH k )/4.
 
For the levels 1≦k≦v, apply the filters in the following way:
   Filter I k+1  vertically by f L  to create vLhLvL k.      Filter vLhLvL k  horizontally by h L  to create vLhLvLhL k .   Filter vLhH k  vertically by f L  to create vLhHvL k .   Filter vLhHvL k  horizontally by h H  to create vLhHvLhH k .   Filter vHhL k  vertically by f H  to create vHhLvH k .   Filter vHhLvH k  horizontally by h L  to create vHhLvHhL k .   Filter vHhH k  vertically by f H  to create vHhHvH k .   Filter vHhHvH k  horizontally by h H  to create vHhHvHhH k .   I k =(vLhLvLhL k +vLhHvLhH k +vHhLvHhL k +vHhHvHhH k )/16
 
The spatially filtered frame  114  is I s =I 1 . A color frame contains three channels: Y, U, V. The spatial filter is applied on each color channel independently.
       
 
         [0046]    In addition to accounting for and applying the spatial filter, the noise filter also estimates the noise. The noise estimation contains three steps, which are described herein below. 
         [0047]    First is estimating the noise for each block/line. The frame is processed either block-by-block or line-by-line. So we first estimate a noise level N i  for i-th block or line. In one embodiment, one of two methods may be utilized to estimate N i . One method is based on spatial information and the other is based on temporal information. They can be chosen based on the application. 
         [0048]    In N i  estimation based on spatial information, N i  is the mean absolute value of the coefficient array given at the first level of the hierarchical representation. 
         [0000]        N   i =mean(| vHhH   1i |). 
         [0049]    vHhH 1i  is the i-th block or line of the coefficient array vHhH 1 . 
         [0000]    In the N i  estimation based on temporal information, N i  is the mean absolute difference between the input frame I  110  and a reference frame I p    116 . 
         [0000]        N   i =mean(| I   i   −I   pi |). 
         [0050]    I i  is the i-th block or line of the input frame I  110 . I pi  is the i-th block or line of the reference frame I p    116 . 
         [0051]    Second is estimating noise for a frame. After we have N i  for all i, the noise level of the frame is the mean, or the median, or the minimum of N i . They can be chosen based on the application.
       N=mean(N i ) for all i.   Or N=median(N i ) for all i.   Or N=min(N i ) for all i.       
 
         [0055]    Third, the noise level should change slowly in a video sequence. So a low-pass IIR filter is applied on the noise level. N(n) denotes the noise level of the n-th frame and N f (n) denotes the noise level after the low-pass filtration. 
         [0000]        N   f ( n )=β N   f ( n− 1)+(1−β) N ( n ). 
         [0000]    β is the coefficient of the IIR filter which controls how fast the noise level changes frame-to-frame. The noise estimation is performed on each color channel independently. Each color channel has its own noise level. 
         [0056]    There are three steps for the temporal filtration. The temporal filter can also be applied block-by-block or line-by-line, which are motion detection, Motion-to-blending factor conversion and frame blending. 
         [0057]    In the Motion detection, the reference frame I p (x,y,n)  116  is the previous output frame stored in the buffer  108 . 
         [0000]        I   p ( x,y,n )= I   o ( x,y,n −1). 
         [0000]    The motion value at (x, y) is just the absolute difference between the spatially filtered frame  114  and the reference frame  116  for all three color channels: 
         [0000]        m ( x,y,n )=| I   s     —     Y ( x,y,n )− I   p     —     Y ( x,y,n )|+| I   s     —     U ( x,y,n )− I   p     —     U ( x,y,n )|+| I   s     —     V ( x,y,n )− I   p     —     V ( x,y,n )|. 
         [0000]    I s     —     Y , I s     —     U , I s     —     v  are the three color channels of I s    114  I p     —     Y , I p     —     U , I p     —     V  are the three color channels of I p    116 . 
         [0058]    Since the motion detection is working on the spatially filtered frames I s    114  and the previously filtered frame I p    116 , it is much more robust than the motion detection working on original noisy frames. 
         [0059]    In the motion-to-blending factor conversion step, if there is little motion, the temporal filtration result is more reliable. If there is large motion, the spatial filtration result is more reliable.  FIG. 2  is an embodiment of a blending factor controlled by the motion value. As shown in  FIG. 2 , a blending factor for each pixel at x, y is defined as: 
         [0000]    
       
         
           
             
               
                 
                   
                     α 
                      
                     
                       ( 
                       
                         x 
                         , 
                         y 
                         , 
                         n 
                       
                       ) 
                     
                   
                   = 
                     
                    
                   
                     
                       α 
                       0 
                     
                     + 
                     
                       
                         ( 
                         
                           1 
                           - 
                           
                             α 
                             0 
                           
                         
                         ) 
                       
                        
                       
                         
                           m 
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                               , 
                               n 
                             
                             ) 
                           
                         
                         / 
                         
                           T 
                           m 
                         
                       
                     
                   
                 
               
               
                 
                     
                    
                   
                     
                       
                         if 
                          
                         
                             
                         
                          
                         
                           m 
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                               , 
                               n 
                             
                             ) 
                           
                         
                       
                       &lt; 
                       
                         T 
                         m 
                       
                     
                     , 
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   1 
                 
               
               
                 
                     
                    
                   
                     else 
                     . 
                   
                 
               
             
           
         
       
     
         [0060]    T m  is an input parameter of the temporal filter. Flat areas look smoother when T m  increases. But larger T m  causes more “ghosting” artifacts on moving areas. α 0  is the offset of the motion-blending factor function in  FIG. 2 . 
         [0061]      FIG. 3  is an embodiment of an offset α 0  is controlled by the total noise level N total . As shown in  FIG. 3 , it is controlled by the total noise level of the three color channels: 
         [0000]    
       
         
           
             
               
                 
                   
                     α 
                     0 
                   
                   = 
                     
                    
                   
                     1 
                     - 
                     
                       
                         N 
                         total 
                       
                       / 
                       
                         T 
                         
                           α 
                            
                           
                               
                           
                            
                           0 
                         
                       
                     
                   
                 
               
               
                 
                     
                    
                   if 
                 
               
               
                 
                     
                    
                   
                     
                       N 
                       total 
                     
                     &lt; 
                     
                       T 
                       
                         
                           α 
                            
                           
                               
                           
                            
                           0 
                         
                         , 
                       
                     
                   
                 
               
             
             
               
                 
                   = 
                     
                    
                   0 
                 
               
               
                 
                     
                    
                   
                     else 
                     . 
                   
                 
               
               
                 
                     
                 
               
             
           
         
       
     
         [0000]    N total  is the total noise level of all the three channels: 
         [0000]        N   total   =N   f     —     Y   +N   f     —U     +N   f     —     V . 
         [0062]    T α0  is a register to control the slope of the function in  FIG. 3 . This function makes α 0  to be close to 1 if the noise level is low, and therefore the temporal filter to be very weak to avoid ghosting artifacts. 
         [0063]    In the frame blending, the output frame  112  is an weighted averaging of I s (x,y,n)  114  and I p (x,y,n)  116 : 
         [0000]    
       
         
           
             
               
                 
                   
                     
                       
                         I 
                         o 
                       
                        
                       
                         ( 
                         
                           x 
                           , 
                           y 
                           , 
                           n 
                         
                         ) 
                       
                     
                     = 
                       
                      
                     
                       
                         α 
                          
                         
                             
                         
                          
                         
                           
                             I 
                             s 
                           
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                               , 
                               n 
                             
                             ) 
                           
                         
                       
                       + 
                       
                         
                           ( 
                           
                             1 
                             - 
                             α 
                           
                           ) 
                         
                          
                         
                           
                             I 
                             o 
                           
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                               , 
                               
                                 n 
                                 - 
                                 1 
                               
                             
                             ) 
                           
                         
                       
                     
                   
                   , 
                 
               
             
             
               
                 
                   = 
                     
                    
                   
                     
                       α 
                        
                       
                           
                       
                        
                       
                         
                           I 
                           s 
                         
                          
                         
                           ( 
                           
                             x 
                             , 
                             y 
                             , 
                             n 
                           
                           ) 
                         
                       
                     
                     + 
                     
                       
                         ( 
                         
                           1 
                           - 
                           α 
                         
                         ) 
                       
                        
                       
                         
                           
                             I 
                             p 
                           
                            
                           
                             ( 
                             
                               x 
                               , 
                               y 
                               , 
                               n 
                             
                             ) 
                           
                         
                         . 
                       
                     
                   
                 
               
             
           
         
       
     
         [0064]    The spatial filter may or may not be the same as the image filter used. In one embodiment, the horizontal level and vertical level (u and v) may be different. The image filter used may only handle the case when u=v. 
         [0065]      FIG. 4  is a flow diagram depicting an embodiment of a filtering noise method  400  utilizing both spatial filtration and temporal filtration. The method starts at step  402  and proceeds to step  404 . At step  404 , a new frame is received. At step  406 , the method  400  performs a noise estimation, which is better discussed in  FIG. 1  and  FIG. 6 . At step  408 , the method  400  performs spatial filtration, which is better described in  FIG. 1  and  FIG. 5 . At step  410 , the method performs motion detection, as described in  FIG. 1 . At step  412 , the method  400  performs motion-to-blending factor conversion as described in  FIG. 1 ,  FIG. 2  and  FIG. 3 . At step  414 , the method  400  outputs a filtered frame. At step  418 , the method  400  determines if the frame processed is the last frame. If the frame is not the last frame, the method  400  proceeds from step  418  to step  404 . If there is the last frame, the method  400  ends at step  420 . 
         [0066]      FIG. 5  is a flow diagram depicting an embodiment of a method  500  for generating a spatially filtered frame. The method starts at step  502  and proceeds to step  504 . At step  504 , the method receives new frames. At step  506 , the method creates hierarchical representation. At step  508 , coefficients in k-th level of the created hierarchical representation are modified. At step  510 , the method  500  creates a spatially filtered frame. At step  512 , a spatially filtered frame is outputted. The method  500  ends at step  514 . 
         [0067]      FIG. 6  is a flow diagram depicting an embodiment of a method  600  for noise estimation. The method  600  starts at step  602 . At step  604 , a new frame is received. At step  606 , the method  600  calculates noise level of one or more blocks and/or lines. At step  608 , the method  600  calculates the noise level of the frame. At step  610 , the method  600  applies a low pass filter on the noise level. At step  612 , a noise level is outputted. The method  600  ends at step  614 . 
         [0068]    While the foregoing is directed to embodiments of the present invention, other and further embodiments of the invention may be devised without departing from the basic scope thereof, and the scope thereof is determined by the claims that follow.