Abstract:
An optimal denoising method for video coding. This method makes use of very few pixels and linear operations, and can be embedded into the motion compensation process of video encoders. This method is simple and flexible, yet offers high performance and produces appealing pictures.

Description:
CROSS-REFERENCE TO OTHER APPLICATION 
     The present application claims priority from U.S. provisional application No. 60/801,375 filed May 19, 2006, which is hereby incorporated by reference. 
    
    
     BACKGROUND AND SUMMARY OF THE INVENTIONS 
     The present application relates generally to digital signal processing, and more particularly to a video compression, coding, filtering and representation system utilizing temporal redundancy and having both device and method aspects. It further relates to a computer program product, such as a recording medium, carrying program instructions readable by a computing device to cause the computing device to carry out methods according to the inventions. 
     BACKGROUND 
     Noising in Video Coding 
     The importance of video technology is constantly growing with the ever increasing use of television and video systems in consumer, commercial, medical and scientific applications. 
     Due to the huge size of the raw digital video data (or image sequences), compression must be applied to such data so that they may be transmitted and stored. Hybrid video coding scheme is the most common video coding scheme. It utilizes motion estimation (ME), discrete cosine transform (DCT)-based transform and entropy coding to exploit temporal, spatial and data redundancy, respectively. Most of the existing video coding standards conform to this hybrid scheme, such as the ISO/IEC MPEG-1, MPEG-2, MPEG-4 standards, the ITU-T H.261, H.263, H.264 standards and AVS or related video coding. 
     Video coding performance is typically evaluated in terms of two measures: coding distortion and compression capability. Due to internal quantization step, the hybrid coding scheme is a lossy coding process, which means that some information is lost in the coding process. Therefore, the decoded (reconstructed) video has some distortion compared to the original (uncoded) video. The best measure of coding distortion is subjective visual quality of the reconstructed video. But subjective quality evaluation is time consuming and requires many observers (both trained and untrained). An alternative is to use objective visual quality measure. A common objective visual quality measure is the peak signal-to-noise ratio (PSNR), defined as: 
             PSNR   =     20   ⁢           ⁢   log   ⁢           ⁢     255       [       1   MNT     ⁢       ∑     t   =   1     T     ⁢       ∑     i   =   1     M     ⁢       ∑     j   =   1     N     ⁢       [       f   ⁡     (     i   ,   j   ,   t     )       -       f   ^     ⁡     (     i   ,   j   ,   t     )         ]     2             ]       1   2                 
Where f (i, j, t) is the pixel at location (i,j) in frame t of the original video sequence, {circumflex over (f)}(i, j, t) is the co-located pixel in the decoded video sequence (at location (i,j) in frame t). M and N are frame width and height (in pixels), respectively. T is the total number of frames in the video sequence. Typically, the higher the PSNR is, the higher subjective visual quality is. Compression capability is usually measured in terms of bit rate, which is the number of bits used to encode video bit stream per second.
 
     Unfortunately, digital video sequences are almost always corrupted by noise due to video acquisition, recording, processing and transmission. A main source of the noise is the noise introduced by capture device (e.g. CCD or CMOS sensor), especially when the scene is dark leading to low signal-to-noise ratio. Such noise introduced in video sequence is undesirable. It degrades the subjective quality of the video. It also affects the accuracy of motion estimation. In video coding systems such as MPEG-1/2/4 and H.261/3/4, temporal redundancy is exploited by motion estimation (ME) and motion compensation (MC) to achieve high coding efficiency. However, the noise is inherently independent among frames. Thus the presence of noise reduces temporal redundancy among consecutive frames and this can decrease the accuracy of the motion vectors obtained in motion estimation. Moreover, even if the motion vectors are accurate, the noise will make the residue frame noisy. As the noise typically exhibits little spatial redundancy, the energy of the noisy residue frame cannot be compacted by DCT. The noisy DCT coefficients would require significantly more bits to compress. As a result coding performance could drop very significantly due to the presence of noise, with a large part of the bandwidth or bit rate wasted to represent the undesirable noise. It is thus highly desirable to remove the noise before (purely pre-processing) or during (encoder-embedded denoising) the video encoding while preserving the original video contents. 
     Many denoising methods have been previously proposed, such as 2-D Kalman filter, Spatial Varying Filter (SVF), Spatial Temporal Varying Filter (STVF), Wavelet Shrinkage, neural network, adaptive wavelet filter, and motion-compensated Kalman filter. These filters are designed as purely pre-processing schemes independent of the video coding process and are to be cascaded with the encoders. Therefore, they require extra computation on top of the video encoder. 
     Optimal Denoising for Video Coding 
     The present application proposes a new purely pre-processing denoising method named Multi-Hypothesis Motion Compensated Filter (MHMCF). This pre-processing denoising method is based on the temporal linear minimum mean square error (LMMSE) estimator and requires much fewer pixels (in our simulation 3 pixels are enough) as input than most existing denoising filters to achieve same or better performance. Based on the MHMCF, the application further proposes an improved denoising method named Embedded Optimal Denoising Filter (EODF). Unlike the purely pre-processing approach, this filter can be seamlessly embedded into the motion compensation process of video encoders with nearly no extra computation introduced. 
     The advantages of the proposed approach are highlighted as follows:
         This approach is simple to implement because it requires much fewer pixels as input with some linear operations;   This approach achieves better performance because the filter coefficients are determined based on linear minimum mean squared error estimator;   This approach is flexible because there is no limitation on the number of references involved in the filtering, and the more the references are, the better the performance is. Further, it allows the use of both past frames and future frames, of both previously denoised frames and the undenoised frames;   This approach can be seamlessly embedded into the motion compensation process of video encoders. Therefore, unlike the standalone purely pre-processing approach, this approach introduces nearly no extra computation;   For P frames, video encoder with this approach embedded is compatible with MPEG-1, MPEG-2, MPEG-4, H.261, H.263, H.264 or AVS standards The standard can be modified easily to accommodate B frames; and   The approach produces encoded frame more visually pleasing. That is because the traditional methods preserve large amplitude High Frequency (HF) coefficients caused by noise. After inverse quantization and inverse transform, these HF coefficients would cause “mosaic” phenomena, which are very visually annoying.
 
Note that many of these advantages are obtained by the cooperation among the various elements of the invention, which work synergistically with each other.
       

    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       The disclosed inventions will be described with reference to the accompanying drawings, which show important sample embodiments of the invention and which are incorporated in the specification hereof by reference, wherein: 
         FIG. 1  shows a video sequence composed of consecutive frames. 
         FIG. 2  illustrates a video encoder with EODF embedded (1 reference frame). 
         FIG. 3  illustrates Rate Distortion performance of the video encoded by JM with EODF embedded.  FIG. 3(   a ) is RD Curve for sample one “Akiyo” frame,  FIG. 3(   b ) is RD Curve for sample two “Foreman” frame, and  FIG. 3(   c ) is RD Curve for sample three “Stephen” frame. 
     
    
    
     DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
     The numerous innovative teachings of the present application will be described with particular reference to the presently preferred embodiment (by way of example, and not of limitation). 
     The Multi-hypothesis Motion Compensated Filter (MHMCF) 
     As shown in  FIG. 1 , a video sequence is composed of many consecutive frames. In the video sequence, assume there is a point object which can be tracked in all the frames. Suppose the original intensity value of this point object in frame k is X k , and the observed value is
 
 Y   k   =X   k   +N   k .  (1)
 
where N k  is the undesirable noise. N k  is assumed zero mean, with variance σ n   2 . It is also assumed that N k  is independent over space and time and independent of X k .
 
     Ideally, X k  should be the same for all k. However, due to many reasons including inaccurate motion estimation, changing light conditions, and deforming objects, X k  changes from one frame to another. This can be described as
 
 X   k   =X   i   +Z   i,k   (2)
 
where Z i,k  represents the innovation in X k  compared to X i . Z i,k  is a random variable with mean  Z i,k    and variance σ z(i,k)   2 ·Z i,k  is assumed to be independent of N k  and X i .
 
     A preferred embodiment will use some or all of {Y 1 , Y 2 , . . . } to estimate X k . Note that the pixels of those {Y 1 , Y 2 , . . . } are selected as the current pixel&#39;s temporal predictions. And these temporal predictions are defined as the hypotheses of the current pixel. How to select these temporal predictions is out of the scope of this application. 
     Suppose a class of preferred embodiments chooses {Y ref 1 , Y ref 2 , . . . , Y refin , Y k } to estimate X k . The proposed non-homogenous linear estimator can be expressed as 
                       X   _     k     =         ∑     j   =   1     m     ⁢       a   j     ⁢     Y   refj         +       a   c     ⁢     Y   k       +   c             (   3   )               
Based on the LMMSE estimator, the coefficients are determined as follows:
 
                     a   i     =           σ     z   ⁡     (     refi   ,   k     )         -   2             ∑     j   =   1     m     ⁢     σ     z   ⁡     (     refj   ,   k     )         -   2         +     σ   n     -   2           ⁢           ⁢   i     =     1   ⁢           ⁢   …   ⁢           ⁢   m               (   4   )                 a   c     =       σ   n     -   2             ∑     j   =   1     m     ⁢     σ     z   ⁡     (     refj   ,   k     )         -   2         +     σ   n     -   2                   (   5   )               c   =       ∑     j   =   1     m     ⁢       a   j     ⁢       Z     refj   ,   k       _                 (   6   )               
and the estimation error variance is
 
     
       
         
           
             
               
                 
                   
                     E 
                     ⁡ 
                     
                       [ 
                       
                         
                           ( 
                           
                             
                               
                                 X 
                                 _ 
                               
                               k 
                             
                             - 
                             
                               X 
                               k 
                             
                           
                           ) 
                         
                         2 
                       
                       ] 
                     
                   
                   = 
                   
                     1 
                     
                       
                         
                           ∑ 
                           
                             j 
                             = 
                             1 
                           
                           m 
                         
                         ⁢ 
                         
                           σ 
                           
                             z 
                             ⁡ 
                             
                               ( 
                               
                                 refj 
                                 , 
                                 k 
                               
                               ) 
                             
                           
                           
                             - 
                             2 
                           
                         
                       
                       + 
                       
                         σ 
                         n 
                         
                           - 
                           2 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
           
         
       
     
     Details of MHMCF 
     Suppose the current frame is Y k . Let ref 1 , ref 2 , . . . ref m  be some previously denoised frames which are used as references (hypotheses of the current frame). And the undesirable noise variance in Y k  is denoted as σ n   2 . Here details a preferred embodiment of applying MHMCF for Y k :
     1. Divide the current frame Y k  into rectangle blocks.   2. The blocks are processed in raster-scan order. Steps 3-10 are the detailed steps for each block. Suppose the current block is B(i, j) and (i,j) is the coordinates of the pixel at the upper left corner. Initialize p=1 (index for ref1).   3. Set ref p  as the reference frame and perform motion estimation (using some appropriate distortion measure and search strategy). Denote R p (i p , j p ) as the chosen block by motion estimation with regards to ref p .   4. Denote residue block of B(i, j) with respect to ref p  as Res p  (i, j), which is derived as follows: Res p (i, j)=B(i, j)−R p (i p , j p )   5. Calculate the mean and the variance of Res p (i, j). Denote c 1  as the mean and σ p     —     n   2  as the variance. Define σ z(refp,k)   2 =max(0,σ p     —     n   2 −σ n   2 ).   6. Change p to 2-m, and repeat step 3-5. After this step, c 1  . . . c k , σ z(ref 1,k)   2  . . . σ z(refm,k)   2  and R 1 (i 1 , j 1 ) . . . R m (i m , j m ) are supposed to be available.   7. Compute the filter coefficients a i , i=1 . . . m:   

     
       
         
           
             
               a 
               i 
             
             = 
             
               
                 σ 
                 
                   z 
                   ⁡ 
                   
                     ( 
                     
                       refi 
                       , 
                       k 
                     
                     ) 
                   
                 
                 
                   - 
                   2 
                 
               
               
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     m 
                   
                   ⁢ 
                   
                     σ 
                     
                       z 
                       ⁡ 
                       
                         ( 
                         
                           refj 
                           , 
                           k 
                         
                         ) 
                       
                     
                     
                       - 
                       2 
                     
                   
                 
                 + 
                 
                   σ 
                   n 
                   
                     - 
                     2 
                   
                 
               
             
           
         
       
         
         8. Compute filter coefficient a c : 
       
    
     
       
         
           
             
               a 
               c 
             
             = 
             
               
                 σ 
                 n 
                 
                   - 
                   2 
                 
               
               
                 
                   
                     ∑ 
                     
                       j 
                       = 
                       1 
                     
                     m 
                   
                   ⁢ 
                   
                     σ 
                     
                       z 
                       ⁡ 
                       
                         ( 
                         
                           refj 
                           , 
                           k 
                         
                         ) 
                       
                     
                     
                       - 
                       2 
                     
                   
                 
                 + 
                 
                   σ 
                   n 
                   
                     - 
                     2 
                   
                 
               
             
           
         
       
         
         9. Compute mean compensation c as follows: 
       
    
     
       
         
           
             c 
             = 
             
               
                 ∑ 
                 
                   j 
                   = 
                   1 
                 
                 m 
               
               ⁢ 
               
                 
                   a 
                   j 
                 
                 ⁢ 
                 
                   c 
                   j 
                 
               
             
           
         
       
         
         10. Denote the denoised output of current block B(i, j) as B′ (i, j), which is calculated as follows: 
       
    
     
       
         
           
             
               B 
               ⁡ 
               
                 ( 
                 
                   i 
                   , 
                   j 
                 
                 ) 
               
             
             = 
             
               
                 
                   a 
                   c 
                 
                 ⁢ 
                 
                   B 
                   ⁡ 
                   
                     ( 
                     
                       i 
                       , 
                       j 
                     
                     ) 
                   
                 
               
               + 
               
                 
                   ∑ 
                   
                     j 
                     = 
                     1 
                   
                   m 
                 
                 ⁢ 
                 
                   
                     a 
                     j 
                   
                   ⁢ 
                   
                     
                       R 
                       j 
                     
                     ⁡ 
                     
                       ( 
                       
                         
                           i 
                           j 
                         
                         , 
                         
                           j 
                           j 
                         
                       
                       ) 
                     
                   
                 
               
               + 
               c 
             
           
         
       
         
         11. If all the blocks in Y k  have been denoised, go to step 12. Otherwise, go to step 3 to process the next block. 
         12. Add the current denoised frame into reference buffer to facilitate denoising for the next frame. 
       
    
     MHMCF Simulation Results 
     Experiments have been conducted to evaluate the performance of MHMCF. Four samples of test video sequences named “Akiyo”, “Foreman”, “Children” and “News” are in CIF format. Gaussian noise with variance 169 is added to the luminance components. In the tests both MHMCF with 1 reference (1HMCF) and MHMCF with 2 references (2HMCF) are based on our embodiments, and STVF is not. 1HMCF, 2HMCF and STVE are used to process the first 100 frames of each sequence. 
     The following Table I shows the denoising performance of different filters in terms of PSNR. It can be seen that both of our MHMCF, i.e. 1HMCF and 2HMCF, have better denoising performance than STVF and 2HMCF has the best performance (for sample “Akiyo”, the PSNR gain can be 8.85 dB). This is consistent with equation (7), which indicates that the more the references (hypotheses) are used, the better denoising performance can be achieved. If 3 or more reference frames were to be used, it should have achieved better performance. 
     
       
         
               
             
               
               
               
               
             
               
               
               
               
               
               
             
               
               
               
               
               
               
             
           
               
                 TABLE 1 
               
             
             
               
                   
               
               
                 Denoising performance 
               
             
          
           
               
                   
                 Sample 
                 PSNR(dB) 
                   
               
             
          
           
               
                   
                 Name 
                 Unfiltered 
                 STVF 
                 1HMCF 
                 2HMCF 
               
               
                   
                   
               
             
          
           
               
                   
                 Akiyo 
                 25.86 
                 30.61 
                 33.37 
                 34.71 
               
               
                   
                 Foreman 
                 25.89 
                 30.12 
                 31.24 
                 32.01 
               
               
                   
                 Children 
                 25.9 
                 29.53 
                 29.90 
                 30.58 
               
               
                   
                 News 
                 25.89 
                 30.02 
                 31.45 
                 32.80 
               
               
                   
                 Average 
                 0 
                 4.19 
                 5.61 
                 6.64 
               
               
                   
                 Gain 
               
               
                   
                   
               
             
          
         
       
     
     To evaluate the subjective quality of denoised video, some frames of sample “Akiyo” and sample “Foreman” processed by different denoising methods are shown in Appendix A. Test results show that MHMCF can dramatically increase the subjective quality. After filtering, most of the noise is removed, but fine details are still well preserved (e.g. hair of person in sample “Akiyo”). On the contrary, STVF cannot remove large amplitude noise and the picture is not visually pleasing. 
     To evaluate the improvement of coding efficiency, MHMCF filtered video sequences are coded using H.264 software, JM 8.3. The first frame is I frame and the rest are P frames. Only 16×16 block size and 1 reference frame are used in motion estimation. The coding performance of samples “Akiyo” and “Foreman” are given out in the following Table II and III respectively. Obviously, the coding efficiency is dramatically improved for those MHMCF filtered video sequences, especially when bit rate budget is adequate. 
     
       
         
               
             
               
               
               
               
             
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE II 
               
             
             
               
                   
               
               
                 Sample “Akiyo” Test Results 
               
             
          
           
               
                   
                 Unfiltered 
                 1HMCF 
                 2HMCF 
               
             
          
           
               
                   
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
               
               
                 QP 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
               
               
                   
               
             
          
           
               
                 19 
                 26.54 
                 12590 
                 33.73 
                 3471.86 
                 34.80 
                 1481.06 
               
               
                 25 
                 27.03 
                 8631 
                 34.3 
                 883.78 
                 34.49 
                 431.65 
               
               
                 31 
                 27.66 
                 4063 
                 33.82 
                 177.87 
                 33.23 
                 142.82 
               
               
                 37 
                 33.55 
                 147 
                 31.53 
                 84.68 
                 31.20 
                 82.01 
               
               
                   
               
             
          
         
       
     
                                                                                                   TABLE III                   Sample “Foreman” Test Results                Unfiltered   1HMCF   2HMCF                PSNR   Bitrate   PSNR   Bitrate   PSNR   Bitrate       QP   (dB)   (kbps)   (dB)   (kbps)   (dB)   (kbps)                    19   26.56   12426   31.45   5246.34   32.01   3219.29       25   27.02   8467   31.59   2061.73   31.72   1117.72       31   27.51   4175   31.36   503.58   30.76   371.76       37   30.95   293   29.55   173.27   29.06   160.19                    
Motion Compensation in Hybrid Video Coding Scheme
 
     Motion compensation is a critical part of hybrid video coding scheme. The input of motion compensation is the residue and the reconstructed reference frames. These reference frames are combined by linear averaging to generate a predicted frame. Residue is the difference between the current frame and the predicted frame, and would be encoded and transmitted to the decoder for the reconstruction of the current frame. In motion compensation process, the current frame is reconstructed by adding the residue frame back to the predicted frame. 
     Taking bi-direction prediction into consideration, the motion compensation of hybrid coding scheme can be generalized as follows (the effect of quantization on residue is ignored):
 
 Y   k ( i,j )= b   1   Y   ref1 ( i,j )+ b   2   Y   ref2 ( i,j )+res( i,j )  (8)
 
where Y k (i, j) is the pixel with coordinates (i, j) in the current frame which is to be reconstructed; Y ref 1 (i, j) and Y ref 2 (i, j) are the hypotheses (temporal predictions) of Y k (i, j) found by ME in two reference frames, respectively. Linear combination of these two hypotheses, i.e. b 1 Y ref 1 (i, j)+b 2 Y ref 2 (i, j) is used as the prediction of the current video Y k (i, j). As stated before, residue res(i, j) is the difference between Y k (i, j) and its prediction b 1 Y ref 1 (i, j)+b 2 Y ref2 (i, j). res(i, j) can be expressed as follows:
 
res( i,j )= Y   k ( i,j )−( b   1   Y   ref1 ( i,j )+ b   2   Y   ref2 ( i,j ))  (9)
 
For P frame, only one reference frame, say, Y ref 1  is used, which indicates that b 2 =0. For B frames, both Y ref 1  and Y ref 2  are utilized and b 1 +b 2 =1.
 
     The Encoder-embedded Optimal Denoising Filter (EODF) 
     The proposed embodiment for an EODF is based on MHMCF. In the embodiment, a MHMCF filter with 2 references is expressed as follows: 
                         X   k     ⁡     (     i   ,   j     )       =         a   1     ⁢       Y     ref   ⁢           ⁢   1       ⁡     (     i   ,   j     )         +       a   2     ⁢       Y     ref   ⁢           ⁢   2       ⁡     (     i   ,   j     )         +       a   c     ⁢       Y   k     ⁡     (     i   ,   j     )         +   c       ⁢     
     ⁢         Let   ⁢           ⁢     b   1       =         a   1       (     1   -     a   c       )       =       σ     z   ⁡     (       ref   ⁢           ⁢   1     ,   k     )         -   2           ∑     j   =   1     2     ⁢     σ     z   ⁡     (     refj   ,   k     )                 ,     
     ⁢       b   2     =         a   2       (     1   -     a   c       )       =       σ     z   ⁡     (       ref   ⁢           ⁢   2     ,   k     )         -   2           ∑     j   =   1     2     ⁢     σ     z   ⁢     (     refj   ,   k     )         -   2                         (   10   )               
As defined before,
 
res( i,j )= Y   k ( i,j )−(b 1   Y   ref1 ( i,j )+ b   2   Y   ref2 ( i,j ))
 
Combining equations (9) and (10), the following equation can be obtained:
 
 {tilde over (X)}   k ( i,j )= b   1   Y   ref1 ( i,j )+ b   2   Y   ref2 ( i,j )+ a   c res( i,j )+ c   (11)
 
     Comparing equation (11) with equation (8), one of ordinary skill in the art can see the only difference is that for the former, res (i, j) is scaled by a c  and an extra constant c is added. Therefore, this filter can be seamlessly incorporated into bi-direction motion compensation in video encoder. 
     In the embodiment, for 1 reference case, the MHMCF has the following form:
 
 {tilde over (X)}   k ( i,j )= a   1   Y   ref1 ( i,j )+ a   c   Y   k ( i,j )+ c  
 
It&#39;s easy to show that a 1 +a c =1 which indicates that b 1 =1. The following equation can be obtained:
 
 {tilde over (X)}   k ( i,j )= Y   ref1 ( i,j )+ a   c res( i,j )+ c   (12)
 
     This filter only needs to scale down the residue and add an extra constant c, and can be easily incorporated into one-direction motion compensation process. Such a video encoder is illustrated in  FIG. 2 . 
     To summary, the preferred embodiment presents EODF for motion compensation with one and two reference frames in equation (11) and (12) respectively. At most two reference frames are used in current video coding standards. However, the proposed EODF can be easily extended to be embedded into motion estimation with more reference frames. 
     DCT Domain Implementation of Encoder-embedded Optimal Denoising Filter (EODF) 
     The previous section discussed an embodiment of EODF implementation on residue domain, operating on residue coefficients. Another embodiment of EODF can also be on Discrete Cosine Transform (“DCT”) domain. 
     Let Res(i, j) be the residue block of size N×M (i=1, . . . , N, j=1, . . . , M). Define TRes(i, j) to be the DCT transform of residue block Res(i, j). Obviously, TRes(i, j) is also of size N×M. 
     DCT domain EODF modifies DCT coefficients as follows,
 
 T Res′( i,j )= T Res( i,j )  i= 1 and  j= 1
 
 T Res′( i,j )= T Res( i,j )* a   c   , i≠ 1 or  j≠ 1
 
where TRes (i, j) is the modified DCT coefficient, a c  is the scale factor defined in the previous section.
 
     EODF Details 
     The preferred embodiment here only describes the detailed steps of EODF with 1 reference frame. The detailed steps of EODF with 2 reference frames are similar and should be obvious to one of ordinary skill in the art. 
     Let Y k  be the current frame to be processed and ref 1  be the reference frame. Suppose the noise variance in the current frame is σ n   2 .
     1. Denote the current residue block to be denoised as Res(i, j) step 2-6 are performed:   2. Calculate the mean and the variance of Res (i, j). Denote c 1  as the mean and σ 1     —     n   2  as the variance. Define σ z(ref 1,k)   2 =max(0,σ 1     —     n   2 ).   3. Compute scaling factor a c :   

     
       
         
           
             
               a 
               c 
             
             = 
             
               
                 σ 
                 n 
                 
                   - 
                   2 
                 
               
               
                 
                   σ 
                   
                     z 
                     ⁡ 
                     
                       ( 
                       
                         
                           ref 
                           ⁢ 
                           
                               
                           
                           ⁢ 
                           1 
                         
                         , 
                         k 
                       
                       ) 
                     
                   
                   
                     - 
                     2 
                   
                 
                 + 
                 
                   σ 
                   n 
                   
                     - 
                     2 
                   
                 
               
             
           
         
       
         
         4. Compute mean compensation c:
 
 c =(1− a   c ) c   1  
 
         5. Compute the denoised residue block Res′(i, j) as follows:
 
Res′( i,j )= a   c *Res( i,j )+ c  
 
         6. Apply transform and quantization to the denoised residue block Res′(i, j). Then write it into bit stream. 
         7. Use the denoised residue to reconstruct the denoised current block. 
         8. If all the residue blocks of the current frame are processed, go to step 9; otherwise, go to step 1 to process the next residue block. 
         9. Add the current denoised frame into reference buffer to facilitate the denoising of the next frame. 
       
    
     EODF Simulation Results 
     In the preferred embodiment the proposed EODF is embedded into H.264 reference software, JM 8.3 and simulated using various QP, bit rate and video sequences. Another encoder-embedded denoising filter—transform domain temporal filter (TDTF) is also implemented for comparison in the simulation. Three sample test sequences named “Akiyo”, “Foreman” and “Stefan” are in CIF format. Gaussian noise with variance 169 is added to the luminance components of these test sequences. The first 100 frames are encoded by the original JM 8.3 (JMO), JM8.3 with TDTF (JMT) and JM8.3 with the proposed EODF (JMP). The first frame is I frame and the rest are P frames. 
     Tables IV-VI compare Rate-distortion (RD) performance with QP ranging from 19-37. 
     
       
         
               
             
               
               
               
               
             
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE IV 
               
             
             
               
                   
               
               
                 Sample “Akiyo” Test Results 
               
             
          
           
               
                   
                 JMO 
                 JMT 
                 JMP 
               
             
          
           
               
                   
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
               
               
                 QP 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
               
               
                   
               
             
          
           
               
                 19 
                 26.54 
                 12590 
                 32.78 
                 6409 
                 35.91 
                 1544 
               
               
                 25 
                 27.03 
                 8631 
                 33.65 
                 1907 
                 35.59 
                 470 
               
               
                 31 
                 27.66 
                 4063 
                 33.2 
                 213 
                 33.22 
                 218 
               
               
                 37 
                 33.55 
                 147 
                 30.72 
                 101 
                 31.29 
                 103 
               
               
                   
               
             
          
         
       
     
     
       
         
               
             
               
               
               
               
             
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE V 
               
             
             
               
                   
               
               
                 Sample “Foreman” Test Results 
               
             
          
           
               
                   
                 JMO 
                 JMT 
                 JMP 
               
             
          
           
               
                   
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
               
               
                 QP 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
               
               
                   
               
             
          
           
               
                 19 
                 26.56 
                 12426 
                 31.52 
                 6910 
                 32.77 
                 3074 
               
               
                 25 
                 27.02 
                 8467 
                 31.46 
                 2500 
                 32.33 
                 1038 
               
               
                 31 
                 27.51 
                 4175 
                 30.68 
                 337 
                 31.1 
                 356 
               
               
                 37 
                 30.95 
                 293 
                 28.1 
                 150 
                 29.16 
                 166 
               
               
                   
               
             
          
         
       
     
     
       
         
               
             
               
               
               
               
             
               
               
               
               
               
               
               
             
               
               
               
               
               
               
               
             
           
               
                 TABLE VI 
               
             
             
               
                   
               
               
                 Sample “Stefan” Test Results 
               
             
          
           
               
                   
                 JMO 
                 JMT 
                 JMP 
               
             
          
           
               
                   
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
                 PSNR 
                 Bitrate 
               
               
                 QP 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
                 (dB) 
                 (kbps) 
               
               
                   
               
             
          
           
               
                 19 
                 26.54 
                 13172 
                 29.7 
                 7875 
                 31.23 
                 4986 
               
               
                 25 
                 26.97 
                 8892 
                 29.41 
                 3196 
                 30.5 
                 2234 
               
               
                 31 
                 27.28 
                 4499 
                 27.64 
                 656 
                 28.73 
                 832 
               
               
                 37 
                 28.37 
                 601 
                 23.74 
                 261 
                 26.05 
                 327 
               
               
                   
               
             
          
         
       
     
       FIG. 3  shows the comparison of RD performance with fixed bit rate. It can be seen JMP (EODF) has much better noise suppress performance compared to JMT and JMO. It can dramatically increase PSNR and reduce bit rate for noisy video coding, especially for low QP (high bit rate). Some reconstructed frames are shown in Appendix B to evaluate the subjective quality. The test results show that the frame encoded by JMP is obviously less noisy and more visually pleasing. 
     When QP is large (bit rate is low), the gaps between these three curves become smaller. This is because some small amplitude high frequency (HF) coefficients caused by noise will be quantized to 0 when QP is large. In some extreme cases, the performance of JMO is even better than JMP in terms of PSNR. However, with JMP, the encoded frame is more visually pleasing. The reason is that, for JMO, although HF coefficients with small amplitude are removed, those with large amplitude can survive quantization. After inverse quantization and inverse transform, these HF coefficients will cause “mosaic” phenomena which are very visually annoying. Tests are also performed on a frame encoded at low bit rate by JMO and JMP, respectively. Appendix C shows the test results. Although PSNR is 0.47 dB lower, the frame encoded by JMP looks better. 
     Modifications and Variations 
     As will be recognized by those of ordinary skill in the art, the innovative concepts described in the present application can be modified and varied over a tremendous range of applications, and accordingly the scope of patented subject matter is not limited by any of the specific exemplary teachings given. 
     MHMCF Modifications and Variations 
     For example, in the above embodiment, step 7 uses m previously denoised frames as the reference (hypothesis). However, the reference frames are not necessarily limited to denoised ones. Another class of embodiments can use all frames, denoised or not, in this sequence. In another class of embodiments, the number of references, m, can be any natural number. Moreover, m can be different for different k. 
     For another example, note that in step 1 of the current embodiment, different blocks can have same or different sizes. In different class of embodiments, the blocks may be different, non-rectangle (e.g. triangular, hexagonal, irregular, etc) shapes. Further, the blocks can be disjoint or overlapped. And in extreme cases, the block can contain only 1 pixel. 
     Yet, for another example, there are various ways to implement the motion estimation in step 3. For instance, it can be block-based motion estimation such as full search, 3-step search, or other search. Further, such block-based motion estimation may use pixels in the block and outside the block. For color video, the motion estimation may be based on only one color component, but can use more than one component also. Apart from block-based motion estimation, in other class of embodiments the motion can also be obtained using other methods such as optical flow, or even manually input. 
     Note that in step 2, the blocks are processed in raster scan order in current embodiment. Yet in other embodiments, the blocks can be processed in other order. 
     Note that in step 5, σ z(refi,k)   2  is defined as σ z(refi,k)   2=max( 0,σ i     —     n   2 −σ n   2 ). This can be generalized as a σ z(refi,k)   2 =d*max(0,σ i     —     n   2 −σ n   2 ) to achieve more accurate parameter estimation in other class of embodiments. 
     In video sequences with scene changes, yet another class of embodiments might suggest that frames in different scenes should not be used to denoise each other even if some frames happen to be similar. 
     Finally, MHMCF can be implemented using recursive or non-recursive means. 
     EODF Modifications and Variations 
     In the preferred embodiment, EODF is embedded into video encoder with one or two reference frames. However, other embodiments can also embed EODF into video encoder with more than 2 reference frames. 
     There are other classes of EODF embodiments. For example, in step 1, the residue block to be denoised can be of any size and any shape. 
     For another example, in step 2, σ z(ref 1,k)   2  is calculated as σ z(ref 1,k)   2 =max(0,σ 1     —     n   2 −σ n   2 ). Again, this can be generalized in other class of embodiments as σ z(ref 1,k)   2 =d*max(0,σ 1     —     n   2 −σ n   2 ) to achieve more accurate parameter estimation in practice. 
     None of the description in the present application should be read as implying that any particular element, step, or function is an essential element which must be included in the claim scope: THE SCOPE OF PATENTED SUBJECT MATTER IS DEFINED ONLY BY THE ALLOWED CLAIMS. Moreover, none of these claims are intended to invoke paragraph six of 35 USC section 112 unless the exact words “means for” are followed by a participle. 
     The claims as filed are intended to be as comprehensive as possible, and NO subject matter is intentionally relinquished, dedicated, or abandoned.