Abstract:
This invention provides a method of interpolating a color filter array (CFA), which combines the advantages of the interpolation method based on edge detection and the interpolation method which utilizes the original color of each CFA pixel as reference, in order to ensure that the edges of the output image are clear and there are no obvious color artifacts. The interpolation method in this invention is computationally efficient and appropriate for real-time applications, in that only a 5×5 linear interpolation template is calculated and applied for each pixel.

Description:
FIELD OF THE INVENTION  
       [0001]     This invention is concerned with image creation in digital cameras, and in particular, with a color filter array (CFA) interpolation method.  
       BACKGROUND  
       [0002]     Most digital cameras utilize a structure of single image sensors fitted with a color filter array (CFA), in which there is only one color value at each pixel to capture a color representation of a scene using a single sensor. One such CFA is described in U.S. Pat. No. 3,971,065 to Bayer, which is hereby incorporated by reference. In the Bayer CFA, each pixel only captures one color: red, green or blue. To render a full-color image, an interpolation process is required. The efficacy of the interpolation algorithm determines the quality of the output image.  
         [0003]     One interpolation method, based on edge detection, requires that interpolation proceed along the direction of the edge, rather than perpendicular to the edge direction, in order to ensure that the interpolation result edge is clear. However, this interpolation method often produces obvious color artifacts at the high frequency image details.  
         [0004]     Under another interpolation method, which utilizes the original color of each CFA pixel as reference, the CFA color values corresponding to the original pixels before the interpolation process are considered to be correct, and therefore are used as the reference for the interpolation. However, this method often produces obvious zigzag effects at the image edges.  
       SUMMARY OF THE INVENTION  
       [0005]     This invention is to provide an interpolation method for CFA demosaicking. This method combines the advantages of the interpolation method based on edge detection and the interpolation method which utilizes the original color of CFA pixel as reference, in order to ensure that edges in the output image are clear and that there are no obvious color artifacts in the output color image.  
         [0006]     In one embodiment as presented in the following part of this invention, the Bayer CFA is referred to as the CFA pattern. However, as a person skilled in the art will recognize, other CFA patterns may also be used within the principle of this invention and therefore should fall into the category of this invention. References to the Bayer CFA herein should be understood to be merely an example.  
         [0007]     In order to solve the above problem, this invention provides a method for interpolating color values in a CFA pattern. The method for interpolating the green value at red or blue pixels comprises the steps described below.  
         [0008]     First, use edge detection methods to decide the direction of green value interpolation at red or blue pixels.  
         [0009]     If the interpolation should be applied in the vertical direction, the green value at the center pixel can be interpolated by 5 neighboring pixels which have this pixel as the center in the vertical direction:  
         G   13     =         -     I   3       +     2   ⨯     I   8       +     2   ⨯     I   13       +     2   ⨯     I   18       -     I   23       4         
 
         [0010]     where I i  is the pixel value of each neighboring pixel and G 13  is the green value of the central pixel.  
         [0011]     If the interpolation should be applied in the horizontal direction, the green value at the central pixel is interpolated by the five neighboring pixels which have this pixel as the center in the horizontal direction.  
         G   13     =         -     I   11       +     2   ⨯     I   12       +     2   ⨯     I   13       +     2   ⨯     I   14       -     I   15       4         
 
         [0012]     where I i  is the pixel value of each neighboring pixel and G 13  is the green value of the central pixel.  
         [0013]     If the interpolation should be applied in both vertical and horizontal directions, the green value at the central pixel is interpolated by the nine neighboring pixels which have this pixel as the center both in the vertical direction and in the horizontal direction.  
         G   13     =               -     I   3       +     2   ⨯     I   8       +     2   ⨯     I   12       +     2   ⨯     I   14       -                 I   15     -     I   11     +     2   ⨯     I   18       +     4   ⨯     I   13       -     I   23             4         
 
         [0014]     where I i  is the pixel value of each neighboring pixel and G 13  is the green value of the central pixel 13.  
         [0015]     The interpolation method in this invention will produce high-quality output images with clear edges and no color artifacts. It is computationally efficient and appropriate for real-time applications, in that only a 5×5 linear interpolation template is calculated and applied for each pixel. 
     
    
     BRIEF DESCRIPTION OF THE DRAWINGS  
       [0016]      FIG. 1  is a 5×5 original Bayer CFA image in which the center is a red pixel.  
         [0017]      FIG. 2  is the green value interpolation template in the vertical direction.  
         [0018]      FIG. 3  is the green value interpolation template in the horizontal direction.  
         [0019]      FIG. 4  is the green value interpolation template in both vertical and horizontal directions.  
         [0020]      FIG. 5  is the “#”-shaped blue value interpolation template.  
         [0021]      FIG. 6  is a 5×5 original Bayer CFA image in which the center is a green pixel.  
         [0022]      FIG. 7  is the double-cross-shaped blue value interpolation template when the center is a green pixel.  
         [0023]      FIG. 8  is the double-cross-shaped red value interpolation template when the center is a green pixel.  
     
    
     DESCRIPTION OF THE PREFERRED EMBODIMENT  
       [0024]      FIG. 1  shows a 5×5 Bayer CRA image in which the center is a red pixel. The method is implemented differently depending on whether the center is a red, blue or green pixel. The following will describe the method for each situation.  
         [0025]     1. Interpolate the green value and blue value at the red pixel (e.g. The central pixel 13, as shown on  FIG. 1 )  
         [0026]     1) Step 1, interpolate the green value at the central pixel 13.  
         [0027]     First, determine the direction of green value interpolation of the central pixel 13 (horizontal, vertical or both) by determining whether there is an edge present. First, gradients near the pixel are calculated according to: 
 
Δ h =|2× I   13   −I   11   −I   15   |+|I   12   −I   14 |  [1]
 
Δ v =|2× I   13   −I   3   −I   23   |+|I   8   −I   18 |  [2]
 
         [0028]     where I i  is the pixel value at each pixel in  FIG. 1 , Δ h  is the second order gradient value in the horizontal direction, and Δ v  is the second order gradient value in the vertical direction.  
         [0029]     If Δ h −Δ v  is larger than a positive threshold T, interpolate in the vertical direction;  
         [0030]     If Δ h −Δ v  is smaller than a negative threshold −T, interpolate in the horizontal direction;  
         [0031]     In other cases, interpolate in both directions.  
         [0032]     The edge direction estimation method described herein is provided merely as an example and should not be construed as limiting the scope of the invention. Other edge direction estimation methods can also be applied to this invention and therefore should fall into the category of this invention.  
         [0033]     1.1) If Δ h −Δ v &gt;T  
         [0034]     Based on the “original CFA color reference” interpolation method (the second method described in the background section), we use the original red value at the center as a reference. The color difference of the green value and red value should be equal to the mean of the green value and the red value differences of the upper and lower neighbor pixels.  
           G   13     -     R   13       =         (       G   8     -     R   8       )     +     (       G   18     -     R   18       )       2         
 
         [0035]     Simplifying the formula yields:  
         G   13     =       R   13     +         (       G   8     -     R   8       )     +     (       G   18     -     R   18       )       2           
 
         [0036]     Where R 8  and R 18  and are unknown, they can be estimated by the mean of the red values of the upper and lower neighbor pixels.  
         R   8     =         R   3     +     R   13       2         
         R   18     =         R   13     +     R   23       2         
 
         [0037]     Simplifying the formula, we get:  
               G   13     =         -     I   3       +     2   ⨯     I   8       +     2   ⨯     I   13       +     2   ⨯     I   18       -     I   23       4             [   3   ]             
 
         [0038]     Therefore, it can be seen that the green value of the central pixel 13 can be interpolated by the five neighboring pixels 3, 8, 13, 18, 23, which have pixel 13 as the center at the vertical direction. Herein I i  is the pixel value at each pixel in  FIG. 1 , and G 13  is the green value at the central pixel 13.  
         [0039]     The formula [3] can be represented as the linear interpolation template in  FIG. 2 . Each value on the template multiplied by ¼ is the interpolation coefficient of the corresponding pixel in formula [3] (e.g. the coefficient of the pixel 3 in the formula [3] is −¼, while that of the pixel 8 is 2/4).  
         [0040]     1.2) Similarly, if Δ h −Δ v &lt;−T, the green value of the central pixel 13 should be interpolated by the five neighboring pixels 11, 12, 13, 14, 15 which have pixel 13 as the center in the horizontal direction.  
               G   13     =         -     I   11       +     2   ⨯     I   12       +     2   ⨯     I   13       +     2   ⨯     I   14       -     I   15       4             [   4   ]             
 
         [0041]     Herein I i  is the pixel value at each pixel in  FIG. 1 , and G 13  is the green value at the central pixel 13.  
         [0042]     The formula [4] can be represented as the linear interpolation template in  FIG. 3 . Each value on the template multiplied by ¼ is the interpolation coefficient of the corresponding pixel in formula [4] (e.g. the coefficient of the pixel 11 in the formula [4] is −¼, while that of the pixel 12 is 2/4).  
         [0043]     1.3) Similarly, if |Δ h −Δ v |&lt;T, the green value of the central pixel 13 should be interpolated by nine pixels 3, 8, 11, 12, 13, 14, 15, 18, 23 which have pixel 13 as the center in both directions.  
               G   13     =               -     I   3       +     2   ⨯     I   8       +     2   ⨯     I   12       +     2   ⨯     I   14       -                 I   15     -     I   11     +     2   ⨯     I   18       +     4   ⨯     I   13       -     I   23             4             [   5   ]             
 
         [0044]     Herein I i  is the pixel value at each pixel in  FIG. 1 , and G 13  is the green value at the central pixel 13.  
         [0045]     The formula [5] can be represented as the linear interpolation template in  FIG. 4 . The value at each pixel on the template multiplied by ¼ is the interpolation coefficient of the corresponding pixel in formula [4] (e.g. the coefficient of the pixel 11 in the formula [5] is −¼, while that of the pixel 12 is 2/4).  
         [0046]     2) Step 2: interpolate the blue value at the central pixel 13.  
         [0047]     The blue value may then be calculated in a similar fashion. Based on the “original CFA color reference” interpolation method (the second method described in the background section), we use the original green value at the center as a reference. The color difference of its blue value and green value should be equal to the mean of the blue value and the green value differences of its four neighboring pixels.  
           B   13     -     G   13       =         ∑       i   =   7     ,   9   ,   17   ,   19       ⁢     (       B   i     -     G   i       )       4         
 
         [0048]     Simplifying the formula, we get:  
         B   13     -     G   13     +         ∑       i   =   7     ,   9   ,   17   ,   19       ⁢     (       B   i     -     G   i       )       4         
 
         [0049]     Herein G 7 , G 9 , G 17 , and G 19  can be estimated by its own neighboring pixel (using the same “color difference mean” method.)  
         G   7     =       B   7     +               (       G   2     -     B   7       )     +     (       G   6     -     B   7       )     +                 (       G   8     -         B   7     +     B   9       2       )     +               (       G   12     -         B   7     +     B   17       2       )           4           
         G   9     =       B   9     +               (       G   4     -     B   9       )     +     (       G   10     -     B   9       )     +                 (       G   8     -         B   7     +     B   9       2       )     +               (       G   14     -         B   9     +     B   19       2       )           4           
         G   17     =       B   17     +               (       G   22     -     B   17       )     +     (       G   16     -     B   17       )     +                 (       G   18     -         B   17     +     B   19       2       )     +               (       G   12     -         B   7     +     B   17       2       )           4           
         G   19     =       B   19     +               (       G   24     -     B   19       )     +     (       G   20     -     B   19       )     +                 (       G   18     -         B   17     +     B   19       2       )     +               (       G   14     -         B   9     +     B   19       2       )           4           
 
         [0050]     The formula can be simplified as:  
               B   13     =       G   13     +       1   16     ⁢     (       -     (       I   2     +     I   4     +     I   4     +     I   10     +     I   16     +     I   20     +     I   22     +     I   24       )       -     2   *     (       I   8     +     I   12     +     I   14     +     I   18       )       +     4   *     (       I   7     +     I   9     +     I   17     +     I   19       )         )                 [   6   ]             
 
         [0051]     Formula [6] can then be simplified as B 13 =G 13 +A, where B 13  is the blue value of the central pixel 13 which is to be calculated and A can be interpolated from the 16 pixels 2, 4, 6, 7, 8, 9, 10, 12, 14, 16, 17, 18, 19, 20, 22, 24, which represent a “#” shape centered at the pixel 13.  
         [0052]     The formula [6] can be represented as the “#”-shaped interpolation template in  FIG. 5 . Each value on the template multiplied by 1/16 is the interpolation coefficient of the pixel in formula [6] respectively (e.g. the coefficient of the pixel 7 in the formula [6] is 4/16, while that of the pixel 8 is − 2/16).  
         [0053]     2. Interpolate the green value and red value at the blue pixel:  
         [0054]     The method of interpolating the green value at the blue pixel is the same as that of interpolating the green value at the red pixel and therefore will not be repeated herein.  
         [0055]     The method of interpolating the red value at a blue pixel is the same as that of interpolating the blue value at a red pixel:  
         [0056]     R 13 =G 13 +A, where R 13  is the red value at the pixel 13 to be calculated, G 13  is the already interpolated green value at the pixel 13, and A is interpolated by the 16 neighboring pixels 2, 4, 6, 7, 8, 9, 10, 12, 14, 16, 17, 18, 19, 20, 22, 24, which represent a “#” shape centered at the pixel 13.  
       A   =       1   16     ⁢     (       -     (       I   2     +     I   4     +     I   6     +     I   10     +     I   16     +     I   20     +     I   22     +     I   24       )       -     2   *     (       I   8     +     I   12     +     I   14     +     I   18       )       +     4   *     (       I   7     +     I   9     +     I   17     +     I   19     +     )         )           
 
         [0057]     where I i  is the value at the each pixel.  
         [0058]     3. To interpolate Red and Blue values at G pixel (in G R G R line), as the center pixel in  FIG. 6 .  
         [0059]     3.1 To interpolate the blue value at pixel 13.  
         [0060]     Based on the “original CFA color reference” interpolation method (the second method described in the background section), we use the original green value at the center pixel as a reference. The color difference of the blue value and green value should be equal to the mean of the blue value and the green value differences of the two neighboring pixels.  
         B   13     =       G   13     +         ∑       i   =   8     ,   18       ⁢     (       B   i     -     G   i       )       2           
 
         [0061]     Herein G 8  and G 18  can be estimated by its own neighboring pixel (the same “color mean idea”).  
         G   8     =       B   8     +               (       G   3     -     B   8       )     +     (       G   7     -         B   6     +     B   8       2       )     +                 (       G   9     -         B   8     +     B   10       2       )     +               (       G   13     -         B   8     +     B   18       2       )           4           
         G   18     =       B   18     +               (       G   23     -     B   18       )     +     (       G   17     -         B   16     +     B   18       2       )     +                 (       G   19     -         B   18     +     B   20       2       )     +               (       G   13     -         B   8     +     B   18       2       )           4           
 
         [0062]     The above formula can be simplified as:  
               B   13     =       G   13     +       1   8     ⁢     (       -     (       I   3     +     I   7     +     I   9     +     I   17     +     I   19     +     I   23       )       +       1   2     *     (       I   6     +     I   10     +     I   16     +     I   20       )       +     3   *     (       I   8     +     I   18       )       -     2   *     I   13         )                 [   7   ]             
 
         [0063]     Herein B 13 =G 13 +A, where B 13  is the blue value at the central pixel 13 which is to be calculated, G 13  is the green value at this pixel, and A is interpolated from the neighboring 13 pixels 3, 6, 7, 8, 9, 10, 13, 16, 17, 18, 19, 20, 23, which represent a double cross shape centered at the pixel 13.  
         [0064]     The formula [7] can be represented as the double cross shaped interpolation template in  FIG. 7 . Each value on the template multiplied by ⅛ is the interpolation coefficient of the corresponding pixel in formula [7] (e.g. the coefficient of the pixel 7 in the formula [7] is −⅛, while that of the pixel 8 is ⅜).  
         [0065]     3.2) Similarly, interpolate the red value at the central pixel 13.  
         [0066]     Herein R 13 =G 13 +A, where R 13  is the red value at the central pixel 13 which is to be calculated, G 13  is the green value at this pixel, and A is interpolated from the neighboring 13 pixels 2, 4, 7, 9, 11, 12, 13, 14, 15, 17, 19, 22, 24, which represent a double cross shape centered at the pixel 13.  
             A   =       1   8     ⁢     (       -     (       I   7     +     I   9     +     I   11     +     I   15     +     I   17     +     I   19       )       +       1   2     *     (       I   2     +     I   4     +     I   22     +     I   24       )       +     3   *     (       I   12     +     I   14       )       -     2   *     I   13         )               [   8   ]             
 
         [0067]     The formula [8] can be represented as the double cross shaped interpolation template in  FIG. 8 . The value at each pixel on the template multiplied by ⅛ is the interpolation coefficient of the corresponding pixel in formula [8] (e.g. the coefficient of the pixel 7 in the formula [7] is −⅛, while that of the pixel 12 is ⅜).  
         [0068]     4. To interpolate Red and Blue values at green pixel in B G B G line is the same as to interpolate red and blue values in G R G R line.  
         [0069]     The exemplary embodiments described herein are provided merely to illustrate the principles of the invention and should not be construed as limiting the scope of the invention. Any improvement or modification of this invention by a person skilled in the art based on the principle of this invention should fall into the category of this invention.