Abstract:
A device for acquiring color images using a color filter array. The color filter array includes a pattern of red array elements, green array elements, and blue array elements. The red array elements and green array elements occur in the color filter array more frequently than the blue array elements. This multiplicity of red, green, and blue color filters adheres closely to known human visual system (HVS) chrominance and luminance responses. Also, the pattern has periodicity in the diagonal directions and is relatively immune to Moire artifacts that arise due to interference between periodic sampling patterns of the CFA and periodic features in imaged scenes.

Description:
BACKGROUND 
       [0001]    The present disclosure relates to the field of image capture, and, in particular, to a color filter array and a method for performing color interpolation, also known as demosaicking. 
         [0002]    Digital image acquisition devices, such as digital cameras, utilize optical sensors to capture images. The optical sensing elements are typically multi-pixel arrays of charge coupled devices (CCD) or complementary metal oxide semiconductors (CMOS). Both of these sensor types are inherently monochromatic. The incident radiation at each pixel of a sensor-array is integrated over a range of wavelengths (in which the device is sensitive) to give intensity values over the sensor-array. To acquire color images, cameras utilize filters that are sensitive in particular ranges in the visible spectrum that are placed before the optical sensor-array in the imaging pipeline. The sensor-array output is then the image band corresponding to the spectral transmission of the color filter. Since at least three color bands are required to represent an image for the human visual system, at least three sensor-arrays with three different color filters are required to acquire a color image. Although other color combinations are used, arrangements of color filters typically extract the three primary colors: red, green, and blue. 
         [0003]    Such multi-sensor acquisition schemes have several drawbacks. For example, multi-sensor cameras typically include one or more beam-splitters that send the light to the different color sensors which contribute substantially to the cost of the camera. Also, since the color bands are acquired at different planes, a post-processing operation is required to correct for the associated misregistration. To avoid the cost and complexity of multi-sensor acquisition systems, most consumer-level digital color cameras employ only one optical sensor. The sensor is overlaid with a color filter array (CFA) such that only one color is sampled at each pixel location. The full-color image is reconstructed from the sub-sampled data in a later step commonly referred to as demosaicking. Demosaicking depends on the pattern that defines the layout of the filters on the pixels of the sensor. 
         [0004]    The most significant desirable feature of a CFA pattern, particularly for devices that have limited computational capabilities (cell-phone cameras, low-end digital still cameras, PDA cameras, etc.), is the ease of demosaicking. Regular, repeated CFA patterns work best to satisfy this requirement. Techniques for performing demosaicking, sometimes referred to as “interpolation,” are known in the art. For example, U.S. Pat. No. 4,642,678 to Cok, U.S. Pat. No. 5,373,322 to Laroche et al., and U.S. Pat. No. 5,475,769 to Wober et al. describe various methods for recovering missing pixel values from sampled color image data. The Wober patent describes a common approach using a convolution kernel to compute missing pixel values for each pixel location based on the pixel values of a neighborhood of pixels surrounding each pixel location. For an n×m neighborhood of pixels, the convolution kernel is an n×m set of coefficients. 
         [0005]    Another useful feature of uniform CFA patterns is their relative immunity to optical and electrical cross-talk among pixels in the sensor array. Cross-talk or leakage between adjoining differently colored pixels can significantly alter the effective spectral transmittance function of a pixel. Regular patterns ensure a measure of consistency in the transmittances of similarly colored pixels across a sensor-array. A drawback of regular arrays is that they may suffer from Moire artifacts, or beats, in cases where the scene has periodic patterns similar in frequency to the period of the CFA pattern. 
         [0006]    A successful CFA pattern must adhere to the properties of the human visual system (HVS). One common periodic pattern is known as the Bayer pattern. In a Bayer pattern, green (50%) is sampled at twice the density of either red (25%) or blue (25%) since the luminance response of the human visual system (HVS) corresponds closely with the HVS response to the green range of the spectrum. 
         [0007]    The most distinctive features of the HVS are its low-pass response to luminance and chrominance components, and the fact that the cut-off for the luminance component is between two and three times as high as the cut-off for the chrominance components. These properties have been used to arrive at the ideal NTSC human visual R:G:B ratios of 30:59:11. These ratios are difficult to achieve with periodic CFAs formed by repeating 2×2 patterns like the Bayer array. 
         [0008]    It is desirable to provide for a CFA with a suitable arrangement of color filters that closely adheres to the properties of the HVS, and also addresses the above-described factors that affect CFA performance. 
       SUMMARY 
       [0009]    According to one aspect of the present disclosure, there is provided a device, and method thereof, for acquiring and reconstructing CFA images. An image recording module generates image data of sampled values output from a color filter array. The device includes an image recording module having an image sensor covered by a color filter array having a novel filter mosaic or pattern for receiving an incident source illumination to produce image data. Advantageously, this pattern adheres closer to the human visual system (HVS) Red:Green:Blue (R:G:B) ratios of 30:59:11 than that of current commercially available CFA patterns. 
         [0010]    The pattern of RGB color filters in the pattern of the color filter array is periodic in at least one diagonal direction. Natural scenes have periodic features predominant in the horizontal and vertical directions (a phenomenon which can be attributed largely to the effect of gravity). The pattern of the color filter array thus offers more immunity to Moire as compared to the standard Bayer array. 
         [0011]    When the image sensor is exposed to light, each pixel value in the image sensor records either a red, green, or blue color value depending on the color filter applied to that pixel location. The image data is input to an image reconstruction module which performs a demosaicking or interpolation algorithm on the image data to produce a reconstructed image. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0012]    Various embodiments of the present disclosure will be described herein below with reference to the figures wherein: 
           [0013]      FIG. 1  illustrates a block diagram of a digital image acquisition device incorporating the present disclosure; 
           [0014]      FIG. 2  is a diagram of a novel three color pattern or mosaic of a color filter array according to embodiments of the present disclosure; 
           [0015]      FIG. 3  is a unit array of the pattern or mosaic of  FIG. 2 ; 
           [0016]      FIG. 4  is an illustration of the pattern or mosaic in which selected neighborhoods and corresponding center pixels are illustrated; and 
           [0017]      FIG. 5  illustrates a flow diagram that sets forth the demosaicking algorithm performed by the image reconstruction module shown in  FIG. 1  to produce a reconstructed image from a CFA image. 
       
    
    
     DETAILED DESCRIPTION 
       [0018]    In reference to the drawings,  FIG. 1  illustrates a block diagram of a digital image acquisition device  100  incorporating the present disclosure. Device  100  may be, for example, a camera or a scanner. Device  100  includes in part an image recording module  102  that outputs image data  104  to an image reconstruction module  106 . As is typical in the art, image recording module  102  may be, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS). Image recording module  102  includes in part an image sensor (not shown) for recording an image through conventional optics. The sensor is typically covered by a filter mosaic or pattern commonly referred to as a color filter array (CFA). 
         [0019]      FIG. 2  illustrates a 5×5 mosaic or pattern of a novel CFA  110  in accordance with the present disclosure for receiving incident light (not shown) to produce image data  104 . When the image sensor is exposed to light, each pixel value in the image sensor records either a red, green, or blue color value depending on the color filter applied to that pixel location. These color values are converted using a conventional analog to digital converter to generate image data  104 , which is input to image reconstruction module  106 . Image data  104  may be, for example, a CFA image. It should be appreciated that additional processing may be performed before the CFA image is output to the to image reconstruction module  106 . For example, the CFA image may be compressed, transmitted and decompressed prior to reaching image reconstruction module  106 . 
         [0020]    A processor (not shown) with operating memory (not shown) controls the operation of device  100 . The processor receives input signals from a user to record an image by engaging the image recording module. Once part, or all, of the CFA image is produced, the processor engages reconstruction module  106 . 
         [0021]    Reconstruction module  106  performs a demosaicking or interpolation algorithm on image data  104  to produce a reconstructed image (not shown). The reconstructed image output by reconstruction module  106  may be stored in memory  108  as a full-color or gray-scale full-resolution image for later display or transfer to another device. It should be appreciated that additional processing may be performed before the image is stored in memory  108 . For example, the image may be either compressed using a compression method, such as the JPEG standard, or sub-region sampled prior to being stored in memory  108 . The demosaicking algorithm is discussed in further detail below. 
         [0022]    In accordance with the present disclosure, imaging device  100  further includes a three color Red, Green, Blue (RGB) tiling pattern for a CFA in which the ratio of red, green, and blue filters is approximately 2:2:1. As understood by those skilled in the art, this ratio may be approximate due to “edge effects.” 
         [0023]      FIG. 3  illustrates a 5-pixel unit array of CFA  110 , where R n , G n , and B denote red, green, and blue pass filters, respectively. CFA  110  is provided by repeating the unit array of  FIG. 3  in both the row and column directions to tessellate a larger area. As illustrated in  FIG. 2 , for example, CFA  110  may be a 5×5 mosaic or pattern resulting from the combination of parts of nine of the unit arrays of  FIG. 3 . Since the unit array is not rectangular, the array formed by tessellating the unit array has “incomplete” units at its edges. Still referring to the unit array FIG.  3 , CFA  110  ( FIG. 2 ) includes five distinct color photosites, namely, R 1 , R 2 , G 1 , G 2 , and B. Optical and electronic cross-talk across CFA  110  is assumed to be isotropic and cross-talk between two color sites is assumed to be dependent only on the distance between the sites. 
         [0024]    The colors of CFA  110  according to the present disclosure share neighbors that are similarly spaced and colored. This characteristic of CFA  110  ensures consistent color filter transmittances across the sensor-array. For example, the adjacent neighbors as well as the diagonally adjacent neighbors of both photosites R 1  and R 2  include one red photosite, two green photosites, and one blue photosite. Likewise, the adjacent neighbors as well as the diagonally adjacent neighbors of both photosites G 1  and G 2  include two red photosites, one green photosite, and one blue photosite. 
         [0025]    In this arrangement, each column of CFA  110  includes a RGB ratio of approximately 2:2:1. Similarly, each row of CFA  110  includes a RGB ratio of approximately 2:2:1. Ultimately, CFA  110  includes 40% red pixels R 1 , R 2 , 40% green pixels G 1 , G 2 , and 20% blue pixels B. Advantageously, this R:G:B ratio adheres closer to the HVS R:G:B ratios of 30:59:11 than that of the prior art and commercially available CFA patterns. 
         [0026]    Referring now to  FIG. 5 , a flow chart is shown illustrating an exemplary demosaicking algorithm performed by image reconstruction module  106  for calculating interpolated pixel values using a set of convolution kernels in accordance with the present disclosure.  FIG. 5  is illustrative only and other processes for calculating interpolated pixel values can also be used. In the demosaicking process, the CFA image within image data  104  is separated into three distinct sparsely sampled color channels, namely red, green, and blue. Demosaicking requires a set of coefficients for each color channel in the full color image. A different convolution kernel of coefficients is used for each color filter. Convolution kernels, and their respective coefficients, according to the present disclosure are defined for each color filter. The convolution kernels may be stored in imaging device  100  for processing sensor data subsequently captured by imaging device  100  for generating a full color image. 
         [0027]    The demosaicking algorithm will now be described in detail with reference to  FIGS. 1 ,  4 , and  5 . For purposes of example,  FIG. 4  illustrates a 10×10 CFA  210  in accordance with the present disclosure. Referring first to  FIG. 5 , initially at step  300 , image recording module  102  of imaging device  100  is used to acquire a CFA image to generate image data  104 . Subsequently at step  302 , CFA  210  is divided into a plurality of n×m neighborhoods about a center pixel P B , P G1 , P G2 , P R2 , P R1 , as illustrated in  FIG. 4 . In the present illustration, the missing pixel values to be reconstructed from the neighborhood of pixels are pixel values for the red, green, and blue color channels. 
         [0028]    In accordance with the present disclosure, a neighborhood may be defined, for example, as a center pixel and each pixel immediately surrounding the center pixel in each adjacent row and column. The neighborhood thus defined may be a 3×3 array of pixels (i.e., n=3) centered on each center pixel. In the present example illustrated in  FIG. 4 , arbitrarily selected neighborhoods  250 ,  255 ,  260 ,  265 , and  270  of CFA  210  are centered about center pixels P B , P G1 , P G2 , P R2 , P R1 , respectively. At step  304 , for each color channel to be reconstructed, a corresponding n×m convolution kernel for the color filter associated with center pixel P B , P G1 , P G2 , P R2 , P R1  of each of the n×m neighborhoods identified in step  302  is selected. 
         [0029]    The following are an exemplary set of convolution kernels according to the present disclosure utilizing a weighted average approach applied to a linear interpolation scheme. Such a scheme may include constants factored into each convolution kernel as illustrated below. The following convolution kernels are illustrative only and other process for calculating interpolated pixel values are envisioned for use with the present disclosure, e.g., edge-adaptive interpolation using convolution with directional kernels, as is understood by those skilled in the art. 
         [0030]    For the reconstruction of the red color channel: 
         [0000]    
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               B 
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       
                         2 
                       
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       1 
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       
                         2 
                       
                     
                   
                 
                 ] 
               
             
           
         
       
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               
                 G 
                  
                 
                     
                 
                  
                 1 
               
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       0 
                     
                     
                       1 
                     
                     
                       0 
                     
                   
                   
                     
                       1 
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       
                         2 
                       
                     
                     
                       0 
                     
                     
                       
                         2 
                       
                     
                   
                 
                 ] 
               
             
           
         
       
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               
                 G 
                  
                 
                     
                 
                  
                 2 
               
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       
                         2 
                       
                     
                     
                       0 
                     
                     
                       
                         2 
                       
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                   
                     
                       0 
                     
                     
                       1 
                     
                     
                       0 
                     
                   
                 
                 ] 
               
             
           
         
       
     
         [0031]    For the reconstruction of the green color channel: 
         [0000]    
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               B 
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       0 
                     
                     
                       1 
                     
                     
                       
                         2 
                       
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       
                         2 
                       
                     
                     
                       1 
                     
                     
                       0 
                     
                   
                 
                 ] 
               
             
           
         
       
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               
                 R 
                  
                 
                     
                 
                  
                 1 
               
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       
                         2 
                       
                     
                   
                   
                     
                       1 
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                   
                     
                       0 
                     
                     
                       1 
                     
                     
                       
                         2 
                       
                     
                   
                 
                 ] 
               
             
           
         
       
       
         
           
             
               At 
             
              
             
                 
             
              
             
               P 
               
                 R 
                  
                 
                     
                 
                  
                 2 
               
             
              
             
               : 
             
              
             
               
                 1 
                 
                   ( 
                   
                     2 
                     + 
                     
                       2 
                        
                       
                         2 
                       
                     
                   
                   ) 
                 
               
                
               
                 [ 
                 
                   
                     
                       
                         2 
                       
                     
                     
                       1 
                     
                     
                       0 
                     
                   
                   
                     
                       0 
                     
                     
                       0 
                     
                     
                       1 
                     
                   
                   
                     
                       
                         2 
                       
                     
                     
                       0 
                     
                     
                       0 
                     
                   
                 
                 ] 
               
             
           
         
       
     
         [0032]    For the reconstruction of the blue color channel: 
         [0000]    
       
         
           
             
               
                 
                   
                     At 
                   
                    
                   
                       
                   
                    
                   
                     P 
                     
                       R 
                        
                       
                           
                       
                        
                       1 
                     
                   
                    
                   
                     : 
                   
                    
                   
                     
                       1 
                       
                         ( 
                         
                           1 
                           + 
                           
                             2 
                           
                         
                         ) 
                       
                     
                      
                     
                       [ 
                       
                         
                           
                             
                               2 
                             
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             1 
                           
                         
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   
                     At 
                   
                    
                   
                       
                   
                    
                   
                     P 
                     
                       R 
                        
                       
                           
                       
                        
                       2 
                     
                   
                    
                   
                     : 
                   
                    
                   
                     
                       1 
                       
                         ( 
                         
                           1 
                           + 
                           
                             2 
                           
                         
                         ) 
                       
                     
                      
                     
                       [ 
                       
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                         
                           
                             1 
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             
                               2 
                             
                           
                         
                       
                       ] 
                     
                   
                 
               
             
             
               
                 
                   
                     At 
                   
                    
                   
                       
                   
                    
                   
                     P 
                     
                       G 
                        
                       
                           
                       
                        
                       1 
                     
                   
                    
                   
                     : 
                   
                    
                   
                     
                       1 
                       
                         ( 
                         
                           1 
                           + 
                           
                             2 
                           
                         
                         ) 
                       
                     
                      
                     
                       [ 
                       
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             
                               2 
                             
                           
                         
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                         
                           
                             0 
                           
                           
                             1 
                           
                           
                             0 
                           
                         
                       
                       ] 
                     
                   
                 
               
               
                 
                   
                     At 
                   
                    
                   
                       
                   
                    
                   
                     P 
                     
                       G 
                        
                       
                           
                       
                        
                       2 
                     
                   
                    
                   
                     : 
                   
                    
                   
                     
                       1 
                       
                         ( 
                         
                           1 
                           + 
                           
                             2 
                           
                         
                         ) 
                       
                     
                      
                     
                       [ 
                       
                         
                           
                             0 
                           
                           
                             1 
                           
                           
                             0 
                           
                         
                         
                           
                             0 
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                         
                           
                             
                               2 
                             
                           
                           
                             0 
                           
                           
                             0 
                           
                         
                       
                       ] 
                     
                   
                 
               
             
           
         
       
     
         [0033]    The interpolated pixel value for center pixel P B , P G1 , P G2 , P R2 , P R1  can now be calculated. In step  306 , for each color channel, each pixel&#39;s color value is computed out of the original color and the colors of the pixels in the surrounding n×m neighborhood of pixel values, such as neighborhoods  250 ,  255 ,  260 , and  265 . Specifically, for each pixel an n×m convolution kernel is multiplied to the corresponding values of the pixel to be reconstructed and its surrounding neighborhood  250 ,  255 ,  260 ,  265 , and  270 . The result is a n×m product matrix of missing color samples. The elements of the product matrix are summed and outputted as the interpolated pixel value for center pixel P B , P G1 , P G2 , P R2 , P R1  for each color channel. Then in step  308 , the missing color samples computed at step  306  are combined with pixels of the CFA image to define a reconstructed image (not shown). 
         [0034]    The demosaicking algorithm continues with the next color channel until all the interpolated pixel values for all pixel locations in image data  104  for all color channels are derived. The demosaicking algorithm may then proceed to calculate the interpolated pixel value for the next center pixel of CFA  210 . In this manner, interpolated pixel values for all pixel locations in CFA  210  for all color channels can be calculated. 
         [0035]    It will be appreciated that variations of the above-disclosed and other features and functions, or alternatives thereof, may be desirably combined into many other different systems, methods and/or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.