Abstract:
A method and apparatus of combining multiple exposure images by applying a transfer function to pixel output signals from pixels in a pixel array, the pixel output signals from each pixel including at least a first pixel output signal generated in response to a first exposure time and a second pixel output signal generated in response to a second exposure time.

Description:
FIELD OF THE INVENTION 
       [0001]    The embodiments disclosed herein relate to generally semiconductor imagers and more specifically to multi-exposure imaging. 
       BACKGROUND OF THE INVENTION 
       [0002]    The dynamic range of an imaging or camera system may be defined by the maximum and minimum illumination levels effectively captured in a single image or frame. A desired imaging device is sensitive to a broad illumination range. Unfortunately, designing an imaging device to be equally sensitive to both low and high illumination levels is limited by currently used photosensors. As a result, several techniques have been developed for extending the dynamic range of imaging devices. Some of the most common techniques include increasing the capacity of a pixel well, multi-exposure image capture, using pixel arrays containing varying pixel areas and/or pixel sensitivity, using logarithmic or other non-linear pixel response to light, and pixel-by-pixel adaptive exposure time. 
         [0003]    Multi-exposure image capture is an attractive technique for extending the dynamic range of an imaging device. Multi-exposure image capture produces a known piecewise linear relationship between exposures and may be implemented using common imaging device architectures. In multi-exposure image capture, the same image is captured using more than one exposure time. A final image is created by summing weighted pixel values from each of the exposures. In this way, a final image output may be constructed from the linear combination of several images of varying exposure times. Unfortunately, however, the final image output is affected by a non-linear signal-to-noise ratio SNR. Due to photon shot noise limitations, as explained below, the signal-to-noise ratio SNR in multi-exposure image capture generally does not scale linearly. 
         [0004]    Photon shot noise σ ph  is characterized by statistical fluctuations in the rate photons are received by a pixel. Photon shot noise σ ph  is a function of the number of photoelectrons P generated in a pixel as shown in Equation 1 below. The signal-to-noise ratio SNR of a pixel is limited by photon shot noise σ ph  when detected signals are large (i.e., when the number of generated photoelectrons P is large). Even when photon shot noise σ ph  is not a significant factor, however (e.g., when the detected signals are small), additional noise sources must be considered. These additional noise sources make up the read noise floor σ read  which refers to the residual noise of the image sensor when photon shot noise is excluded. The read noise floor σ read  limits the image quality in the dark regions of an image. Thus, pixel noise σ is a combination of photon shot noise σ ph  and the read noise floor σ read , as illustrated in Equation 2 below. The signal-to-noise ratio SNR is dependent upon the signal level (via both the numerator and the photon shot noise σ ph  in the denominator) in addition to the read noise floor σ read  of the sensor as shown in Equation 3 below. 
         [0000]    
       
         
           
             
               
                 
                   
                     σ 
                     
                       p 
                        
                       
                           
                       
                        
                       h 
                     
                   
                   = 
                   
                     
                       P 
                     
                     . 
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   1 
                 
               
             
             
               
                 
                   σ 
                   = 
                   
                     
                       
                         
                           σ 
                           
                             p 
                              
                             
                                 
                             
                              
                             h 
                           
                           2 
                         
                         + 
                         
                           σ 
                           read 
                           2 
                         
                       
                     
                     = 
                     
                       
                         
                           P 
                           + 
                           
                             σ 
                             read 
                             2 
                           
                         
                       
                       . 
                     
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   2 
                 
               
             
             
               
                 
                   SNR 
                   = 
                   
                     
                       P 
                       σ 
                     
                     = 
                     
                       
                         P 
                         
                           
                             P 
                             + 
                             
                               σ 
                               read 
                               2 
                             
                           
                         
                       
                       . 
                     
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   3 
                 
               
             
           
         
       
     
         [0005]    Based on the signal-to-noise ratio SNR model of Equation 3, multi-exposure image capture produces a signal-to-noise ratio SNR response that contains discontinuities, meaning there are abrupt changes in the signal-to-noise ratio SNR when multiple exposures are used—the signal-to-noise ratio SNR for a dynamic range is not linear, but discontinuous. The result of the discontinuities is a visible change in the final image signal quality between regions of varying illumination (acquired through different exposure times). The discontinuities occur when the pixels saturate during a given exposure time and a transition is made to use a shorter exposure for increased light levels.  FIGS. 1A ,  1 B and  1 C demonstrate an example of the signal-to-noise ratio SNR discontinuities that occur for multiple exposure imaging. As seen in  FIG. 1A , a longer exposure time (e.g., Exposure  1 ) is used to capture dark areas of an image (areas where the light intensity is low). The shortest exposure time (Exposure  3 ) is used to capture the brightest areas of the image. Other intervening exposure times may also be used (e.g., Exposure  2 ). The total number of exposure times used is dependent upon two values: the maximum signal-to-noise ratio SNR max  and the minimum acceptable signal-to-noise ratio level SNR lim . The maximum signal-to-noise ratio SNR max  represents the signal-to-noise ratio SNR of a saturated photosensor. Although higher signal-to-noise ratios SNRs may be desired, the maximum signal-to-noise ratio SNR max  is limited by a maximum number of photoelectrons that a photosensor is able to collect. Using Equation 3, the maximum signal-to-noise ratio SNR max  is determined when the photoelectrons P are at a maximum P max . The minimum acceptable signal-to-noise ratio SNR lim  is a predetermined quality-control value. On the one hand, high quality standards would require that the minimum acceptable signal-to-noise ratio SNR lim  be as high as possible, close to the value of the maximum signal-to-noise ratio SNR max . If the minimum acceptable signal-to-noise ratio SNR lim  were shifted towards the maximum signal-to-noise ratio SNR max , the result is a high-valued signal-to-noise ratio SNR with many small discontinuities, as illustrated in  FIG. 1B . Unfortunately, in order to achieve the high signal-to-noise ratio SNR, a high number of exposure times is required. If only a few exposure times were used (e.g., Exposures  1  and  2 ), the dynamic range of the imaging device would be severely limited. On the other hand, if the minimum acceptable signal-to-noise ratio SNR lim  were lowered, as illustrated in  FIG. 1C , only a few exposure times would be required. However, the signal-to-noise ratio SNR will vary greatly and there will be at least one large discontinuity that will result in differences in image quality among image regions with different illumination levels. A minimum acceptable signal-to-noise ration SNR lim  that reduces both the number of exposure times required and the size of the discontinuities between exposure times is preferred. 
         [0006]    One well known method for combining multiple exposure image data is to use simple image addition and an exposure ratio factor to compensate for exposure differences.  FIG. 2  shows a block diagram of a circuit  10  used to add two exposures. In  FIG. 2 , the photoelectrons accumulated in a pixel P(i, j) in row m of an imager are measured in response to two different exposure times, Exposure  1  and Exposure  2 . A signal representing the number of collected photoelectrons in pixel P(i, j) in response to Exposure  1  is output as signal P 1 (i, j). A signal representing the number of collected photoelectrons in pixel P(i, j) in response to Exposure  2  is output as signal P 2 (i, j). The two output signals P 1 (i, j), P 2 (i, j) are summed after applying an exposure weighting factor α to signal P 2 (i, j). The resulting output signal is P out (i, j), which is equal to P 1 (i, j)+αP 2 (i, j). The resulting signal-to-noise ratio SNR from combining different exposures using addition is shown below in Equation 4. The exposure ratio factor α doesn&#39;t change the signal-to-noise ratio SNR since both the signal and noise are multiplied by the same factor. Thus, the exposure factor is not included in Equation 4. 
         [0000]    
       
         
           
             
               
                 
                   SNR 
                   = 
                   
                     
                       
                         
                           P 
                           1 
                         
                         + 
                         
                           P 
                           2 
                         
                       
                       
                         
                           
                             P 
                             1 
                           
                           + 
                           
                             P 
                             2 
                           
                           + 
                           
                             2 
                              
                             
                               σ 
                               read 
                               2 
                             
                           
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   4 
                 
               
             
           
         
       
     
         [0007]    Equation 4 may be plotted against Equation 3 in order to demonstrate the negative aspects of using simple image addition in multi-exposure imaging.  FIGS. 3A-3C  illustrate the use of Equation 3 to plot the signal-to-noise ratio for both a long exposure P 1  and a short exposure P 2 . Equation 4 is also used to plot a summed exposure P 1 +P 2 . The comparison shows that in low-illumination levels, the signal-to-noise ratio is decreased when the two signals P 1 , P 2  are summed. The comparison also shows that summing signals P 1 , P 2  results in an increase in the discontinuity that occurs at the transition from signal P 1  to signal P 2 . The plots in  FIGS. 3A-3C  were made using an exposure ratio α of 10, a photosensor full well of 10,000 e −  and a readout noise floor σ read  of 10 e − . 
         [0008]    As another example, consider the low-light case when P 1 =100 e − , P 2 =10 e − , σ read =10 e −  and σ=10. When just using the long exposure signal P 1 , for low light situations, the signal-to-noise ratio SNR is 7.07, as shown below in Equation 5. However, when both exposures are added, the signal-to-noise ratio SNR is reduced to 6.25, as shown below in Equation 6. 
         [0000]    
       
         
           
             
               
                 
                   SNR 
                   = 
                   
                     
                       
                         P 
                         1 
                       
                       
                         
                           
                             P 
                             1 
                           
                           + 
                           
                             σ 
                             read 
                             2 
                           
                         
                       
                     
                     = 
                     
                       7.07 
                       . 
                     
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   5 
                 
               
             
             
               
                 
                   SNR 
                   = 
                   
                     
                       
                         
                           P 
                           1 
                         
                         + 
                         
                           P 
                           2 
                         
                       
                       
                         
                           
                             P 
                             1 
                           
                           + 
                           
                             P 
                             2 
                           
                           + 
                           
                             2 
                              
                             
                               σ 
                               read 
                               2 
                             
                           
                         
                       
                     
                     = 
                     
                       6.25 
                       . 
                     
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   6 
                 
               
             
           
         
       
     
         [0009]    The above example shows that for low light levels where photo shot noise doesn&#39;t dominate the signal-to-noise ratio SNR, the overall signal-to-noise ratio SNR is reduced when adding the two exposures. 
         [0010]    There is a need and desire, therefore, to achieve a desired dynamic range increase while avoiding signal-to-noise ratio SNR discontinuity artifacts in the resulting images. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0011]      FIGS. 1A-1C  are graphs that illustrate the signal-to-noise ratio SNR discontinuities that occur during multiple exposure imaging. 
           [0012]      FIG. 2  is a summing circuit for combining multiple exposure image data. 
           [0013]      FIGS. 3A-3C  are graphs that illustrate the signal-to-noise ratio SNR resulting from use of the summing circuit of  FIG. 2 . 
           [0014]      FIG. 4  is a weighted transfer function circuit for combining multiple exposure image data according to a disclosed embodiment. 
           [0015]      FIG. 5  is graph that illustrates the signal-to-noise ratio SNR resulting from the use of the weighted transfer function circuit of  FIG. 4 , according to a disclosed embodiment. 
           [0016]      FIG. 6  is a graph of a weighted transfer function according to a disclosed embodiment. 
           [0017]      FIG. 7  is a block diagram of a CMOS semiconductor imager according to a disclosed embodiment. 
           [0018]      FIG. 8  is a block diagram of a processing system that includes an imaging device according to a disclosed embodiment. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0019]    In order to achieve improved signal-to-noise ratio SNR performance across the entire dynamic range available via multi-exposure imaging, a transfer function is applied to both long and short exposure signals so that only the long exposure signal is used for low light intensity (low signal levels), only the short exposure signal is used for high signal levels, and both signals are mixed close to the exposure transition points (the points at which a discontinuity exists between the signal-to-noise ratios SNRs of two different exposures). The block diagram of  FIG. 4  shows the circuit  20  used to combine exposures using the transfer functions. 
         [0020]    In  FIG. 4 , a transfer function β(P) is applied to signals from pixel P(i, j). A signal representing the number of collected photoelectrons in pixel P(i, j) in response to Exposure  1  is output as signal P 1  (for convenience, the indices (i, j) are omitted). Similarly, a signal representing the number of collected photoelectrons in pixel P(i, j) in response to Exposure  2  is output as signal P 2 . The pixel output P 2  is weighted by exposure factor α. If desired, pixel output P 1  may also be weighted by a different exposure factor. The transfer function β(P) is applied to signal P 1  to yield transfer signal β(P 1 ). In one branch of  FIG. 4 , the transfer signal β(P 1 ) is multiplied with the pixel output signal P 1  to create signal P 1 ·β(P 1 ). In another branch, the transfer signal β(P 1 ) is subtracted from 1 to create an inverse function 1−β(P 1 ). Inverse function 1−β(P 1 ) is applied to the weighted pixel output α·P 2  to yield a signal α·P 2 [1−β(P 1 )]. The resulting signal α·P 2 [1−β(P 1 )] is summed with signal P 1 ·β(P 1 ) to create output signal P out (i, j), which is equal to P 1 ·β(P 1 )+α·P 2 [1−(P 1 )]. 
         [0021]    The transfer function β(P) may be generated on the fly using a function generator and a known explicit equation or may be a look-up table LUT of values. The output range of the transfer function is zero to one. Thus, the function 1−β(P) is an inverse transfer function of function β(P). The transfer and inverse transfer functions act as weighting functions providing varying weights to either signal P 1  or P 2 , depending on the signal level. One skilled in the art will recognize that the transfer function β(P) may alternatively be applied to signal P 1 , with the inverse transfer function being applied to P 2 , as long as the transfer function β(P) is modified appropriately. 
         [0022]    The technique and circuit  20  described in relation to  FIG. 4  allows multiple exposures to be combined so that the signal-to-noise ratio SNR is improved with reduced discontinuities across the dynamic range of the system. For example, the transfer function β(P) may be designed to output a 1 for the long exposure signal P 1  and a 0 for the short exposure signal P 2  when the long exposure signal P 1  is small in order to avoid adding noise from the short exposure signal P 2 . Other transfer functions β(P) may of course be used as long as the function results in the improvement of the signal-to-noise ratio SNR and reduced discontinuities over the entire dynamic range of the image sensors. 
         [0000]    
       
         
           
             
               
                 
                   SNR 
                   = 
                   
                     
                       
                         
                           
                             P 
                             1 
                           
                           · 
                           
                             β 
                              
                             
                               ( 
                               
                                 P 
                                 1 
                               
                               ) 
                             
                           
                         
                         + 
                         
                           
                             P 
                             2 
                           
                           · 
                           
                             [ 
                             
                               1 
                               - 
                               
                                 β 
                                  
                                 
                                   ( 
                                   
                                     P 
                                     1 
                                   
                                   ) 
                                 
                               
                             
                             ] 
                           
                         
                       
                       
                         
                           
                             
                               ( 
                               
                                 
                                   P 
                                   1 
                                 
                                 + 
                                 
                                   σ 
                                   R 
                                   2 
                                 
                               
                               ) 
                             
                             · 
                             
                               
                                 β 
                                 2 
                               
                                
                               
                                 ( 
                                 
                                   P 
                                   1 
                                 
                                 ) 
                               
                             
                           
                           + 
                           
                             
                               ( 
                               
                                 
                                   P 
                                   2 
                                 
                                 + 
                                 
                                   σ 
                                   R 
                                   2 
                                 
                               
                               ) 
                             
                             · 
                             
                               
                                 [ 
                                 
                                   1 
                                   - 
                                   
                                     β 
                                      
                                     
                                       ( 
                                       
                                         P 
                                         1 
                                       
                                       ) 
                                     
                                   
                                 
                                 ] 
                               
                               2 
                             
                           
                         
                       
                     
                     . 
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   7 
                 
               
             
           
         
       
     
         [0023]    Equation 7 above shows the signal-to-noise ratio SNR after combining signals P 1 , P 2  using a weighted transfer function. Equation 7 may be used to plot signal-to-noise ratio SNR results in order to demonstrate the effect of transfer function β(P).  FIG. 5  illustrates the signal-to-noise ratio SNR using a weighted transfer function as defined below in Equation 8 and illustrated in  FIG. 6 . In  FIG. 5 , the signal-to-noise ratio SNR resulting from the weighted transfer function is compared with the signal-to-noise ratio SNR resulting from basic summing of exposure signals. It is apparent that the signal-to-noise ratio SNR resulting from a weighted transfer function is generally improved across the entire dynamic range of the system while the discontinuity at the exposure signal transition point is less. 
         [0024]    The signal-to-noise ratio SNR resulting from the transfer function plotted in  FIG. 5  is derived from the transfer function in Equation 8 below and illustrated in  FIG. 6 . The transfer function of Equation 8 is an example of a linear transfer function for a defined transition region S 1  to S 2  with a value of 1 for input values less than S 1  and 0 for input values greater than S 2 . The transition region S 1  to S 2  is a range of signal levels that includes the signal level at which a transition point or discontinuity exists between signal-to-noise ratios SNRs of different exposure times. The transition region boundaries S 1 , S 2  may be equidistant from the transition point, or may be shifted so that the transition point is closer to one of the boundaries S 1 , S 2 . The boundaries S 1 , S 2  or methods of determining the boundaries S 1 , S 2  are selected in advance. 
         [0000]    
       
         
           
             
               
                 
                   
                     β 
                      
                     
                       ( 
                       
                         P 
                         1 
                       
                       ) 
                     
                   
                   = 
                   
                     { 
                     
                       
                         
                           1 
                         
                         
                           
                             
                               P 
                               1 
                             
                             &lt; 
                             
                               S 
                               1 
                             
                           
                         
                       
                       
                         
                           
                             
                               
                                 S 
                                 2 
                               
                               - 
                               
                                 P 
                                 1 
                               
                             
                             
                               
                                 S 
                                 2 
                               
                               - 
                               
                                 S 
                                 1 
                               
                             
                           
                         
                         
                           
                             
                               S 
                               1 
                             
                             ≤ 
                             
                               P 
                               1 
                             
                             ≤ 
                             
                               S 
                               2 
                             
                           
                         
                       
                       
                         
                           0 
                         
                         
                           
                             
                               S 
                               2 
                             
                             &lt; 
                             
                               
                                 P 
                                 1 
                               
                               . 
                             
                           
                         
                       
                     
                   
                 
               
               
                 
                   Equation 
                    
                   
                       
                   
                    
                   8 
                 
               
             
           
         
       
     
         [0025]    The circuit  20  illustrated in  FIG. 4 , including the transfer function β(P) may be implemented using either hardware or software or via a combination of hardware and software. For example, in a semiconductor CMOS imager  100 , as illustrated in  FIG. 7 , the circuit  20  may be implemented within the image processor  180 .  FIG. 7  illustrates a simplified block diagram of a semiconductor CMOS imager  100  having a pixel array  140  including a plurality of pixel cells arranged in a predetermined number of columns and rows. Each pixel cell is configured to receive incident photons and to convert the incident photons into electrical signals. Pixel cells of pixel array  140  are output row-by-row as activated by a row driver  145  in response to a row address decoder  155 . Column driver  160  and column address decoder  170  are also used to selectively activate individual pixel columns. A timing and control circuit  150  controls address decoders  155 ,  170  for selecting the appropriate row and column lines for pixel readout. The control circuit  150  also controls the row and column driver circuitry  145 ,  160  such that driving voltages may be applied. Each pixel cell generally outputs both a pixel reset signal v rst  and a pixel image signal v sig , which are read by a sample and hold circuit  161  according to a correlated double sampling (“CDS”) scheme. The pixel reset signal v rst  represents a reset state of a pixel cell. The pixel image signal v sig  represents the amount of charge generated by the photosensor in the pixel cell in response to applied light during an integration period. The pixel reset and image signals v rst , v sig  are sampled, held and amplified by the sample and hold circuit  161 . The sample and hold circuit  161  outputs amplified pixel reset and image signals V rst , V sig . The difference between V sig  and V rst  represents the actual pixel cell output with common-mode noise eliminated. The differential signal (e.g., V rst −V sig ) is produced by differential amplifier  162  for each readout pixel cell. The differential signals are digitized by an analog-to-digital converter  175 . The analog-to-digital converter  175  supplies the digitized pixel signals to an image processor  180 , which forms and outputs a digital image from the pixel values. The output digital image is a result of the combination of multiple exposures in the circuit  20  of the or at least controlled by the image processor  180 . 
         [0026]    The circuit  20  and transfer function β(P) of  FIG. 4  may be used in any system which employs an imager device, including, but not limited to a computer system, camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other imaging systems. Example digital camera systems in which the invention may be used include both still and video digital cameras, cell-phone cameras, handheld personal digital assistant (PDA) cameras, and other types of cameras.  FIG. 8  shows a typical processor system  1000  which is part of a digital camera  1001 . The processor system  1000  includes an imaging device  100  which includes either software or hardware to implement multi-exposure imaging in accordance with the embodiments described above. System  1000  generally comprises a processing unit  1010 , such as a microprocessor, that controls system functions and which communicates with an input/output (I/O) device  1020  over a bus  1090 . Imaging device  100  also communicates with the processing unit  1010  over the bus  1090 . The processor system  1000  also includes random access memory (RAM)  1040 , and can include removable storage memory  1050 , such as flash memory, which also communicates with the processing unit  1010  over the bus  1090 . Lens  1095  focuses an image on a pixel array of the imaging device  100  when shutter release button  1099  is pressed. 
         [0027]    The processor system  1000  could alternatively be part of a larger processing system, such as a computer. Through the bus  1090 , the processor system  1000  illustratively communicates with other computer components, including but not limited to, a hard drive  1030  and one or more removable storage memory  1050 . The imaging device  100  may be combined with a processor, such as a central processing unit, digital signal processor, or microprocessor, with or without memory storage on a single integrated circuit or on a different chip than the processor. 
         [0028]    It should again be noted that although the embodiments of the invention have been described with specific reference to CMOS imaging devices, the embodiments have broader applicability and may be used in any imaging apparatus which generates pixel output values, including charge-coupled devices CCDs and other imaging devices.