Abstract:
A method of reading pixel data from a pixel array includes exposing each one of a plurality of regions of pixels a respective exposure time. Pixel data is read from the plurality of regions of pixels. The pixel data is interpolated from a first one of the plurality of regions of pixels to determine the pixel data of the regions of pixels other than the first one of the plurality of regions of pixels to generate a first image having the first exposure time. The pixel data is interpolated from the second one of the plurality of regions of pixels to determine the pixel data of the regions of pixels other than the second one of the plurality of regions to generate a second image having the second exposure time. The images are combined to produce a high dynamic range image.

Description:
CROSS-REFERENCE TO RELATED APPLICATION 
       [0001]    This application claims the benefit of U.S. Provisional Application No. 61/825,419, filed May 20, 2013. 
     
    
     BACKGROUND INFORMATION 
       [0002]    1. Field of the Disclosure 
         [0003]    The present invention relates generally image sensors. More specifically, examples of the present invention are related to image processing and signal processing techniques utilized in a high dynamic range (HDR) image sensors 
         [0004]    2. Background 
         [0005]    Standard image sensors have a limited dynamic range of approximately 60 to 70 dB. However, the luminance dynamic range of the real world is much larger. Natural scenes often span a range of 90 dB and over. In order to capture the highlights and shadows simultaneously, HDR technologies have been used in image sensors to increase the captured dynamic range. The most common techniques to increase dynamic range is to merge multiple exposures captured with standard (low dynamic range) image sensors into a single linear HDR image, which has much larger dynamic range than a single exposure image. 
         [0006]    One of the most common HDR sensor solutions would be having multiple exposures into one single image sensor. With different exposure integration times or different sensitivities (for example by inserting neutral density filters), one image sensor could have 2, 3, 4 or even more different exposures in a single image sensor. Multiple exposure images are available in a single shot using this HDR image sensor. However, overall image resolution is decreased using this HDR sensor compared to a normal full resolution image sensor. For example, for an HDR sensor that combines 4 different exposures in one image sensor, each HDR image would be only a quarter resolution of the full resolution image. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0007]    Non-limiting and non-exhaustive examples of the present invention are described with reference to the following figures, wherein like reference numerals refer to like parts throughout the various views unless otherwise specified. 
           [0008]      FIG. 1A  is a diagram illustrating one example of a four exposure HDR image sensor in accordance with the teachings of the present invention. 
           [0009]      FIG. 1B  is a diagram illustrating one example of pixel regions in which there is a first exposure in an example four exposure HDR image sensor in accordance with the teachings of the present invention. 
           [0010]      FIG. 2  is a diagram illustrating one example of a four exposure HDR image sensor in which the full resolution of the image sensor is captured utilizing an example upscaling technique in accordance with the teachings of the present invention. 
           [0011]      FIG. 3  is a diagram illustrating one example of a four exposure HDR image sensor in which the full resolution of the image sensor is captured utilizing an example pixel data substitution technique in accordance with the teachings of the present invention. 
           [0012]      FIG. 4  is a diagram illustrating one example of an imaging system including an image sensor pixel array having fast intra-frame autofocus in accordance with the teachings of the present invention. 
       
    
    
       [0013]    Corresponding reference characters indicate corresponding components throughout the several views of the drawings. Skilled artisans will appreciate that elements in the figures are illustrated for simplicity and clarity and have not necessarily been drawn to scale. For example, the dimensions of some of the elements in the figures may be exaggerated relative to other elements to help to improve understanding of various embodiments of the present invention. Also, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are often not depicted in order to facilitate a less obstructed view of these various embodiments of the present invention. 
       DETAILED DESCRIPTION 
       [0014]    As will be shown, methods and apparatuses directed to an HDR image sensor having full resolution are disclosed. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. In the following description numerous specific details are set forth to provide a thorough understanding of the embodiments. One skilled in the relevant art will recognize, however, that the techniques described herein can be practiced without one or more of the specific details, or with other methods, components, materials, etc. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring certain aspects. 
         [0015]    Reference throughout this specification to “one embodiment,” an embodiment, “one example,” or “an example” means that a particular feature, structure, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. Thus, the appearances of the phrases such as “in one embodiment” or “in one example” in various places throughout this specification are not necessarily all referring to the same embodiment or example. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments or examples. The following is a detailed description of the terms and elements used in the description of examples of the present invention by referring to the accompanying drawings. 
         [0016]    As will be shown, examples in accordance with the teachings of the present invention provide image processing techniques that recover the full resolution for an HDR image sensor that include two or more exposures in one single image sensor chip. It is noted that the example techniques described below utilize a four exposure HDR image sensor as an example. It is appreciated of course that an HDR image sensor having any plurality of exposures may also be utilized in accordance with the teachings of the present invention. 
         [0017]    In the example depicted in  FIG. 1A , an example four exposure HDR image sensor  100  is illustrated. In the illustrated example, a Bayer pattern is shown for explanation purposes. However it is appreciated that any other color or non-color pattern may also be utilized in accordance with the teachings of the present invention. As shown in the depicted example, image sensor  100  includes a plurality of pixel regions T0, T1, T2 and T3 arranged as shown in a two dimensional array. In the example, it is assumed that images obtained with the plurality of pixel regions T0, T1, T2, and T3 are acquired with exposures E0, E1, E2 and E3, respectively. Referring to the example illustrated in  FIG. 1B , it can be appreciated that if an image is captured using only the T0 pixel regions of image sensor  100  in which there is only the exposure E0, only a quarter of the total number of pixels of the image sensor  100  are available. In other words, the other three-quarters of the pixel regions other than the T0 pixel regions are not utilized to capture the image at exposure E0, and as a consequence, the full resolution of image sensor  100  is not captured directly. 
         [0018]    There are a variety of techniques that may be utilized in accordance with the teachings of the present invention to capture the full resolution of image sensor  100  at HDR. For instance, in one example, the full resolution of image sensor  100  may be captured at HDR with upscaling. Scaling is a non-trivial process that involves a trade-off between smoothness and sharpness. Various scaling methods could be utilized, such as for example nearest-neighbor interpolation, bilinear interpolation, spline interpolation, or other vector-based interpolation. In one example, bi-cubic interpolation method could be used to get an up-scaled image for each of the exposures E0, E1, E2, and E3. Assuming that: 
         [0000]        T 0′=upscale( T 0),
 
         [0000]        T 1′=upscale( T 1),
 
         [0000]        T 2=upscale( T 2), and 
         [0000]        T 3′=upscale( T 3),
 
         [0000]    the image data for the other pixel areas that have different exposures can be recovered in accordance with the teachings of the present invention. To illustrate,  FIG. 2  shows an example image sensor  200  in which in the which image data is recovered using an example upscaling technique for the missing pixel locations, (e.g., the non-T0 pixel regions) with the recovered T0′ data for the E0 exposure as shown in accordance with the teachings of the present invention. It is appreciated that recovered T1′ data can be used similarly for the non-T1 pixel regions for the E1 exposure, that recovered T2′ data can be used similarly for the non-T2 pixel regions for the E2 exposure, and that recovered T3′ data can be used similarly for the non-T3 pixel regions for the E3 exposure in accordance with the teachings of the present invention. Therefore, the full resolution of the image sensor  100  can be obtained for all of the exposures E0, E1, E2 and E3 to obtain full resolution HDR information using an example upscaling technique in accordance with the teachings of the present invention. 
         [0019]    In another example, pixel substitution could be utilized in the HDR image sensor, where it is assumed that the pixel response is linear with respect to exposure time. In the example, the pixel values of the missing pixel locations can be estimated from the other exposures. For instance, assuming that: 
         [0000]    
       
         
           
             
               r 
                
               
                   
               
                
               1 
             
             = 
             
               
                 E 
                  
                 
                     
                 
                  
                 0 
               
               
                 E 
                  
                 
                     
                 
                  
                 1 
               
             
           
         
       
       
         
           
             
               r 
                
               
                   
               
                
               2 
             
             = 
             
               
                 E 
                  
                 
                     
                 
                  
                 0 
               
               
                 E 
                  
                 
                     
                 
                  
                 2 
               
             
           
         
       
       
         
           
             
               r 
                
               
                   
               
                
               3 
             
             = 
             
               
                 E 
                  
                 
                     
                 
                  
                 0 
               
               
                 E 
                  
                 
                     
                 
                  
                 3 
               
             
           
         
       
     
         [0000]    then it can be assumed that 
         [0000]        T 0 — 1″= r 1× T 1
 
         [0000]        T 0 — 2″= r 2× T 2
 
         [0000]        T 0 — 3″= r 3× T 3
 
         [0000]    By integrating the estimated values from other exposures and the substituting the missing pixel locations with the estimated value from the other exposures, the full resolution image for T0 can therefore be recovered, as shown in the example illustrated in  FIG. 3 , which shows an example image sensor  300  in which the image data is recovered using an example pixel data substitution technique for the missing pixel locations in accordance with the teachings of the present invention. 
         [0020]    For the remaining exposures E1, E2 and E3, similar pixel data substitution techniques may be utilized to recover the T1, T2 and T3 pixel data in accordance with the teachings of the present invention. However, it is noted that if the longer exposure is saturated, the estimated exposure pixel value is lost since the useful pixel information is clipped by the full well capacity. In this case, the upscaled image pixel value may be utilized instead, using the previously discussed technique, in accordance with the teachings of the present invention. 
         [0021]    It is appreciated that the technique of pixel substitution suffers from a poor signal-to-noise ratio (SNR) as the exposure ratio increase. For instance, if E0/E3=64, then E3 has to apply 64× digital gain to get the same digital level of E0, which could have an 18 dB lower SNR compared to E0. Furthermore, if E3 falls into the nonlinear response region of the image sensor, the estimated values for E0 — 3″ will be inaccurate. On the other hand, the technique of upscaling suffers from decreased sharpness and/or invalid high frequency estimation, which result in zigzag artifacts due to the interpolation. 
         [0022]    In another example, the upscaling technique as well as the pixel data substitution techniques may both be utilized based on the exposure ratios to recover the missing pixel information in accordance with the teachings of the present invention. For instance, in one example, 
         [0000]    
       
         
           
             
               
                 T 
                  
                 
                     
                 
                  
                 
                   0 
                   recovered 
                 
               
               = 
               
                 T 
                  
                 
                     
                 
                  
                 
                   0 
                    
                   &#39;&#39; 
                 
               
             
             , 
             
               
 
             
              
             
               
                 
                   if 
                    
                   
                       
                   
                    
                   
                     
                       E 
                        
                       
                           
                       
                        
                       0 
                     
                     
                       E 
                        
                       
                           
                       
                        
                       3 
                     
                   
                 
                 &lt; 
                 
                   R 
                   
                     th 
                      
                     
                         
                     
                      
                     0 
                   
                 
               
               ; 
             
           
         
       
       
         
           
             
               
                 T 
                  
                 
                     
                 
                  
                 
                   0 
                   recovered 
                 
               
               = 
               
                 
                   
                     ( 
                     
                       1 
                       - 
                       w 
                     
                     ) 
                   
                   × 
                   T 
                    
                   
                       
                   
                    
                   
                     0 
                      
                     &#39;&#39; 
                   
                 
                 + 
                 
                   w 
                   × 
                   T 
                    
                   
                       
                   
                    
                   
                     0 
                     &#39; 
                   
                 
               
             
             , 
             
               
 
             
              
             
               
                 
                   if 
                    
                   
                       
                   
                    
                   
                     R 
                     
                       th 
                        
                       
                           
                       
                        
                       0 
                     
                   
                 
                 &lt; 
                 
                   
                     E 
                      
                     
                         
                     
                      
                     0 
                   
                   
                     E 
                      
                     
                         
                     
                      
                     3 
                   
                 
                 &lt; 
                 
                   R 
                   
                     th 
                      
                     
                         
                     
                      
                     1 
                   
                 
               
               ; 
               
                 
 
               
                
               
                 
                   where 
                    
                   
                       
                   
                    
                   w 
                 
                 = 
                 
                   
                     ( 
                     
                       
                         
                           E 
                            
                           
                               
                           
                            
                           0 
                         
                         
                           E 
                            
                           
                               
                           
                            
                           3 
                         
                       
                       - 
                       
                         R 
                         
                           th 
                            
                           
                               
                           
                            
                           0 
                         
                       
                     
                     ) 
                   
                   
                     ( 
                     
                       
                         R 
                         
                           th 
                            
                           
                               
                           
                            
                           1 
                         
                       
                       - 
                       
                         R 
                         
                           th 
                            
                           
                               
                           
                            
                           0 
                         
                       
                     
                     ) 
                   
                 
               
               ; 
               
                 
                   and 
                    
                   
                     
 
                   
                    
                   T 
                    
                   
                       
                   
                    
                   
                     0 
                     recovered 
                   
                 
                 = 
                 
                   T 
                    
                   
                       
                   
                    
                   
                     0 
                     &#39; 
                   
                 
               
             
             , 
             
               
 
             
              
             
               
                 if 
                  
                 
                     
                 
                  
                 
                   
                     E 
                      
                     
                         
                     
                      
                     0 
                   
                   
                     E 
                      
                     
                         
                     
                      
                     3 
                   
                 
               
               &gt; 
               
                 
                   R 
                   
                     th 
                      
                     
                         
                     
                      
                     1 
                   
                 
                 . 
               
             
           
         
       
     
         [0000]    Thus, for an HDR scene, the technique of upscaling may be utilized, while for a low dynamic range scene, the technique of pixel data substitution may be utilized to recover the full resolution of the HDR image sensor in accordance with the teachings of the present invention. 
         [0023]    In one example, after recovering the full resolution images for T0, T1, T2 and T3, an HDR combination process may be used to combine the full resolution images for T0, T1, T2 and T3 into a single HDR image. In one example, a tone-mapping process can be performed to compress the dynamic range and produce a normal image that is for suitable for display in accordance with the teachings of the present invention. 
         [0024]      FIG. 4  is a diagram illustrating one example of an imaging system  491  including an example HDR image sensor  492  having a plurality of image sensor pixels cells in accordance with the teachings of the present invention. As shown in the depicted example, imaging system  491  includes an HDR image sensor  492  coupled to control circuitry  498  and readout circuitry  494 , which is coupled to function logic  496 . 
         [0025]    In one example, HDR image sensor  492  is a two-dimensional (2D) array of image sensor pixel cells (e.g., pixels P1, P2, P3, . . . Pn). It is noted that HDR image sensor  492  may be an example of image sensor  100  of  FIGS. 1A-1B  or an example of image sensor  200  of  FIG. 2 , or an example of image sensor  300  of  FIG. 3 , and that similarly named and numbered elements referenced below are coupled and function similar to as described above. As illustrated, each pixel cell is arranged into a row (e.g., rows R1 to Ry) and a column (e.g., column C1 to Cx) to acquire image data of a person, place, object, etc., which can then be used to render a 2D image of the person, place, object, etc. 
         [0026]    In one example, after each pixel cell P1, P2, P3, . . . , Pn has acquired its image data or image charge, the image data is readout by readout circuitry  494  and then transferred to function logic  496 . In various examples, readout circuitry  494  may include HDR processing circuitry as discussed above, amplification circuitry, analog-to-digital (ADC) conversion circuitry, or otherwise. Function logic  496  may simply store the image data or even manipulate the image data by applying post image effects (e.g., HDR processing, tone-mapping processing, crop, rotate, remove red eye, adjust brightness, adjust contrast, or otherwise). In one example, readout circuitry  494  may readout a row of image data at a time along readout column lines (illustrated) or may readout the image data using a variety of other techniques (not illustrated), such as a serial readout or a full parallel readout of all pixels simultaneously. 
         [0027]    In one example, control circuitry  498  is coupled to HDR image sensor  492  to control operational characteristics of image sensor  492 . For example, control circuitry  498  may generate a shutter signal for controlling image acquisition. In one example, the shutter signal is a global shutter signal for simultaneously enabling all pixels within image sensor  492  to simultaneously capture their respective image data during a single acquisition window. In another example, the shutter signal is a rolling shutter signal such that each row, column, or group of pixels is sequentially enabled during consecutive acquisition windows. 
         [0028]    The above description of illustrated examples of the present invention, including what is described in the Abstract, are not intended to be exhaustive or to be limitation to the precise forms disclosed. While specific embodiments of, and examples for, the invention are described herein for illustrative purposes, various equivalent modifications are possible without departing from the broader spirit and scope of the present invention. Indeed, it is appreciated that the specific example voltages, currents, frequencies, power range values, times, etc., are provided for explanation purposes and that other values may also be employed in other embodiments and examples in accordance with the teachings of the present invention. 
         [0029]    These modifications can be made to examples of the invention in light of the above detailed description. The terms used in the following claims should not be construed to limit the invention to the specific embodiments disclosed in the specification and the claims. Rather, the scope is to be determined entirely by the following claims, which are to be construed in accordance with established doctrines of claim interpretation. The present specification and figures are accordingly to be regarded as illustrative rather than restrictive.