Abstract:
An object perceived by a lateral sensor array effected by parallax is shifted to correct for parallax error. A void resulting from said shift is filled by examining and interpolating image and color content from other locations.

Description:
BACKGROUND OF THE INVENTION 
       [0001]    1. Field of the Invention 
         [0002]    Embodiments of the invention relate generally to digital image processing and more particularly to methods and apparatuses for image pixel signal readout. 
         [0003]    2. Background of the Invention 
         [0004]    There is a current interest in using CMOS active pixel sensor (APS) imagers as low cost imaging devices. An example pixel  10  of a CMOS imager  5  is described below with reference to  FIG. 1 . Specifically,  FIG. 1  illustrates an example 4T pixel  10  used in a CMOS imager  5 , where “4T” designates the use of four transistors to operate the pixel  10  as is commonly understood in the art. The 4T pixel  10  has a photosensor such as a photodiode  12 , a transfer transistor  11 , a reset transistor  13 , a source follower transistor  14 , and a row select transistor  15 . It should be understood that  FIG. 1  shows the circuitry for the operation of a single pixel  10 , and that in practical use there will be an M×N array of identical pixels arranged in rows and columns with the pixels of the array being accessed by row and column select circuitry, as described in more detail below. 
         [0005]    The photodiode  12  converts incident photons to electrons that are transferred to a storage node FD through the transfer transistor  11 . The source follower transistor  14  has its gate connected to the storage node FD and amplifies the signal appearing at the node FD. When a particular row containing the pixel  10  is selected by the row select transistor  15 , the signal amplified by the source follower transistor  14  is passed to a column line  17  and to readout circuitry (not shown). It should be understood that the imager  5  might include a photogate or other photoconversion device, in lieu of the illustrated photodiode  12 , for producing photo-generated charge. 
         [0006]    A reset voltage Vaa is selectively coupled through the reset transistor  13  to the storage node FD when the reset transistor  13  is activated. The gate of the transfer transistor  11  is coupled to a transfer control line, which serves to control the transfer operation by which the photodiode  12  is connected to the storage node FD. The gate of the reset transistor  13  is coupled to a reset control line, which serves to control the reset operation in which Vaa is connected to the storage node FD. The gate of the row select transistor  15  is coupled to a row select control line. The row select control line is typically coupled to all of the pixels of the same row of the array. A supply voltage Vdd, is coupled to the source follower transistor  14  and may have the same potential as the reset voltage Vaa. Although not shown in  FIG. 1 , column line  17  is coupled to all of the pixels of the same column of the array and typically has a current sink transistor at one end. 
         [0007]    As known in the art, a value is read from the pixel  5  using a two-step process. During a reset period, the storage node FD is reset by turning on the reset transistor  13 , which applies the reset voltage Vaa to the node FD. The reset voltage actually stored at the FD node is then applied to the column line  17  by the source follower transistor  14  (through the activated row select transistor  15 ). During a charge integration period, the photodiode  12  converts photons to electrons. The transfer transistor  11  is activated after the integration period, allowing the electrons from the photodiode  12  to transfer to and collect at the storage node FD. The charges at the storage node FD are amplified by the source follower transistor  14  and selectively passed to the column line  17  via the row select transistor  15 . As a result, two different voltages—a reset voltage (Vrst) and the image signal voltage (Vsig)—are readout from the pixel  10  and sent over the column line  17  to readout circuitry, where each voltage is sampled and held for further processing as known in the art. 
         [0008]      FIG. 2  shows a CMOS imager integrated circuit chip  2  that includes an array  20  of pixels and a controller  23  that provides timing and control signals to enable the reading out of the above described voltage signals stored in the pixels in a manner commonly known to those skilled in the art. Typical arrays have dimensions of M×N pixels, with the size of the array  20  depending on a particular application. Typically, in color pixel arrays, the pixels are laid out in a Bayer pattern, as is commonly known. The imager  2  is read out a row at a time using a column parallel readout architecture. The controller  23  selects a particular row of pixels in the array  20  by controlling the operation of row addressing circuit  21  and row drivers  22 . Charge signals stored in the selected row of pixels are provided on the column lines  17  to a readout circuit  25  in the manner described above. The signals (reset voltage Vrst and image signal voltage Vsig) read from each of the columns are sampled and held in the readout circuit  25 . Differential pixel signals (Vrst, Vsig) corresponding to the readout reset signal (Vrst) and image signal (Vsig) are provided as respective outputs Vout 1 , Vout 2  of the readout circuit  25  for subtraction by a differential amplifier  26 , and subsequent processing by an analog-to-digital converter  27  before being sent to an image processor  28  for further processing. 
         [0009]    In another aspect, an imager  30  may include lateral sensor arrays as shown in  FIG. 3 . This type of imager is also known as an “LSA” or “LiSA” imager, has color planes separated laterally into three distinct imaging arrays. As depicted in the top plan view of  FIG. 3 , the imager  30  has three M×N arrays  50 B,  50 G,  50 R, one for each of the three primary colors Blue, Green, and Red, instead of the having one Bayer patterned array. The distance between the arrays  50 B,  50 G,  5 OR shown as distance A. An advantage of using an LSA imager is that part of the initial processing for each of the colors is done separately; as such, there is no need to adjust the processing circuits (for gain, etc.) for differences between image signals from different colors. The distance between the arrays shown as distance A. 
         [0010]    A disadvantage of using an LSA imager is the need to correct for increased parallax error that often occurs. Parallax is generally understood to be an array displacement divided by the projected (object) pixel size. In a conventional pixel array that uses Bayer patterned pixels, four neighboring pixels are used for imaging the same image content. Thus, two green pixels, a red pixel, and a blue pixel are co-located in one area. With the four pixels being located close together, parallax error is generally insignificant. In LSA imagers, however, the parallax error is more pronounced because each color is spread out among three or more arrays.  FIG. 4  depicts a top plan view of a portion of an LSA imager  30  and an object  66 . Imager  30  includes three arrays  50 B,  50 G,  5 OR, and lenses  51 B,  51 G,  51 R for each of the arrays, respectively. 
         [0011]    Parallax geometry is now briefly explained. In the following equations, δ is the width of one pixel in an array  50 R,  50 G,  50 B, D is the distance between the object  66  and a lens (e.g., lenses  51 R,  51 G,  51 B), and d is the distance between a lens and an associated array. Δ is the projection of one pixel in an array, where object  66  embodies that projection. Δ decreases as D increases. Σ is the physical shift between the centers of the arrays  50 R,  50 G,  50 B. Σ is calculated as follows: Σ=A·N·δ, where A is the gap between the pixel arrays, and N is the number of pixels in the array. 
         [0012]    If the green pixel array  50 G is in between the blue pixel array  50 B and the red pixel array  50 R, as depicted in  FIG. 4 , and used as a reference point, then −Σ is the shift from the green pixel array  50 G to the red pixel array  50 R. Furthermore, +Σ is the shift from the green pixel array  50 G to the blue pixel array  50 B. Γ is the angular distance between similar pixels in different color channels to the object  66 . Γ changes as D changes. θ, is the field of view (FOV) of the camera system. γ is the angle that a single pixel in an array subtends on an object  66 . Imager software can correlate the separation between the pixel arrays in an LSA imager  30 . σ is sensor shift that software in an imager applies to correlate corresponding pixels. σ is generally counted in pixels and can be varied depending on the content of the image. P is the number of pixels of parallax shift. P can be computed based on the geometric dimensions of the imager  30  and the object  66 , as depicted in  FIG. 4 . Parallax can be calculated from the spatial dimensions as follows: 
         [0000]    
       
         
           
             
               
                 
                   P 
                   = 
                   
                     
                       
                         A 
                         · 
                         N 
                         · 
                         δ 
                       
                       Δ 
                     
                     = 
                     
                       Σ 
                       Δ 
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
             
               
                 
                   Δ 
                   = 
                   
                     
                       δ 
                       · 
                       D 
                     
                     d 
                   
                 
               
               
                 
                   ( 
                   2 
                   ) 
                 
               
             
             
               
                 
                   P 
                   = 
                   
                     
                       
                         A 
                         · 
                         N 
                         · 
                         δ 
                       
                       D 
                     
                     ≈ 
                     
                       
                         A 
                         · 
                         N 
                         · 
                         f 
                       
                       Δ 
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
             
               
                 
                   Δ 
                   = 
                   
                     
                       2 
                       · 
                       D 
                       · 
                       
                         tan 
                          
                         
                           ( 
                           
                             Θ 
                             2 
                           
                           ) 
                         
                       
                     
                     N 
                   
                 
               
               
                 
                   ( 
                   4 
                   ) 
                 
               
             
             
               
                 
                   P 
                   = 
                   
                     
                       A 
                       · 
                       δ 
                       · 
                       
                         N 
                         2 
                       
                     
                     
                       2 
                       · 
                       D 
                       · 
                       
                         tan 
                          
                         
                           ( 
                           
                             Θ 
                             2 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   5 
                   ) 
                 
               
             
             
               
                 
                   
                     P 
                     - 
                     σ 
                   
                   = 
                   
                     
                       Σ 
                       · 
                       f 
                     
                     
                       δ 
                       · 
                       D 
                     
                   
                 
               
               
                 
                   ( 
                   6 
                   ) 
                 
               
             
           
         
       
     
         [0013]    Parallax can also be calculated from the angular dimensions as follows: 
         [0000]    
       
         
           
             
               
                 
                   P 
                   = 
                   
                     Γ 
                     γ 
                   
                 
               
               
                 
                   ( 
                   7 
                   ) 
                 
               
             
             
               
                 
                   
                     Γ 
                     ≈ 
                     
                       
                         A 
                         · 
                         N 
                         · 
                         γ 
                       
                       D 
                     
                   
                   = 
                   
                     Σ 
                     D 
                   
                 
               
               
                 
                   ( 
                   8 
                   ) 
                 
               
             
             
               
                 
                   
                     γ 
                     ≈ 
                     
                       Δ 
                       D 
                     
                   
                   = 
                   
                     δ 
                     d 
                   
                 
               
               
                 
                   ( 
                   9 
                   ) 
                 
               
             
             
               
                 
                   P 
                   = 
                   
                     
                       
                         A 
                         · 
                         N 
                         · 
                         d 
                       
                       D 
                     
                     ≈ 
                     
                       
                         A 
                         · 
                         N 
                         · 
                         f 
                       
                       Δ 
                     
                   
                 
               
               
                 
                   ( 
                   10 
                   ) 
                 
               
             
             
               
                 
                   γ 
                   ≈ 
                   
                     
                       2 
                       · 
                       
                         tan 
                          
                         
                           ( 
                           
                             Θ 
                             2 
                           
                           ) 
                         
                       
                     
                     N 
                   
                 
               
               
                 
                   ( 
                   11 
                   ) 
                 
               
             
             
               
                 
                   P 
                   = 
                   
                     
                       A 
                       · 
                       δ 
                       · 
                       
                         N 
                         2 
                       
                     
                     
                       2 
                       · 
                       D 
                       · 
                       
                         tan 
                          
                         
                           ( 
                           
                             Θ 
                             2 
                           
                           ) 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   12 
                   ) 
                 
               
             
           
         
       
     
         [0014]    Thus the number of pixels of parallax shift P is calculated with the same parameters for both spatial and angular dimensions. 
         [0015]    Hyperparallax, or Hyperparallax distance, is the distance at which a pixel shift of one occurs.  FIG. 5   a  depicts a top down block representational view of an image scene perceived by an imager with a shift σ of 0. According to equation 6, P equals 0 when D=∞, P equals 1 when D=D HP , P equals 2 when D=D HP /2. Thus, in images received by the imager having arrays  50 R,  50 G,  50 B from an object at a distance D=∞, there is no parallax shift. In images received from an object at a distance D=2*D HP , there is a ½ pixel of parallax shift. In images received from an object at distance D=D HP , there is a 1 pixel parallax shift. In images received from an object at distance D=D HP /2, there are 2 pixels of parallax shift. 
         [0016]      FIG. 5   b  depicts a top down block representational view of an image scene perceived by an imager with a shift σ of 1. According to equation 6, P equals −1 when D=∞, P equals 0 when D=D HP , P equals 1 when D=D HP /2. Thus, in images received by the imager having arrays  50 R,  50 G,  50 B from an object at distance D=∞, there is a −1 pixel parallax shift. In images received from an object at distance D=2*D HP , there is a ½ pixel parallax shift. In images received from an object at distance D=D HP , there is no parallax shift. In images received from an object at distance D=D HP /2, there is a 1 pixel parallax shift. 
         [0017]    Imager shift&#39;s σ can be applied selectively to image content, where none, some, or all of the image content is adjusted. In an image that has objects at different distances from an imager, different σ&#39;s can be applied depending on the perceived distance of the object. 
         [0018]    However, when applying a parallax shift to an image, there is a void that occurs in the area behind the shifted pixels. For example, if an image is shifted 2 pixels to the left, there will be portions of 2 columns that will be missing image content because of the shift. Thus, there is a need to correct for the lost image content due to a shift. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0019]      FIG. 1  is an electrical schematic diagram of a conventional imager pixel. 
           [0020]      FIG. 2  is a block diagram of a conventional imager integrated chip. 
           [0021]      FIG. 3  is a block diagram of a conventional lateral sensor imager. 
           [0022]      FIG. 4  depicts a top down view of a block representation of an image scene perceived by a lateral sensor imager 
           [0023]      FIGS. 5   a  and  5   b  depict a top down block representation of an image scene perceived by a lateral sensor imager. 
           [0024]      FIG. 6  depicts objects perceived by a lateral sensor array. 
           [0025]      FIG. 7  depicts objects perceived by a lateral sensor array. 
           [0026]      FIG. 8  depicts objects perceived by a lateral sensor array that are shifted, resulting in voids. 
           [0027]      FIG. 9  depicts shifted objects perceived by a lateral sensor array, voids and image content correction regions. 
           [0028]      FIG. 10  depicts shifted objects perceived by a lateral sensor array and patched voids. 
           [0029]      FIG. 11  is a block diagram representation of a system incorporating an imaging device constructed in accordance with an embodiment described herein. 
       
    
    
     DETAILED DESCRIPTION OF THE INVENTION 
       [0030]    In the following detailed description, reference is made to the accompanying drawings, which are a part of the specification, and in which is shown by way of illustration various embodiments of the invention. These embodiments are described in sufficient detail to enable those skilled in the art to make and use them. It is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes, as well as changes in the materials used, may be made. 
         [0031]    Embodiments disclosed herein provide de-parallax correction, which includes interpreting and replacing image and color content lost when performing a de-parallax shifting of image content. An embodiment of the invention there are four steps of the de-parallax correction process: identification, correlation, shifting, and patching. 
         [0032]    The method is described with reference to  FIGS. 6-10  which depicts three lateral sensor arrays  50 R,  50 G,  50 B representing three color planes red, green, blue, respectively. Each array  50 R,  50 G,  50 B has a respective center line  91 R,  91 G,  91 B used as a reference point for the following description. The center array, i.e., array  50 G, serves as a reference array. Typically an image represented in array  50 G is shifted by an amount ±X in arrays  50 R,  50 B. Depicted in each array  50 R,  50 G,  50 B are images  97 R,  97 G,  97 B and  95 R,  95 G,  95 B, respectively, corresponding to two images captured by the imager. The object corresponding to images  95 R,  95 G,  95 B is farther away from the arrays  50 R,  50 G,  50 B when compared to the object corresponding to images  97 R,  97 G,  97 B; thus, there is little to no shift of the images  95 R,  95 G,  95 B from the respective center lines  91 R,  91 G,  91 B. Because the object corresponding to images  97 R,  97 G,  97 B is closer to the arrays  50 R,  50 G,  50 B, there is a noticeable shift of the red and blue images  95 R,  95 B from the respective center lines  91 R,  91 B. As image  95 G is the reference point there should be no shift in the green array  50 G. 
         [0033]    A first step of the de-parallax correction process is to identify the sections of the scene content that are affected by the parallax problem. This is a generally known problem with various known solutions. The presumptive first step in image processing is the recognition of the scene, separating and identifying content from the background and the foreground. Thus, with respect to the image scenes depicted in  FIG. 6 , conventional image processing would identify the scene content as having object images  97 R,  97 G,  97 B and  95 R,  95 G,  95 B. 
         [0034]    A second step of the de-parallax correction process is to correlate the parts of the identified object images. For example, image  97 R is to be aligned with image  97 G and image  97 B is to be aligned with image  97 G. Therefore, image  97 R would be correlated to image  97 G and image  97 B would be correlated to image  97 G. Thus, the left side of image  97 R would be correlated to the left side of image  97 G and the right side of image  97 R would be correlated to the right side of image  97 G. In addition, the left side of image  97 B would be correlated to left side of image  97 G and the right side of image  97 B would be correlated to right side of image  97 G. 
         [0035]    Similarly, image  95 R is lined up with image  95 G and image  95 B is lined up with image  95 G. Therefore, image  95 R would be correlated to image  95 G and image  95 B would be correlated to image  95 G. Thus, the left side of image  95 R would be correlated to the left side of image  95 G and the right side of image  95 R would be correlated to the right side of image  95 G. In addition, the left side of image  95 B would be correlated to the left side of image  95 G and the right side of image  95 B would be correlated to the right side of image  95 G. 
         [0036]    There are many different known techniques for correlating color planes. For example, there are known stereoscopic correlation processes or other processes that look for similar spatial shapes and forms. The correlation step results in an understanding of the relationship between corresponding image found in each of the arrays  50 R,  50 G,  50 B. 
         [0037]    The next step of the de-parallax correction process is to shift the images in the red and blue arrays  50 R,  50 B such that they line up with the images in the green array  50 G. Initially, the processing system of the imager are device housing the imager determines the number of pixels that need to be shifted. Presumably, image content in the red and blue color planes are shifted the absolute value of the same number of pixels. For example, red may be shifted to the right and blue may be shifted to the left, so that the image content is aligned.  FIG. 7  depicts arrays  50 R,  50 G,  50 B having images  97 R,  97 G,  97 B and  95 R,  95 G,  95 B. Arrays  50 R,  50 G,  50 B are shown with 18 rows and 18 columns of pixels, but it should be appreciated that this is a mere representation of pixel arrays having any number of rows and columns. 
         [0038]    As noted above, the amount of shifting of an image object typically depends on its distance from the imager. The closer to the imager, the greater the shifting required. Thus, images  97 R,  97 G,  97 B are not aligned and require shifting. The farther away from the imager, generally less shifting is required. Thus, images  95 R,  95 G,  95 B are substantially aligned and require substantially no shifting. As seen in  FIG. 7 , to shift image  97 R to align it with image  97 G, image  97 R should be shifted 2 pixels to the right. To shift image  97 B to align it with object  97 G, image  97 B should be shifted 2 pixels to the left. 
         [0039]    Shifting scene content in the red and blue arrays  50 R,  50 B results in some blank or “null” space in their columns.  FIG. 8  illustrates arrays  50 R,  50 G,  50 B having images  97 R,  97 G,  97 B and  95 R,  95 G,  95 B after images  97 R,  97 B were shifted. As seen in the red array  50 R, there is a void  98 R resulting from image  97 R being shifted 2 pixels to the right. Void  98 R is the width of the shift, i.e., 2 pixels, and the height of object  97 R, i.e., 4 pixels. Similarly, in array  50 B, there is a void  98 B resulting from image  97 B being shifted 2 pixels to the left. Void  98 B is the width of the shift, i.e., 2 pixels, and the height of object  98 R, i.e., 4 pixels. 
         [0040]    A fourth step of the de-parallax correction process is to patch all voids created by shifts. The patch occurs in two steps: patching image content and patching color content. The image information for a void can be found in the comparable section of at least one of the other arrays. The correlated image information contains pertinent information about picture structure, e.g., scene brightness, contrast, saturation, and highlights, etc. For example, as depicted in  FIG. 9 , image information for void  98 R in array  5 OR can be filled in from correlated image content  99 GR of array  5 OG and/or from correlated image content  99 B of array  50 B. Similarly, image information for void  98 B in array  50 B can be filled in from correlated image content  99 GB of array  5 OG and/or from correlated image content  99 R of array  50 R. Therefore, an image information patch is applied to the voids  98 R,  98 B from correlated image content  99 B,  99 R and/or correlated image content  99 GR,  99 GB, respectively. 
         [0041]    Although correlated image content  99 B,  99 R and/or correlated image content  99 GR,  99 GB are used to supply missing image information, they do not have correlated color content. The correlated color content must be interpolated. One approach to determining color content is to apply a de-mosaic process to suggest what the desired color is, e.g., red based on a known color, such as e.g., green. For example, green pixels may be averaged to determine missing red information. Another approach looks at other image content in the neighborhood of the desired pixel. 
         [0042]    Another approach is to use information from neighboring pixels. For example, a patching color content process for patching red color would interpolate color information in pixels of the array, e.g., array  50 R, surrounding the void, e.g., void  98 R and apply the information to the void, e.g., void  98 R. This approach may require recognizing and compensating for pixels having a different parallax than that of the void  98 R. An additional approach is to interpolate color values from the shifted pixels, e.g.,  97 R, and apply this color content information to the void, e.g., void  98 R. 
         [0043]    Referring to  FIG. 10 , at the completion of the patching process, a void, e.g., void  98 R, of the array, e.g., array  50 R, has been filled in with image and color content, e.g., content  98 R′, and the de-parallax correction process is completed. Information can be patched from one or a plurality of other arrays. Likewise, the blue void  98 B may be filled with image and color content  98 B: 
         [0044]    Generally, shifting and patching only applies to a small number of pixels. Thus, differences between actual and interpolated image and color content should be negligible. There are several approaches to applying a de-parallax correction process: no correction, some correction, and most (if not all) correction. With no correction, a resulting image from an imager array has parallax problems, which may or may not be noticeable, or which may be significant depending on the context of the scene. With some correction, a de-parallax correction process is applied to only certain objects in the scene and a resulting image from an imager array may still have parallax problems, which may or may not be noticeable, or which may be significant depending on the context of the scene. With most correction, a de-parallax correction process is applied to most if not all of the image, e.g., “locally,” and a resulting image from an imager array should have no parallax problems, which should not be noticeable, or which may be significant depending on the context of the scene. 
         [0045]    The above described image processing may be employed in an image processing circuit as part of an image device, which may be part of a processing system.  FIG. 11  shows a camera system  1100 , which includes an imaging device  1101  employing the processing described above with respect to  FIGS. 1-10  The system  1100  is an example of a system having digital circuits that could include image sensor devices. Without being limiting, such a system could include a computer system, camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other image acquisition or processing system. 
         [0046]    System  1100 , for example a camera system, generally comprises a central processing unit (CPU)  1110 , such as a microprocessor, that communicates with an input/output (I/O) device  1150  over a bus  1170 . Imaging device  1101  also communicates with the CPU  1110  over the bus  1170 . The system  1100  also includes random access memory (RAM)  1160 , and can include removable memory  1130 , such as flash memory, which also communicate with the CPU  1110  over the bus  1170 . The imaging device  1100  may be combined with a processor, such as a CPU, digital signal processor, or microprocessor, with or without memory storage on a single integrated circuit or on a different chip than the processor. In operation, an image is received through lens  1194  when the shutter release button  1192  is depressed. The illustrated camera system  1190  also includes a view finder  1196  and a flash  1198 . 
         [0047]    It should be appreciated that other embodiments of the invention include a method of manufacturing the system  1   100 . For example, in one exemplary embodiment, a method of manufacturing a CMOS readout circuit includes the steps of fabricating, over a portion of a substrate an integrated single integrated circuit, at least an image sensor with a readout circuit as described above using known semiconductor fabrication techniques.