Patent Abstract:
An imager has first and second photosensitive sites and an interpolator located in a semiconductor substrate. The first photosensitive site is configured to receive light having a spectral component, and the second photosensitive site is configured to measure the level of the spectral component in light received by the second photosensitive site. The interpolator is configured to estimate the level of the spectral component in the light received by the first photosensitive site based on the measurement by the second photosensitive site.

Full Description:
BACKGROUND 
       [0001]    The invention relates to color interpolation. 
         [0002]      FIG. 1  shows a semiconductor imager  10  (e.g., a complementary metal-oxide semiconductor (CMOS) imager) might be used to electrically capture “snapshots” of an optical image. The imager is used to convert an optical image into an electrical representation. The imager  10  accomplishes this conversion through the use of an array of sensing elements arranged as pixel cells  12  that sense the intensity of light coming from the image. The “exposure time” for each snapshot depends on an integration interval during which each pixel cell  12  integrates an indication of the number of photons of light striking the cell  12  (i.e., measures an intensity of light striking the cell  12 ) and provides an indication of the integrated value via an analog output signal. For CMOS imagers, on-chip analog conditioning circuitry  14  (e.g., circuitry to perform correlated double sampling and gain control) and an analog-to-digital converter (ADC)  16  process the analog outputs of the pixel cells  12  to provide a digital representation of the image which can be retrieved from the imager  10  through a parallel port interface  18 . 
         [0003]    The pixel cells  12  provide an indication of the intensity of light striking the cell  12 . Hence, the above-described arrangement may be used to produce a monochrome or luminance only representation of the image. However, to produce color representations of the image, the imager also needs to provide information about primary colors (e.g., red, green and blue colors) of the image. To accomplish this, each pixel cell  12  is configured to sense the intensity level of light in one of the primary color bands. A typical way to accomplish this is to cover each pixel cell  12  with a spectrum-discriminating filter (e.g., a filter that only allows a red, green or blue color band to pass through the filter). As a result, some pixel cells  12  sense red light, some pixel cells  12  sense green light and some pixel cells  12  sense blue light. As an example, a multi-band filter pattern  20  (see  FIG. 2 ) placed over the array of pixel cells  12  may have alternating red, green and blue filter stripes that extend along the columns of the array. Thus, each filter stripe of the pattern  20  configures one of the columns of the array to sense light in one of the primary color bands. As another example, the filter pattern may be checkered, instead of striped. 
         [0004]    Each pixel cell  12  captures a portion of the image. To maximize the resolution of the image when reproduced on a display, it is desirable to form a one-to-one correspondence between the pixel cells  12  of the imager  10  and pixels of the display. However, with color imagers, three adjacent pixel cells  12  (each pixel cell  12  sensing a different primary color band) are typically used to provide the information needed to form one pixel on the display. Thus, when used to capture color images, the effective display pixel resolution of the imager  10  typically is one third of the actual pixel cell  12  resolution. 
         [0005]    For purposes of preserving a one-to-one correspondence between the pixel cells  12  and the pixels of the display, one solution is to form an imager having three times as many pixel cells as corresponding pixels of the display to compensate for the three primary colors. Referring to  FIG. 3 , another solution is to use three imagers  22 ,  24 , and  28 , one for each primary color band of the image. Thus, for example, one imager  22  (covered by a red filter) senses red light, one imager  24  (covered by a green filter) senses green light, and one-imager  26  (covered by a blue filter) senses the blue light coming from the image. Dichroic plates  28  may be used to split the light into beams into its primary colors. 
         [0006]    Referring to  FIG. 4 , a third solution might be to use an off chip discrete-time signal processing (DSP) engine  30  to interpolate the two missing colors for each pixel cell  12 . To accomplish this, the DSP engine  30  processes the color information provided by adjacent pixel cells  12 . Typically, nearest neighbors are weighted with predetermined coefficients and averaged to determine a color at a particular pixel cell location. For example, referring back to  FIG. 1 , a pixel cell  12   a  that is covered by a red filter provides a representation of a red color of the portion of the image striking the cell  12   a . To ascertain the blue color of the portion of the image otherwise striking the cell  12   a  (if not for the red filter), the DSP engine  30  averages (a weighted representation of) the outputs of adjacent pixel cells  12   b  and  12   c  (i.e., adjacent pixel cells covered by a blue filter) to interpolate the missing blue color. The DSP engine  30  also interpolates the green color of the portion of the image that would other strike the cell  12   a  in a similar manner. 
       SUMMARY OF THE INVENTION 
       [0007]    In general, in one aspect, the invention features an imager that has first and second photosensitive sites and an interpolator located in a semiconductor substrate. The first photosensitive site is configured to receive light having a spectral component, and the second photosensitive site is configured to measure the level of the spectral component in light received by the second photosensitive site. The interpolator is configured to estimate the level of the spectral component in the light received by the first photosensitive site based on the measurement by the second photosensitive site. 
         [0008]    Implementations of the invention may include one or more of the following. The first and/or second photosensitive sites may include a pixel cell and a filter that covers the pixel cell. The filter covering the first photosensitive site may be configured to prevent the spectral component from striking the pixel cell, and the filter covering the second photosensitive site may be configured to allow the spectral component to strike the pixel cell. The first photosensitive site may also be configured to measure the level of another spectral component in light received by the first photosensitive site, and the interpolator may be also configured to estimate the level of the another spectral component in the light received by the second photosensitive site based on the measurement by the first photosensitive site. 
         [0009]    The imager may also include a third photosensitive site (also located in the substrate) that is configured to measure the level of the other spectral component in light received by the third photosensitive site. The first photosensitive site may also be configured to receive light having the another spectral component, and the interpolator may also be configured to estimate the level of the spectral components in the light received by the first photosensitive site based on the measurements by the second and third photosensitive sites. 
         [0010]    In general, in another aspect, the invention features an imager that has first and second photosensitive sites and an interpolator located in a semiconductor substrate. Each first photosensitive site is configured to receive light having a spectral component, and each second photosensitive site is configured to measure the level of the spectral component in light received by the second photosensitive site. The interpolator is configured to estimate the level of the spectral component in the light received by at least one of the first photosensitive sites based on the measurements by the second photosensitive sites. 
         [0011]    Implementations of the invention may include one or more of the following. The interpolator may include an averaging circuit that is configured to perform the estimation by averaging some of the measurements by the second photosensitive sites. The interpolator may also include a scaling circuit that is configured to scale some of the measurements by predetermined coefficients before being averaged by the averaging circuit. The scaling circuit may be programmable to change one or more of the coefficients. The first and second photosensitive sites may be part of an array of photosensitive sites (e.g., located in a column of the array, a row of the array, or arranged in a rectangular block of an array). 
         [0012]    In general, in another aspect, the invention features a color imager for use with light having first, second and third primary color bands. The imager has first, second and third photosensitive sites and an interpolator located in a semiconductor substrate. Each first photosensitive site is configured to receive a portion of the light and measure a level of the first primary color band in the portion of light received by the first photosensitive site. Each second photosensitive site is configured to receive a portion of the light and measure a level of the second primary color band in the portion of light received by the second photosensitive site. Each third photosensitive site is configured to receive a portion of the light and measure a level of the third primary color band in the portion of light received by the third photosensitive site. The interpolator is configured to estimate the levels of the second and third primary color bands in the light received by the first photosensitive sites based on the measurements by the second and third photosensitive sites; estimate the levels of the first and third primary color bands in the light received by the second photosensitive sites based on the measurements by the first and third photosensitive sites; and estimate the levels of the first and second primary color bands in the light received by the third photosensitive sites based on the measurements by the first and second photosensitive sites. 
         [0013]    Implementations of the invention may include one or more of the following. The interpolator may be also configured to furnish a representation of the levels of the first, second and third primary color bands for each of the first, second and third photosensitive sites. The representation for each site may include a representation (e.g., a true color representation) of the color of the light received by the site. 
         [0014]    In general, in another aspect, the invention features a method that includes using a first photosensitive site located in a semiconductor substrate to receive light having a spectral component. A second photosensitive site located in the substrate is used to measure the level of the spectral component in light received by the second photosensitive site. An interpolator located in the substrate is used to estimate the level of the spectral component in the light received by the first photosensitive site based on the measurement by the second photosensitive site. 
         [0015]    In general, in another aspect, the invention features a method that includes using first photosensitive sites located in a semiconductor substrate to receive light having a spectral component. Second photosensitive sites located in the substrate are used to measure the level of the spectral component in light received by each of the second photosensitive sites. An interpolator located in the substrate is used to estimate the level of the spectral component in the light received by at least one of the first photosensitive sites based on the measurements by the second photosensitive sites. 
         [0016]    Among the advantages of the invention are one or more of the following. True color imaging occurs on a single semiconductor chip. The pixel cells of the imager and the pixels of the display have a one-to-one correspondence Only one imager is required. The imager may be used with many commonly used color filter patterns. 
         [0017]    Other advantages will become apparent from the following description and from the claims. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWING 
         [0018]      FIG. 1  is a schematic view of a semiconductor imager. 
           [0019]      FIG. 2  is a schematic view of color filters. 
           [0020]      FIG. 3  is a block diagram of a system to interpolate color. 
           [0021]      FIG. 4 . is a schematic view of an optical system to separate light into primary color components. 
           [0022]      FIG. 5  is a schematic view of a semiconductor imager. 
           [0023]      FIG. 6  is an electrical schematic diagram of circuitry of the imager of  FIG. 5 . 
           [0024]      FIG. 7A  is a representation of the contents of the serial register of  FIG. 6 . 
           [0025]      FIG. 7B  is a representation of the contents of the buffer of  FIG. 6 . 
           [0026]      FIG. 8  is a electrical schematic diagram of another imager. 
       
    
    
     DESCRIPTION OF THE PREFERRED EMBODIMENTS 
       [0027]      FIG. 5  shows a CMOS imager  50  located on a monolithic semiconductor substrate, or chip. The illustrated embodiment is constructed to furnish twenty-four bit True Color data, e.g., eight bits representing a red color, eight bits representing a green color, and eight bits representing a blue color, for every photosensitive site  51 . Each photosensitive site  51  is a region of the imager  50  that includes a pixel cell  52 . As a result of this arrangement, a one-to-one correspondence between pixel cells  52  of the imager  50  and pixels of a display used to display the image captured by the imager  50  is preserved without requiring a larger imager, complicated optics, or off-chip color interpolation. 
         [0028]    The imager  50  has an on chip color interpolator  58  which, for each photosensitive site  51 , estimates the level of the primary colors that are not sensed by the pixel cell  52  at that photosensitive site  51 . The color sensed by the pixel cell  52  is determined by a primary color filter of the site  51  that covers the pixel cell  52 . In this manner, the primary color filter (which is a red, green or blue filter) covers the cell  52 . Each cell  52  senses the level of light by measuring the intensity of the light in one of the primary color bands (e.g., red, green or blue) but does not sense the level of light in the other two primary color bands. The interpolator  58  estimates the missing color levels for the site  51  by using the outputs of pixel cells  52  in adjacent photosensitive sites  51  that are sensing these color levels. 
         [0029]    The photosensitive sites  51  (and corresponding pixel cells  52 ) are arranged in a rectangular array of rows and columns. To estimate the missing color levels for a given photosensitive site  51  (i.e., to estimate the color levels not sensed by the site  51 ), the interpolator  58  may be configured to use pixel cells  52  in the same row, same column, or both (e.g., the interpolator  58  may use a block of pixel cells  52  that surround the given photosensitive site  51 ). Although may configurations are possible, as discussed below, a multi-band column oriented filter pattern (See  FIG. 2 ) is assumed, and pixel cells  52  from the same row are used in the interpolation. 
         [0030]      FIG. 6  shows analog conditioning circuitry  54  to perform correlated double sampling of the analog outputs of the pixel cells  52  and provide gain control. This circuitry receives the analog outputs of the pixel cells  52 . The circuitry  54  furnishes its output to an analog-to-digital converter  56  which converts the analog outputs of the pixel cells  52  into digital data and supplies the digital data to the interpolator  58 . After an integration interval has passed, the pixel cells  52  have captured a snapshot of the image. At that time a column decoder  64  begins routing the outputs of the pixel cell  52  to the analog conditioning circuitry  54  for processing. The decoder  64  sequentially selects one row of pixels  52  and serially provides the analog outputs of the pixel cells  52  of the row that is selected (i.e., provides all of the columns of the selected row) to the analog conditioning circuitry  54 . A control circuit  62  controls the integration of the light by the pixel cells  52  and the overall timing of the imager  50 . The True Color data may be read from the imager  50  at a parallel port interface  60 . 
         [0031]    The interpolator  58  estimates the levels of the missing color levels for a given photosensitive site  51  using the outputs of other pixel cells  52  that are close to the given photosensitive site  51 . As one example, the interpolator  58  may be configured to use a one dimensional approach by serially processing photosensitive sites  51  and the corresponding pixel cells  52  at the photosensitive sites  51  from the same row of the array. The processing of a given photosensitive site  51  includes retrieving the color level sensed by the pixel cell  52  of the given photosensitive site  51  and estimating the missing color levels. The estimation uses the interpolator  58  to form the outputs of the last two pixel cells  52  that were processed and the next two pixel cells  52  to be processed to estimate the two missing color levels for the photosensitive site  51  currently being processed. The interpolator  58  performs a weighted average of the outputs from the pixel cells  52  to estimate the missing color levels. 
         [0032]    For example,  FIG. 7A  shows a photosensitive site  51   a  is covered by a blue filter which filters out red and green light from striking the corresponding pixel cell  52 . To estimate the red light that would otherwise strike the pixel cell  52  if not for the blue filter (i.e., to estimate the level of red light striking the photosensitive site  51   a ), the interpolator  58  forms a weighted average of the outputs of pixel cells  52  in adjacent photosensitive sites  51   b  and  51   c  that are covered by a red filter. Similarly, to estimate the green light that would otherwise strike the pixel cell  52  if not for the blue filter (i.e., to estimate the level of green light striking the photosensitive site  51   a ), the interpolator  58  uses a weighted average of the outputs of pixel cells  52  in adjacent photosensitive sites  51   d  and  51   e  that are covered by a green filter. 
         [0033]    The estimate of color level for a given photosensitive site  51  uses a number of different values. The weight given by the interpolator  58  to the actual color level from another photosensitive site  51  is a function of the distance between the given photosensitive site  51  and the photosensitive site  51  furnishing the actual color level. For example, to estimate the level of green light striking the photosensitive site  51   a  (see  FIG. 7A ), the interpolator  58  might be configured to apply twice as much weight to the output of the pixel cell  52  in adjacent photosensitive site  51   d  than to the output of the pixel cell  52  twice as far away, such as pixel cell  52   e.    
         [0034]      FIG. 6  shows the hardware of the interpolator  58  including a five stage serial register  66 . The least significant bits zero to fifteen of the register contain eight bit digital representations of actual color levels for the last two photosensitive sites  51  and corresponding pixel cells  52  processed. The most significant bits twenty-four to thirty-nine of the register  66  contain eight bit digital representations of actual color levels for the next two photosensitive sites  51  and corresponding pixel cells  52  to be processed. The other bits sixteen to twenty-three of the register  66  contain an eight bit representation of the actual color level for the photosensitive site  51  and corresponding pixel cell  52  being processed. 
         [0035]    Each photosensitive site  51  assembles the twenty-four bit True Color representation in a buffer  74  (of the parallel port interface  60 ) as follows. The interpolator  58  transfers the bits  16 - 23  of the register  66  which are representative of an actual color level, to the buffer  74  without any further processing. The interpolator  58  assigns a weight via scalar multipliers to the values represented by the bits  32 - 39  and  8 - 15  of the register  66 . The interpolator  58  also averages (via adders  70  and a “divide-by-two” circuitry  72 ) these values to estimate one of the missing color values, and stores the resultant eight bit color value in the buffer  74 . The twenty-four bit representation is completed by the interpolator  58  assigning a weight to the values represented by the bits  24 - 31  and  0 - 7 , average these values together, and stores the resultant eight bit color value in the buffer  75 . The twenty-four bit True Color value may then be retrieved from the buffer  74  (and from the parallel port interface  60 ) via an I/O interface  76  that is configured to communicate with off chip devices. 
         [0036]      FIGS. 7A and 7B  show the red-green-blue (“RGB”) byte ordering of the stored twenty-four bit color values  69  circularly rotates, and the most significant byte of the color value  69  corresponds to the actual color level sensed by the pixel cell  52  in the corresponding photosensitive site  51 . As an example, for the twenty-four bit color value  69   a  representative of the color sensed by the pixel cell  52  in photosensitive site  51   a , the most significant byte represents the actual blue color level (B 1 ) sensed by the pixel cell in photosensitive site  51   a , the next significant byte represents the estimated red color level for the photosensitive site  51   a , and the least significant byte represents the estimated green color level for the photosensitive site  51   a.    
         [0037]    The gains of the scalar multipliers  68  (i.e., the weighting applied by the interpolator  58 ) may either be fixed or programmable.  FIG. 6  shows the gains being programmable, with the I/O interface  76  having writable and readable registers used to program the gains of the multipliers. 
         [0038]    The one dimensional color interpolation approach discussed above can be extended to two dimensional interpolation. In such an approach, the outputs from pixel cells  52  from more than one row are used to estimate the missing color levels of a photosensitive site  51 . For example,  FIG. 8  shows another interpolator  90  of another imager  100  having three serial, five stage registers  92 . Similar to the register  66 , each register  92  contains digital representations of five adjacent pixel cells  52  of one of three adjacent different rows. Each register  92  has representations from the same column of pixel cells  52 . Thus, the bits of the registers  92  represent the outputs of a 5×3 block of pixel cells  52 . The interpolator  90  includes analog conditioning circuitry  95  and an A/D converter  97  for each register  92 . The integrator  90  also has a gain circuit  94  (e.g., scalar multipliers) and an averaging circuit  96  to provide weighted averaging for the interpolation. The imager  100  uses a column decoder  91  that has three serial outputs associated with three different adjacent rows of pixel cells  52 . A control circuit  94  controls the integration of the light by the pixel cells  52  and the overall timing of the imager  100 . 
         [0039]    Other embodiments are within the scope of the following claims. For example, other filter patterns, such as a checkered filter pattern may cover the array of pixel cells. The array may have more pixel cells dedicated to sensing one of the primary colors than to the other primary colors. For example, to improve the perceived luminance of the reproduced image, the array may have more pixel cells dedicated to sensing green (a color that closely matches the luminance of the human eye) color levels. The imager may represent color in a format other than a True Color representation. For example, six bits may be used to represent a green color level, five bits may be used to represent a blue color level, and five bits may be used to represent a red color level.

Technology Classification (CPC): 7