Abstract:
An image sensor device has a first region configured to sense only infrared illumination and a second region configured to not sense visible and infrared illumination.

Description:
RELATED APPLICATIONS 
       [0001]    This application is a continuation of U.S. application Ser. No. 14/063,069, filed Oct. 25, 2013 (allowed), which is a continuation of U.S. application Ser. No. 12/813,705, filed Jun. 11, 2010 and issued as U.S. Pat. No. 8,569,681 on Oct. 29, 2013, which is a divisional application of application Ser. No. 11/513,582, filed Aug. 31, 2006 and issued as U.S. Pat. No. 7,737,394 on Jun. 15, 2010, which applications are assigned to the assignee of the present invention and the entire contents of which are incorporated herein by reference. 
     
    
     TECHNICAL FIELD 
       [0002]    The present invention relates generally to optical devices and in particular the present invention relates to image sensors. 
       BACKGROUND 
       [0003]    Image sensors are used in many different types of electronic devices to capture an image. For example, consumer devices such as video cameras and digital cameras as well as numerous scientific applications use image sensors to capture an image. An image sensor is comprised of photosensitive elements that sense incident illumination and produces an electrical signal indicative of that illumination. Each photosensitive element is typically referred to as a picture element or pixel. 
         [0004]    Image sensors include charge coupled devices (CCD) and complementary metal oxide semiconductor (CMOS) sensors. Imaging sensors may be capable of capturing grayscale and/or color images. Imaging sensors capable of capturing color images often employ a color filter array (CFA) to separate visible illumination of different color. For example, each of the pixels can be covered with a red, green, or blue filter according to a specific pattern. For example, the Bayer pattern has a repeating pattern of an alternating row of green and red and an alternating row of blue and green. As a result of the filtering, each pixel of the color image captured by a CMOS sensor with CFA responds to only the illumination of wavelengths determined by the color filter of the three primary light colors. 
         [0005]    CFA color filters are typically transparent to infrared illumination. In video and still digital cameras, an infrared-cutoff filter is typically placed in front of the sensor to ensure that only visible wavelength illumination reaches the sensor. This is done to make possible accurate color reproduction in captured images. 
         [0006]    White balance is the compensation of an image for variation in color temperature of scene illuminant. Images are adjusted in such a way that gray objects will look gray when displayed on a standard output device with a predefined color temperature. Other colors displayed on the output device will also appear as they appeared during image capture. Without white balance, images captured using illuminants having a low temperature will exhibit a reddish cast when displayed on a standard RGB monitor with the color temperature of the white point of 6500 K. Illuminants of higher color temperature can create a bluish cast. The human eye can compensate for the different color temperatures. Both film and digital cameras, however, need to correct for different light sources in order to render an image in its true colors. 
         [0007]    In a film camera, the color correction can be accomplished manually by the camera user attaching a color correction filter over the lens. A digital camera can correct the color temperature automatically by estimating the color temperature of the scene illuminant. Then the entire image is adjusted by the difference between the scene illuminant color temperature and the color temperature of the white point of the target output device (e.g., a standard display). One problem with this approach is that the white balance algorithm has to be informed as to the correct color temperature of the scene illuminant. 
         [0008]    Various techniques exist in the art to determine the color temperature and spectral content of the illuminant. The color temperature of the illuminant can be reliably identified if one knows which objects in the scene are white. Determining the current lighting conditions in a digital camera may be difficult since cameras cannot reliably automatically identify which objects in the scene are white and which are not and thus must rely on other estimation techniques. However, illumination sources often vary with respect to how much infrared radiation they emit. Having an estimate of how much infrared radiation is present in the scene compared to visible illumination can provide information about the nature of the illuminant. For example, incandescent lamps emit visible light accompanied with strong infrared content. Office fluorescent lighting typically emits very little infrared when compared to scene luminance. Outdoor daylight contains moderate amounts of infrared compared to scene luminance. For the reasons stated above, and for other reasons stated below which will become apparent to those skilled in the art upon reading and understanding the present specification, there is a need in the art for detecting ambient infrared in a sensor equipped with an IR cut-off filter. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0009]      FIG. 1  shows a cross-sectional view of one embodiment of a portion of an image sensor. 
           [0010]      FIG. 2  shows a top view of one embodiment of an image sensor. 
           [0011]      FIG. 3  shows a top view of the image sensor that is beneath the infrared filter. 
           [0012]      FIG. 4  shows a flowchart of one embodiment of an IR sensing method. 
           [0013]      FIG. 5  shows a flowchart of one embodiment for determining IR avg . 
           [0014]      FIG. 6  shows a block diagram of one embodiment of an imaging system of the present invention. 
           [0015]      FIG. 7  shows a graph of pixel response for the IR-sensitive and IR-shielded pixels of an image sensor. 
       
    
    
     DETAILED DESCRIPTION 
       [0016]    In the following detailed description of the invention, reference is made to the accompanying drawings that form a part hereof and in which is shown, by way of illustration, specific embodiments in which the invention may be practiced. In the drawings, like numerals describe substantially similar components throughout the several views. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments may be utilized and structural, logical, and electrical changes may be made without departing from the scope of the present invention. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims and equivalents thereof. 
         [0017]      FIG. 1  illustrates a cross-sectional view of one embodiment of an image sensor. For purposes of clarity, not all of the layers are shown in this figure. For example, there may be metal interconnect layers formed between the layers shown as well as dielectric layers for insulation purposes. 
         [0018]    The sensor is comprised of a substrate  130  that incorporates a plurality of pixels or photodiodes  101 - 104 . The photodiodes  101 - 104  are responsible for converting light into an electrical signal for use by the circuitry that reads the photodiode information. The higher the intensity of the light that strikes the photodiode  101 - 104 , the greater the charge collected. 
         [0019]    Optionally, a color filter array (CFA)  112  is positioned over the photodiodes  101 - 104 . This layer comprises the different color filters required for the color system that is used. For example, the filters may be red  107 , green  106 , and blue  108  for an additive RGB system or cyan, yellow, magenta, and an extra green for a subtractive CYMG system. Each filter color separates out that particular color for that photodiode. In one embodiment, the color filters are formed in a “Bayer” color filter pattern that is well known in the art. 
         [0020]    An IR cutoff filter  120  is positioned over the CFA  112 . This filter blocks IR light from reaching certain ones of the photodiodes  101 - 104  such that only visible light reaches the desired photodiodes  101 - 104 . A top view of the embodiments of the IR cutoff filter  120  of the present invention are illustrated in  FIGS. 2 and 3  that are described subsequently. 
         [0021]    A lens  113  is positioned over the IR cutoff filter  120 . The lens  113  is responsible for focusing light on the photodiodes  101 - 104 . 
         [0022]    Optionally, a plurality of micro-lenses  150  can be formed over the photodiodes  101 - 104 . Each micro-lens  150  can be formed over a corresponding photodiode  101 - 103 . Each micro-lens  150  focuses the incoming light rays onto its respective photodiode  101 - 104  in order to increase the light gathering efficiency of the photodiode  101 - 104 . 
         [0023]      FIG. 2  illustrates a top view of one embodiment of the IR cutoff filter  203  over an image sensor. In this embodiment, the IR filter  203  is deposited directly on the sensor  200  or its protective cover in a patterned fashion. The active array pixel filter  203  blocks IR from reaching the active pixel array under the filter  203 . A slot  205  is formed in the periphery of the filter  203  over the array to allow IR to reach the pixels under the slot  205 . Optically “black” IR-sensitive (OBIR) pixels  300  as shown in  FIG. 3  are formed around the periphery of the sensor  200 . The OBIR pixels  300  are covered with a filter that blocks all visible wavelengths of light from hitting the pixels but is transparent to IR illumination. 
         [0024]    The slot  205  of  FIG. 2  is only one embodiment of the IR cutoff filter. The opening or openings in the IR filter can be of any shape that can be formed into the filter, thus exposing a predetermined quantity of pixels to IR illumination. For example, a slot can be positioned along only one side of the sensor. 
         [0025]      FIG. 3  illustrates the composition of the sensor pixel array lying directly beneath the IR filter. This figure shows the regular active array of pixels  301  that is covered by an IR cutoff filter. This array  301  is surrounded by a ring of OBIR pixels  300  that must be under the opening in the IR filter. The OBIR pixels  300  are separated from the active pixel array  301  by a ring of barrier pixels  305 . The barrier pixels  305  shield the active pixel array  301  from IR leaking into periphery active pixels in the active array  301  due to the long wavelength of IR. The barrier pixels  305  may be shielded from both visible and IR illumination (using a metal layer) and also act to substantially stop any charge leakage from the OBIR pixels  300  into the active pixels  301 . This avoids contaminating the response of active pixels by IR generated charge. 
         [0026]    The OBIR pixels  300  are the same type of pixels that are illustrated in  FIG. 1  but are not covered by the IR cutoff filter as is the active array of pixels  301 . This allows the OBIR pixels  300  to receive IR illumination. 
         [0027]      FIG. 3  also includes optically and IR black (OIRB) pixels  303  that are shielded from all light, both visible and IR. This can be accomplished by a metal layer. These pixels  303  may or may not be under the slot in the IR filter. The active pixel array  301  is covered with the patterned array of color filters (e.g., Bayer CFA). 
         [0028]      FIG. 4  illustrates a flowchart of one embodiment of a method for IR detection in accordance with the image sensors of  FIGS. 2 and 3 . This method reads the light signal from the OBIR pixels  401 . These signals are averaged and normalized by analog gain  403  to obtain IR avg  as described later with reference to  FIG. 5 . 
         [0029]    After the averaging and normalization of the OBIR signals  403 , the light signals from the active pixels adjacent to the IR slot shown in  FIG. 3  are analyzed  405 . These signals are averaged and normalized  407  by analog gains to generate G avg . G avg  is then used to calculate the ratio  409  of G avg /IR avg . 
         [0030]    Averaging the signals is accomplished by the equation: 
         [0000]        G   avg =(avg( P   ij  green1)/Gain G 1+avg( P   ij  green2)/Gain G 2)/2 /T   int activepixels    (1)
 
         [0000]    where ‘P ij  green1’ is a light signal from a pixel located in row i and column j in the sensor array, covered by a green CFA filter and located in such rows i that contain pixels covered with green and red CFA filters, ‘P ij  green2’ is a light signal from a pixel located in row i and column j in the sensor array, covered by a green CFA filter and located in such rows i that contain pixels covered with green and blue CFA filters, ‘GainG1’ is the gain applied to the green1 pixels, ‘GainG2’ is the gain applied to the green2 pixels, and ‘T int activepixels ’ is the integration time the active pixels are receiving. 
         [0031]    During a normalization process, GainR, GainB, GainG1, and GainG2 for the Bayer pattern are the four typical gains of an image sensor. The gains are applied to corresponding pixels in the Bayer pattern. Since OBIR pixels may only differ from regular pixels by having a black filter, the red, green1, green2, and blue pixel gains keep getting applied to OBIR pixels as well in order to simplify sensor design. 
         [0032]    Referring again to  FIG. 4 , after the averaging and normalizing of the OBIR signals  403 , the light signals from the active pixels adjacent to the IR slot shown in  FIG. 3  are analyzed  405 . These signals are then averaged and normalized  407  by analog gain to generate G avg . IR avg  is discussed subsequently with reference to  FIG. 5 . IR avg  is then used to calculate the ratio  409  of G avg /IR avg . Alternate embodiments can calculate other ratios such as Y avg /IR avg  where Y avg  is the average luminance of pixels after demosaicing and, optionally, color correction. 
         [0033]      FIG. 5  illustrates a flowchart of one embodiment for determining IR avg . The method begins with zeroing a row counter, r  501 . The signals from the OBIR pixels are then read  503 . These signals are used to determine four averages of P ij  from the OBIR pixels and are affected by Blue/Red/Green1/Green2 gain. These averages are denoted as IR avgR , IR avgB , IR avgG . 
         [0000]        IR   avgG ( r )=avg(avg( P   ij  green1)/Gain G 1, avg( P   ij  green2)/(Gain G 2)/ T   int IR    (2)
 
         [0000]        IR   avgR ( r )=avg( P   ij  red)/Gain R/T   int IR    (3)
 
         [0000]        IR   avgB ( r )=avg( P   ij  blue)/Gain B/T   int IR    (4)
 
         [0000]    where P ij  are OBIR pixels from the r th  inner pixel ring as detailed below. T int IR  is the integration time OBIR and OIRB pixels are receiving, which is typically equal to T int  active pixels. If pixel ring r contains no pixels of type green1 or green2, the corresponding term is excluded from the calculation of the average in (2). 
         [0034]    These averages are compared to a saturation threshold  507 , T sat . This threshold is determined by the maximum pixel value D max  that can be read multiplied by a saturation constant, K sat  (i.e., T sat =K sat *D max ). For example, a maximum pixel value of 1023 multiplied by a constant of 18%. If ring r contains no pixels affected by the blue or red gains, the comparison for the average value of that type of pixel is not performed. 
         [0035]    If none of the averages are less than T sat    507 , the pixels being considered are all close to IR saturation. Therefore, “more inner” OIRB pixels that are further from the OBIR pixels are then read  509 . The “more inner” OIRB pixels are read by incrementing the row counter  511  and repeating the OBIR pixel read  503 . This is repeated until non-saturated pixels are found. Since these pixels are shielded from visible and IR illumination, the signal generated by these pixels is due largely to charge and light leakage from adjacent OBIR pixels, see  FIG. 7 . Hence such signal may be substantially attenuated compared to OBIR pixel response thus allowing to assess IR illumination even when OBIR pixels are overexposed. 
         [0036]    Charge leakage is the result of the fact that infrared photons are absorbed at a higher depth than the visual wavelength photons due to solid state sensor construction typically having a longer average travel distance before the infrared photo-generated carriers are collected. Thus, the infrared-generated carriers can leak under the optical shield of the OIRB pixels. The OIRB pixels collect the infrared-generated charge and their response is increased. Illumination free of infrared will have no effect on OIRB pixel response. The magnitude of that component can then be determined in comparison to the visible light illumination of the edge pixels. 
         [0037]    OIRB pixels are particularly effective in cases where levels of IR radiation are so high as to saturate responses of OBIR pixels. For example, as the image sensor operates, the exposure for active pixels is typically selected to result in capture of visually pleasing images. Since OBIR pixels differ from active pixels only by having a different CFA filter, their exposure would typically be identical to that of the active pixels. This is true unless, optionally, the sensor is specifically designed to allow setting exposure of OBIR pixels independently from active pixels. In such an embodiment, the sensor would require a more complicated design. 
         [0038]    When active pixels are exposed in a way optimal for image capture, OBIR pixel response can be saturated if the scene illuminant has a high IR content. In such case, values of OIRB pixels should be read out instead of OBIR. 
         [0039]    After non-saturated pixels are found, the average of the averages that are less than the saturation threshold T sat  is determined  513 . This is determined by the following equation where, typically, T intIR =T intactivepixels  and K ir  is an IR sensitivity coefficient: 
         [0000]        IR   avg ( r )=avg( IR   avgR ( r ),  IR   avgB ( r ),  IR   avgG ( r ))/Kir( r )   (5)
 
         [0040]    In the formula above, if IR avgR , IR avgG , and/or IR avgB  are less than T sat , or pixels of a certain type (e.g., affected by the blue or red gains) do not exist in ring r, these averages are excluded from the average. 
         [0041]    If IR coefficient is set to 1 for OBIR, K ir (0) is defined as 1 for OBIR pixels, K ir (0)=1. K ir  for inner OIRB pixels is typically substantially less than 1. For example, the inner row of OIRB pixels could have K ir (1)=0.05. The next row, K ir (2) might have K ir (2)=0.0025. The coefficient would get progressively smaller due to reduced sensitivity of inner OIRB pixels. K ir (r) can be determined using a calibration procedure. 
         [0042]    One such calibration procedure illuminates the sensor with IR that has spectra expected to be encountered in actual operation. A flat field is then imaged. The sensor exposure and gains are adjusted to make responses of OIRB pixels equal to a constant (e.g., T sat ). Exposure T int (r) is recorded for r=0, as T int (0). Gains are then frozen. For each inner ring of OIRB pixels, exposure is adjusted to make their responses equal to T sat . The OIRB pixels probably saturate in this condition. T int (r) is recorded for r=1, 2, etc. If K ir (0)=1, K ir (r)=T int (0)/T int (r). 
         [0043]    The embodiment of the sensor of  FIGS. 2 and 3  is optimized for use during manufacture of a sensor array. The opening in the IR cutoff filter can be added when the IR filter is formed over the sensor array. 
         [0044]    The IR detection embodiments can also be used when a sensor array already has an IR cutoff filter covering the entire array of pixels. Such an IR detection embodiment relies on the fact that spectral transmittance of reflective IR-cutoff filters is highly angle dependent. Such filters allow a small amount of IR illumination through when illuminated at an angle other than perpendicular to the filter plane. 
         [0045]    One problem solved by the present embodiments in IR sensing in using a visible light sensor is that the level of IR can substantially differ from the level of visible illumination, sometimes by many orders of magnitude. By using OBIR and OIRB pixels the IR is sensed in a high-dynamic-range fashion. The present embodiments require only a few changes to the sensor such as forming a black filter over some pixels, a metal layer over other pixels, and forming “barrier” pixels. 
         [0046]    There are also alternative embodiments for IR sensing. For example, the exposure of IR pixels can be controlled independently from active pixels. In this embodiment, the exposure is varied to avoid overexposure of IR pixels and the OIRB pixels are optional. 
         [0047]    In yet another embodiment, the method for IR sensing is similar to the embodiment of  FIGS. 2-5  but not all of the OIRB pixels are shielded completely. The pixels adjacent to the OIRB pixels have a certain area (e.g., 50%) covered by metal. The next inner row is covered by metal to a greater extent (e.g., 75%) and the next row is covered to an even greater amount (e.g., 100%). 
         [0048]    In still another embodiment, the OIRB pixels are not used. In this embodiment, special high dynamic range pixels are used to sense IR. 
         [0049]      FIG. 6  illustrates a block diagram of one embodiment of an imaging system of the present invention. The system comprises an image sensor  600  as described previously, coupled to a control circuit  601 . This system can represent a camera or some other imaging device. 
         [0050]    In one embodiment, the control circuit  601  is a processor, microprocessor, or other controller circuitry that reads and processes the image from the image sensor device  600 . For example, the imaging system can be a digital camera in which the image sensor device  600  is exposed to an image for recording. The control circuitry  601  executes the above-described embodiments and reads the accumulated charges from the photodiodes of the image sensor device  600 . The control circuitry  601  may then process this data into an image or simply store the data. 
         [0051]    The data from the system can be output to other systems over an I/O circuit  603 . The I/O circuit  603  may be a Universal Serial Bus (USB) or some other type of bus that can connect the imaging system to a computer or other system. 
       Conclusion 
       [0052]    In summary, the embodiments discussed herein provide a way to measure the amount of infrared illumination that reaches an image sensor. Knowing the amount of infrared illumination is useful in automatic white balance algorithms. A large amount of infrared illumination informs the white balance algorithm that the scene is illuminated by an incandescent light source. The white balance algorithm can then shift the color temperature of the received image to adjust for the extra red color cast. A low amount of infrared informs the white balance algorithm that the camera is imaging a scene illuminated by a fluorescent light source, such as office lighting, and the color temperature of the received image must be shifted to remove a greenish cast. A moderate amount of infrared indicates that the camera is used outdoors in daylight conditions. In this condition, the white balance would typically remove a blue color cast from the image, if any. 
         [0053]    Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement that is calculated to achieve the same purpose may be substituted for the specific embodiments shown. Many adaptations of the invention will be apparent to those of ordinary skill in the art. Accordingly, this application is intended to cover any adaptations or variations of the invention. It is manifestly intended that this invention be limited only by the following claims and equivalents thereof.