Imaging systems with visible light sensitive pixels and infrared light sensitive pixels

An image sensor may have an array of image pixels arranged in color filter unit cells that each have at least one red image pixel that generates red image signals, at least one blue image pixel that generate blue image signals, at least one clear image pixels that generate clear image signals, at least one infrared image pixel that generates infrared image signals, and optionally at least one green image pixel that generates green image signals. The image sensor may be coupled to processing circuitry that performs chroma demosaicking operations on the image signals. The processing circuitry may generate an infrared image using the infrared image signals and a luminance value using the clear, red, blue, and infrared image signals. The processing circuitry may perform point filter operations on the image signals based on the generated luminance value to produce corrected visible light image signals having improved image quality.

BACKGROUND

This relates generally to imaging devices, and more particularly, to imaging devices with broadband image pixels and infrared image pixels.

Image sensors are commonly used in electronic devices such as cellular telephones, cameras, and computers to capture images. In a typical arrangement, an electronic device is provided with an array of image pixels arranged in pixel rows and pixel columns. Circuitry is commonly coupled to each pixel column for reading out image signals from the image pixels.

Conventional imaging systems employ a single image sensor in which the visible light spectrum is sampled by red, green, and blue (RGB) image pixels arranged in a Bayer mosaic pattern. The Bayer Mosaic pattern consists of a repeating cell of two-by-two image pixels, with two green pixels diagonally opposite one another, and the other corners being red and blue. However, the Bayer pattern does not readily enable further miniaturization of image sensors via smaller image pixel sizes because of limitations of signal to noise ratio (SNR) in the image signals captured from the image pixels.

In some cases, imaging systems may be provided with infrared imaging capabilities for generating infrared images of a scene. In conventional imaging systems with infrared imaging capabilities, one of the green image pixels in the Bayer Mosaic pattern is replaced with an infrared image pixel. However, forming a Bayer Mosaic pattern having an infrared image sensitive pixel can limit the resolving power and reduce the luminance response and SNR of the image sensor when capturing visible light images.

It would therefore be desirable to be able to provide imaging devices with improved means of capturing and processing image signals.

DETAILED DESCRIPTION

Electronic devices such as digital cameras, computers, cellular telephones, and other electronic devices may include image sensors that gather incoming light to capture an image. The image sensors may include arrays of image pixels. The pixels in the image sensors may include photosensitive elements such as photodiodes that convert the incoming light into image signals. Image sensors may have any number of pixels (e.g., hundreds or thousands or more). A typical image sensor may, for example, have hundreds of thousands or millions of pixels (e.g., megapixels). Image sensors may include control circuitry such as circuitry for operating the image pixels and readout circuitry for reading out image signals corresponding to the electric charge generated by the photosensitive elements. Readout circuitry may include selectable readout circuitry coupled to each column of pixels that can be enabled or disabled to reduce power consumption in the device and improve pixel readout operations.

FIG. 1is a diagram of an illustrative electronic device that uses an image sensor to capture images. Electronic device10ofFIG. 1may be a portable electronic device such as a camera, a cellular telephone, a tablet computer, a webcam, a video camera, or other imaging device that captures digital image data. Camera module12may be used to convert incoming light into digital image data. Camera module12may include one or more lenses14and one or more corresponding image sensors16. During image capture operations, light from a scene may be focused onto image sensor16by lens14. Image sensor16may include circuitry for converting analog pixel data into corresponding digital image data to be provided to storage and processing circuitry18. If desired, camera module12may be provided with an array of lenses14and an array of corresponding image sensors16.

Processing circuitry18may include one or more integrated circuits (e.g., image processing circuits, microprocessors, storage devices such as random-access memory and non-volatile memory, etc.) and may be implemented using components that are separate from camera module12and/or that form part of camera module12(e.g., circuits that form part of an integrated circuit that includes image sensors16or an integrated circuit within module12that is associated with image sensors16). Image data that has been captured by camera module12may be processed and stored using processing circuitry18. Processed image data may, if desired, be provided to external equipment (e.g., a computer or other device) using wired and/or wireless communications paths coupled to processing circuitry18.

As shown inFIG. 2, image sensor16may include a pixel array20containing image sensor pixels22(sometimes referred to herein as image pixels22) and control and processing circuitry24. Array20may contain, for example, hundreds or thousands of rows and columns of image sensor pixels22. Control circuitry24may be coupled to row decoder circuitry26and column decoder circuitry28. Row decoder circuitry26may receive row addresses from control circuitry24and supply corresponding row control signals such as reset, row-select, transfer, and read control signals to pixels22over control paths32. One or more conductive lines such as column lines32may be coupled to each column of pixels22in array20. Column lines32may be used for reading out image signals from pixels22and for supplying bias signals (e.g., bias currents or bias voltages) to pixels22. During pixel readout operations, a pixel row in array20may be selected using row decoder circuitry26and image data associated with image pixels22in that pixel row can be read out along column lines32.

Column decoder circuitry28may include sample-and-hold circuitry, amplifier circuitry, analog-to-digital conversion circuitry, bias circuitry, column memory, latch circuitry for selectively enabling or disabling the column circuitry, or other circuitry that is coupled to one or more columns of pixels in array20for operating pixels22and for reading out image signals from pixels22. Column decoder circuitry28may be used to selectively provide power to column circuitry on a selected subset of column lines32. Readout circuitry such as signal processing circuitry associated with column decoder circuitry28(e.g., sample-and-hold circuitry and analog-to-digital conversion circuitry) may be used to supply digital image data to control and processing circuitry24and/or processor18(FIG. 1) over path25for pixels in chosen pixel columns.

Image sensor pixels such as image pixels22are conventionally provided with a color filter array which allows a single image sensor to sample red, green, and blue (RGB) light using corresponding red, green, and blue image sensor pixels arranged in a Bayer mosaic pattern. The Bayer mosaic pattern consists of a repeating unit cell of two-by-two image pixels, with two green image pixels diagonally opposite one another and adjacent to a red image pixel diagonally opposite to a blue image pixel. However, limitations of signal to noise ratio (SNR) that are associated with the Bayer Mosaic pattern make it difficult to reduce the size of image sensors such as image sensor16. It may therefore be desirable to be able to provide image sensors with an improved means of capturing images.

In one suitable arrangement that is sometimes discussed herein as an example, the green pixels in a Bayer pattern are replaced by broadband image pixels as shown inFIG. 3. As shown inFIG. 3, a unit cell34of image pixels22may be formed from two clear (C) image pixels (sometimes referred to as white image pixels or clear filter image pixels) that are diagonally opposite one another and adjacent to a red (R) image pixel that is diagonally opposite to a blue (B) image pixel. Clear image pixels22in unit cell34may be formed with a visibly transparent color filter that transmits light across the visible light spectrum (e.g., clear pixels22can capture white light). Clear image pixels22may have a natural sensitivity defined by the material that forms the transparent color filter and/or the material that forms the image sensor pixel (e.g., silicon). In another suitable arrangement, clear image pixels22may be formed without any color filter elements. The sensitivity of clear image pixels22may, if desired, be adjusted for better color reproduction and/or noise characteristics through use of light absorbers such as pigments. Clear image pixels22may be sensitive to light across the entire visible light spectrum or may be sensitive to broad portions of the visible light spectrum. Unit cell34may be repeated across image pixel array20to form a mosaic of red, clear, and blue image pixels22. In this way, red image pixels may generate red image signals in response to red light, blue image pixels may generate blue image signals in response to blue light, and clear image pixels may generate clear image signals in response to white light (e.g., any desired broad-spectrum visible light). Clear image pixels22may have a substantial response to any suitable combination of light of at least two of the following colors: red, green, and blue. In this way, clear image pixels22may have a broadband response relative to the non-clear image pixels in each unit cell (e.g., clear pixels22may therefore sometimes be referred to as broadband image pixels22). If desired, broadband pixels22may have clear color filter elements in which a pigment such as a yellow pigment has been added to clear color filter element material (e.g., so that the color filter of broadband pixels22pass red and green light). The example ofFIG. 3is merely illustrative. If desired, unit cell34may have any desired size, may be rectangular, and may include any desired colored pixels.

Image pixels22in pixel array20may have a native sensitivity that extends into the infrared (IR) region of the electromagnetic spectrum (e.g., image pixels22may have a sensitivity to light that has a wavelength of greater than 700 nm). In visible imaging applications, this unwanted infrared sensitivity may be suppressed by forming an infrared cutoff filter in camera module12. For example, an infrared cutoff filter may be interposed between lens14and image sensor16(FIG. 1), may be formed as a coating on one or more surfaces of lenses14, etc. In one suitable arrangement, an infrared notch filter may be interposed between lens14and image sensor16across each image pixel22in array20. In another suitable arrangement, an infrared cutoff filter may be coated only over the pixels in array20that are sensitive to visible light.

In some scenarios, device10may include separate arrays of visible light sensitive image pixels (e.g., red image pixels, green image pixels, blue image pixels, etc.) and infrared light sensitive image pixels. However, forming a separate infrared pixel array on device10may occupy an excessive amount of space within device10. If desired, device10may include one or more infrared light sensitive image pixels22(sometimes referred to herein as infrared-sensitive pixels, infrared pixels, or infrared image pixels22) for capturing infrared images. Device10having infrared pixels22may, for example, be formed as a part of a surveillance imaging system, gaming imaging system, automotive imaging system, or any other desired imaging systems. If desired, device10may include an infrared light emitter that illuminates a scene using infrared light centered about a selected infrared wavelength.

Pixel array20having infrared pixels22may be used to perform both visible light and infrared light imaging operations. Pixel array20having infrared pixels22may generate infrared light images in response to infrared light and may generate visible light images in response to visible light. Infrared sensitive image pixels22in pixel array20may be sensitive to infrared light (e.g., light having a wavelength of greater than or equal to 700 nm). Infrared sensitive image pixels22may be insensitive to visible light (e.g., light having a wavelength of between 400 and 700 nm). For example, infrared pixels22may include a visible light cutoff filter that blocks visible light from being captured by the infrared pixels.

If desired, each unit cell of image pixels22in array20may include one or more infrared image pixels22(e.g., one of the image pixels in unit cell34ofFIG. 3may be replaced by an infrared image pixel). Repeating unit cells in array20having infrared image pixels may each include any desired number of image pixels22. For example, array20may include two-pixel by four-pixel unit cells, two-pixel by two-pixel unit cells, four-pixel by four-pixel unit cells, or unit cells of any other desired size that each include broadband pixels and infrared image pixels.

FIG. 4shows a diagram of an illustrative unit cell in array20that includes an infrared image pixel22. In the example ofFIG. 4, array20includes a repeating unit cell40of image pixels22. Unit cell40may be a two-pixel by four-pixel unit cell of eight image pixels22. As shown inFIG. 4, unit cell40may include four broadband image pixels (C), one red image pixel (R), one green image pixel (G), one blue image pixel (B), and one infrared image pixel (IR) Infrared image pixel22may include, for example, a black color filter element that blocks visible light while transmitting infrared light.

The broadband image pixels (C) in unit cell40may be formed diagonally adjacent to one another in a zig-zag (checker-board) pattern. The first row of unit cell40may include a first broadband image pixel formed adjacent to the green image pixel. A second broadband image pixel may be interposed between the green image pixel and the infrared image pixel. The second row of unit cell40may include the red image pixel formed adjacent to a third broadband image pixel. The blue image pixel may be interposed between the third broadband image pixel and a fourth broadband image pixel. Unit cell40may be repeated throughout array20in a repeating pattern. Array20having repeating unit cell40may thereby include broadband image pixels in each row and each column of the pixel array. Array20may include infrared image pixels22in every four columns. Half of the image pixels in array20having unit cell40may be broadband image pixels, allowing for improved light capture abilities relative to conventional Bayer filter image sensors. Array20having unit cells40may thereby gather image signals having improved SNR, improved luminance channel image fidelity, and improved resolving power relative to Bayer filter image sensors.

The example ofFIG. 4is merely illustrative. If desired, the image pixels22within unit cell40may be arranged in any order. For example, the positions of the red, green, blue, and infrared pixels22in unit cell40may be permuted (e.g., the location of the red image pixel in unit cell40may be swapped with the location of the green image pixel, the blue image pixel, or the infrared image pixel, the location of the green image pixel in unit cell40may be swapped with the location of the red image pixel, blue image pixel, or infrared image pixel, the location of the infrared image pixel in unit cell40may be swapped with the location of the red image pixel, blue image pixel, or green image pixel, etc.).

FIG. 5shows a diagram of another illustrative unit cell having infrared and clear image pixels22that may be formed on array20. In the example ofFIG. 5, pixel array20includes a four-pixel by four-pixel unit cell42of image pixels22. As shown inFIG. 5, unit cell42may include eight broadband image pixels (C), two red image pixels (R), two blue image pixels (B), two infrared image pixels (IR), and two green image pixels (G). The broadband image pixels (C) in unit cell42may be formed diagonally adjacent to one another in a checker-board pattern. The first row of unit cell42may include a first broadband image pixel formed adjacent to a first green image pixel. A second broadband image pixel may be interposed between the first green image pixel and a first infrared image pixel. The second row of unit cell42may include a first red image pixel formed adjacent to a third broadband image pixel. A first blue image pixel may be interposed between the third broadband image pixel and a fourth broadband image pixel. The third row of unit cell42may include a fifth broadband image pixel formed adjacent to a second infrared image pixel. A sixth broadband image pixel may be interposed between the second infrared image pixel and a second green image pixel. The fourth row of unit cell42may include a seventh broadband image pixel interposed between a second blue image pixel and a second red image pixel. The second red image pixel may be interposed between the seventh broadband image pixel and an eighth broadband image pixel.

Unit cell42may be repeated throughout array20in a repeating pattern. In this way, pixel array20may include broadband image pixels in each row and column of the pixel array and may include infrared image pixels in every two columns of the pixel array. The red, green, and blue image pixels in pixel array20having unit cell40may be located in every two columns of pixel array20. Half of the image pixels22in array20having unit cell42may be broadband image pixels, allowing for improved light capture abilities relative to conventional Bayer filter image sensors. Array20having unit cells42may thereby gather image signals having improved SNR, improved luminance channel image fidelity, and improved resolving power relative to Bayer filter image sensors. Pixel array20having unit cells42may have improved chromatic resolving power and alignment relative to pixel arrays having unit cell40ofFIG. 4because red, blue, green, and infrared image pixels22are located in every two columns of pixel arrays having unit cell42, whereas the red, blue, green, and infrared image pixels22are located in every four columns of pixel arrays having unit cell40. However, pixel arrays20having unit cell42may require a more image processing resources than pixel arrays20having unit cell40ofFIG. 4(e.g., image sensors having unit cell40may perform image processing operations faster than image sensors having unit cell42).

The example ofFIG. 5is merely illustrative. If desired, image pixels22within unit cell42may be arranged in any order. For example, the positions of the red, green, blue, and infrared pixels22in unit cell42may be permuted (e.g., the location of the red image pixels in unit cell42may be swapped with the location of the green image pixels, the location of one of the red image pixels in unit cell42may be swapped with the location of one of the blue image pixels, etc.).

FIG. 6shows a diagram of another illustrative unit cell having infrared and broadband image pixels22that may be formed on pixel array20. In the example ofFIG. 6, pixel array20includes a two-pixel by four-pixel unit cell44of image pixels22. As shown inFIG. 6, unit cell44may include four broadband image pixels (C), one red image pixel (R), one blue image pixel (B), and two infrared image pixels (IR). Image pixels22in unit cell44may be arranged similar to unit cell40ofFIG. 4but with the green image pixel of unit cell40replaced with a second infrared image pixel. In other words, the broadband image pixels22in unit cell42may be formed diagonally adjacent to one another in a zig-zag (checker-board) pattern. The first row of unit cell44may include a first broadband image pixel formed adjacent to a first infrared image pixel. A second broadband image pixel may be interposed between the first infrared image pixel and a second infrared image pixel. The second row of unit cell40may include the red image pixel formed adjacent to a third broadband image pixel. The blue image pixel may be interposed between the third broadband image pixel and a fourth broadband image pixel.

Unit cell44may be repeated throughout array20in a repeating pattern. In this way, pixel array20may include broadband image pixels in each row and column of the pixel array and may include infrared image pixels in every second column (e.g., half of the image pixels22in array20may be broadband image pixels). Pixel array20having unit cells44may have improved infrared resolving power and aliasing relative to pixel arrays having unit cell40ofFIG. 4because the number density of infrared image pixels in unit cell44is greater than the number density of infrared image pixels in unit cell40(e.g., because infrared image pixels22are located in every two columns of arrays having unit cell44whereas infrared image pixels22are located in every four columns of arrays having unit cell40). Image sensor16having unit cell44may compute a green image signal by performing a weighted subtraction of red image signals and blue image signals from the broadband image signals generated by array20. Performing the weighted subtraction to generate the green image signals may amplify noise generated by arrays having unit cell44relative to arrays having unit cell40ofFIG. 4.

The example ofFIG. 6is merely illustrative. If desired, image pixels22within unit cell44may be arranged in any order. For example, the positions of the red, blue, and infrared pixels22in unit cell44may be permuted (e.g., the location of the red image pixels in unit cell44may be swapped with the location of the blue image pixel, etc.) Arrays formed with unit cell44may exhibit improved imaging performance when formed with an infrared cut-off filter that is individually coated over visible light sensitive image pixels22(e.g., without being coated over infrared image pixels22) relative to scenarios when an infrared notch filter is provided for all image pixels22across array20. In scenarios where an infrared cut-off filter is individually coated over visible light sensitive image pixels22, unit cell44may provide improved color reproduction for image sensor16relative to unit cells40and42. However, when the visible light image pixels are individually covered with infrared cutoff filters, the image pixels may not be used as infrared-sensitive pixels when infrared light dominates an imaged scene, such as indoor scenes with active infrared illumination (e.g., for video gaming applications). Thus, the higher number density of infrared pixels as shown inFIG. 6can be advantageous in scenarios where pixel-wise infrared cutoff filters are formed.

FIG. 7shows a diagram of another illustrative unit cell having infrared and broadband image pixels22that may be formed on pixel array20. In the example ofFIG. 7, pixel array20includes a two-pixel by two-pixel unit cell46of image pixels22. As shown inFIG. 7, unit cell46may include one broadband image pixel (C), one infrared image pixel (IR), one red image pixel (R), and one blue image pixel (B). The broadband image pixel may be formed diagonally opposite to blue image pixel. The red image pixel may be formed adjacent to the broadband image pixel and blue image pixel and diagonally opposite to the infrared image pixel. Unit cell46may be repeated throughout array20in a repeating pattern. In this way, broadband image pixels, infrared image pixels, blue image pixels, and red image pixels22may be formed in every second column of pixel array20.

Pixel array20having unit cell46may perform a weighted subtraction to generate green image signals. Performing the weighted subtraction to generate the green image signals may amplify noise generated by arrays having unit cell46relative to arrays having unit cell40ofFIG. 4. Pixel array20having unit cells46may have improved chromatic resolving power and aliasing relative to pixel arrays having unit cell40ofFIG. 4because the number density of infrared, red, and blue image pixels in unit cell46is greater than the number density of infrared, red, and blue image pixels in unit cell40. Arrays20having unit cell46may have reduced luminance resolving power and signal-to-noise-ratio relative to arrays having unit cells40,42, or44(e.g., because arrays having unit cell46include broadband image pixels22in every other column and row whereas arrays having unit cells40,42, and44have broadband image pixels22in every column and row). For example, arrays20having unit cell46may have approximately 1 dB of signal-to-noise ratio improvement over pixel arrays without broadband image pixels, whereas arrays20having unit cells40,42, or44may have approximately 3 dB of signal-to-noise ratio improvement over Bayer filter pixel arrays. However, pixel arrays20having unit cell46may require fewer image processing resources relative to pixel arrays having unit cells40,42, and44, because unit cell46has fewer total pixels than unit cells40,42, and44(e.g., image sensor16having unit cells46may require less time and power for image processing than image sensors having unit cells40,42, or44).

The example ofFIG. 7is merely illustrative. If desired, image pixels22within unit cell46may be arranged in any order. For example, the positions of the red, blue, broadband, and infrared pixels22in unit cell46may be permuted (e.g., the location of the red image pixel in unit cell46may be swapped with the location of the blue image pixel, etc.). If desired, the red, green, and/or blue pixels22ofFIGS. 3-7may be replaced by image pixels of any desired color (e.g., yellow, magenta, cyan, or any other desired color that has a narrower spectral response than broadband image pixels22).

Broadband image pixels22in array20can help increase the signal-to-noise ratio (SNR) of image signals captured by image pixels22by gathering additional light in comparison with image pixels having a narrower color filter (e.g., a filter that transmits light over a subset of the visible light spectrum), such as green image pixels. Broadband image pixels22may particularly improve SNR in low light conditions in which the SNR can sometimes limit the image quality of images. Image signals gathered from image pixel array20having broadband image pixels (e.g., as shown inFIGS. 3-7) may be converted to red, green, and blue image signals to be compatible with circuitry and software that is used to drive most image displays (e.g., display screens, monitors, etc.). This conversion generally involves the modification of captured image signals using a color correction matrix (CCM). If care is not taken, color correction operations can undesirably amplify noise.

In one suitable arrangement, noise generated by the CCM may be reduced by implementing strong de-noising (e.g., chroma de-noising) prior to applying the CCM to gathered image signals. Chroma de-noising may be performed by processing circuitry18(FIG. 1) by applying a chroma filter to image signals gathered by image pixels22. The chroma filter may serve to increase noise correlation between image signals from different colored image pixels (e.g., red, white, and blue image signals). Increasing noise correlation between image signals from different colored image pixels may reduce noise amplification by the CCM, leading to improved final image quality. In another arrangement, noise amplified by the CCM may be compensated for by applying a so-called “point filter” to the captured image signals. The point filter may use high fidelity broadband image signals to enhance the quality of red, green, and blue image signals produced using the CCM. If desired, image sensor16may implement both chroma de-noising and the point filter to reduce noise amplification by the CCM to yield improved luminance performance in the final image.

FIG. 8shows a flow chart of illustrative steps that may be performed by processing circuitry such as processing circuitry18ofFIG. 1to process image signals gathered by a filtered pixel array such as pixel array20(e.g., a pixel array having broadband and infrared image pixels). The steps ofFIG. 8may, for example, be performed by processing circuitry18to generate visible light and/or infrared light images while reducing noise in image signals captured using unit cells having broadband image pixels and infrared image pixels such as those shown inFIGS. 3-7.

At step100, image sensor16may capture image signals from a scene. The image signals captured by image sensor16may include clear (broadband) image signals generated in response to light gathered with broadband image pixels22and may include infrared image signals gathered in response to infrared light gathered by infrared image pixels22. If desired, the image signals may also include one or more of red image signals, blue image signals, or green image signals (e.g., depending on which pixel unit cell is used). For example, if array20is formed with unit cells40ofFIG. 4or unit cells42ofFIG. 5, then the image signals may include red, clear, infrared, and blue image signals, and if array20is formed with unit cells44ofFIG. 6or unit cells46ofFIG. 7, then the image signals may additionally include green image signals (e.g., green image signals generated in response to light gathered with green image pixels22).

In the example ofFIG. 8, image sensor16may capture red (R′), broadband (C′), blue (B′), infrared (IR′), and optionally green (G′) image signals. The red image signals may have a first spectral response value (a signal power level as a function of the wavelength of light received by red image sensor pixels), the blue image signals may have a second spectral response value, and the broadband image signals may have a fourth spectral response value that is, for example, greater than a sum of the first and second spectral response values (e.g., the broadband image signals may have a broadband sensitivity for an equal energy radiator over the visible light spectrum with standard CIE illuminant E). The image signals may have image values corresponding to light captured by each image pixel22(e.g., red image signals may include a red image value, blue image signals may include a blue image value, etc.). The captured image signals may be conveyed to processing circuitry18for image processing.

At step102, processing circuitry18may perform white balance operations on the captured image signals. In the example ofFIG. 6, processing circuitry18may produce a white-balanced red image signal (R), white-balanced broadband image signal (C), white-balanced blue image signal (B), white-balanced infrared image signal (IR), and optionally a white-balanced green image signal (G) using the captured image signals.

At step104, processing circuitry18may demosaic and apply a chroma filter to the white-balanced image signals to extract demosaicked red, broadband, infrared, blue, and optionally green image data from the white-balanced image signals. The chroma filter may be applied to chroma de-noise the white-balanced image signals. Processing circuitry18may, for example, demosaic the image signals and apply the chroma filter simultaneously, sequentially, or in an interspersed manner. This process of applying a chroma filter and demosaicking the image signals may sometimes be referred to herein as “chroma demosaicking” The chroma filter may increase noise correlation between image signals of each color (e.g., noise fluctuations in the red, broadband, infrared, and blue channels may increase or decrease together in a correlated manner). For example, processing circuitry18may increase the correlated noise between the red, broadband, blue, infrared, and green image signals to as much as 70% or more of all noise associated with the red, broadband, blue, infrared, and green image signals. By increasing noise correlation, processing circuitry18may reduce the amount of noise amplification generated when a CCM is applied to the image signals. Chroma demosaicking the image signals may allow missing color image signals (e.g., image signals of colors not generated by the image pixels) to be determined from available color image signals (e.g., by demosaicking missing pixel values for each of the pixels in array20).

Image pixels22that are sensitive to light in the visible light spectrum (e.g., red pixels, green pixels, blue pixels, broadband pixels, etc.) may sometimes be referred to herein as visible light sensitive pixels22or visible light pixels22. In scenarios where an infrared notch filter is formed across each of the image pixels22in array20, a portion of the image signals captured by the visible light sensitive image pixels22(e.g., the red, broadband, blue, and green image pixels22in array20) may be generated in response to infrared light. Infrared light captured by visible light-sensitive image pixels22can cause visible light-sensitive image pixels22to generate erroneous pixel values.

At optional step106, processing circuitry18may perform infrared image signal contamination subtraction operations on the chroma demosaicked image data. Processing circuitry18may, for example, perform optional step106in scenarios where an array-wide infrared notch filter is formed for all image pixels on array20. Processing circuitry18may determine an estimated value of the portion of the image signal of each color that was generated in response to infrared light. Processing circuitry18may subtract the estimated value from the image signals to remove the infrared light contribution to the image signals. In the example ofFIG. 8, processing circuitry18may generate infrared-subtracted image data R-IRR, B-IRB, C-IRC, and G-IRGby subtracting the estimated value from the red, blue, broadband, and green image signals, respectively.

If desired, processing circuitry18may estimate the infrared light contribution (portion) of the visible light image signals using an infrared subtraction matrix (e.g., processing circuitry may determine estimation values of infrared portions of the visible light image signals using an infrared subtraction matrix). The estimated values may be subtracted from the pixel values generated by each image pixel22of the corresponding color. In the examples ofFIGS. 6 and 7in which no green image pixels22are formed on pixel array20, undesirable infrared portions of the red, broadband, and blue image signals may be estimated using a three-row by four-column infrared subtraction matrix. Estimated infrared light contributions for the red, blue, and broadband image signals may, for example, be determined using the following equation:

[IRRIRCIRB]=[kRRkRCkRBkRIkCRkCCkCBkCIkBRkBCkBBkBI]⁡[RCBIR](1)
where R, C, B, and IR are respective red, broadband, blue, and infrared pixel values, where IRRis the estimated value of the infrared portion of the red pixel values (e.g., IRRis an estimate of the portion of the red pixel values that was generated by red image pixels in response to infrared light), where IRCis an the estimated value of the infrared portion of the broadband pixel values, where IRBis the estimated value of the infrared portion of the blue pixel values, and where kRR, kRC, . . . , kBIare constant values for estimating the infrared contamination of the red, broadband, and blue image signals.

The estimated values IRR, IRC, and IRBmay be subtracted from the visible light pixel values to generate infrared-subtracted pixel values R-IRR, B-IRB, C-IRC, and G-IRG. In the examples ofFIGS. 6 and 7in which no green image pixels22are formed on pixel array20, infrared-subtracted pixel values (infrared-subtracted image data) R-IRR, B-IRB, and C-IRCmay, for example, be determined using the following equation:

Infrared subtracted pixel values R-IRR, B-IRB, and C-IRCmay be free from undesirable infrared light contamination. Subtracting infrared contributions from the visible light image signals can reduce the signal-to-noise ratio of the captured image data. If desired, processing circuitry18may mitigate noise clipping of the infrared subtracted image data by performing dark current subtraction after subtracting the estimated infrared values from the visible light image signals. If desired, processing circuitry18may perform additional noise filtering (e.g., chroma noise filtering) to mitigate noise amplification effects caused by performing infrared signal subtraction.

This example is merely illustrative. If desired, processing circuitry18may perform infrared subtraction operations on any visible light image signals (e.g., based on the color filter arrangement of array20). In the examples ofFIGS. 4 and 5in which green image pixels22are formed on array20, processing circuitry18may generate estimated infrared values IRR, IRC, IRB, and IRGusing a four-row by five-column infrared subtraction matrix. In scenarios where individual pixels22that are sensitive to visible light are coated with an infrared cutoff filter (e.g., instead of forming a filter coating across all of the pixels on array20), visible light sensitive pixels22may generate image signals that are free from infrared light contributions (e.g., because the associated cutoff filters adequately block infrared light from being captured by pixels22), and performing the infrared subtraction operations of step106may be omitted.

At optional step108, processing circuitry18may generate luminance channel image data L (e.g., one or more luminance values sometimes referred to herein as original luminance values) by performing a linear combination (e.g., a weighted sum) of the infrared, red, broadband, blue, and green pixel values associated with each pixel22in array20. By performing a weighted sum that includes infrared pixel values associated with each image pixel22, processing circuitry22may reduce noise in a pixel value luminance channel relative to imaging systems that do not include infrared image pixels22. Luminance channel image data L may be subsequently used for performing additional image processing operations such as applying a point filter on the image data.

At step110, processing circuitry18may apply a color correction matrix (CCM) to the infrared-subtracted red image data, infrared-subtracted blue image data, infrared-subtracted clear (broadband) image data, and may optionally apply a CCM to the infrared-subtracted green image data (e.g., in scenarios where array20includes green pixels22). In the examples ofFIGS. 6 and 7in which array20is formed without green image pixels, the CCM may extract green image data from the infrared-subtracted broadband image data to generate red, green, and blue image data. For example, the CCM may convert the image data into standard red, standard green, and standard blue image data (sometimes referred to collectively as linear sRGB image data or simply sRGB image data). In the examples ofFIGS. 4 and 5in which array20includes green image pixels22, the CCM may convert the infrared-subtracted red, blue, green, and broadband image data into standard red, standard green, and standard blue image data. If desired, gamma correction processes may be performed on the linear sRGB image data. After gamma correction, the sRGB image data may be used for display using an image display device. In some cases, it may be desirable to provide additional noise reduction (e.g., by applying a point filter to the sRGB image data) to further mitigate the noise amplification generated by applying the CCM to the red, broadband, blue, and green image data. Processing circuitry18may preserve the broadband image data and the luminance channel image data (e.g., luminance channel image data L as generated at step108) for further processing of the sRGB image data during step112.

At step112, processing circuitry18may apply a point filter to the image data (e.g., to the sRGB image data produced after applying the CCM to the infrared-subtracted image data). The point filter may operate on the sRGB image data to generate corrected sRGB data. The point filter may serve to further reduce noise amplification caused by applying the CCM to the infrared-subtracted image data. When displayed using a display system, the corrected sRGB data thereby provide improved image quality (e.g., better luminance performance) when compared to the sRGB data prior to applying the point filter.

FIG. 9shows a flow chart of illustrative steps that may be performed by processing circuitry18to demosaic and filter image signals received from image pixel array20. The steps ofFIG. 9may, for example, be performed by processing circuitry18to perform chroma demosaicking on red, broadband, and blue image signals gathered by image pixels22to generate sufficient noise correlation in red, broadband, and blue image data. The steps ofFIG. 9may, for example, be performed as part of step104ofFIG. 8.

At step120, processing circuitry18may demosaic the infrared image signal to produce infrared image data (e.g., an infrared image (pixel) value for each image pixel). For example, processing circuitry18may interpolate infrared pixel values for each non-infrared image pixel22in array20using the infrared pixel values captured by infrared image pixels22. In this way, processing circuitry18may fill in missing infrared pixel values for each visible light-sensitive pixel in array20. The demosaicked infrared image data may be used to perform infrared subtraction operations on the red, blue, broadband, and green image data (e.g., while processing step106ofFIG. 8). In another suitable arrangement, infrared image values may be produced for a combination of available image pixels22. If desired, processing circuitry18may pass the demosaicked infrared image data to other processing circuitry for additional image processing (e.g., processing circuitry18may generate an infrared image using the demosaicked infrared image data).

In scenarios where an infrared filter coating is provided for all image pixels22across array20(e.g., when all image pixels in array20are sensitive to infrared light), processing circuitry18may perform interpolation on infrared image signals based on cross-channel noise correlations. For example, in low light-level situations in which most of the image light from a scene is infrared (e.g., if there is active infrared illumination of a scene by an infrared emitter on device10while at low ambient light levels), each of the different color image signals (e.g., each of the image signal color channels) may be highly correlated. Processing circuitry18may characterize the amount of correlation between each image signal channel by computing scene statistics (e.g., average pixel ratios). If desired, processing circuitry18may use identified correlations between the image signal channels to determine an interpolation direction for infrared image pixels22and/or to determine infrared pixel values (e.g., by computing Laplacian gradients between the image signal channels or by assuming constant local hue and using color ratios from nearby pixel locations on array20).

At step122, processing circuitry18may demosaic the broadband image signal to produce broadband image data (e.g., a broadband image value for each image pixel). In another suitable arrangement, broadband image values may be produced for a combination of available image pixels22. The broadband image values may be used to compute difference values using the red and blue image signals to increase noise correlation between the red, broadband, and blue image signals.

At step124, processing circuitry18may generate red difference values by subtracting the broadband image values from the red image values for each pixel. Processing circuitry18may generate blue difference values by subtracting the broadband image values from the blue image values. The red difference values may, for example, be computed for each red image pixel and the blue difference values may be computed for each blue image pixel in array20.

At step126, processing circuitry18may filter the red difference values and the blue difference values using a chroma filter. The chroma filter may be applied to the red and blue difference values by, for example, performing a weighted average of difference values computed over a kernel of image pixels22(e.g., a weighted average of a group of difference values that were computed by performing step122). The kernel of image pixels may be defined as a subset of the image pixels in image pixel array20over which the chroma filtering is being performed (e.g., the kernel may include some or all of the image pixels in image pixel array20). For example, when a five-pixel by five-pixel kernel is used, a weighted average of difference values is calculated for a five-pixel by five-pixel subset of image pixels22in image pixel array20when performing chroma filtering (e.g., a weighted sum of difference values may be computed for a given image pixel22using difference values at25surrounding image pixels in image pixel array20). In general, a kernel of any desired size may be used.

At step128, the broadband image values may be added to the chroma filtered red difference values and the chroma filtered blue difference values to generate chroma filtered red image values and chroma filtered blue image values, respectively.

At step130, processing circuitry18may demosaic the chroma filtered red image values and the chroma filtered blue image values to produce chroma-demosaicked red image data and chroma-demosaicked blue image data (e.g., red and blue image data that has been chroma demosaicked) with increased correlated noise. The demosaicked broadband image data and the chroma demosaicked red and blue image data may then be operated on to generate infrared-subtracted broadband, red, and blue image data as described above in connection with step106ofFIG. 8.

The example ofFIG. 9is merely illustrative. If desired, processing circuitry18may demosaic the chroma filtered red and blue image values prior to generating the red and blue difference values (e.g., processing circuitry18may perform step118prior to step112). In scenarios where array20includes green image pixels (e.g., as shown inFIGS. 4 and 5), processing circuitry18may generate chroma demosaicked green image data.

If chroma filtering of the difference values is performed over a sufficiently large kernel of image pixels22, minimal noise from the red and blue image signals may remain in the red and blue difference values after chroma filtering (e.g., after performing step114). For example, if the kernel has a size of 15-pixels by 15-pixels or greater, chroma filtering may reduce noise in the red and blue chroma filtered difference values to negligible levels. If desired, the kernel of image pixels22may include image pixels located in multiple image pixel arrays20, image pixels located in multiple image sensors16, and/or image pixels used during multiple time frames (e.g., to allow for temporal denoising). When the broadband image values are added to the chroma filtered difference values, noise in the broadband image values may dominate over noise in the difference values. In this way, noise in the red and blue image data produced at step126may be substantially equal to noise in the broadband image data. Noise in the red and blue image data may thereby be highly correlated, resulting in reduced noise amplification by the CCM.

The CCM may operate on the infrared-subtracted red, broadband, and blue image data to produce linear sRGB data at step110(FIG. 6). In examples where array20does not include green image pixels22, the CCM may extract information from the broadband image data to generate the standard green data. The broadband image data (e.g., the demosaicked broadband image data produced at step120) may be preserved after operating on the image data with the CCM. The sRGB image data may be represented in other three-dimensional spaces such as a luminance-chroma-hue (LCH) space. In an LCH space, the luminance channel may be related to the brightness of an image captured by image sensor16, the chroma channel may be related to the color saturation of an image, and the hue channel may be related to the specific color of the image (e.g., red, purple, yellow, green, etc.). The perception of noise and sharpness in a displayed image may be affected by noise and signal variations in the luminance channel. The SNR in the image data may be improved by transforming the sRGB data to LHC data, replacing a luminance value in the luminance channel with a broadband image value (which correlates well with overall image brightness due to the broad spectrum of the white image signal), and transforming the LHC data back to sRGB data. In this way, noise amplification caused by the CCM may be suppressed in the luminance channel, where noise is particularly noticeable to a viewer when viewing a displayed image.

As described above in connection with step112ofFIG. 8, a point filter may be applied to the linear sRGB data to produce corrected sRGB data using the broadband image data. The point filter may operate on a single image pixel22without information from adjacent image pixels22, whereas chroma demosaicking may require image signals (e.g., difference values) from multiple image pixels (e.g., a kernel of image pixels) when being applied to image signals at a single image pixel22. For example, the point filter may operate on a standard red value, standard green value, and standard blue value for each image pixel. To perform point filter operations on the sRGB data, processing circuitry18may use the red image data, broadband image data, and blue image data (e.g., the image data prior to applying the CCM) to compute an original (raw) luminance signal. The original luminance signal may be a linear combination (e.g., a weighted sum) of the broadband image data, red image data, and blue image data. In scenarios where the luminance signal is generated using infrared image signals (e.g., in scenarios in which processing circuitry18performs optional step108ofFIG. 8), the original luminance signal may be a linear combination of the broadband, red, blue, and infrared image data (e.g., the original luminance signal may include the luminance values determined while processing step108ofFIG. 8).

Processing circuitry18may compute an implied luminance signal that is a linear combination of the standard red, standard green, and standard blue image data (e.g., after applying the CCM to the image data). If desired, weights in the linear combination used to compute the implied luminance signal may be substantially similar to the weights used to compute the original luminance signal. The weights may be adjusted to modify the “strength” of the point filter (e.g., the degree to which the point filter transforms or corrects the sRGB data).

Processing circuitry18may generate a scaling value (e.g., a scaling factor to be applied to color corrected image values) by, in a simplest case, dividing the original luminance signal by the implied luminance signal. If desired, the scaling factor may include a numerator and denominator. The numerator and/or the denominator of the scaling value may include a weighted sum of the original luminance signal and the implied luminance signal. The scaling value may include adjustable weighting parameters that can be varied to adjust the strength of the point filter (e.g., the weighting parameters may be continuously varied to adjust the strength of the point filter from zero to a full strength). To apply the point filter to the sRGB data (e.g., to the standard red, green, and blue image data), processing circuitry18may multiply the sRGB data by the scaling value to produce the corrected sRGB data. For example, processing circuitry18may multiply the standard red image data by the scaling value, the standard green image data by the scaling value, etc. If desired, the corrected sRGB data may have hue and chroma channels that are approximately preserved from before applying the point filter (e.g., upon conversion of the corrected sRGB data to LCH space). The corrected sRGB data may have improved noise and/or sharpness due to inherited fidelity of the broadband image signals.

FIG. 10shows a flow chart of illustrative steps that may be performed by processing circuitry18to apply a point filter to sRGB data after applying the CCM to the red, broadband, and blue image data (as an example). Processing circuitry18may, for example, apply the point filter to sRGB data for each image pixel22in image pixel array20. The steps ofFIG. 10may, for example, be performed as part of step112ofFIG. 8.

At step140, processing circuitry18may generate an implied luminance value (e.g., a luminance value in LCH space) for a given image pixel22by combining the standard red, green, blue image data (e.g., after applying a CCM). The implied luminance value may, for example, be computed as a linear combination of the standard red, green, and blue image data.

At step142, processing circuitry18may generate a scaling value by dividing the original luminance value (e.g., the original luminance value as generated at step108ofFIG. 8) by a linear combination of the implied luminance value and the original luminance value. The scaling value may include adjustable weighting parameters that can be varied to adjust the strength of the point filter (e.g., the weighting parameters may be varied continuously to adjust the strength of the point filter from zero to a full strength). If desired, processing circuitry18may select the weighting parameters based on the illuminant of the imaged scene. The scaling value may, for example, be an operator that operates on the sRGB data.

At step144, processing circuitry18may multiply the sRGB data by the scaling value to produce corrected sRGB data (e.g., corrected standard red, green, and blue image data). For example, processing circuitry18may multiply the standard red image data by the scaling value, the standard green image data by the scaling value, etc. The corrected sRGB data may, if desired be provided to an image display. The corrected sRGB data may have improved noise and/or sharpness when compared with the sRGB data prior to applying the point filter.

The examples ofFIGS. 1-10are merely illustrative. If desired, any other suitable three-dimensional spaces may be used for performing the point filter operation. Any desired color filters may be used in conjunction with the broadband and infrared color filters shown inFIGS. 4-7for obtaining color image signals. Unit cells40ofFIG. 4 and 46ofFIG. 6may, if desired, be arranged in a four-pixel by two-pixel pattern (e.g., so that each unit cell includes pixels from two columns and four rows of the array).

FIG. 11shows in simplified form a typical processor system300, such as a digital camera, which includes an imaging device2000(e.g., an imaging device2000such as device10ofFIGS. 1-10employing broadband and infrared color filters and the techniques for operations described above). The processor system300is exemplary of a system having digital circuits that could include imaging device2000. Without being limiting, such a system could include a computer system, still or video camera system, scanner, machine vision, vehicle navigation, video phone, surveillance system, auto focus system, star tracker system, motion detection system, image stabilization system, and other systems employing an imaging device.

The processor system300generally includes a lens396for focusing an image on pixel array20of device2000when a shutter release button397is pressed, central processing unit (CPU)395, such as a microprocessor which controls camera and one or more image flow functions, which communicates with one or more input/output (I/O) devices391over a bus393. Imaging device2000also communicates with the CPU395over bus393. The system300also includes random access memory (RAM)392and can include removable memory394, such as flash memory, which also communicates with CPU395over the bus393. Imaging device2000may be combined with the CPU, with or without memory storage on a single integrated circuit or on a different chip. Although bus393is illustrated as a single bus, it may be one or more busses or bridges or other communication paths used to interconnect the system components.

Various embodiments have been described illustrating image sensors having broadband image pixels and infrared-sensitive image pixels and image processing techniques (e.g., chroma demosaicking, applying a point filter, etc.) for reducing noise in the image signals.

An image sensor may have an array of image sensor pixels and processing circuitry. The array of image sensor pixels may include a first group of image sensor pixels that generate first image signals in response to light of a first color (e.g., red image sensor pixels that generate red image signals in response to red light), a second group of image sensor pixels that generate second image signals in response to light of a second color (e.g., blue image sensor pixels that generate blue image signals in response to blue light), a third group of image sensor pixels that generate third image signals in response to light of at least two of first color, the second color, and a third color that is different from the first and second colors (e.g., clear image sensor pixels may have a significant response to at least two of red light, green light, and blue light, and may generate the third image signals in response to light of at least the first and second colors, light of the second and third colors, light of the first, second, and third colors, light of the first and third colors, etc.). The array of image sensor pixels may include a group of infrared image sensor pixels that generate infrared image signals in response to infrared light. As an example, clear image pixels in the third group of image sensor pixels may generate clear image signals in response to red and green light, in response to red and blue light, in response to green and blue light, etc. The first image signals may, for example, have a first spectral response level (e.g., a signal power level as a function of the wavelength of light received by the first group of image sensor pixels), the second image signals may have a second spectral response level (e.g., a signal power level as a function of the wavelength of light received by the second group of image sensor pixels), and the third image signals may have a third spectral response level (e.g., a signal power level as a function of the wavelength of light received by the third group of image sensor pixels). The third image signals may have a spectral response level that is greater than the first and second spectral response levels. In other words, the third image signals may be captured in response to a broader range of light wavelengths than the first and second image signals.

If desired, the array of image sensor pixels may be arranged in a plurality of repeating unit cells of image sensor pixels. Each unit cell in the plurality of unit cells may include a first image sensor pixel from the first group of image sensor pixels (e.g., each unit cell may include a red image sensor pixel), a second image sensor pixel from the second group of image sensor pixels (e.g., each unit cell may include a blue image sensor pixel), a third image sensor pixel from the third group of image sensor pixels (e.g., each unit cell may include a clear image sensor pixel), and an infrared image sensor pixel from the group of infrared image sensor pixels. If desired, the array of image sensor pixels may include an additional group of image sensor pixels that generate image signals in response to a third color of light (e.g., green image sensor pixels). Each unit cell may, if desired, include one or more green image sensor pixels, additional infrared image sensor pixels, additional clear image sensor pixels, etc.

In one suitable arrangement, each unit cell in the array may include eight image sensor pixels arranged in two rows or two columns (e.g., as shown inFIGS. 4 and 6). In another suitable arrangement, each unit cell in the array may include sixteen image sensor pixels arranged in four rows or two columns (e.g., as shown inFIG. 5). Each row or column in the unit cell may include a respective pair of clear (broadband) image pixels (e.g., image sensor pixels from the third group of image sensor pixels). In another suitable arrangement, each unit cell in the array may include four image sensor pixels arranged in two rows or two columns (e.g., as shown inFIG. 7).

If desired, the processing circuitry may perform chroma demosaicking operations on the first, second, and third image signals that increase noise correlations associated with the first, second, and third image signals. The processing circuitry may perform infrared light contamination subtraction operations on the first, second, and third image signals be subtracting portions of the first, second, and third image signals that were generated in response to infrared light. If desired, the processing circuitry may generate an original luminance value by combining the first, second, third, and infrared image signals (e.g., by performing a weighted sum of the first, second, third, and infrared image signals). The processing circuitry may demosaic the infrared image signals using the first, second, and third image signals.

The processing circuitry may apply a color correction matrix to the first, second, and third image signals to generate standard red, standard green, and standard blue image data. If desired, the processing circuitry may perform point filter operations on the standard red, standard green, and standard blue image data. For example, the processing circuitry may generate an implied luminance value by combining the standard red, standard green, and standard blue image data, may generate a scaling value by dividing the original luminance value by a linear combination of the original and implied luminance values (e.g., in a simplest case, the processing circuitry may divide the original luminance value by the implied luminance value to generate the scaling value). The processing circuitry may generate corrected standard red, green, and blue image data by multiplying the standard red, standard green, and standard blue image data by the scaling value.