Imaging devices having arrays of image sensors and lenses with multiple aperture sizes

An array camera may be formed from an array of lenses, an array of corresponding apertures, and an array of corresponding image sensors. The array of apertures may be configured so that some image sensors receive light through apertures of different size than other image sensors. Providing apertures of smaller size increases the F/# of an array camera and increases the depth-of-field in a captured image. The array of image sensors may include a near-infrared image sensor. Providing an image sensor array with a near-infrared image sensor may enhance depth information in captured images or increase night vision capabilities of an array camera. Combining an array of image sensors that includes a near-infrared sensor with an array of apertures having different aperture diameters may allow increased depth-of-field imaging, enhanced extraction of depth information from an image, improved night vision, enhanced image clarity or other improvements.

BACKGROUND

This relates generally to imaging devices, and more particularly, to imaging devices with multiple lenses and image sensors.

Image sensors are commonly used in electronic devices such as cellular telephones, cameras, and computers to capture images. In a typical arrangement, an electronic device is provided with a single image sensor and a single corresponding lens. Some electronic devices use arrays of image sensors and corresponding lenses to gather image data. This type of system, which is sometimes referred to as an array camera, may be used to extend depth of focus, increase output resolution through super-resolution processing, and capture depth information from a scene. Array cameras may also be used to improve image processing and information gathering processes such as gesture control, image segmentation or other image processing operations.

In a conventional array camera, image sensors associated with red, green, and blue color filters are used along with corresponding lenses having equal sized apertures. Array cameras having red, green and blue color image sensors with equal sized apertures have certain depth of focus that is determined by the aperture size. Reducing the aperture size increases the range of depth of focus. However, reducing all aperture sizes may have undesired consequences such as reduced sensitivity and reduced signal to noise ratio.

It would therefore be desirable to be able to provide improved imaging devices with array cameras with multiple aperture sizes.

DETAILED DESCRIPTION

Digital camera modules are widely used in electronic devices such as digital cameras, computers, cellular telephones, or other electronic devices. These electronic devices may include image sensors that gather incoming light to capture an image. The image sensors may include arrays of image pixels. The pixels in the image sensors may include photosensitive elements such as photodiodes that convert the incoming light into digital data. Image sensors may have any number of pixels (e.g., hundreds or thousands or more). A typical image sensor may, for example, have hundreds of thousands or millions of pixels (e.g., megapixels).

FIG. 1is a diagram of an illustrative electronic device that uses an image sensor to capture images. Electronic device10ofFIG. 1may be a portable electronic device such as a camera, a cellular telephone, a video camera, or other imaging device that captures digital image data. Camera module12may be used to convert incoming light into digital image data. Camera module12may include an array of lenses13, a corresponding array of apertures14, and a corresponding array of image sensors16. Lens array13and image sensor array16may be mounted in a common package and may provide image data to processing circuitry18. Processing circuitry18may include one or more integrated circuits (e.g., image processing circuits, microprocessors, storage devices such as random-access memory and non-volatile memory, etc.) and may be implemented using components that are separate from camera module12and/or that form part of camera module12(e.g., circuits that form part of an integrated circuit that includes image sensors16or an integrated circuit within module12that is associated with image sensors16). Image data that has been captured by camera module12may be processed and stored using processing circuitry18. Processed image data may, if desired, be provided to external equipment (e.g., a computer or other device) using wired and/or wireless communications paths coupled to processing circuitry18.

There may be any suitable number of lenses13iin lens array13, any suitable number of apertures14iin aperture array14, and any suitable number of image sensors16iin image sensor array16. Lens array13may, as an example, include N*M individual lenses arranged in an N×M two-dimensional array. The values of N and M may be equal to or greater than one, may be equal to or greater than two, may exceed 10, or may have any other suitable values. The physical size of each lens13iin lens array13may be substantially different from the physical size of any other lens in lens array13or may be substantially the same as the size of other lenses in lens array13. Each image sensor16iand lens13imay have an associated aperture14i. The aperture14i(sometimes called a lens aperture) associated with each image sensor16imay have a different size than that of other apertures in aperture array14or may have the same size as other apertures in aperture array14.

Image sensor array16may contain a corresponding N×M two-dimensional array of individual image sensors. The image sensors may be formed on one or more separate semiconductor substrates. With one suitable arrangement, which is sometimes described herein as an example, the image sensors are formed on a common semiconductor substrate (e.g., a common silicon image sensor integrated circuit die). Each image sensor may be identical or, if desired, some image sensors may be different (e.g., some image sensors may have different pixel sizes, shapes or sensitivity than other image sensors). For example, each image sensor may be a Video Graphics Array (VGA) sensor with a resolution of 480×640 sensor pixels (as an example). Other types of image sensor may also be used for the image sensors if desired. For example, images sensors with greater than VGA resolution or less than VGA resolution may be used, image sensor arrays in which the image sensors are not all identical may be used, etc.

The use of a camera module with an array of lenses and an array of corresponding image sensors (i.e., an array camera) having associated apertures of various sizes, may allow images to be captured with increased depth-of-field. This is because images captured using a smaller aperture have a larger depth-of-field (DOF) than images captured using larger apertures. Depth-of-field may be defined to be the physical distance between the farthest and nearest objects in a real-world scene that appear in-focus to a viewer of an image of the scene. Providing an array camera with one or more lenses having smaller apertures allows the DOF to be increased relative to that of a conventional single-lens configuration. Cameras having a smaller aperture have a correspondingly larger “F-Number” (F/#) relative to other cameras. Larger F/# (or smaller aperture) leads to less light allowed through the aperture to the image sensor. Array cameras having multiple image sensors associated with multiple lenses and differing aperture sizes may combine the benefits of large and small apertures (or small and large F/#'s, respectively). Large DOF images may be combined with smaller DOF images to produce images with enhanced clarity or to obtain depth information from the images. In comparison with conventional devices, color cross-talk may also be reduced, because a single color filter can be used for each sub-array instead of using a conventional Bayer pattern or other multiple-color color filter array pattern. With a single color filter arrangement of this type, there is no opportunity for color information to bleed from one channel to another. As a result, signal-to-noise ratio and color fidelity may be improved (e.g., having an equal number of green, red and blue image pixels may help avoid an imbalance of green pixels as in a Bayer color filer array). A single color filter arrangement may also allow increased resolution as the pixels of a single image sensor are not subdivided into multiple colors (as in the case of a Bayer color filter array). The color filters that are used for the image sensor pixel arrays in the image sensors may, for example, be red filters, blue filters, and green filters. Each filter may form a color filter layer that covers the image sensor pixel array of a respective image sensor in the array. Other filters such as infrared-blocking filters, filters that block visible light while passing infrared light, ultraviolet-light blocking filters, white color filters, dual-band IR cutoff filters (e.g., dual-band NIR image sensors having filters that allow visible light and a range of infrared light emitted by LED lights), etc. may also be used.

In an array with numerous image sensors, some of the image sensors may have red filters, some may have blue color filters, some may have green color filters, some may have patterned color filters (e.g., Bayer pattern filters, etc.), some may have infrared-blocking filters, some may have ultraviolet light blocking filters, some may be visible-light-blocking-and-infrared-passing filters, etc. The image sensor integrated circuit may have combinations of two or more, three or more, or four or more of these filters or may have filters of only one type.

Processing circuitry18(e.g., processing circuitry integrated onto sensor array integrated circuit16and/or processing circuitry on one or more associated integrated circuits) can select which digital image data to use in constructing a final image for the user of device10. For example, circuitry18may be used to blend image data from red, blue, and green sensors to produce full-color images, may be used to select an infrared-passing filter sensor when it is desired to produce infrared images, may be used to produce 3-dimensional (sometimes called stereo) images using data from two or more different sensors that have different vantage points when capturing a scene, may be used to produce increased DOF images using data from two or more image sensors with apertures of differing sizes (e.g., differing diameters in circular aperture configurations), etc. In some modes of operation, all of the sensors on array16may be active (e.g., when capturing high-quality images). In other modes of operation (e.g., a low-power preview mode), only a subset of the image sensors may be used. Other sensors may be inactivated to conserve power (e.g., their positive power supply voltage terminals may be taken to a ground voltage or other suitable power-down voltage and their control circuits may be inactivated or bypassed).

Circuitry in an illustrative pixel of one of the image sensors in sensor array16is shown inFIG. 2. As shown inFIG. 2, pixel190includes a photosensitive element such as photodiode22. A positive power supply voltage (e.g., voltage Vaa) may be supplied at positive power supply terminal30. A ground power supply voltage (e.g., Vss) may be supplied at ground terminal32. Incoming light is collected by photodiode22after passing through a color filter structure. Photodiode22converts the light to electrical charge.

Before an image is acquired, reset control signal RST may be asserted. This turns on reset transistor28and resets charge storage node26(also referred to as floating diffusion FD) to Vaa. The reset control signal RST may then be deasserted to turn off reset transistor28. After the reset process is complete, transfer gate control signal TX may be asserted to turn on transfer transistor (transfer gate)24. When transfer transistor24is turned on, the charge that has been generated by photodiode22in response to incoming light is transferred to charge storage node26.

Charge storage node26may be implemented using a region of doped semiconductor (e.g., a doped silicon region formed in a silicon substrate by ion implantation, impurity diffusion, or other doping techniques). The doped semiconductor region (i.e., the floating diffusion FD) exhibits a capacitance that can be used to store the charge that has been transferred from photodiode22. The signal associated with the stored charge on node26is conveyed to row select transistor36by source-follower transistor34.

When it is desired to read out the value of the stored charge (i.e., the value of the stored charge that is represented by the signal at the source S of transistor34), row select control signal RS can be asserted. When signal RS is asserted, transistor36turns on and a corresponding signal Vout that is representative of the magnitude of the charge on charge storage node26is produced on output path38. In a typical configuration, there are numerous rows and columns of pixels such as pixel140in the image sensor pixel array of a given image sensor. A vertical conductive path such as path40can be associated with each column of pixels.

When signal RS is asserted in a given row, path40can be used to route signal Vout from that row to readout circuitry. If desired, other types of image pixel circuitry may be used to implement the image pixels of sensors16-1, . . .16-N. For example, each image sensor pixel140(see, e.g.,FIG. 1) may be a three-transistor pixel, a pin-photodiode pixel with four transistors, a global shutter pixel, a time-of-flight pixel, etc. The circuitry ofFIG. 2is merely illustrative.

A diagram of a conventional array camera with an array of identical lenses and corresponding image sensors having apertures of equal size is shown inFIG. 3. In the example ofFIG. 3, array camera (camera module120) has a lens array140that is made up of three lenses: lenses140A,140B, and140C. Lenses140A,140B, and140C each focus image light from an objects such as objects,200,205, and210, onto a respective image sensor in image sensor array160. In particular, lens140A may be used to focus image light onto image sensor160A, lens140B may be used to focus image light onto image sensor160B, and lens140C may be used to focus image light onto image sensor160C. Each image sensor is also associated with a color filter. In a typical arrangement, color filter180A is a red color filter, color filter180B is a green color filter, and color filter180C is a blue color filter. With a camera array of the type shown inFIG. 3, images may only be able be focused on objects located within a small range of distances from the camera, such as object210. Images of objects at other distances may be blurred due the limited depth-of-field. Similarly, 3-dimensional depth information may only be obtained from captured images for objects located within a short distance from the camera.

FIG. 4is a perspective view of an illustrative camera module having an array14of apertures (e.g., apertures such as apertures14(1,1), and14(4,4)). The array of apertures may, for example, be a rectangular array having rows and columns of apertures. The apertures may all have equal size or may have different sizes. For example, aperture14(4,4) may have an aperture diameter d44that is the same as or different than aperture diameter d21corresponding to aperture14(2,1). There may be any suitable number of apertures14in the array. In theFIG. 4example, there are four rows and four columns of apertures.

An illustrative sensor array of the type that may be used with the lens array ofFIG. 4is shown inFIG. 5. As shown inFIG. 5, sensor array16may include image sensors such as sensor16(1,1),16(4,1), and16(4,4). The array ofFIG. 5has sixteen image sensors, but, in general, array16may have any suitable number of image sensors (e.g., two or more sensors, four or more sensors, ten or more sensors, 20 or more sensors, etc.).

FIG. 6is a cross-sectional view of a camera module such as camera module12ofFIG. 1. As shown inFIG. 6, camera module12may be configured such that image sensors receive light through lenses and apertures of the same or differing size. In the illustrative embodiment ofFIG. 6, image sensor16A receives light through lens13A and aperture14A. Image sensor16B receives light through lens13B and aperture14B. Image sensor16A and16B may be configured to receive light of different colors (i.e., image sensors16A and16B may be covered with color filters19A and19B respectively). As shown inFIG. 6, apertures14A and14B have aperture diameters dAand dB, respectively. To help increase the depth-of-field of camera module12, aperture diameter dAmay be smaller than aperture diameter dB. In camera module12ofFIG. 6, apertures14A and14B limit the amount of light reaching image sensors16A and16B (i.e., light from a wider range of angles will pass through aperture14B than will pass through aperture14A). In this arrangement, image sensor16A will capture an image with a larger DOF than images captured using image sensor16B even though lenses13A and13B have substantially the same physical size. Images captured using image sensor16B will have better signal to noise ratio (SNR) than images captured by image sensor16A due to the larger amount of light allowed through aperture14B (assuming the same pixel design), as compared with aperture14A. In some cases, image sensors receiving light through a smaller aperture, such as aperture14A may include high sensitivity image pixels to compensate for smaller signal (light) levels. Images captured by image sensors16A and16B may be combined using processing circuitry such as processing circuitry18ofFIG. 1. In other embodiments, aperture diameter dBmay be smaller than aperture diameter dAor may be substantially the same as aperture diameter dA.

FIG. 7is a cross-sectional top view of an illustrative camera module12. The image sensor array16ofFIG. 7may be formed from a single integrated circuit die (as an example). As shown in the illustrative arrangement ofFIG. 7, image sensor array16includes (as an example) two image sensor pixel arrays16A and16B. Each pixel array may have rows and columns of image pixels such as image pixel190ofFIG. 2or other suitable image pixels. Image sensor pixel arrays16A and16B may have any suitable resolution (e.g., 640×480, 1920×1080 etc.).

Image sensor array16may also include support circuitry64(e.g., row select and control driver circuitry). Support circuitry64may be used to issue reset signals, row select signals, etc. for the image sensor pixel arrays. Support circuitry64may likewise be used for reading out image sensor data along associated output lines such as output line40ofFIG. 2).

Using color filters such as color filters19A and19B as inFIG. 6, image pixel arrays16A and16B may be configured to receive light of various colors. For example, image pixel array16A may be a near-infrared (NIR) image sensor (i.e., a pixel array that receives light through a filter that blocks visible light while allowing NIR light to pass, thereby gathering near-infrared image data) while image pixel array16B may be a Bayer image sensor (i.e., a pixel array that receives light through a Bayer color filter). In one suitable arrangement, image sensor16A may be a NIR image sensor and image sensor16B may be a Bayer image sensor. This type of arrangement may improve night (or low-light) imaging capabilities or improve image depth estimation (may be aided by using NIR structure lighting) and object recognition in the image captured by the Bayer image sensor. In another embodiment, image sensor16A may be a clear image sensor (i.e., a non-Bayer image sensor that is free of color filter elements so that its pixel array receives light through clear structures that pass all light or through IR cutoff filters that pass all visible light) and image sensor16B may be a Bayer image sensor to improve low-light imaging capabilities or to improve image resolution.

In other arrangements in which image sensor16B is a Bayer image sensor, image sensor16A may be a dual-band NIR image sensor (i.e., an image sensor having a color filter that allows visible light and a narrow range of infrared light emitted by LED lights), a second Bayer image sensor, or other color image sensor. Other embodiments may include arrangements in which image sensor16A is a clear image sensor and image sensor16B is a clear image sensor, a NIR image sensor or a dual-band NIR image sensor for black-and-white, day and night imaging (e.g., for a surveillance camera). In this type of arrangement, aperture diameter dAof aperture14A may be chosen to provide a corresponding (as an example) F/# of F/2.8 while aperture14B may aperture diameter dBof aperture14B may be chosen to provide a corresponding with an F/# larger than F/2.8.

Image pixel arrays16A and16B may include pixels with single photosensor such as photosensor22ofFIG. 2or may include stacked color pixels (i.e., pixels with several photosensors located at various depths in the silicon circuit die). Stacked color pixels may be used to exploit the absorbing properties of silicon to form, e.g., stacked red/blue, stacked red/green/blue, stacked NIR/blue or other stacked pixels in which pixels are located at differing depths below an image sensor surface to provide differing amounts of color filtering (resulting in differing amounts of light of each color reaching the pixels).

As shown inFIG. 7, image pixel arrays16A and16B may receive light through apertures14A and14B, respectively. Apertures14A and14B may have associated aperture diameters dAand dB, respectively that are different or aperture diameters that are substantially the same.

In one suitable arrangement, image sensor16A may be a NIR image sensor having a corresponding aperture14A with an aperture diameter dA. Aperture diameter dAmay be smaller than aperture diameter dBof aperture14B. Aperture14B may be associated with a Bayer image sensor such as image sensor16B. In an arrangement in which aperture14A is associated with NIR image sensor16A, and aperture diameter dAis smaller than aperture dB, improved night (or low-light) imaging in addition to enhanced depth-of-field and image depth estimation may be achieved. The embodiments described in connection withFIGS. 6 and 7are merely illustrative and other embodiments may be used.

FIG. 8is a cross-sectional top view of another illustrative embodiment of camera module12. In the embodiment ofFIG. 8, camera module12includes image sensor array16which may be formed from a single integrated circuit die (as an example). As shown in the illustrative arrangement ofFIG. 8, image sensor array16may include (as an example) three image sensor pixel arrays16A,16B, and16C. Each pixel array may have rows and columns of image pixels such as image pixel190ofFIG. 2or other suitable image pixels. Image sensor pixel arrays16A,16B, and16C may have any suitable resolution (e.g., 640×480, etc.).

Image sensor array16may also include support circuitry64(e.g., row select and control driver circuitry). Support circuitry64may be used to issue reset signals, row select signals, etc. for the image sensor pixel arrays. Support circuitry64may likewise be used for reading out image sensor data along associated output lines such as output line40ofFIG. 2).

Various embodiments of camera module12may include arrangements in which image sensors16A,16B, and16C may be NIR image sensors, dual-band NIR image sensors, red image sensors, blue image sensors, green image sensors, clear image sensors, Bayer color image sensors, or other color image sensors. Image sensors16A,16B, and16C may include single color pixels and a color filter or may include stacked red/blue (R/B) image pixels or stacked NIR/blue image pixels to form R/B image sensors or NIR/blue image sensors, respectively. Image sensors may be provided with combined single color and stacked image pixels. As an example, image sensor16A may be a stacked R/B image sensor, image sensor16B may be a green image sensor, and image sensor16C may be a NIR image sensor. In another example, image sensor16A may be a red image sensor, image sensor16B may be a green image sensor, and image sensor16C may be a stacked NIR/blue image sensor.

Image sensor16A,16B, and16C may have corresponding apertures14A,14B and14C having aperture diameters dA, dB, and dC, respectively. Aperture diameters dA, dB, and dCmay be substantially the same or may be different. As an example, aperture diameter dA, may be larger than aperture diameter dB, while aperture diameter dBis equal to aperture diameter dC. In another arrangement aperture diameter dA, may be equal to aperture diameter dB, while aperture diameter dBis smaller than aperture diameter dC. Other arrangements are possible in which no aperture diameter dA, dB, or dCis equal to any other.

FIG. 9is a cross-sectional top view of another illustrative embodiment of camera module12. In the embodiment ofFIG. 9, camera module12includes image sensor array16which may be formed from a single integrated circuit die (as an example). As shown in the illustrative arrangement ofFIG. 9, image sensor array16may include (as an example) four image sensor pixel arrays16(1,1),16(2,2),16(1,2), and16(2,1). Each pixel array may have rows and columns of image pixels such as image pixel190ofFIG. 2or other suitable image pixels. Image sensor pixel arrays16(1,1),16(2,2),16(1,2), and16(2,1) may have any suitable resolution (e.g., 640×480, etc.).

Image sensor array16may also include support circuitry64(e.g., row select and control driver circuitry). Support circuitry64may be used to issue reset signals, row select signals, etc. for the image sensor pixel arrays. Support circuitry64may likewise be used for reading out image sensor data along associated output lines such as output line40ofFIG. 2).

In the illustrative embodiment ofFIG. 9, image pixel array16(1,1) is a red image sensor (i.e., an array of image pixels that receive incident light through a filter than allows passage of red light and blocks other colors). Image pixel array16(1,2) may be a blue image sensor, image pixel array16(2,1) may be a green image sensor and image pixel array16(2,2) may be a NIR image sensor. Image sensors16(1,1),16(1,2), and16(2,1) may have associated aperture diameters d11, d12, and d21respectively that are all substantially the same. Providing image sensors16(2,2),16(1,2), and16(2,1) with aperture diameters d11, d12, and d21respectively that are all substantially the same allows combination of the images captured by image sensors16(2,2),16(1,2), and16(2,1) into a single color image using processing circuitry such as processing circuitry18ofFIG. 1. In order to provide a larger DOF and enhanced depth information, NIR image sensor16(2,2) may have an associated aperture diameter d22that is smaller than aperture diameters d11, d12, and d21.

Processing circuitry18may also be used to combine images captures by image sensors16(1,1),16(2,2),16(1,2), and16(2,1) into a color image with enhanced low-light representation (due to inclusion of a NIR image). Images captured by image sensors16may also provide enhanced depth-of-field and may provide enhanced data for depth estimation.

Various embodiments have been described illustrating array cameras that include arrays of image sensors, lenses and apertures having apertures of various size. Array cameras may include image sensors sensitive to varying combinations of colors of light including NIR light. Apertures associated with the image sensors of varying color sensitivity may have aperture diameters of varying size to enhance the DOF in captured images, to improve image signal-to-noise-ratio, to provide enhanced clarity images, to capture improved data for depth estimation of objects in a scene, etc. Processing circuitry may be used to process images captured by different image sensors to form a composite color image, a stereoscopic image, a low (or night) light image or to extract depth information about a scene.

The foregoing is merely illustrative of the principles of this invention which can be practiced in other embodiments.