Patent Description:
In optical systems, imaging lenses are utilized to collimate light, focus light, and the like. Despite the progress made in the development of optical systems, there is a need in the art for improved imaging lenses.

The present invention relates to the imaging systems and methods as claimed.

Numerous benefits are achieved by way of the present invention over conventional techniques. For example, embodiments of the present invention provide an imaging lens that may be characterized by a lower f-number for NIR light and a higher f-number for visible light by utilizing a wavelength-selective filter at its aperture stop. Moreover, embodiments of the present invention provide an image sensor that may be operated at a lower resolution mode for NIR light using pixel binning and at a higher resolution mode for visible light using native pixel resolution. The imaging lens and the image sensor may be suitable for use as a TOF depth sensor with active illumination in the NIR wavelength range where a faster lens and more light integration are desired, as well as a computer vision sensor with passive illumination in the visible wavelength range where higher image resolution and greater depth of field are desired. The imaging lens may be suitable for use for both imaging visible light at a lower photo speed and imaging IR light at a faster photo speed. These and other embodiments of the invention along with many of its advantages and features are described in more detail in conjunction with the text below and attached figures.

The present invention relates generally to imaging systems with a multiple f-number lens. In optics, the f-number (sometimes referred to as the focal ratio, f-ratio, f-stop, or relative aperture) of a lens is the ratio of the lens's focal length to the diameter of the entrance pupil. The f-number is a dimensionless number that is a quantitative measure of lens speed. Thus, the f-number or f/# is given by: <MAT> where f is the focal length, and D is the diameter of the entrance pupil (effective aperture). A higher f-number implies a smaller diameter stop for a given focal-length lens. Since a circular stop has area A = πr<NUM>, doubling the aperture diameter and therefore halving the f-number will admit four times as much light into the system. Conversely, increasing the f-number of an imaging lens decreases the amount of light entering a camera by decreasing the aperture size. For example, doubling the f-number will admit ¼ as much light into the system.

To maintain the same photographic exposure when doubling the f-number, the exposure time would need to be four times as long, or alternatively, the illumination would need to be increased to a level four times as high as the original level. Increasing the f-number may have the benefit of increasing the depth of field (DoF) and increasing the spatial resolution of an image (e.g., as measured by modulation transfer function or MTF).

<FIG> illustrates schematically a system <NUM> that includes an imaging system <NUM> and an illumination source <NUM> according to an embodiment of the present invention. The system <NUM> may be integrated in a goggle, as illustrated in <FIG>, that can be worn by a user for virtual reality (VR) or augmented reality (AR) experiences. The system <NUM> may include other optical and electronic components for creating VR and AR experiences.

In one embodiment, the imaging system <NUM> and the illumination source <NUM> may be used for time-of-flight (TOF) depth sensing. The illumination source <NUM> can be configured to emit a plurality of laser pulses. A portion of each of the plurality of laser pulses may be reflected off of an object in front of the user. The portion of each of the plurality of laser pulses reflected off of one or more objects may be received and imaged by the imaging system <NUM>. The imaging system <NUM> can be configured to determine a time of flight for each of the laser pulses from emission to detection, thereby determining the distance of the object from the user. The illumination source <NUM> may comprise a laser source, such as a vertical-cavity surface-emitting laser (VCSEL). In some embodiments, the laser source may be configured to emit laser pulses in the near infrared (NIR) wavelength range, for example in the wavelength range from about <NUM> to about <NUM>. The illumination source <NUM> may also include a collimation lens for collimating the plurality of laser pulses.

In some embodiments, the imaging system <NUM> may also be used for computer vision. When used for computer vision, the imaging system <NUM> is configured to image objects in front of the user that are illuminated by passive ambient light in the visible wavelength range. By using a shared imaging system for both TOF depth sensing and computer vision, lower cost and more compact system design may be realized. It should be understood that, although the imaging system <NUM> is described above as part of an AR or VR system, the imaging system <NUM> may be used in other systems. In other embodiments, the world cameras (WC) <NUM> and <NUM>, as well as the picture camera <NUM>, may also be configured for dual functions, i.e., for imaging both visible and infrared light.

In some embodiments, the system <NUM> may operate the imaging system <NUM> in a time-shared fashion such that depth sensing and computer vision are alternately performed at different time slots. In some embodiments, the duration of each time slot may range from about <NUM> to about <NUM>, so that there is no significant latency in either depth sensing or computer vision. In other embodiments, the system <NUM> may operate the imaging system <NUM> to perform depth sensing and computer vision simultaneously, as described in more detailed below.

<FIG> illustrates schematically an imaging system <NUM> that may be used for dual-wavelength sensing according to some embodiments of the present invention. For example, the imaging system <NUM> may be used for both TOF depth sensing in the NIR wavelength range and computer vision in the visible wavelength range. The imaging system <NUM> includes an imaging lens <NUM> and an image sensor <NUM> positioned at an image plane of the imaging lens <NUM>. The imaging lens <NUM> may include one or more lens elements 216a-216e disposed along an optical axis. The imaging lens may further include an aperture stop <NUM> that may define the entrance pupil size. In a lens system, the limiting diameter that determines the amount of light that reaches the image is called the aperture stop. In some embodiments, the aperture stop may be positioned near the front of a compound imaging lens. In some other embodiments, the aperture stop may be positioned between two groups of lens elements of a compound imaging lens (e.g., as illustrated in <FIG>). In such cases, the entrance pupil size is determined by the image of the aperture stop formed by the lens elements preceding the aperture stop. In the following, it is assumed that the entrance pupil size is the same as the aperture stop size.

When the imaging system <NUM> is used for TOF depth sensing, it may be advantageous to configure the imaging lens <NUM> as a fast lens so that a relatively low power laser source may be used for active illumination. Lower power illumination may lead to lower cost, smaller form factor, and lower power consumption, among other advantages. In some cases, a relatively low f/#, for example in a range from about f/<NUM> to about f/<NUM>, may be desirable for TOF depth sensing. In contrast, when the imaging system <NUM> is used for computer vision, it may be advantageous to configure the imaging lens <NUM> as a slow lens so that higher spatial resolution and greater depth of field (DoF) may be achieved. In some cases, a relatively high f/#, for example in a range from about f/<NUM> to about f/<NUM>, may be desirable for computer vision. The imaging system <NUM> may be applied to other applications where it may be desirable to have different lens speeds for sensing light in different wavelength ranges (e.g., for infrared sensing and visible light sensing).

According to an embodiment of the present invention, the imaging lens <NUM> includes a filter <NUM> positioned at the aperture stop <NUM> that functions as a wavelength selective filter. <FIG> shows a schematic plan view of a filter <NUM> that is used in the imaging lens <NUM> according to an embodiment of the present invention. The filter <NUM> includes two regions: a central (e.g., circular) region <NUM> with a first diameter D<NUM>, and an outer (e.g., annular) region <NUM> surrounding the central region <NUM>. The outer region <NUM> is characterized by a second diameter D<NUM> as its outer diameter. The second diameter D<NUM> may be substantially the same as the diameter of the aperture stop <NUM>. It should be understood that, although the central region <NUM> is depicted as having a circular shape in <FIG>, other shapes, such as elliptical, square, rectangular shapes can also be used. Similarly, although the outer region <NUM> is depicted as having an annular shape in <FIG>, other shapes are also possible.

<FIG> is a plot of an exemplary transmittance curve as a function of wavelength for the central region <NUM> of the filter <NUM> according to an embodiment of the present invention. <FIG> is a plot of an exemplary transmittance curve as a function of wavelength for the outer region <NUM> of the filter <NUM> according to an embodiment of the present invention. As illustrated in <FIG>, the central region <NUM> of the filter <NUM> is configured to have a first transmission band <NUM> in the NIR wavelength range (e.g., from about <NUM> to about <NUM>) and a second transmission band <NUM> in the visible (VIS) wavelength range (e.g., from about <NUM> to about <NUM>). Accordingly, the central region <NUM> is characterized by high transmittance values in both the NIR and the visible wavelength ranges. As illustrated in <FIG>, the outer region <NUM> is configured to have only one transmission band <NUM> in the NIR wavelength range (e.g., from about <NUM> to about <NUM>), such that the outer region <NUM> is characterized by high transmittance values in the NIR wavelength range but low transmittance values in the visible wavelength range.

In some embodiments, the filter <NUM> may comprise a multilayer thin film stack formed on a surface of a transparent substrate such as glass. A multilayer thin film may comprise a periodic layer system composed from two or more materials of differing indices of refraction. This periodic system may be engineered to significantly enhance the transmittance of the surface in one or more desired wavelength ranges, while suppressing the transmittance of the surface in other wavelength ranges. The maximum transmittance may be increased up to nearly <NUM>% with increasing number of layers in the stack. The thicknesses of the layers making up the multilayer thin film stack are generally quarter-wave, designed such that transmitted beams constructively interfere with one another to maximize transmission and minimize reflection. In one embodiment, the multilayer thin film stack in the central region <NUM> may be engineered to have two high transmittance bands, one in the visible wavelength range and the other in the NIR wavelength range, and have low transmittance for all other wavelengths. The multilayer thin film stack in the annular region <NUM> may be engineered to have only one high transmittance band in the NIR wavelength range, and have low transmittance for all other wavelengths. In other embodiments, other types of bandpass filters, such as metasurface filter, may be used.

<FIG> illustrates a schematic cross-sectional view of a wavelength-selective filter <NUM> according to some embodiments of the present invention. The filter <NUM> may include a transparent substrate <NUM> such as a piece of glass, a first multilayer thin film <NUM> disposed on a front surface of the substrate <NUM>, and a second multilayer thin film <NUM> disposed on the first multilayer thin film <NUM>. The first multilayer thin film <NUM> may have a circular shape with a diameter D<NUM>. The second multilayer thin film <NUM> may have an annular shape with an inner diameter D<NUM> and an outer diameter D<NUM>. In some embodiments, the filter <NUM> may further include an anti-reflective coating <NUM> on the back surface of the substrate <NUM>.

The first multilayer thin film <NUM> may be configured to have a transmittance curve that exhibits a first transmission band <NUM> in the NIR wavelength range (e.g., about <NUM> to about <NUM>) and a second transmission band <NUM> in the visible (VIS) wavelength range (e.g., about <NUM> to about <NUM>), as illustrated in <FIG>. The second multilayer thin film <NUM> may be configured as a high-pass filter that transmits light in the NIR wavelength range and blocks light in the visible wavelength range, as illustrated by the dashed curve <NUM> in <FIG>. As such, the combination of the first multilayer thin film <NUM> and the second multilayer thin film <NUM> may result in an effective transmittance curve <NUM> as illustrated in <FIG> for the outer region of the filter <NUM>. Thus, the outer region of the filter <NUM> may effectively transmit only light in the NIR wavelength range, while the central region of the filter <NUM> may transmit light in both visible and NIR wavelength ranges.

When the filter <NUM> or <NUM> is positioned at the aperture stop <NUM> in the imaging lens <NUM> as illustrated in <FIG>, the filter <NUM> or <NUM> may effectively give rise to two different apertures for the imaging lens <NUM> depending on the wavelength range of the light being imaged. Referring to <FIG> and <FIG>, when the imaging lens <NUM> is used for imaging NIR light, for example for TOF depth sensing where the illumination laser source <NUM> (as illustrated in <FIG>) operates in the NIR wavelength range, the NIR light is transmitted through both the central region <NUM> and the outer region <NUM> of the filter <NUM>. Thus, the effective aperture of the imaging lens <NUM> for NIR light is the second diameter D<NUM>. When the imaging lens <NUM> is used for imaging visible light, for example for computer vision where the illumination is from the ambient visible light, the visible light is transmitted only through the central region <NUM>. Thus, the effective aperture of the imaging lens <NUM> for visible light is the first diameter D<NUM>. The imaging lens <NUM> with the wavelength-selective filter <NUM> may be applied to other applications where it may be desirable to have different lens speeds for sensing light in different wavelength ranges.

Assume that the imaging lens <NUM> has a focal length f. When the imaging lens is used for imaging visible light, the imaging lens <NUM> may be characterized by a first f/# for visible light given by, <MAT> When the imaging lens is used for imaging NIR light, the imaging lens <NUM> may be characterized by a second f/# for NIR light given by, <MAT>.

Thus, the imaging lens <NUM> can be configured to have a relatively low f/#NIR for TOF depth sensing in the NIR wavelength range, and a relatively high f/#VIS for computer vision in the visible wavelength range. For TOF depth sensing, a lower f/# means that more active illumination NIR light can pass through the imaging lens <NUM>. Therefore a relatively low power laser source may be used for illumination, which may lead to lower cost, smaller form factor, and lower power consumption, among other advantages. In some embodiments, the value of D<NUM> may be chosen such that f/#NIR is in a range from about f/<NUM> to about f/<NUM>.

For computer vision in the visible wavelength rage, a higher f/# may afford higher spatial resolution at the image plane (e.g., as measured by MTF) and greater DoF, among other advantages. In fact, a lower f/# may not be desired when imaging visible light in some cases. As described more fully below, image sensors typically have higher quantum efficiencies in the visible wavelength range than in the NIR wavelength range. Thus, the image sensor may be saturated when a fast lens is used for imaging visible light. In some embodiments, the value of D<NUM> may be chosen such that f/#VIS is in a range from about f/<NUM> to about f/<NUM>. The intensity ratio between VIS and NIR modes can be controlled by setting the ratio D<NUM>/D<NUM> accordingly. In some embodiments, a ratio of D<NUM>/D<NUM> may be chosen to be in the range from about <NUM> to about <NUM>. In one embodiment the ratio of D<NUM>/D<NUM> may be chosen to be about <NUM>, so that the value of f/#VIS is about twice as large as the value of fl#NIR.

<FIG> illustrates a schematic imaging system according to some embodiments. The imaging system includes a wavelength-selective filter <NUM>, an optical lens <NUM>, and an image sensor <NUM>. Although a single lens element is depicted for the optical lens <NUM> in <FIG> for simplicity of illustration, the optical lens <NUM> may include several lens elements. The filter <NUM> may include a transparent substrate <NUM> such as a piece of glass, a first multilayer thin film <NUM> that has a circular shape with a first diameter D<NUM>, and a second multilayer thin film <NUM> that has an annular shape surrounding the first multilayer thin film <NUM> with an outer diameter of D<NUM>. The first multilayer thin film <NUM> may be configured to have high transmittance for both the visible and NIR wavelength ranges, and the second multilayer thin film <NUM> may be configured to have high transmittance for only the NIR wavelength range, as discussed above.

As illustrated in <FIG>, an incoming light ray in the visible wavelength range may be transmitted by the first multilayer thin film <NUM> and form an image spot <NUM> at the image sensor, as illustrated by the light path represented by the solid arrows. A portion of the incoming light may be reflected by the image sensor <NUM> and incident on a back side of the second multilayer film <NUM>, as illustrated by the light path represented by the dashed arrows. For incoming light in the visible wavelength range, the reflected light may be reflected by the second multilayer thin film <NUM>, as the second multilayer thin film <NUM> is configured to have low transmittance values and high reflectance values in the visible wavelength range. The light reflected by the second multilayer thin film <NUM> may form a ghost image <NUM> at the image sensor <NUM>. Note that, for incoming light in the NIR wavelength range, the portion of the light reflected by the image sensor <NUM> and incident on the back side of the second multilayer thin film <NUM> will be mostly transmitted by the second multilayer thin film <NUM>, as the second multilayer thin film <NUM> is configured to have high transmittance values in the NIR wavelength range. Thus, the filter <NUM> may not present a significant ghost image problem for light in the NIR wavelength range.

<FIG> shows a ray tracing diagram of an exemplary imaging system for a field point (e.g., collimated rays at a certain incidence angle) according to some embodiments. The image system may include a wavelength-selective filter <NUM>, an optical lens <NUM>, and an image sensor <NUM>. <FIG> shows intensity distributions at the image sensor <NUM> as simulated by the ray tracing. As illustrated, the intensity distributions show an image point <NUM>, as well as a ghost image <NUM>. The ghost image may obscure the real image. Therefore, it may be desirable to prevent the formation of the ghost image.

<FIG> illustrates a schematic cross-sectional diagram of a wavelength-selective filter <NUM> that may be used in an imaging system and may prevent ghost image formation according to some embodiments. Similar to the wavelength-selective filter <NUM> illustrated in <FIG>, the filter <NUM> includes a transparent substrate <NUM>, a first multilayer thin film <NUM> formed on a front side of the substrate <NUM> having a circular shape with a first diameter D<NUM>, and a second multilayer thin film <NUM> formed on the front side of the substrate <NUM> having an annular shape surrounding the first multilayer thin film <NUM> with an outer diameter of D<NUM>. The first multilayer thin film <NUM> may be configured to have high transmittance values in both the visible and NIR wavelength ranges, and the second multilayer thin film <NUM> may be configured to have high transmittance values in only the NIR wavelength range, as discussed above.

The filter <NUM> may further include a third thin film <NUM> formed on a back side of the substrate <NUM>. The third thin film <NUM> may have an annular shape with an outer diameter D<NUM> and an inner diameter D<NUM>. D<NUM> may be slightly greater than the inner diameter D<NUM> of the second multilayer thin film <NUM>, so as not to block incoming light rays entering the imaging system through the central region (e.g., the first multilayer thin film <NUM>) of the wavelength-selective filter <NUM>. In some embodiments, the value of D<NUM> may depend on the thickness of the substrate <NUM>. For a relatively thin substrate <NUM>, D<NUM> may be comparable to D<NUM>. The third thin film <NUM> may be configured to have high absorption coefficients in the visible wavelength range and high transmittance values in the NIR wavelength range. Thus, the third thin film <NUM> may be referred to as a "black coating. " As visible light reflected off of the image sensor <NUM> incident on the third thin film <NUM>, a significant portion of it may be absorbed by the third thin film <NUM>, and only a small portion of it may be transmitted by the third thin film <NUM> and incident on the back surface of the second multilayer thin film <NUM> as illustrated by the light path represented by the thinner dashed arrows in <FIG>. Therefore, the intensity of the ghost image <NUM> may be significantly reduced as compared to the case where the filter <NUM> without the "black coating" is used as illustrated in <FIG>.

<FIG> shows the intensity distribution of a ghost image from ray tracing simulation using the wavelength-selective filter <NUM> illustrated in <FIG> according to some embodiments. <FIG> shows the intensity distribution of a ghost image from ray tracing simulation using the wavelength-selective filter <NUM> illustrated in <FIG> that includes the "black coating" <NUM> according to some embodiments. As illustrated, the intensity of the ghost image may be significantly reduced by including the "black coating" <NUM> in the wavelength-selective filter <NUM>. <FIG> shows the ratio of the ghost image intensity using the wavelength-selective filter <NUM> that does not include a "black coating" and the ghost image intensity using the wavelength-selective filter <NUM> with the "black coating" <NUM>. As illustrated, the ghost image intensity can be reduced by as much as <NUM> fold by including the "black coating" <NUM> in the wavelength-selective filter <NUM>.

<FIG> illustrates a schematic cross-sectional diagram of a wavelength-selective filter <NUM> according to some other embodiments. The filter <NUM> may include a transparent substrate <NUM>, a first multilayer thin film <NUM> formed on a front surface of the substrate <NUM>. The first multilayer thin film <NUM> may be configured to have a first transmission band <NUM> in the NIR wavelength range and a second transmission band <NUM> in the visible wavelength range, as illustrated in <FIG>. The filter <NUM> may further include a second multilayer thin film <NUM> formed on the outer region of the first multilayer thin film <NUM>. The second multilayer thin film <NUM> may be configured to be a high-pass filter similar to the wavelength-selective filter <NUM> illustrated in <FIG>. The filter <NUM> may further include an anti-reflective coating <NUM> formed on a back surface of the substrate <NUM>. The anti-reflective coating <NUM> can prevent or reduce the amount of incoming light being reflected off of the back surface of the substrate <NUM>. The filter <NUM> may further include a "black coating" <NUM> formed on the back surface of the anti-reflective coating <NUM>. The "black coating" <NUM> may be configured to absorb visible light and transmit NIR light as discussed above.

<FIG> shows a transmittance curve <NUM> and a reflectance curve <NUM> of the "black coating" <NUM> as a function of wavelength according to some embodiments. A transmittance curve <NUM> of the first multilayer thin film <NUM> is also shown. As illustrated, the "black coating" <NUM> can be configured to have low transmittance values for the visible wavelength range from about <NUM> to about <NUM>, and high transmittance values in the NIR wavelength range from about <NUM> to about <NUM>. The "black coating" <NUM> may have relatively high reflectance values in the wavelength range from about <NUM> to about <NUM>, but this may not significantly affect the performance of the wavelength-selective filter <NUM> as light in this wavelength range is mostly blocked by the first multilayer thin film <NUM> as evidenced by the transmittance curve <NUM> of the first multilayer thin film <NUM>.

Note that the "black coating" <NUM> has both low reflectance values and low transmittance values in the visible wavelength range. Thus, the "black coating" <NUM> may substantially absorb visible light, thereby preventing visible light reflected off of the image sensor <NUM> (as illustrated in <FIG>) from being transmitted and incident on the back side of the second multilayer thin film <NUM> to form a ghost image <NUM> on the image sensor <NUM>. In contrast, the anti-reflective coating <NUM> is normally configured to have low reflectance values but high transmittance values. Thus, visible light reflected off of the image sensor <NUM> may be transmitted by the anti-reflective coating <NUM> and be reflected by the second multilayer thin film <NUM> to form the ghost image <NUM> on the image sensor <NUM> in absence of the "black coating" <NUM>.

<FIG> shows a reflectance curve <NUM> of the second multilayer thin film <NUM> as a function of wavelength according to some embodiments. As illustrated, the second multilayer thin film <NUM> may be configured to have low reflectance values (thus high transmittance values) only in the NIR wavelength range from about <NUM> to about <NUM>, and relatively high reflectance values for all other wavelengths. <FIG> also shows the reflectance curve <NUM> of the "black coating" <NUM>, as well as the transmittance curve <NUM> of the first multilayer thin film <NUM>. As illustrated, the low reflectance values of the "black coating" <NUM> in the visible wavelength range may reduce reflection of light in the visible wavelength range, thereby reduce the intensity of the ghost image.

<FIG> shows an exemplary quantum efficiency (Q. ) curve <NUM> as a function of wavelength of an image sensor <NUM> that may be used in the imaging system <NUM> as illustrated in <FIG>, according to an embodiment of the present invention. As illustrated, the quantum efficiency of the image sensor <NUM> in the visible (VIS) wavelength range can be as much as four times of the quantum efficiency in the NIR wavelength range. Therefore, a low f/# lens may allow too much visible light to pass through the imaging lens <NUM> to the image sensor <NUM> and may saturate the image sensor <NUM>.

In some embodiments, the image sensor <NUM> in the imaging system <NUM> illustrated in <FIG> may comprise a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) device that converts light into electrons in a two-dimensional array of pixel cells. <FIG> illustrates schematically a plan view of the image sensor <NUM> according to an embodiment of the present invention. The image sensor <NUM> may include a two-dimensional array of pixel cells <NUM>. The value of the accumulated charge of each pixel cell <NUM> may be read out to obtain an intensity distribution of the image. When the imaging system <NUM> is used for computer vision in the visible wavelength range, it may be desirable to have the highest possible spatial resolution at the image sensor <NUM>. On the other hand, when the imaging system <NUM> is used for TOF depth sensing in the NIR wavelength range, it may be advantageous to have more light integration at the expense of pixel resolution to achieve better signal to noise ratio (SNR).

According to some embodiments of the present invention, the image sensor <NUM> may be operated at different resolution modes for the visible wavelength range and the NIR wavelength range. In one embodiment, the image sensor <NUM> may be operated at the native resolution for the visible wavelength range, i.e., at the maximum possible resolution that the physical pixel size of the image sensor can support. Thus, for computer vision in the visible wavelength range, the image sensor <NUM> may be operated such that the accumulated charge in each pixel cell <NUM> is read out.

For the NIR wavelength range, the image sensor <NUM> may be operated at a resolution that is lower than the native resolution for greater light integration. <FIG> illustrates schematically a mode of operating the image sensor <NUM> according to an embodiment of the present invention. The two-dimensional array of pixel cells <NUM> may be binned into <NUM>×<NUM> groups <NUM>. Each group <NUM> includes four pixel cells 222a-222d. This mode of operation can be referred to as image sensor pixel binning. In other embodiments, other binning configurations may be used. For example, the pixel cells <NUM> of the image sensor <NUM> may be binned into n×n groups, where n is an integer greater than one. The pixels of the image sensor may also be binned into m×n groups, where m and n are integers and at least one of m and n is greater than one, and m may or may not be equal to n. By binning the pixels, the spatial resolution may be reduced as compared to the native resolution. When the image sensor <NUM> is used in an imaging system that includes the wavelength-selective filter <NUM>, <NUM>, <NUM>, <NUM>, or <NUM>, since the spatial resolution of the imaging system (e.g., as measured by modulation transfer function or MTF) may be lower in the NIR wavelength range because of the greater effective aperture size, the reduction of spatial resolution at the image sensor may not be detrimental. With the greater light integration afforded by binning, a relatively low power laser source may be used for active illumination. Lower power illumination may lead to lower cost, smaller form factor, and lower power consumption, among other advantages.

In one embodiment, binning may be performed at the analog level, where the value of the total accumulated charge for the m×n pixels in each group is read out. In such cases, the readout noise is not added. In another embodiment, binning may be performed at the digital level, where the value of the accumulated charge for each pixel is read out, and the readout values for the m×n pixels in each group are then summed. In such cases, the readout noise is added in the summation process. Thus, the later embodiment may be more appropriate where the readout noise is relatively low.

As described above, the imaging system <NUM> illustrated in <FIG> includes an imaging lens <NUM> that may be characterized by a lower f-number for NIR light and a higher f-number for visible light by utilizing a wavelength-selective filter <NUM> at its aperture stop, and an image sensor <NUM> that may be operated at a lower resolution mode for NIR light using pixel binning and at a higher resolution mode for visible light. The imaging system <NUM> may be suitable for use as a TOF depth sensor with active illumination in the NIR wavelength range where a faster lens and more light integration are desired, as well as a computer vision sensor with passive illumination in the visible wavelength range where higher image resolution and greater depth of field are desired.

<FIG> is a schematic diagram illustrating an imaging system <NUM> according to another embodiment of the present invention. The imaging system <NUM> may include a plurality of lens elements 1702a-1702f, and a filter <NUM> positioned at the aperture stop <NUM>. The imaging system <NUM> may further include a dichroic beam splitter <NUM> positioned in the optical path after the filter <NUM>. The dichroic beam splitter <NUM> may be configured to transmit visible light along a first optical path, and reflect IR light along a second optical path. The imaging system <NUM> may further include a first image sensor <NUM> (VIS sensor) for visible light, and a second image sensor <NUM> (IR sensor) for IR light. The first image sensor <NUM> is disposed along the first optical path and configured to receive the visible light transmitted by the dichroic beam splitter <NUM>. The second image sensor <NUM> is disposed along the second optical path and configured to receive the IR light reflected by the dichroic beam splitter <NUM>. In this fashion, visible light and IR light may be imaged by the first image sensor <NUM> and the second image sensor <NUM>, respectively, at the same time. In this configuration, the first optical path to the first image sensor <NUM> and the second optical path to the second image sensor <NUM> are perpendicular to each other.

<FIG> is a schematic diagram illustrating an imaging system <NUM> according to yet another embodiment of the present invention. The imaging system <NUM> is similar to the imaging system <NUM> in that it also includes a dichroic beam splitter <NUM> positioned after the filter <NUM>, and configured to transmit visible light along a first optical path and to reflect IR light along a second optical path. The imaging system <NUM> further includes a mirror <NUM> positioned along the second optical path and configured to reflect IR light toward the second image sensor <NUM>. In this configuration, the first optical path to the first image sensor <NUM> and the second optical path to the second image sensor <NUM> are parallel to each other. The imaging system <NUM> may further include a lens element <NUM> positioned after the mirror <NUM> along the second optical path for refocusing IR light at the second image sensor <NUM>.

<FIG> is a simplified flowchart illustrating a method <NUM> of operating an imaging system according to an embodiment of the present invention. The method <NUM> includes performing three-dimensional sensing using the imaging system. In some embodiments, performing the three-dimensional sensing is performed in a first time slot. The imaging system may include a near infrared (NIR) light source, an imaging lens, and an image sensor positioned at an image plane of the imaging lens.

In an embodiment, three-dimensional sensing may be performed by: emitting, using the NIR light source, a plurality of NIR light pulses toward one or more first objects (<NUM>). A portion of each of the plurality of NIR light pulses may be reflected off of the one or more first objects. The method also includes receiving and focusing, using the imaging lens, the portion of each of the plurality of NIR light pulses reflected off of the one or more first objects onto the image sensor (<NUM>). The imaging lens may include an aperture stop and a wavelength-selective filter positioned at the aperture stop. The wavelength-selective filter may have a first region and a second region surrounding the first region. In one embodiment, the wavelength-selective filter is configured to transmit NIR light through both the first region and the second region, and to transmit visible light through the first region only. The method further includes detecting, using the image sensor, a three-dimensional image of the one or more first objects by determining a time of flight for the portion of each of the plurality of NIR light pulses from emission to detection (<NUM>).

The method <NUM> further includes performing computer vision in a second time slot using the imaging system. Performing computer vision may be performed in a second time slot following the first time slot. In an embodiment, computer vision may be performed by receiving and focusing, using the imaging lens, visible light from an ambient light source reflected off of one or more second objects onto the image sensor (<NUM>), and detecting, using the image sensor, a two-dimensional intensity image of the one or more second objects (<NUM>). In some embodiments, some of the second objects can be the same as some of the first objects that were imaged in steps <NUM>-<NUM> described above.

According to an embodiment of the present invention, the image sensor includes a two dimensional array of pixels. In some embodiments, detecting the three-dimensional image of the one or more first objects is performed by reading out a total amount of charge for each group of m×n pixels, where m and n are integers, and at least one of m and n is greater than one. In some other embodiments, detecting the three-dimensional image of the one or more first objects is performed by reading out an amount of charge for each pixel of the two-dimensional array of pixels, and calculating a total amount of charge for each group of m×n pixels by summing the amount of charge of the m×n pixels in each group, where m and n are integers, and at least one of m and n is greater than one.

In one embodiment, detecting the two-dimensional intensity image of the one or more second objects is performed by reading out an amount of charge for each pixel of the two-dimensional array of pixels.

In some embodiments, the method <NUM> may include alternately performing three-dimensional sensing and computer vision in sequential time slots, and the duration of each time slot may range from about <NUM> to about <NUM>.

In some other embodiments, the method <NUM> may include performing three-dimensional sensing and computer vision simultaneously using an imaging system such as that illustrated in <FIG> or <FIG>.

It should be appreciated that the specific steps illustrated in <FIG> provide a particular method of <NUM> according to an embodiment of the present invention. Other sequences of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Moreover, the individual steps illustrated in <FIG> may include multiple sub-steps that may be performed in various sequences as appropriate to the individual step. Furthermore, additional steps may be added or removed depending on the particular applications. One of ordinary skill in the art would recognize many variations, modifications, and alternatives.

Claim 1:
An imaging system (<NUM>, <NUM>, <NUM>, <NUM>) comprising:
a near infrared (NIR) light source configured to emit a plurality of NIR light pulses toward one or more first objects, wherein a portion of each of the plurality of NIR light pulses is reflected off of the one or more first objects;
an imaging lens (<NUM>, <NUM>, <NUM>) including:
one or more lens elements (<NUM>, <NUM>, <NUM>) configured to receive and focus the portion of each of the plurality of NIR light pulses reflected off of the one or more first objects onto an image plane, and to receive and focus visible light reflected off of one or more second objects onto the image plane;
an aperture stop (<NUM>); and
a filter (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) positioned at the aperture stop (<NUM>), the filter (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>) including:
a central region (<NUM>) with a first linear dimension, the central region (<NUM>) being characterized by high transmittance values in two wavelength ranges, wherein the two wavelength ranges consist in an NIR wavelength range and a visible wavelength range; and
an outer region (<NUM>) surrounding the central region (<NUM>) with a second linear dimension greater than the first linear dimension, the outer region (<NUM>) being characterized by high transmittance values in the NIR wavelength range and low transmittance values in all the other wavelength ranges; and
an image sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) positioned at the image plane, the image sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) including a two-dimensional array of pixels (<NUM>), wherein the image sensor (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>, <NUM>) is configured to:
detect a two-dimensional intensity image of the one or more second objects in an unbinned pixel mode, wherein the two-dimensional intensity image is formed by light in the visible wavelength range transmitted through only the central region (<NUM>) of the filter; and
detect a time-of-flight three-dimensional image of the one or more first objects in a binned pixel mode in which each respective group of two or more adjacent pixels are binned as a binned pixel, wherein the time-of-flight three-dimensional image is formed by light in the NIR wavelength range transmitted through both the central region (<NUM>) and the outer region (<NUM>) of the filter (<NUM>, <NUM>, <NUM>, <NUM>, <NUM>).