Patent Description:
Examples are disclosed that relate to using a time-of-flight (TOF) camera to perform HDR imaging. One example provides a method of generating an HDR image via a differential TOF pixel with two polysilicon gates (polyfingers) controlled by complementary clock signals during an integration period. The method comprises, during the integration period, controlling the first polyfinger for a first exposure time, and during the integration period, controlling the second polyfinger for a second exposure time, the second exposure time being shorter than the first exposure time. The method further comprises, for each pixel of a plurality of pixels, comparing a charge collected at the first polyfinger and a charge collected at the second polyfinger to a threshold, and selecting one of the charge collected at the first polyfinger and the charge collected at the second polyfinger for inclusion in the HDR image based upon the comparing.

HDR imaging techniques may be used to capture HDR images using image sensors that do not have the native capability to adequately capture both low light and high light portions of the scene in a same image. As one example, HDR imaging techniques may be used in an automobile image sensor system to obtain images in conditions having both bright and low-light aspects (e.g. when facing the sun, when encountering headlights at nighttime, etc.).

Various different HDR imaging techniques may be used. For example, HDR images may be produced via a traditional RGB camera by acquiring multiple exposures at different exposure settings in a time-sequential manner, and then merging the sequence of multiple exposures. This may be used to generate an HDR image with a higher luminance range than any individual exposure taken at a single exposure setting. However, as different exposures are taken at different times, any movement within the imaged scene between frames may produce misalignment and motion artifacts (e.g. ghosting) in the resulting HDR image.

To address such problems with motion artifacts, other HDR imaging techniques may utilize spatial multiplexing approaches. As one example, a global shutter multi-photodiode image sensor architecture may help to avoid ghosting associated with time-multiplexed HDR approaches. A global shutter multi-diode image sensor may include two or more photodiodes per pixel to capture different light levels at each photodiode. For example, each pixel may include a relatively larger photodiode to capture lower light scenes and a relatively smaller photodiode to capture brighter scenes. With global shutter, the first photodiode acquires long exposures and the second photodiode acquires short exposures during a same integration period. However, the use of multiple photodiodes per pixel may result in an undesirably large pixel size and complexity, and also may introduce spectral sensitivity mismatch between the differently sized photodiodes.

Another approach to spatial multiplexing is interleaved HDR, in which an image sensor separately acquires long and short exposures at odd and even rows of pixels in a sensor array. However, such techniques reduce image resolution. For example, using only odd rows for long exposures and only even rows for short exposures reduces image resolution by half.

Accordingly, examples are disclosed that relate to using a global shutter TOF pixel configured for depth image sensing (e.g. in a differential TOF camera) to obtain a two-dimensional HDR image of a surrounding environment. As described in more detail below, each pixel of the sensor includes a pair of polysilicon gates (referred to herein as a first polyfinger and a second polyfinger) that are individually controllable to integrate charge from light exposure during an integration period. During the integration period, the first polyfinger integrates light for a first exposure time and the second polyfinger integrates light for a second, shorter exposure time. This allows the sensor array to collect high light and low light data from a scene during a same integration period. At readout, charges collected at each polyfinger are compared to one or more thresholds, such as a pixel saturation value, to determine whether to include the charge collected at the first polyfinger or the charge collected at the second polyfinger in an HDR image. Pixels selected for inclusion in the HDR image are then stitched together to form the HDR image. The dual polyfinger architecture for each pixel of the image sensor array may help to avoid size, complexity, and spectral sensitivity mismatch issues found with multi-photodiode approaches. Further, the same sensor array may also capture a depth image by altering the controlling clock signals duty cycle at which each polyfinger integrates active light for depth images compared to HDR images.

<FIG> depicts an example use scenario <NUM> in which a depth image sensor <NUM> of a first automobile <NUM> captures both a depth image and an HDR image of a surrounding real-world environment. As described in more detail below, the depth image sensor <NUM> comprises an array of pixels each having two independently clocked polyfingers configured for time-of-flight (TOF) depth imaging. The depth image sensor <NUM> further comprises an electrically controlled optical filter configured to switch between passing visible light or infrared light (IR), which may help to switch between depth imaging and visible light imaging modes. The depth image sensor <NUM> is operably coupled to a computing system, which may be integrated with or located remotely from the depth image sensor (e.g. elsewhere in automobile <NUM>). Example computing systems are described below with reference to <FIG>.

In this example, as the first automobile <NUM> approaches a second automobile <NUM>, depth images acquired via the depth image sensor <NUM> during a first set of integration periods may help to monitor a proximity of the first automobile <NUM> to the second automobile <NUM>. During a depth sensing integration period, the first polyfinger is controlled to integrate charge for a first exposure time, and the second polyfinger is controlled to integrate charge for a second exposure time, which may be non-overlapping with the first exposure time. In depth sensing, each polyfinger may integrate charge for approximately <NUM>% of the integration period. Then, for each pixel, the difference between the charge integrated at the first polyfinger and the charge integrated at the second polyfinger may be used to compute a depth value for the pixel.

HDR images obtained via the depth sensor <NUM> may be used for display on an in-vehicle display system. During an HDR integration period, the computing system may control the first polyfinger to integrate charge for a relatively longer exposure time and control the second polyfinger to integrate charge for a relatively shorter exposure time. In some examples, the relatively longer exposure time may comprise <NUM>-<NUM>% of the integration period, whereas the shorter exposure time may comprise <NUM>-<NUM>% of the integration period. After integration, for each pixel of the pixel array, the computing system compares the charge collected at the first polyfinger and the charge collected at the second polyfinger to one or more thresholds (e.g. by converting the charges to digital signals via an analog-to-digital converter) to determine which charge to include in the two-dimensional image. The computing system then stitches together selected data for all of the pixels to form an HDR image. As shown in <FIG>, the HDR image <NUM> generated in the use scenario <NUM> of <FIG> may be displayed via an interior display of the first automobile <NUM>.

<FIG> depicts an example use scenario <NUM> in which a first automobile <NUM> is backing into a parallel parking space between two parked automobiles <NUM> and <NUM>. A depth image sensor <NUM> located at a rear of the first automobile <NUM> acquires, during a first set of integration periods, depth images of real-world surroundings, which are used to monitor a distance between the first automobile <NUM> and other objects (e.g. the parked vehicle <NUM>). During each of a different, second set of integration periods, a first polyfinger and second polyfinger of each pixel of the depth image sensor <NUM> are controlled to collect long and short exposures, as described above. The depth image sensor <NUM> may switch an electrically controlled optical filter between passing infrared light for the first set of integration periods and visible light for the second set of integration periods. The two exposures for each pixel are then compared to one or more thresholds to determine which charge for each pixel to include in an HDR image, and the selected exposures for each pixel are stitched to other exposures to form the HDR image. As shown in <FIG>, the HDR image <NUM> of the surrounding environment may be output to a display, e.g. to provide visual feedback to a driver of the first automobile <NUM>.

A depth image sensor may generate depth images and HDR images in any suitable sequence and proportion. In some examples, a depth image sensor may be configured to alternately generate HDR images and depth images at a <NUM>:<NUM> ratio. In other examples, a depth image sensor may be configured to generate two or more depth images per each HDR image.

While the examples of <FIG> use machine vision in automotive contexts, in other examples a depth image sensor may be used to acquire HDR images in any other suitable context. For example, a depth image sensor may be used as an eye of a robotic system. In one such example, a robotic surgical system may perform depth sensing between a surgical instrument and an anatomical reference point for instrument tracking, and may utilize HDR imaging for displaying visible images to a surgeon. As another example, a depth image sensor may be integrated with a wearable device, such as a head-mounted display device, to perform depth sensing and present video feedthrough augmented reality (AR) imagery, among other possible uses.

<FIG> illustrates an example pixel <NUM> of a time-of-flight (TOF) image sensor that may be used as the depth image sensor <NUM> and/or <NUM>. An image sensor array comprising pixel <NUM> may be fabricated on a silicon wafer using a standard complementary metal-oxide-semiconductor (CMOS) process to pattern the wafer. The CMOS process comprises front-end-of-line (FEOL) and back-end-of-line (BEOL) stages, which form a structure having an FEOL (or silicon EPI) layer <NUM> and a BEOL layer <NUM>. Pixel <NUM> comprises a first independently controllable polyfinger photoelectron collector 506A and a second independently controllable polyfinger photoelectron collector 506B, which may be formed on the BEOL layer <NUM> in the BEOL stage. Further, pixel <NUM> includes an optical stack <NUM> comprising one or more dielectric anti-reflection layers. The optical stack <NUM> further may include a micro-lens array.

An electrical bias applied to the first polyfinger 506A is shifted in time with respect to the electrical bias applied to the second polyfinger 506B. For depth imaging, the shift may be performed in a manner to achieve an approximately <NUM>% duty cycle for each polyfinger 506A, 506B. In contrast, for HDR imaging, the shift may be performed in a manner to achieve a longer exposure at one polyfinger and a shorter exposure at the other polyfinger. In either case, photoelectrons a and b generated by absorption of light are injected into the silicon FEOL layer <NUM> and move toward one or the other of the independently controllable polyfingers 506A, 506B, depending upon the applied bias.

<FIG> schematically shows aspects of an example TOF depth image sensor <NUM> comprising an image sensor array <NUM> having a plurality of pixels <NUM>. The depth image sensor <NUM> includes an emitter <NUM> configured to irradiate a subject <NUM> with modulated light-sinusoidally modulated, pulse modulated, or modulated according to any other periodic waveform. In a more particular embodiment, the emitter <NUM> may be a programmable near-infrared laser capable of emitting in a continuous-modulation mode or in a repeating-burst mode. In one specific example, the emitter <NUM> is configured to operate at a modulation frequency of <NUM>-<NUM> megahertz (MHz).

Some modulated light from the emitter <NUM> reflects back from the subject <NUM> to the image sensor array <NUM>. Because the light pulses received at the photodetector array <NUM> have traveled out to the subject <NUM> and back, they differ in phase from the pulse train released by the emitter <NUM>. The phase difference varies (e.g., increases) in proportion to the distance the subject <NUM>, allowing a distance to be determined at each pixel <NUM> when depth imaging. Likewise, during HDR imaging, the controller <NUM> controls the first and second polyfingers of each pixel <NUM> to respectively collect and read out charge from longer and shorter exposures, and one of the longer or shorter exposure at each pixel is selected to be included in an HDR image, for example, based upon thresholding.

To provide some measure of ambient-light rejection, the image sensor array <NUM> may be arranged behind an optical filter <NUM>. As mentioned above, the optical filter <NUM> may be electrically switchable to selectively pass either visible light or IR light at any one time, thereby configuring the image sensor array <NUM> to switch between visible light imaging and depth imaging contexts. In one embodiment, the emitter <NUM> may be a narrow-band infrared (IR) emitter such as an IR laser or IR light-emitting diode (LED). Irradiance and photo-detection in the IR provides an additional potential advantage in that a human subject will not detect the irradiance from the emitter <NUM>. When controlled to pass IR light, the passband may be chosen to match the emission wavelength band of emitter <NUM>.

<FIG> shows a schematic diagram of an example electrical circuit <NUM> for driving and reading pixel <NUM>, and <FIG> shows a timing diagram illustrating an example method for controlling circuit <NUM>. Circuitry for the first polyfinger (illustrated as photogate A, or PGA) is collectively indicated at <NUM>, and circuitry for the second polyfinger (illustrated as photogate B, or PGB) is collectively indicated at <NUM>. As mentioned above, PGA and PGB are individually controllable, so that a duty cycle for integrating charge at each polyfinger can be adjusted between depth imaging and intensity imaging. In some examples, a size of each pixel <NUM> is on the order of <NUM> micrometers.

The timing scheme <NUM> may be implemented as stored instructions executable by the control circuitry of the depth image sensor <NUM> and/or a host computing system, in various examples. To initiate an integration/readout cycle, a global reset <NUM> may be used to clear any remaining charge at floating diffusion nodes FDA, FDB and the polyfingers PGA, PGB (<FIG>). Next, in an integration period <NUM>, for each polyfinger a lower voltage is applied to a first transfer gate TG1, and a higher voltage is applied to a second transfer gate TG2. During the integration period <NUM>, the first polyfinger (PGA) is biased for a first longer exposure time to collect charge for the longer exposure time, and the second polyfinger (PGB) is biased for a second, shorter exposure time to collect charge for the shorter exposure time. In one example where the integration period <NUM> is <NUM> microseconds, the first polyfinger may be controlled to achieve a total exposure of <NUM> to <NUM> microseconds during the integration period <NUM> for a <NUM>-<NUM>% duty cycle, and the second polyfinger may be controlled for <NUM> to <NUM> seconds during the integration period <NUM> for a <NUM>-<NUM>% duty cycle. In the depicted example, the exposures of PGA and PGB are interleaved. This helps to avoid motion artifacts when the captured scene includes moving objects.

To help prevent blooming-the bleeding of photo charge from an overexposed pixel into other nearby pixels-caused by oversaturation, timing scheme <NUM> comprises an anti-blooming period <NUM>. For example, if the sun is within an imaged scene, without the anti-blooming period <NUM>, an oversaturated pixel(s) may cause may blooming in a resulting image. The depicted anti-blooming period <NUM> clears the charges collected at the floating diffusion nodes FDA, FDB and polyfingers PGA, PGB, and flushes the charges out through the reset gate, such that any remaining charges do not affect the charge collected in the in-pixel memory that is readout during the readout period <NUM>.

Next, during row readout <NUM>, all rows of pixels are reset and sampled before charge is transferred from in-pixel memory of each polyfinger PGA, PGB to the respective floating diffusion node FDA, FDB and sampled. The charges are amplified and converted to digital signals via an analog-to-digital converter. In some examples, the received light levels are amplified by a multiplier of <NUM> to <NUM>, which may occur before or after analog-to-digital signal conversion.

<FIG> depicts an example of a monochrome image <NUM> acquired via exposures captured by a first polyfinger (e.g. PGA) of each pixel of a depth image sensor, and <FIG> depicts an example of a monochrome image <NUM> acquired via exposures captured by a second polyfinger (e.g. PGB) of each of the same pixels. In <FIG>, the imaged environment outside the window appears saturated while the imaged office interior includes adequate detail. In contrast, more detail of the imaged outdoor environment is visible in the image <NUM>. Thus, selected pixels of the outdoor environment from <FIG> may be combined with selected pixels of the interior from <FIG> to form a single HDR image.

To select which data from each pixel to include in an HDR image, the charge collected at each pixel for each of the first image <NUM> and the second image <NUM> may be compared to one or more threshold values (e.g. by converting the charge at each pixel of each image to a digital signal using an analog-to-digital converter). As one example, a threshold near a pixel saturation value may be used. In such an example, the signal from the longer exposure may be selected for a pixel unless the signal exceeds the threshold, indicating that the signal is close to or at saturation, in which case the signal from the shorter exposure may be selected. Based on the comparison, one of the charge collected at PGA and the charge collected at the PGB is selected for inclusion in the HDR image. After performing the pixel-by-pixel threshold comparison, the selected pixels from each image are stitched together to form the HDR image.

<FIG> illustrates an example method <NUM> of generating an HDR image via a TOF pixel comprising an array of pixels, each pixel having a first polyfinger and a second polyfinger that are independently controllable to integrate charge during an integration period <NUM>. During the integration period <NUM>, method <NUM> comprises, at <NUM>, controlling the first polyfinger for a first exposure time by biasing the first polyfinger to integrate charge during the first exposure time. In some instances, the first exposure time comprises <NUM>-<NUM>% of the integration period, as indicated at <NUM>. During the integration period <NUM>, method <NUM> also comprises, at <NUM>, controlling the second polyfinger for a second exposure time by biasing the second polyfinger to integrate charge for the second exposure time. In some instances, the second exposure time comprises <NUM>-<NUM>% of the integration period, as indicated at <NUM>.

After the integration period <NUM>, in some examples, method <NUM> may comprise, at <NUM>, clearing charge from floating diffusion nodes, the first polyfinger, and the second polyfinger to prevent blooming. Method <NUM> further comprises, for each pixel of the plurality of pixels, comparing a charge collected at the first polyfinger and a charge collected at the second polyfinger to a threshold, as indicated at <NUM>. The threshold may be near or at a pixel saturation value (e.g. within <NUM>-<NUM>% of the pixel saturation value), as indicated at <NUM>, or may be set at any other suitable level. Based on the comparison, at <NUM> method <NUM> comprises, for each pixel of the plurality of pixels, selecting one of the charge collected at the first polyfinger and the charge collected at the second polyfinger for inclusion in the HDR image. Selecting may comprise selecting the charge collected at the first polyfinger when the charge collected at the first polyfinger does not exceed the threshold, and selecting the charge collected at the second polyfinger when the charge collected at the first polyfinger exceeds the threshold, as indicated at <NUM>.

At <NUM>, method <NUM> comprises, for each pixel of the plurality of pixels, computing an HDR value for the pixel based on the charges integrated during the integration period. Method <NUM> further comprises generating an HDR image, as indicated at <NUM>.

In some examples, the integration period comprises a first integration and method <NUM> comprises, during a second integration period <NUM>, controlling the first polyfinger to integrate charge for a third exposure time <NUM> (e.g. <NUM>% of the second integration period), and controlling the second polyfinger to integrate charge for a fourth exposure time <NUM> (e.g. <NUM>% of the second integration period). Further, for each pixel of the plurality of pixels, method <NUM> comprises computing a depth value for the pixel based on the charge integrated during the third exposure time and the charge integrated during the fourth exposure time to generate a depth image, as indicated at <NUM>. Method <NUM> further comprises generating the depth image, as indicated at <NUM>.

HDR and depth images may be generated in any suitable order. In some instances, method <NUM> comprises generating depth images and HDR images alternately. In other examples, method <NUM> comprises generating two or more depth images per HDR image.

The logic machine <NUM> may include one or more processors configured to execute software instructions.

The term "program" may be used to describe an aspect of computing system <NUM> implemented to perform a particular function. In some cases, a program may be instantiated via logic machine <NUM> executing instructions held by storage machine <NUM>. It will be understood that different programs may be instantiated from the same application, service, code block, object, library, routine, API, function, etc. Likewise, the same program may be instantiated by different applications, services, code blocks, objects, routines, APIs, functions, etc. The term "program" may encompass individual or groups of executable files, data files, libraries, drivers, scripts, database records, etc..

Another example provides a method of generating a high dynamic resolution (HDR) image via a depth image sensor comprising an array of pixels each having a first polyfinger and a second polyfinger, the first polyfinger and the second polyfinger being independently controllable to integrate current during an integration period, the method comprising, during the integration period, controlling the first polyfinger for a first exposure time, during the integration period, controlling the second polyfinger for a second exposure time, the second exposure time being shorter than the first exposure time, and for each pixel of a plurality of pixels, comparing a charge collected at the first polyfinger and a charge collected at the second polyfinger to a threshold, and selecting one of the charge collected at the first polyfinger and the charge collected at the second polyfinger for inclusion in the HDR image based upon the comparing. In such an example, the first exposure time may additionally or alternatively comprise <NUM>-<NUM>% of the integration period, and the second exposure time may additionally or alternatively comprise <NUM>-<NUM>% of the integration period. In such an example, the integration period may additionally or alternatively comprise a first integration period, and the method may additionally or alternatively comprise: during a second integration period, controlling the first polyfinger to integrate charge for a third exposure time, during the second integration period, controlling the second polyfinger to integrate charge for a fourth exposure time, and for each pixel of the plurality of pixels, computing a depth value for the pixel based on the charge integrated during the third exposure time and the charge integrated during the fourth exposure time to generate a depth image. In such an example, depth images and HDR images may additionally or alternatively be generated alternately. In such an example, two or more depth images may additionally or alternatively be generated per HDR image. In such an example, the threshold may additionally or alternatively be within <NUM>-<NUM> of a pixel saturation value. In such an example, selecting the one of the charge collected at the first polyfinger and the charge collected at the second polyfinger for inclusion in the HDR image may additionally or alternatively comprise selecting the charge collected at the first polyfinger when the charge collected at the first polyfinger does not exceed the threshold, and selecting the charge collected at the second polyfinger when the charge collected at the first polyfinger exceeds the threshold. In such an example, the method may additionally or alternatively comprise clearing charge from floating diffusion nodes, the first polyfinger, and the second polyfinger of each pixel at an end of the integration period to prevent blooming.

Another example provides an apparatus comprising a differential TOF pixel comprising an array of pixels each having a first polyfinger and a second polyfinger, the first polyfinger and the second polyfinger being independently controllable to integrate charge during an integration period, a logic subsystem, and a storage subsystem storing instructions executable by the logic subsystem to: during the integration period, control the first polyfinger to integrate charge for a first exposure time, during the integration period, control the second polyfinger to integrate charge for a second exposure time, the second exposure time being shorter than the first exposure time, and for each pixel of the plurality of pixels, compare the charge integrated at the first polyfinger and the charge at the second polyfinger to a threshold, and based on the comparing, select one of the charge integrated at the first polyfinger and the charge integrated at the second polyfinger for inclusion in a high dynamic range (HDR) image. In such an example, the first exposure time may additionally or alternatively comprise <NUM>-<NUM>% of the integration period and the second exposure time may additionally or alternatively comprise <NUM>-<NUM>% of the integration period. In such an example, the integration period may additionally or alternatively comprise a first integration period, and the instructions may additionally or alternatively comprise be executable to: during a second integration period, control the first polyfinger to integrate charge for a third exposure time, during the second integration period, control the second polyfinger to integrate charge for a fourth exposure time, and for each pixel of the plurality of pixels, assign a depth value to the pixel based on the charge integrated during the third exposure time and the charge integrated during the fourth exposure time to generate a depth image. In such an example, the instructions may additionally or alternatively be executable to generate depth images and HDR images alternately. In such an example, the instructions may additionally or alternatively be executable to generate two or more depth images per each HDR image. In such an example, the threshold may additionally or alternatively comprise a pixel saturation value. In such an example, the instructions may additionally or alternatively be executable to select the charge collected at the first polyfinger when the charge collected at the first polyfinger does not exceed the threshold, and select the charge collected at the second polyfinger when the charge collected at the first polyfinger exceeds the threshold. In such an example, the instructions may additionally or alternatively be executable to clear charge at floating diffusion nodes, the first polyfinger, and the second polyfinger of each pixel at an end of the integration period to prevent blooming. In such an example, each pixel may additionally or alternatively comprise a size equal to or less than <NUM> micrometers.

Claim 1:
A method of generating a high dynamic range, HDR, image via a depth image sensor comprising an array of pixels, each pixel having a first charge collection point and a second charge collection point, the first charge collection point and the second charge collection point being independently controllable to integrate current during an integration period, the method comprising:
during a first integration period, controlling the first charge collection point for a first exposure time;
during the first integration period, controlling the second charge collection point for a second exposure time, the second exposure time being shorter than the first exposure time; and
for each pixel of the plurality of pixels,
comparing a charge collected at the first charge collection point and a charge collected at the second charge collection point to a threshold, and
selecting one of the charge collected at the first charge collection point and the charge collected at the second charge collection point for inclusion in the HDR image based upon the comparing; and
during a second integration period, controlling the first charge collection point to integrate charge for a third exposure time of <NUM>% of the second integration period;
during the second integration period, controlling the second charge collection point to integrate charge for a fourth exposure time of <NUM>% of the second integration period; and
for each pixel of the plurality of pixels, computing a depth value for the pixel based on the charge integrated during the third exposure time and the charge integrated during the fourth exposure time to generate a depth image.