Patent Description:
Depth imaging techniques are implemented in time-of-flight cameras and range finders. Depth imaging techniques include pulsing light sources, and measuring reflected light to sense the presence, distance information, depth information, and/or speed information of an object. These optical systems have a wide range of uses, such as security systems, medical systems, automotive systems, aerospace systems, consumer electronic devices, etc..

Depth imagers can include a light source (e.g., light emitting diodes, lasers, etc.). Depth imagers can also include an array of sensors that create a depth image of a scene, where each pixel stores the distance to the object in the field of view of that pixel. Such imagers may also simultaneously output a regular image of the scene, such as a red-green-blue (RGB) image or infrared image.

An example of a depth imaging technique for creating depth images is time-of-flight imaging. Time-of-flight operates by having a light source emit light onto the scene. Light is then reflected off the objects in the scene. Reflected light hits the array of sensors. The signals generated by the array of sensors can be used to calculate distance information and form a depth image.

<CIT> describes depth sensor module and depth sensing method. Here, depth measurements derived by triangulation can be used to calibrate depth maps generated by time-of-flight measurements. <CIT> describes a method and device for determining the distance to a retrospective object. According to <CIT>, an amplitude control regulates a transmitter light source corresponding associated light paths have approximately the same value at the input of a comparator. <CIT> describes a distance measuring device which uses a round-trip time of light to measure a distance to an object. Patent <CIT> describes a method for subtracting background light from an exposure value of a pixel in an imaging array. Patent Publication Number <CIT> describes optical proximity sensors using echo cancellation techniques to detect one or more objects. In particular, an echo canceller is adapted to produce an echo cancellation signal that is combined with the detection signal produced by the light detector to produce an echo cancelled detection signal.

In accordance with a first aspect of the invention, there is provided a method in accordance with claim <NUM>.

In accordance with a third aspect of the invention, there is provided a device in accordance with claim <NUM>.

Depth imagers can implement time-of-flight operations to measure depth or distance of objects. A depth imager can emit light onto a scene and sense light reflected back from the objects in the scene using an array of sensors. Timing of the reflected light hitting the array of sensors gives information about the depth or distance of objects in the scene. In some cases, corrupting light that is outside of a field of view of a pixel in the array of sensors can hit the pixel due to internal scattering or internal reflections occurring in the depth imager. The corrupting light can corrupt the depth or distance measurement. To address this problem, an improved depth imager can isolate and measure the corrupting light due to internal scattering or internal reflections occurring in the depth imager, and systematically remove the measured corrupting light from the depth or distance measurement.

Typically, a light source of a depth imager is close to (or adjacent to) the array of sensors and moves along with the array of sensors if the array of sensors is mobile (e.g., the light source has a fixed position with respect to the array of sensors). For simplicity, the array of sensors is sometimes herein referred to as "sensor". In some other cases, one could consider a static light source as well, or any light source whose position is known over time.

Referring to the case with the light source of a depth imager close to the sensor, it can be understood that the time it takes for the light to return and hit the sensor is proportional to the distance to the object. More precisely, the depth D of an object is given by the equation: <MAT> where c is the speed of light, tR is the time it takes for the light to travel to the object, return, and hit the sensor. The factor of <NUM> in equation (<NUM>) is due to the light having to travel to the object and return from the object. The relationship between tR and D can be different, in some cases more complicated, than equation (<NUM>) depending on the location of the light source relative to the sensor.

<FIG> illustrates a depth imager and time-of-flight operations, according to some embodiments of the disclosure. The depth imager includes light source <NUM> for emitting light, array of sensors <NUM>, and lens <NUM>. The scene may have a plurality of objects, e.g., object <NUM>, object <NUM>, and object <NUM>. The light source <NUM> can include one or more laser diodes. In some embodiments, the light source <NUM> includes light emitters such as vertical cavity surface-emitting lasers or edge-emitting lasers, in some cases, combined with a diffuser to affect the parts of the scene that the light emitters are sending are light to. The array of sensors <NUM> can include charge-coupled device (CCD) sensors or CMOS sensors arranged over a two-dimensional area. The sensors in the array of sensors <NUM> are referred to as pixels. The depth imager may include a driver <NUM> (e.g., electronic driving circuitry) for controlling and driving the light source <NUM> and the array of sensors <NUM>. The depth imager may include processing circuitry <NUM> for receiving and processing signals from the array of sensors <NUM> to form depth estimates.

The emitted light bounces off the objects, and can return and hit the array of sensors <NUM>. Besides the return light from the objects, the array of sensors <NUM> can further receive other extraneous light which may corrupt the time-of-flight measurement. One form of extraneous light is background (BG) light. BG light may come from other sources such as sunlight, other light sources not associated with the system, and light sources associated with other depth imagers. It is desirable to design time-of-flight depth imagers which have some tolerance for BG light.

Depth imagers can have an operating depth range defined by a minimum distance Dmin and a maximum distance Dmax. Outside of the operating depth range, the depth imager does not report distance. One reason for Dmin not being zero could be that the return light for objects of a certain reflectivity closer than Dmin is so strong that the return light saturates the array of sensors <NUM>. Dmax is typically the distance beyond which not enough light returns to the array of sensors <NUM> to determine the distance. In one example, suppose a depth imager has a minimum distance Dmin for its operating depth range, and the minimum time tmin for light to travel to and back from the minimum distance Dmin is thus <MAT>. Dmin and tmin can depend on the individual depth imager.

To obtain a depth for each pixel in the array of sensors <NUM>, a set of measurements are carried out at each pixel. The set of measurements provide information to compute a depth. Each pixel can enable a respective depth to be computed or estimated; the array of sensors <NUM> can generate a depth image based on the respective depths. <FIG> illustrates a measurement that can be performed by a pixel in the array of sensors, according to some embodiments of the disclosure. A measurement can be defined by emitted light (e.g., plot <NUM>), light arriving at the pixel (e.g., plot <NUM>), sensor signal (e.g., plot <NUM>), and collected signal (e.g., plot <NUM>). Emitted light is dictated by when the light source (e.g., light source <NUM>) is turned on and off. In the example shown, emitted light is turned on between times t = <NUM> and t = TL (e.g., duration is TL). The light source is off outside of the interval between times t = <NUM> and t = TL. The start time, when the emitted light is turned on, is set to be t = <NUM> as a reference upon which time instants are defined. The light arriving at the pixel can include BG light (which is assumed to be slowly varying over the course of the measurement), and return light reflected from an object. The light reflected from an object can arrive at the pixel at t = tR. The sensor signal dictates a time interval for the pixel to sense light arriving at the pixel. The sensor signal can define a period of integration for the pixel, where the pixel collects or accumulates charge based on photons of light arriving at the pixel. In the example shown, the pixel is sensing light between t = t<NUM> and t = t<NUM> + Ts (e.g., duration is Ts). In some cases, t<NUM> = tmin. In some cases, Ts = TL. The collected signal can include BG light and return light reflected from an object hitting the pixel during the period of integration. For simplicity, only one sensor signal is shown for controlling a pixel to make one measurement. In practice, a pixel can have multiple charge storage units for collecting multiple light measurements respectively, and thus multiple separate sensor signals can be used to control the charge storage units individually with different timings.

The notation for a measurement done by a pixel is M(X, Y, Z), where X specifies the time when the light source <NUM> turns off (or duration of emitted light from t = <NUM>), Y specifies the start time when the pixel begins sensing light, and Z specifies the duration of the time interval where the pixel is sensing light. The measurement illustrated in <FIG> is M(TL, t<NUM>, Ts). Y can be negative, depending on the measurement.

To achieve accurate measurements, it is desirable to have precise control of the light source <NUM> and be capable of turning light source <NUM> on or off. For example, it is desirable to have driver <NUM> in the depth imager to be able to turn on the light source <NUM> at t = <NUM> and turn off the light source at t = TL. The driver <NUM> preferably can generate a signal to drive the light source <NUM> with sufficiently short rise and fall times.

Referring back to <FIG>, the pixel can convert incoming photons into electrons and store those electrons in charge storage units. The charge storage units can be controlled individually with different sensor signals to achieve different sensor timings and make different measurements. The charge storage units for a given pixel are denoted as Si(i = <NUM>,. ,N - <NUM>) where N is the number of charge storage units per pixel. For example, each pixel include one or more photodiodes (or photosensitive element) for sensing photons and converting the photons into electrons and one or more capacitors for storing the electrons, acting as charge storage unit(s). Sensor signals can start and stop the collection of electrons by the capacitors with specified timings. The series of capacitors can be considered as memory elements, i.e., charge storage units, which store different collected light measurements. To achieve accurate measurements, it is desirable to have precise control of the sensor signals, and be able to have the charge storage units start and stop sensing light at precise timings. The amount of charge stored in the charge storage units of a pixel can be read by processing circuitry <NUM> and processed by the processing circuitry <NUM> to compute depth.

In some cases, depth can be estimated from three light measurements collected by three charge storage units. The depth can be estimated based on a scheme that is tolerant to BG light. A pixel can include three charge storage units, S<NUM>, S<NUM>, and S<NUM>, which can store different collected light measurements. The light source <NUM> can be on from times t = <NUM> to t = TL. S<NUM> can collect light from times t = t<NUM> to t = t<NUM>+TL. The measurement being made by S<NUM> is M(TL, t<NUM>, TL). S<NUM> can collect light from times t = t<NUM> to t = t<NUM> + TL. The measurement being made by S<NUM> is M(TL, t<NUM>, TL). In one case, t<NUM> = t<NUM> + TL. S<NUM> can collect BG light for a time TL with the light source <NUM> off. S<NUM> can collect light from times t = -t<NUM> to t = -t<NUM> + TL. The measurement being made by S<NUM> is M(TL, -t<NUM>, TL). In one case, t<NUM> = TL, and thus S<NUM> can collect light from times t = -t<NUM> = -TL to t = <NUM>. In another case, S<NUM> is M(<NUM>,<NUM>,TL), and the light source is kept off during the measurement, thus S<NUM> is only capturing BG light. Because the light source is not emitting light, the S<NUM> does not collect any reflected light. Generally speaking, the measurement S<NUM> collects BG light for a period of time before the light source is on. A depth D for the pixel can be obtained by computing the following exemplary equation: <MAT> which is obtained when t<NUM> = t<NUM> + Ts, <MAT>. Equation (<NUM>) is obtained when the durations of the time interval where the pixel is sensing light for the three charge storage is the same TL for all three measurements, which need not be the case in general.

The collected light by S<NUM> and S<NUM> are individually adjusted for BG light by subtracting the BG light collected by S<NUM>. The depth is computed based on the ratio of collected light by S<NUM> over the total amount of collected light by S<NUM> and S<NUM>, after the adjustment for BG light S<NUM> is performed. Accordingly, the depth D computed with equation (<NUM>) and the described measurements has the advantage of being tolerant/insensitive to background light, since S<NUM> measuring BG light is subtracted from the measurements done by S<NUM> and S<NUM>.

Measurements made by charge storage units S<NUM>, S<NUM>, and S<NUM> can be repeated a plurality of times and the collected light can be added/accumulated in the respective charge storage units (or in other circuitry) to increase signal strength. The added repeated measurements in a charge storage unit can be used for the depth calculation. If the number of repetitions is different for each charge storage unit, which can be advantageous in some cases to increase the signal-to-noise ratio, the values in the storage units may be divided by the number of repetitions, and the resulting values can be used for depth estimation.

The charge storage units can perform the addition or accumulation of repeated measurements in hardware, using suitable sensor signals. The same charge storage unit can be turned on multiple times with an appropriate sensor signal to make multiple measurements without dumping the electrons or resetting the charge storage unit in between the measurements. As the measurements are repeated, the electrons continue to be added/accumulated in the same charge storage unit.

Referring back to <FIG>, the plots shown have ideal/perfect square pulses, and the computation in equation (<NUM>) assumes the light source <NUM> is turned on with fast (zero) rise time and off with fast (zero) fall time and the charge storage unit is either completely off or completely on. In practice, due to electronics hardware limitations on actual rise and fall times of signals and sensor sensitivities, the plot <NUM> for emitted light and plot <NUM> for sensor signal are likely to not have perfect square pulse shapes. <FIG> illustrates an exemplary measurement that can be performed by a pixel in the array of sensors, according to some embodiments of the disclosure. The emitted light (e.g., plot <NUM>) by the light source <NUM> has an intensity which can vary depending on time. This is due to the physical limitations of the light source <NUM> as well as the limited rise and fall times of the signal driving the light source <NUM>. The emitted light is represented as fTL(t), which is output power as a function of time. TL is the amount of time the light source <NUM> is on. The light arriving at the pixel (e.g., plot <NUM>), which is in part a result of the emitted light, would also have a similar shape as the emitted light. The light arriving at the pixel is represented as a <MAT>, where a is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude), b is the amount of BG light (e.g., constant over time), and tR is the amount of time for the light to travel to and from the object in the scene. The sensor signal (e.g., plot <NUM>) can also have time varying photon conversion efficiency (e.g., number of electrons collected per incoming photon) and the sensor signal may also have limited rise and fall times. Photon conversion efficiency can increase with the strength of the sensor signal. The sensor signal (e.g., representing its photon conversion efficiency as a function of time) is represented by gTS(t - t<NUM>), where t<NUM> is the time when a charge storage unit starts to turn on and TS is an amount of time that the charge storage unit is on. The collected signal (e.g., plot <NUM>), which is a result of the light arriving at the pixel and the sensor signal, would also have a non-ideal shape. A collected signal as a function of time is represented by (a fTL(t - tR) + b) gTS(t - t<NUM>), which is a combination of the function representing light arriving at the pixel and the function representing the sensor signal. Accordingly, the measurement representing a number of collected electrons is the integral of the collected signal: ∫(a fTL(t - tR) + b) gTS(t - t<NUM>)dt.

<FIG> illustrates exemplary measurements that can be performed by a pixel in the array of sensors, accounting for background light, according to some embodiments of the disclosure. As discussed previously, it is possible to take three measurements with charge storage units S<NUM>, S<NUM>, and S<NUM> to subtract out the BG light in the measurements when performing depth estimation. The emitted light (e.g., plot <NUM>) by the light source <NUM> is represented as fTL(t). TL is the amount of time the light source <NUM> is on. The light arriving at the pixel (e.g., plot <NUM>) is represented as a fTL(t - tR) + b. Three sensor signals are used to make three measurements with three charge storage units S<NUM>, S<NUM>, and S<NUM>.

S<NUM> is controlled by sensor signal (e.g., plot <NUM>), which is represented by gTS(t - t<NUM>), where t<NUM> is the time when the charge storage unit starts to turn on and TS is an amount of time that the charge storage unit is on. The measurement made by S<NUM> is M(TL, t<NUM>, TS). The collected signal for S<NUM> can be represented by (a fTL(t - tR) + b) gTS(t - t<NUM>). A measurement by S<NUM> representing a number of collected electrons is the integral of the collected signal: ∫(a fTL(t - tR) + b) gTS(t - t<NUM>)dt.

S<NUM> is controlled by sensor signal (e.g., plot <NUM>), which is represented by gTS(t - t<NUM>), where t<NUM> is the time when the charge storage unit starts to turn on and TS is an amount of time that the charge storage unit is on. The measurement made by S<NUM> is M(TL, t<NUM>, TS). The collected signal for S<NUM> can be represented by (a fTL(t - tR) + b) gTS(t - t<NUM>). S<NUM> can collect light from times t = t<NUM> to t = t<NUM> + TS. A measurement representing a number of collected electrons is the integral of the collected signal: ∫(a fTL(t - tR) + b) gS(t - t<NUM>)dt. In one case, t<NUM> = t<NUM> + TS. In some cases t<NUM> is slightly less than t<NUM> + TS by a predefined amount to allow for some overlap of the two time intervals. The overlapping can be particularly advantageous when the pulse shapes are not ideal. In some cases, the sensor signal for making measurement S<NUM> may have relatively small values close to the edges of the time interval during which the pulse is non-zero, such that a part of the measurement S<NUM> is small and therefore noisy. In such cases, changing the start time of t<NUM> to be slightly less than than t<NUM> + TS by a predefined amount may increase the return signal and therefore the signal-to-noise ratio over a range of depth that is of interest.

S<NUM> can collect BG light for a time TS with the light source <NUM> off. S<NUM> sensor signal (e.g., plot <NUM>) is represented by gTS(t + t<NUM>), where -t<NUM> is the time when the charge storage unit starts to turn on and TS is an amount of time that the charge storage unit is on. The measurement made by S<NUM> is M(TL, -t<NUM>, TS). S<NUM> can collect light from times t = -t<NUM> to t = -t<NUM> + TS. The collected signal for S<NUM> can be represented by b gTS(t + t<NUM>). A measurement representing a number of collected electrons is the integral of the collected signal: ∫ b gTS(t + t<NUM>)dt. In one case, t<NUM> = TS, and thus S<NUM> can collect light from times t = -t<NUM> = -TS to t = <NUM>.

The charge storage units do not all necessarily turn on for the same amount of time TS, and the durations can differ from each other. In some cases, one or more of the durations TS can equal to TL. Before computing a ratio for the depth estimation, the measurement by S<NUM> is subtracted from S<NUM> and S<NUM> to remove the contribution from BG light to the measurements. If S<NUM> and S<NUM> are controlled by sensor signals that do not have the same pulse shape, which in some cases may be the case if they are not pulses of the same width, then the amount of background light they capture may differ, in which case S<NUM> may be multiplied by a factor which is different for S<NUM> and S<NUM> respectively, and subsequently subtracted from S<NUM> and S<NUM> respectively to subtract the correct amount of background light.

To take into account that the plots do not have perfect square pulses and obtain accurate depth estimates, it is possible to measure the ratio <MAT> as a function of depth and store the values in a lookup table. Different values of the ratio serves as the index of the lookup table. The lookup table implements a function H(x) such that <MAT> and allows the estimated depth to be adjusted for the non-ideal pulse shapes. In other words, the result from the lookup table based on the ratio <MAT> can output a corrected depth estimate. Values for H(x) can be extrapolated between the values of x for which H(x) is known. The depth equation becomes: <MAT>.

One of the issues with time-of-flight imagers is the following: the assumption is that each pixel receives light only from the part of the scene that corresponds to its field of view. Phrased differently, the assumption is that each pixel captures light from objects within its field of view. All the light that enters the lens from objects in its field of view are focused onto that pixel alone. However, if light from an object somewhere else in the scene undergoes multiple internal scattering events or multiple internal reflections, then this light can end up hitting a pixel whose field of view it is not in. Accordingly, it is possible for light from one part of the scene to corrupt measurements from pixels whose field of view corresponds to a different part of the scene.

This means that, besides BG light, the light arriving at a pixel can include other extraneous light, such as light due to internal scattering events or multiple internal reflections. This light, i.e., a corrupting light or extraneous light, can significantly impact the depth estimate of the pixel that the light ends up hitting. This effect can be strong for objects nearby the camera, where the amount of light hitting the sensor is large. For a handheld device, fingers or other parts of a hand can be very close to the camera (unintentionally), where the effect can be strong. In a scenario where a pixel's field of view is pointing at a faraway object, and in another part of the scene outside of the pixel's field of view is an object very close to the array of sensors <NUM>, the effect can be pronounced. The amount of light returning to the sensor can decay quadratically with distance, i.e., the light hitting a pixel whose field of view is a given object is proportional to <NUM>/D<NUM> where D is the distance to the object. Therefore, the nearby object may lead to a large amount of light coming back to the sensor relative to light returning from the faraway object. Since the amount of signal from the faraway object is relatively low in comparison, the corrupting light could significantly impact the depth estimate. Multiple internal reflections can lead to an object corrupting many or all of the pixels in the sensor.

Note that this internal reflections problem is not the same as the multipath problem, where the reflections are from multiple objects in the scene that may be centimeters or meters away from each other. The internal reflection problem is quite a different challenge from the multipath problem because the assumed reflections can happen at much closer distances (within centi- or milli-meters) to the depth imager. This difference makes most approaches for the multipath problem impractical for addressing internal reflections, in some cases, those approaches can potentially requiring far more measurements than the methods described herein.

<FIG> illustrate exemplary scenarios for extraneous sources of light hitting a pixel, according to some embodiments of the disclosure. In <FIG>, it can be seen that light reflecting off a nearby object <NUM> undergoes multiple bounces between the lens <NUM> and the array of sensors <NUM>. The light can go past the lens, bounces off the array of sensors <NUM>, bounces off the lens, and hits the array of sensors <NUM> on a different pixel. In <FIG>, it can be seen that light reflecting off a nearby object <NUM> (e.g., an object outside of the field of view of the pixel) undergoes multiple bounces inside the lens <NUM>. In some depth imagers, careful optical design (e.g., using a set of lenses and antireflective coating) could mitigate the issue, but it can be challenging to completely remove the effect.

In <FIG>, it can be seen that the light can reflect off a smudge <NUM> on a screen <NUM>. The smudge <NUM> can be an oil film or residue left on a screen <NUM> that is protecting the lens. This scenario can be common for handheld devices having a depth imager. The smudge <NUM> is just one example of undesirable material on screen <NUM>. Other undesirable materials which can cause internal reflections can include dust and condensation. For instance, when there is dust close to the sensor, due to blurring of nearby objects, the dust can cause big disks to appear in the depth image (e.g., a system may interpret that there are big disks in the scene).

It is advantageous to ensure that the corrupting light from internal reflections does not affect the depth estimates.

As discussed previously the internal reflection problem is most serious when there are objects close to the sensor. Such nearby objects are closer than a distance DIR (IR stands for internal reflection). DIR can be on the order of centimeters. Typically, the minimum distance Dmin of the operational range of the depth imager is larger than DIR.

One technique for making sure that the corrupting light does not affect the depth calculation is to separately measure the corrupting light and removing its impact or contribution to the depth estimation. <FIG> illustrates a technique for isolating and measuring the corrupting light, according to some embodiments of the disclosure. For instance, it is possible to make the width or duration of the emitted light TL short enough so that the light from the corrupting object outside of the field of view of the pixel and reflected light from the object or objects within the field of view of the pixel are separated in time. A charge storage unit of a pixel can then be controlled to sense and collect light during the time period when the charge storage unit is expected to capture BG light and the corrupting light, and no reflected light from object(s) within the field of view of the pixel. Once the measurement of BG light and the corrupting light is made, it is possible to apply an appropriate gain factor and subtract the measurement of BG light and the corrupting light from other measurements that are affected by the corrupting light.

The emitted light (e.g., plot <NUM>) by light source <NUM> is represented by fTC(t). TC is the amount of time the light source <NUM> is on, and TC is short enough to temporally separate the return light reflecting off object or objects within the field of view of the pixel and the corrupting light. Return light reflecting off object (e.g., plot <NUM>) within the field of view of the pixel is represented by aR fTC(t - tR), where aR is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tR is the amount of time for the light to travel to and from the object or objects within the field of view. Corrupting light due to internal reflections (e.g., plot <NUM>) is represented by aB fTC(t - tB), where aB is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tB is the amount of time for the light return to the pixel after one or more internal reflection(s). One can see that the return light from the object and the corrupting light do not overlap in time. Light arriving at the pixel (e.g., plot <NUM>) is represented as aR fTC(t - tR) + aB fTC(t - tB) + b, which is a combination of the return light reflecting off object, the corrupting light, and BG light (represented by b). A charge storage unit can be controlled by sensor signal (e.g., plot <NUM>), to start sensing light at -tc and for a duration of TS. The sensor signal is represented by gTS(t + tC). The measurement by the charge storage unit is F = M(TC, -tC, TS). In one case, TS = tC + tmin, and accordingly the measurement is M(TC, -tC, tC + tmin). In another case, tC < TS < tC + tmin but TS is chosen large enough to collect all the light due to internal reflections, assuming some maximum distance between nearby objects leading to corrupting light and the sensor. If DC,max is the maximum distance between the sensor and objects leading to corrupting light, then TS may be chosen to be larger than <MAT>. The collected signal (e.g., plot <NUM>), which is a result of the corrupting light and BG light, is represented by (aR fTC(t - tR) + aB fTC(t - tB) + b) gTS(t + tc). The function (aR fTC(t - tR) + aB fTC(t - tB) + b) gTS(t + tc) is a combination of the function representing light arriving at the pixel and the function representing the sensor signal. Accordingly, the measurement representing a number of collected electrons is the integral of the collected signal: ∫(aR fTC(t - tR) + aB fTC(t - tB) + b) gTS(t + tc)dt. This measurement (which is referred herein as F and can be denoted as M(TC, -tC, TS)) can include a contribution from BG light b (referred herein as BG), a contribution from corrupting light aB fTC(t - tB) due to internal reflections (referred herein as corrupting light C), and a contribution ∫ aR fTC(t - tR) gTS(t + tc)dt from the return light reflecting off the object or objects within the field of view of the pixel, which may be zero or small if the sensor signal shuts off before light returning from the object or objects within the field of view of the pixel starts to hit the pixel. The contribution to the measurement solely from the corrupting light is equal to ∫ aB fTC(t - tB) gTS(t + tc)dt. If the contribution from the object or objects within the field of view can be ignored, then the measurement becomes of the form F = C + BG, where BG = ∫ b gTS(t + tc)dt is the contribution from the background light. If a measurement for BG light is made separately, it is possible to subtract or remove the contribution from BG light from the measurement F (e.g., F - BG = C) to obtain corrupting light C.

A given measurement M, depending on its parameters, can have a certain contribution due to corrupting light: αFC (the contribution is C multiplied by a gain factor αF). Fortunately, the impact/contribution from the corrupting light αFC can be removed by subtracting the measurement M by αFC. C can be derived by the technique illustrated in <FIG>, e.g., through making measurement F and removing the contribution from BG light from F. The gain factor αF for corrupting light can be determined a priori by knowing the shapes of the light pulse and the sensor signal, and can depend weakly on distance, or the distance to the corrupting object may be known a priori as in case of smudge on a screen at a fixed distance from the sensor. As a result, the quantity M - αFC would be substantially independent from the corrupting light.

Determining the gain factor αF corresponding to the corrupting light C is not trivial. For example, if the light source pulse and the sensor signal pulse are perfectly square, then the amount of corrupting light in a given measurement M would be proportional to the amount of time that M is collecting corrupting light. For example if M(TL, <NUM>, TS) and TL < TS then M(TL, <NUM>, TS) can collect corrupting light from time <MAT> to TL where DC is the distance between the corrupting object and the sensor. If <MAT> is negligible compared to TL, then the amount of corrupting light in a given measurement M is approximately proportional to TL.

Referring back to <FIG>, the amount of time corrupted light is being captured by C is TC (which is also the duration of the emitted light used for the measurement illustrated in <FIG>, M(TC, -tC, TS)). Therefore, if the gain factor αF is set to a ratio of the duration of the emitted light used for a given measurement M(TL, <NUM>, TS) and a duration of the emitted light used in the measurement M(TC, -tC, TS) illustrated by <FIG>, <MAT>, then the quantity M - αFC would be a quantity in sensitive to corrupting light, if the pulses were perfectly square. The ratio effectively finds a gain factor αF that accounts for the contribution from the corrupting light captured in a given measurement M relative to the amount of corrupting light captured in C. The gain factor αF is thus a multiplier multiplying the amount of corrupting light captured in C, and the result of the multiplication yields an approximation of the amount of corrupting light captured in a given measurement M(TL, <NUM>, TS). If the pulses are not perfectly square the quantity M - αFC may, to some satisfactory level of accuracy, still be approximately insensitive to corrupting light, for an appropriately chosen gain factor αF. If fC,M is the amount of corrupting light collected by a given measurement M, and fC is the amount of corrupting light collected by F (as illustrated by the scheme seen in <FIG>), then choosing αF equal to the ratio <MAT> would lead to M - αFC being insensitive to corrupting light.

It is possible to use the measurement illustrated in <FIG> and derivation for C to remove the impact of corrupting light (e.g., M - αFC) from the measurements illustrated by <FIG>. Referring back to the example illustrated by <FIG>, S<NUM> is M(TL, t<NUM>, TS), S<NUM> is M(TL, t<NUM>, TS), and S<NUM> is M(TL, -t<NUM>, TS). S<NUM> is likely to be the measurement most affected by corrupting light. The gain factor αF can be determined, and C can be derived using the example illustrated in <FIG>. S<NUM> can be subtracted by the αFC to make S<NUM> insensitive to corrupting light.

If there is no BG light (BG = <NUM>), then the measurement F includes the corrupting light due to internal reflections only (e.g., F = C + BG = C + <NUM> = C). S<NUM> can be adjusted by computing <MAT>. The measurement by S<NUM> is subtracted by F = C multiplied by a gain factor αF for the corrupting light. The resulting quantity <MAT> can be insensitive to corrupting light. Measurement by S<NUM> can be less affected by corrupting light, and adjustment for the corrupting light may not be necessary to reach a satisfactory degree of accuracy. When the measurements are applied to equation (<NUM>) or equation (<NUM>), <MAT> replaces S<NUM>. The equation can be applied in the same manner without further adjustments, if there is no BG light.

The depth estimation calculation can be different, if there is BG light (BG ≠ <NUM>). When there is BG light, then measurement F is a combination of corrupting light C and BG light BG, e.g., F = C + BG. To obtain a measurement Si that is insensitive to corrupting light C and BG light, the measurement Si can remove contribution from the corrupting light αFC and BG light S<NUM> through subtraction: Si - αF,iC - S<NUM>. This setup ensures that the corrupting light due to internal reflections is removed once (through -αF,iC) and the BG light is removed once (through -S<NUM>). Plugging in equation C = F - BG, and assuming BG = S<NUM>, Si - αF,iC - S<NUM> can be rewritten as Si - αF,iC - S<NUM> = Si - αF,i(F - BG) - S<NUM> = Si - αF,i(F - S<NUM>) - S<NUM> = Si - αF,iF + αF,iS<NUM> - S<NUM> = Si - αF,iF + (αF,i - <NUM>)S<NUM>. Equation (<NUM>) with S<NUM> adjusted for corrupting light and BG light can become the following, where S<NUM> - αF,<NUM>F + (αF,<NUM> - <NUM>)S<NUM> replaces S<NUM> -S<NUM>: <MAT>.

This adjusted ratio <MAT> for depth calculation can grow with depth, and can be provided as input to a lookup table implementing H(x) to obtain an actual depth D.

More generally, for different depth estimation equations and techniques for measuring light, some contribution by the corrupting light measurement αF,iF can be subtracted from a measurement made by a given charge storage unit Si, as seen in Si - αF,iF + (αF,i - <NUM>)S<NUM>. The adjusted measurement <MAT> is thus equal to Si - αF,iF + (αF,i - <NUM>)S<NUM>, where the corrupting light measurement F multiplied by a gain factor αF,i is subtracted from the original measurement Si and background measurement S<NUM> multiplied by a gain factor (αF,i - <NUM>) is added to the original measurement Si. A specific gain factor αF,i for the corrupting light corresponding to the measurement can be determined and used. For example, it can be determined that the amount of corrupting light in Si is equal to fc,i and the amount of corrupting light in F is equal to fC, in which case <MAT>. For measurement S<NUM>, the gain factor αF,<NUM>, can thus be a ratio between the amount of corrupting light captured in S<NUM> and an amount of corrupting light captured in measurement F. For measurement S<NUM>, the gain factor αF,<NUM>, can thus be a ratio between the amount of corrupting light captured in S<NUM> and an amount of corrupting light captured in measurement F. When the measurements are applied to equation (<NUM>) or equation (<NUM>), <MAT> replaces Si - S<NUM> for measurements which can be impacted by a component of the corrupting light. Measurement <MAT> is uncorrupted by the corrupting light and BG light and can be used directly in a depth estimation equation, such as equation (<NUM>).

To compute the depth equation seen in equation <NUM>, it is possible to utilize one charge storage unit for S<NUM>, one charge storage unit for S<NUM>, one charge storage unit for S<NUM>, and one charge storage unit for F. In other words, the technique illustrated by <FIG> in combination with the measurements seen in <FIG> may utilize one additional charge storage unit to make the corrupting light measurement F.

In some embodiments, both the measurement S<NUM> and the measurement S<NUM> are impacted by some amount of corrupting light. Accordingly, both the measurement S<NUM> and the measurement S<NUM> are adjusted for both corrupting light and background light. In other words, in equation (<NUM>), <MAT> replaces S<NUM> - S<NUM>, and <MAT> replaces S<NUM> - S<NUM>.

In some embodiments, the start time of the measurement S<NUM>, i.e., t<NUM>, is set to begin collecting charge when no more corrupting light is hitting the pixel (or is expected to hit the pixel). Accordingly, the measurement S<NUM> is impacted by corrupting light and background light, and the measurement S<NUM> is impacted by background light and not by corrupting light. Accordingly, the measurement S<NUM> is adjusted for both corrupting light and background light, and the measurement S<NUM> is adjusted only for background light. In other words, in equation (<NUM>), <MAT> replaces S<NUM> - S<NUM>, and S<NUM> - S<NUM> stays as S<NUM> - S<NUM>. Note that the gain factor αF,<NUM> in <MAT>, which is a ratio between the amount of corrupting light captured in S<NUM> and an amount of corrupting light captured in measurement F, is zero if no amount of corrupting light is captured in S<NUM>, which effectively makes <MAT>.

In some cases, it is possible to avoid the need to have an additional charge storage unit. Note that [Si - αF,iF + (αF,i - <NUM>)S<NUM>] (a quantity seen in equation (<NUM>)) can be rewritten as (Si + (αF,i - <NUM>)S<NUM>) - αF,iF. Assuming αF,i - <NUM> > <NUM>, it is possible to perform the measurement (Si + (αF,i - <NUM>)S<NUM>) in hardware. As mentioned previously, the charge storage units in a pixel can be controlled to make and accumulate multiple measurements performed over time. In some cases, the measurements can be repeated based on different light pulse widths of the light source <NUM> and/or different sensor signal pulse widths/durations using a same charge storage unit. Specifically, the same charge storage unit can perform the addition of measurements, by letting the same charge storage unit continue to collect and accumulate electrons (e.g., an amount of charge) for all the measurements, and not reset the charge storage unit after each measurement).

A same charge storage unit can be used to make a measurement corresponding to Si and the measurement is stored in the charge storage unit. The same charge storage unit can make the BG measurement S<NUM>, αF,i - <NUM> number of times, and add/accumulate the αF,i - <NUM> number of BG measurements in the same charge storage unit. αF,i - <NUM> is equal to the gain factor αF,i for the corrupting light measurement corresponding to a given measurement Si minus <NUM>. The measurements can be made in any suitable order. Effectively, a single charge storage unit can accumulate charge that correspond to (Si + (αF,i - <NUM>)S<NUM>). If αF,i - <NUM> is not an integer, it is possible to change the pulse width (or duration) of the BG measurements S<NUM> to produce a αF,i - <NUM> multiple of a single BG measurement S<NUM>. The pulse width can be pre-determined and programmed based on the gain factor αF,i. By manipulating the sensor signal for the same charge storage unit to perform and accumulate the measurement Si done once and the BG measurement S<NUM> done αF,i - <NUM> number of times, the same charge storage unit can make the measurement equivalent to (Si + (αF,i - <NUM>)S<NUM>). A separate charge storage unit is no longer needed for separately making the BG measurement S<NUM>. This frees up a "free" charge storage unit (or eliminates the need for an additional charge storage unit) that can be used to isolate and measure the corrupting light F as illustrated by <FIG>.

Using various techniques described herein, a measurement S<NUM> can be adjusted for corrupting light and BG light. Moreover, a pixel can make measurement S<NUM> (or any measurement Si that is impacted by corrupting light), a BG measurement S<NUM>, and a corrupting light measurement F, by utilizing one charge storage unit for S<NUM> and S<NUM>, and one charge storage unit for F.

Applying the same techniques, a measurement S<NUM> can also be adjusted for corrupting light and BG light. Specifically, the quantity S<NUM> - αF,<NUM>F + (αF,<NUM> - <NUM>)S<NUM> can be free of corrupting light and BG light. Note that the gain factor αF,<NUM>, in this case, can be a ratio between the amount of corrupting light captured in S<NUM> and an amount of corrupting light captured in measurement F. The need to have three separate charge storage units for S<NUM>, S<NUM> and F is obviated, since S<NUM> + (αF,<NUM> - <NUM>)S<NUM> can be performed in hardware (i.e., using the same charge storage unit). Accordingly, the pixel can make measurement S<NUM> (which can also be impacted by corrupting light), a BG measurement S<NUM>, and a corrupting light measurement F, by utilizing one charge storage unit for S<NUM> and S<NUM>, and one charge storage unit for F.

Equation (<NUM>) can be modified as follows, to ensure that both measurement S<NUM> and measurement S<NUM> can be insensitive to corrupting light and BG light: <MAT>.

The result is a pixel which can account for corrupting light and BG light with three charge storage units: one charge storage unit for [S<NUM> + (αF,<NUM> - <NUM>)S<NUM>], one charge storage unit for [S<NUM> + (αF,<NUM> - <NUM>)S<NUM>], and one charge storage unit for F. The three charge storage units are sufficient to measure the quantities for computing depth based on equation <NUM>. The result is a pixel which can advantageously remove the contributions from corrupting light and BG light, and enable more accurate depth estimates to be computed.

In the technique illustrated by <FIG>, the temporal separation of the corrupting light due to internal reflections and the return light from the object or objects within the field of view makes use of a short pulse of duration TC. Suppose the corrupting objects are within <NUM> of the depth imager, and the goal is to measure depth of objects that are more than <NUM> away. To achieve temporal separation of the corrupting light due to an object less than <NUM> away and the return light reflecting off objects more than <NUM> away, TC has to be short enough to make sure that the corrupting light returns to the pixel before any light returns from an object at <NUM> away. This means that the width of the light pulse TC has to satisfy <MAT>, which gives TC∼<NUM>ns. Some drivers for driving light source <NUM> or the light source <NUM> itself cannot achieve a nanosecond width pulse of emitted light. Even if the driver can generate a short nanosecond pulse, it is possible that the output power of the emitted light is too low at that short light pulse width to get much signal back.

To address this issue, the charge storage units can be controlled in such a way to still isolate and measure the corrupting light. <FIG> illustrates another technique for isolating and measuring the corrupting light, according to some embodiments of the disclosure.

The emitted light (e.g., plot <NUM>) by light source <NUM> is represented by fTL(t). TL is the amount of time the light source <NUM> is on, and TL does not have to be short enough to temporally separate the return light reflecting off object or objects within the field of view of the pixel and the corrupting light. Return light reflecting off object (e.g., plot <NUM>) within the field of view of the pixel is represented by aR fTL(t - tR), where aR is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tR is the amount of time for the light to travel to and from the object or objects within the field of view. Corrupting light due to internal reflections (e.g., plot <NUM>) is represented by aB fTL(t - tB), where aB is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tB is the amount of time for the light return to the pixel after one or more internal reflection(s). One can see that the return light from object and the corrupting light overlap in time (both can hit the pixel during a period of time). Light arriving at the pixel (e.g., plot <NUM>) is represented as aR fTL(t - tR) + aB fTL(t - tB) + b, which is a combination of the return light reflecting off object, the corrupting light, and BG light (represented by b). A charge storage unit can be controlled by sensor signal (e.g., plot <NUM>), to start sensing light at -t<NUM> and stops sensing light at a delay time td. The charge storage unit thus sense for a duration of TS = t<NUM> + td. The sensor signal is represented by gTS(t + t<NUM>). The measurement by the charge storage unit is F = M(TL, -t<NUM>, TS). The collected signal (e.g., plot <NUM>), which is a result of the corrupting light and BG light, is represented by (aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t + t<NUM>). The function (aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t + t<NUM>) is a combination of the function representing light arriving at the pixel and the function representing the sensor signal. Accordingly, the measurement representing a number of collected electrons is the integral of the collected signal: ∫(aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t + t<NUM>)dt. This measurement includes a contribution from BG light and a portion of corrupting light due to internal reflections. Delay time td for turning off the sensor can be selected such that the pixel is only collecting light reflected off nearby (corrupting) objects and BG light.

As illustrated by the collected signal in <FIG>, the charge storage unit stops sensing at td, where td is after the time tB the corrupting light starts to hit the pixel and before the time tR that the return light reflecting off object hits the pixel. The result is equivalent to having sent out a light pulse of width td (referred to as the effective pulse width) or shorter because no return light reflecting off the object in the field of view of the pixel is captured, and the corrupting light is effectively isolated and measured by turning off the sensor at td. By using an appropriate gain factor, the measurement seen in <FIG> can be used to remove the corrupting light from a measurement affected by the corrupting light.

This technique is effective if the corrupting object is known, e.g., at some fixed distance. For instance, the corrupting object could be a screen <NUM> at some fixed distance from the pixel. In another instance, the corrupting object can be at a distance short enough that it can be approximated to be zero distance. Otherwise, the effective pulse width captured from the corrupting object or objects would depend on the distance to the corrupting object.

For example, if the corrupting object is at a distance DC, and the emitted light pulse is active for a time interval TL, then the return light from the corrupting object would be active from time 2DC/c to TL + <NUM>DC/c. If the sensor is active from time - t<NUM> to td with <MAT>, then the sensor would be collecting light between times <MAT> and td. Therefore the time interval during which the sensor collects light has a width of td - <NUM>DC/c. This quantity depends on the distance Dc between sensor and corrupting object. If DC is negligible compared to td, as could be the case for corrupting objects very near the camera compared to c td/<NUM>, then this quantity would be only weakly dependent on <NUM>DC. Another case would be a corrupting object such as a screen where DC is constant, therefore there is no variation of td - <NUM>DC/c with DC since DC is constant. In both these cases, the effective pulse width can be determined a priori. If the quantity DC varies to the point where the amount of corrupting light depends strongly on DC, then it may be the case that a gain factor for removing the corrupting light cannot be determined a priori.

When the distance to the corrupting object can vary, a different technique can be used to effectively achieve short light pulse without having to generate one. <FIG> illustrates a technique for effectively getting arbitrarily short pulses, according to some embodiments of the disclosure. The technique involves two light pulses of width TL of the same shape but shifted in time by some delay time td. The first emitted light (e.g., plot <NUM>) by light source <NUM> is represented by fTL (t). The first emitted light starts/begins at t = <NUM>, and has a duration of TL. In this example, the first emitted light has a first start time at t = <NUM> (or some suitable time Tstart within the frame of reference). The second emitted light (e.g., plot <NUM>) by light source <NUM> is represented by fTL(t - td). The second emitted light starts/begins at t = td, and has a duration of TL. Note that td « TL, or in other words, the duration TL is significantly greater than td. The second emitted light pulse has a same pulse shape as the first emitted light pulse, and has the same duration as the first emitted light pulse. The second emitted light pulse has a second start time that offset or delayed by a pre-determined amount of time td relative to the first start time. If the first start time is at t = Tstart, then the second start time is at t = Tstart + tq. When the second emitted light is subtracted from the first emitted light, the difference (e.g., plot <NUM>) would appear as a short positive pulse <NUM> of width td, followed by a short negative pulse <NUM> of width td, after a time TL - td of having minimal signal. The light output power is minimal between the positive pulse <NUM> and the negative pulse <NUM>. A charge storage unit can be controlled by sensor signal (e.g., plot <NUM>), to start sensing light at -t<NUM> and for a duration of TS. The sensor signal is represented by gTS(t + t<NUM>). It is possible to make two measurements: (<NUM>) a first measurement with the first emitted light and the sensor signal starting to sense at -t<NUM> and turning off at some time before the negative pulse <NUM>, and (<NUM>) a second measurement with the second emitted light and the sensor signal start sensing at -t<NUM> and turning off at some time before the negative pulse <NUM>. The sensor signal used for the first measurement has the same start time, duration, and shape as the sensor signal used for the second measurement. The sensor signals can turn on at or sometime before the start time of the first emitted light pulse (e.g., t = <NUM>), and turn off sometime before TL or TL - td, and after td (or after the start time of the second emitted light pulse). The first and second measurements defined by -t<NUM> and TS of the sensor signal can be determined based on TL and td. Subtracting the first measurement by the second measurement would yield an equivalent result to what would be obtained by exposing the scene with a short light pulse width td. Phrased differently, it is possible to obtain a measurement as if a short light pulse of width td was used by (<NUM>) performing two measurements with the same sensor signal (e.g., plot <NUM>) with light pulses which are slightly delayed with respect to each other, and (<NUM>) obtaining a difference of the two measurements. Using this technique, arbitrarily small light pulses can be simulated, even if the driver or the light source cannot output short light pulses directly. This resulting measurement can thus capture corrupting light and BG light, and is referred to below as the resulting corrupting light measurement.

This simulation technique may use two charge storage units to obtain the first and the second measurement, which may require an additional charge storage unit for the pixel. However, one can remove the need to include an additional charge storage unit by recognizing that part of the subtraction can be reformulated as addition. A given charge storage unit can be operated with a proper sensor signal such that the charge storage unit can be reused to make multiple measurements and implement addition/accumulation of the measurements. Suppose the first measurement is denoted as F<NUM> and the second measurement is denoted as F<NUM>. The resulting corrupting light measurement after subtraction, each capturing corrupting light and BG light, is denoted as F<NUM> - F<NUM>. Because both the first measurement F<NUM> and the second measurement F<NUM> include a component of BG light, the subtraction of F<NUM> by F<NUM> removes/cancels out the common component of BG light, leaving just the component of corrupting light. To ensure that the depth estimation can be tolerant to corrupting light (i.e., F<NUM> - F<NUM>) and BG light (i.e., S<NUM>), it is possible to remove or adjust for the corrupting light and BG light by subtracting a measurement Si by the resulting corrupting light measurement F<NUM> - F<NUM> multiplied by a gain factor αF,i and by a BG light measurement S<NUM>, that is: <MAT> αF,i(F<NUM> - F<NUM>) - S<NUM>. A depth D for the pixel can be obtained by computing the following exemplary equation: <MAT>.

This ratio <MAT> for depth calculation can grow with depth, and can be provided as input to a look up table implementing H(x) to obtain an actual depth D.

Note that <MAT> can be rewritten as <MAT> (αF,iF<NUM> + S<NUM>). Equation <NUM> can be rewritten as follows: <MAT>.

With this rewritten quantity, <MAT>), it is possible to use a first charge storage unit to obtain (Si + αF,iF<NUM>) and second charge storage unit to obtain (αF,iF<NUM> + S<NUM>). Specifically, to form a depth estimate that is tolerant to corrupting light (i.e., F<NUM> - F<NUM>) and BG light (i.e., S<NUM>), the depth imager can use a total of four charge storage units to obtain the quantities used in equation (<NUM>). A first charge storage unit can obtain (S<NUM> + αF,<NUM>F<NUM>). A second charge storage unit can obtain (αF,<NUM>F<NUM> + S<NUM>). A third charge storage unit can obtain (S<NUM> + αF,<NUM>F<NUM>). A fourth charge storage unit can obtain (αF,<NUM>F<NUM> +S<NUM>).

The quantity Si + αF,iF<NUM> can be stored in the charge storage unit that was originally used for Si. Accordingly, Si + αF,iF<NUM> can be implemented by controlling a first charge storage unit for measuring Si in a suitable manner to make and add the αF,iF<NUM> measurement directly in the same charge storage unit. αF,iF<NUM> can be added directly to the same charge storage unit in hardware. The same charge storage unit can be used to make a measurement corresponding to Si and the measurement is stored in the charge storage unit. The same charge storage unit can make the second measurement F<NUM>, αF,i number of times, and add/accumulate the αF,i number of measurement F<NUM> in the same charge storage unit. In this case, the gain factor αF,i can be a ratio between the amount of corrupting light captured in Si and an amount of corrupting light captured in measurement F<NUM> - F<NUM>. Effectively, a single charge storage unit can make the measurement Si + αF,iF<NUM>. If αF,i is not an integer, it is possible to change the pulse width (or duration) of the second measurement F<NUM> to produce a integer multiple of a single second measurement F<NUM>. An exemplary approach would be to simply approximate αF,i to the nearest integer. The pulse width, associated with the gain factor αF,i can be pre-determined and programmed by determining the amount of corrupting light captured in Si, the amount of corrupting light captured in F<NUM> - F<NUM>, and taking ratio of the two amounts. Manipulating the sensor signal for the same charge storage unit to perform and accumulate the measurement Si done once and the second measurement F<NUM> done αF,i number of times, the same charge storage unit can make the measurement equivalent to Si + αF,iF<NUM>. A separate charge storage unit is no longer needed for separately making the corrupting light measurement F<NUM>.

The quantity αF,iF<NUM> + S<NUM> can be stored in the charge storage unit that was originally used for S<NUM>, once again by altering the pulse width of the F<NUM> measurement and repeating its measurement an integer number of times such that its contribution being accumulated in the charge storage unit is approximately αF,iF<NUM>. Accordingly, αF,iF<NUM> + S<NUM> can be implemented by controlling a second charge storage unit for measuring S<NUM> in a suitable manner to make and add the αF,iF<NUM> measurement directly in the same charge storage unit. αF,iF<NUM> can be added directly to the same charge storage unit in hardware. The same charge storage unit can be used to make a measurement corresponding to S<NUM> and the measurement is stored in the charge storage unit. The same charge storage unit can make the second measurement F<NUM>, αF,i number of times, and add/accumulate the αF,i number of measurement F<NUM> in the same charge storage unit. In this case, the gain factor αF,i can be a ratio between the amount of corrupting light captured in Si and an amount of corrupting light captured in measurement F<NUM> - F<NUM>. Effectively, a single charge storage unit can make the measurement αF,iF<NUM> + S<NUM>. If αF,i is not an integer, it is possible to change the pulse width (or duration) of the first measurement F<NUM> to produce a integer multiple of a single first measurement F<NUM>. An exemplary approach would be to simply approximate αF,i to the nearest integer. The pulse width, associated with the gain factor αF,i can be pre-determined and programmed by determining the amount of corrupting light captured in Si, the amount of corrupting light captured in F<NUM> - F<NUM>, and taking ratio of the two amounts. Manipulating the sensor signal for the same charge storage unit to perform and accumulate the measurement S<NUM> done once and the first measurement F<NUM> done αF,i number of times, the same charge storage unit can make the measurement equivalent to αF,iF<NUM> + S<NUM>. A separate charge storage unit is no longer needed for separately making the corrupting light measurement F<NUM>.

In some embodiments, the measurement S<NUM> has some amount of corrupting light or is significantly affected by the corrupting light, and the measurement S<NUM> does not have any corrupting light, or is minimally affected by the corrupting light. For example, the measurement S<NUM> can begin to collect charge at t = t<NUM>, and the measurement S<NUM> can begin to collect charge much later, at t = t<NUM>. The start time of the measurement S<NUM> can be set to ensure that the measurement S<NUM> does not collect any corrupting light. The gain factor αF,<NUM>, seen in equations (<NUM>) and (<NUM>), is thus zero. If the measurement S<NUM> has no contribution from corrupting light, then equation (<NUM>) and (<NUM>) can be formulated as follows: <MAT>.

Equation (<NUM>) can be rewritten as: <MAT>.

With this rewritten equation, to form a depth estimate where the measurement S<NUM> is tolerant to corrupting light (i.e., F<NUM> - F<NUM>) and BG light (i.e., S<NUM>), and the measurement S<NUM> is tolerant to BG light, the depth imager can use a total of three charge storage units to obtain the quantities used in equation (<NUM>). A first charge storage unit can obtain (S<NUM> + αF,<NUM>F<NUM>). A second charge storage unit can obtain (S<NUM> + αF,<NUM>F<NUM>). A third charge storage unit can obtain (S<NUM> + αF,<NUM>F<NUM>).

As explained herein, it is possible to adjust measurements by measuring the corrupting light through the use of short light pulses as seen in <FIG> or through the techniques illustrated by <FIG> and <FIG>. Given two measurements S<NUM> and S<NUM>, where each measurement has collected some amount of corrupting light C<NUM> and C<NUM>, respectively, the quantity <MAT> can be free of corrupting light, as the amount of corrupting light in the first term is C<NUM>, and the amount of corrupting light in the second term is <MAT>, thus they cancel out. The quantity is equal to the measurement S<NUM> subtracted by the measurement S<NUM> multiplied by a ratio of the corrupting light C<NUM> and the corrupting light C<NUM>. The ratio is between an amount of corrupting light collected in S<NUM> and an amount of corrupting light collected in S<NUM>. With the appropriate ratio, the contribution from corrupting light can be removed from the measurement S<NUM> by subtracting <MAT> from S<NUM>. Based on the shape pulses, it is possible to know or estimate corrupting light C<NUM> and C<NUM> a priori, thus the ratio can be determined a priori. The amount of corrupting light captured can, in some cases, depend on the amount of time that both the sensor signal and return corrupting light are on at the same time. The ratio of corrupting light captured in the two measurements S<NUM> and S<NUM> can thus be used as the gain factor in many of the operations described herein.

For example, if S<NUM> corresponds to the measurement M(TL<NUM>, t<NUM>, TS<NUM>), then the sensor is on from time t<NUM> to t<NUM> + TS<NUM>. If the corrupting object is at distance Dc, the return corrupting light is on from times <MAT> to <MAT>. If, for example <MAT> <MAT>, then the sensor and return corrupting light are both on from times <MAT> to t<NUM> + Ts<NUM>. If, furthermore, S<NUM> corresponds to the measurement M(TL<NUM>, t<NUM>, TS<NUM>), and the corrupting object is as before at distance Dc, then the return corrupting light is on from times <MAT> to <MAT>. If, for example <MAT>, then the sensor and corrupting light are both on from time t<NUM> to <MAT>.

If, all the pulses are square, then the amount of corrupting light captured in S<NUM> is proportional to the time amount of time <MAT> that both the sensor and return corrupting light are active during the S<NUM> measurement, while the amount of corrupting light in S<NUM> is proportional to the amount of time <MAT> that both the corrupting light and sensor are active during the S<NUM> measurement. In that case, <MAT> is a quantity where the corrupting light has been removed.

If, the pulses are not exactly square but approximately square, then this equation may still be approximately insensitive to corrupting light. If the pulses deviate significantly from square, then the amount of corrupting light will be a function of the shape of the pulses. By determining the shape of the pulses, or by empirically adding a corrupting object in the scene, performing a measurement, then removing the corrupting object in the scene and performing the same measurement, and subtracting the first measurement from the second measurement, one obtains purely the corrupting light. By performing this scheme for the S<NUM> measurement one obtains C<NUM>, and by performing this scheme for S<NUM> one obtains C<NUM>. Then the gain factor or ratio <MAT> can be stored. For example, if the distance to the corrupting object is known a priori this scheme for obtaining gain factor α is carried out with the corrupting object at the known distance to the corrupting object.

If the effect of corrupting light is to be removed for corrupting objects in a pre-determined interval of distances then α can be obtained for different distances to corrupting objects within the pre-determined interval of distances, and it can be verified whether α varies a small enough amount in the interval, in which case, one would choose for α a value that is close to the values obtained in the interval. Once gain factor α is pre-determined, and new S<NUM> and S<NUM> measurements are performed, then the quantity S<NUM> - αS<NUM> can be obtained, which will be either fully or approximately free of corrupting light.

Accordingly, it is possible to form quantities which are uncorrupted and use the uncorrupted quantities for uncorrupted depth estimates. If it is possible additional charge storage unit(s) can be used to store the corrupted light. Alternatively, whenever there is a quantity in the depth estimation equation of the form A - B + C - D + ···, the depth estimation equation can be rewritten as (A + C + ···) - (B + D + ···), and the summations can be performed directly in hardware (i.e., by reusing the same charge storage unit to make multiple measurements). As a result, additional charge storage units to make and store the corrupting light measurement are no longer needed.

One other possible way of making sure that the depth images are not corrupted by internal scattering or diffraction is to turn on sensor(s) past the point where the last corrupting light (due to internal scattering) is hitting the sensor, or at a point where the last corrupting light hitting the sensor is negligible. This methodology avoids the corrupting light altogether, by design, and the resulting depth estimate computed from the sensor measurements would be uncorrupted.

<FIG> illustrates turning on a sensor (or starting to collect light) after the light pulse is off, according to some embodiments of the disclosure. The emitted light (e.g., plot <NUM>) by light source <NUM> is represented by fTL(t). TL is the amount of time the light source <NUM> is on. Return light reflecting off an object (e.g., plot <NUM>) or objects within the field of view of the pixel is represented by aR fTL(t - tR), where aR is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tR is the amount of time for the light to travel to and from the object or objects within the field of view. Corrupting light due to internal reflections (e.g., plot <NUM>) is represented by aB fTL(t - tB), where aB is a gain factor to account for the reduction in light returning to the pixel (e.g., reduction in amplitude) and tB is the amount of time for the light return to the pixel after one or more internal reflection(s). Light arriving at the pixel (e.g., plot <NUM>) is represented as aR fTL(t - tR) + aB fTL(t - tB) + b, which is a combination of the return light reflecting off object, the corrupting light, and BG light (represented by b). A charge storage unit can be controlled by a sensor signal (e.g., plot <NUM>), to start sensing light at t<NUM> and for a duration of TS. The sensor signal is represented by gTS(t - t<NUM>). The measurement by the charge storage unit is M(TL, t<NUM>, TS). t<NUM> is preferably greater than TL, meaning that the sensor starts collecting light after the light pulse is off, or smaller than TL but large enough so that the amount of corrupting light received is negligible within some tolerance. The collected signal (e.g., plot <NUM>), is represented by (aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t - t<NUM>). The function (aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t + t<NUM>) is a combination of the function representing light arriving at the pixel and the function representing the sensor signal. Accordingly, the measurement representing a number of collected electrons is the integral of the collected signal: ∫(aR fTL(t - tR) + aB fTL(t - tB) + b) gTS(t + t<NUM>)dt. By design (i.e., selecting the appropriate t<NUM>), aR fTL(t - tR), i.e., the contribution from the corrupting light due to internal reflections, is zero or minimal during the period of time the sensor is on. As a result, the measurement avoids corrupting light (but would have some contribution from BG light).

If tB,max is the maximum travel time of a pulse going to a nearby corrupting object and returning to the sensor, a measurement S<NUM> = M(TL,t<NUM>, TS) = M(TL, tB,max, TS) would not collect any light from the nearby corrupting object, as long as t<NUM> ≥ TL + tB,max. The quantity tB,max can be determined to be either a time after light reflecting off object(s) with a known distance to sensor, such as a glass cover, returns to the sensor, or a time such that object(s) that contribute to corruption do so in a small enough amount so as not to significantly affect depth estimates. If the distance dB,max is the largest distance between the object(s) and sensor for which the object(s) can still lead to significant corruption, then setting tB,max to <NUM> dB,max/c or larger, where c is the speed of light, will lead to S<NUM> not having significant contribution from corrupting light. For example if objects further than <NUM> centimeters away from the sensor do not lead to significant corruption due to internal reflection, then setting tB,max ≥ <NUM> * <NUM> cm/c will lead to S<NUM> being approximately free of corruption. To make the measurement S<NUM>, a charge storage unit can start collecting light after the amount of time TL the light source <NUM> is on, plus the maximum travel time tB,max, and capture light for a duration of TS.

If tmin is the travel time for a pulse to travel to object(s) at the minimum distance Dmin at which the depth camera would begin to report depths (e.g., Dmin = <NUM>), another measurement S<NUM> can be made where t<NUM> = TL + tmin. That is, S<NUM> = M(TL, TL + tmin, TS). To make the measurement S<NUM>, a charge storage unit can start capturing light after the amount of time TL the light source <NUM> is on plus the travel time tmin, and capture light for a duration of TS. tB,max is shorter than tmin, which means that the charge storage unit making the measurement S<NUM> starts collecting light after the charge storage unit making the measurement S<NUM> starts collecting light. This also means that the measurement S<NUM> would also not collect any light from the nearby corrupting object.

A further measurement S<NUM> can be made to capture BG light. S<NUM> can collect BG light for a time TS with the light source <NUM> off. S<NUM> can collect light from times t = -t<NUM> to t = -t<NUM> + TS. The measurement being made by S<NUM> is M(TL, -t<NUM>, TS). In one case, t<NUM> = TS, and thus S<NUM> can collect light from times t = -t<NUM> = -TS to t = <NUM>. In another case, S<NUM> is M(<NUM>,<NUM>,TS),and the light source is kept off during the S<NUM> measurement, thus only capturing BG light. Generally speaking, the measurement S<NUM> collects BG light for a period of time where no light is received from the light source the light source <NUM>.

<FIG> illustrates measuring distance based on a technique which turns on a sensor (or starting to collect light) after the light pulse is off, according to some embodiments of the disclosure. By design, the three measurements S<NUM>, S<NUM>, and S<NUM> do not capture any light from nearby corrupting objects or corrupting light caused by internal reflections. Therefore, a depth estimate can be obtained that is insensitive to corrupting light based on these three measurements S<NUM>, S<NUM>, and S<NUM>. With the three measurements S<NUM>, S<NUM>, and S<NUM>, it is possible to obtain a depth estimate by designing and computing a quantity based on the three measurements S<NUM>, S<NUM>, and S<NUM> that grows with or is (positively) related to depth. Such a quantity can indicate depth. One exemplary quantity based on the three measurements S<NUM>, S<NUM>, and S<NUM> is (S<NUM> - S<NUM>)/(S<NUM> - S<NUM>). The quantity is a ratio of (<NUM>) the measurement S<NUM> minus the BG light measurement S<NUM> and (<NUM>) the measurement S<NUM> minus the measurement S<NUM>. To illustrate, <FIG> shows the case where the laser pulse and sensor pulses are perfectly square. Plot <NUM> illustrates the emitted light. Plot <NUM> illustrates return light reflecting off an object or objects within the field of view of the pixel and background light. Plot <NUM> illustrates corrupting light due to internal reflections. Plot <NUM> illustrates the sensor signal for making measurement S<NUM>. Plot <NUM> illustrates the sensor signal for making measurement S<NUM>. Plot <NUM> illustrates the sensor signal for making BG light measurement S<NUM>.

To illustrate measuring distance using the technique for turning on the sensor after the light pulse is off to avoid corrupting light, <FIG> shows how the quantity (S<NUM> - S<NUM>)/(S<NUM> - S<NUM>) is proportional to the depth. The return light from the light source (whose emitted light is shown as plot <NUM>) is on from times tR to tR + TL, with some amount of light per unit time aR arriving at the sensor. The background light arriving at the sensor per unit time is denoted by b. The combination of return light from the light source and background light is shown as plot <NUM>. The corrupting light shown as plot <NUM> arriving at the sensor is on from times tB to tB + TL, and the amount of corrupting light per unit time when the corrupting light is being received is denoted by aB. The sensor signal shown as plot <NUM> for S<NUM> is on from times TL + tB,max to TL + tB,max + TS. Therefore, denoting Ceff as the photoconversion efficiency of the sensor which, when multiplied by the number of photons hitting the sensor, gives the final output value of the sensor, S<NUM> = Ceff(aR(tR - tB,max) + bTS). aR(tR - tB,max) represents the amount of return light hitting the sensor during the time when the sensor is on. bTS represents the amount of BG light hitting the sensor during the time when the sensor is on. The sensor signal shown as plot <NUM> for S<NUM> is on from times TL + tmin to TL + tmin + TS. Therefore, S<NUM> = Ceff(aR(tR - tmin) + bTS). aR(tR - tmin) represents the amount of return light hitting the sensor during the time when the sensor is on. bTS represents the amount of BG light hitting the sensor during the time when the sensor is on. The sensor signal shown as plot <NUM> for S<NUM> is on for a time TS when no light is received from the light source, and therefore S<NUM> = Ceff b TS.

Based on these measurements, the amount of time for the light to return from the object can be derived based on tmin, tB,max, S<NUM>, S<NUM>, and S<NUM> according to the following equation. Moreover, the depth of the object of interest can then given by <MAT>.

Note that the numerator of the ratio, S<NUM> - S<NUM>, is a quantity that represents the measurement S<NUM> with the contribution from BG light subtracted out. The quantity S<NUM> or S<NUM> - S<NUM>, would get larger as the distance/depth of the object grows. Note that the denominator of the ratio, S<NUM> - S<NUM>, is a quantity that represents a difference between the measurement S<NUM> and the measurement S<NUM>. S<NUM> - S<NUM> = Ceff(aR(tR - tB,max) + bTS) - Ceff(aR(tR - tmin) + bTS) = Ceff (aR(-tB,max + tmin)). Note that this quantity is independent of tR, which means that this quantity is independent of the distance/depth of the object. Accordingly, the ratio <MAT> is positively related to the distance/depth of the object grows.

In the case where the pulses are not perfectly square, it is possible to measure the ratio (S<NUM> - S<NUM>)/(S<NUM> - S<NUM>) as a function of depth and store the values in a lookup table. The lookup table implements a function H(x) such that <MAT> and can convert the ratio into a depth estimate. In other words, the result from the lookup table based on the ratio (S<NUM> - S<NUM>)/(S<NUM> - S<NUM>) can output a corrected depth estimate. Values for H(x) can be extrapolated between the values of x for which H(x) is known. The depth equation becomes: <MAT>.

Generally speaking, the embodiments disclosed herein are applicable to depth imagers which suffer from the internal reflection problem. These depth imagers can be found in optical systems include time-of-flight, and range finding systems. Optical systems designed for determining depth, distance, and/or speed can also be found in many applications, including sports electronics, consumer electronics, medical equipment, aerospace/military equipment, automotive electronics, security systems, etc..

The present disclosure includes apparatuses which can include means to implement any one or more parts of the various techniques disclosed herein.

The embodiments described herein (e.g., illustrated in <FIG>) for addressing the internal reflection issues are distinguished from systems which record a corrupted depth image and performs image processing on the corrupted depth image to filter out the effects caused by internal reflection. The embodiments described herein are in contrast to those systems because a pixel itself is controlled in such a way isolate and measure the effect caused by internal reflections, and a depth calculation takes the corrupting light measurement into account by appropriately removing the effect. In particular, the scheme illustrated by <FIG> and <FIG> can be particularly good at getting rid of the "flying dust" problem: when dust flies close to the sensor the dust diffracts light, creating a big semi-transparent circle. Industrial applications can be particularly susceptible to the flying dust problem. The flying dust problem goes away completely with the scheme illustrated by <FIG> and <FIG>, because no light is captured in the measurements from nearby diffracting objects like dust.

The embodiments described herein (e.g., illustrated in <FIG>) can address the internal reflection problem and output uncorrupted depth estimates as close as centimeters away from the depth imager.

In some cases, the charge storage unit that makes a measurement F of corrupting light can be used as a corrupting light detector. The detector can provide useful information to the user, if sufficient corrupting light is collected by the pixels in the sensor. The depth imager can cause an output to be generated to notify the user that there is corrupting light, and ask the user to remove the cause for the internal reflections (e.g., smudge, finger, dust, condensation, etc.).

Moreover, the detector can modify the operating mode of the depth imager to account for corrupting light. For instance, if no corrupting light is detected, the depth imager can compute depth estimates based on any one or more exemplary embodiments accompanying equation (<NUM>) or (<NUM>). If corrupting light is detected, the depth imager can be configured to make measurements and compute depth estimates based on any one or more exemplary embodiments accompanying equations (<NUM>)-(<NUM>).

For detecting whether there is sufficient corrupting light, it is possible to sense corrupting light hitting any of the pixels based on the techniques described herein and sense all light hitting the pixels. If the ratio of the corrupting light and all light hitting the pixels is greater than a threshold, the corrupting light detector can output a signal indicating that there is sufficient corrupting light. If it is less than the threshold, there is insufficient corrupting light.

To sum all the corrupting light hitting an array of sensors, electrons from a group of charge storage units measuring the corrupting light can be collected (e.g., by columns) so that it is not necessary to read out each pixel. Similarly, to sum all the light hitting the pixels, electrons from a group of charge storage units measuring all the light can be collected (e.g., by columns) so that it is not necessary to read out each pixel.

It is also imperative to note that all of the specifications, dimensions, and relationships outlined herein (e.g., circuit components) have only been offered for purposes of example and teaching only. Such information may be varied considerably without departing from the spirit of the present disclosure, or the scope of the appended claims and/or examples. The specifications apply only to one non-limiting example and, accordingly, they should be construed as such. In the foregoing description, example embodiments have been described with reference to particular component arrangements. Various modifications and changes may be made to such embodiments without departing from the scope of the appended claims and/or examples. The description and drawings are, accordingly, to be regarded in an illustrative rather than in a restrictive sense.

Note that with the numerous examples provided herein, interaction may be described in terms of two, three, four, or more electrical components. However, this has been done for purposes of clarity and example only. It should be appreciated that the system can be consolidated in any suitable manner. Along similar design alternatives, any of the illustrated components of the FIGURES may be combined in various possible configurations, all of which are clearly within the broad scope of this Specification. In certain cases, it may be easier to describe one or more of the functionalities of a given set of flows by only referencing a limited number of electrical elements. It should be appreciated that the electrical circuits of the FIGURES and its teachings are readily scalable and can accommodate a large number of components, as well as more complicated/sophisticated arrangements and configurations. Accordingly, the examples provided should not limit the scope or inhibit the broad teachings of the electrical circuits as potentially applied to a myriad of other architectures.

Note that in this Specification, references to various features (e.g., elements, structures, modules, components, steps, operations, characteristics, etc.) included in "one embodiment", "example embodiment", "an embodiment", "another embodiment", "some embodiments", "various embodiments", "other embodiments", "alternative embodiment", and the like are intended to mean that any such features are included in one or more embodiments of the present disclosure, but may or may not necessarily be combined in the same embodiments.

It is also important to note that the functions related to making measurements and depth estimation illustrate only some of the possible functions that may be carried out by the circuits illustrated in the FIGURES. Some of these operations may be deleted or removed where appropriate, or these operations may be modified or changed considerably without departing from the scope of the present disclosure. In addition, the timing of these operations may be altered considerably. The operational flows have been offered for purposes of example and discussion. Substantial flexibility is provided by embodiments described herein in that any suitable arrangements, chronologies, configurations, and timing mechanisms may be provided without departing from the teachings of the present disclosure.

Claim 1:
A method to measure depth that is insensitive to corrupting light due to internal reflections, the method comprising:
making a first measurement of corrupting light due to internal reflection with a first emitted light pulse (<NUM>) having a first start time and a first duration;
making a second measurement of corrupting light due to internal reflection with a second emitted light pulse (<NUM>) having a second duration, wherein the second emitted light pulse has a same pulse shape as the first emitted light pulse, and the second emitted light pulse has a second start time that is offset by a pre-determined amount of time (td) relative to the first start time, wherein the predetermined amount of time is less than the first duration (TL) and the second duration (TL);
determining a contribution from the corrupting light based on the first corrupting light measurement and the second corrupting light measurement; and
determining the depth based on one or more measurements with the contribution from the corrupting light removed.