Phase compensation in a time of flight system

Systems and methods are provided for imaging a surface via time of flight measurement. An illumination system includes an illumination driver and an illumination source and is configured to project modulated electromagnetic radiation to a point on a surface of interest. A sensor system includes a sensor driver and is configured to receive and demodulate electromagnetic radiation reflected from the surface of interest. A temperature sensor is configured to provide a measured temperature representing a temperature at one of the illumination driver and the sensor driver and located at a position remote from the one of the illumination driver and the sensor driver. A compensation component is configured to calculate a phase offset between the illumination system and the sensor system from at least the measured temperature and a model representing transient heat flow within the system.

TECHNICAL FIELD

This application relates generally to imaging systems, and more specifically, to phase compensation in a time of flight system.

BACKGROUND

A time-of-flight camera (ToF camera) is a range imaging camera system that resolves distance based on the known speed of light, measuring the time-of-flight of a light signal between the camera and the subject for each point of the image. A time-of-flight camera generally includes an illumination unit that illuminates the subject with light modulated with frequencies up to 100 MHz. The illumination unit normally uses infrared light to make the illumination unobtrusive. A lens can be used to gather the reflected light and images the environment onto an image sensor, with an optical band-pass filter passing only the light with the same wavelength as the illumination unit. This helps suppress non-pertinent light and reduce noise. At the image sensor, each pixel measures the time the light has taken to travel from the illumination unit to the object and back to the sensor. From this time, a distance to the subject at that point can be determined.

SUMMARY

In accordance with one example, a time of flight system is provided. An illumination system includes an illumination driver and an illumination source and is configured to project modulated electromagnetic radiation to a point on a surface of interest. A sensor system includes a sensor driver and is configured to receive and demodulate electromagnetic radiation reflected from the surface of interest. A temperature sensor is configured to provide a measured temperature representing a temperature at one of the illumination driver and the sensor driver and located at a position remote from the one of the illumination driver and the sensor driver. A compensation component is configured to calculate a phase offset between the illumination system and the sensor system from at least the measured temperature and a model representing transient heat flow within the system.

In accordance with another example, a method is provided for determining a distance to a surface of interest with a time of flight system. Modulated electromagnetic radiation is projected on a point on the surface of interest via an illumination system. Electromagnetic radiation reflected from the point on the surface of interest is received at a sensor system. The received electromagnetic radiation is demodulated at the sensor system. A temperature is measured at a position remote from respective drivers associated with each of the illumination system and the sensor system. A phase offset between the illumination system and the sensor system is calculated from at least the measured temperature and a model representing transient heat flow within the system. A location of the point on the surface of interest, relative to the time of flight system, is determined from the received electromagnetic radiation and the calculated phase offset.

In accordance with yet another example, a time of flight system is provided. An illumination system includes an illumination driver and an illumination source is configured to project modulated electromagnetic radiation to a point on a surface of interest. A sensor system includes a sensor driver and is configured to receive and demodulate electromagnetic radiation reflected from the surface of interest. A temperature sensor is configured to provide a measured temperature representing a temperature at one of the illumination driver and the sensor driver and located at a position remote from the one of the illumination driver and the sensor driver. A compensation component is configured to calculate a numerical representation of the time derivative of the measured temperature for at least one time as well as a phase offset between the illumination system and the sensor system. The phase offset is calculated from at least the measured temperature, the calculated time derivative of the measured temperature, and a model representing transient heat flow within the system. A time of flight calculation component is configured to determine a distance to the point of the surface of interest from the demodulated electromagnetic radiation and the phase offset.

DETAILED DESCRIPTION

Described herein are systems and methods for maintaining an accurate phase offset in a time of flight imaging system. In a time of flight system, both the illumination unit and the image sensor have to be controlled by high-speed signals, provided by associated driver electronics, and synchronized with high accuracy to maintain to obtain a high resolution. For example, if the signals between the illumination unit and the sensor shift by only ten picoseconds, the measured distance changes by one and a half millimeters. At these levels of precision, changes in the temperature of the system components can affect this timing sufficiently to introduce significant inaccuracies. A correction for any differences in timing between the illumination unit and the image sensor, referred to as a phase offset, can be obtained via a calibration process for a range of expected temperatures.

It can be difficult to directly measure the temperature at the driver electronics, as there will be a finite distance between the driver and the sensor. Accordingly, in the systems described herein, it is assumed that the temperature information used for correction is provided from a location spatially remote from the drivers. When the system has been operating for some length of time, the temperature is relatively constant across the circuitry, and the remote measurement can be relied upon for calculating the phase offset. During a transition in the temperature of these components, as might occur during initiation of the system or when the power provided to the illumination is changed, the temperature at the remote location could vary significantly and nonlinearly from the temperature at the driver. As a result, a system relying solely on the measured temperature can experience significant inaccuracies in the phase offset and the corresponding measurement until a steady state temperature is reached.

FIG. 1illustrates an example of a time of flight system10. The system10includes an illumination system12having an associated illumination driver14and an illumination source15. The illumination system12is configured to project electromagnetic radiation, modulated by the illumination driver14, to a point on a surface of interest. In one example, the illumination source15can provide infrared light. After reflecting from the surface of interest, the light is received at a sensor20, configured to receive and demodulate electromagnetic radiation reflected from the surface of interest. To this end, the sensor20includes a sensor driver22to demodulate the received electromagnetic radiation and provide the demodulated signal to a time of flight (TOF) calculation component24. From the time of flight, a distance to the point of the surface of interest can be determined. For example, a difference in the phase of the transmitted signal and the demodulated signal can be determined. From this system, a depth mapping of a region of interest within the surface can be obtained.

Each of the illumination driver14and the sensor driver22can be provided with a timing reference from a common timing generator11, although it will be appreciated that small differences in the provided reference can be introduced via differences in the path lengths between the drivers and the timing generator. The accuracy of the time of flight calculation can be affected by any difference in the timing of the illumination system12and the sensor20, and more specifically by differences in the timing their respective drivers14and22. Accordingly, a phase offset between the illumination driver14and the sensor driver22can be provided to the time of flight calculation component24from a compensation component26to account for any differences in timing. The phase offset is sensitive to a die temperature of the circuit board or boards containing the illumination driver14and the sensor driver22. When the system10has reached a steady state temperature, the phase offset tends to stabilize to a standard value. During transitions, such as starting the system or changing the power of the illumination system12, the offset value can vary significantly. It will be appreciated that it is undesirable to utilize the standard correction for the system during these times.

In the illustrated system10, a direct measurement of the temperature at the illumination driver14or the sensor driver22is not available. Instead, one or more temperature sensors28, each located remote from each of the illumination driver14or the sensor driver22but on a same circuit board as one of the drivers14and22, each provide a measured temperature to the compensation component26. From the measured temperature or temperatures and a transient heat flow model of the system10, the phase offset can be corrected during transitions, maintaining an accurate time of flight measurement. In one implementation, the transient heat flow model can be a specific instance of a parameterized general model, with a set of parameters for a particular system configuration being determined via curve fitting of the results of a reference circuit board to the general model.

The illustrated system10takes into account the transient variation of temperatures rather than just the instantaneous values to calibrate the phase offset, providing faster convergence of the measured value of phase to the correct value. Current time of flight systems sometimes take several minutes to give stable value of phase as the thermal time constants are quite long. The illustrated system converges to the correct phase offset in only a few seconds, allowing the system to remain useful when the power provided to the illumination system12is altered or the temperature at the drivers14and22is otherwise changed.

FIG. 2illustrates an example model of a circuit50containing a sensor driver52and an illumination driver54. It will be appreciated that, while the model is shown as having the sensor driver, the illumination driver, and two temperature sensors,56and58, on a single board, in other implementations, the two drivers and their respective remote temperature sensors can be located on two separate boards. Further, while the offset will depend on the temperature at each of the photosensor driver52and the illumination driver54, for the sake of simplicity of explanation, the following model will focus on the effect of the temperature at the sensor driver on the phase offset. It will be appreciated, however, that the following analysis can be extended in a similar manner to determine the effect of the temperature at the illumination driver54on the phase offset. In such a case, the model would likely include additional measured temperatures for at least the second temperate sensor58, as well as additional parameters, such as thermal time constants for the second temperature sensor58and the illumination driver54. Further, in this specific example, it will be assumed that the sensor associated with the sensor driver52is detecting one of infrared, ultraviolet, or visible light. Accordingly, this sensor will be referred to as the “photosensor” to distinguish it from the temperature sensors56and58.

The inventors have determined that, for the illustrated system50and similarly configured systems, the relationship between the phase offset and the temperature at the photosensor driver52is substantially linear. Specifically, the phase offset due to the temperature at the photosensor driver52at a given time, t, can be expressed as:
P0(t)=K(TS(t)−TSC)  Eq. 1

where P0is the phase offset, TS, is the photosensor temperature, TSCis the temperature for which the phase offset is calibrated to be zero, and K is a constant static coefficient determined for the system.

The temperature of the photosensor driver will vary from an initial temperature, TS0, to a final stable temperature, TSF, in an essentially exponential fashion, such that:
TS(t)TSF+(TS0−TSF)e−t/τSEq. 2

where τSis a thermal time constant, measured for each system, associated with the temperature change at the photosensor.

From Eqs. 1 and 2, the change in the phase offset over time can be expressed, in terms of the photosensor temperature, as:
P0(t)=K(TSFTS0−TSF]e−t/τS−TSC)  Eq. 3

The measured temperature, T1, and the photosensor temperature, are also generally linearly related. In the illustrated system50, however, during transitions, the presence of the capacitors CPand CSintroduce nonlinearities into the relationship between the measured temperature and the temperature at the photosensor. Accordingly, while the measured temperature varies exponentially with time, it may due so with a different thermal time constant, τ1, such that:
T1(t)=T1F+(T10−T1F)e−t/τ1Eq. 4

where T10is an initial temperature at the temperature sensor, T1Fis a final stable temperature, τ1is a thermal time constant, measured for each system, associated with the temperature change at the temperature sensor.

From Eq. 4, we can determine a time derivative of the temperature at the temperature sensor as:

Since the temperature at the photosensor and the temperature at the temperature sensor are linearly related outside of temperature transients, Eq. 3 can be rewritten as:
P0(t)=K(T1FT10−T1F]e−t/τS−T1C)  Eq. 6

where T1Cis the temperature for which the phase offset is calibrated to be zero.

It will be appreciated that the time derivative of the measured temperature can be determined numerically from the measured temperature data. In many systems, it can be assumed that the time constants for the temperature at the photosensor driver52and the temperature sensor56are reasonably similar in value, such the term

(1τS-tτ1)
is sufficiently close to zero to allow the exponential term to be linearly approximated near zero as

1-(1τS-tτ1).
Accordingly, the model of Eq. 7 can be approximated as:

In other systems, it can be assumed that the time constants for the temperature at the photosensor driver52and the temperature sensor56are substantially similar in value, such that such the term

(1τS-tτ1)
is sufficiently close to zero to allow the exponential term to be ignored. In this case, the model of Eq. 7 can be approximated as:

It will be appreciated that the time constants, τ1and τs, as well as the constant K will differ among systems. To this end,FIGS. 3-6graphically illustrate the determination of these parameters. In these examples, each parameter is determined via a curve fitting procedure applied to one of the models of Eqs. 7-9 for data collected from a reference system. It will be appreciate that, for systems different from that ofFIG. 2that additional or different parameters may be used within the general model.

FIG. 3illustrates a chart80of calibration data obtained for calculating the static coefficient, K. The horizontal axis82represents a difference between a measured temperature at the temperature sensor and a calibrated temperature for which the phase offset is calibrated to be zero. The vertical axis84represents the change in the phase offset due to temperature. The phase is measured as a digital value for which 4096 units is equal to a full period (e.g., 2π radians). As can be seen from the chart80, the plotted curve86is substantially linear. The static coefficient, K, can be determined as the slope of this curve or a line fitted to the curve, when the linear relationship is insufficient to extract a slope from the curve. Accordingly, the static coefficient represents an expected amount of change in the phase offset associated with a given deviation of the measured temperature from the calibration value when the temperature of the system is at a steady state, that is, not in transition.

FIG. 4illustrates a chart100of calibration data obtained for determining the time constant, τs, of the temperature at the photosensor. The vertical axis102represents the phase offset, represented as a digital value for which 4096 units is equal to a full period (e.g., 2π radians). The horizontal axis104represents time, measured in seconds. Each dataset106-111represents data taken from the reference system at a specific temperature, specifically twenty-two degrees Celsius106, twenty-four degrees Celsius107, twenty-seven degrees Celsius108, thirty degrees Celsius109, thirty-three degrees Celsius110, and thirty-five degrees Celsius111. For each dataset106-111, an exponential curve, TS(t)=TSF+(TS0−TSF)e−t/τS, can be fitted to the dataset to obtain a time constant associated with the represented temperature. In the illustrated implementation, the time constant is substantially equal across temperatures, and the same value can be used across a temperature range of interest.

FIG. 5illustrates a chart120of calibration data obtained for determining the time constant, τ1, of the temperature at the temperature sensor. The vertical axis122represents the measured temperature in Celsius. The horizontal axis124represents time, measured in seconds. Each dataset126-131represents data taken from the reference system at a specific temperature, specifically twenty-two degrees Celsius126, twenty-four degrees Celsius127, twenty-seven degrees Celsius128, thirty degrees Celsius129, thirty-three degrees Celsius130, and thirty-five degrees Celsius131. For each dataset126-131, an exponential curve, T1(t)=T1F+(T10−T1F)e−t/τ1, can be fitted to the dataset to obtain a time constant associated with the represented temperature. In the illustrated implementation, the time constant is substantially equal across temperatures, and the same value can be used across a temperature range of interest.

FIG. 6illustrates a chart140of a modeled phase offset142for an example time of flight system142and a measured phase offset144. The vertical axis146represents the phase offset, represented as a digital value for which 4096 units is equal to a full period (e.g., 2π radians). The horizontal axis148represents time, measured in seconds. It can be seen from the chart140that the predicted value tracks closely with the measured value even during an initial transition150, allowing for an increase in the accuracy of the time of flight calculation.

FIG. 7illustrates an example of a method170for determining a distance to a surface of interest with a time of flight system. At172, modulated electromagnetic radiation is projected to a point on the surface of interest via an illumination system. In one implementation, infrared light modulated to include a series of higher intensity pulses and directed at the surface of interest. At174, electromagnetic radiation reflected from the point on the surface of interest is received at a sensor system. At176, the received electromagnetic radiation is demodulated at the sensor system. At178, a temperature is measured at a position remote from respective drivers associated with each of the illumination system and the sensor system. By “remote from,” it is meant that there is a sufficient spatial separation that a temperature change at a driver will not be instantaneously measured at the temperature sensor. For example, a temperature sensor located on a same circuit board as one or both of the sensor driver and the illumination driver, but spatially removed from both components, can be used to measure the temperature.

At180, a phase offset between the illumination system and the sensor system is calculated from at least the measured temperature and a model representing transient heat flow within the system. In one implementation, the model can include a time derivative of the measured temperature, such that calculating the phase offset includes calculating a numerical derivative of the measured temperature for at least one time. Depending on the specifics of the system, the model can include one or more of a nonlinear function of a time since a transition in temperature has begun, a product of the time derivative of the measured temperature and a linear term representing the time since the transition began, and a linear combination the time derivative of the measured temperature, a final stable temperature at the temperature sensor after transition in temperature, and a calibration temperature for the temperature sensor at which the phase offset is expected to be zero.

It will be appreciated that a given model can be parameterized for a general class of systems. To this end, the method can include an additional step of determining a plurality of system parameters for the model from a reference system via a curve fitting analysis. (not shown). In one implementation, the plurality of system parameters can include a calibration temperature for the measured temperature at which the phase offset is expected to be zero, a thermal time constant associated with the driver associated with the sensor system, a thermal time constant associated with the driver associated with the illumination system, a thermal time constant associated with the driver associated with each temperature sensor, and a static coefficient representing an expected amount of change in the phase offset associated with a given deviation of the measured temperature from the calibration value when the temperature of the system is not in transition. At182, a location of the point on the surface of interest, relative to the time of flight system, is determined from the received electromagnetic radiation and the calculated phase offset.

It will be appreciated that the method, including172,174,176,178,180, and182, can be iteratively repeated to provide locations of multiple locations on the surface of interest. It will be appreciated that the calculated phase offset at180can be refined over time to reflective changes in temperature as the time of flight system is operating. In one implementation, measured temperatures from178can be retained across iterations and utilized to provide a more accurate calculation of the phase offset at180. For example, multiple temperature readings over time can be used to calculate a numerical derivative as part of the phase offset calculation.

FIG. 8is a schematic block diagram illustrating an exemplary system200of hardware components capable of implementing examples of the systems and methods disclosed inFIGS. 1-7, such as time of flight compensation component24and the compensation component26. The system200can include various systems and subsystems. The system200can be a personal computer, a laptop computer, a workstation, a computer system, an appliance, a “smart” phone, an application-specific integrated circuit (ASIC), a server, a server blade center, a server farm, etc.

The system200can includes a system bus202, a processing unit204, a system memory206, memory devices208and210, a communication interface212(e.g., a network interface), a communication link214, a display216(e.g., a video screen), and an input device218(e.g., a keyboard and/or a mouse). The system bus202can be in communication with the processing unit204and the system memory206. The additional memory devices208and210, such as a hard disk drive, server, stand-alone database, or other non-volatile memory, can also be in communication with the system bus202. The system bus202interconnects the processing unit204, the memory devices206-210, the communication interface212, the display216, and the input device218. In some examples, the system bus202also interconnects an additional port (not shown), such as a universal serial bus (USB) port.

The processing unit204can be a computing device and can include an application-specific integrated circuit (ASIC). The processing unit204executes a set of instructions to implement the operations of examples disclosed herein. The processing unit can include a processing core.

The additional memory devices206,208and210can store data, programs, instructions, database queries in text or compiled form, and any other information that can be needed to operate a computer. The memories206,208and210can be implemented as computer-readable media (integrated or removable) such as a memory card, disk drive, compact disk (CD), or server accessible over a network. In certain examples, the memories206,208and210can comprise text, images, video, and/or audio, portions of which can be available in formats comprehensible to human beings.

Additionally or alternatively, the system200can access an external data source or query source through the communication interface212, which can communicate with the system bus202and the communication link214.

In operation, the system200can be used to implement one or more parts of a time of flight measurement system. Computer executable logic for implementing the system control126resides on one or more of the system memory206, and the memory devices208,210in accordance with certain examples. The processing unit204executes one or more computer executable instructions originating from the system memory206and the memory devices208and210. The term “computer readable medium” as used herein refers to a medium that participates in providing instructions to the processing unit204for execution, and can include either a single medium or multiple non-transitory media operatively connected to the processing unit204.

The invention has been disclosed illustratively. Accordingly, the terminology employed throughout the disclosure should be read in an exemplary rather than a limiting manner. Although minor modifications of the invention will occur to those well versed in the art, it shall be understood that what is intended to be circumscribed within the scope of the patent warranted hereon are all such embodiments that reasonably fall within the scope of the advancement to the art hereby contributed.