Patent ID: 12196860

DETAILED DESCRIPTION OF EMBODIMENTS

Overview

In some of the embodiments described in the above-mentioned U.S. Patent Application Publication 2020/0256669, SPADs are grouped together into “super-pixels,” wherein the term “super-pixel” refers to a group of mutually-adjacent pixels along with data processing elements that are coupled directly to these pixels. At any instant during operation of the system, only the sensing elements in the area or areas of the array that are to receive reflected illumination from a beam are actuated, for example by appropriate biasing of the SPADs in selected super-pixels, while the remaining sensing elements are inactive. The sensing elements are thus actuated only when their signals provide useful information. This approach reduces the background signal, thus enhancing the signal-to-background ratio, and lowers both the electrical power needs of the detector array and the number of data processing units that must be attached to the SPAD array.

One issue to be resolved in a depth mapping system of this sort is the sizes and locations of the super-pixels to be used. For accurate depth mapping, with high signal/background ratio, it is important that the super-pixels contain the detector elements onto which most of the energy of the reflected beams is imaged, while the sensing elements that do not receive reflected beams remain inactive. A mapping of SPAD pixels to processing units, i.e., the assignment of SPAD pixels to super-pixels, may be determined initially during a factory calibration. Temperature changes during operation, as well as mechanical shocks, however, may alter the mechanical parameters of the mapping, thus modifying the positions of the laser spots on the SPAD array and necessitating recalibration during operation in the field.

In response to this problem, U.S. Patent Application Publication 2020/0256669 describes methods for calibrating the locations of the reflected laser spots on the SPAD array. In these methods, processing and control circuitry receives timing signals from the SPAD array and searches over the sensing elements in order to identify the respective regions of the array on which the light pulses reflected from the target scene are incident. This search process, however, can be time-consuming, and the actual depth mapping operation cannot begin until the search is completed.

Embodiments of the present invention that are described herein provide improved methods for verifying and updating the calibration of the locations of the laser spots on an array of single-photon detectors, as well as devices implementing these improved methods. These methods make use of stray radiation that is reflected from an optical window, such as the window of a housing containing the radiation source and the imaging assembly. The term “stray radiation” refers to the small fraction of photons that are emitted by the radiation source and reflect back directly from the window to particular locations on the array of sensing elements. These stray photons can be identified readily, since their times of flight are much shorter than those of photons reflected from the target.

The locations at which the stray photons are incident on the array of sensing elements are fixed by the relative positions and geometry of the illumination and sensing assemblies in the depth sensing device. Consequently, any change in these locations is a reliable indicator that the internal alignment of the device has changed. Upon detecting such a change, processing and control circuitry in the device can immediately correct the calibration of the locations of the laser spots reflected from the target scene. In some cases, the circuitry is able to compute a coordinate transformation relating the previous locations of the stray reflections on the array of sensing elements, as indicated by the existing calibration, to the new locations following the detected change, and can then apply this coordinate transformation in correcting the calibration. Even when it is not possible to compute and apply such a transformation, the detection of the change in the stray reflections can prompt the processing and control circuitry to initiate a new search in order to update the calibration.

The disclosed embodiments thus provide depth sensing apparatus, which comprises a transparent window, a radiation source, and an imaging assembly. The radiation source emits a first array of beams of light pulses through the window toward a target scene. The imaging assembly comprises a second array of sensing elements, which output signals indicative of respective times of incidence of photons on the sensing elements, along with objective optics, which image the target scene onto the array of sensing elements. Processing and control circuitry in the apparatus stores and uses a dual calibration in processing these signals:A first calibration associates the beams in the first array with respective locations on the second array onto which the beams reflected from the target scene are imaged. The processing and control circuitry processes the signals output by the sensing elements, in accordance with this first calibration in order to measure respective times of flight of the light pulses and thus sense the depths of points in the target scene.A second calibration indicates the locations on which stray radiation is incident on the array of sensing elements due to reflection of the beams from the window. In response to the signals output by the sensing elements, the processing and control circuitry is able to detect changes in the locations on which the stray radiation is incident relative to the second calibration. Upon detecting such a change, the processing and control circuitry corrects the first calibration, as explained above, in order to compensate for the detected change.

System Description

FIG.1is a schematic side view of a depth mapping system20, in accordance with an embodiment of the invention. Depth mapping system20comprises a camera module48, which comprises a radiation source21and an imaging assembly23. The elements of camera module48are contained in a housing25, which comprises a transparent window29through which optical radiation exits and enters the housing.

Radiation source21emits M individual beams30(for example, M may be on the order of 500). The radiation source typically comprises one or more banks of emitters arranged in a two-dimensional emitter array22, together with beam optics37. The emitters typically comprise solid-state devices, such as vertical-cavity surface-emitting lasers (VCSELs) or other sorts of lasers or light-emitting diodes (LEDs). Beam optics37typically comprise a collimating lens and may comprise a diffractive optical element (DOE, not shown), which replicates the actual beams emitted by array22to create the M beams that are projected onto a target scene32. For the sake of simplicity, these internal elements of beam optics37are not shown.

Imaging assembly23comprises a two-dimensional array24of sensing elements, for example single-photon detectors, such as SPADs. Imaging assembly23also comprises J processing units28, along with select and readout circuits (not shown) for coupling the processing units to the sensing elements and to a controller26. Array24comprises a number of sensing elements N that is much larger than M, for example, 100×100 pixels or 200×200 pixels. The number J of processing units28depends on the number of pixels of array24to which each processing unit is coupled. These features of imaging assembly23and their operation are described in greater detail in the above-mentioned U.S. Patent Application Publication 2020/0256669.

Radiation source21emits the M pulsed beams30of light through window29toward target scene32. Although beams30are depicted inFIG.1as parallel beams of constant width, each beam diverges as dictated by diffraction. Furthermore, beams30diverge from each other so as to cover a required area of scene32. Scene32reflects or otherwise scatters those beams30that impinge on the scene. The reflected and scattered beams returning from scene32through window29are collected by objective optics34, represented by a lens inFIG.1, which form an image of scene32on array24. Thus, for example, a small region36on scene32, on which a beam30ahas impinged, is imaged onto a small area38on array24.

For clarity, processing units28are shown as if separate from array24, but in practice they may be integrated with array24along with other processing and readout circuits. Processing units28comprise hardware amplification and logic circuits, which sense and record pulses output by the sensing elements in respective super-pixels, and thus measure the times of arrival of the photons that gave rise to the pulses, as well as the strengths of the optical pulses impinging on array24. The processing units and associated circuits may assemble histograms of the times of arrival of multiple pulses emitted by array22, and thus output signals that are indicative of the distance to respective points in scene32, as well as of signal strength. Circuitry that can be used for this purpose is described, for example, in the above-mentioned U.S. Patent Application Publication 2020/0256669. Alternatively or additionally, some or all of the components of processing units28and other processing circuitry may be separate from array24and may, for example, be integrated with controller26. For the sake of generality, controller26, processing units28, and the associated processing and readout circuitry are collectively referred to herein as “processing and control circuitry.”

Controller26is coupled to both radiation source21and imaging assembly23. Controller26actuates the emitters in array22to emit the pulsed beams. The controller also provides control signals to imaging assembly23and receives output signals from processing units28. The output signals may comprise histogram data, as noted earlier, and may be used by controller26to derive both times of incidence and signal strengths at the location of each laser spot that is imaged onto array24.

To make optimal use of the available sensing and processing resources, controller26identifies the respective locations on array24on which the pulses of optical radiation reflected from corresponding regions of target scene32are imaged by lens34, and groups the sensing elements into super-pixels that correspond to these locations. The signals output by sensing elements outside these areas are not used, and these sensing elements may thus be deactivated, for example by reducing or turning off the bias voltage to these sensing elements. Methods for choosing the super-pixels initially and for verifying and updating the selection of super-pixels are described, for example, in the above-mentioned U.S. Patent Application Publication 2020/0256669.

For clarity, the dimensions of arrays22and24have been exaggerated inFIG.1relative to scene32. The lateral separation between arrays22and24, referred to as the “baseline,” is in reality much smaller than the distance from emitter array22to scene32. Consequently a chief ray40(a ray passing through the center of objective optics34) from scene32to array24is nearly parallel to rays30, leading to only a small amount of parallax.

Controller26typically comprises a programmable processor, which is programmed in software and/or firmware to carry out the functions that are described herein. Alternatively or additionally, controller26comprises hard-wired and/or programmable hardware logic circuits, which carry out at least some of the functions of the controller. Although controller26is shown in the figure, for the sake of simplicity, as a single, monolithic functional block, in practice the controller may comprise a single chip or a set of two or more chips, with suitable interfaces for receiving and outputting the signals that are described herein.

One of the functional units of controller26is a depth processing unit (DPU)27, which processes signals output by processing units28in order to calculate the times of flight of the photons in each of beams30, and thus maps the corresponding distances to the points in target scene32. This mapping is based on the timing of the emission of beams30by emitter array22and from the times of arrival (i.e., times of incidence of reflected photons) measured by processing units28. Controller26typically stores the depth coordinates in a memory, and may output the corresponding depth map for display and/or further processing.

FIG.2is a schematic detail view of camera module48, in accordance with an embodiment of the invention. Emitter array22and sensing array24are mounted side by side on a substrate50, such as a printed circuit board. Beam optics37direct the beams emitted from array22through window29toward the target scene (not shown in this figure). Objective optics34image the target scene via a bandpass filter52onto sensing array24.

A number of the beams emitted from array22, however, are reflected from the outer surface of window29back into module48as stray beams54. These stray beams pass through objective optics34and filter52and are incident on certain sensing elements in array24(typically, although not necessarily, at the edge of array24). The times of flight of the photons in these beams are very short, relative to the times of flight to and from target scene32(as shown inFIG.1). Controller26is thus able to distinguish stray beams54from the light that is actually returned from the target scene and can also identify readily the locations in array24on which the stray beams are incident.

As noted earlier, the locations on array24at which stray beams54are incident can be stored as part of a process of calibration of camera module48; and controller26can use changes in these locations in correcting the calibration when necessary. A calibration and correction procedure of this sort is described in greater detail hereinbelow with reference toFIG.4.

In the pictured example, window29is sufficiently thick and distant from optics34and37so that stray beams54fall within the field of view of objective optics34. When the outer surface of window29is closer to optics34and37(for example because a thinner window is used), the stray beams reflected in a single bounce from the outer surface of the window may fall outside the field of view of the objective optics. Even in this case, however, stray beams with sufficient intensity to be used in calibration may reach sensing array24after multiple bounces within window29or additional reflections from other surfaces within the camera module.

Super-Pixel Selection and Actuation

FIG.3Ais a schematic representation of a pattern of spots70of optical radiation that are projected onto target scene32, in accordance with an embodiment of the invention. Each spot70is cast by a corresponding beam30(FIG.1).

FIG.3Bis a schematic frontal view of sensing array24onto which target scene32is imaged, in accordance with an embodiment of the invention. The sensing elements, such as SPADs, in array24are too small to be seen in this figure. Rather,FIG.3Bshows the locations of spots72that are reflected from target scene32and imaged onto array24by objective optics34. In other words, each spot72is the image on array24of a corresponding spot70that is projected onto scene32by emitter array22. Optics34image a region74of target scene32(FIG.3A), including spots70that the area contains, onto a corresponding area76on array24.

FIG.3Cis a schematic detail view of area76of array24, showing the locations of spots72that are imaged onto the array, in accordance with an embodiment of the invention. In this view, it can be seen that array24comprises a matrix of sensing elements78, such as SPADs. (As noted earlier, sensing elements78in an array are also referred to as “pixels.”) Controller26assigns each processing unit28to a super-pixel80comprising a 2×2 group of the sensing elements78. In this example, it is assumed that during an initial calibration stage, spots72were imaged onto array24at locations72a. Controller26thus selected the sensing elements78to assign to each super-pixel80so as to maximize the overlap between the corresponding spot72and the super-pixel, and thus maximize the signal received from each super-pixel.

At some later stage, however, spots72shifted to new locations72bon array24. This shift may have occurred, for example, due to mechanical shock or thermal effects in camera module48, or due to other causes. Spots72at locations72bno longer overlap with super-pixels80in area76, or overlap only minimally with the super-pixels. Sensing elements78on which the spots are now imaged, however, are inactive and are not connected to any of processing units28. To rectify this situation, controller26corrects the calibration of the locations of super-pixels80, as described below.

FIG.4is a flow chart that schematically illustrates a method for depth mapping, including calibration and correction when required, in accordance with an embodiment of the invention. This method is described here as one example of how direct reflections of stray beams54from a surface such as window29(as shown inFIG.2) can be used in detecting and correcting for certain changes in calibration. Other methods of calibration and correction based on these sorts of direct reflections will be apparent to those skilled in the art after reading the present description and are considered to be within the scope of the present invention.

As an initial step in this method, controller26stores an initial calibration, which associates each beam30that is emitted by radiation source22and forms a spot70on target scene32with a corresponding location on sensing array24onto which the spot is imaged. Procedures for performing this calibration are described in detail in the above-mentioned U.S. Patent Application Publication 2020/0256669. Typically, the calibration is performed initially in the factory, and it may be repeated as and when required in the field, for example when camera module48is found to have undergone a major shift in alignment.

As part of this calibration, controller26also registers and stores the locations on which stray beams54are incident on sensing array24following reflection of the stray beams from window29, at a registration step90. These stray beams are referred to as “direct reflections” (DR), and the locations at which they impinge on array24are referred to as “DR spots.” Controller26may also store other features of the DR spots, such as the magnitude of the corresponding signals output by the sensing array.

Subsequently, whenever camera module48is activated, for example by a user of system20, controller26detects and checks the signals output by sensing array24in response to the DR spots before it begins to track depth coordinates of points in target scene32, at a spot detection step92. As noted earlier, the DR spots can be identified readily on the basis of their short times of flight, as well as the calibrated locations at which they are incident on sensing array24. Controller26compares the current locations of the DR spots to the locations that were stored as part of the initial calibration, at a change assessment step94. Additionally or alternatively, at this step controller26may compare the magnitudes of the signals output by the sensing array in response to the DR spots.

If the locations and/or magnitudes are unchanged (to within a predetermined tolerance), controller26typically goes on to receive and process signals from the appropriate super-pixels80in sensing array24in order to track the depth coordinates of the target scene, at a tracking step96. Alternatively or additionally, controller26may consider other factors, such as prior failures of system20in acquiring a suitable depth map of target scene32, in deciding whether to proceed to tracking step96or to update the calibration of beams30, as described below.

If the DR spot locations have shifted relative to the stored calibration, controller26attempts to compute a coordinate transformation relating the current DR spot locations to the locations that were stored as part of the preceding calibration, at a transform evaluation step98. For example, the controller may compute a homographic transformation, which accounts for rotation and translation of the set of DR spots from the calibrated to the current DR spot locations, and possibly for changes in scale (magnification), as well. Controller26verifies that the transformation is valid, for example by applying the transformation to the calibrated DR spot locations and checking that the variance of the transformed locations relative to the actual, current DR spot locations is within a predefined tolerance. If so, controller26applies the coordinate transformation in correcting and updating the stored calibration of beams30relative to locations on sensing array24, at a calibration correction step100. Thus, for example, the translation and rotation operations of the transformation may be applied to replace locations72awith locations72bin the calibration, as illustrated inFIG.3C. Controller26then goes on to receive and process signals from super-pixels80in sensing array24using the updated calibration in order to track the depth coordinates of the target scene at step96.

On the other hand, if the DR spots have shifted and it is not possible to compute a valid homographic transformation over the new DR spot locations, controller26returns to the initial calibration mode, at a recalibration step102. In this mode, controller26searches over the sensing elements in array24in order to find the locations on sensing array24onto which spots70are imaged. This calibration procedure may follow along the lines described in the above-mentioned U.S. Patent Application Publication 2020/0256669, possibly using the previous calibration as the starting point for the search. The new calibration will also include the updated locations of the DR spots for future reference. Once the calibration is completed, controller26continues to tracking mode at step96.

Alternative Embodiments

FIG.5Ais a schematic side views of a camera module110for use in a depth mapping system, in accordance with an embodiment of the invention. Module110may be used, for example, in place of camera module48in system20, and the operation of the system will remain substantially as described above. This embodiment, as well as the embodiment ofFIG.5B, is shown and described here in order to illustrate that the transparent window from which stray radiation reflects onto array24of sensing elements need not necessarily be a single, unitary window, but may rather be one of a number of windows. In this example, camera module110includes a receive window112and a transmit window114, which are joined together. The stray radiation from emitter array22may be reflected from either or both of windows112and114.

FIG.5Bis a schematic side views of a camera module120for use in a depth mapping system, in accordance with another embodiment of the invention. Module120is similar to module110, except that it comprises a separate receive window122and transmit window124. Typically (although not necessarily), the stray reflections will come from transmit window124.

FIG.5Cis a schematic side views of a camera module130for use in a depth mapping system, in accordance with yet another embodiment of the invention. Camera module130is shown and described here in order to illustrate that a transparent window134from which stray radiation reflects onto array24of sensing elements need not necessarily be a part of a housing132in which the elements of the camera module are contained. In this embodiment, window134is external to the camera module housing but still results in stray reflections of the transmitted beams reaching array24of sensing elements.

It will be appreciated that the embodiments described above are cited by way of example, and that the present invention is not limited to what has been particularly shown and described hereinabove. Rather, the scope of the present invention includes both combinations and subcombinations of the various features described hereinabove, as well as variations and modifications thereof which would occur to persons skilled in the art upon reading the foregoing description and which are not disclosed in the prior art.