Patent Description:
A passive depth sensing system measures ambient light reflected from objects or surfaces in a scene to determine distances between the sensing system and the objects or surfaces. An active depth sensing system emits light pulses into a scene and measures reflections of the light pulses from objects or surfaces in the scene to determine the distances between the sensing system and the objects or surfaces. Some active depth sensing systems may employ diffractive optical elements (DOEs) to diffract the emitted light pulses into additional emissions that can increase the number of light projections onto the scene. In some instances, the additional emissions can be used to create (and replicate) coded patterns of light onto the scene.

The maximum diffraction angle of coded light patterns created by a DOE, and thus the field of view (FOV) of the depth sensing system, may be limited by the DOE's feature size. For depth sensing systems that employ vertical-cavity surface-emitting lasers (VCSELs), distortion increases as the fanout angle increases, which may further limit the FOV. Although the FOV of a depth sensing system may be effectively doubled by using two light projectors, image artifacts associated with stitching together depth information generated from reflections of light emitted from different light projectors may prevent (or at least render impractical) the detection of objects and surfaces positioned along the "stitching area" of the scene.

<CIT> discloses embodiments of systems and methods for identifying features using color information in an image. The image may be formed from one or more display images comprising color information and features or feature components. Because color information may be used to identify features, more than one feature or feature component may be displayed in a display image. Because a plurality of features may be identified in a calibration image, an image system, such as a projector-camera system, can reduce the number of display images needed to calibrate the system.

<CIT> relates to systems and methods for determining one or more depths. An example system includes a time-of-flight receiver configured to sense pulses from reflections of light from a structured light transmitter. An example method includes sensing, by a time-of-flight receiver, pulses from reflections of light from a structured light transmitter.

The invention is defined by the independent claims to which reference should be made. Preferable features are set out in the dependent claims.

Implementations of the subject matter described in this disclosure may allow the field of view (FOV) of an active depth sensing system to be increased beyond the limitations imposed by the feature size of diffractive optical elements (DOEs) and the maximum fanout angles associated with vertical-cavity surface-emitting lasers (VCSELs) without distortion, thereby allowing for wide-angle 3D sensing by a relatively compact active depth sensing system. According to the invention, the active depth sensing system includes a first light projector configured to project light in a first direction, a second light projector configured to project light in a second direction opposite to the first direction, and a reflective component positioned between the first and second light projectors. The reflective component is configured to redirect the light projected by the first light projector onto a first portion of a scene and to redirect the light projected by the second light projector onto a second portion of the scene that is adjacent to, but does not overlap, the first portion of the scene. The active depth sensing system also includes a controller that generates depth information based on reflections of the redirected light from the first and second portions of the scene while optionally correcting for projection distortion based at least in part on one or more angles of refraction associated with the reflective component.

In some implementations, the reflective component may include first and second reflective elements. The first reflective element may be or include a prism configured to fold the optical path of light projected from the first light projector and refract the folded light onto the first portion of the scene based at least in part on the folded optical path. The second reflective element may be or include a prism configured to fold the optical path of light projected from the second light projector and refract the folded light onto the second portion of the scene based at least in part on the folded optical path. The prisms and folded optics employed by the reflective component may allow the active depth sensing system to seamlessly stitch together depth information generated from reflections of light emitted from the first and second light projectors without stitch artifacts.

The active depth sensing system may also include switchable diffusers configured to transition the light projectors between a time-of-flight (ToF) sensing mode and a structured light (SL) sensing mode. In this manner, the active depth sensing system may reap advantages offered by both ToF and SL techniques while minimizing their respective disadvantages.

In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means connected directly to or connected through one or more intervening components or circuits. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring teachings of the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing the terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "settling," or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

In the figures, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by that block may be performed in a single component or across multiple components, and/or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Also, the example devices may include components other than those shown, including well-known components such as a processor, memory and the like.

Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more active depth sensing systems. While described below with respect to a device having or coupled to two light projectors, aspects of the present disclosure are applicable to devices having any number of light projectors, and are therefore not limited to specific devices.

The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, and so on). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific embodiments. For example, a system may be implemented on one or more printed circuit boards or other substrates, and may have movable or static components. While the below description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

<FIG> shows an example ToF system <NUM>. The ToF system <NUM> may be used to generate depth information of a scene including a surface <NUM>, or may be used for other applications for ranging surfaces or other portions of the scene. The ToF system <NUM> may include a transmitter <NUM> and a receiver <NUM>. The transmitter <NUM> may be referred to as a "light projector," "transmitter," "projector," "emitter," and so on, and should not be limited to a specific transmission component. Similarly, the receiver <NUM> may be referred to as a "light sensor," "detector," "sensor," "sensing element," "photodetector," and so on, and should not be limited to a specific receiving component.

The transmitter <NUM> may be configured to transmit, emit, or project signals (such as a field of light) onto the scene. While ToF systems are described in the examples as emitting light (which may include near-infrared (NIR)), signals at other frequencies may be used, such as microwaves, radio frequency signals, sound, and so on. The present disclosure should not be limited to a specific range of frequencies for the emitted signals.

The transmitter <NUM> transmits light <NUM> toward a scene including a surface <NUM>. The transmitted light <NUM> includes light pulses <NUM> at known time intervals (such as periodically). The receiver <NUM> includes a sensor <NUM> to sense the reflections <NUM> of the transmitted light <NUM>. The reflections <NUM> include the reflected light pulses <NUM>, and the ToF system <NUM> determines a round trip time <NUM> for the light by comparing the timing <NUM> of the transmitted light pulses to the timing <NUM> of the reflected light pulses <NUM>. The light pulses may be modulated continuous-wave (AMCW) pulses of light. In some instances, the light reflected back to the receiver <NUM> may have a different phase than the light emitted from the transmitter <NUM>. The phase difference may be used to determine the round trip time of the emissions. The distance of the surface <NUM> from the ToF system <NUM> may be calculated to be half the round trip time multiplied by the speed of the emissions (such as the speed of light for light emissions).

The sensor <NUM> may include an array of photodiodes to measure or sense the reflections. Alternatively, the sensor <NUM> may include a CMOS sensor or other suitable photo-sensitive sensor including a number of pixels or regions for sensing. The ToF system <NUM> identifies the reflected light pulses <NUM> as sensed by the sensor <NUM> when the magnitudes of the pulses are greater than a value. For example, the ToF system <NUM> measures a magnitude of the ambient light and other interference without the signal and determines if further measurements are greater than the previous measurement by a value.

In some implementations, the sensor <NUM> may include a sensor pixel including a photodiode (not shown for simplicity) for converting photons from the reflections <NUM> into electrical current. The sensor pixel may include one or more capacitors to store energy from the current. The ToF system <NUM> may calculate a distance between the ToF system <NUM> and the surface <NUM> in part by comparing the voltages with their corresponding phases. The ToF system <NUM> may open and close a shutter to expose the sensor <NUM> at a number of particular phase offsets relative to the pulsed signal. During each exposure cycle, electrical charge may be stored by one or more storage elements, such as by capacitors.

As a non-limiting example, during a first exposure cycle, a first capacitor (C1) may store a charge (Q1) and a second capacitor (C2) may store a charge (Q2), where Q1 is the accumulated charge from the reflected signal when the shutter is open at a <NUM>° phase offset, and where Q2 is the accumulated charge from the reflected signal when the shutter is open at a <NUM>° phase offset. During a second exposure cycle, C1 may store a charge (Q3) and C2 may store a charge (Q4), where Q3 is the accumulated charge from the reflected signal when the shutter is open at a <NUM>° phase offset, and where Q4 is the accumulated charge from the reflected signal when the shutter is open at a <NUM>° phase offset. The ToF system <NUM> can calculate the phase offset (ϕ) between the pulsed signal and the reflected signal based on the charges stored across C1 and C2 for each of the exposure cycles: <MAT>.

The calculated phase offset ϕ between the pulsed signal and the reflected signal is proportional to the distance d between the corresponding sensor pixel and the surface <NUM>: <MAT>.

where c is the speed of light and f is the frequency of the modulated signal. Based on the determined distances from each pixel of the sensor <NUM> to the surface <NUM>, the ToF system <NUM> may generate depth information for the surface <NUM>.

Some environments (e.g., with corners, convex areas, and/or reflective surfaces) may cause different pulses of light to arrive at the ToF system <NUM> along multiple reflection paths and recombine at the sensor <NUM>, which is known as MPI. For purposes of discussion herein, MPI may also be referred to as "multipath effects" or "MPI effects. " MPI may cause the ToF system to overestimate the amount of charge being accumulated for one or more phase offsets of the corresponding pulsed signal. The overestimation may cause the ToF system to inaccurately calculate the corresponding phase shift ϕ between the pulsed and reflected signals. Thus, the ToF system may inaccurately calculate the corresponding distance d from one or more of the sensor pixels to the object or scene, which may cause distortions (or "bumps") in corresponding depth information.

<FIG> shows an example environment <NUM> in which MPI may affect ToF depth sensing. The ToF system includes an emitter <NUM> and a sensor <NUM>. The scene includes an object <NUM> and an object <NUM>. The object <NUM> may have a mirror-like surface. The emitter <NUM> transmits a pulsed signal <NUM> and a pulsed signal <NUM> toward the object <NUM>. The pulsed signal <NUM> and the reflected signal <NUM> follow a direct path <NUM> to the sensor <NUM>. In contrast, the pulsed signal <NUM> and the reflected signal <NUM> follow an indirect path <NUM> (e.g., reflecting off of the object <NUM>) to the sensor <NUM> such that the reflected signal <NUM> may arrive at the sensor <NUM> at a different time than the reflected signal <NUM>. In some aspects, the sensor <NUM> may interpret the reflected signals <NUM> and <NUM> as being reflected from the same location on object <NUM>. As such, when the reflected signals <NUM> and <NUM> arrive at the sensor <NUM> at two different times, the sensor <NUM> may generate two different distances for the location on object <NUM>, causing MPI.

<FIG> shows another example environment <NUM> in which MPI may affect ToF depth sensing. The ToF system includes an emitter <NUM> and a sensor <NUM>. The scene includes an object <NUM> and an object <NUM>. The object <NUM> may have a semitransparent surface. The emitter <NUM> transmits a pulsed signal <NUM> and a pulsed signal <NUM>. The pulsed signal <NUM> and the reflected signal <NUM> follow path <NUM> (e.g., reflecting off of the object <NUM>). The pulsed signal <NUM> and the reflected signal <NUM> follow path <NUM> (e.g., reflecting off of the object <NUM>). The reflected signal <NUM> may arrive at the sensor <NUM> at a different time than the reflected signal <NUM>. In some aspects, the sensor <NUM> may interpret the reflected signals <NUM> and <NUM> as being reflected from the same location on object <NUM>. As such, when the reflected signals <NUM> and <NUM> arrive at the sensor <NUM> at two different times, the sensor <NUM> may generate two different distances for the location on object <NUM>, causing MPI.

<FIG> shows another example environment <NUM> in which MPI may affect ToF depth sensing. The ToF system includes an emitter <NUM> and a sensor <NUM>. The scene includes an object <NUM> and an object <NUM>, which may represent two walls that intersect at a corner point. The emitter <NUM> transmits a pulsed signal <NUM>, a pulsed signal <NUM>, and a pulsed signal <NUM>, and the sensor <NUM> receives corresponding reflected signals <NUM>. Possibly due to the reflective properties of the object <NUM> or the object <NUM>, two or more of the reflected signals <NUM> may arrive at the sensor <NUM> at a different time. In some aspects, the sensor <NUM> may interpret each of the reflected signals <NUM> as being reflected from the same location on object <NUM> or object <NUM>. As such, if two or more of the reflected signals <NUM> arrive at the sensor <NUM> at different times, the sensor <NUM> may generate two or more different distances for the location on object <NUM> or <NUM>, causing MPI. For example, a conventional ToF system may superimpose multiple of the reflected signals <NUM>, resulting in accurate distance calculations for the corresponding location on object <NUM> or object <NUM>, and ultimately resulting in one or more regions in a corresponding depth map (not shown for simplicity) inaccurately appearing to have a uniform depth.

<FIG> shows an example SL system <NUM>. A SL system may transmit light in a distribution of points (or another suitable shape of focused light). For purposes of discussion herein, the distribution of points may be referred to as a "pattern," a "SL pattern," a "dot pattern," or the like, and the pattern may be predefined. The points of light may be projected on to a scene, and the reflections of the points of light may be received by the SL system. Depths of objects in a scene may be determined by comparing the pattern of the received light and the pattern of the transmitted light. In comparing the patterns, a portion of the predefined distribution for the transmitted light may be identified in the received light. A SL system may project a distribution of light (such as a distribution of light points or other shapes) using a SL projector.

The SL system <NUM> (which herein may also be called a SL system) may be used to generate depth information for a scene <NUM>. For example, the scene <NUM> may include a face, and the SL system <NUM> may be used for identifying or authenticating the face. The SL system <NUM> may include a transmitter <NUM> and a receiver <NUM>. The transmitter <NUM> may be referred to as a "transmitter," "projector," "emitter," and so on, and should not be limited to a specific transmission component. Throughout the following disclosure, the terms projector and transmitter may be used interchangeably. The receiver <NUM> may be referred to as a "detector," "sensor," "sensing element," "photodetector," and so on, and should not be limited to a specific receiving component.

While the disclosure refers to the distribution as a light distribution, any suitable signals at other frequencies may be used (such as radio frequency waves, sound waves, etc.). Further, while the disclosure refers to the distribution as including a plurality of light points, the light may be focused into any suitable size and dimensions. For example, the light may be projected in lines, squares, or any other suitable dimension. In addition, the disclosure may refer to the distribution as a codeword distribution, where a defined portion of the distribution (such as a predefined patch of light points) is referred to as a codeword. If the distribution of the light points is known, the codewords of the distribution may be known. However, the distribution may be organized in any way, and the present disclosure should not be limited to a specific type of distribution or type of signal or pulse.

The transmitter <NUM> may be configured to project or transmit a distribution <NUM> of light points onto the scene <NUM>. The white circles in the distribution <NUM> may indicate where no light is projected for a possible point location, and the black circles in the distribution <NUM> may indicate where light is projected for a possible point location. In some example implementations, the transmitter <NUM> may include one or more light sources <NUM> (such as one or more lasers), a lens <NUM>, and a light modulator <NUM>. The transmitter <NUM> may also include an aperture <NUM> from which the transmitted light escapes the transmitter <NUM>. In some implementations, the transmitter <NUM> may further include a diffractive optical element (DOE) to diffract the emissions from one or more light sources <NUM> into additional emissions. In some aspects, the light modulator <NUM> may include a DOE, for example, to adjust the intensity of the emission. In projecting the distribution <NUM> of light points onto the scene <NUM>, the transmitter <NUM> may transmit one or more lasers from the light source <NUM> through the lens <NUM> (and/or through a DOE or light modulator <NUM>) and onto the scene <NUM>. The transmitter <NUM> may be positioned on the same reference plane as the receiver <NUM>, and the transmitter <NUM> and the receiver <NUM> may be separated by a distance called the baseline (<NUM>).

In some example implementations, the light projected by the transmitter <NUM> may be infrared (IR) light. IR light may include portions of the visible light spectrum and/or portions of the light spectrum that is not visible to the naked eye. In one example, IR light may include NIR light, which may or may not include light within the visible light spectrum, and/or IR light (such as far infrared (FIR) light) which is outside the visible light spectrum. The term IR light should not be limited to light having a specific wavelength in or near the wavelength range of IR light. Further, IR light is provided as an example emission from the transmitter. In the following description, other suitable wavelengths of light may be used, such as, for example, light in portions of the visible light spectrum outside the IR light wavelength range or ultraviolet light. Alternatively, other signals with different wavelengths may be used, such as microwaves, radio frequency signals, and other suitable signals.

The scene <NUM> may include objects at different depths from the SL system (such as from the transmitter <NUM> and the receiver <NUM>). For example, objects 306A and 306B in the scene <NUM> may be at different depths. The receiver <NUM> may be configured to receive, from the scene <NUM>, reflections <NUM> of the transmitted distribution <NUM> of light points. To receive the reflections <NUM>, the receiver <NUM> may capture an image. When capturing the image, the receiver <NUM> may receive the reflections <NUM>, as well as (i) other reflections of the distribution <NUM> of light points from other portions of the scene <NUM> at different depths and (ii) ambient light. Noise may also exist in the captured image.

In some example implementations, the receiver <NUM> may include a lens <NUM> to focus or direct the received light (including the reflections <NUM> from the objects 306A and 306B) on to the sensor <NUM> of the receiver <NUM>. The receiver <NUM> may also include an aperture <NUM>. Assuming for the example that only the reflections <NUM> are received, depths of the objects 306A and 306B may be determined based on the baseline <NUM>, displacement and distortion of the light distribution <NUM> (such as in codewords) in the reflections <NUM>, and intensities of the reflections <NUM>. For example, the distance <NUM> along the sensor <NUM> from location <NUM> to the center <NUM> may be used in determining a depth of the object 306B in the scene <NUM>. Similarly, the distance <NUM> along the sensor <NUM> from location <NUM> to the center <NUM> may be used in determining a depth of the object 306A in the scene <NUM>. The distance along the sensor <NUM> may be measured in terms of number of pixels of the sensor <NUM> or a distance (such as millimeters).

In some example implementations, the sensor <NUM> may include an array of photodiodes (such as avalanche photodiodes) for capturing an image. To capture the image, each photodiode in the array may capture the light that hits the photodiode and may provide a value indicating the intensity of the light (a capture value). The image therefore may be the capture values provided by the array of photodiodes.

In addition or alternative to the sensor <NUM> including an array of photodiodes, the sensor <NUM> may include a complementary metal-oxide semiconductor (CMOS) sensor. To capture the image by a photosensitive CMOS sensor, each pixel of the sensor may capture the light that hits the pixel and may provide a value indicating the intensity of the light. In some example implementations, an array of photodiodes may be coupled to the CMOS sensor. In this manner, the electrical impulses generated by the array of photodiodes may trigger the corresponding pixels of the CMOS sensor to provide capture values.

The sensor <NUM> may include at least a number of pixels equal to the number of possible light points in the distribution <NUM>. For example, the array of photodiodes or the CMOS sensor may include a number of photodiodes or a number of pixels, respectively, corresponding to the number of possible light points in the distribution <NUM>. The sensor <NUM> logically may be divided into groups of pixels or photodiodes (such as 4x4 groups) that correspond to a size of a bit of a codeword. The group of pixels or photodiodes may also be referred to as a bit, and the portion of the captured image from a bit of the sensor <NUM> may also be referred to as a bit. In some example implementations, the sensor <NUM> may include the same number of bits as the distribution <NUM>.

If the light source <NUM> transmits IR light (such as NIR light at a wavelength of, e.g., <NUM>), the sensor <NUM> may be an IR sensor to receive the reflections of the NIR light. As illustrated, the distance <NUM> (corresponding to the reflections <NUM> from the object 306B) is less than the distance <NUM> (corresponding to the reflections <NUM> from the object 306A). Using triangulation based on the baseline <NUM> and the distances <NUM> and <NUM>, the differing depths of objects 306A and 306B in the scene <NUM> may be determined in generating depth information for the scene <NUM>. Determining the depths may further include determining a displacement or a distortion of the distribution <NUM> in the reflections <NUM>.

Although multiple separate components are illustrated in <FIG>, one or more of the components may be implemented together or include additional functionality. All described components may not be required for a SL system <NUM>, or the functionality of components may be separated into separate components. Additional components not illustrated may also exist. For example, the receiver <NUM> may include a bandpass filter to allow signals having a determined range of wavelengths to pass onto the sensor <NUM> (thus filtering out signals with a wavelength outside of the range). In this manner, some incidental signals (such as ambient light) may be prevented from interfering with the captures by the sensor <NUM>. The range of the bandpass filter may be centered at the transmission wavelength for the transmitter <NUM>. For example, if the transmitter <NUM> is configured to transmit NIR light with a wavelength of <NUM>, the receiver <NUM> may include a bandpass filter configured to allow NIR light having wavelengths within a range of, e.g., <NUM> to <NUM>. Therefore, the examples described regarding <FIG> are for illustrative purposes, and the present disclosure should not be limited to the example SL system <NUM>.

For a light projector (such as the transmitter <NUM>), the light source may be any suitable light source. In some example implementations, the light source <NUM> may include one or more distributed feedback (DFB) lasers. In some other example implementations, the light source <NUM> may include one or more vertical-cavity surface-emitting lasers (VCSELs).

A DOE is a material situated in the projection path of the light from the light source. The DOE may be configured to split a light point into multiple light points. For example, the material of the DOE may be a translucent or a transparent polymer with a known refractive index. The surface of the DOE may include peaks and valleys (varying the depth of the DOE) so that a light point splits into multiple light points when the light passes through the DOE. For example, the DOE may be configured to receive one or more light points from one or more lasers and project an intended distribution with a greater number of light points than emitted by the one or more lasers. While the Figures may illustrate the depth of a DOE changing along only one axis of the DOE, the Figures are only to assist in describing aspects of the disclosure. The peaks and valleys of the surface of the DOE may be located at any portion of the surface of the DOE and cause any suitable change in the depth of portions of the DOE, and the present disclosure should not be limited to a specific surface configuration for a DOE.

<FIG> shows a block diagram of an example device <NUM> configured for active depth sensing using ToF and SL techniques. It will be understood that ToF and SL are example active depth techniques and that the device <NUM> may use other active depth techniques in some implementations. In some embodiments, the device <NUM> may be configured to generate depth information using ToF techniques while using SL techniques to mitigate the effects of MPI in the depth information. The device <NUM> may include or be coupled to an emitter <NUM> (a "first emitter"), a sensor <NUM>, a processor <NUM>, a memory <NUM> storing instructions <NUM>, and an active depth controller <NUM> (which may include one or more signal processors <NUM>). The emitter <NUM> may include or be coupled to a DOE <NUM>. The DOE <NUM> may optionally be included in or coupled to the device <NUM>. The emitter <NUM> may include or be coupled to a diffuser <NUM>. The diffuser <NUM> may optionally be included in or coupled to the device <NUM>.

The device <NUM> may further include or be coupled to an emitter <NUM> (a "second emitter"). In some implementations, the emitter <NUM> may be the same or similar to the emitter <NUM>. The emitter <NUM> may include or be coupled to a DOE <NUM>, which may be the same or similar to the DOE <NUM>. The DOE <NUM> may optionally be included in or coupled to the device <NUM>. The emitter <NUM> may include or be coupled to a diffuser <NUM>, which may be the same or similar to the diffuser <NUM>. The diffuser <NUM> may optionally be included in or coupled to the device <NUM>. Aspects of the present disclosure described with respect to one or more of the emitter <NUM>, the DOE <NUM>, or the diffuser <NUM> may also apply for one or more of the emitter <NUM>, the DOE <NUM>, or the diffuser <NUM>, respectively.

A reflective component <NUM> may be positioned between the emitter <NUM> and the emitter <NUM>. The reflective component <NUM> may optionally be included in or coupled to one or more of the device <NUM>, the emitter <NUM>, or the emitter <NUM>. In some implementations, the reflective component <NUM> may be configured to redirect light projected by the emitter <NUM> onto a first portion of a scene and to redirect light projected by the emitter <NUM> onto a second portion of the scene, as is further described with respect to <FIG> and <FIG>.

For purposes of discussion herein, the device <NUM> may be referred to as a "ToF and SL system. " Further for purposes of discussion herein, the "ToF and SL system" may instead refer to just one or more components of the device <NUM> (e.g., one or more of the active depth controller <NUM>, the emitter <NUM>, the sensor <NUM>, the DOE <NUM>, the diffuser <NUM>, the emitter <NUM>, the DOE <NUM>, the diffuser <NUM>, or the reflective component <NUM>) or any other components that may be used for active depth sensing.

In some embodiments, one or more of the emitter <NUM> or the emitter <NUM> may be a single, hybrid laser projector capable of switching between projecting a first distribution of light (e.g., with use of one or more of the diffuser <NUM> or the diffuser <NUM>) during a first projection mode (e.g., a ToF projection mode) of one or more of the emitter <NUM> or the emitter <NUM> and projecting a second distribution of light (e.g., with use of one or more of the DOE <NUM> or the DOE <NUM>) during a second projection mode (e.g., a SL projection mode) of one or more of the emitter <NUM> or the emitter <NUM>. When operating in the SL projection mode, the DOEs <NUM> and <NUM> may enable the emitters <NUM> and <NUM>, respectively, to transmit the second distribution of light, which may be, for example, a known DOE dot pattern, a codeword DOE projection, or the like. The diffusers <NUM> and <NUM> may be switchable such that the diffuser is "off" (or "disabled" or "switched off) when the device <NUM> operates in the SL projection mode and is "on" (or "enabled" or "switched on") when the device <NUM> operates in the ToF projection mode.

More specifically, when operating in the ToF projection mode, the respective diffuser of the diffusers <NUM> and <NUM> is switched on, which causes the respective emitter of the emitters <NUM> and <NUM> to transmit the second distribution of light (e.g., flood distribution). Accordingly, the emitters <NUM> and <NUM> may be synchronized to project a second distribution of light (e.g., a DOE distribution) during the SL projection mode and a second distribution of light (e.g., a full flood frame) during a ToF projection mode. In some aspects, one or more of the distributions of light may be time-modulated, as described with respect to <FIG>. In some embodiments, one or more of the emitter <NUM> and the emitter <NUM> may include multiple projectors.

In some embodiments, the sensor <NUM> may be a single, hybrid ToF and SL sensor for receiving reflected light according to ToF and SL sensing (or "readout") modes. The sensor <NUM> may be configured to switch between operating in a first sensing mode (e.g., a ToF sensing mode) and a second sensing mode (e.g., a SL sensing mode). For example, the sensor <NUM> may be a composite CMOS image sensor configured to switch between operating in (or alternating between) the ToF and SL sensing modes. The sensing mode may depend on which distribution (e.g., DOE or flood) the respective emitter of the emitters <NUM> and <NUM> are projecting. In some aspects, the sensor <NUM> may be based on a monolithic pixel array architecture, for example, with Time-Division Multiplexed Read (TDMR) capabilities. In other embodiments, the sensor <NUM> may include one or more generic ToF sensors operating in conjunction with multiple projectors.

In some embodiments, the active depth controller <NUM> may be a computation element for calculating depth information. The active depth controller <NUM> may be configured to alternate between computing depth information using ToF techniques and computing depth information using SL techniques. For purposes of discussion herein, depth information calculated using SL techniques may also be referred to as "SL depth information," "SL information," or the like. Similarly, for purposes of discussion herein, depth information calculated using ToF techniques may also be referred to as "ToF depth information," "ToF information," or the like. In some aspects, the active depth controller <NUM> may use SL depth information as reference for calculating or supplementing ToF depth information, which may help compensate for MPI errors in the ToF depth information. That is, the active depth controller <NUM> may use the sparse depth information from the SL mode as a baseline reference to compensate for multipath effects in the depth information from the ToF mode. In this manner, the active depth controller <NUM> may generate high-resolution and high-accuracy depth information without MPI artifacts. In some embodiments, the sensor <NUM> may be a Reconfigurable Instruction Cell Array (RICA), which is a proprietary, real-time, low-power, (re)programmable, image signal processing (ISP), active sensing, processing engine. In some aspects, stacking the RICA programmable implementation with the hybrid NIR sensor described herein may enable the active depth controller <NUM> to switch programming "on-the-fly" to toggle computing SL depth information and ToF depth information while reducing a number of components for a sensor (e.g., the sensor <NUM>). In other embodiments, the active depth controller <NUM> may be a generic sensor.

In some aspects, the active depth controller <NUM> may be configured to control (or otherwise operate) one or more of the emitter <NUM>, the emitter <NUM>, or the sensor <NUM> to synchronize their respective operating modes, such that the sensor <NUM> and the emitters <NUM> and <NUM> concurrently operate in either their respective SL modes or ToF modes. In some aspects, the active depth controller <NUM> may be controlled, work in conjunction with, or otherwise be operated by one or more other components of the device <NUM>, such as at least one of the processor <NUM> or the memory <NUM>.

The device <NUM> may optionally include or be coupled to a display <NUM> and a number of input/output (I/O) components <NUM>. The sensor <NUM> may be, or otherwise may be coupled to, a camera, such as a single camera, a dual camera module, or a module with any number of other camera sensors (not pictured). The signal processor <NUM> may be configured to process captures from the sensor <NUM>. The device <NUM> may further include one or more optional sensors <NUM> (such as a gyroscope, magnetometer, inertial sensor, NIR sensor, and so on) coupled to the processor <NUM>. The device <NUM> may also include a power supply <NUM>, which may be coupled to or integrated into the device <NUM>. The device <NUM> may include additional features or components not shown.

The memory <NUM> may be a non-transient or non-transitory computer readable medium storing computer-executable instructions <NUM> to perform all or a portion of one or more operations described in this disclosure. The processor <NUM> may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions <NUM>) stored within the memory <NUM>. In some aspects, the processor <NUM> may be one or more general purpose processors that execute instructions <NUM> to cause the device <NUM> to perform any number of functions or operations. In additional or alternative aspects, the processor <NUM> may include integrated circuits or other hardware to perform functions or operations without the use of software. While shown to be coupled to each other via the processor <NUM> in the example of <FIG>, the processor <NUM>, the memory <NUM>, the active depth controller <NUM>, the optional display <NUM>, the optional I/O components <NUM>, and the optional sensors <NUM> may be coupled to one another in various arrangements. For example, the processor <NUM>, the memory <NUM>, the active depth controller <NUM>, the optional display <NUM>, the optional I/O components <NUM>, and/or the optional sensors <NUM> may be coupled to each other via one or more local buses (not shown for simplicity).

The display <NUM> may be any suitable display or screen allowing for user interaction and/or to present items (such as depth information or a preview image of the scene) for viewing by a user. In some aspects, the display <NUM> may be a touch-sensitive display. The I/O components <NUM> may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and to provide output to the user. For example, the I/O components <NUM> may include (but are not limited to) a graphical user interface, keyboard, mouse, microphone and speakers, squeezable bezel or border of the device <NUM>, physical buttons located on device <NUM>, and so on. The display <NUM> and/or the I/O components <NUM> may provide a preview image or depth information for the scene to a user and/or receive a user input for adjusting one or more settings of the device <NUM> (such as adjusting an intensity of emissions by one or more of the emitter <NUM> or the emitter <NUM>, determining or switching one or more operating modes of the device <NUM>, adjusting a field of emission of by one or more of the emitter <NUM> or the emitter <NUM>, and so on).

The active depth controller <NUM> may include, or may otherwise be coupled to, a signal processor <NUM>, which may be one or more processors to process captures from the sensor <NUM>. The active depth controller <NUM> may be configured to switch at least one of the emitter <NUM>, the emitter <NUM>, or the sensor <NUM> between one or more operating modes. The active depth controller <NUM> may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

One or more of the emitter <NUM> or the emitter <NUM> may vary its field of emission for different operating modes. In some example implementations, one or more of the emitter <NUM> or the emitter <NUM> may include a focusing apparatus for adjusting the size of the field of emission / transmission. In one example, mirrors attached to actuators (such as microelectromechanical systems (MEMS) actuators) may adjust the focus of the light emissions from the respective emitter of the emitters <NUM> and <NUM>. In another example, an adjustable holographic optical element (HOE) may adjust the focus of the light emissions from the respective emitter of the emitters <NUM> and <NUM>. In a further example, a formable DOE (such as a piezoelectric material to adjust the shape) may be adjusted to focus the diffracted points of light emitted.

In some other example implementations, the device <NUM> may emit light using a plurality of light emitters (not shown) instead of, or in combination with, the emitters <NUM> and <NUM>. The emitters may include a first group of light emitters (e.g., of a first array of light emitters) for emitting light with a first field of transmission. The emitters may further include a second or different group of light emitters (e.g., of a second array of light emitters) for emitting light with a second field of transmission. The first field may be larger than the second field at a common depth from one or more of the emitter <NUM> or the emitter <NUM>. In some example implementations, the first group of light emitters may be active for a first mode of the respective emitter of the emitters <NUM> and <NUM>, and the second group of light emitters may be active for a second mode of the respective emitter of the emitters <NUM> and <NUM>.

<FIG> shows a timing diagram <NUM> illustrating an example operation of a ToF and SL system including a sensor <NUM>, emitters <NUM>, and a controller <NUM>. The sensor <NUM>, emitters <NUM>, and controller <NUM> may be example embodiments of the sensor <NUM>, emitters <NUM> and <NUM>, and active depth controller <NUM>, respectively, of <FIG>. It will be understood that ToF and SL are example active depth techniques and that the system may use other active depth techniques in some implementations.

The example timing diagram <NUM> shows three projection cycles for the emitters <NUM>: a first projection cycle ending at time <NUM>, a second projection cycle ending at time <NUM>, and a third projection cycle ending at time <NUM>. Each of the emitters <NUM> may project a first distribution of light during each of the projection cycles. The first distribution of light may be a flood distribution for a first projection mode, such as a ToF projection mode. For example, each of the emitters <NUM> may project a flood distribution <NUM>, a flood distribution <NUM>, and a flood distribution <NUM> during a ToF projection mode for each of the first, second, and third projection cycles, respectively. For purposes of discussion herein, a flood distribution may also be referred to as a "flood illumination" or a "diffuse light. " In some aspects, one or more of the flood distributions may be time-modulated, as described with respect to <FIG> and <FIG>. Each of the emitters <NUM> may also project a second distribution of light during each of the projection cycles. The second distribution of light may be a DOE distribution for a second projection mode, such as a SL projection mode. For example, each of the emitters <NUM> may project a DOE distribution <NUM>, a DOE distribution <NUM>, and a DOE distribution <NUM> during the SL projection mode for each of the first, second, and third projection cycles, respectively. For purposes of discussion herein, a DOE distribution may also be referred to as a "DOE pattern," a "DOE projection," a "SL distribution," a "SL pattern," and/or a "SL projection.

The example timing diagram <NUM> shows three sensing cycles for the sensor <NUM>: a first sensing cycle ending at time <NUM>, a second sensing cycle ending at time <NUM>, and a third sensing cycle ending at time <NUM>. The sensor <NUM> may read out two frames of ToF sensor data (during a ToF sensing mode) and one frame of SL sensor data (during a SL sensing mode) for each sensing cycle. The sensor <NUM> may be configured to operate in the ToF sensing mode at the same time as when the emitters <NUM> are configured to operate in the ToF projection mode. The sensor <NUM> may be configured to operate in the SL sensing mode at the same time as when the emitters <NUM> are configured to operate in the SL projection mode. During the ToF sensing mode, the emitters <NUM> may emit laser pulses, and the sensor <NUM> may be exposed at a number of particular laser pulse phase offsets (e.g., Phase <NUM>°, Phase <NUM>°, Phase <NUM>°, and Phase <NUM>°) relative to a respective pulsed signal from each of the emitters <NUM>. The sensor <NUM> may accumulate and store an amount of charge (Q) for each of the particular laser pulse phase offsets (or "phase offsets").

For example, during a first exposure, the sensor <NUM> may read out the first frame of ToF sensor data <NUM> based on a Q1 and a Q2, where Q1 is the charge accumulated at <NUM>° phase offset, and where Q2 is the charge accumulated at <NUM>° phase offset. During a second exposure, the sensor <NUM> may read out a second frame of ToF sensor data <NUM> based on a Q3 and a Q4, where Q3 is the charge accumulated at <NUM>° phase offset, and where Q4 is the charge accumulated at <NUM>° phase offset. Similarly, the sensor <NUM> may read-out a first frame of ToF sensor data <NUM> and a second frame of ToF sensor data <NUM> during the second sensing cycle, and the sensor <NUM> may read-out a first frame of ToF sensor data <NUM> and a second frame of ToF sensor data <NUM> during the third sensing cycle. The sensor <NUM> may read-out a frame of SL sensor data <NUM> during the first sensing cycle, a frame of SL sensor data <NUM> during the second sensing cycle, and a frame of SL sensor data <NUM> during the third sensing cycle.

After each sensing cycle, the controller <NUM> may calculate SL depth information (Z(SL)) using SL sensor data. For example, the controller <NUM> may calculate SL depth information (Z(SL)) <NUM>, Z(SL) <NUM>, and Z(SL) <NUM> after each of the first, second, and third sensing cycle, respectively.

After calculating Z(SL), the controller <NUM> may use Z(SL) to reduce, filter, and/or eliminate MPI associated with the ToF sensor data for the corresponding sensing cycle. For purposes of discussion herein, the improved depth information may be referred to as Z(ToF + SL). For example, the controller <NUM> may calculate Z(ToF + SL) <NUM>, Z(ToF + SL) <NUM>, and Z(ToF + SL) <NUM> after calculating each of Z(SL) <NUM>, Z(SL) <NUM>, and Z(SL) <NUM>, respectively. In some aspects, the controller <NUM> may calculate Z(ToF + SL) using one frame of ToF sensor data for the corresponding sensing cycle, or, since ToF sensing techniques are susceptible to noise, the controller <NUM> may calculate Z(ToF + SL) using more than one frame of ToF sensor data for the corresponding sensing cycle. In some implementations, the controller <NUM> may use the first frame of ToF sensor data and the second frame of ToF sensor data to calculate Z(SL) and Z(ToF + SL) any time during the next sensing cycle. As a non-limiting example, the controller <NUM> may average the first frame of ToF sensor data <NUM> with the second frame of ToF sensor data <NUM> to calculate Z(SL) <NUM> and Z(ToF + SL) <NUM> between time <NUM> and time <NUM>. In some aspects, the controller may use frames of ToF sensor data in a different manner to calculate Z(SL) and/or Z(ToF + SL).

In this manner, the system may generate high-resolution and high-accuracy depth information without MPI artifacts using the sparse depth information from the SL mode as a baseline to eliminate multipath effects from the ToF mode. In accordance with the embodiments described herein, the system may generate the depth information without MPI artifacts using a single sensor (e.g., the sensor <NUM>), a pair of emitters (e.g., the emitters <NUM>), and/or a single controller (e.g., the controller <NUM>).

<FIG> shows an example ToF and SL light projection system <NUM>. The system <NUM> includes a first emitter <NUM> and a second emitter <NUM>, which may be example embodiments of the emitter <NUM> and the emitter <NUM> of <FIG>, respectively. The system <NUM> further includes a reflective component <NUM> positioned between the first emitter <NUM> and the second emitter <NUM>. The reflective component <NUM> redirects light projected by the first emitter <NUM> and the second emitter <NUM> onto a scene <NUM>, which may include an object (not shown for simplicity). In some implementations, the system <NUM> may include one or more image capturing devices, such as a ToF camera and a SL camera (not shown for simplicity), for detecting reflections of the redirected light. In some aspects, the ToF camera and the SL camera may include an image sensor, such as the sensor <NUM> of <FIG>. A first DOE <NUM> is coupled to the front of the first emitter <NUM>, and a first diffuser <NUM> is coupled to the front of the first DOE <NUM>. Similarly, a second DOE <NUM> is coupled to the front of the second emitter <NUM>, and a second diffuser <NUM> is coupled to the front of the second DOE <NUM>. The first DOE <NUM>, the first diffuser <NUM>, the second DOE <NUM>, and the second diffuser <NUM> may be example embodiments of the DOE <NUM>, the diffuser <NUM>, the DOE <NUM>, and the diffuser <NUM> of <FIG>, respectively. A first lens (not shown for simplicity) may be situated between the first emitter <NUM> and the first DOE <NUM>, and a second lens (not shown for simplicity) may be situated between the second emitter <NUM> and the second DOE <NUM>. For purposes of discussion herein, the first emitter <NUM>, the first lens, the first DOE <NUM>, and the first diffuser <NUM> may collectively be referred to as a first light projector <NUM>, and the second emitter <NUM>, the second lens, the second DOE <NUM>, and the second diffuser <NUM> may collectively be referred to as a second light projector <NUM>. In some aspects, the first light projector <NUM> and the second light projector <NUM> may be identical (or nearly identical).

The first light projector <NUM> projects a first light towards the second light projector <NUM>, and the second light projector <NUM> projects a second light towards the first light projector <NUM>. In some implementations, the first projected light <NUM> is projected along an axis (not shown for simplicity) in a first direction, and the second projected light <NUM> is projected along the axis in a second direction opposite the first direction. In some aspects, the first and second light projectors may be physically separated by a distance, and the reflective component <NUM> may be positioned along the axis between the first and second light projectors.

The reflective component <NUM> may include a first reflective element <NUM> that optically folds a first optical path of the light projected from the first light projector <NUM> and a second reflective element <NUM> that optically folds a second optical path of the light projected from the second light projector <NUM>. In some implementations, the first reflective element <NUM> may be a first prism including a first reflective surface <NUM> to receive the light projected from the first light projector <NUM>, and the second reflective element <NUM> may be a second prism including a second reflective surface <NUM> to receive the light projected from the second light projector <NUM>. In some other implementations, one or more portions of the reflective component <NUM> may be glass. The reflective component <NUM> redirects the first projected light <NUM> and the second projected light <NUM> onto a first portion of the scene <NUM> and a second portion of the scene <NUM>, respectively. Specifically, the first reflective element <NUM> refracts the light folded by the first reflective element <NUM> onto the first portion of the scene <NUM> based at least in part on the first folded optical path, and the second reflective element <NUM> refracts the light folded by the second reflective element <NUM> onto the second portion of the scene <NUM> based at least in part on the second folded optical path.

In some implementations, the first reflective element <NUM> and the second reflective element <NUM> may have a same (or nearly same) refractive index. Thus, the reflective component <NUM> may symmetrically (or nearly symmetrically) refract the light projected by the first light projector <NUM> and the second light projector <NUM> onto respective portions of the first portion of the scene <NUM> and the second portion of the scene <NUM>. In this manner, the first portion of the scene <NUM> and the second portion of the scene <NUM> may be adjacent to one another. In some implementations, the first portion and the second portion are non-overlapping relative to one another. That is, the first portion of the scene <NUM> and the second portion of the scene <NUM> may be aligned in such a manner that they seamlessly cover the scene <NUM> with a negligible amount (such as less than a first value) of gap and with a negligible amount (such as less than a second value) of overlap.

The example of <FIG> shows four example beams (A, B, C, and D) of light projected from the first light projector <NUM> and four example beams (E, F, G, and H) of light projected from the second light projector <NUM>. For example, the first light projector <NUM> projects beam A, which enters the first reflective element <NUM> at point A. Based on the refractive properties of the first reflective element <NUM>, beam A optically folds to point A' on the first reflective surface <NUM> and exits the first reflective element <NUM> at a first angle (e.g., <NUM>°) towards the scene <NUM>. Similarly, the second light projector <NUM> projects beam E, which enters the second reflective element <NUM> at point E. Based on the refractive properties of the second reflective element <NUM>, beam E optically folds to point E' on the second reflective surface <NUM> and exits the second reflective element <NUM> at the first angle towards the scene <NUM>. Since the first reflective element <NUM> and the second reflective element <NUM> have the same refractive index, points A and A' are symmetrical to points E and E', respectively.

As another example, the first light projector <NUM> projects beam B, which enters the first reflective element <NUM> at point B. Based on the refractive properties of the first reflective element <NUM>, beam B optically folds to point B', then point B", and exits the first reflective element <NUM> at a second angle (e.g., greater than <NUM>°) towards the scene <NUM>. In a symmetrical fashion, the second light projector <NUM> projects beam F, which enters the second reflective element <NUM> at point F. Based on the refractive properties of the second reflective element <NUM>, beam F optically folds to point F', then point F", and exits the second reflective element <NUM> at the second angle towards the scene <NUM>.

Similarly, the first light projector <NUM> projects beam C, which enters the first reflective element <NUM> at point C. Based on the refractive properties of the first reflective element <NUM>, beam C optically folds to point C', then point C", and exits the first reflective element <NUM> at a third angle (e.g., greater than <NUM>°) towards the scene <NUM>. In a symmetrical fashion, the second light projector <NUM> projects beam G, which enters the second reflective element <NUM> at point G. Based on the refractive properties of the second reflective element <NUM>, beam G optically folds to point G', then point G", and exits the second reflective element <NUM> at the third angle towards the scene <NUM>.

Similarly, the first light projector <NUM> projects beam D, which enters the first reflective element <NUM> at point D. Based on the refractive properties of the first reflective element <NUM>, beam D optically folds to point D', then point D", and exits the first reflective element <NUM> at a third angle (e.g., <NUM>°) towards the scene <NUM>. In a symmetrical fashion, the second light projector <NUM> projects beam H, which enters the second reflective element <NUM> at point H. Based on the refractive properties of the second reflective element <NUM>, beam H optically folds to point H', then point H", and exits the second reflective element <NUM> at the third angle towards the scene <NUM>.

Thus, the reflective component <NUM> symmetrically refracts beams A-D and beams E-H, respectively, onto the scene <NUM>. In this manner, beams are projected onto the first portion of the scene <NUM> and the second portion of the scene <NUM>, and the system <NUM> has a wide (e.g., greater than <NUM> degrees) field of view (FOV) of the scene <NUM>.

<FIG> shows an example ToF and SL light projection system <NUM> operating in a first mode (e.g., a ToF mode, as shown on the left) and a second mode (e.g., a SL mode, as shown on the right). The ToF and SL system <NUM> may be a simplified example embodiment of the ToF and SL system <NUM> of <FIG>. That is, the first light projector <NUM>, the second light projector <NUM>, and the reflective component <NUM> may be example embodiments of the first light projector <NUM>, the second light projector <NUM>, and the reflective component <NUM> of <FIG>, respectively. The reflective component <NUM> may redirect light projected by the first light projector <NUM> and the second light projector <NUM> onto a scene <NUM>, which may include an object (not shown for simplicity). In some implementations, the system <NUM> may include one or more image capturing devices, such as a ToF camera and a SL camera (not shown for simplicity), for detecting reflections of the redirected light. In some aspects, the ToF camera and the SL camera may include an image sensor, which may be an example embodiment of the sensor <NUM> of <FIG>. The system <NUM> may also include a controller (not shown for simplicity), such as the active depth controller <NUM> of <FIG>.

In some implementations, each of the first diffuser and the second diffuser may be a switchable diffuser configured to transition the first light projector <NUM> and the second light projector <NUM>, respectively, between a ToF sensing mode and a SL sensing mode. During the ToF mode, the first emitter and the second emitter may operate in a ToF projection mode, and the sensor <NUM> may operate in a ToF sensing mode, as described with respect to <FIG>. During the SL mode, the first emitter and the second emitter may operate in a SL projection mode, and the sensor <NUM> may operate in a SL sensing mode, as also described with respect to <FIG>.

During the ToF mode, the first diffuser and the second diffuser may be switched on, as indicated with solid gray. Thus, when the laser of the first light projector <NUM> emits light through the first DOE, the DOE distribution from the first DOE is diffused when passing through the first diffuser, which provides a flood illumination of a first portion of the scene <NUM>. Similarly, when the laser of the second light projector <NUM> emits light through the second DOE, the DOE distribution from the second DOE is diffused when passing through the second diffuser, which provides a flood illumination of a second portion of the scene <NUM>. Specifically, during the ToF mode, the first emitter transmits a first pulsed signal, and the first diffuser diffuses the first projected light to project a uniform flood distribution onto the first portion of the scene <NUM>.

Similarly, during the ToF mode, the second emitter transmits a second pulsed signal, and the second diffuser diffuses the second projected light to project a uniform flood distribution onto the second portion of the scene <NUM>. That is, the reflective component <NUM> may symmetrically (or nearly symmetrically) refract the light projected by each of the first light projector <NUM> and the second light projector <NUM> such that the portion of the scene upon which light is projected from the first light projector <NUM> (e.g., the first portion of the scene) can be adjacent to, yet non-overlapping with, the portion of the scene upon which light is projected from the second light projector <NUM> (e.g., the second portion of the scene). In this manner, the projections of light from the first and second light projectors <NUM> and <NUM>, as refracted by the reflective component <NUM>, may be aligned in such a manner that they seamlessly cover the scene <NUM> with a negligible amount (such as less than a first value) of gap and with a negligible amount (such as less than a second value) of overlap. Referring back to <FIG>, a first reflected signal and a second reflected signal of the first pulsed signal and the second pulsed signal, respectively, may arrive at the sensor <NUM>, and the sensor <NUM> captures frames including the pulsed signals. The active depth controller <NUM> (such as the signal processor <NUM>) may calculate ToF depth information based on the captured frames by determining an amount of time for the light to be reflected back to the sensor <NUM>.

During the SL mode, the first diffuser and the second diffuser may be switched off (as indicated with white). In some implementations, the first diffuser and the second diffuser function as transparent. Thus, when the laser of the first light projector <NUM> emits light through the first DOE during the SL mode, the DOE distribution from the first DOE passes through the first diffuser unaffected, which projects the first projected light onto the first portion of the scene <NUM> (e.g., as a first distribution of light points, such as a dot matrix pattern). Similarly, when the laser of the second light projector <NUM> emits light through the second DOE during the SL mode, the DOE distribution from the second DOE passes through the second diffuser unaffected, which projects the second projected light onto the second portion of the scene <NUM> (e.g., a second distribution of light points, such as a as a dot matrix pattern). In some implementations, the first distribution of light points and the second distribution of light points may be the same or similar.

Referring back to <FIG>, the sensor <NUM> may capture frames including reflections of the redirected light (where the light is reflected by objects in the scene <NUM>). The active depth controller <NUM> (such as the signal processor <NUM>) may detect, in the captured frames, the redirected light projected by the first light projector <NUM> and the second light projector <NUM> as a first reflected light and a second reflected light, respectively. The active depth controller <NUM> (such as the signal processor <NUM>) may then calculate SL depth information based on the manner in which the first projected light and the second projected light distorts on the first portion of the scene <NUM> and the second portion of the scene <NUM>, respectively.

The active depth controller <NUM> may be configured to generate depth information based on the detected reflections of redirected light and correct for projection distortion in the generated depth information based at least in part on one or more angles of refraction associated with the reflective component <NUM>. The controller <NUM> may use the generated SL depth information during calculation of the ToF depth information to reduce or eliminate multipath artifacts in the ToF depth information.

For instance, projecting light at an angle through a prism, rather than directly at the scene <NUM>, may result in some level (or value) of distortion or skew in the resultant SL depth information. For purposes of discussion herein, the distortion or skew may be referred to as "projection distortion. " In addition, it may take more time for the first projected light <NUM> and the second projected light <NUM> to be redirected through the reflective component <NUM> than if the first projected light <NUM> and the second projected light <NUM> were projected directly towards the scene <NUM>. Thus, ToF depth information generated based on light redirected by the reflective component <NUM> may also include some level (or value) of projection distortion.

In some aspects of the present disclosure, the system <NUM> may be configured to negate or compensate for projection distortion. In some implementations, the reflective component <NUM> may distort the projected light in a known manner. Thus, at least one of the first light projector <NUM> or the second light projector <NUM> may include a predistortion pattern to negate the projection distortion. That is, the predistortion pattern may be incorporated into at least one of the first DOE <NUM> or the second DOE <NUM> such that the respective DOE pattern negates (or "cancels out") the known projection distortion. In addition or in the alternative, the system <NUM> may be configured to digitally compensate for projection distortion after detecting the reflections of the redirected light ("post-capture"), such as by using the active depth controller <NUM> of <FIG>.

In some implementations, the system <NUM> may be included in or coupled to a device, such as a camera or a cellphone. Referring to <FIG>, with device <NUM> as an example device, the active depth controller <NUM> may be configured to identify a face of a user of the device based at least in part on the generated depth information. In some aspects, the controller <NUM> may be further configured to resolve multipath interference (MPI) in a first portion of the generated depth information based at least in part on a second portion of the generated depth information.

<FIG> shows an example electrical circuit diagram for a demodulation pixel cell <NUM>. In some implementations, the demodulation pixel cell <NUM> may be included in or coupled to a sensor, such as the sensor <NUM> of <FIG>, and may be used for generating ToF depth information. The demodulation pixel cell <NUM> may include a photodiode <NUM> coupled to a ground potential <NUM>. The photodiode <NUM> may convert light (e.g., photons) from reflected signals to electrical current, which flows to a transistor <NUM> and a transistor <NUM> coupled in parallel to the photodiode <NUM>. The transistor <NUM> and the transistor <NUM> may block the current from flowing to a capacitor (C1) and a capacitor (C2), respectively. C1 may be coupled to a ground potential <NUM>, and C2 may be coupled to a ground potential <NUM>. In some aspects, at least one of the transistor <NUM> or the transistor <NUM> may be field-effect transistors (FETs). In some aspects, at least one of the transistor <NUM> and the transistor <NUM> may be metal-oxide-semiconductor field-effect transistors (MOSFETs).

For example, during a first exposure cycle, C1 may store a first charge (Q1) from a reflected signal when the shutter is opened at a first phase offset (e.g., Φ<NUM> = <NUM>°) relative to the transmitted signal, and C2 may store a second charge (Q2) from the reflected signal when the shutter is opened at a second phase offset (e.g., Φ<NUM> = <NUM>°) relative to the transmitted signal. During a second exposure cycle, C1 may store a third charge (Q3) from the reflected signal when the shutter is opened at a third phase offset (e.g., Φ<NUM> = <NUM>° relative to the transmitted signal, and C2 may store a fourth charge (Q4) from the reflected signal when the shutter is opened at a fourth phase offset (e.g., Φ<NUM> = <NUM>°) relative to the transmitted signal. The phase offset, ϕ, between the transmitted signal and the reflected signal may be calculated based on the charges stored across C1 and C2 for each of the exposure cycles, which allows calculation of corresponding ToF depth information: <MAT> <MAT> <MAT> where D represents depth information, c is the speed of light (i.e., <MAT>), fmod represents the modulation frequency of the transmitted signal, V<NUM> - V<NUM> represents the integrated electrical signals for Φ<NUM> and Φ<NUM> during the first exposure cycle, V<NUM> - V<NUM> represents the integrated electrical signals for Φ<NUM> and Φ<NUM> during the second exposure cycle, and σdepth represents a depth accuracy. Accordingly, the demodulation pixel cell <NUM> may capture ToF sensor data for generating ToF depth information.

<FIG> shows an example electrical circuit diagram for a global shutter (GS) pixel array <NUM>. The GS pixel array <NUM> may also be referred to herein as an NIR GS imager. The GS pixel array <NUM> includes two shared GS photodiodes, PD1 and PD2. Each of PD1 and PD2 may absorb photons (e.g., from light reflected back from a scene and/or an object) during a SL sensing mode. Each of PD1 and PD2 is coupled to a floating storage diode, SD1 and SD2, respectively. SD1 and SD2 may operate as storage node elements for charge accumulation and readout from the photodiodes PD1 and PD2. Each of the storage diodes SD <NUM> and the SD2 is coupled to a transfer gate, TG1 and TG2, respectively. TG1 and TG2 may be transistors with relatively low voltage drops. Charge from PD1 and PD2 may flow to a transistor LOD1 and a transistor LOD2, respectively, which are each coupled to a supply voltage, Vddpix <NUM>.

The GS pixel array <NUM> includes a capacitor FD for accumulating charge. The capacitor FD is coupled to a transistor TS1, which is coupled to, for example, the storage diode SD1. The capacitor FD is also coupled to a transistor TS2, which is coupled to, for example, the storage diode SD2. The capacitor FD is further coupled to a reset switch, RST. When RST is closed, charge may flow to Vddpix <NUM>. When RST is open, charge may flow to a source follower amplifier, SF_AMP. Because the source voltage of SF_AMP remains proportional to the gate voltage, SF_AMP may convert charge to voltage and toggle a Select switch, SEL. When SEL is open (e.g., during a SL mode, when TS1 and TS2 are open), Vddpix <NUM> may be isolated, and a relatively small amount of charge from each of a series of signal pulses (e.g., a pulse train) may accumulate across the capacitor FD. When SEL is closed (e.g., after each of the SL modes, when TS1 and TS2 are closed), the series of accumulated signal pulses may be transferred from each of TG1 and TG2 to an output terminal, Vout. Vout may be coupled to a current source, I_bias. Accordingly, the GS pixel array <NUM> may capture SL sensor data for generating SL depth information.

<FIG> shows an example electrical circuit diagram for a GS pixel array <NUM>. The GS pixel array <NUM> may also be referred to herein as an NIR GS imager. The GS pixel array <NUM> includes two shared GS photodiodes, PD1 and PD2. Each of PD1 and PD2 may absorb photons (e.g., from light reflected back from a scene and/or an object) during a SL sensing mode. Each of PD1 and PD2 is coupled to a CCD-readout memory, MEM1 and MEM2, respectively. MEM1 and MEM2 may operate as storage node elements for charge accumulation and readout from the photodiodes PD1 and PD2. Each of MEM1 and MEM2 is coupled to a transfer gate, TG1 and TG2, respectively. TG1 and TG2 may be transistors with relatively low voltage drops. Charge from PD1 and PD2 may flow to a transistor LOD1 and a transistor LOD2, respectively, which are each coupled to a supply voltage, Vddpix <NUM>.

The GS pixel array <NUM> includes a capacitor FD for accumulating charge. The capacitor FD is coupled to a transistor TS1, which is coupled to, for example, the CCD-readout memory, MEM1. The capacitor FD is also coupled to a transistor TS2, which is coupled to, for example, the CCD-readout memory, MEM2. The capacitor FD is further coupled to a reset switch, RST. When RST is closed, charge may flow to Vddpix <NUM>. When RST is open, charge may flow to a source follower amplifier, SF_AMP. Because the source voltage of SF_AMP remains proportional to the gate voltage, SF_AMP may convert charge to voltage and toggle a Select switch, SEL. When SEL is open (e.g., during a SL mode, when TS1 and TS2 are open), Vddpix <NUM> may be isolated, and a relatively small amount of charge from each of a series of signal pulses (e.g., a pulse train) may accumulate across the capacitor FD. When SEL is closed (e.g., after each of the SL modes, when TS1 and TS2 are closed), the series of accumulated signal pulses may be transferred from each of TG1 and TG2 to an output terminal, Vout. Vout may be coupled to a current source, I_bias. Accordingly, the GS pixel array <NUM> may capture SL sensor data for generating SL depth information.

<FIG> shows an example electrical circuit diagram for a rolling shutter (RS) pixel array <NUM>. The pixel array <NUM> may also be referred to herein as a hybrid NIR RS imager and may be capable of operating in a ToF sensing mode and a SL sensing mode. The RS pixel array <NUM> may be an example embodiment of the sensor <NUM> of <FIG>. The pixel array <NUM> may be configured to read-out signal line-by-line and thus, to operate in a constant wave mode (e.g., at a particular duty cycle) so as to expose each line of the RS for an equal amount of time. The RS pixel array <NUM> includes four shared RS photodiodes, PD1-PD4. Each of PD1-PD4 may absorb photons (e.g., from light reflected back from a scene and/or an object) during a SL sensing mode. PD1 is coupled to two transfer gates, TG1 and TG2. PD2 is coupled to two transfer gates, TG3 and TG4. PD3 is coupled to two transfer gates, TG5 and TG6. PD4 is coupled to two transfer gates, TG7 and TG8. Each of TG1-TG8 may be a transistor with a relatively low voltage drop.

The RS pixel array <NUM> includes capacitors FD1 and FD2 for accumulating charge from reflected signals. Each of FD1 and FD2 is coupled to a reset switch, RST1 and RS2, respectively. When either of RST1 and RS2 is closed, charge may flow to Vddpix <NUM> and Vddpix <NUM>, respectively. When either of RST1 and RST2 is open, charge may flow to a source follower amplifier, SF_AMP1 and SF_AMP2, respectively. Because the source voltage of SF_AMP1 and SF_AMP2 remains proportional to the gate voltage, SF_AMP1 and SF_AMP2 may convert charge to voltage and toggle a corresponding Select switch, SEL1 and SEL2, respectively.

During the ToF sensing mode, each of TG1-TG8 may be closed (activated) and the pixel array <NUM> may demodulate multiple phases of reflected signals. SEL1 and SEL2 may be open during the ToF sensing mode, which may isolate Vddpix <NUM> and Vddpix <NUM>, allowing a relatively small amount of charge from each of a series of signal pulses (e.g., a pulse train) to accumulate across FD1 and FD2. When SEL1 is closed, a series of accumulated signal pulses may be transferred from TG1, TG3, TG5, and TG7 to output terminal, Vout1. When SEL2 is closed, a series of accumulated signal pulses may be transferred from TG2, TG4, TG6, and TG8 to output terminal, Vout2. Vout1 may be coupled to a current source, I_bias1, and Vout2 may be coupled to a current source, I_bias2. Accordingly, the RS pixel array <NUM> may capture ToF sensor data for generating ToF depth information.

During the SL sensing mode, each of TG1, TG4, TG5, and TG8 (e.g., half of the read-out circuitry) may be closed, and each of TG2, TG3, TG6, and TG7 (e.g., the other half of the read-out circuitry) may be open. In this manner, reflected signals may be captured in dual-phase (e.g., one on the left, and one on the right) at different time frames. SEL1 and SEL2 may also be open during the SL sensing mode, which may isolate Vddpix <NUM> and Vddpix <NUM>, allowing a relatively small amount of charge from each of a series of signal pulses (e.g., a pulse train) to accumulate across FD1 and FD2.

In this manner, the pixel array <NUM> may operate as a hybrid RS sensor for a mixed-mode ToF and SL system for generating high-resolution and high-accuracy depth information without MPI artifacts using the sparse depth information from the SL mode as a baseline to eliminate multipath effects from the ToF mode.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. At block <NUM>, the device projects light from a first light projector of the device towards a second light projector of the device. At block <NUM>, the device projects light from the second light projector towards the first light projector. At block <NUM>, the device redirects, via a reflective component positioned between the first and second light projectors, the light projected by the first light projector onto a first portion of a scene and the light projected by the second light projector onto a second portion of the scene, the first and second portions of the scene being adjacent to one another and non-overlapping relative to one another. At block <NUM>, the device detects reflections of redirected light projected by the first and second light projectors.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In some implementations, the process <NUM> begins after the process <NUM> described with reference to <FIG>. For example, at block <NUM>, after detecting the reflections of the redirected light projected by the first and second light projectors in block <NUM> of the process <NUM>, the device generates depth information based on the detected reflections of redirected light, as described with respect to <FIG>. At block <NUM>, the device corrects for projection distortion in the generated depth information based at least in part on one or more angles of refraction associated with the reflective component, as also described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In some implementations, the process <NUM> begins after the process <NUM> described with reference to <FIG>. For example, at block <NUM>, after correcting for the projection distortion in block <NUM> of the process <NUM>, the device identifies a face of a user of the device based at least in part on the generated depth information, as described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In some implementations, the process <NUM> begins after generating the depth information in block <NUM> of the process <NUM>. At block <NUM>, the device resolves multipath interference (MPI) in a first portion of the generated depth information based at least in part on a second portion of the generated depth information, as described with respect to <FIG> and <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In various aspects, the example process <NUM> of <FIG> may be one implementation for redirecting the light in block <NUM> of the process <NUM> of <FIG>. At block <NUM>, the device symmetrically refracts the light projected by the first and second light projectors onto respective portions of the first and second portions of the scene, as described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In various aspects, the example process <NUM> of <FIG> may be one implementation for redirecting the light in block <NUM> of the process <NUM> of <FIG>. At block <NUM>, the device optically folds, via a first reflective element of the reflective component, a first optical path of the light projected from the first light projector, as described with respect to <FIG>. At block <NUM>, the device optically folds, via a second reflective element of the reflective component, a second optical path of the light projected from the second light projector, as also described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In some implementations, the process <NUM> begins after optically folding the first and second optical paths in blocks <NUM> and <NUM> of the process <NUM> described with reference to <FIG>. At block <NUM>, the device refracts the light folded by the first reflective element onto the first portion of the scene based at least in part on the first folded optical path, as described with respect to <FIG>. At block <NUM>, the device refracts the light folded by the second reflective element onto the second portion of the scene based at least in part on the second folded optical path, as also described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In some implementations, the process <NUM> begins before optically folding the first and second optical paths in blocks <NUM> and <NUM> of the process <NUM> described with reference to <FIG>. At block <NUM>, the device receives, via a first reflective surface of a first prism of the first reflective element, the light projected from the first light projector, as described with respect to <FIG>. At block <NUM>, the device receives, via a second reflective surface of a second prism of the second reflective element, the light projected from the second light projector, as also described with respect to <FIG>.

<FIG> shows a flowchart illustrating an example process <NUM> for depth sensing that may be performed by a device, according to some implementations. The process <NUM> may be performed by a device such as the device <NUM> described above with reference to <FIG>. In various aspects, the example process <NUM> of <FIG> may be one implementation for projecting the light from one or more of the first and second light projectors in one or more of blocks <NUM> and <NUM> of the process <NUM> of <FIG>. At block <NUM>, the device transitions, via a switchable diffuser provided within each light projector of the first and second light projectors, the respective light projector between a structured light (SL) sensing mode and a time-of-flight (ToF) sensing mode, as also described with respect to <FIG>.

The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium (such as the memory <NUM> in the device <NUM> of <FIG>) including instructions <NUM> that, when executed by the processor <NUM> (or the active depth controller <NUM>), cause the device <NUM> to perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

The non-transitory processor-readable storage medium may include random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, other known storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer or other processor.

The various illustrative logical blocks, modules, circuits and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors, such as the processor <NUM> or the active depth controller <NUM> in the device <NUM> of <FIG>. Such processor(s) may include but are not limited to one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The term "processor," as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured as described herein.

Claim 1:
A depth sensing device (<NUM>), comprising:
a first light projector (<NUM>) and a second light projector (<NUM>), the first light projector (<NUM>) configured to project light (<NUM>) towards the second light projector (<NUM>), the second light projector (<NUM>) configured to project light (<NUM>) towards the first light projector (<NUM>);
a reflective component (<NUM>) positioned between the first and second light projectors (<NUM>, <NUM>), the reflective component (<NUM>) configured to redirect the light (<NUM>) projected by the first light projector (<NUM>) onto a first portion (<NUM>) of a scene (<NUM>) and to redirect the light (<NUM>) projected by the second light projector (<NUM>) onto a second portion (<NUM>) of the scene (<NUM>), wherein the first and second portions of the scene (<NUM>, <NUM>) are adjacent to one another and are non-overlapping relative to one another;
a receiver configured to detect reflections of redirected light projected by the first and second light projectors (<NUM>, <NUM>); and
a controller configured to generate depth information based on the detected reflections of redirected light.