Patent Description:
A device may determine depths or distances of its surroundings using different active depth sensing systems. In determining depths or distances of objects from the device, the device may transmit one or more wireless signals and measure reflections of the wireless signals. The device may then use the reflections to generate a depth map illustrating or otherwise indicating the depths of objects from the device. One depth sensing system is a structured light system that transmits a known distribution of light.

For example, a structured light system may transmit light in a known distribution or pattern of points. The light may be near-infrared (NIR) signals or other frequency signals of the electromagnetic spectrum. The reflections of the transmitted light may be captured, and the captured signals may be processed in determining depths of objects from the structured light system. Constraints on the resolution and the transmission power of the transmitted distribution for conventional structured light systems limit the accuracy and effective operation for active depth sensing.

US patent application <CIT> discloses a receiver sensor to capture a plurality of images, at two or more different exposure times, of a scene onto which a code mask is projected. Two or more of the plurality of images are combined by extracting decodable portions of the code mask from each image to generate a combined image. Depth information for the scene is then ascertained based on the combined image using the code mask.

<CIT> discloses an electronic device that may increase the frequency of depth image capture so as to more rapidly accumulate sufficient depth data for the local environment through which it is travelling.

The article "<NPL> teaches dividing an image into several zones and calculating an optimal exposure time for each zone. Fringe images are captured using the optimal exposure time of each zone and combined in image fusion.

US patent <CIT> discloses a real-time structured light depth extraction system including a projector, a camera, and an image processor wherein the camera samples light reflected from an object during projection of first and second structured light patterns. To cope with saturation of light-receiving cells in the camera, the integration time is varied in accordance with the intensity of the reflected signal.

This Summary is provided to introduce in a simplified form a selection of concepts that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

The present disclosure provides a method for active depth sensing according to claim <NUM>, a device for active depth sensing according to claim <NUM>, and a non-transitory computer-readable medium according to claim <NUM>. Specific embodiments are subject of the dependent claims.

Aspects of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements.

Aspects of the present disclosure relate to active depth sensing systems. A structured light system may transmit light in a predefined distribution of points (or another suitable shape of focused light, such as lines, arcs, etc.). The points of light may be projected on to a scene, and the reflections of the points of light may be received by the structured light system. Depths of objects in a scene may be determined by comparing the pattern of the received light and the pattern of the transmitted light. In comparing the patterns, a portion of the predefined distribution for the transmitted light may be identified in the received light.

<FIG> is a depiction of an example structured light system <NUM>. The structured light system <NUM> may be used in generating a depth map (not pictured) or otherwise determining depths of objects in a scene <NUM>. The structured light system <NUM> may include at least a projector or transmitter <NUM> and a receiver <NUM>. The projector or transmitter <NUM> may be referred to as a "transmitter," "projector," "emitter," and so on, and should not be limited to a specific transmission component. Similarly, the receiver <NUM> may also be referred to as a "detector," "sensor," "sensing element," "photodetector," and so on, and should not be limited to a specific receiving component.

The transmitter <NUM> may be configured to project a codeword distribution <NUM> of light points (or other suitable distribution and type of focused light) onto the scene <NUM>. In some example implementations, the transmitter <NUM> may include one or more laser sources <NUM>, a lens <NUM>, and a light modulator <NUM>. The transmitter <NUM> also may include an aperture <NUM> from which the transmitted light escapes the transmitter <NUM>. In some implementations, the transmitter <NUM> further may include a diffractive optical element (DOE) to diffract the emissions from one or more laser sources <NUM> into additional emissions. In some aspects, the light modulator <NUM> (to adjust the intensity of the emission) may comprise a DOE. The codeword distribution <NUM> may be hardcoded on the structured light system <NUM> (e.g., at the transmitter <NUM>) so that the pattern and other characteristics of the codeword distribution <NUM> do not vary. In projecting the codeword distribution <NUM> of light points onto the scene <NUM>, the transmitter <NUM> may transmit one or more lasers from the laser source <NUM> through the lens <NUM> (and/or through a DOE or light modulator <NUM>) and onto the scene <NUM>. The transmitter <NUM> may be positioned on the same reference plane as the receiver <NUM>, and the transmitter <NUM> and the receiver <NUM> may be separated by a distance called the baseline (<NUM>).

The scene <NUM> may include objects at different depths from the structured light system (such as from the transmitter <NUM> and the receiver <NUM>). For example, objects 106A and 106B in the scene <NUM> may be at different depths. The receiver <NUM> may be configured to receive, from the scene <NUM>, reflections <NUM> of the transmitted codeword distribution <NUM> of light points. To receive the reflections <NUM>, the receiver <NUM> may capture an image frame (frame). When capturing the frame, the receiver <NUM> may receive the reflections <NUM>, as well as (i) other reflections of the codeword distribution <NUM> of light points from other portions of the scene <NUM> at different depths and (ii) ambient light. Noise also may exist in the captured frame.

In some example implementations, the receiver <NUM> may include a lens <NUM> to focus or direct the received light (including the reflections <NUM> from the objects 106A and 106B) on to the sensor <NUM> of the receiver <NUM>. The receiver <NUM> also may include an aperture <NUM> to restrict the direction from which the receiver <NUM> may receive light. Assuming for the example that only the reflections <NUM> are received, depths of the objects 106A and 106B may be determined based on the baseline <NUM>, displacement and distortion of the codeword distribution <NUM> in the reflections <NUM>, and intensities of the reflections <NUM>. For example, the distance <NUM> along the sensor <NUM> from location <NUM> to the center <NUM> may be used in determining a depth of the object 106B in the scene <NUM>. Similarly, the distance <NUM> along the sensor <NUM> from location <NUM> to the center <NUM> may be used in determining a depth of the object 106A in the scene <NUM>. The distance along the sensor <NUM> may be measured in terms of number of pixels of the sensor <NUM> or a distance (such as millimeters).

In some example implementations, the sensor <NUM> may include an array of photodiodes (such as avalanche photodiodes) for capturing an image. To capture the image, each photodiode in the array may capture the light that hits the photodiode and may provide a value indicating the intensity of the light (a capture value). The frame therefore may be the capture values provided by the array of photodiodes.

In addition or alternative to the sensor <NUM> including an array of photodiodes, the sensor <NUM> may include a complementary metal-oxide semiconductor (CMOS) sensor or other suitable type of sensor. To capture the image by a photosensitive CMOS sensor, each pixel of the sensor may capture the light that hits the pixel and may provide a value indicating the intensity of the light. In some examples, the light intensity may be indicated in terms of luminance, lumens, lumens per square meter (lux), or other suitable measurement of light intensity. In some example implementations, an array of photodiodes may be coupled to the CMOS sensor. In this manner, the electrical impulses generated by the array of photodiodes may trigger the corresponding pixels of the CMOS sensor to provide capture values for a frame.

The sensor <NUM> may include at least a number of pixels equal to the number of light points in the codeword distribution <NUM>. For example, the array of photodiodes or the CMOS sensor may include at least a number of photodiodes or a number of pixels, respectively, corresponding to the number of light points in the codeword distribution <NUM>. The sensor <NUM> logically may be divided into groups of pixels or photodiodes that correspond to a size of a bit of the codeword distribution if the transmitted distribution is logically divided into codewords. The group of pixels or photodiodes also may be referred to as a bit, and the portion of the captured frame from a bit of the sensor <NUM> also may be referred to as a bit. In some example implementations, the sensor <NUM> may include the same number of bits as the codeword distribution <NUM>.

As illustrated, the distance <NUM> (corresponding to the reflections <NUM> from the object 106B) is less than the distance <NUM> (corresponding to the reflections <NUM> from the object 106A). Using triangulation based on the baseline <NUM> and the distances <NUM> and <NUM>, the differing depths of objects 106A and 106B in the scene <NUM> may be determined. However, in order to be able to determine the distance <NUM> and the distance <NUM>, the portion of the codeword distribution <NUM> in the reflections <NUM> needs to be identified. If a sufficient number of points in the codeword distribution <NUM> are not recognized in the portions of codeword distribution <NUM> included in the reflections <NUM>, the portions of the codeword distribution <NUM> in the reflections may not be identified. For example, the amount of ambient light received when capturing the frame by the receiver <NUM> may obfuscate the codeword distribution <NUM> in the reflections <NUM>.

A structured light transmitter may include one or more laser diodes to transmit light with a wavelength of <NUM> nanometers (nm) or <NUM> (or another suitable wavelength in the NIR spectrum). As stated above, a structured light receiver may capture reflections of the light distribution transmitted by the structured light transmitter and ambient light not associated with the transmitted light distribution. The captured frame also may include noise.

<FIG> is a depiction of an example captured frame <NUM> including light from the scene <NUM>, an example reflection of the light distribution <NUM>, and an example noise <NUM>. If the intensity of the light from the scene <NUM> (ambient light) increases, the light distribution <NUM> may be more difficult to identify in the frame <NUM>. <FIG> is a depiction of an example captured frame <NUM> including light from the scene <NUM>, the example reflection of the light distribution <NUM> (from <FIG>), and the example noise <NUM> (from <FIG>). The intensity of the light in the scene <NUM> is greater than the intensity of the light in the scene <NUM>. As a result, portions of the light distribution <NUM> in the frame <NUM> may be more difficult to identify.

The amount of ambient light may affect the ability to determine depths using a structured light system, and a structured light system may be more effective in low light scenarios (such as indoors or at nighttime) than in bright light scenarios (such as outdoors during sunny days). A device may compensate for ambient light by including a light filter in the structured light receiver. Referring back to <FIG>, if the transmitter <NUM> transmits light with a <NUM> wavelength, the receiver <NUM> may include a filter before the lens <NUM> that blocks light with a wavelength outside of a range around <NUM>. For example, the filter may block light with a wavelength outside of an example range of <NUM> to <NUM>. However, ambient light still exists within the range of wavelengths that may interfere with operation of the structured light system.

Additionally or alternatively, a device may compensate for ambient light by adjusting the exposure time of the structured light receiver sensor. Referring back to <FIG>, the amount of time that each pixel of the sensor <NUM> receives light for a frame capture may be adjustable. If the exposure time is increased, the amount of light received by a pixel during the exposure time for a frame capture increases (and the measured overall light intensity for the pixel increases).

<FIG> is a depiction of an example graph <NUM> illustrating measured overall light intensity <NUM> in relation to exposure time <NUM> for a pixel. The overall light intensity the pixel measures increases more quickly in relation to the exposure time if the light includes light from the light distribution reflection (<NUM>) than if the light does not include light from the light distribution reflection (<NUM>). The measured overall light intensity without a light distribution reflection may be from the ambient light. The slope of lines <NUM> and <NUM> may increase as the amount of ambient light increases.

For depth sensing, the difference in overall measured light intensity with and without the light distribution reflection may be used to determine whether light of the reflected light distribution is received at the pixel. For example, a device may use an intensity threshold to determine if a light point exists at the pixel. The intensity threshold for an exposure time may be between the light intensity value without a light distribution reflection and the light intensity value with a light distribution reflection (such as between lines <NUM> and <NUM>). In this manner, the device may determine that a light point of the light distribution reflection is received at the pixel when the measured overall light intensity is greater than the intensity threshold, and the device may determine that a light point of the light distribution reflection is not received at the pixel when the measured overall light intensity is less than the intensity threshold. The threshold may be based on the exposure time for the pixel.

The difference between the measured overall light intensity with and without the light distribution reflection may need to be sufficiently large for a device to accurately identify a light point at the pixel using the intensity threshold. In this manner, the structured light receiver may be configured to have a minimum exposure time or a fixed exposure time.

Since the measured overall light intensity <NUM> without a light distribution reflection (<NUM>) may be from ambient light, the slope of lines <NUM> and <NUM> may increase as the amount of ambient light increases. The difference in overall measured light intensity <NUM> between the two lines <NUM> and <NUM> may remain the same as the amount of ambient light increases. <FIG> is a depiction of an example graph <NUM> illustrating measured overall light intensity <NUM> in relation to exposure time <NUM> for a pixel when more ambient light exists than for the graph <NUM> in <FIG>. As shown, the slope of the line <NUM> is greater than the slope of the line <NUM> (in <FIG>), and the slope of the line <NUM> is greater than the slope of the line <NUM> (in <FIG>). The difference between the measured overall light intensity <NUM> with and without the light distribution reflection (<NUM> and <NUM>, respectively) and the difference between the measured overall light intensity <NUM> (in <FIG>) with and without the light distribution reflection (<NUM> and <NUM>, respectively, in <FIG>) may be approximately the same for the same exposure time. The magnitude of the measured overall light intensity <NUM>, though, is greater than the magnitude of the measured overall light intensity <NUM> in <FIG> for the same exposure time.

For the measured overall light intensity for a pixel, the light distribution reflection intensity portion reduces in relation to the ambient light intensity portion as the ambient light increases. As a result, increased ambient light may cause the device to have more difficulty in identifying whether a light point exists at the pixel, and the device may make more errors in determining where light from the light distribution is received at the structured light receiver sensor.

A device may attempt to compensate for an increase in ambient light by increasing the transmission power for the structured light transmitter. In this manner, the intensity of the light transmission may be increased, thus increasing the intensity of the reflections received by the structured light receiver. However, eye safety concerns and various regulations may limit the intensity of the structured light transmissions. Further, increased transmission power requires additional power, which may be undesirable for power constrained devices (such as battery powered devices, including smartphones, tablets, and other handheld devices).

In addition or alternative to increasing the transmission power for the structured light transmitter, the exposure time for the structured light receiver sensor when capturing a frame may be increased when the ambient light increases. However, each sensor pixel or photodiode may be able to measure a defined range of light intensities. If the exposure time is increased, the overall intensity of the light received by a pixel over the exposure time increases. As a result, the pixel (such as a photodiode or CMOS sensor pixel) may become saturated (reaching the upper limit of the range of intensities able to be measured for the pixel) before reaching the end of the exposure time. In addition, the ambient light may be significant enough to saturate pixels of the structured light receiver sensor even when the exposure time is fixed. For example, depth sensing using a structured light system in bright daylight may be ineffective as a result of the amount of ambient light. The device may not receive an accurate measurement of the overall intensity, and the device is unable to determine whether light from the light distribution is received at the pixel.

For frames captured with scenes having more ambient light, the device may use larger codewords to identify portions of the light distribution in the frame. The light distribution logically may be divided into a plurality of portions (such as codewords). The codewords may be larger (include more points of light) to increase the probability of identifying a codeword in the frame. Alternatively, the codewords may be smaller (include less points of light) to increase the number of depths that may be determined for the scene (to increase resolution). Smaller codewords are considered finer scale codewords than larger codewords (coarser scale codewords). When the measured light intensities from the captured frame are compared to the predefined light distribution transmitted by the structured light transmitter, codewords of one size or scale are attempted to be identified in the received light. The size of the codewords causes a trade-off between reliably identifying portions of the light distribution by using coarser scale codewords and increasing the resolution by using finer scale codewords. Since larger codewords include more light points than smaller codewords, more tolerance for errors in identifying light points in the frame exists when identifying larger codewords than when identifying smaller codewords. However, the resolution decreases as the size of the codewords increase. Further, the ambient light may be significant enough that even the largest codewords for the light distribution are not identifiable. For example, a significant portion of the structured light receiver sensor may be saturated when capturing a frame, thus preventing any size codewords from being identified in the saturated portions of the frame.

In the present disclosure, a sensor exposure time or the frame capture rate for a structured light receiver are adjustable. A device increases the frame capture rate, thus reducing the exposure time for the sensor when the amount of ambient light increases. For example, the structured light receiver may have a higher frame capture rate (thus shortening the exposure time of the sensor for each frame capture) when the structured light system is used in bright daylight than when used indoors or during nighttime. If the frame capture rate for the structured light receiver is increased (thus decreasing the exposure time per frame capture), the device aggregates the measured overall light intensities across multiple frames. The device generates an aggregated frame with aggregated light intensity measurements across multiple frames. In this manner, the device may increase in the aggregated frame the difference between measured overall light intensities with and without the light distribution, and individual frames of the aggregate frame may include a short enough exposure time to prevent saturating portions of the structured light receiver sensor. In some example implementations, a device may measure the amount of ambient light to determine when and/or how much to increase the frame capture rate. In some examples, the increased frame capture rate may be an integer multiple of a base frame capture rate for the structured light receiver. In some further examples, the frame captures at an increased rate of the structured light receiver (which may include a NIR sensor) may be synchronized with frame captures from a camera sensor (such as an RGB sensor).

In the following description, numerous specific details are set forth, such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term "coupled" as used herein means connected directly to or connected through one or more intervening components or circuits. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the teachings disclosed herein. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring teachings of the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processes, and other symbolic representations of operations on data bits within a computer memory. In the present disclosure, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system.

Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present application, discussions utilizing the terms such as "accessing," "receiving," "sending," "using," "selecting," "determining," "normalizing," "multiplying," "averaging," "monitoring," "comparing," "applying," "updating," "measuring," "deriving," "settling" or the like, refer to the actions and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

In the figures, a single block may be described as performing a function or functions; however, in actual practice, the function or functions performed by that block may be performed in a single component or across multiple components, and/or may be performed using hardware, using software, or using a combination of hardware and software. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are described below generally in terms of their functionality. Also, the example devices may include components other than those shown, including well-known components such as a processor, memory and the like.

Aspects of the present disclosure are applicable to any suitable electronic device (such as security systems, smartphones, tablets, laptop computers, vehicles, drones, or other devices) including or coupled to one or more structured light systems. While described below with respect to a device having or coupled to one structured light system, aspects of the present disclosure are applicable to devices having any number of structured light systems, and are therefore not limited to specific devices.

The term "device" is not limited to one or a specific number of physical objects (such as one smartphone, one controller, one processing system, etc.). As used herein, a device may be any electronic device with one or more parts that may implement at least some portions of this disclosure. While the below description and examples use the term "device" to describe various aspects of this disclosure, the term "device" is not limited to a specific configuration, type, or number of objects. Additionally, the term "system" is not limited to multiple components or specific embodiments. For example, a system may be implemented on one or more printed circuit boards or other substrates, and may have movable or static components. While the below description and examples use the term "system" to describe various aspects of this disclosure, the term "system" is not limited to a specific configuration, type, or number of objects.

<FIG> is a block diagram of an example device <NUM> including a structured light system (such as the structured light system <NUM> in <FIG>). In some other examples, the structured light system may be coupled to the device <NUM>. The example device <NUM> may include or be coupled to a transmitter <NUM> (such as transmitter <NUM> in <FIG>) and a receiver <NUM> (such as receiver <NUM> in <FIG>). The transmitter <NUM> and the receiver <NUM> may be separated by a baseline <NUM>. The example device <NUM> also may include a processor <NUM>, a memory <NUM> storing instructions <NUM>, and a camera controller <NUM> (which may include one or more image signal processors <NUM>). The device <NUM> optionally may include (or be coupled to) a display <NUM>, a number of input/output (I/O) components <NUM>, and a camera <NUM>. The device <NUM> may include additional features or components not shown. For example, a wireless interface, which may include a number of transceivers and a baseband processor, may be included for a wireless communication device.

The transmitter <NUM> and the receiver <NUM> may be part of a structured light system (such as the structured light system <NUM> in <FIG>) controlled by the camera controller <NUM> and/or the processor <NUM>. The device <NUM> may include or be coupled to additional structured light systems or may include a different configuration for the structured light system. For example, the device <NUM> may include or be coupled to additional receivers (not shown) for capturing multiple frames of a scene at different perspectives. The disclosure should not be limited to any specific examples or illustrations, including the example device <NUM>.

The transmitter <NUM> may be configured to transmit a distribution of light in the NIR range. For example, the transmitter <NUM> may include a laser to transmit light with a wavelength of <NUM> or <NUM>. The receiver <NUM> may include a NIR sensor for capturing frames. The receiver <NUM> may have a base exposure time for frame capture or may have a base frame capture rate, and the exposure time and/or the frame capture rate are adjustable (such as increased from a base frame capture rate).

The memory <NUM> may be a non-transient or non-transitory computer readable medium storing computer-executable instructions <NUM> to perform all or a portion of one or more operations described in this disclosure. The device <NUM> also may include a power supply <NUM>, which may be coupled to or integrated into the device <NUM>.

The processor <NUM> may be one or more suitable processors capable of executing scripts or instructions of one or more software programs (such as instructions <NUM>) stored within the memory <NUM>. In some aspects, the processor <NUM> may be one or more general purpose processors that execute instructions <NUM> to cause the device <NUM> to perform any number of functions or operations. In additional or alternative aspects, the processor <NUM> may include integrated circuits or other hardware to perform functions or operations without the use of software. While shown to be coupled to each other via the processor <NUM> in the example of <FIG>, the processor <NUM>, the memory <NUM>, the camera controller <NUM>, the optional display <NUM>, and the optional I/O components <NUM> may be coupled to one another in various arrangements. For example, the processor <NUM>, the memory <NUM>, the camera controller <NUM>, the optional display <NUM>, and/or the optional I/O components <NUM> may be coupled to each other via one or more local buses (not shown for simplicity).

The display <NUM> may be any suitable display or screen allowing for user interaction and/or to present items (such as a depth map or a preview image of the scene) for viewing by a user. In some aspects, the display <NUM> may be a touch-sensitive display. The I/O components <NUM> may be or include any suitable mechanism, interface, or device to receive input (such as commands) from the user and to provide output to the user. For example, the I/O components <NUM> may include (but are not limited to) a graphical user interface, keyboard, mouse, microphone and speakers, squeezable bezel or border of the device <NUM>, physical buttons located on device <NUM>, and so on. The display <NUM> and the I/O components <NUM> may provide a preview image or depth map of the scene to a user and/or receive a user input for adjusting one or more settings of the device <NUM> (such as adjusting the intensity of the emissions by transmitter <NUM>, adjusting the scale of the codewords used for determining depths, adjusting the frame capture rate of the receiver <NUM>, and so on).

The camera controller <NUM> may include an image signal processor <NUM>, which may be one or more processors to process the frames captured by the receiver <NUM>. The image signal processor <NUM> also may control the transmitter <NUM> (such as control the transmission power) and control the receiver <NUM> (such as control the frame capture rate or control the exposure time for a frame capture). In some aspects, the image signal processor <NUM> may execute instructions from a memory (such as instructions <NUM> from the memory <NUM> or instructions stored in a separate memory coupled to the image signal processor <NUM>). In other aspects, the image signal processor <NUM> may include specific hardware for operation. The image signal processor <NUM> may alternatively or additionally include a combination of specific hardware and the ability to execute software instructions.

If the device <NUM> includes the optional camera <NUM>, the camera controller <NUM> (such as the image signal processor <NUM>) may control the camera <NUM> and process image frames captured by the camera <NUM>. In some alternative example implementations of the device <NUM>, the processor <NUM> or another suitable device component may be configured to control the camera <NUM> and/or process image frames captured by the camera <NUM>. For example, a separate camera controller (not shown) may include one or more image signal processors to process the image frames and control the camera <NUM>.

The camera <NUM> may be oriented to have approximately the same field of capture as the receiver <NUM>. For example, a smartphone may include a camera <NUM> and a receiver <NUM> on the front side of the smartphone with the display <NUM>, and the camera <NUM> and the receiver <NUM> may be oriented in the same direction. In some example implementations, the camera <NUM> includes an image sensor for capturing visible light (such as an RGB sensor).

The following examples are described in relation to the device <NUM> to explain some concepts in the present disclosure. However, other suitable devices may be used in performing aspects of the present disclosure, and the present disclosure should not be limited to a specific device or configuration of components, including device <NUM>.

<FIG> is an illustrative flow chart depicting an example operation <NUM> for adjusting an exposure time for frame capture of a structured light receiver sensor (such as a sensor of the receiver <NUM> in <FIG>). Beginning at <NUM>, the device <NUM> determines an amount of ambient light for the scene to be captured by the receiver <NUM>. The device <NUM> then adjusts the exposure time of the sensor for the receiver <NUM> for one or more frame captures based on the amount of ambient light, wherein the exposure time is inversely related to the amount of ambient light (<NUM>). In this manner, the device <NUM> may reduce the exposure time when the ambient light increases, and may increase the exposure time when the ambient light decreases.

If the exposure time for frame capture is decreased, the difference between measurements (in a captured frame) with and without reflections of the light distribution from the transmitter <NUM> is smaller. For example, when the exposure time is decreased, a sensor pixel receiving reflected light from the light distribution from the transmitter <NUM> (such as light from a light point of the distribution) measures a smaller intensity difference from a sensor pixel not receiving reflected light from the light distribution. Referring back to <FIG>, the difference between lines <NUM> and <NUM> or lines <NUM> and <NUM> decrease when moving left (reducing the exposure time <NUM> or <NUM>) in the graph <NUM> or <NUM>, respectively.

The device <NUM> may amplify the smaller differences in the measurements to identify which portions of a captured frame include reflections of the light distribution. In some example implementations, the device <NUM> may aggregate multiple captured frames into an aggregated frame to amplify the differences. <FIG> is an illustrative flow chart depicting an example operation <NUM> for generating an aggregated frame. Beginning at <NUM>, the device <NUM> receives a plurality of captured frames from the structured light receiver based on the adjusted exposure time. For example, if the device <NUM> decreases the exposure time for each frame capture based on an increase in ambient light (<NUM> in <FIG>), the structured light receiver <NUM> may capture frames using the shortened exposure time, and the device <NUM> may receive the captured frames from the receiver <NUM>.

The device <NUM> aggregates values across the plurality of captured frames in generating the aggregated frame (<NUM>). A pixel value at a corresponding location in the plurality of frames is added across the plurality of frames. In this manner, the aggregated frame includes aggregated values across the plurality of frames for each pixel. As a result, the difference between measurements with and without light distribution reflections may be amplified for the device <NUM> to more easily identify the light distribution reflections. Other suitable processes for amplifying the difference in measurements may be used, and the present disclosure should not be limited to the examples in <FIG>. For example, the device may multiply the values in a captured frame by a constant value (thus logically aggregating multiple instances of the same captured frame. In another example, the captured frames may be sufficient without requiring aggregation.

Referring back to <FIG>, in some example implementations of determining an amount of ambient light (<NUM>), the device <NUM> may determine if the ambient light saturates the receiver sensor based on one or more captured frames from the receiver <NUM>. Additionally, the device <NUM> may determine the extent of the sensor saturation. <FIG> is an illustrative flow chart depicting an example operation <NUM> for determining a saturation of the receiver sensor for frame capture. Beginning at <NUM>, the device <NUM> (such as the camera controller <NUM> or the image signal processor <NUM>) may receive one or more frames from the receiver <NUM>. The device <NUM> may identify, for each of the one or more frames, whether one or more portions of the frame are saturated (<NUM>). For example, the device <NUM> may determine, for each pixel of a frame, if the pixel is saturated (<NUM>). In one example, a pixel may be determined to be saturated if the pixel value in the frame is a maximum intensity value. In another example, a pixel may be determined to be saturated if the pixel has a maximum intensity value and a number of neighboring pixels have a maximum intensity value. As an illustrative example, if the frame values may be in a range from <NUM> to <NUM>, the device <NUM> may determine that a pixel is saturated if the pixel value (and optionally a number of neighboring pixel values) is <NUM>. The device <NUM> may determine an amount of saturated pixels for each frame (<NUM>). In one example, the device <NUM> may determine the number of saturated pixels for the frame. Additionally or alternatively, the device <NUM> may determine the proportion of saturated pixels to total pixels of the frame.

Proceeding to <NUM>, the device <NUM> may accumulate the identified saturations across the one or more frames. In one example, the device <NUM> may determine the total amount of saturated pixels across the one or more frames (<NUM>). For example, the device <NUM> may determine the total number of instances of pixel saturation across the one or more frames. Additionally or alternatively, the device <NUM> may determine a combined proportion of saturated pixels to total pixels across the one or more frames (such as a simple average proportion, median proportion, weighted average proportion, or other suitable proportion).

In some additional or alternative aspects to determining the total amount of saturated pixels, the device <NUM> may determine an increase or decrease in the amount of saturated pixels between consecutive frames (<NUM>). The device <NUM> may determine a change between consecutive frames in the total number of saturated pixels or the proportion of saturated pixels to total pixels. In some examples, the device <NUM> may determine a trend in changes between consecutive frames. As an illustrative example, if four frames are received (frame <NUM>, frame <NUM>, frame <NUM>, and frame <NUM>), and the device <NUM> determines that the changes between frames <NUM> and <NUM>, frames <NUM> and <NUM>, and frames <NUM> and <NUM> are an increase in saturation, the device <NUM> may determine a trending increase in saturation across frames. Additionally or alternatively, the device <NUM> may determine if the magnitude of the change is increasing or decreasing. For example, the device <NUM> may determine whether the increase or decrease in the amount of saturated pixels is accelerating or decelerating across a plurality of consecutive frames.

The device <NUM> may accumulate statistics regarding saturation in frames for one frame or for a plurality of frames. If accumulated for a plurality of frames, the device <NUM> may update the accumulations each time a frame is received. In some example implementations, the saturation statistics may be accumulated across a defined number of frames (such as <NUM>, <NUM>, <NUM>, or other suitable number of frames). For example, the device <NUM> may include or be configured to buffer the amount of saturation of the last <NUM> frames from the receiver <NUM>. When a new frame is received, and an amount of saturation is determined for the frame, the device <NUM> may remove the oldest buffered amount of saturation and buffer the amount of saturation for the new frame. In this manner, the accumulated identified saturations may be a moving window across a number of frames as the frames are received from the receiver <NUM>. In some example implementations, the size of the window (e.g., the size of the buffer) may be fixed. In some other example implementations, the size of the window (e.g., the size of the buffer) may be adjustable. In one example, the device <NUM> may increase the window size when the variance in total amount of saturated pixels among frames increases, and the device <NUM> may decrease the window size when the variance in total amount of saturated pixels among the frames decreases.

While some examples of accumulating saturation statistics across one or more frames are described in relation to <FIG>, any suitable accumulation of saturation statistics may be used. The present disclosure should not be limited to the above examples regarding <FIG>.

In some further example implementations, the device <NUM> may accumulate other statistics in addition or alternative to saturation statistics. In one example, the device <NUM> may determine the average measured intensity for a frame (such as the average lumens or lux for the frame). The device <NUM> then may buffer a number of the average intensities and/or determine an overall average intensity across the frames. The exposure time may be inversely related to the average intensity. Similar to using saturation statistics, the device <NUM> may use one or more intensity thresholds to determine when and how much to adjust the exposure time.

In another example, the device <NUM> may determine the number of decoding errors that occur when processing a frame to identify one or more codewords of the light distribution. The device <NUM> may determine the number of decoding errors by determining the number of misidentified codewords or the portions of the frame for which a codeword is not identified. The device may determine a total number of decoding errors, an average or median number of decoding errors, or a trend in decoding errors across one or more frames. The number of decoding errors may be buffered for a number of frames. The exposure time may be inversely related to the number of decoding errors. The device <NUM> may use one or more decoding error thresholds to determine when and how much to adjust the exposure time.

In another example, the device <NUM> may determine the signal-to-noise ratio (SNR) for one or more frames. Ambient light may be considered noise. In this manner, the SNR is inversely related to the amount of ambient light. For example, frames captured indoors or during nighttime may have a higher SNR than frames captured during bright daylight. The device may determine an average SNR, median SNR, or a trend in SNR across one or more frames. The SNRs may be buffered for a number of frames. The device <NUM> may use one or more SNR thresholds to determine when and how much to adjust the exposure time.

Other suitable accumulation statistics may be used, and the present disclosure should not be limited to the provided examples. Further, the device <NUM> may use a combination of accumulation statistics in determining when and how much to adjust the exposure time. For example, the device <NUM> may use an intensity statistic and a decoding error statistic to determine when and how much to adjust the exposure time. The combination of accumulation statistics may be, e.g., a vote checker, where any of the statistics may cause the device <NUM> to determine when and how much to adjust the exposure time. Other suitable combinations of statistics may be used, and the present disclosure should not be limited to the provided examples.

Referring back to <FIG> for the above examples of accumulation statistics, the amount of ambient light may be determined by the device <NUM> (in <NUM>) to be, e.g., at least one of a saturation statistic, an intensity statistic, a decoding error statistic, an SNR statistic, or other suitable statistic. In some example implementations of adjusting the exposure time of the receiver sensor for frame capture (in <NUM>), the device <NUM> may determine when and the amount to adjust the exposure time based on the accumulated statistics.

The exposure time may be based on or corresponding with the frame capture rate of the receiver <NUM>. For example, the device <NUM> may decrease the exposure time for a frame capture by increasing a frame capture rate for the receiver <NUM>. The receiver <NUM> may have a base frame capture rate (e.g., <NUM> fps, <NUM> fps, <NUM> fps, etc.). The base frame capture rate may be fixed, or the device <NUM> may adjust the base frame capture rate based on, e.g., one or more operations in the image processing pipeline for the device <NUM>. For example, if the device is slowed in processing frames from the receiver <NUM>, the device <NUM> may decrease the base frame capture rate to compensate.

In some example implementations of decreasing the exposure time, the device <NUM> may increase the frame capture rate for the receiver <NUM> (such as from a base frame capture rate). In some aspects, the device <NUM> may increase the frame capture rate to an integer multiple of the base frame capture rate based on at least one accumulation statistic. As an example, the receiver <NUM> may have a base frame capture rate of <NUM> fps. The receiver <NUM> captures a frame approximately every <NUM> at the base frame capture rate, and the exposure time for each frame capture is less than <NUM>. If the receiver <NUM> increases the frame capture rate from <NUM> fps (the base frame capture rate) to, e.g., <NUM> fps (an integer multiple of the base frame capture rate) based on at least one accumulation statistic, the exposure time for each frame capture is less than approximately <NUM>. In some other examples, the exposure time may be adjusted without adjusting the frame capture rate of the receiver <NUM>.

With a shorter exposure time (such as based on a higher frame capture rate), the receiver sensor is less likely to be saturated during frame capture. However, the determined difference between the received light with and without reflections of the light distribution is smaller when the exposure time is decreased. Referring back to <FIG>, the difference between lines <NUM> and <NUM> or lines <NUM> and <NUM> decrease when moving left (reducing the exposure time <NUM> or <NUM>) in the graph <NUM> or <NUM>, respectively.

In some example implementations, the device <NUM> may increase the differences in measured overall light intensities with and without the light distribution reflection by aggregating the measured light intensities across multiple frames. In this manner, the device <NUM> may compensate for decreasing the exposure time (such as based on the frame capture rate). Aggregated light intensities may be used in generating the aggregated frame (such as described above regarding <FIG>), and the device may use the aggregated frame for active depth sensing (such as in generating a depth map). While the examples of aggregating the measured light intensities for an aggregated frame are described below regarding increasing the frame capture rate, the measure light intensities may be aggregated without increasing the frame capture rate, and the present disclosure should not be limited to the following examples.

<FIG> is an illustrative flow chart depicting an example operation <NUM> for generating an aggregated frame based on an increased frame capture rate. The aggregated frame includes aggregated measured light intensities across multiple frames based on an adjusted frame capture rate. Beginning at <NUM>, the device <NUM> adjusts the frame capture rate of the receiver <NUM> based on the accumulation statistics. The device <NUM> increases the frame capture rate from a base frame capture rate to an integer multiple of the base frame capture rate (<NUM>). In some instances, the current frame capture rate already may be an integer multiple (greater than one) of the base frame capture rate. The device <NUM> thus may decrease the frame capture rate to a lower integer multiple of the base frame capture rate (including the base frame capture rate itself), or the device <NUM> may increase the frame capture rate to a higher integer multiple of the base frame capture rate.

After adjusting the frame capture rate, the device <NUM> receives a plurality of frame captures from the receiver <NUM> using the adjusted frame capture rate (<NUM>), and the device <NUM> generates an aggregated frame from the received frames (<NUM>). The device <NUM> aggregates values for a corresponding pixel across a number of received frames (<NUM>). For example, the device <NUM> may stack the number of received frames. The number of received frames for which to aggregate (stack) may be the integer multiple number of the base frame capture rate. In one example, if the base frame capture rate is <NUM> fps and the current frame capture rate is <NUM> fps, the number of received frames to aggregate may be <NUM> (since the current frame capture rate is 2X the base frame capture rate). In another example, if the current frame capture rate is <NUM> fps, the number of received frames to aggregate may be <NUM> (since the current frame capture rate is 4X the base frame capture rate). While the examples describe aggregating a number of frames equal to an integer multiple of the base frame capture rate, the device <NUM> may aggregate any number of frames. Further, the frame capture rate may be a non-integer multiple of the base frame capture rate, and the present disclosure should not be limited to specific multiples or specific frame capture rates in generating an aggregated frame.

If the generated aggregated frame includes an aggregated number of frames equal to the integer multiple of the base frame capture rate for the current frame capture rate, the rate of aggregated frames (generated by the device <NUM>) may be the same as the base frame capture rate. For example, if the base frame capture rate is <NUM> fps, the current frame capture rate is <NUM> fps, and each aggregated frame includes <NUM> frame captures (at the current frame capture rate), the periodicity of aggregated frames may equal the base frame capture rate.

In some example implementations, the total (aggregated) exposure time of the receiver sensor across the captured frames aggregated for an aggregate frame may be approximately equal to the exposure time of the receiver sensor for a frame at the base frame capture rate. In this manner, any thresholds that may be used in determining to adjust the frame capture rate may be used independent of the frame capture rate. Further, the device <NUM> may treat the aggregated frame the same as a frame captured at the base frame capture rate in performing active depth sensing. In this manner, the device <NUM> may not need to perform any calculations or conversions in using aggregated frames instead of captured frames (at the base frame capture rate) for active depth sensing or frame capture rate adjustment. In some other example implementations, the device <NUM> may adjust the active depth sensing operations or the frame capture rate adjustment operations based on the frame capture rate or the type of aggregated frame generated.

<FIG> is a block diagram <NUM> of example components for generating aggregated frames. The sensor <NUM> from receiver <NUM> may be a NIR sensor with a base frame capture rate of <NUM> fps. The sensor <NUM> may provide to the device <NUM> a stream of captured frames <NUM> at a rate of <NUM> fps. The device <NUM> may include a statistic(s) collection module <NUM> to determine and accumulate one or more statistics to be used for determining a frame capture rate for the sensor <NUM>. The accumulated statistics <NUM> may be received by the frame capture rate determination module <NUM>, and the frame capture rate determination module <NUM> may use the statistics <NUM> to determine whether to adjust the frame capture rate of the sensor <NUM> and how much to adjust the frame capture rate. For example, the module <NUM> may compare the statistics <NUM> to one or more thresholds (such as a saturation threshold, intensity threshold, or other suitable threshold) to determine whether and how much to adjust the frame capture rate of the sensor <NUM>. If the module <NUM> determines that the frame capture rate is to be adjusted, the module <NUM> may send instructions <NUM> for adjusting the frame capture rate. The instructions <NUM> may indicate a multiple N (where N is an integer equal to or greater than <NUM>) of the base frame capture rate. The instructions <NUM> may be received by the receiver <NUM> or a controller controlling the receiver <NUM> (such as the camera controller <NUM>) to adjust the frame capture rate of the sensor <NUM> to N times the base frame capture rate (such as 30N where <NUM> fps is the base frame capture rate). The determination module <NUM> also may send instructions <NUM> as to the number of frames to be aggregated to a frame aggregation module <NUM>. For example, the determination module <NUM> may send instructions to the aggregation module <NUM> to aggregate N number of frames. The aggregation module <NUM> may aggregate (such as stack) a consecutive N number of frames from the stream of capture frames <NUM> based on the instructions <NUM>, and the aggregation module <NUM> may generate a stream of aggregated frames <NUM>. The stream of aggregated frames <NUM> may be at a rate equal to the base frame capture rate of the sensor <NUM> (such as <NUM> fps for a base frame capture rate of <NUM> fps).

Determining and accumulating statistics, determining to adjust the frame capture rate, instructing to adjust the frame capture rate, and generating aggregated frames (such as modules <NUM>, <NUM>, and <NUM> in <FIG>) may be performed, e.g., by the camera controller <NUM> (such as the image signal processor <NUM>), the processor <NUM>, and/or other suitable components of the device <NUM>. The operations may be embodied in the instructions <NUM> stored in the memory <NUM> (or another suitable memory of the device <NUM>) and executed by a processor, may be performed using dedicated hardware (such as in the image signal processor <NUM> or the processor <NUM>), or may be performed using a combination of hardware and software.

In addition to determining the number of captured frames to aggregate in generating an aggregated frame, the device <NUM> may determine which captured frames to aggregate for an aggregated frame. In the above example for <FIG>, the device <NUM> may aggregate N consecutive frames (where N is the integer multiple of the base frame capture rate for the current frame capture rate).

In some example implementations, the device may synchronize which captured frames from the receiver <NUM> to aggregate based on image frame captures from a camera. Referring back to <FIG>, the device <NUM> may include or be coupled to a camera <NUM>. The camera <NUM> may include an image sensor, such as an RGB sensor, for capturing images using the received visible light. The receiver <NUM> (which may receive NIR) may assist the camera <NUM> (which may receive visible light). In one example, NIR measurements from the receiver <NUM> may be used in helping determine a focal length for the camera <NUM>. In another example, NIR measurements from the receiver <NUM> may be used in color balancing a captured image frame from the camera <NUM> in low light scenarios. If the receiver <NUM> and the camera <NUM> are to operate concurrently, the device <NUM> may be configured to synchronize frame capture of the receiver <NUM> with image frame capture of the camera <NUM>. In some example implementations, the camera controller <NUM> may synchronize the frame captures of the receiver <NUM> and the camera <NUM>. In some other example implementations, the processor <NUM> or another suitable component of the device <NUM> may synchronize the frame captures of the receiver <NUM> and the camera <NUM>. The present disclosure should not be limited to a specific component or combination of components performing the example operations for synchronizing frame captures.

The receiver <NUM> may be operating at a frame capture rate greater than the frame capture rate of the camera <NUM>. In synchronizing captures from the receiver <NUM> and the camera <NUM>, the device <NUM> may synchronize the timing of multiple exposure windows of the receiver <NUM> sensor to the timing of the exposure window of the camera <NUM> image sensor. The camera <NUM> may have a global shutter (where all pixels of the image sensor are scanned at the same time) or a rolling shutter (where the pixels of the image sensor are scanned sequentially).

<FIG> is an illustrative depiction of synchronizing receiver <NUM> sensor exposure windows <NUM> with image sensor exposure windows <NUM> for the receiver <NUM> and the camera <NUM> of the device <NUM>. Begin image sensor exposure <NUM> is a timing illustration when the first pixels of the image sensor begin (or end) being exposed for frame capture. End image sensor exposure <NUM> is a timing illustration when the last pixels of the image sensor begin (or end) being exposed for image frame capture. Global shutter 1008A, 1008B, and 1008C indicate when the image sensor pixels are scanned for capturing an image frame (thus indicating the end of the previous exposure window). For example, global shutter 1008B indicates an end of the exposure window 1002A, and global shutter 1008C indicates an end of the exposure window 1002B. While there may exist an amount of time between ending an exposure window of the image sensor for a first image frame and beginning an exposure window of the image sensor for a next image frame, <FIG> illustrates an end of the exposure window 1002A and illustrates a beginning of the exposure window 1002B as occurring at the same time (at global shutter 1008B) for example purposes.

The camera <NUM> may provide a synchronization (sync) strobe signal <NUM> indicating when the shutter for the image sensor is used. In this manner, the device <NUM> may know the end of the exposure window for a captured frame. For example, sync strobe 1010A may indicate when global shutter 1008A occurs, sync strobe 1010B may indicate when global shutter 1008B occurs, and sync strobe 1010C may indicate when global shutter 1008C occurs. For a global shutter, the end of the exposure window is at the same time for all pixels of the image sensor (as indicated by vertical lines for the global shutter 1008A - 1008C).

The device <NUM> may know the length of the exposure windows for the image sensor based on the type of camera <NUM> and other device configurations. Using the known length of the exposure window, the device <NUM> may use the sync strobe signal <NUM> to synchronize the timing of frame captures (or the exposure times) for the receiver <NUM>. For example, the device <NUM> may use the sync strobe 1010A to align the exposure window 1003A with the exposure window 1002A of the image sensor (by adjusting the start of the exposure window <NUM>). The device <NUM> may determine the number of captured frames for the receiver <NUM> corresponding to an exposure window of the image sensor for the camera <NUM> based on the known length of the exposure window. In the example in <FIG>, the device <NUM> may determine to aggregate <NUM> consecutive frames corresponding to <NUM> exposure windows <NUM> to generate an aggregated frame (based on the length of the exposure window and the timing of the shutter for the image sensor of the camera <NUM>). In this manner, the aggregated frame exposure window 1012A (for a first aggregated frame) corresponds to the exposure window 1002A, and the aggregated frame exposure window 1012B (for a second aggregated frame) corresponds to the exposure window 1002B. In some example implementations, the base frame capture rate of the receiver <NUM> may be equal to the image frame capture rate of the camera <NUM>.

While the provided example in <FIG> illustrates an integer multiple of exposure windows <NUM> occurring during the exposure windows <NUM>, the exposure windows <NUM> and <NUM> may not align as illustrated. In some example implementations, the device <NUM> may determine the exposure windows <NUM> most aligned with an exposure window <NUM>. For example, if <NUM> exposure windows <NUM> occur during an exposure window <NUM>, the device <NUM> may use three or four exposure windows <NUM> for an aggregated frame. In some other example implementations, the device <NUM> may delay frame captures by the receiver <NUM> to align a beginning of an exposure window <NUM> with a beginning of an exposure window <NUM>. Any other suitable alignment techniques may be used, and the present disclosure should not be limited to the provided examples.

Alternative to a global shutter, the camera <NUM> may include a rolling shutter. As a result, the timing of an exposure window may differ for different image sensor pixels of the camera <NUM> when capturing an image frame. <FIG> is an illustrative depiction of synchronizing sensor exposure windows <NUM> for the receiver <NUM> with image sensor exposure windows <NUM> for the camera <NUM> with a rolling shutter. Begin image sensor exposure <NUM> is a timing illustration when the first pixels of the image sensor begin (or end) being exposed for frame capture. End image sensor exposure <NUM> is a timing illustration when the last pixels of the image sensor begin (or end) being exposed for image frame capture. Since the camera <NUM> includes a rolling shutter for <FIG>, the lines for rolling shutter 1108A, 1108B, and 1108C are slanted to indicate when the image sensor pixels are scanned for capturing an image frame (with the first pixels being scanned before the last pixels). For example, rolling shutter 1108B indicates an end of an exposure window, which differs for different pixels of the image sensor, that corresponds to a determined exposure window 1102A. The rolling shutter 1108C indicates an end of an exposure window that corresponds to a determined exposure window 1102B. The time from beginning to end of scanning for a rolling shutter may be known or determined based on the camera <NUM> and the device <NUM>.

For an image sensor exposure window <NUM>, one or more pixels of the image sensor may not be exposed during an end of the exposure window, such as illustrated by interval <NUM>, when the pixels of the image sensor are being scanned. For example, the exposure window for the first pixels scanned may end before the exposure window for the last pixels scanned from the image sensor. The scan time for the image sensor using a rolling shutter (such as indicated by interval <NUM>) may be known or determined by the device <NUM> based on the camera <NUM>. For example, the camera <NUM> may have a pixel scan rate of its image sensor, and the scan time may be the number of pixels of the image sensor divided by the pixel scan rate.

Similar to the example in <FIG> regarding a global shutter, the camera <NUM> may provide a sync strobe signal <NUM> indicating when the rolling shutter for the image sensor begins (when the pixels of the image sensor begin to be scanned for an image frame). Sync strobe 1110A may indicate when rolling shutter 1108A begins, sync strobe 1110B may indicate when rolling shutter 1108B begins, and sync strobe 1110C may indicate when rolling shutter 1108C begins.

The device <NUM> may determine the portion of the exposure when all pixels of the image sensor are exposed (such as illustrated by the determined exposure windows 1102A and 1102B, where the interval (such as interval <NUM>) during the rolling shutters 1108A, 1108B, and 1108C are removed from the corresponding exposure windows <NUM>. In some example implementations, the device <NUM> may synchronize the receiver sensor exposure windows <NUM> to the determined exposure windows of the image sensor (e.g., excluding the intervals during the rolling shutter). The device <NUM> thus may exclude receiver sensor exposure windows <NUM> occurring during a rolling shutter or outside of a determined exposure window (excluding the time interval for the rolling shutter) for the image sensor.

For example, in <FIG>, the device <NUM> may align the beginning of the exposure window 1103B with the beginning of the determined exposure window 1102A. In this manner, the device <NUM> may exclude the exposure window 1103A from being used for the aggregated frame corresponding to the exposure window 1112A (and the previous aggregated frame). The exposure window 1112B for the aggregated frame corresponds to the determined exposure window 1102B, and the aggregated frames for the receiver <NUM> each include <NUM> captured frames and exclude <NUM> captured frame. In the example in <FIG>, the time for the rolling shutter approximately equals one exposure window for the receiver <NUM>. However, the rolling shutter may take a longer or shorter amount of time.

While the provided example in <FIG> illustrates an integer multiple of exposure windows <NUM> occurring during the determined exposure windows 1102A - 1102B, the exposure windows may not align as illustrated. In some example implementations, the device <NUM> may determine the exposure windows <NUM> most aligned with the determined exposure window 1102A. For example, the device <NUM> may exclude any exposure windows of the receiver <NUM> with any beginning portion of the exposure window during the rolling shutter of the camera <NUM>. In some other example implementations, the device <NUM> may delay frame captures by the receiver <NUM> to align a beginning of an exposure window <NUM> with a beginning of a determined exposure window <NUM> (excluding the intervals during a rolling shutter). Any other suitable alignment techniques may be used, and the present disclosure should not be limited to the provided examples.

In some example implementations, the device <NUM> may use the sync strobe signal to determine when to start collecting captured frames from the receiver <NUM> for a frame of the camera <NUM>. For example, the device <NUM> may begin collecting (for aggregation) frames captured at or after receipt of the sync strobe for the image frame. If the camera <NUM> includes a rolling shutter, the device <NUM> may begin collecting frames (for aggregation) after the known time interval for scanning from the sync strobe (such as interval <NUM> after sync strobe 1110A before beginning to collect frames from the receiver <NUM> for aggregation). In this manner, the frame captures of the receiver <NUM> may be synchronized to the image frames of the camera <NUM>, ensuring that the scene information in the aggregated frame corresponds to the scene information in the captured image frame. While a sync strobe signal <NUM> and <NUM> is described for the above examples, any suitable signal or indication of a shutter for the camera <NUM> may be used, and the present disclosure should not be limited to use of a sync strobe signal.

The techniques described herein may be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a specific manner. Any features described as modules or components may also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the techniques may be realized at least in part by a non-transitory processor-readable storage medium (such as the memory <NUM> in the example device <NUM> of <FIG>) comprising instructions <NUM> that, when executed by the processor <NUM> (or the camera controller <NUM> or the image signal processor <NUM>), cause the device <NUM> to perform one or more of the methods described above. The non-transitory processor-readable data storage medium may form part of a computer program product, which may include packaging materials.

The non-transitory processor-readable storage medium may comprise random access memory (RAM) such as synchronous dynamic random access memory (SDRAM), read only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM), FLASH memory, other known storage media, and the like. The techniques additionally, or alternatively, may be realized at least in part by a processor-readable communication medium that carries or communicates code in the form of instructions or data structures and that can be accessed, read, and/or executed by a computer or other processor.

The various illustrative logical blocks, modules, circuits and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors, such as the processor <NUM> or the image signal processor <NUM> in the example device <NUM> of <FIG>. Such processor(s) may include but are not limited to one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), application specific instruction set processors (ASIPs), field programmable gate arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. The term "processor," as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated software modules or hardware modules configured as described herein.

Claim 1:
A method for active depth sensing, comprising:
determining (<NUM>) an amount of ambient light for a scene to be captured by a structured light receiver (<NUM>; <NUM>);
adjusting (<NUM>) a frame capture rate of the structured light receiver, wherein adjusting (<NUM>) the frame capture rate comprises increasing (<NUM>) the frame capture rate from a base frame capture rate to an integer multiple of the base frame capture rate;
adjusting (<NUM>) an exposure time for frame capture of a sensor (<NUM>; <NUM>) of the structured light receiver based on the determined amount of ambient light, wherein the adjusted exposure time of the sensor of the structured light receiver is inversely related to the determined amount of ambient light and wherein the adjusted exposure time is based on the adjusted frame capture rate;
receiving (<NUM>; <NUM>; <NUM>) a plurality of frames (<NUM>; <NUM>; <NUM>) captured by the structured light receiver at the increased frame capture rate and using the adjusted exposure time; and
generating (<NUM>; <NUM>) an aggregated frame (<NUM>), wherein generating the aggregated frame includes aggregating (<NUM>), for each pixel, corresponding pixel values across the plurality of captured frames.