Patent Description:
Depth estimation is a central problem in computer vision. Determinations of depth are fundamental to many applications, such as facial recognition, face tracking, camera pose estimation and geometric modeling of scenes which drive many augmented (AR) applications.

The technical challenges associated with implementing depth estimation include, without limitation, simultaneously pushing the performance envelope of depth estimation across multiple dimensions of system performance, such as latency, processor power consumption, performance across varying light conditions and the ability to accurately estimate depth in the absence of motion in the frame of the sensor.

Often, robustness of performance, speed and low processor power consumption adhere to an "iron triangle" relationship, whereby a sensor or system can only have, at most, two of these three desirable performance properties. Thus, simultaneous improvement of the performance of depth estimation systems across multiple performance parameters remains a source of technical challenges and opportunities for improvement in the field of computer vision.

<NPL>, describes a method of stereo-matching using a mirror-galvanometer driven laser.

For a more complete understanding of this disclosure and its advantages, reference is now made to the following description, taken in conjunction with the accompanying drawings, in which:.

This disclosure provides systems and methods for semi-dense depth estimation from a Dynamic Vision Sensor (DVS) stereo pair and a pulsed speckle pattern projector.

In a first aspect, a method for semi-dense depth estimation is provided as set out in claim <NUM>. Additional features are set out in claims <NUM> to <NUM>.

In a second aspect, an apparatus is provided as set out in claim <NUM>. Additional features are set out in claims <NUM> to <NUM>.

In a third aspect, a non-transitory computer-readable medium is provided as set out in claim <NUM>.

A "non-transitory" computer readable medium excludes wired, wireless, optical, or other signals.

<FIG>, discussed below, and the various embodiments used to describe the principles of this disclosure in this patent document are by way of illustration only and should not be construed in any way to limit the scope of the disclosure. Those skilled in the art will understand that the principles of this disclosure may be implemented in any suitably arranged wireless communication system.

<FIG> illustrates a non-limiting example of a device for performing semi-dense depth estimation according to some embodiments of this disclosure. The embodiment of device <NUM> illustrated in <FIG> is for illustration only, and other configurations are possible. However, suitable devices come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular implementation of a device.

As shown in the non-limiting example of <FIG>, the device <NUM> includes a communication unit <NUM> that may include, for example, a radio frequency (RF) transceiver, a Bluetooth® transceiver, or a Wi-Fi® transceiver, etc., transmit (TX) processing circuitry <NUM>, a microphone <NUM>, and receive (RX) processing circuitry <NUM>. The device <NUM> also includes a speaker <NUM>, a main processor <NUM>, an input/output (I/O) interface (IF) <NUM>, input/output device(s) <NUM>, and a memory <NUM>. The memory <NUM> includes an operating system (OS) program <NUM> and one or more applications <NUM>.

Applications <NUM> can include games, social media applications, applications for geotagging photographs and other items of digital content, virtual reality (VR) applications, augmented reality (AR) applications, operating systems, device security (e.g., anti-theft and device tracking) applications or any other applications which access resources of device <NUM>, the resources of device <NUM> including, without limitation, speaker <NUM>, microphone <NUM>, input/output devices <NUM>, and additional resources <NUM>. According to some embodiments, applications <NUM> include applications which can consume or otherwise utilize depth estimation data regarding physical objects in a field of view of electronic device <NUM>.

The communication unit <NUM> may receive an incoming RF signal, for example, a near field communication signal such as a Bluetooth® or Wi-FiTM signal. The communication unit <NUM> can down-convert the incoming RF signal to generate an intermediate frequency (IF) or baseband signal. The IF or baseband signal is sent to the RX processing circuitry <NUM>, which generates a processed baseband signal by filtering, decoding, or digitizing the baseband or IF signal. The RX processing circuitry <NUM> transmits the processed baseband signal to the speaker <NUM> (such as for voice data) or to the main processor <NUM> for further processing (such as for web browsing data, online gameplay data, notification data, or other message data). Additionally, communication unit <NUM> may contain a network interface, such as a network card, or a network interface implemented through software.

The TX processing circuitry <NUM> receives analog or digital voice data from the microphone <NUM> or other outgoing baseband data (such as web data, e-mail, or interactive video game data) from the main processor <NUM>. The TX processing circuitry <NUM> encodes, multiplexes, or digitizes the outgoing baseband data to generate a processed baseband or IF signal. The communication unit <NUM> receives the outgoing processed baseband or IF signal from the TX processing circuitry <NUM> and up-converts the baseband or IF signal to an RF signal for transmission.

The main processor <NUM> can include one or more processors or other processing devices and execute the OS program <NUM> stored in the memory <NUM> in order to control the overall operation of the device <NUM>. For example, the main processor <NUM> could control the reception of forward channel signals and the transmission of reverse channel signals by the communication unit <NUM>, the RX processing circuitry <NUM>, and the TX processing circuitry <NUM> in accordance with well-known principles. In some embodiments, the main processor <NUM> includes at least one microprocessor or microcontroller.

The main processor <NUM> is also capable of executing other processes and programs resident in the memory <NUM>. The main processor <NUM> can move data into or out of the memory <NUM> as required by an executing process. In some embodiments, the main processor <NUM> is configured to execute the applications <NUM> based on the OS program <NUM> or in response to inputs from a user or applications <NUM>. Applications <NUM> can include applications specifically developed for the platform of device <NUM>, or legacy applications developed for earlier platforms. Additionally, main processor <NUM> can be manufactured to include program logic for implementing methods for monitoring suspicious application access according to certain embodiments of the present disclosure. The main processor <NUM> is also coupled to the I/O interface <NUM>, which provides the device <NUM> with the ability to connect to other devices such as laptop computers and handheld computers. The I/O interface <NUM> is the communication path between these accessories and the main processor <NUM>.

The main processor <NUM> is also coupled to the input/output device(s) <NUM>. The operator of the device <NUM> can use the input/output device(s) <NUM> to enter data into the device <NUM>. Input/output device(s) <NUM> can include keyboards, touch screens, mouse(s), track balls or other devices capable of acting as a user interface to allow a user to interact with electronic device <NUM>. In some embodiments, input/output device(s) <NUM> can include a touch panel, a virtual reality headset, a (digital) pen sensor, a key, or an ultrasonic input device.

Input/output device(s) <NUM> can include one or more screens, which can be a liquid crystal display, light-emitting diode (LED) display, an optical LED (OLED), an active matrix OLED (AMOLED), or other screens capable of rendering graphics.

The memory <NUM> is coupled to the main processor <NUM>. According to certain embodiments, part of the memory <NUM> includes a random access memory (RAM), and another part of the memory <NUM> includes a Flash memory or other read-only memory (ROM). Although <FIG> illustrates one example of a device <NUM>. Various changes can be made to <FIG>.

For example, according to certain embodiments, device <NUM> can further include a separate graphics processing unit (GPU) <NUM>.

According to certain embodiments, electronic device <NUM> includes a variety of additional resources <NUM> which can, if permitted, be accessed by applications <NUM>. According to certain embodiments, additional resources <NUM> include an accelerometer or inertial motion unit <NUM>, which can detect movements of the electronic device along one or more degrees of freedom. Additional resources <NUM> include a dynamic vision sensor (DVS) stereo pair <NUM>, one or more cameras <NUM> of electronic device <NUM>, and a speckle pattern projector (SPP) <NUM>. DVS stereo pair <NUM> comprises a pair of dynamic vision sensors spaced at a stereoscopically appropriate distance for estimating depth at over a field of depth of interest. As a non-limiting example, if the field of depth of interest is close to device <NUM>, the DVS sensors may be spaced comparatively closely, and if the field of depth of interest is far from device <NUM>, the individual sensors of DVS stereo pair <NUM> may be spaced further apart. SPP <NUM> projects a spatially non-repetitive pattern of dots of light (also known as "speckles") at one or more wavelengths which can be detected by the sensors of DVS stereo pair <NUM>. According to certain embodiments, SPP <NUM> projects a pattern at a wavelength at the edge of what the human eye can see, for example, at or around ∼<NUM>. In various embodiments according to this disclosure SPP <NUM> utilizes a laser-based diffractive optical element ("DOE") to project a speckle pattern.

Although <FIG> illustrates one example of a device <NUM> for performing semi-dense depth estimation, various changes may be made to <FIG>. For example, the device <NUM> could include any number of components in any suitable arrangement. In general, devices including computing and communication systems come in a wide variety of configurations, and <FIG> does not limit the scope of this disclosure to any particular configuration. While <FIG> illustrates one operational environment in which various features disclosed in this patent document can be used, these features could be used in any other suitable system.

<FIG> illustrates an example of an apparatus <NUM> for obtaining projected speckle pattern and scene data according to certain embodiments of this disclosure. The components shown in the non-limiting example of <FIG> are in some embodiments, configured within a single device. In some embodiments, the components described with reference to <FIG> are implemented on physically separate, but communicatively coupled, devices.

Referring to the non-limiting example of <FIG>, apparatus <NUM> comprises a first dynamic vision sensor 205a and a second dynamic vision sensor 205b, which together, comprise a DVS stereo pair. In some embodiments, apparatus <NUM> comprises a separate red-green-blue ("RGB") camera <NUM>, a speckle pattern projector <NUM> and an inertial motion unit ("IMU") <NUM>. As shown in this particular example, each of sensors <NUM>-<NUM> is communicatively connected to a processor <NUM>.

According to certain embodiments, first dynamic vision sensor 205a and second dynamic vision sensor 205b are configured to provide stereoscopic image data over a shared portion of each sensor's field of view. In some embodiments, dynamic vision sensors 205a and 205b each comprise a lens for receiving light and a pixelated sensor upon which the received light is focused. Each pixelated sensor of dynamic vision sensors 205a and 205b is configured to generate an output in response to a change in the intensity of the light received at the sensor. The light received at the sensor includes scene light and light from SPP <NUM>. As used in this disclosure, the term scene light encompasses light from objects within the field of view which is not provided by the SPP. Examples of scene light include, without limitation, sunlight reflected off of objects (for example, people, plants and buildings) in a scene, as well as artificial light, such as from a flash or on-camera light, used to illuminate a dark scene.

The output of each pixelated sensor is an event stream of time-mapped binary values reflecting changes in the intensity of the light received at the sensor at known points in time (for example, "<NUM>" for a decrease in the intensity of light received at the sensor, and "<NUM>" for an increase in the intensity of light received at the sensor). In some embodiments, first and second dynamic vision sensors 205a and 205b can respond to changes in light intensity occurring at wavelengths just outside of the visible spectrum.

In some embodiments, apparatus <NUM> includes RGB camera <NUM>, which is a digital camera comprising a lens for receiving light and a pixelated sensor upon which the received light is focused. According to various embodiments, the pixelated sensor of RGB camera 210is a complementary metal oxide semiconductor (CMOS) sensor, which periodically outputs frames of raw received light data from each of pixel of the sensor. The output of RGB camera <NUM> is, in certain embodiments, provided to processor <NUM> for calibrating first and second DVS sensors 205a and 205b, and for generating image data with additional chromatic detail. According to certain embodiments, the field of view of RGB camera <NUM> includes at least part of the overlap in the field of view of first dynamic vision sensor 205a and second dynamic vision sensor 205b.

As shown in the non-limiting example of <FIG>, apparatus <NUM> includes speckle pattern projector ("SPP") <NUM>. SPP <NUM> projects patterned light onto a portion of a scene falling within the stereoscopic field of first dynamic vision sensor 205a and second dynamic vision sensor 205b. As used in this disclosure, the term stereoscopic field encompasses a region of overlap in the field of view of first dynamic vision sensor 205a and the field of view of second dynamic vision sensor 205b. The patterned light projected by SPP <NUM> comprises a non-repeating pattern of individual light elements, such as a speckle pattern of dots, for which locations within the pattern can be identified in event stream data from each sensor of the DVS stereo pair. The pattern is pulsed according to a time-mapped control signal known to processor <NUM>. In some embodiments, the projected speckle pattern is projected at a wavelength that is invisible to the human eye, but visible to the DVS stereo pair, thereby avoiding the distraction of seeing flickering dots on a scene.

In certain embodiments, apparatus <NUM> includes IMU <NUM>, which is configured to move in concert with first and second DVS sensors 205a and 205b, and output data reflecting the motion, orientation and acceleration of apparatus <NUM> to processor <NUM>. In some embodiments, IMU <NUM> is a six degree-of-freedom sensor, capturing acceleration data across three axes as well as rotational (yaw) acceleration across three axes. In certain embodiments, IMU <NUM> may have greater or fewer than six degrees of freedom.

According to certain embodiments, the data output by IMU <NUM> is used for motion stabilization or motion correction on the outputs from the DVS stereo pair and/or RGB camera <NUM>. For example, while the individual pixel sensors of first and second dynamic vision sensors 205a and 205b are very fast, and configured to record events associated with changes in the intensity of light at the sensor more frequently than RGB sensors having a predetermined frame rate, such dynamic vision sensor event stream data may be aggregated over longer time scales, during which the position of apparatus <NUM> may change over the duration of an aggregation period. According to certain embodiments, processor <NUM> utilizes data from IMU <NUM> to compensate for changes in camera position over event aggregation intervals.

Referring to the non-limiting example of <FIG>, processor <NUM> is a processor communicatively connected to each of sensors <NUM>-<NUM> (for example, main processor <NUM> in <FIG>), and configured to execute instructions for performing semi-dense depth estimation according to certain embodiments of this disclosure. In some embodiments, processor <NUM> is a purpose-specific processor (for example, a programmable field gate array, or system on a chip ("SOC") device) specifically configured to perform depth estimation and output depth estimation and image data from a DVS stereo pair and/or RGB camera <NUM> to another processor.

<FIG> illustrates an example of a sensor configuration in an apparatus <NUM> for obtaining projected speckle pattern and scene data according to certain embodiments of this disclosure. In this illustrative example, apparatus <NUM> is depicted from the perspective over a viewer in the stereoscopic field of apparatus <NUM>.

Referring to the non-limiting example of <FIG>, apparatus <NUM> includes a first sensor 305a of a DVS stereo pair, a second sensor 305b of a DVS stereo pair, a speckle pattern projector <NUM>, and an internal motion unit sensor <NUM>.

According to certain embodiments, first sensor 305a and second sensor 305b are each DVS sensors comprising a lens and an array of pixelated sensors, each of which configured to provide a time-mapped event stream of changes in the intensity of light at the sensor. As shown in the illustrative example of <FIG>, first sensor 305a and second sensor 305b are separated by a distance <NUM>, which is a stereoscopically appropriate distance for performing stereo matching and depth estimation over a field of depth of interest. In some embodiments, apparatus <NUM> may be configured to adjust distance <NUM> depending on the field of depth of interest (for example, by increasing distance <NUM> to increase a parallax angle for distant objects). According to various embodiments, apparatus <NUM> may include three or more DVS sensors disposed a polygonal fashion on the face of apparatus <NUM>, thereby providing a variety of parallax axes and distances between sensors.

As shown in the non-limiting example of <FIG>, SPP <NUM> is depicted as residing squarely between first sensor 305a and second sensor 305b along a common centerline. Embodiments according to this disclosure are not so limited, and SPP <NUM> may be positioned at generally any location where it can project a light pattern (for example, a light pattern pulsed on and off according to a time-synchronized control signal) into the stereographic field.

According to certain embodiments, IMU <NUM> is placed close to first sensor 305a and second sensor 305b and coplanar with the DVS stereo pair. In some embodiments, IMU <NUM> is located anywhere within apparatus <NUM>, where the movement data obtained by IMU <NUM> reflects the motion of the DVS stereo pair.

<FIG> illustrates aspects of the operation of a dynamic vision sensor ("DVS") <NUM> according to certain embodiments of this disclosure. The embodiment shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, DVS <NUM> is, in certain embodiments, one sensor of a DVS stereo pair. In some embodiments, DVS <NUM> is one sensor of a set of three or more DVS sensors (for example, a set of DVS sensors disposed along multiple parallax angles, and at multiple sensor spacings). In certain embodiments, DVS <NUM> is a single DVS sensor.

According to various embodiments, DVS <NUM> comprises a lens assembly <NUM>, and a pixelated array <NUM> of light intensity sensors, such as light intensity sensor <NUM>. In some embodiments, lens assembly <NUM> comprises an optical lens having a focal length corresponding to a distance between lens assembly <NUM> and pixelated array <NUM>. In various embodiments according to this disclosure, lens assembly <NUM> comprises an aperture for adjusting (such as by stepping down an f-stop) the overall intensity of light provided to pixelated array <NUM>.

As shown in the non-limiting example of <FIG>, pixelated array <NUM> of light intensity sensors comprises an array of light intensity sensors (for example, light intensity sensor <NUM>) substantially covering an area in the focal plane of a lens in lens assembly <NUM>. Further, the output each light intensity sensor of pixelated array <NUM> is mapped to a spatial coordinate value.

Light intensity sensor <NUM> comprises a photo sensor configured to output a signal corresponding to a direction of change in the measured intensity of light received at light intensity sensor <NUM>. The output of light intensity sensor is a binary signal, for example "<NUM>" for an increase in the measured intensity of light, and "<NUM>" for a decrease in the measured intensity of light. When there is no changed in the measured intensity of light at light intensity sensor <NUM>, no signal is output. Signals output by light intensity sensor <NUM> are time-coded or time-mapped to a time value by pixelated array <NUM> or by another downstream component (such as processor <NUM> in <FIG>).

Referring to the non-limiting example of <FIG>, at a high level, DVS <NUM> operates by receiving light <NUM> through lens assembly <NUM>, and converting the received light into one or more time coded, or time-synchronized event stream <NUM>, by using the output of the constituent light intensity sensors of pixelated array <NUM>.

Event stream <NUM> comprises a time-coded or time-synchronized stream of light intensity change events output by light intensity sensors of pixelated array <NUM>. An individual light intensity change event <NUM> comprises data indicating a change (for example, an increase or decrease) in the measured intensity of the light measured at a particular light intensity sensor (e.g., a pixel) of pixelated array <NUM>. For example, in this illustrative example, light intensity change event <NUM> corresponds to a change in the measured light intensity at light intensity sensor <NUM>. Further, each individual light intensity change event <NUM> is time-coded or otherwise mapped to an event time based on a common timescale for each sensor of pixelated array <NUM>. Each individual light intensity change event <NUM> is also mapped to a value in a spatial coordinate system (for example, a coordinate system based on the rows and columns of pixelated array <NUM>).

<FIG> illustrates an example of a speckle pattern projected onto a scene which comprises a subject and a background. The embodiment shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, scene <NUM> depicts objects from the perspective of the stereoscopic field of a DVS stereo pair. In this particular example, the foreground of scene <NUM> comprises a ¾ (or "head and shoulders") view of a static subject and the background of scene <NUM> comprises an off-axis view of a flat wall <NUM> and ceiling <NUM>. As indicated by the intersection line <NUM> of the wall and ceiling, the perspective of the DVS stereo pair is such that vanishing point (not shown) is off to the lower left of the figure, meaning that portions of wall <NUM> on the left hand side of scene <NUM> are farther from the DVS stereo pair than portions of wall <NUM> on the right hand side of scene <NUM>. In this non-limiting example, wall <NUM> is generally featureless, except for placard <NUM> on wall <NUM>. An SPP (for example, SPP <NUM> in <FIG>) has projected speckles of light, including speckles 530a, 530b, and 530c, on both the foreground and background of scene <NUM>. According to various embodiments, speckles 530a, 530b, and <NUM> comprise elements of a non-repeating pattern of points of light at a wavelength which is visible, at a minimum, to the sensors of a DVS stereo pair. According to some embodiments, speckles 530a, 530b, and 530c are also visible to the human eye. However, in some embodiments, speckles 530a, 530b, and 530c are invisible to the human eye, as discussed above.

Scene <NUM> presents several technical challenges for any apparatus attempting to perform depth estimation of the features of subject <NUM> and wall <NUM>. As one example of the technical challenges posed by scene <NUM>, with the exception of placard <NUM>, wall <NUM> is a generally featureless surface, such as a section of dry wall to which a layer of primer has been applied. This paucity of features makes can make estimation of the distance between a DVS stereo pair and a point on wall <NUM> based on scene light very difficult. This is because stereo matching data from one sensor of a DVS stereo pair to the other sensor of the DVS stereo pair can be very difficult, if not outright impossible when one patch of wall <NUM> appears identical to many other patches of wall <NUM>.

As another example of the technical challenges associated with performing depth estimation on scene <NUM>, subject <NUM> is static. Depth estimation can be more readily calculated on moving subjects, because, in addition to using the parallax angle between the sensors of the DVS stereo pair as data from which depth can be estimated, a moving subject's location in the frame of a single sensor as a function of time can also provide data from which depth can be estimated.

The present invention addresses the challenges of scene <NUM> and even more challenging scenes (for example, a static white ball positioned in front of a white wall in a dimly lit room) to provide, at a minimum, for a wide range of scene elements and light conditions, semi-dense depth estimates. As used in this disclosure, the term semi-dense depth estimate encompasses an estimate of the depth of at least some of the points in a scene. Depending on, for example, the density of features in a scene, and the density of dots of light in a projected speckle pattern, depth estimates for more points in the scene may be calculated, and a denser depth estimate obtained.

<FIG> illustrates an example of components for implementing semi-dense depth estimation from a dynamic vision sensor stereo pair and a pulsed speckle pattern projector according to various embodiments of the present disclosure.

Referring to the non-limiting example of <FIG>, a system for implementing semi-dense depth estimation comprises a sensor block <NUM>, a speckle pattern projector (SPP) <NUM> and a depth estimation pipeline <NUM>. The embodiment shown in <FIG> is for illustration only and other embodiments could be used without departing from the scope of the present disclosure.

Referring to the non-limiting example of <FIG>, sensor block <NUM> comprises DVS stereo pair <NUM>, clock <NUM> and IMU sensor <NUM>. According to various embodiments, DVS stereo pair <NUM> is a pair of DVS sensors (for example, sensor <NUM> in <FIG>), wherein each DVS sensor has its own field of view, and the DVS sensors of DVS stereo pair <NUM> are spaced such that their respective fields of view overlap to produce a stereoscopic field. As shown in the non-limiting example of <FIG>, DVS stereo pair <NUM> is connected to clock <NUM> and the projected light filtering stage <NUM> of depth estimation pipeline <NUM>. In this illustrative example, DVS stereo pair <NUM> outputs one or more event streams of pixel intensity change data, wherein each event stream is time synchronized with a control signal for speckle pattern projector <NUM>.

In certain embodiments, clock <NUM> is a common clock for assigning time values to light intensity change events (for example, light intensity change event <NUM> in <FIG>) in the event streams from each sensor of DVS stereo pair <NUM>, as well as time values for the on/off state of a predetermined light pattern projected by the SPP on a field of view (for example, the stereoscopic field of DVS stereo pair <NUM>). According to various embodiments, clock <NUM> also time codes, or assigns time values to motion event data from IMU <NUM>. In some embodiments, clock <NUM> has a clock speed of <NUM> megahertz (<NUM>) or greater, reflecting the fact that, in certain embodiments, the sensors of DVS stereo pair <NUM> can respond to changes in the received intensity of light on microsecond time scales.

As shown in the non-limiting example of <FIG>, IMU <NUM> is connected to clock <NUM> and projected light filtering stage <NUM> of depth estimation pipeline <NUM>. According to various embodiments, IMU <NUM> provides depth estimation pipeline <NUM> with time-synchronized acceleration and orientation data for performing motion stabilization and correction for changes in vantage point.

According to various embodiments, speckle pattern projector <NUM> comprises speckle pattern controller <NUM> and projector <NUM>. In various embodiments, speckle pattern controller <NUM> controls an on/off state of one or more speckle patterns of light projected into some or all of the stereoscopic field of DVS pair <NUM>. In some embodiments, speckle pattern controller <NUM> is implemented as software executed by a separate processor (for example, main processor <NUM> in <FIG>). In certain embodiments, speckle pattern controller <NUM> is implemented as a purpose-specific controller for projector <NUM>. According to various embodiments, speckle pattern controller <NUM> pulses speckled light projected into the stereoscopic field between an "on" state and an "off' state according to a control signal which is time-synchronized to common clock <NUM>. According to various embodiments, the control signal of speckle pattern controller <NUM> is provided to both speckle projector <NUM> and the projected light filtering stage <NUM> of depth estimation pipeline <NUM>.

Referring to the non-limiting example of <FIG>, projector <NUM> comprises a light source which projects one or more non-repeating patterns of points of light ("speckles") into some portion of the field of view of at least one sensor of DVS stereo pair <NUM> (for example, the stereoscopic field of DVS stereo pair <NUM>). In certain embodiments, projector <NUM> comprises a diffractive optical element ("DOE") and a laser which is switched between an "on" state and an "off'" state according to a control signal from speckle pattern controller <NUM>.

In certain embodiments according to this disclosure, depth estimation pipeline <NUM> comprises a series of processing stages, which in this illustrative example, are numbered <NUM> through <NUM>, and produce a depth map or depth data which, in certain embodiments, are consumed, or utilized by an augmented reality (AR) or virtual reality (VR) application. According to certain applications, each stage of depth estimation pipeline <NUM> is implemented as a module of program code in a computer program executed by a processor (for example, main processor <NUM> in <FIG>). In some embodiments, each stage of depth estimation pipeline <NUM> is implemented via processing logic implemented in hardware (for example, as a programmable field gate array or system on a chip "SOC"). In certain embodiments, depth estimation pipeline is implemented through a combination of hardware and software modules.

Referring to the non-limiting example of <FIG>, depth estimation pipeline includes a projected light filtering stage <NUM>, which receives time synchronized (by, for example, assigning event times from common clock <NUM>) event stream data from DVS stereo pair <NUM>, speckle pattern control signal data from speckle pattern controller <NUM>, and IMU <NUM>. As received by projected light filtering stage <NUM>, the event stream for each individual light intensity change sensor (for example, light intensity sensor <NUM>) comprises a set of time coded events associated with one or more of scene light, switching one or more speckle patterns on and off. Projected light filtering stage <NUM> labels each of the time-coded events of each event stream as being associated with scene light, or light associated with the pulsing of a particular speckle pattern. Projected light filtering stage <NUM> separates each event stream into channels associated with the labels, such as a "scene" channel or a "first speckle pattern" channel.

According to various embodiments, depth estimation pipeline <NUM> includes motion stabilization stage <NUM>. As discussed with respect to pattern frame synthesis and scene frame synthesis stages <NUM> and <NUM>, in certain embodiments, multiple streams of event stream data are accumulated (for example, in a buffer) and synthesized over specified time intervals to generate synthesized event image data. According to various embodiments, the synthesized event image data comprises multi-channel histograms. Each multi-channel histogram comprises a spatially mapped representation of the light belonging to a specified channel (for example, scene light, or light from a projected speckle pattern) received at one of the DVS sensors of DVS stereo pair <NUM>. In certain embodiments, the length of the specified interval may be such that the motion of DVS stereo pair <NUM> over the interval degrades the image quality of the multi-channel histograms, such as by introducing blur or other motion-related effects (for example, effects associated with DVS stereo pair <NUM> moving closer to, or further away from, objects in the stereoscopic field). In some embodiments, motion stabilization stage <NUM> aggregates time-synchronized motion data from IMU <NUM> to determine motion stabilization corrections to be applied to multi-channel histograms generated at pattern frame synthesis stage <NUM> and scene frame synthesis stage <NUM>.

As shown in the non-limiting example of <FIG>, depth estimation pipeline <NUM> includes pattern frame synthesis stage <NUM>. According to certain embodiments, pattern frame synthesis stage <NUM> receives the output of projected light filtering stage <NUM> and motion stabilization stage <NUM>. The pattern frame synthesis stage <NUM> accumulates and synthesizes the output of projected light filtering stage <NUM>, and applies corrections determined by motion stabilization stage <NUM> to generate one or more channels of synthesized event image data. According to various embodiments, the synthesized event image data comprises, for each speckle pattern projected into the stereoscopic field, a multi-dimensional histogram (for example, histograms <NUM> and <NUM> of <FIG>) which provides a spatial representation of how the light associated with the projected speckle pattern (in certain embodiments, the scene light is excluded from the histogram) is received at a DVS sensor of DVS stereo pair <NUM>. As shown in the non-limiting example of <FIG>, at least two multi-dimensional histograms (one for each sensor of DVS stereo pair <NUM>) are generated, so that depth estimation may be performed based on stereo matching of histograms.

Referring to the non-limiting example of <FIG>, scene frame synthesis stage <NUM> receives the outputs of projected light filtering stage <NUM> and motion stabilization stage <NUM>, and accumulates and synthesizes, for each DVS sensor of DVS stereo pair <NUM>, light intensity change events, to generate two sets of synthesized image event data (one set for each DVS sensor of DVS stereo pair <NUM>) based on scene light. According to various embodiments, the synthesized image event data comprises two multi-dimensional histograms (such as histogram <NUM> in <FIG>) which, depending on conditions, settings and computational resources, may be stereo matched to obtain depth map data. As discussed elsewhere in this disclosure, in cases where the scene is largely featureless (for example, a white object in front of a smooth white wall in a room illuminated by a diffused light source, such as a flash bounced off a ceiling or a large soft box), or to conserve resources, scene frame synthesis stage <NUM> may be omitted, and semi-dense depth estimation based only on stereoscopic analysis of synthesized event image data from projected light patterns may be performed instead.

Depth estimation pipeline <NUM> includes stereo matching stage <NUM>. For a given channel (for example, a first projected speckle pattern) of synthesized event image data, the synthesized event image data for a first DVS of DVS stereo pair <NUM> is mapped to the synthesized event image data for a second DVS of DVS stereo pair <NUM>, to identify the locations of image features (for example, representations of projected speckles, or objects appearing in a scene) within the synthesized event image data. According to various embodiments, at stereo matching stage <NUM>, for each channel of the synthesized event image data, a patch scan of a histogram from each DVS sensor's synthesized data is performed to identify matching image patches in each histogram. In certain embodiments, the identification of matching image patches in each histogram can be performed by first generating a binary representation of the synthesized event image data in the patches, and then calculating the Hamming distance between patches. In this way, matching image patches can be recast as a search problem to identify patches with the lowest calculated Hamming distances.

Referring to the non-limiting example of <FIG>, depth estimation pipeline <NUM> includes depth mapping stage <NUM>, wherein a depth map comprising data representing the distance of one or more locations in the stereoscopic field of DVS stereo pair <NUM> is calculated based on the physical distance (for example, distance <NUM> in <FIG>) between the sensors of DVS stereo pair <NUM>, the focal length of the lens of the sensors of the DVS pair, and differences in frame location between image patches mapped in stereo matching stage <NUM>. In some embodiments, the depth map is determined by applying a lookup table to differences in frame location between image patches mapped in stereo matching stage <NUM>. In certain embodiments, a depth map is determined through a combination of calculation and identifying values in a lookup table.

According to various embodiments, the depth map determined by depth mapping stage <NUM> is output to one or more augmented reality (AR) or virtual reality (VR) applications <NUM>.

<FIG> illustrates an example of projected light filtering on an event stream of pixel intensity change data according to various embodiments of this disclosure.

Referring to the non-limiting example of <FIG>, an event stream <NUM> of light change intensity from an individual sensor (e.g., a pixel) of a DVS sensor (for example, DVS <NUM> in <FIG>) is shown. In this example, event stream <NUM> comprises events detected at the sensor during an interval in which light pattern in the field of view of the sensor was pulsed on and off. In the illustrative example of <FIG>, the constituent events of event stream <NUM> are shown as upward pips, indicating an increase in the measured intensity of light at the sensor (e.g., an increase in brightness), and downward pips, indicating a decrease in the measured intensity of light at the sensor (e.g., a darkening). As shown in <FIG>, each of these events is mapped to a timescale <NUM> provided by a common clock (for example, clock <NUM> in <FIG>).

Event stream <NUM> includes events associated with changes in scene light, as well as events associated with the pulsing of the speckle pattern. To separate the component of event stream <NUM> associated with light from an SPP from scene light, projected light filtering is performed.

Projected light filtering is performed by identifying conjugate sets of events occurring when the light pattern from the SPP is pulsed, and subtracting the identified events to obtain a scene light-only event stream. Referring to the non-limiting example of <FIG>, the pulsing of the light pattern of the SPP can be determined from SPP control signal <NUM>, which, like event stream <NUM>, is time-synchronized, or mapped to a timescale <NUM> provided by a common clock. In this example, SPP control signal <NUM> is a square wave, whose high and low values represent the on and off states of the light pattern. Events associated with light from the SPP can be identified from conjugate events occurring at the leading and trailing edges of the square wave of SPP control signal <NUM>. Two positive light intensity change events 720a occur immediately after the SPP control signal <NUM> switches to a value associated with a "light on" state, and two negative light intensity change events 720b occur immediately after the SPP control signal <NUM> switches to a value associated with a "light of" state. Events 720a and 720b comprise a conjugate pair of events which, by their timing and content, can be ascribed to light generated by an SPP. By identifying such conjugate pairs, labels <NUM> can be assigned to light intensity change events. In this non-limiting example, events 720a and 720b are assigned the label "A," corresponding to projected speckle pattern "A. " Similarly, based on their timing, other events are labeled "S," corresponding to events detected in response to scene light.

<FIG> illustrates an example of an event stream of scene-only pixel intensity change data obtained by performing projected light filtering according to certain embodiments of this disclosure.

As shown in the non-limiting example of <FIG>, by performing projected light filtering, an event stream of light intensity change events associated with one category, or channel, of light can be obtained. In this non-limiting example, event stream <NUM> is mapped to a timescale <NUM> (which in this non-limiting example, is the same as timescale <NUM> in <FIG>), and comprises only events associated with scene light, as shown by the single "S" label <NUM>. While not shown in <FIG>, a similar, single-channel event stream comprising only light intensity change events associated with speckles projected by an SPP could be generated using time-based projected light filtering according to certain embodiments of this disclosure. Similarly, time-based projected light filtering as described with reference to the illustrative examples of <FIG> and <FIG> is performed for each pixel of each DVS of a DVS sensor pair.

<FIG> illustrates an example of projected light filtering on an event stream of pixel intensity change data, as well as multi-channel histograms generated from filtered and synthesized light intensity change event streams. In some embodiments, the density of a depth estimation can be increased (in the sense that the depth of more points in a scene is determined) through pulsed projection of two or more different speckle patterns on to the scene. Additionally, the confounding effects of flickering lights can, in some cases, be mitigated by projecting two or more different speckle patterns on to the scene at different pulse rates or different phases. <FIG> illustrates an example of time-based projected light filtering where two or more speckle patterns are projected.

Referring to the non-limiting example of <FIG>, an event stream <NUM> of light intensity change events at an individual light intensity change sensor (e.g., a pixel) of a DVS sensor is shown. As in the non-limiting examples of <FIG> and <FIG>, in event stream <NUM> positive light change events are shown as upward pips, and negative light change events are shown as downward pips. Further, the constituent events of event stream <NUM> are mapped to a timescale <NUM> provided by a common clock (for example, clock <NUM> in <FIG>). In this example, two speckle patterns "SPP #<NUM>" and "SPP #<NUM>" are pulsed on and off in the scene. The on/off state of speckle pattern "SPP #<NUM>" is represented by a first control signal <NUM>, and the on/off state of speckle pattern "SPP #<NUM>" is represented by a second control signal <NUM>. Each of control signals <NUM> and <NUM> are mapped to timescale <NUM> provided by the common clock. Here, as in the example of <FIG>, the portions of event stream <NUM> associated with the respective contributions of scene light, speckle pattern "SPP #<NUM>" and speckle pattern "SPP #<NUM>" are determined from conjugate pairs of positive and negative light intensity change events detected immediately after the leading and trailing edges of first control signal <NUM> and second control signal <NUM>, and events can be assigned labels <NUM>.

After generating labels <NUM> for each event stream of each pixel of a pixelated array (for example, pixelated array <NUM> in <FIG>) of a DVS array, synthesized event image data is generated. In some embodiments, synthesized event image data comprises two-dimensional histograms providing, for each channel of light received at the DVS sensor, a two-dimensional mapping of the received light at the DVS sensor.

Referring to the non-limiting example of <FIG>, examples, <NUM>, <NUM> and <NUM> of histogram-format synthesized event image data from each channel of light received at a DVS sensor are provided. Histogram <NUM> comprises a two-dimensional mapping of the scene light as received at a DVS sensor. Histogram <NUM> comprises a two-dimensional mapping of light from speckle pattern "SPP #<NUM>" as received at the DVS sensor. Histogram <NUM> comprises a two-dimensional mapping of light from speckle pattern "SPP #<NUM>" as received at the DVS sensor.

<FIG> illustrates an example of performing stereo matching on synthesized event image data according to various embodiments of this disclosure.

Referring to the non-limiting example of <FIG> synthesized event image data 1005a from the left DVS of a DVS stereo pair, and synthesized event image data 1005b from the right DVS of a DVS stereo pair are shown at the top of the figure. In this non-limiting example, synthesized event image data 1005a & 1005b are represented as a superposition of two-dimensional histograms for multiple channels, such as a channel of scene light, a channel associated with a first projected speckle pattern, and a channel associated with a second projected speckle pattern.

According to certain embodiments, stereo matching (for example, as described in the non-limiting examples of <FIG>, <FIG> and <FIG> of this disclosure) comprises matching image patches in the histograms of "Left" synthesized event image data 1005a and "Right" synthesized event image data 1005b. In some embodiments, to save time and computational resources stereo matching comprises matching non-zero pixels in "Left" synthesized event image data 1005a to non-zero pixels in "Right" synthesized event image data 1005b.

According to certain embodiments, image patches are generated at predetermined points along a scanning pattern for both "Left" synthesized event image data 1005a and "Right" synthesized event image data 1005b. As shown in the example of <FIG>, a patch 1010a comprising a <NUM>×<NUM>×<NUM> pixel "slice" of "Left" synthesized event image data 1005a, which has a coordinate value within the synthesized event image data, is taken at a point on a scanning pattern which includes scanline <NUM>. Similarly, patches, including patch 1010b are taken from "Right" event image data 1005b. As shown in this non-limiting example, patch 1010a and patch 1010b were obtained at the same location within a scanning pattern.

As shown in the non-limiting example of <FIG>, patch 1010a comprises non-zero pixels in two of the three channels of light in the synthesized event image data 1005a and 1005b. For example, X_S^L is a <NUM>×<NUM> pixel grid representation 1020a of the scene light in patch 1010a. Similarly, X_S^R is a <NUM>×<NUM> pixel grid representation 1020b of the scene light in patch 1010b. As shown in <FIG>, patch 1010a includes scene light from the vertex of a triangle in the field of view of a DVS sensor, while patch 1010b, obtained from the same point along scanline <NUM> does not include the vertex of the triangle, reflecting the difference in perspective between the DVS sensors of a DVS sensor pair.

Additionally, patch 1010a comprises X_1^L, which is a <NUM>×<NUM> representation 1025a of light from a first speckle pattern, and X_2^L, which is a <NUM>×<NUM> representation 1030a of light from a second speckle pattern. Similarly patch 1010b comprises X_1^R, which is a <NUM>×<NUM> representation 1025b of light from a first speckle pattern, and X_2^R, which is a <NUM>×<NUM> representation 1030b of light from a second speckle pattern.

To reduce the computational load associated with identifying non-zero pixels across three channels (including the potentially pixel-rich data in the scene light channel) within "Left" synthesized event image data 1005a and "Right" synthesized event image data 1005b, the dimensionality of the representations of the patches of image data are reduced from matrices, or other <NUM> element representations of three <NUM>×<NUM> grids (e.g., <NUM>×<NUM>×<NUM>) of pixels, to a binary representation. In the non-limiting example of <FIG>, a compact binary representation algorithm <NUM> is applied to the representations 1020a, 1025a and 1030a of each channel in patch 1010a, to obtain a binary representation 1040a of patch 1010a. Similarly, the compact binary representation algorithm <NUM> is, in certain embodiments, applied to the representations 1020b, 1025b and 1030b to obtain a binary representation 1040b of the constituent channels of patch 1010b.

According to various embodiments, the distance between binary representations of patches obtained from "Left" synthesized event image data 1005a and "Right" synthesized event image data 1005b is calculated. In certain embodiments, the distance between representations is calculated based on a Hamming distance between the elements of the binary representations. In this example, the values of binary representation 1040a match the values of binary representation <NUM> in five of the seven places of each binary representation, resulting in a Hamming value of two. However, a binary representation of a patch obtained from slightly to the right of patch 1010b (and including the vertex of the triangle) might result in a smaller Hamming value. In this way, stereo matching of elements in "Left" synthesized event image data 1005a and "Right" synthesized event image data 1005b becomes a search for the lowest values of the calculated distance between binary representations of patches.

<FIG> illustrates an example of performing stereo matching on synthesized event image data. In certain embodiments, the speed with which stereo matching is performed, as well as battery consumption, can be improved by implementing a "low power mode" in which only synthesized event image data from projected light (thereby excluding scene light) and omitting the step of reducing the dimensionality (for example by applying compact binary representation algorithm <NUM> in <FIG>). <FIG> provides a non-limiting example of stereo matching in a "low power mode.

Referring to the non-limiting example of <FIG>, synthesized event image data from each DVS of a DVS pair is shown at the top of the figure as "Left" synthesized event image data 1105a and "Right" synthesized event image data 1105b. A <NUM>×<NUM>×<NUM> pixel patch 1110a of "Left synthesized image data 1105a is shown as having been obtained at a predetermined location along a scanning pattern which includes scanline <NUM>.

As shown in the non-limiting example of <FIG>, patch 1110a comprises non-zero pixels in one of the two channels of light in the synthesized event image data 1105a and 1105b. As shown in <FIG>, patch 1110a comprises X_1^L, which is a <NUM>×<NUM> representation 1120a of light from a first speckle pattern, and X_2^L, which is a <NUM>×<NUM> representation 1125a of light from a second speckle pattern. Similarly patch 1110b comprises X_1^R, which is a <NUM>×<NUM> representation 1120b of light from a first speckle pattern, and X_2^R, which is a <NUM>×<NUM> representation 1125b of light from a second speckle pattern. According to various embodiments, the representations of each channel within patch 1110a are provided to a binary representation algorithm <NUM>, which generates a binary representation 1135a. As depicted in the non-limiting example of <FIG>, binary representation 1135a can be generated without undergoing a dimensionality reduction process with respect to the <NUM>×<NUM>×<NUM> patch 1110a. Similarly, the binary representations of each channel within patch 1110b are provided to binary representation algorithm <NUM>, which generates a second binary representation 1135b of patch 1110b. Depending on the channels and available processing resources, it can be computationally less expensive to perform stereo matching based on calculating the distances between unreduced binary representations of certain channels, rather than stereo matching based on dimensionally reduced binary representations of all of the channels.

According to certain embodiments, having obtained binary representations 1135a and 1135b, the distance between elements within patches is calculated. According to certain embodiments, the distance is calculated based on the Hamming distance or as the sum of squares. After calculating distances across a patch set obtained from "Left" synthesized event image data 1105a and the patch set obtained from "Right" synthesized event image data 1105b, stereo matching can be treated as a search problem to identify, for each patch from "Left" synthesized event image data 1105a, the patch from "Right" synthesized event image data 1105b.

<FIG> illustrates operations of a method <NUM> for performing depth estimation. While the flow chart depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process depicted in the example depicted is implemented by, for example, main processor <NUM>.

Referring to <FIG>, method <NUM> includes operation <NUM>, wherein a processor (for example, main processor <NUM>, or a dedicated depth estimation apparatus, such as a system on a chip (SOC)) receives a control signal (for example, control signal <NUM> in <FIG>). The control signal indicates the on/off state of a predetermined light pattern (for example, the speckle pattern depicted in <FIG>) projected by an SPP on a field of view (for example, the stereoscopic field of a DVS stereo pair).

Method <NUM> includes operation <NUM>, wherein the processor receives, from each sensor of a DVS stereo pair (for example, DVS stereo pair <NUM> in <FIG>), an event stream (for example, event stream <NUM> in <FIG>) of pixel intensity change data. The event stream is time-synchronized, for example, by being time-stamped on a common timescale (such as timescale <NUM> in <FIG>) with the control signal of the SPP. Further, the event stream comprises a first portion associated with light from a scene in the field of view (for example, events labeled "S" in labels <NUM> of <FIG>) and a second portion associated with the predetermined light pattern projected by the SPP.

Method <NUM> includes operation <NUM>, wherein the processor performs projected light filtering, by identifying conjugate pairs of events following the trailing and leading edges of an SPP control signal, to generate synthesized event image data. The synthesized event image data has one or more channels (for example, the channels represented by histograms <NUM>, <NUM> and <NUM> in <FIG>), each of which is based on an isolated portion of the event stream of pixel intensity change data (for example, isolated portions of event stream <NUM> in <FIG>).

Method <NUM> further includes operation <NUM>, wherein the processor performs stereo matching on at least one channel (for example, a channel associated with a first speckle pattern, in synthesized event image data 1105a and 1105b in <FIG>) to obtain a depth map (for example, as performed by depth mapping stage <NUM> of depth estimation pipeline <NUM> in <FIG>) for at least a portion of a field of view.

<FIG> illustrates operations of a method for performing depth estimation according to various embodiments of this disclosure. While the flow chart depicts a series of sequential steps, unless explicitly stated, no inference should be drawn from that sequence regarding specific order of performance, performance of steps or portions thereof serially rather than concurrently or in an overlapping manner, or performance of the steps depicted exclusively without the occurrence of intervening or intermediate steps. The process depicted in the example depicted is implemented by, for example, main processor <NUM>. In certain embodiments, some, or all, of the operations described with reference to <FIG> are performed in conjunction with the operations of method <NUM> of <FIG>.

Referring to the non-limiting example of <FIG>, at operation <NUM>, a processor (or dedicated depth estimation unit) receives IMU data (for example, from IMU <NUM> in <FIG>) indicating the motion and/or change of orientation of the DVS stereo pair. According to certain embodiments, the IMU data is time synchronized with an event stream of pixel intensity change data (for example, by being time-mapped to a common timescale, such as timescale <NUM> in <FIG>). Further, at operation <NUM>, the processor performs motion stabilization (for example, motion stabilization as performed by motion stabilization stage <NUM> of depth estimation pipeline <NUM>) based on the IMU data.

At operation <NUM>, the processor receives a second control signal (for example, second control signal <NUM> in <FIG>), which is associated the on/off state of a second predetermined pattern of light projected into a field of a view (for example, a stereoscopic field) of a DVS stereo pair. The second control signal turns the second predetermined pattern of light on and off at distinguishable (for example, distinguished by frequency or phase) from a first control signal.

As shown in the non-limiting example of <FIG>, at operation <NUM>, the processor (or depth estimation hardware, or a combination thereof) generates synthesized event image data comprising histograms of accumulated pixel intensity change data (for example, histogram <NUM> in <FIG>). According to certain embodiments, histograms generated at operation <NUM> may comprise event data accumulated across each pixel of a pixelated sensor array (for example, pixelated array <NUM> in <FIG>) over a predetermined interval. Further, the histograms generated at operation may be motion stabilized or otherwise corrected to offset the effects in changes of DVS sensor position and orientation over an accumulation interval.

In certain embodiments according to this disclosure, at operation <NUM>, in preparation for performing stereo matching, the processor generates binary representations of the histograms of accumulated pixel intensity change data determined at operation <NUM>. In some embodiments, the dimensionality of the binary representation is less than the dimensionality of the represented portion of the histogram (for example, as is the case for binary representation 1040a in <FIG>. In various embodiments, the dimensionality of the binary representation relative to the dimensionality of the represented portion of the histogram is not reduced (for example, as is the case for binary representation 1135a in <FIG>).

At operation <NUM>, the processor performs projected light filtering on an event stream (for example, event stream <NUM> in <FIG>) to generate synthesized event image data comprising a channel associated with a second predetermined light pattern (for example, the synthesized event image data of histogram <NUM> of <FIG>).

At operation <NUM>, the processor performs stereo matching by calculating the Hamming distances between binary representations of histograms (or patches thereof) and matching patches by based on minima of the calculated Hamming distances.

Claim 1:
A method for semi-dense depth estimation, the method comprising:
receiving, at an electronic device, a control signal of a speckle pattern projector, SPP, the control signal indicating an on/off state, as a function of time, of a predetermined light pattern projected by the SPP on a field of view;
receiving from each sensor of a dynamic vision sensor, DVS, stereo pair, an event stream of pixel intensity change data, wherein the event stream is time-synchronized with the control signal of the SPP and comprises a first portion associated with light from a scene in the field of view and a second portion associated with the predetermined light pattern projected by the SPP;
performing projected light filtering that separates the second portion determined from the control signal of the SPP from the first portion on the event stream of pixel intensity change data for each sensor of the DVS stereo pair;
generating synthesized event image data, the synthesized event image data having one or more channels, each channel based on an isolated portion of the event stream of pixel intensity change data; and
performing stereo matching on at least one channel of the synthesized event image data for each sensor of the DVS stereo pair to generate a depth map for at least a portion of the field of view,
wherein the second portion comprises a conjugate pair of events first occurring after the control signal of the SPP is switched and the first portion is determined by excluding the second portion from the event stream.