PHOTOMETRIC MASKS FOR SELF-SUPERVISED DEPTH LEARNING

A method estimating a depth of an environment includes generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The method also includes generating, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume, the cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. The method further includes controlling an action of the vehicle based on the depth estimate.

BACKGROUND

Field

Certain aspects of the present disclosure generally relate to depth estimates, and more specifically to systems and methods for self-supervised depth estimation.

Background

Autonomous agents (e.g., vehicles, robots, etc.) rely on machine vision for constructing a three-dimensional (3D) representation of a surrounding environment. The 3D representation may be used for various tasks, such as localization and/or autonomous navigation. In some examples, the 3D representation may be generated from a depth estimate of an environment. Therefore, an accuracy of the 3D representation may be based on an accuracy of the depth estimate. Thus, improving an accuracy of the depth estimate may improve the accuracy of the 3D representation, which in turn, improves an ability of the autonomous agent to perform various tasks.

In some cases, a multi-frame network may use cost volumes may be used to estimate depth for a 3D image of a scene. In some examples, the cost volume is generated by combining information from multiple images onto a single 3D structure and evaluating a similarity metric between all pixel pairs given a series of possible depth ranges. Pixel pairs with a highest similarity may be referred to as correct pixel pairs. A depth estimation network (e.g., artificial neural network) may leverage activations associated with the correct pixel pairs to generate depth estimates. In some examples, an accuracy of the depth estimate generated by the multi-frame network may increase by using a single-frame network as a teacher for the multi-frame network.

SUMMARY

In one aspect of the present disclosure, a method for generating a depth estimate of an environment includes generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The method further includes generating, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume. The cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. The method still further includes controlling an action of the vehicle based on the depth estimate.

Another aspect of the present disclosure is directed to an apparatus including means for generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The apparatus further includes means for generating, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume. The cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. The apparatus still further includes means for controlling an action of the vehicle based on the depth estimate.

In another aspect of the present disclosure, a non-transitory computer-readable medium with non-transitory program code recorded thereon is disclosed. The program code is executed by a processor and includes program code to generate, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The program code further includes program code to generate, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume. The cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. The program code still further includes program code to control an action of the vehicle based on the depth estimate.

Another aspect of the present disclosure is directed to an apparatus having a processor, and a memory coupled with the processor and storing instructions operable, when executed by the processor, to cause the apparatus to generate, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. Execution of the instructions also cause the apparatus to generate, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume. The cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. Execution of the instructions further cause the apparatus to control an action of the vehicle based on the depth estimate.

DETAILED DESCRIPTION

The ability to perceive distances through depth estimation based on sensor data provides an ability to plan/estimate ego-motion through the environment. Therefore, an agent, such as an autonomous agent, may generate a 3D representation of an environment based on one or more images obtained from a sensor. The 3D representation may also be referred to as a 3D model, a 3D scene, or a 3D map. 3D representations may facilitate various tasks, such as scene understanding, motion planning, and/or obstacle avoidance. For example, the agent may autonomously navigate through an environment based on the 3D representation.

In some cases, a single frame may be used to estimate a depth of an environment. In other cases, multiple frames (multi-frame) may be used to estimate the depth. Multi-frame depth estimation may be considered an improvement of over single frame depth estimation because multi-frame depth estimation may leverage geometric relationships between images via feature matching, in addition to learning appearance-based features. In some examples, cost volumes may be used by a multi-frame depth estimation network (e.g., a multi-frame monocular depth estimation network) to estimate a depth of an environment. In some examples, the cost volume is generated by combining information from multiple images onto a single 3D structure and evaluating a similarity metric between all pixel pairs given a series of possible depth ranges. Pixel pairs with a highest similarity may be referred to as correct pixel pairs. A depth estimation network (e.g., artificial neural network) may leverage activations associated with the correct pixel pairs to generate depth estimates.

Cost volumes may increase an accuracy of depth estimates for static objects. However, the use of cost volumes in a multi-frame depth estimation network may reduce an accuracy of the depth estimates associated with dynamic objects, low texture areas, and/or occluded objects. Therefore, it may be desirable to use self-supervised learning to improve the accuracy of the depth estimates generated based on cost volumes.

Deep learning approaches, such as self-supervised learning, may eliminate hand-engineered features (e.g., labeled data) and improve depth estimates as well as 3D model reconstruction. For example, self-supervised learning improves the reconstruction of textureless regions and/or geometrically under-determined regions. Aspects of the present disclosure are directed to self-supervised depth estimates based on cost volumes.

Particular aspects of the subject matter described in this disclosure may be implemented to realize one or more of the following potential advantages. In some examples, the described techniques may provide ground-truth data for training a cost volume-based depth estimation network in a self-supervised manner. In such examples, the overall accuracy of the depth estimates generated based on the cost volumes may improve. Specifically, the accuracy of the depth estimates for dynamic objects, textureless objects, and/or occluded objects may improve.

Aspects of the present disclosure are not limited to an autonomous agent. Aspects of the present disclosure also contemplate an agent operating in a manual mode or a semi-autonomous mode. In the manual mode, a human driver manually operates (e.g., controls) the agent. In the autonomous mode, an agent control system operates the agent without human intervention. In the semi-autonomous mode, the human may operate the agent, and the agent control system may override or assist the human. For example, the agent control system may override the human to prevent a collision or to obey one or more traffic rules.

FIG.1Ais a diagram illustrating an example of a vehicle100in an environment150, in accordance with various aspects of the present disclosure. In the example ofFIG.1A, the vehicle100may be an autonomous vehicle, a semi-autonomous vehicle, or a non-autonomous vehicle. As shown inFIG.1A, the vehicle100may be traveling on a road110. A first vehicle104may be ahead of the vehicle100and a second vehicle116may be adjacent to the ego vehicle100. In this example, the vehicle100may include a 2D camera108, such as a 2D red-green-blue (RGB) camera, and a LIDAR sensor106. Other sensors, such as RADAR and/or ultrasound, are also contemplated. Additionally, or alternatively, although not shown inFIG.1A, the vehicle100may include one or more additional sensors, such as a camera, a RADAR sensor, and/or a LIDAR sensor, integrated with the vehicle in one or more locations, such as within one or more storage locations (e.g., a trunk). Additionally, or alternatively, although not shown inFIG.1A, the vehicle100may include one or more force measuring sensors.

In one configuration, the 2D camera108captures a 2D image that includes objects in the 2D camera's108field of view114. The LIDAR sensor106may generate one or more output streams. The first output stream may include a 3D cloud point of objects in a first field of view, such as a 360° field of view112(e.g., bird's eye view). The second output stream124may include a 3D cloud point of objects in a second field of view, such as a forward facing field of view126.

The 2D image captured by the 2D camera includes a 2D image of the first vehicle104, as the first vehicle104is in the 2D camera's108field of view114. As is known to those of skill in the art, a LIDAR sensor106uses laser light to sense the shape, size, and position of objects in the environment150. The LIDAR sensor106may vertically and horizontally scan the environment150. In the current example, the artificial neural network (e.g., autonomous driving system) of the vehicle100may extract height and/or depth features from the first output stream. In some examples, an autonomous driving system of the vehicle100may also extract height and/or depth features from the second output stream.

The information obtained from the sensors106,108may be used to evaluate a driving environment. Additionally, or alternatively, information obtained from one or more sensors that monitor objects within the vehicle100and/or forces generated by the vehicle100may be used to generate notifications when an object may be damaged based on actual, or potential, movement.

FIG.1Bis a diagram illustrating an example the vehicle100, in accordance with various aspects of the present disclosure. It should be understood that various aspects of the present disclosure may be applicable to/used in various vehicles (internal combustion engine (ICE) vehicles, fully electric vehicles (EVs), etc.) that are fully or partially autonomously controlled/operated, and as noted above, even in non-vehicular contexts, such as, e.g., shipping container packing.

The vehicle100may include drive force unit165and wheels170. The drive force unit165may include an engine180, motor generators (MGs)182and184, a battery195, an inverter197, a brake pedal186, a brake pedal sensor188, a transmission152, a memory154, an electronic control unit (ECU)156, a shifter158, a speed sensor160, and an accelerometer162.

The engine180primarily drives the wheels170. The engine180can be an ICE that combusts fuel, such as gasoline, ethanol, diesel, biofuel, or other types of fuels which are suitable for combustion. The torque output by the engine180is received by the transmission152. MGs182and184can also output torque to the transmission152. The engine180and MGs182and184may be coupled through a planetary gear (not shown inFIG.1B). The transmission152delivers an applied torque to one or more of the wheels170. The torque output by engine180does not directly translate into the applied torque to the one or more wheels170.

MGs182and184can serve as motors which output torque in a drive mode, and can serve as generators to recharge the battery195in a regeneration mode. The electric power delivered from or to MGs182and184passes through the inverter197to the battery195. The brake pedal sensor188can detect pressure applied to brake pedal186, which may further affect the applied torque to wheels170. The speed sensor160is connected to an output shaft of transmission152to detect a speed input which is converted into a vehicle speed by ECU156. The accelerometer162is connected to the body of vehicle100to detect the actual deceleration of vehicle100, which corresponds to a deceleration torque.

The transmission152may be a transmission suitable for any vehicle. For example, transmission152can be an electronically controlled continuously variable transmission (ECVT), which is coupled to engine180as well as to MGs91and92. Transmission20can deliver torque output from a combination of engine180and MGs91and92. The ECU156controls the transmission152, utilizing data stored in memory154to determine the applied torque delivered to the wheels170. For example, ECU156may determine that at a certain vehicle speed, engine180should provide a fraction of the applied torque to the wheels170while one or both of the MGs182and184provide most of the applied torque. The ECU156and transmission152can control an engine speed (NE) of engine180independently of the vehicle speed (V).

The ECU156may include circuitry to control the above aspects of vehicle operation. Additionally, the ECU156may include, for example, a microcomputer that includes a one or more processing units (e.g., microprocessors), memory storage (e.g., RAM, ROM, etc.), and I/O devices. The ECU156may execute instructions stored in memory to control one or more electrical systems or subsystems in the vehicle. Furthermore, the ECU156can include one or more electronic control units such as, for example, an electronic engine control module, a powertrain control module, a transmission control module, a suspension control module, a body control module, and so on. As a further example, electronic control units can be included to control systems and functions such as doors and door locking, lighting, human-machine interfaces, cruise control, telematics, braking systems (e.g., anti-lock braking system (ABS) or electronic stability control (ESC)), battery management systems, and so on. These various control units can be implemented using two or more separate electronic control units, or using a single electronic control unit.

The MGs182and184each may be a permanent magnet type synchronous motor including for example, a rotor with a permanent magnet embedded therein. The MGs182and184may each be driven by an inverter controlled by a control signal from ECU156so as to convert direct current (DC) power from the battery195to alternating current (AC) power, and supply the AC power to the MGs182and184. In some examples, a first MG182may be driven by electric power generated by a second MG184. It should be understood that in embodiments where MGs182and184are DC motors, no inverter is required. The inverter, in conjunction with a converter assembly may also accept power from one or more of the MGs182and184(e.g., during engine charging), convert this power from AC back to DC, and use this power to charge battery195(hence the name, motor generator). The ECU156may control the inverter, adjust driving current supplied to the first MG182, and adjust the current received from the second MG184during regenerative coasting and braking.

The battery195may be implemented as one or more batteries or other power storage devices including, for example, lead-acid batteries, lithium ion, and nickel batteries, capacitive storage devices, and so on. The battery195may also be charged by one or more of the MGs182and184, such as, for example, by regenerative braking or by coasting during which one or more of the MGs182and184operates as generator. Alternatively (or additionally, the battery195can be charged by the first MG182, for example, when vehicle100is in idle (not moving/not in drive). Further still, the battery195may be charged by a battery charger (not shown) that receives energy from engine180. The battery charger may be switched or otherwise controlled to engage/disengage it with battery195. For example, an alternator or generator may be coupled directly or indirectly to a drive shaft of engine180to generate an electrical current as a result of the operation of engine180. Still other embodiments contemplate the use of one or more additional motor generators to power the rear wheels of the vehicle100(e.g., in vehicles equipped with 4-Wheel Drive), or using two rear motor generators, each powering a rear wheel.

The battery195may also power other electrical or electronic systems in the vehicle100. In some examples, the battery195can include, for example, one or more batteries, capacitive storage units, or other storage reservoirs suitable for storing electrical energy that can be used to power one or both of the MGs182and184. When the battery195is implemented using one or more batteries, the batteries can include, for example, nickel metal hydride batteries, lithium ion batteries, lead acid batteries, nickel cadmium batteries, lithium ion polymer batteries, and other types of batteries.

FIG.2is a block diagram illustrating a software architecture200that may modularize artificial intelligence (AI) functions for planning and control of an autonomous agent, according to aspects of the present disclosure. Using the architecture, a controller application202may be designed such that it may cause various processing blocks of a system-on-chip (SOC)220(for example a central processing unit (CPU)222, a digital signal processor (DSP)224, a graphics processing unit (GPU)226and/or an network processing unit (NPU)228) to perform supporting computations during run-time operation of the controller application202.

The controller application202may be configured to call functions defined in a user space204that may, for example, provide for taillight recognition of ado vehicles. The controller application202may make a request to compile program code associated with a library defined in a taillight prediction application programming interface (API)206to perform taillight recognition of an ado vehicle. This request may ultimately rely on the output of a convolutional neural network configured to focus on portions of the sequence of images critical to vehicle taillight recognition.

A run-time engine208, which may be compiled code of a runtime framework, may be further accessible to the controller application202. The controller application202may cause the run-time engine208, for example, to take actions for controlling the autonomous agent. When an ado vehicle is detected within a predetermined distance of the autonomous agent, the run-time engine208may in turn send a signal to an operating system210, such as a Linux Kernel212, running on the SOC220. The operating system210, in turn, may cause a computation to be performed on the CPU222, the DSP224, the GPU226, the NPU228, or some combination thereof. The CPU222may be accessed directly by the operating system210, and other processing blocks may be accessed through a driver, such as drivers214-218for the DSP224, for the GPU226, or for the NPU228. In the illustrated example, the deep neural network may be configured to run on a combination of processing blocks, such as the CPU222and the GPU226, or may be run on the NPU228, if present.

FIG.3is a diagram illustrating an example of a hardware implementation for a vehicle control system300, according to aspects of the present disclosure. The vehicle control system300may be a component of a vehicle, a robotic device, or other device. For example, as shown inFIG.3, the vehicle control system300is a component of a vehicle100. Aspects of the present disclosure are not limited to the vehicle control system300being a component of the vehicle100, as other devices, such as a bus, boat, drone, or robot, are also contemplated for using the vehicle control system300. In the example ofFIG.3, the vehicle system may include a depth estimation system390. In some examples, depth estimation system390is configured to perform operations, including operations of the process800described with reference toFIG.8.

The vehicle control system300may be implemented with a bus architecture, represented generally by a bus330. The bus330may include any number of interconnecting buses and bridges depending on the specific application of the vehicle control system300and the overall design constraints. The bus330links together various circuits including one or more processors and/or hardware modules, represented by a processor320, a communication module322, a location module318, a sensor module302, a locomotion module323, a planning module324, and a computer-readable medium313. The bus330may also link various other circuits such as timing sources, peripherals, voltage regulators, and power management circuits, which are well known in the art, and therefore, will not be described any further.

The vehicle control system300includes a transceiver314coupled to the processor320, the sensor module302, a comfort module308, the communication module322, the location module318, the locomotion module323, the planning module324, and the computer-readable medium313. The transceiver314is coupled to an antenna333. The transceiver314communicates with various other devices over a transmission medium. For example, the transceiver314may receive commands via transmissions from a user or a remote device. As another example, the transceiver314may transmit driving statistics and information from the comfort module308to a server (not shown).

In one or more arrangements, one or more of the modules302,313,314,318,320,322,323,324,390, can include artificial or computational intelligence elements, such as, neural network, fuzzy logic or other machine learning algorithms. Further, in one or more arrangements, one or more of the modules302,313,314,318,320,322,323,324,390can be distributed among multiple modules302,313,314,318,320,322,323,324,390described herein. In one or more arrangements, two or more of the modules302,313,314,318,320,322,323,324,390of the vehicle control system300can be combined into a single module.

The vehicle control system300includes the processor320coupled to the computer-readable medium313. The processor320performs processing, including the execution of software stored on the computer-readable medium313providing functionality according to the disclosure. The software, when executed by the processor320, causes the vehicle control system300to perform the various functions described for a particular device, such as the vehicle328, or any of the modules302,313,314,318,320,322,323,324,390. The computer-readable medium313may also be used for storing data that is manipulated by the processor320when executing the software.

The sensor module302may be used to obtain measurements via different sensors, such as a first sensor303A and a second sensor303B. The first sensor303A and/or the second sensor303B may be a vision sensor, such as a stereoscopic camera or a red-green-blue (RGB) camera, for capturing 2D images. In some examples, one or both of the first sensor303A or the second sensor303B may be used to identify an intersection, a crosswalk, or another stopping location. Additionally, or alternatively, one or both of the first sensor303A or the second sensor303B may identify objects within a range of the vehicle100. In some examples, one or both of the first sensor303A or the second sensor303B may identify a pedestrian or another object in a crosswalk. The first sensor303A and the second sensor303B are not limited to vision sensors as other types of sensors, such as, for example, light detection and ranging (LiDAR), a radio detection and ranging (radar), sonar, and/or lasers are also contemplated for either of the sensors303A,303B. The measurements of the first sensor303A and the second sensor303B may be processed by one or more of the processor320, the sensor module302, the comfort module308, the communication module322, the location module318, the locomotion module323, the planning module324, in conjunction with the computer-readable medium313to implement the functionality described herein. In one configuration, the data captured by the first sensor303A and the second sensor303B may be transmitted to an external device via the transceiver314. The first sensor303A and the second sensor303B may be coupled to the vehicle328or may be in communication with the vehicle328.

Additionally, the sensor module302may configure the processor320to obtain or receive information from the one or more sensors303A and303B. The information may be in the form of one or more two-dimensional (2D) image(s) and may be stored in the computer-readable medium313as sensor data. In the case of 2D, the 2D image is, for example, an image from the one or more sensors303A and303B that encompasses a field-of-view about the vehicle100of at least a portion of the surrounding environment, sometimes referred to as a scene. That is, the image is, in one approach, generally limited to a subregion of the surrounding environment. As such, the image may be of a forward-facing (e.g., the direction of travel) 30, 90, 120-degree field-of-view (FOV), a rear/side facing FOV, or some other subregion as defined by the characteristics of the one or more sensors303A and303B. In further aspects, the one or more sensors303A and303B may be an array of two or more cameras that capture multiple images of the surrounding environment and stitch the images together to form a comprehensive 330-degree view of the surrounding environment. In other examples, the one or more images may be paired stereoscopic images captured from the one or more sensors303A and303B having stereoscopic capabilities.

The location module318may be used to determine a location of the vehicle328. For example, the location module318may use a global positioning system (GPS) to determine the location of the vehicle328. The communication module322may be used to facilitate communications via the transceiver314. For example, the communication module322may be configured to provide communication capabilities via different wireless protocols, such as Wi-Fi, long term evolution (LTE), 3G, etc. The communication module322may also be used to communicate with other components of the vehicle328that are not modules of the vehicle control system300. Additionally, or alternatively, the communication module322may be used to communicate with an occupant of the vehicle100. Such communications may be facilitated via audio feedback from an audio system of the vehicle100, visual feedback via a visual feedback system of the vehicle, and/or haptic feedback via a haptic feedback system of the vehicle.

The locomotion module323may be used to facilitate locomotion of the vehicle328. As an example, the locomotion module323may control movement of the wheels. As another example, the locomotion module323may be in communication with a power source of the vehicle328, such as an engine or batteries. Of course, aspects of the present disclosure are not limited to providing locomotion via wheels and are contemplated for other types of components for providing locomotion, such as propellers, treads, fins, and/or jet engines.

The vehicle control system300also includes the planning module324for planning a route or controlling the locomotion of the vehicle328, via the locomotion module323. A route may be planned to a passenger based on compartment data provided via the comfort module308. In one configuration, the planning module324overrides the user input when the user input is expected (e.g., predicted) to cause a collision. The modules may be software modules running in the processor320, resident/stored in the computer-readable medium313, one or more hardware modules coupled to the processor320, or some combination thereof.

The depth estimation system390may be in communication with the sensor module302, the transceiver314, the processor320, the communication module322, the location module318, the locomotion module323, the planning module324, and the computer-readable medium313. In some examples, the behavior planning system may be implemented as a machine learning model, such as a vehicle control system300as described with reference toFIG.3. Working in conjunction with one or more of the sensors303A,303B, the sensor module302, and/or one or more other modules313,314,318,320,322,323,324, the depth estimation system390may generate, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. Additionally, the depth estimation system390may generate, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume, the cross-attention model having been trained using a photometric loss associated with a single-frame depth estimation model. Finally, the depth estimation system390may control an action of the vehicle100based on the depth estimate.

FIG.4Aillustrates an example of a target image400of a scene404according to aspects of the present disclosure. The target image400may be captured by a monocular camera or may be one image of a multi-frame image captured by one or more cameras. The one or more cameras may capture a forward-facing view of an agent (e.g., a vehicle). In one configuration, the one or more cameras are integrated with the vehicle, such as the vehicle100described with reference toFIGS.1A and1B. For example, the one or more cameras may be defined in a roof structure, windshield, grill, or other portion of the vehicle. The target image400may also be referred to as a current image. The target image400captures a 2D representation of a scene.

FIG.4Billustrates an example of a depth map440of the scene404according to aspects of the present disclosure. The depth map440may be estimated from the target image400and one or more source images. The source images may be images captured at a previous time step in relation to the target image400. The depth map440provides a depth of a scene. The depth may be represented as a color or other feature.

FIG.4Cillustrates an example of a 3D reconstruction450of the scene404according to aspects of the present disclosure. The 3D reconstruction may be generated from the depth map440as well as a pose of the target image400and a source image. As shown inFIGS.4A and4C, the viewing angle of the scene404in the 3D reconstruction450, is different from the viewing angle of the scene404in the target image400. Because the 3D reconstruction450is a 3D view of the scene404, the viewing angle may be changed as desired. The 3D reconstruction450may be used to control one or more actions of the agent.

Depth estimation systems use one or more sensors to build three-dimensional (3D) representations of a local environment. In some cases, a depth estimation sensor may use a LIDAR sensor. Additionally, or alternatively, depth estimation systems may use cameras, such as a red-green-blue (RGB) camera. Aspects of the present disclosure are directed to a system for training and using a depth network to build 3D representation from two or more images captured by one or more sensors associated (e.g., integrated) with an agent. In some examples, each image captured by the one or more sensors may include different objects, such as dynamic and/or static objects, at different depths with respect to a reference location. In the present disclosure, a depth of an object in an image may refer to a distance of the object points (for example, pixels) from a reference location, such as a camera location.

Feature matching is a fundamental component of Structure-from-Motion (SfM). By establishing correspondences between points across frames, a wide range of tasks can be performed, including depth estimation ego-motion estimation, keypoint extraction, calibration, optical flow, and scene flow. Within these tasks, self-supervision enables learning without explicit ground-truth, by using view synthesis losses obtained via the warping of information from one image onto another, obtained from multiple cameras or a single moving camera. While more challenging from a training perspective, self-supervised methods can leverage arbitrarily large amounts of unlabeled data, which has been shown to achieve performance comparable to supervised methods, while enabling new applications such as test-time refinement and unsupervised domain adaptation.

In some conventional systems, single-frame self-supervised methods use multi-view information at training time, as part of the loss calculation. In contrast, multi-frame systems use multi-view information at inference time. For example, conventional systems may build cost volumes or correlation layers. These multi-frame systems learn geometric features in addition to appearance-based ones, which leads to better performance relative to single-frame methods.

However, multi-frame calculation relies heavily on feature matching to establish correspondences between frames, using only image information. Because of that, correspondences will be noisy and often inaccurate due to ambiguities and local minima caused by lack of texture, repetitions, luminosity changes, dynamic objects, and so forth.

Various aspects of the present disclosure improve self-supervised feature matching, focusing on the depth estimation, such as monocular depth estimation or multi-frame monocular depth estimation. In some implementations, a cost volume between target image features and context image features may be used to estimate a depth of an environment. Such depth estimates may be used for 3D reconstruction. In some examples, a 3D model of a scene may be estimated by determining a stereo correspondence between a first image and a second image of the scene. The stereo correspondence may be based on a cross-attention cost volume generated at a cross-attention cost volume generation model (e.g., cost volume model). Each image may be captured by a sensor associated with an agent, such as the vehicle100described with reference toFIGS.1A and1B. The stereo correspondence may be determined by matching pixels between the first image and the second image. The pixels may be matched based on a similarity metric for all pixel pairs given a series of possible depth ranges. The similarity metric of matching pixel pairs may satisfy a matching condition, such as a value of the similarity metric being less than a threshold. A low similarity metric value associated with two pixels may correspond to a high similarity between the two pixels.

FIG.5Ais a block diagram illustrating an example of a cross-attention cost volume generation model500, in accordance with various aspects of the present disclosure. For ease of explanation, the cross-attention cost volume generation model500may be referred to as the cost volume model500. As shown in the example ofFIG.5A, the cost volume model500may receive two input images502and504, having dimensions H×W×3, where H represents a height, W represents a width, and 3 represents a number of channels, such as red, green, and blue. A first image502may be a target image Itand a second image504may be a context image Ic. Each image502and504may be encoded by an encoder network514to produce C-dimensional target features506) and context features508(Fc) at a fraction of an original resolution, such as ¼ the original resolution. For each target feature ftuv∈Ft, corresponding to pixel Ft={u,v}, matching candidates may be sampled from context features508Fcalong an epipolar line εt→cuvby an epipolar sampler520(seeFIG.5B). In the present application, context features Fcmay also be represented as F(Ic). Additionally, target features Ftmay also be represented as F(It). In some examples, spatial-increasing discretization (SID) may be used to uniformly sample depth value in log space. Assuming D bins ranging from dminto dmax, each depth value diis given by:

In the example ofFIG.5A, a feature volume510(Ct→c) may be generated from the matching candidates. Dimensions of the feature volume510(Ct→c) may be H/4×W/4×D×C. Each cell (u, v, i) receives sampled features Ft→cuv=Fc(u′i,v′i), for i∈[0, . . . , D] whereis a bilinear sampling operator and (u′i, v′i) represent projected pixel coordinates, such that:

In Equation 2, Rt→cand tt→crepresent relative rotation (Rt→cand translation (tt→c) between frames, and K∈R3×3represent pinhole camera intrinsics. In practice, the relative rotation and translation may be predicted by a pose network, and pinhole camera intrinsics (K) may be a known constant.

In some examples, an attention model (not shown inFIG.5A) may compute a similarity between the target features506(Ft) and the feature volume510(Ct→c). In some such examples, L multi-head attention layers may be used to split the C feature channel dimensions into Nhgroups, such that Ch=C/Nh. Feature updates may be computed per head h of the L multi-head attention layers, each update may have different representations. For each attention head h, a set of linear projections are used to compute queries Qhfrom the target features Ft, and keys Khand values Vhfrom the feature volume Ct→c:

The output values V∈RCmay be obtained as a weighted concatenation of per-head output values:

In Equation 7, WO∈RC×C, bO∈RC, and ⊕ is a concatenation operation. Per-bin attention values

may be obtained by averaging over the number of heads. The process of obtaining per-bin attention values may be repeated L times, each using the output values V to update the feature volume for key and value calculation, such that Ct→cl+1=Vl. Final attention values may be used to populate a cross-attention cost volume512(At→c: The cross-attention cost volume512may be a H/4×W/4×D structure that encodes a similarity between each feature from the target features506) and candidates from the feature volume510(Ct→c). The feature volume510may also be referred to as sampled context features (Ct→c). Each cell (u, v, i) in the cross-attention cost volume512receives a corresponding attention value α(u′i, v′i) from a last cross-attention layer as the similarity metric for feature matching.

In some examples, the cross-attention may be alternated between the target features506) and candidates from the feature volume510(Ct→c) with self-attention among epipolar-sampled context features. In this setting, queries Q′hmay be calculated from the feature volume510(Ct→c), such that, Q′h=Ct→cW′Qh+b′Qh. The self-attention refinement step may take place after each cross-attention layer, and may be repeated L-1 times. It may be omitted from a last iteration because cross-attention weights a from the last layer L may be used to populate At→c, as opposed to output values V. Therefore, self-attention updates are not used for the last layer L.

FIG.5Bis a block diagram illustrating an example of an epipolar sampler520, in accordance with various aspects of the present disclosure. The example ofFIG.5Bmay be an example of depth-discretized epipolar sampling. In the example ofFIG.5B, for each target feature F(u, v), D matching candidates F(u′i, v′i) are sampled from a depth-discretized epipolar line εt→cuv. That is, for each target feature ftuv∈Ft, corresponding to pixel Ft={u, v} in a target image502, matching candidates may be sampled from context features508(Fc) along an epipolar line εt→cuvin a context image504.

The process for generating cost volumes from monocular information as described with reference toFIGS.5A and5Bmay fail if a camera associated with an ego vehicle is static between frames. Additionally, or alternatively, the process for generating the cost volumes may assume a static world, and may fail in the presence of dynamic objects. To circumvent these limitations, various aspects are directed to combining multi-frame cost volumes with features from a single-frame depth network. These features are then decoded jointly, which makes predicted depth maps robust to multi-frame failure cases.

According to various aspects of the present disclosure, a single-frame depth network is used as the teacher and a multi-frame depth network is used as the student. For ease of explanation, the single-frame depth network may be referred to as a single-frame network and the multi-frame depth network may be referred to as a multi-frame network. As discussed, depth estimates generated by the multi-frame networks may be more accurate than depth estimates generated by the single-frame network. Still, multi-frame networks may be more susceptible to dynamic objects and occlusions. Such that the depth estimates may not accurately account for dynamic objects and/or occlusions.

A single-frame network may remedy the shortcomings of the multi-frame network in the presence of dynamic objects and/or occlusions. Still, dynamic objects and/or occlusions may also cause the single-frame network to fail, or generate less accurate depth estimates, during training time when two or more frames are used to produce the selfsupervised objective for minimization. Such errors may not be present at test time, when only a single frame is considered. In some examples, during training, a photometric loss (e.g., photometric error) in areas of an image that contain a dynamic object and/or occlusions may be greater than a photometric loss in areas of the image that include static objects. A location of a static object does not change between subsequent frames. In contrast, a location of a dynamic object may change between subsequent frames.

In some aspects, a mask may be used to remove areas associated with a photometric error that satisfies a removal condition. In some examples, the removal condition may be satisfied based on the photometric error being greater than or equal to an error threshold. In some such examples, the mask may discard pixels that may cause a production of an incorrect selfsupervisory signal. Conventional systems may mask pixels via re-projection and auto-masking. In the present disclosure, the photometric loss itself may be used as a mask to improve the quality of the multi-frame network.

FIG.6Aillustrates an example of a pipeline for a single-frame depth estimation network600, in accordance with various aspects of the present disclosure. As shown inFIG.6A, the training pipeline may be used to train a depth network630and a pose network650. The depth network300receives a target image (II)502. The pose network400receives the target image502and one or more source images (Is)606. The source images606are also referred to as context images. The single-frame depth estimation network600may also be referred to as a single-frame depth estimation model.

As shown inFIG.6A, the depth network630generates a depth map (Dt)608of the target image502. As discussed, the depth map608may be a per-pixel depth map. A view estimation module610receives the output of the depth network300and the six DoF transformation (e.g., relative pose) output of the pose network400. As discussed, six DoF transformation determined by the pose network400is a transformation between the target image502and the context image504. The view estimation module610warps the context image504to reconstruct the target image502. The reconstructed target image may be referred to as the warped image612(Ît).

Specifically, the view estimation module610generates a transformation matrix between frames. In one configuration, the transformation matrix is a per pixel transformation matrix. The view estimation module610warps the pixels based on the per pixel transformations to generate an image reconstruction. The reconstruction may be considered a reconstruction of the context image504as viewed by the target image502. The context image504may also be referred to as a source image. In some aspects, the local transformation individually warp each pixel from the target image502. Each pixel may be warped with a corresponding depth estimate to reconstruct the context image504.

That is, the warped image612is generate by sampling pixels from the target image502based on the predicted depth map608and the transformations (e.g., global transformation and local transformation). For example, each point (e.g., pixel) in the target image502is projected onto the context image504based on the predicted depth map608and the transformation matrix. Bilinear interpolation of pixels neighboring the target image pixel projected onto the context image504may approximate a value of the target pixel. The approximated value may be used as the value of the pixel in the warped image612. In some examples, the bilinear sampling mechanism linearly interpolates values of multiple pixel neighbors (such as, top-left, top-right, bottom-left, and bottom-right) of the target pixel projected onto the source the context image504. That is, the color of the pixel in the warped image612may be based on neighboring pixels in the context image504. The warped image612may be a 3D reconstruction of a 2D target image.

During training of the single-frame depth estimate network, a photometric loss is calculated based on the difference between the target image502and the warped image612(e.g., the warped source image that approximates the target image). As shown inFIG.6A, the training pipeline600determines a photometric loss616based on a comparison of the warped image612and the target image502. The photometric loss616may be used to update the depth network630, the view estimation module610, and/or the pose network650.

The photometric loss616(Lp) may be determined as follows:

where SSIM( ) is a function for estimating a structural similarity (SSIM) between the target image502and the warped image612. SSIM may be determined as follows:

where s( ) determines a structure similarity, c( ) determines a contrast similarity, and l( ) determines a luminance similarity. α, β, and γ are parameters for adjusting a relative importance of each component, and each parameter is greater than zero. The photometric loss616(Lp) consists of a weighted sum between a structural similarity (SSIM) and absolute error (L1) terms.

In some examples, the photometric loss616may be used to guide the generation of correct pixel matches to increase an accuracy of cost volumes (e.g., multi-frame network700described with reference toFIG.7). In such examples, the photometric loss may bounded between two values, such as [0,1]. In such examples, a photometric loss616with a value of one may produce a soft mask. The soft mask may include low values, such as zero, where the photometric loss616is greater than a threshold, and high values, such as one, where the photometric loss616is less than a threshold. That is, a photometric loss616that is greater than a threshold may yield low mask values. The soft mask may then be multiplied with the multi-frame photometric loss, such that areas where the single frame network does not model correctly are weighed less. In some examples, the soft mask may be multiplied with the multi-frame photometric loss after detachment, such that gradients are not back-propagated. An accuracy of the depth estimates of the multi-frame network may increase as a result of using the soft mask.

During a testing stage, the training pipeline600may generate the warped image612as described above. The photometric loss616may not be calculated during a testing stage. The warped image612may be used for localization and/or other vehicle navigation tasks.

FIG.6Billustrates an example of a depth network630for a single-frame depth estimate network, in accordance to aspects of the present disclosure. As shown inFIG.6B, the depth network630includes an encoder632and a decoder634. The depth network630generates a per-pixel depth map, such as the depth map440ofFIG.4B, of an input image502.

The encoder632includes multiple encoder layers632a-d. Each encoder layer632a-dmay be a packing layer for downsampling features during the encoding process. The decoder634includes multiple decoder layers634a-d. InFIG.6B, each decoder layer634a-dmay be an unpacking layer for upsampling features during the decoding process. That is, each decoder layer634a-dmay unpack a received feature map.

Skip connections636transmit activations and gradients between encoder layers632a-dand decoder layers634a-d. The skip connections636facilitate resolving higher resolution details. For example, a gradient may be directly back-propagated to layers via the skip connections636, thereby improving training. Additionally, the skip connections636directly transmit image details (e.g., features) from convolutional layers to deconvolutional layers, thereby improving image recovery at higher resolutions.

The decoder layers634a-dmay generate intermediate inverse depth maps310. Each intermediate inverse depth map640may be upsampled before being concatenated with a corresponding skip connection636and feature maps unpacked by a corresponding decoder layer634a-d. The inverse depth maps640also serve as the output of the depth network from which the loss is calculated. In contrast to conventional systems that incrementally super-resolve each inverse depth map640. Aspects of the present disclosure upsample each inverse depth map640to a highest resolution using bilinear interpolation. Upsampling to the highest resolution reduces copy-based artifacts and photometric ambiguity, thus improving depth estimates.

FIG.6Cillustrates an example of a pose network650for ego-motion estimation, in accordance with various aspects of the present disclosure. In contrast to conventional pose networks, the pose network650ofFIG.4does not use explainability masks.

As shown inFIG.6C, the pose network650includes multiple convolutional layers652, a final convolutional layer654, and a multi-channel (e.g., six-channel) average pooling layer656. The final convolutional layer654may be a 1×1 layer. The multi-channel layer656may be a six-channel layer.

In one configuration, a target image (It)502and a source image (Is)606are input to the pose network650. The target image502and source image410may be concatenated when input to the pose network650. During training, one or more source images606may be used during different training epochs. The source images606may include an image at a previous time step (t-1) and an image at a subsequent time step (t+1). The output is a set of six DoF transformations between the target image502and the context image504. The process may be repeated for each context image504if more than one context image504is considered.

As discussed, a teacher-student training procedure may be used to improve the performance of multi-frame predictions via the supervision of the single-frame network600in areas where cost volume generation fails. This single-frame network600is trained jointly, sharing the same pose predictions, and discarded during evaluation.

FIG.7is a block diagram illustrating an example of a depth-prediction architecture700, in accordance with various aspects of the present disclosure. As shown inFIG.7, the depth-prediction architecture700receives a context image504and a target image502. Target features) (not shown inFIG.7) may be generated by processing the target image502by a multi-frame encoder514. Additionally, context features (Fc) (not shown inFIG.7) may be generated by processing the context image504by a multi-frame encoder514. The target features and context features may be processed via epipolar sampler520. Cross-attention matching702may be performed based on the output of the epipolar sampler520to generate a cross-attention cost volume512. The cross-attention matching702is performed as described with reference toFIG.5Ain relation to the target features506) and the feature volume510(Ct→c). The cross-attention cost volume512may be processed by a depth decoder, such as the depth decoder634discussed with reference toFIG.6B, to generate a depth map704. In some examples, a multi-frame network may include the multi-frame encoder500and the depth decoder645.

Additionally, as shown inFIG.7, during training, the target image502and the context image504may be processed by a single-frame network600. As discussed, a mask generated based on the photometric loss616of the single-frame network600may be multiplied with a photometric loss of the multi-frame network. In the example ofFIG.7, implementation, a teacher network is used to supervise the cost volume. In some such implementations, the teacher network is single-frame network600described with reference toFIGS.6A,6B,6C, and7. As discussed, an accuracy of depth estimate of a multi-frame depth estimate network (e.g., cost volume-based depth estimate network) may be reduced due to the presence of dynamic objects, changes in viewpoints, errors in pose estimation, object occlusion, and/or textureless areas. In contrast, during testing, an accuracy of a depth estimate of the single-frame network600(e.g., single-frame depth estimation network) may not be affected by the presence of dynamic objects, changes in viewpoints, errors in pose estimation, object occlusion, and/or textureless areas. However, during training, the accuracy of the depth estimate of the single-frame network600may be reduced due to the presence of dynamic objects, changes in viewpoints, errors in pose estimation, object occlusion, and/or textureless areas. Therefore, as discussed, a mask may be generated based on the photometric loss616of the single-frame network600, and the mask may be used to guide the photometric loss of the multi-frame encoder500.

During training, the multi-frame network may determine a loss based on a difference between a ground-truth depth estimate and the depth estimate704. In some examples, the multi-frame network may be trained to minimize the loss between the ground-truth depth estimate and the depth estimate704. In some such examples, the ground-truth depth estimate may be provided by a sensor, such as a LIDAR sensor, or another source, such as previously determined depth estimates. In other such examples, a teacher network may be used to provide the ground-truth depth estimates.

As discussed, in some aspects, the ground-truth depth estimate may be provided by a sensor, such as a LIDAR sensor, or another source, such as previously determined depth estimates. As an example, a LIDAR sensor associated with an agent, such as a vehicle100as described with reference toFIGS.1A and1B, may generate depth estimates for a scene. The depth estimates generated by the LIDAR sensor may be used as ground-truth depth estimates for a depth estimation network, such as a depth estimation network that estimates depth based on a cost volume. The depth estimates may also be referred to as a depth map.

The multi-frame network may be trained end-to-end using a photometric reprojection loss, consisting of a weighted sum between a structural similarity (SSIM) and absolute error (L1) terms. The photometric loss (Lp) may be determined as follows:

where SSIM( ) is a function for estimating a structural similarity (SSIM) between the target image502and a warped image. The SSIM may be determined as follows:

where s( ) determines a structure similarity, c( ) determines a contrast similarity, and l( ) determines a luminance similarity. α, β, and γ are parameters for adjusting a relative importance of each component, and each parameter is greater than zero. As discussed, the photometric reprojection loss of the multi-frame network may be multiplied by the mask generated based on photometric loss616of the single-frame network600to improve an accuracy of a depth estimate704. In some examples, a reconstructed pointcloud706(e.g., 3D image) may be generated based on the depth estimate704. In such examples, an accuracy of the pointcloud704is increased based on an increase in the accuracy of the depth estimate704.

In some examples, a photometric loss may be determined based on two frames, by warping information. In some aspects, the photometric loss616associated with the single-frame network600may be used to improve the performance (e.g., depth estimate accuracy) of the multi-frame depth network700, by weighting a photometric loss of the multi-frame depth network700based on the photometric loss616from the single-frame network600. The single-frame photometric loss616may be applied as weights to the per-pixel multi-frame photometric loss associated with the multi-frame depth network700, such that higher errors lead to smaller weights. That is, weights associated with dynamic objects may be lowered. By lowering the weights associated with dynamic objects, the the multi-frame depth network700may increase an accuracy of depth estimates of static objects as well as dynamic objects.

Specifically, as previously discussed, the photometric loss616may be used to guide the generation of correct pixel matches to increase an accuracy of cost volumes determined by the multi-frame network700. In such examples, the photometric loss may bounded between two values, such as [0,1]. In such examples, a photometric loss616with a value of one may produce a soft mask. The soft mask may include low values, such as zero, where the photometric loss616is greater than a threshold, and high values, such as one, where the photometric loss616is less than a threshold. That is, a photometric loss616that is greater than a threshold may yield low mask values. The soft mask may then be multiplied with the multi-frame photometric loss, such that areas where the single frame network does not model correctly are weighed less. In some examples, the soft mask may be multiplied with the multi-frame photometric loss after detachment, such that gradients are not back-propagated. An accuracy of the depth estimates of the multi-frame network may increase as a result of using the soft mask.

FIG.8is a diagram illustrating an example process800performed in accordance with various aspects of the present disclosure. The process800may be performed by a vehicle, such as a vehicle100as described with reference toFIGS.1A and1B, and/or a depth estimation module of a vehicle, such as the depth estimation system390as described with reference toFIG.3. The vehicle may be referred to as an agent. The example process800is an example of depth estimation. As shown in the example ofFIG.8, the process800begins at block802by generating, via a cross-attention model, a cross-attention cost volume based on a current image of the environment and a previous image of the environment in a sequence of images. The cross-attention model may be an example of the multi-frame network700described with reference toFIG.7. The current image and the previous image may be obtained from a monocular camera associated with the vehicle. In some examples, the current image and the previous image are two-dimensional (2D) images.

In some examples, the process800also generates current image features from the current image via a feature extraction network. Each one of the current image features corresponds to a current image pixel in the current image. Furthermore, the process800generates previous image features from the previous image via the feature extraction network. Each one of the previous image features corresponds to a previous image pixel in the previous image. In such examples, the cross-attention cost volume is generated based on cross-attention matching each feature from the current image features with one or more features of the previous image features.

The cross-attention matching may include sampling, for each current image pixel, one or more candidate pixels from the previous image corresponding to the current image pixel along an epipolar line. Additionally, the cross-attention matching may include matching, for each current image pixel, a current image feature associated with the current image pixel with each previous image feature associated with the one or more sampled candidate pixels corresponding to the current image pixel.

At block804the process generates, via the cross-attention model, a depth estimate of the current image based on the cross-attention cost volume. The cross-attention model may be trained using a photometric loss associated with a single-frame depth estimation model. In some examples, the process800generates a three-dimensional (3D) reconstruction of the environment via the depth estimate. The single-frame depth estimation model may be an example of the single-frame depth estimation network600described with reference toFIG.6A. In some examples, the photometric loss is based on a difference between a target image and a reconstruction of the target image. The photometric loss may be an example of the photometric loss616described with reference toFIG.6A.

Finally, at block806, the process800controls an action of the vehicle based on the depth estimate. The vehicle may be an example of an autonomous vehicle or a semi-autonomous vehicle. In some examples, the action may include planning a route and/or navigating through the environment.

Based on the teachings, one skilled in the art should appreciate that the scope of the present disclosure is intended to cover any aspect of the present disclosure, whether implemented independently of or combined with any other aspect of the present disclosure. For example, an apparatus may be implemented or a method may be practiced using any number of the aspects set forth. In addition, the scope of the present disclosure is intended to cover such an apparatus or method practiced using other structure, functionality, or structure and functionality in addition to, or other than the various aspects of the present disclosure set forth. It should be understood that any aspect of the present disclosure may be embodied by one or more elements of a claim.

The various illustrative logical blocks, modules and circuits described in connection with the present disclosure may be implemented or performed with a processor specially configured to perform the functions discussed in the present disclosure. The processor may be a neural network processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array signal (FPGA) or other programmable logic device (PLD), discrete gate or transistor logic, discrete hardware components or any combination thereof designed to perform the functions described herein. Alternatively, the processing system may comprise one or more neuromorphic processors for implementing the neuron models and models of neural systems described herein. The processor may be a microprocessor, controller, microcontroller, or state machine specially configured as described herein. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or such other special configuration, as described herein.

The steps of a method or algorithm described in connection with the present disclosure may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in storage or machine readable medium, including random access memory (RAM), read only memory (ROM), flash memory, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, a hard disk, a removable disk, a CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer. A software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media. A storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.

The processor may be responsible for managing the bus and processing, including the execution of software stored on the machine-readable media. Software shall be construed to mean instructions, data, or any combination thereof, whether referred to as software, firmware, middleware, microcode, hardware description language, or otherwise.

If implemented in software, the functions may be stored or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media include both computer storage media and communication media including any storage medium that facilitates transfer of a computer program from one place to another.

Further, it should be appreciated that modules and/or other appropriate means for performing the methods and techniques described herein can be downloaded and/or otherwise obtained by a user terminal and/or base station as applicable. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described herein. Alternatively, various methods described herein can be provided via storage means, such that a user terminal and/or base station can obtain the various methods upon coupling or providing the storage means to the device. Moreover, any other suitable technique for providing the methods and techniques described herein to a device can be utilized.