3D SENSING AND VISIBILITY ESTIMATION

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for determining the visibility of query points using depth estimates generated by a neural network.

BACKGROUND

This specification relates to predicting visibility of three-dimensional query points in an environment based on a sensor image.

The environment can be a real-world environment, and the query point can be any point in the environment surrounding an agent, e.g., an autonomous vehicle. Predicting visibility of the query point can be a task required for motion planning of the agent.

Autonomous vehicles include self-driving cars, trucks, boats, and aircraft. Autonomous vehicles use a variety of on-board sensors and computer systems to detect nearby objects and use such detections to make control and navigation decisions.

Some autonomous vehicles have on-board computer systems that implement neural networks or other types of machine learning models for various prediction tasks, e.g., object classification within images. For example, a neural network can be used to determine that an image captured by an on-board camera is likely to be an image of a nearby obstruction, e.g., a car or a pedestrian.

SUMMARY

This specification describes how a computer system can use a neural network to generate visibility predictions for three-dimensional query points located in an environment surrounding an agent. The agent may be any type of vehicle including, for example, cars, trucks, motorcycles, buses, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, trams, golf carts, trains, and trolleys. The agent may be an autonomous vehicle or a semi-autonomous vehicle.

The visibility prediction can be generated from an input image, e.g., a single image, captured by a sensor of the agent, e.g., a camera sensor. The visibility prediction can be a prediction of whether a three-dimensional query point in the environment is visible or not in the single image. In other words, the visibility prediction indicates whether the three-dimensional query point is visible in the image, i.e., whether or not the three-dimensional query point is occluded by an object, out of range of the sensor, or is otherwise not able to be seen in the image.

An autonomous or semi-autonomous vehicle system can use a fully-learned neural network to generate visibility predictions of a scene from an input image captured by a sensor that characterizes the scene. This can allow the vehicle to plan a reliable and safe future motion trajectory. Such visibility prediction advantageously relies on monocular depth estimation based on an image captured by the camera, and does not require any additional depth information of the scene, such as depth maps or lidar or radar readings.

Because cameras can sense significantly farther and at higher angular resolution than lasers or radars, the visibility prediction generated from the camera image can allow prediction for locations that cannot be sensed by lasers or radars. In addition, cameras can be less impacted by adverse weather conditions such as fog or rain than lasers or radars, so that visibility predictions using camera images can be utilized under a wider variety of driving conditions. By using two-dimensional camera pixels and mapping them into three dimensional space in relation to the three-dimensional query points, the computer system can make camera signals more accessible to perception systems, e.g., those that are deployed on-board autonomous vehicles.

The described techniques can also estimate the uncertainty associated with the visibility predictions. Such uncertainty can be used as a quantitative indication of the expected relative amount of error in visibility prediction, and can be used in combination with the visibility predictions for making more robust, reliable, and safer control of the agent, i.e., control that is robust against prediction errors. Further, a safety margin can be customized and multiplied to the estimated uncertainty to enable the agent to move more conservatively or aggressively when needed.

The neural network is advantageously trained using ground truth depths obtained from LiDAR images that can be more accurate and reliable than depths generated using other image sources. The training samples include camera images with a variety of driving conditions and situations, such as with the presence of cyclists or pedestrian crowds, adverse weather, and night scenes.

Further, the neural network can include a depth estimation head and an uncertainty estimation head, and the two heads can be trained on two different parts of the training examples to improve the network’s depth and uncertainty estimation.

DETAILED DESCRIPTION

This specification describes how a computer system can use a neural network to generate visibility predictions for three-dimensional query points located in an environment surrounding an agent. The agent may be any type of vehicle including, for example, cars, trucks, motorcycles, buses, recreational vehicles, amusement park vehicles, farm equipment, construction equipment, trams, golf carts, trains, and trolleys. As a particular example, the agent may be, e.g., an autonomous vehicle.

The visibility prediction can be generated using an input image, e.g., a single image, captured by a sensor of the agent, e.g., a front-facing camera. The visibility prediction can be a prediction of whether a three-dimensional query point in the environment is visible or not. In other words, the visibility prediction indicates whether the three-dimensional query point is visible in the image, i.e., whether the three-dimensional query point is occluded by an object, out of range of the sensor, or is otherwise not able to be seen in the image.

FIG.1is a diagram of an example computer system100.

The computer system100can include a training system110and an on-board system120. The on-board system120can be physically located on-board an agent122. Being on-board the agent122means that the on-board system120includes some or all of components that travel along with the agent122, e.g., power supplies, computing hardware, and sensors.

The agent122inFIG.1is illustrated as an automobile, but the on-board system110can be located on-board any appropriate vehicle or agent type.

In some cases, the agent122is an autonomous vehicle. An autonomous vehicle can be a fully autonomous vehicle that determines and executes fully-autonomous driving decisions in order to navigate through an environment. An autonomous vehicle can also be a semi-autonomous vehicle that uses predictions to aid a human driver. For example, the vehicle can autonomously apply the brakes if a prediction indicates that a human driver is about to collide with another vehicle. As another example, the vehicle can have an advanced driver assistance system (ADAS) that assists a human driver of the vehicle in driving the vehicle by detecting potentially unsafe situations and alerting the human driver or otherwise responding to the unsafe situation. As a particular example, the vehicle can alert the driver of the vehicle or take an autonomous driving action when an obstacle is detected, when the vehicle departs from a driving lane, or when an object is detected in a blind spot of the human driver.

Generally, the agent122uses visibility outputs to inform fully-autonomous or semi-autonomous driving decisions. For example, the agent122can autonomously apply the brakes if a visibility output indicates with an uncertainty satisfying a threshold that a human driver is about to navigate onto static obstacles, e.g., a paved sidewalk or other non-road ground surface. As another example, for automatic lane changing, the agent122can use visibility output(s) to analyze available space surrounding a target lane to ensure that there is no fast approaching traffic before starting a lane changing operation. The agent122can also use the visibility output to identify situations when the road is not occluded and thus trigger alerts to the driver.

The on-board system120can include one or more sensor subsystems132. The sensor subsystems132include a combination of components that receive reflections of electromagnetic radiation. In particular, the sensor subsystems include one or more camera systems that capture images155, i.e., that detect reflections of visible light and optionally one or more other types of systems e.g., laser systems that detect reflections of laser light, radar systems that detect reflections of radio waves, and camera systems that detect reflections of visible light.

The sensor subsystems132provide the captured images155to a depth prediction system134.

The depth prediction system134implements the operations of each layer of a neural network trained to generate depth prediction outputs for some or all pixels in an input image155. That is, the neural network receives as input a single image155and generates as output a depth prediction output for the image155.

Each depth prediction output can include a respective estimated depth that estimates a distance between the sensor, i.e., the camera system that captured the image, and a portion of the scene depicted at the pixel in the image.

For example, the neural network can be a convolutional neural network that includes a first subnetwork that processes the image to generate a feature representation of the image. For example, the feature representation can be a feature map that includes a respective feature vector for each of a plurality of regions of the image155.

The neural network can also include a depth estimate neural network head that processes the feature representation to generate the respective estimated depths for each of the plurality of pixels.

Optionally, the neural network can also include an uncertainty neural network head that processes the feature representation to generate a respective estimated uncertainty for each of the plurality of pixels. The estimated uncertainty for a given pixel is a score that measures how confident the neural network is in the estimated depth for the given pixel. For example, each uncertainty estimate can be a score between zero and one, inclusive, with a score of one indicating that the neural network is completely uncertain about the estimated depth and a score of zero indicating that the neural network is completely certain about the estimated depth.

The neural network is described in more detail below with reference toFIGS.2A-4.

The depth prediction system134can implement the operations of the neural network by loading a collection of model parameter values172that are received from the training system110. Although illustrated as being separated, the model parameter values170and the software or hardware modules performing the operations may actually be located on the same computing device or, in the case of an executing software module, stored within the same memory device.

The depth prediction system134can use hardware acceleration or other special-purpose computing devices to implement the operations of one or more layers of the neural network. For example, some operations of some layers may be performed by highly parallelized hardware, e.g., by a graphics processing unit or another kind of specialized computing device. In other words, not all operations of each layer need to be performed by central processing units (CPUs) of the depth prediction system134.

The depth prediction system134can communicate the depth prediction outputs to a planning subsystem136. Optionally, the sensor subsystem132may communicate the captured image155to the planning subsystem136.

The planning subsystem136can obtain a three-dimensional (3D) query point. For example, the 3D query point can be a point on a planned motion path of the agent122or another point of interest to the future navigation of the agent122. The 3D query point can correspond to one or more pixels of the captured image155. That is, the planning subsystem136can receive both a query point and data specifying the corresponding pixel for the query point. Alternatively, such correspondence between a 3D query point and a pixel in the image can be determined by the planning subsystem136, for example, by performing a 3D calibration that registers the 3D query point in a world coordinate system to the pixel(s) in the image coordinate system. That is, the system can map the query point from a specified three-dimensional coordinate system, e.g., one that is centered at the agent122or at another 3d point, to the two-dimensional image coordinate system using calibration data that “calibrates” 2d points in the image coordinate system.

The planning subsystem136can then determine whether the 3D query point is visible or not in the captured image155.

For example, the visibility output can be a binary indicator that has one value, e.g., one or zero, when the system predicts that the point is visible in the image and another, different value, e.g., zero or one, when the system predicts that the point is not visible in the image.

The planning subsystem136can compare the depth prediction output165which includes an estimated depth, d(u, v), for a corresponding pixel, at coordinates (u,v), of the captured image155of the 3D query point with a depth of the 3D query point, x′, i.e., a distance between the 3D query point and the sensor. In particular, if d(u, v) > x′, the query point is visible or in free space. If d(u, v) ≤ x′, the query point is not visible or occluded, where u and v are coordinates of the corresponding pixel in the image coordinate system, d() is the depth, and x′ represents the distance from the sensor to the query point in the sensor coordinate system.

When the neural network also generates uncertainty estimates, the system136can also use the uncertainty estimate for the corresponding pixel to generate the visibility output. In particular, the system136may calculate a modified estimated depth using the estimated depth and the estimated uncertainty. The modified estimated depth can be negatively correlated with the uncertainty so that the depth estimation can be more robust and reliable. The system134can compare the modified estimated depth with the distance between the query point and the sensor to determine the visibility output. A similar binary indicator can be used for the visibility output.

The planning subsystem136can then use the visibility output165to make fully-autonomous or semi-autonomous driving decisions. For example, the planning subsystem136can generate a fully-autonomous plan to navigate on a highway or other road by querying the visibility output(s) to differentiate free or visible areas in the vicinity of the agent122from areas where there are occlusions. By identifying occlusions, during a turn operation, the vehicle can perform a necessary yield operation to a potential object, e.g., a car, a cyclist, or a pedestrian. As another example, the planning subsystem136can generate a semi-autonomous plan for a human driver to navigate the car using the visibility output(s).

A user interface subsystem138can receive depth prediction output165and can generate a user interface display, e.g., on a graphic user interface (GUI) that indicates the depth map of nearby objects, e.g., a road or a nearby vehicle. For example, the user interface subsystem138can generate a user interface presentation having image or video data containing a representation of the regions of space that have depth value satisfying a certain threshold. An on-board display device can then display the user interface presentation for passengers or drivers of the agent122.

The depth prediction system134can also use the image data155to generate training data123. The on-board system120can provide the training data123to the training system110in offline batches or in an online fashion, e.g., continually whenever it is generated.

The training system110can be hosted within a data center112, which can be a distributed computing system having hundreds or thousands of computers in one or more locations.

The training system110can include a training neural network subsystem114that can implement the operations of each layer of a neural network that is designed to make depth predictions from input image data. The training neural network subsystem114can include a plurality of computing devices having software or hardware modules that implement the respective operations of each layer of the neural network according to an architecture of the neural network.

The training neural network generally has the same architecture and parameters as the on-board neural network. However, the training system110need not use the same hardware to compute the operations of each layer. In other words, the training system110can use CPUs only, highly parallelized hardware, or some combination of these.

The training neural network subsystem114can receive as input training examples123aand123bthat have been selected from a set of labeled training data125.

Each of the training examples123includes a training image and a ground truth depth output for the training image that assigns a respective ground truth depth to at least a portion of the pixels in the training image. The respective ground truth depths can be obtained from light detection and ranging (LiDAR) scans projected onto and matching scenes of the training images in the training examples. The LiDAR images can be taken by the on-vehicle sensor subsystem132, e.g., by laser sensors.

In some cases, the training examples can be separated into two different subsets123aand123b.

For each training example in the first subset123a, the training neural network subsystem114can process the training image in the training example using the neural network to generate a training depth prediction output that includes a respective estimated training depth and a respective estimated training uncertainty for each of the pixels in the training image in the training example.

For each training example in a second subset of the training examples123b, the training neural network subsystem114can process the training image in the training example using the neural network to generate a training depth prediction output that includes a respective estimated training depth and a respective estimated training uncertainty for the pixels of the training image in the training example.

A training engine116can analyze the predictions from the first and second subset of training examples135a,135band compare them to the labels in the training examples123. In particular, the training engine116can, for each training example in the second subset of the training examples123b, compute a respective target uncertainty for each of the pixels in the training image from an error between (i) the respective ground truth depth for the pixel and (ii) the respective estimated training depth for the pixel, i.e., so that the target uncertainty is larger when the error is larger. For example, the target uncertainty can be equal to the error divided by a scaling factor to ensure that the target uncertainty falls in a certain range, e.g., zero to 1.

A training engine116can determine a first update to parameters of the neural network by computing gradients of a depth objective, i.e., an objective function that measures errors between the respective estimated training depths and the respective ground truth depths for the training examples in the first subset. Similarly, the training engine115can determine a second update to the parameters of the neural network by computing gradients of an uncertainty objective, i.e., the same objective function or a different objective function that measures errors between the respective estimated training uncertainties and respective target uncertainties for the training examples in the second subset.

Based on the first update and/or second update of the neural network, the training engine116then generates updated model parameter values145by using an appropriate updating technique, e.g., stochastic gradient descent with backpropagation. The training engine116can then update the collection of model parameter values170using the updated model parameter values145.

The system can repeatedly perform this updating on training examples sampled from the first and second subsets to train the neural network.

In some other cases, the training examples are not separated into subsets, and the training system110trains the neural network by making use of both the depth objective and the uncertainty objective to compute a single update for each training example.

After training is complete, the training system110can provide a final set of model parameter values171to the on-board system120for use in making fully autonomous or semi-autonomous driving decisions. The training system110can provide the final set of model parameter values171by a wired or wireless connection to the on-board system120.

FIGS.2A-2Cillustrate examples of depth prediction outputs and estimated uncertainties associated from images202.

The images202can be camera images taken from the same camera or different cameras in the sensor subsystem120inFIG.1. The different cameras can be positioned at different locations of the agent122. The camera images202can capture a scene of the environment with trees, cars, roads, paved sidewalks, and grass areas, etc.

The camera images202can capture a portion of a road that is relatively far from the location of the camera and is farther than can be sensed by LiDAR or radar sensors. Therefore, it is beneficial to estimate depth thus visibility from the camera images to allow depth values to be available for locations that cannot be sensed by LiDAR or radar.

The system can provide the camera images202as input to a neural network trained to identify depths201and associated uncertainty estimates203.

In particular, inFIGS.2A-2C, larger depth values are represented with darker colors. Thus, for example, as shown inFIG.2B, the estimated depth of the nearby cars are less than the depth of the farther away sidewalk and trees because the cars are represented with a relatively lighter color. More generally, however, depths in a depth image can be color-coded or represented in grayscale in any appropriate way. Different shade or color may represent different depths, and the depth image may be presented by the user interface subsystem138to the user.

For the uncertainty203associated with depth estimation, as shown inFIG.2C, points with greater uncertainty are represented with lighter colors. Thus, the edge of the nearby cars has a higher uncertainty than the other regions on the cars. Such uncertainty may be partly caused by the partial volume effect of the image202, which is that a given pixel may include part of the car and also part of the other objects.

FIG.3Ashows an example architecture300of the neural network.

As shown inFIG.3A, the neural network includes an encoder subnetwork310that receives a camera image155and processes the camera image155to generate a feature representation312of the image155. For example, the feature representation312can be a feature map that includes a respective feature vector for each of multiple regions within the image155. In some cases, each region corresponds to a different pixel of the image155while in other cases, each region includes multiple pixels from the image155. As one example, the encoder subnetwork310can be a convolutional encoder, a self-attention encoder, or an encoder that has both convolutional and self-attention layers.

The neural network also has a depth estimate neural network head320that receives the feature representation312and processes the representation312to generate depth predictions340, i.e., to generate depth estimates for some or all of the pixels of the image155. For example, the depth estimate neural network head320can be a fully-convolutional neural network or can be a neural network that has both convolutional and fully-connected layers.

The neural network also has an uncertainty estimate neural network head330that receives the feature representation312and processes the representation312to generate uncertainty estimates350, i.e., to generate uncertainty estimates for some or all of the pixels of the image155. For example, the uncertainty estimate neural network head330can be a fully-convolutional neural network or can be a neural network that has both convolutional and fully-connected layers.

As described above, the neural network heads320and330can be trained on different subsets, e.g., randomly selected subsets, of a set of training data.

For example, the system can first train the neural network head320and the encoder subnetwork310on a first subset of the training data to generate depth estimates, i.e., on a loss function that measures errors between depth predictions and ground truth depths. The system can then train the neural network head330to generate uncertainty estimates on a second subset of the training data while holding the neural network head320and the encoder subnetwork310fixed, i.e., on a loss function that measures errors between uncertainty estimates and target uncertainty estimates computed as described above.

As another example, each batch of training data can include training examples from both the first and second subsets, and the system can jointly train the neural network head320, the encoder subnetwork310, and the neural network head330on an objective function that measures both (i) errors between depth predictions and ground truth depths and (ii) errors between uncertainty estimates and target uncertainty estimates computed as described above.

As yet another example, the system can alternate between training the neural network on batches of training examples from the first subset and batches of training examples from the second subset.

As yet another example, the training examples are not separated into subsets, and the training system trains the neural network by making use of both the depth objective and the uncertainty objective to compute a single update for each batch of training examples. That is, the system uses an objective function that measures both (i) errors between depth predictions and ground truth depths and (ii) errors between uncertainty estimates and target uncertainty estimates computed as described above for all of the training examples in the batch.

FIG.3shows an example of visibility predictions generated for a region of the environment that is in the vicinity of the planned motion trajectory302of the agent122.

As described above, the system can obtain depth images and 3D query points that are in front of the agent and/or along candidate future motion paths of the agent.

The system can then determine visibility outputs for pixels corresponding to the 3D query points based on the respective estimated depths of the pixels and the distances from the query points to the camera sensor. Based on the visibility outputs, the system can determine whether a 3D query point in the environment is visible or not, thereby generating a future motion trajectory302that includes only 3D query points that are visible as shown inFIG.3B.

FIG.4is a flowchart of an example process400for generating visibility outputs from an image using the neural network. The example process inFIG.4can use a neural network that has already been trained to estimate depths in camera images. The example process can thus be used to make predictions from unlabeled input, e.g., a single image taken by an on-board camera. The process will be described as being performed by an appropriately programmed neural network system.

The system obtains an image captured by a sensor and characterizing a scene in an environment (410). The image can be a camera image generated from the camera subsystem in a sensor subsystem of an agent, e.g., a vehicle.

The system can process the image using a neural network, e.g., a deep neural network, to generate a depth prediction output that includes a respective estimated depth for each of a plurality of pixels in the image (420).

The depth prediction output can estimate a distance between the camera and a portion of the scene depicted at the pixel in the image.

The depth prediction output can further include a respective estimated uncertainty for each of the plurality of pixels in the image that estimates uncertainty associated with the respective estimated depth.

The system can obtain one or more 3D query points (430). For example, the 3D query points can be points along a candidate future motion trajectory of the agent. Each query point can be represented in a three-dimensional coordinate system, e.g., centered at the agent or at a different, fixed point in the environment.

For each query point, the system can identify a corresponding pixel for the query point in the image (440). The corresponding pixel for a given query point is the pixel to which the query point is projected when the query point is mapped from the three-dimensional coordinate system to the two-dimensional image coordinate system.

For each query point, the system can determine a visibility output for the corresponding pixel based on the respective estimated depth for the corresponding pixel and a distance between the three-dimensional query point and the sensor (450). The system can compute the distance between the three-dimensional query point and the sensor by, e.g., computing Euclidean distance between a point on the sensor, e.g., a specified point on the surface of the sensor or at the center of the sensor, in the three-dimensional coordinate system.

The visibility output can characterize whether the three-dimensional query point is visible in the image or not.

For example, the visibility output can be a binary indicator that has one value, e.g., one or zero, when the system predicts that the point is visible in the image and another, different value, e.g., zero or one, when the system predicts that the point is not visible in the image.

As described above, the depth outputs include, for a given pixel at coordinates (u,v) in the image, an estimated depth, d(u,v). To generate the visibility output for the query point corresponding to the given pixel, the system can compare d(u,v) to the depth of the 3D query point, x′, i.e., a distance between the 3D query point and the sensor. In particular, if d(u, v) > x′, the query point is visible or in free space. If d(u, v) ≤ x′, the query point is not visible or occluded, where u and v are coordinates of the corresponding pixel in the image coordinate system, d() is the depth, and x′ represents the distance from the sensor to the query point in the sensor coordinate system.

When the neural network also generates uncertainty estimates, the system can also use the uncertainty estimate for the corresponding pixel to generate the visibility output. In particular, the system136calculate a modified estimated depth using the estimated depth and the estimated uncertainty. The modified estimated depth can be negatively correlated with the uncertainty so that the depth estimation can be more robust and reliable. That is, the modified estimated depth can be smaller when the uncertainty is larger, and vice versa. The system can compare the modified estimated depth with the distance between the query point and the sensor to determine the visibility output. A similar binary indicator can be used for the visibility output.

The visibility outputs can be used by the planning subsystem of the on-board system to control the agent, i.e., to plan the future motion of the vehicle based on the visibility predictions in the environment. As another example, the visibility output may be used in simulation and assist in controlling the simulated vehicle, in testing the realism of certain situations encountered in the simulation, and in ensuring that the simulation includes surprising interactions that are likely to be encountered in the real-world. When the planning subsystem receives the visibility prediction outputs, the planning sub system can use the prediction output to generate planning decisions that plan a robust, safe, and comfortable future trajectory of the autonomous vehicle, i.e., to generate a planned vehicle path.