Understanding Perception and Motion Planning for Autonomous Driving (2021)
By: Betty Ld
Tags: Perception, Autonomous Cars, Computer Vision, AI, Deep Learning

Understanding Perception and Motion Planning for Autonomous Driving (2021)

Deep dive into current SOTA approaches to jointly predict the environment around the car and its behaviors Betty LD Jul 8·14 min read

Autonomous Driving as the future of mobility. Image by @ed259.

Three papers to explain the latest trends in AD

Honoring the CVPR 2021 conference Workshop On Autonomous Driving (WAD), I want to share with you three state-of-the-art approaches in Perception and Motion Planning for Autonomous Driving. I’ve selected recent papers achieving outstanding results in the current benchmarks and whose authors were selected as keynote speakers in CVPR 2021.

In order to explore the subject broadly, these three papers cover different approaches: Wayve (English startup) paper uses camera images as input with supervised learning, Toyota Research Institute for Advance Developpement (TRI-AD) uses unsupervised learning, and Waabi (Toronto startup) a supervised approach with LiDAR and HD Maps as inputs.

Comparison of papers presented in this article. Image by the author.

“ FIERY : Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras.”, Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, ‪Vijay Badrinarayanan, Roberto Cipolla and Alex Kendall, 2021. [1]

: Future Instance Prediction in Bird’s-Eye View from Surround Monocular Cameras.”, Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, ‪Vijay Badrinarayanan, Roberto Cipolla and Alex Kendall, 2021. [1] “ PackNet : 3D Packing for Self-Supervised Monocular Depth Estimation.” (CVPR 2020 oral), Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos and Adrien Gaidon. [2]

: 3D Packing for Self-Supervised Monocular Depth Estimation.” (CVPR 2020 oral), Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos and Adrien Gaidon. [2] “Perceive, Predict, and Plan: Safe Motion Planning Through Interpretable Semantic Representations.” ECCV (2020). Sadat, A., S. Casas, Mengye Ren, X. Wu, Pranaab Dhawan and R. Urtasun. [3]

Why Perception and Motion Planning together:

The goal of Perception for Autonomous Vehicles (AVs) is to extract semantic representations from multiple sensors and fuse the resulting representation into a single “bird’s eye view” (BEV) coordinate frame of the ego-car for the next downstream task: motion planning.

Example of transformation from six cameras (left images) to create a bird’s eye view of the ego-car (right image). We can see that the cars are labeled in blue and their motions are predicted by the model are drawn as lines. Image from NVIDIA paper Lift, Splat, Shoot [4].

In most recent papers the semantic prediction and the bird’s eye view (BEV) are computed jointly (Perception). Then, a temporal state enables to predict jointly the surrounding agents' behavior and the ego-car motion (Motion Planning).

End-to-end is king

In recent years, end-to-end multi-task networks have outperformed sequential training networks. Currently, the inputs are just raw sensor measurements and the outputs are steering wheel commands. In order to avoid the “black-box” effect, the creation of intermediate blocks became necessary for optimization purposes (each block may have its intermediate loss function) and interpretability of the outcomes (especially when it is a bad outcome ). We can therefore access the interpretable intermediate representations such as semantic maps, depth maps, surrounding agents' probabilistic behavior in between the intermediate layer blocks (see image below).

Learning to drive with end-to-end DL and CV. Image from Alex Kendall (Wayve) CVPR keynote presentation [5]

This article dives deep inside the two main sections representative of the current split in AD:

Section 1: models using cameras only as input: ▹ Supervised // ▹ Unsupervised

▹ Supervised // ▹ Unsupervised Section 2: models using LiDAR (laser) and HD Maps as input ▹ Supervised.

Section 1) Using Camera only

Cameras only. Image from @jstepphoto.

Supervised

We’ll take one of the latest models (CVPR 2021) “FIERY”[1], made by the R&D of a start-up called Wayve (Alex Kendall CEO). They aim at learning representations with 3D geometry and temporal reasoning from monocular cameras.

The first challenge for a team having only monocular cameras on their AV is to learn depth. This is an essential step to create the bird’s eye view reference frame where the instances are identified and the motion is planned. This task is all the more difficult since each camera initially outputs its own inference in its own coordinate of reference.

An essential step of the process is to generate a 3D image from a 2D image, so I will first explain the state-of-the-art approach to “lift” the 2D images from the camera rigs to a 3D representation of the world shared by all cameras. This method published by NVIDIA CVPR 2020 paper “Lift, Splat, Shoot: Encoding Images from Arbitrary Camera Rigs by Implicitly Unprojecting to 3D” [4] is used in FIERY as well.

Encoding 2D to 3D with “Lift-Splat-Shoot”

Paper and code available here.

Creating a third dimension on 2D images. The image is illustrating the “lift” step from the “Lift, Splat, Shoot” paper [4].

1. Lift: transforms the local 2D coordinate system to a 3D frame shared across all cameras. They generate representation at all possible (discretized) depths for each pixel. The depth probabilities act as self-attention weights.

The model creates artificially a large point cloud by associating to each pixel from the 2D image a list of discrete depths D. For each pixel p with (r,g,b,a) values, the network predicts a context vector c and a distribution over depth a. The context vector c is then multiplied by each weight a from the distribution D. The result as a matrix is the outer product of a and c. This operation enables to give more attention to a particular depth. If the depth distribution a is all 0 but one element is 1, then this network acts as a pseudolidar. This point cloud tensor for each image feeds an Efficient-Net backbone network pretrained on Image net. The EfficientNet model will output the outer product described above made of the features to be lifted c and the set of discrete depth probabilities a.

2. Splat: Extrinsic and intrinsic camera parameters are used to splat the 3D representation onto the bird’s eye view plane.

They use a pointpillar technique originally used for object detection in LiDAR point clouds. Pointpillar converts the point cloud to a pseudo-image to be able to apply 2D convolutional architecture.

The point cloud is discretized into a grid in the x-y plane, which creates a set of pillars P. Each point in the cloud is transformed into a D-dimensional (D=9) vector made where we add (Xc, Yc, Zc) the distance to the arithmetic mean of all points in the pillar and (Xp, Yp) the distance from the center of the pillar in the x-y coordinate system to the original (x,y,z, reflectance). From this sparse pillar grid, they create a dense tensor of size (D, P, N) with P the number of non-empty pillars (possibly padded with 0) and N the number of points per pillar (possibly sampled if there are too many points in one pillar). This dense tensor feeds a PointNet network to generate a (C, P, N) tensor, followed by a max operation to create a (C, P) tensor. Finally, we use P to scatter back the features to the original pillar location to create a pseudo image of size (C, H, W).

In “Lift, Splat, Shoot”, the author use sum pooling instead of max pooling on the D axis to create C x H x W tensor. They feed this tensor in a second backbone network made of ResNet blocks to convert the point clouds into the bird’s eye’s view image.

3. Shoot: They “shoot” different trajectories for each instance in the BEV, calculate the cost of them and the trajectory with the minimum cost.

For the trajectory planning, let’s see into “FIERY” rather than the Shoot part of the NVIDIA paper.