Deep Learning in Mapping for Autonomous Driving
By: Patrick Langechuan Liu
Tags: Computer Vision, Deep Learning, Autonomous Vehicles

The applications of deep learning has been explored in various components throughout the autonomous driving stack, for example, in perception, prediction, and planning. Deep learning can also be used in mapping, a critical component for higher-level autonomous driving.

Having accurate maps is essential to the success of autonomous driving for routing, localization as well as to ease perception. Maps with varying degrees of information can be obtained through subscribing to the commercially available map service. However, in areas where maps are not available, self-driving vehicles need to rely on their own map building capability to ensure the functionality and safety of autonomous driving.

Offline Mapping vs Online Mapping

In an offline mapping scenario, sensor data are aggregated in a centralized location. The data could either be satellite imagery or data collected by onboard sensors such as cameras or lidars. It could be from multiple passes of the same vehicle through the same location or crowd-sourced to a fleet of cars. A rendering of the map is built offline, and human annotators are required to annotate semantic structures on the map and review the final results. Traditional mapping service works in this offline fashion, then the annotated and curated maps are then served to vehicles on the road.

An example user interface for HD map annotation (source)

Online mapping happens onboard the vehicle and are thus cannot afford to have the human in the loop. Typical examples are SLAM systems for simultaneous localization and mapping. Recently, semantic SLAM focusing on the geometry and semantic meanings of the surface markings on the road are explored as a lightweight solution to mapping. In addition, monocular semantic online mapping (monoSOM) is one trending topic where a neural network is used to fuse temporal sequences of monocular images from multiple camera into a semantic birds-eye-view map. These topics are beyond the scope of this post and will be reviewed elsewhere.

SD Maps vs HD Maps

Depending on the input resolution, there are roughly two types of applications of deep learning in mapping. The first line of work focuses on the discovery of the topology of maps, such as road network, and do not usually contain lane level information. They only require a relatively low-resolution image with roughly meter-level accuracy. The other type of application focuses on extracting lane level information such as lane lines, road surface arrows, and other semantic markings. This requires a higher resolution image with centimeter-level accuracy. Correspondingly, these two types of maps will be loosely referred to as SD maps and HD maps for the rest of this post.

Deep Learning application on SD map (Left, DeepRoadMapper) and HD map (Right, DAGMapper)

This post focuses on the offline generation of HD maps. Note that some of the methods can be applied to online mapping as well, and a short review session is dedicated to some related works of SD mapping.

Annotator-friendly Mapping

In offline mapping, it is affordable and essential to have a human reviewer to review the results of deep learning. Thus not only does the overall accuracy matter but also the presentation of the results in the sense that the results should be easy to be modified by a human annotator.

As most motion planners can only handle lane graphs that are structured and represent the right topology, the output of the mapping algorithm is usually a structured representation (polylines for lane lines, etc). This representation luckily lends itself to efficient modification of the human annotator.