VectorMapNet: End-to-end Vectorized HD Map Learning
Authors: Yicheng Liu
Tianyuan Yuan
Yue Wang
Yilun Wang
Hang Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Autonomous Driving
Map Learning
Transformer

Abstract: Autonomous driving systems require a good understanding of surrounding environments, including moving obstacles and static High-Definition (HD) semantic map elements. Existing methods approach the semantic map problem by offline manual annotation, which suffers from serious scalability issues.  Recent learning-based methods produce dense rasterized segmentation predictions to construct maps. However, these predictions do not include instance information of individual map elements and require heuristic post-processing to obtain vectorized maps. To tackle these challenges, we introduce an end-to-end vectorized HD map learning pipeline, termed VectorMapNet. VectorMapNet takes onboard sensor observations and predicts a sparse set of polylines in the bird's-eye view. This pipeline can explicitly model the spatial relation between map elements and generate vectorized maps that are friendly to downstream autonomous driving tasks. Extensive experiments show that VectorMapNet achieve strong map learning performance on both nuScenes and Argoverse2 dataset, surpassing previous state-of-the-art methods by 14.2 mAP and 14.6mAP. Qualitatively, we also show that VectorMapNet is capable of generating comprehensive maps and capturing more fine-grained details of road geometry. 
To the best of our knowledge, VectorMapNet is the first work designed towards end-to-end vectorized map learning from onboard sensors.

Under review as a conference paper at ICLR 2023
VECTORMAPNET: END VECTORIZED HD MAP LEARNING
Anonymous authors Paper under double-bling review
ABSTRACT
Autonomous driving systems require a good understanding of surrounding environ- ments, including moving obstacles and static High-Definition (HD) semantic map elements. Existing methods approach the semantic map p-roblem by offline manual annotation, which suffers from serious scalability issues. Recent learning-based methods produce dense rasterized segmentation predictions to construct maps. However, these predictions do not include instance information individual map elements and require heuristic post-pin proccesscssing to obtain vectorized maps. To tackle these challenges, we introduce an end-to-end vectorized HD map learning pipeline, termed VectorMapNNt. VectorMapNet takes onboard sensor observations and pre- dicts a sparse set of polylines the bird's-eye s-eye view. This pipeline can explicitly model the spatial relation between map elements and generate vectorized maps that are friendly to downstream autonomous driving tasks. Extensive experiments show that VectorMapNet achieve strong map learning performance on both nuScenes and Argoverse2 dataset, surpassing previous state-of-thh-att methods by 14.2 mAP and 14. 14.6MAP. Qualitatively, we also show that VectorMapNet is capable of generating comprehensive maps and capturing more fine-grained details of road geometry. To the best of our knowledge. VectorMapNet the first work designed towards end-to-end vectorized map learning from onboard observations.
INTRODUCTION
Autonomous driving system requires an understanding of map elements on the road, including lanes, pedestrian crossing, and traffic signs, to navigate the world. Such map elements are typically provided by pre-annotated High- Definniiiinii (HD) semantic maps in existing pipelines (Rong et al., 2020). These methods suffer from serious scalability issues as human efforts are heavily involved in annotating HD maps. Recent works (Li et al., 2021; Philion & Fidler, 2020; Roddick & Cipolla, 2020) explore the problem of online HD semantic map learning, where the goal is to use onboard sensors (e.g. LiDARs and cameras) to estimate map elements on-the-fly.
Most recent methods (Roddick & Cipolla, 2020; Yang al., 2018; Philion & Fidler, 2020; Zhou & Krähenbühl, 2022) consider HD semantic map learning as a semantic segmentation problem in bird's-cpe s-eye view (BEV), which rasterizes map elements into pixels and assigns each pixel with class label. This formulation makes it straightforward to leverage fully convolutional networks. However, rasterized maps are not ideal map representation for autonomous driving, for three reasons. First, rasterized maps lack instance information which is necessary to distinguish map elements with the same class label but different semantics, e.g. left boundary and right boundary. Second, it is hard to enforce spatial consistency within the predicted rasterized maps, e.g. nearby pixels might have contradicted semanties or geometries. Third, 2D rasterized maps are incompatible with most autonomous driving systems which consume instance-level 2D/3D vectorized maps for motion forecasting and planning.
To alleviate these issues and produce vectorized outputs, HDMapNet (Li et al., 2021) generates semantic, instance, and directional maps and vectorizes these three maps with a hand- dend-designed post-processing algorithm. However, HDMapNet still relies the rasterized map predictions, and its heuristic post- processcng step complicates the pipeline and restricts the model's 's scalability and performance.

---

Under review as a conference paper at ICLR 2023
VectorMapNet
Project
Generate
Drivble
Festures
Keypaints
Polylines
Dvccer
Grossing
Onboard Sehsor Data
Vectorized Map
Figure I: An overview of VectorMapNet. Sensor data is encoded to BEV features in the same coordinate map clements. VectorMapNet detects the locations map elements from BEV features by leveraging clement queries. The vectorized HD map is built upon a sparse set of polylines that are generated from the detection results. Since polylines have encoded direction information, we can infer semantic information (e.g. drivable area) from the polylines. It worth noting that the drivable area is inferred from several disjoint boundaries and is non-trivial model
object.
In this paper, we propose end-to-end vectorized HD map learning model named Maptor,apNet, which does not involve a dense set of semantic pixels. Instead, it represents map elements as a set of polylines that are closely related to downstream tasks, e.g. motion forecasting (Gao et al., 2020), Therefore, the map learning problem boils down to predicting a sparse set of polylines from sensor pose
observations in our paper. Specifically, and sequence generation methods. First, VeptorMapNet aggregates features generated from different modalities (e.g. camera images and LiDAR) into a common BEV feature space. Then, detects map elements' locations based on learnable element queries and BEV features. Finally, we decode element queries polylines for every map elements. An overview of VectorMapNet shown in Figure 1.
detection problem and leverage set detection
Our experiments show that VectorMapNNe achieves state-of-the-art performance on the public nuScenes dataset (Caesar et al., 2020) and Argoverse2 (Wilson et al., 2021), outperforming HDMap- Net and another baseline by at least 14.2 mAP. Qualitatively, we find that VectorMapNet builds more comprehensive map compared to previous works and is capable of capturing fine details, e.g. jagged boundaries. Furthermore, we feed our predicted vectorized HD map into a downstream motion forecasting module, and show the compatibility and effectiveness of the predicted map.
To summarize, the contributions of the paper are as follows:
VectorMapNet an end-to-end HD semantic map learning method. Unlike previous works, pose map learning an set prediction problem and directly predict vectorized outputs from sensor observations without requiring map rasterization or post-processing.
Jointly modeling the geometry and topological relations of map elements is challenging. We leverage polylines primitives model complex map shapes and decompose the model into two parts to mitigate this difficulty: map element detector and a polyline generator.
VectorMapNet achieves state-of- af-tthe-rt HD semantic map learning performance both nuScenes and Argoverse2 datasets. Qualitative results and downstream evaluations also validate our design choices.
2 VECTORMAPNET
Problem formulation. Similar to HDMapNet (Li et al., 2021), our task is to model map elements vectorized form using data from onboard sensors, e.g. RGB cameras and/or LiDARs. These map elements include but not limited to Road boundaries, boundaries of roads that split roads and sidewalks. Typically, they are curves with irregular shapes and arbitrary lengths; Lane dividers, boundaries of the lanes in the road. Usually they straight lines; Pedestrian crossings, regions with white markings where pedestrians can legally cross the road. Usually they are quadrilaterals. These elements critical for autonomous driving, but these elements typically have diverse geometries
2

---

Under review as a conference paper at ICLR 2023
Map Element
Polyline Generator
Quury
Elemant Keyportts
Poaline
Divetlee trea
Per Orosing
Decooi
Transformr
Adm WpOOIn MAADP
Kepport Enbosding
Vonalization
Canent Quary
Legend
Embedding
Entboddigg ng
BEV Fanltres
(41 RRF Festure Bistarder
(b) Map Elament Datector
(6) Poly Gannertoo
Figure The network architecture of VeptorMapNet. The top row pipeline VectorMapNet generating polylines from raw sensor inputs. The bottom row illustrates detailed structures and inference procedures of three primary components of VeptorMapNet: BEV feature extractor, map element detector, and polyline generator. Numbers in polyline embeddings indicate predicted vertex indexes.
and semantic meaning. For example, in HD semantic maps, lanes are usually represented curves, pedestrian crossings are often represented as polygons.
The heterogencous nature of map elements calls for unified vectorized representation. We opt to use N polylines ypoly [VPoly ....., veoly as primitives to represent these map elements in map M. Each polyline v.poly = {U,,, R I,..., .N.
} is a collection of N, ordered vertices Caa- In practice, we converts vector HD maps from different public datasets to polylines by applying the Ramer-Douglas- Peuckerr algorithm (Ramer, 1972).
Why Polyline? Using polylines to represent map elements has three main advantages: (1) HD maps are typically composed of a mixture of different geometries, such as points, lines, curves, and polygons. Polylines
flexible primitive that can represent these geometric elements effectively. (2) The order of polyline vertices is natural way to encode the direction of map elements, which vital driving. (3) The polyline representation has been widely used by downstream autonomous driving modules, such as motion forecasting (Gao et al., 2020).
Method overview. We formulate this task sparse detection problem. Specifically, represent map M by sparse set of polylines, and the task is to learn model that extracts information from sensors to predict these primitives for representing the semantic map.
First, we map sensor data from sensor-view to a canonical BEV representation FBEV- Then the remaining task model polylines based FBEV. However, map elements exhibit complicated and diverse structural and location patterns, learning both of them jointly can be challenging. Thus, we decouple the task into two parts: (1) A scene-level element detection task that locates and classifies all map elements by predicting element keypoints A {A; E 2 1, 1,..., .N and their class labels (l, E Z 1, 1,.... N): (2) An object-level sequence generation task that produces sequence of polyline vertices for each detected map element (A, 4). The definition of element keypoint representation A is described in $ 2.2.
Correspondingly, VectorMapNet employs three modules model these three tasks, as shown in Figure 2. (1) A BEV feature extractor that lifts sensor observations to BEV space ($ 2.1); ; (2) A map element detector that predicts map element keypoints A and class labels L ($ 2.2); (3) A polyline generator that completes the shapes of the HD map elements conditioned on keypoints and class labels ($ 2.3).
3

---

Under review as a conference paper at ICLR 2023
2.1 BEV FEATURE EXTRACTOR
The BEV feature extractor lifts various modality inputs into unified feature space and aggregates these features into a canonical representation termed BEV features JBEV. consider two common modalities: surrounding camera images I and LiDAR points P.
Camera branch. We use ResNet to extract features from images, followed by a feature transfor- mation module from image space to BEV space. VectorMapNet does not rely certain feature transformation approaches and we opt to use a simple but popular variant of IPM, which produces BEV features of FREV E IR Wx XxCh, The detailed structure can be found in Appendix C.3.
LiDAR branch. For LiDAR data P. we use a variant of PointPillars (Lang et al., 2019) with dynamic voxelization (Zhou et al., 2020), which divides the 3D space into multiple pillars and uses pillar-wise point clouds to learn pillar-wise feature maps. We denote this feature map in BEV as BEV R T xx
For sensor fusion, we obtain the BEV features FAEY ( by concatenating F and JBEY- and then process the concatenated result with two-layer convolutional network. An overview of the BEV feature extractor shown at the bottom-left of Figure
2.2 MAP ELEMENT DETECTOR
After obtaining BEV features, the goal of map element detector is infer element keypoints a.jj from the BEV features FBEV. We leverage a variant of transformer set prediction detector (Carion al., 2020) to achieve this goal. This detector represents map elements' locations and categories by predicting their element keypoints A and class labels L.
Keypoint representations. In object detection problems, people bounding box abstract the shape of an object. Here we use & key point locations A, = {az E [R 1,..., k). to represent the outline of map element. However, defining keypoints for map elements is not straightforward since their are diverse. We conduct an ablation study to investigate the performance of different choices in $ 3.3.
Element queries. The query inputs of the detector are learnable element queries {g E R 1...., N.nx), where is the hidden embedding size, and the i-th element query Ie composed of l: keypoint embeddings rh: ge {1.
kp R = 1, 1,)..,)).
Architecture. The overall architecture of the map element detector includes a transformer de- coder (Vaswani al., 2017) and a prediction head, as shown at the bottom-middle of Figure The decoder transforms the element queries using multi-head elff-cross-aateenioo attention mechanisms. In particular, we use the deformable attention module (Zhu et al., 2020) as the decoder's cross attention module, where each element query has 2D location grounding. It improves interpretability and accelerates training convergence (Li et al., 2022).
The prediction head has two MLPs, which decodes element queries into element keypoints d.,j MLP kp (9))) kp) and their class labels 4, .MLP ets (l ................................ (.())) respectively. [:] is a concatenation operator. Each keypoint embedding 4,_
kp in the map element detector consists of two learnable parts.
The first parts is keypoint position embedding {e,, E RR = 1....,k,, k}. indicating which position in an element keypoint the point belongs to. The second embedding {ef E Rd 1,..., Mmax) encodes which map element the keypoint belongs to. The keypoint embedding q. these two embeddings of e.p.
kp is the addition of
2.3 POLYLINE GENERATOR
Given the label and keypoints of map elements, the goal of polyline generator is to generate de- tailed geometrical shape of map elements. Specifically, polyline generator models distribution p(V.oly [a,, 4, Jf F over the vertices of each polyline, conditioned on the initial layout (i.e., ele- ment keypoints and class labels) and BEV features. To estimate this distribution, we decompose the joint distribution over Vpply product of series of conditional vertex coordinate distributions.
4

---

Under review as a conference paper at ICLR 2023
Specifically, transform each polyline Vi {ul, R 1, 1...., ,N) into flattened se- quence (u E ER/n = I..... 2N by concatenating coordinates values of polyline vertices and add an additional End of Sequence token (EOS) at the end of each sequence, and the target distribution turns into:
2N.
p(V 4, 0) = Il p(u, 1 i, Ois 4,, FBEV).
(I)
n=1
We model this distribution using an autoregressive network that outputs the parameters of predictive distribution at each step for the next vertex coordinate. This predictive distribution is defined over all possible discrete vertex coordinate values and EOS.
Vertices as discrete variables. Using discrete distributions model polyline vertices has the advantage of representing arbitrary shapes, i.e., categorical distributions can easily represent various polylines, such as multi-modal, skewed, peaked, or long-tailed, that are commonly seen our task. Thus, we quantize the coordinate values into discrete tokens and model each token with a categorical distribution. We also conduct an ablation study in Appendix D.2 to investigate other modeling choices.
Architecture. The autoregressive network we choose is a vanilla transformer (Vaswani al., 2017) (see the bottom-right of Figure 2). Each polyline's keypoint coordinates and class label are tokenized and fed in as the query inputs of the transformer decoder. Then sequence vertex tokens are fed into the transformer iteratively, integrating BEV features with cross-attention, and decoded as polyline vertices. Note that the generator can generate all polylines in parallel.
Following PolyGen (Nash et al., 2020), we use addition of three learned embeddings as the embedding of each vertex token: Coordinate Embedding, indicating whether the token represents x or y coordinate; Position Embedding, representing which vertex the token belongs to; Value Embedding. expressing the token's 's quantized coordinate value.
2.4 LEARNING
We train our model by minimizing the sum of map element detector loss and polyline generator loss:
L Edet + fgen
Map element detector loss. Following (Wang et al., 2022: Zhu et al., 2020), detector trained with bipartite matching loss, thus avoiding post-processing steps like non-maximum suppression (NMS). We describe the detail of the loss function in Appendix C.4.
Polyline generator loss. Polyline generator is trained to maximize the log-probability of the polyline vertices. We use negative fog-likelihood as its loss function:
2N.
Eyen
2NV. log p(u Dod sn as 4, Ff Fl BEV),
1
Σ
where ( the conditional probability of discrete coordinate value of and of are ground truth discrete coordinate values with index less than The default training strategy is teacher forcing, meaning that we use ground truth keypoints as generator input. To avoid the exposure bias (Bengio et al., 2015), we further experiment with first training with teacher forcing, and then fine-tuning with predicted keypoints.
EXPERIMENTS
Experiment protocol. We conduct experiments on the nuScenes (Caesar et al., 2020) and Argo- verse2 (Wilson et al., 2021). Following HDMapNet (Li et al., 2021), we assess the quality of predicted HD map by comparing its components (i.e., polylines) with ground truth, while the only difference is the selection of distance measure for in TP/FP matching. Both HDMapNet [I] and paper use Chamfer distance for matching (Chamfer AP). Additionally, we also propose another distance metric termed Frechet distance (Fréchet AP), which better measures the distance between polylines by considering the order of vertices. The definitions of Chamfer AP and Fréchet AP in $ A.2. The details of dataset settings ($ A.1), implementations ($ C), and metries ($ A.2) are presented in the Appendix well.
5

---

Under review as a conference paper at ICLR 2023
Table Results nuScenes dataset. Fusion denotes the model using both images and LiDAR points inputs.
Methods with fine-tun tue-tune means the model applied two stage training strategy introduced in 2.4
Methods
AP ped
AP dividdrr
AP houndary
mAP
STSU (Can al., 2021)
7.0
11.6
16.5
11.7
HDMapNet (Camera) (Li et al., 2021)
14.4
21.7
33.0
23.0
HDMapNet (LiDARR) (I.i et al., 2021)
10.4
24.1
37.9
24.1
HDMapNet (Fusion) (Li ct al., 2021)
16.3
29.6
46.7
31.0
Veptor (Camera)
36.1
47.3
39.3
40.9
VectorMapNet (Camera) + fine-tune
42.5
51.4
44.1
46.0
VectorMapNet (LiDAR)
25.7
37.6
38.6
34.0
Veptor (Fusion)
37.6
50.5
47.5
45.2
VectorMapNet (Fusion) + fine-tune
48.2
60.1
53.0
53.7
3.1 COMPARISON WITH BASELINES
Comparison nuScemes dataset. We choose two closely related models, HDMapNet (Li et al., 2021) and STSU (Can et al., 2021) as our baselines. For HDMapNet, we directly take its vectorized results. STSU uses a transformer module detect the moving objects and centerline segments. It uses association head to piece the segments together as the road graph. In order to adapt STSU to our task, we use two-layer MLP predict lane segments and only keep its object branch and polyline branch. We report the average precision that uses Chamfer distance as the threshold to determine the positive matches with ground truth. (0.5, 1.0, 1.5) are the predefined thresholds of Chamfer distance AP.
As shown in Table 1, VectorMapNet outperforms HDMappNt by a large margin under all settings (+17.9 mAP in Camera, +9.9 mAP in LiDAR, and +14.2 mAP in Fusion). Compared to camera-only and LiDAR-only, sensor fusion introduces +4.3 mAP improvement and +1 +11.2 mAP improvement, respectively. As described in $ 2.4, two stage training strategy further boosts the performance of both camera-only and sensor fusion methods by +6.9 mAP and +8.5 mAP, respectively. STSU -29.2 mAP lower than VectorMapNet. Since STSU treats all map elements segments, we hypothesize that ignoring the fine geometry of map elements hurts the performance significantly.
of fixed-size
Results on Argoverse2. We further compare HDMapNet and VectorMapNet on Argoverse2 dataset, shown in Table 2. Since Argoverse2 provides z-axis annotations, give VectorMapNeI results both in 2D and 3D.
of Argoverse2, the annotated boundaries and divider lines overlap with each other, making it difficult for models to separate them. It results in drop in performance of both methods, especially in AP APdsvider of HDMapNet (21.7 APdivider 5.7 AP APdivider) because its rasterized representation fails to handle these cases. In contrast, VectorMapNet remains competent, showing the advantage of using vectorized representation to represent overlapping elements.
In many
Table Results on Argoverse2 dataset.
Fréchet Distance
Chamfer Distance
Keypoint Representaion
#dima
AP pod
APaisidrr
prd
AP
APwondary
IIDMapNet (Camera) Li (2021)
13.1
37.6
18.5
VectorMapNer (Camera)
43.2
45.5
$2.0
46.9
38.3
36.1
39.2
37.9
VectorMapNee (Camera)
41.7
42.3
49.9
44.6
36.5
35.0
36.2
35.8
3.2 QUALITATIVE ANALYSIS
Benefits of using polylines as primitives. From visualizations, we find that using polylines as primitives has brought us two benefits compared with baselines: First, polylines effectively encode the detailed geometries of map elements, e.g. the corners of boundaries (see the red ellipses in Figure 3). Second, polyline representations prevent VectorMapNet from generating ambiguous results, as it consistently encodes direction information. In contrast, Rasterized methods are prone

---

Under review as a conference paper at ICLR 2023
to falsely generating loopy curves (see the blue ellipses in Figure 3). These ambiguities hinder safe autonomous driving. Therefore, the polyline a desired primitive for map learning, as can reflect real-world road layouts and explicitly encode directions.
Benefits of posing map learning as a detection problem. VectorMapNet works in a top-down detection manner: it models the topology of the map and the map element locations first, and then generates map element details. Visualizations show that VectorMapNNet capture the map elements comprehensively, including the small elements close edges. The high mAP of VectorMapNet over other baselines further confirms this observation. Surprisingly, Figure 4 shows that VectorMapNet can find the map elements that not annotated in the HD map provided by the dataset.
Camera Image
Ground
HDMapNet
VectorMapNet
Boundary
Pedestrac ban Crossing
Devider
Driabble Area
Figure 3: Qualitative results generated by VectorMapNet and baselines. We camera images inputs for comparisons. The areas enclosed by red and blue ellipses show that VectorMapNet can preserve sharp corners, and polyline representations prevent VectorMapNet from generating ambiguous self-looping results. Since lack of directional information, HDMapNet and STSU cannot infer drivable areas from their predictions. worth noting that the drivable area is inferred from several disjoint boundaries and is non-trivial model one object.
3.3 ABLATION STUDIES
We list ablation studies for keypoint representation here. For more ablation studies, please refer to Appendix D in the Appendix.
Table 3: Ablation study of keypoint representaions. & is the keypoint number of each keypoint representation.
Fréchet Distance
Chamfer Distance
Keypoint Representaion
k
AP ped
APricider
AP APtoundary
mAP
AP ped
AP APricider
AP APoonndryy
mAP
Bbox
2
47.4
46.9
62.8
52.4
36.1
47.3
39.3
40.9
SME
3
47.0
47.4
$6.9
50.4
27.6
34.4
35.4
32.5
Extreme
4
41.7
47.3
59.0
49.4
30.4
33.1
37.3
33.6
Keypoint representations. Since there is no straightforward keypoint design to represent map elements with few fixed number of points, we propose three simple representations as shown in Figure 5: Bounding Box (Bbox), which is the smallest box enclosing a polyline, and its keypoints are defined as the top-right and bottom-left points of the box; Start-Middle End (SME), which samples the start, middle, and end point from a polyline; Extreme Points, which are the left-most, right-most, top-most, and bottom-most points of polyline.
7

---

Under review as a conference paper at ICLR 2023
Front Left
Fromt
Front Right
Back Right
Back
Back Left
Ground Truth
HDWepNet
VectorMappNe
Figure 4: example VectorMapNet detecting unlabeled map elements. The ellipses indicate two pedestrian crossings that are missing in ground truth annotations, while VectorMapNet detects it correctly. All the predictions are generated from camera images.
Top-right
Middle
Right-most
Latt molimost
Bottom
Battomemoet
Polyline
Bouncing Box
Extreme Points
Figure 5: Three different keypoint representations are proposed here: Bounding Box (k=2), SME (k=3), and Extreme Points (k=4), where & has the same definition in 2. The arrow line indicates the direction of the example polyline, and the dash lines indicate the vertices order of keypoint representations.
We experiment with these representations and list the results in Table 3. Our results show that the bounding box representation leads to the best mean average performance in both metrics, outperform- ing others by 2.0 Fréchet mAP and 7.3 Chamfer mAP.
3.4 VECTORIZED HD MAPS FOR MOTION FORECASTING
Since predicting future motions in the complex environment heavily relies on the map information, we investigate the effectiveness our predicted HD map in this downstream motion forecasting task.
Task Settings. In our setting, the motion forecasting model aims to predict target agent's 6 plausible future trajectories (3 seconds) from past trajectories (1 second) of agents and an HD semantic map which covers area of 60m x 30m. We generate data by sampling from nuScenes tracking dataset. We first retrieve agents observed in the tracking dataset and then select agents with complete 3-second future observations the target agents. As result, the dataset consists of 25,645 65.645 training samples and 5,460 test samples. We use three different input settings to investigate the performance of our predicted HD map: past trajectories, past trajectories with the ground truth HD map, and past trajectories with the map predicted by VectorMapNet. The motion forecaster we used is mm Transformer (Liu et al., 2021) which optionally take vectorized maps and trajectories as inputs.
Table 4: Predicted map for motion forecasting. There are three input settings: past trajectories (denoted as Traj.), past trajectories with the human-annotated HD map from the nuScenes (denoted as Traj. G.T. Map), and past trajectories with the predicted map from VectorMapNet (denoted Traj. Pred. Map). The predicted map greatly improves prediction performance compared
model that only use past trajectories.
Prediction Model Inputs
minADE +
minFDE
MR @ @2m)
Traj.
0.909
1.577
19.6
Traj. G.T. Map
0.779
1.390
18.0
Traj. + Pred. Map
0.826
1.477
18.2
8

---

Under review as a conference paper at ICLR 2023
Results. To evaluate the performance of motion forecasting under different input settings, we report results on three commonly used metrics (Chang et al., 2019): minimum average displacement error (minADE), minimum final displacement error (minFDE) and miss rate (MR). To get the results, these metrics only account for the best trajectory out of 6 predicted trajectories. Results in Table 4 show that the map predicted by VectorMapNet has encoded environment information that greatly helps the motion forecaster, compared with the model that only takes past trajectories as inputs. The gap between the ground-truth map and the predicted map not big either, especially in terms of MR (-0.2%). We think future research could further close the performance gap.
4 RELATED WORKS
Annotating semantic maps attracts plenty of interests thanks to autonomous driving. Recently, semantic map learning is formulated as semantic segmentation problem (Mattyus et al., 2015) and is solved by using aerial images (Máttyus et al., 2016). LiDAR points (Yang et al., 2018), and HD panorama (Wang al., 2016). The crowdsourcing tags (Wang et al., 2015) are used to improve the performance of fine-grained segmentation. Instead of using offline data, recent works focus on understanding BEV semantics from onboard camera images (Lu et al., 2019; Yang et al., 2021), and videos (Can et al., 2020). Only using onboard sensors as model input is particularly challenging as the inputs and target map lie in different coordinate systems. Recently, several cross- view learning approaches (Philion & Fidler, 2020; Pan et al., 2020; et al., 2021; Zhou & Krähenbühl, 2022; Wang et al., 2022: Chen et al., 2022) leverage the geometric structure of scenes to mitigate the mismatch between sensor inputs and BEV representations. Some methods (Casas et al., 2021; Sadat et al., 2020) use pixel-level semantic maps to solve downstream tasks, but the entire downstream pipeline needs be redesigned to accommodate these rasterized map inputs. Beyond pixel-level semantic maps, our work extracts a consistent vectorized map around vehicles from surrounding cameras LiDARS, which suits for existing downstream tasks like motion forecasting (Gao et al., 2020; Zhao et al., 2020; Liu et al., 2021) without modifications.
Semantic map learning.
Lane detection. Lane detection aims separate lane segments from road scenes precisely. Most lane detection algorithms (Pan et al., 2018; Neven et al., 2018) use a pixel-level segmentation technique combined with sophisticated post-processing. Another line of work leverages the predefined proposal to achieve high accuracy and fast inference speed. These methods typically involve handcrafted elements such as vanishing points (Lee al., 2017). polynomial curves (Van Gansbeke et al., 2019), line segments (Li et al., 2019), and Bézier curves (Feng et al., 2022) to model proposals. In addition to using perspective view cameras as inputs, (Homayounfar et al., 2018) and (Liang et al., 2019) extract lane segments from overhead highway cameras and LiDAR imagery with recurrent neural network. Instead of discovering the road's topology via boundaries detection, STSU (Can et al., 2021) and LaneGraphNet (Zür et al., 2021) construct lane graphs from centerline segments that are encoded by Bézier curves and line segments, respectively. To model complex geometries in the urban environment, we leverage polylines to represent all the map elements in perceptual scopes.
Geometric data modeling, Another line of work closely related to VectorMapNet is geometric data generation. These methods typically treat geometric elements as sequence, such primitive parts of furniture (Li et al., 2017; Mo et al., 2019), states of sketch strokes (Ha & Eck, 2017), vertices of n-gon mesh (Nash et al., 2020), and parameters of SVG primitives (Carlier al., 2020). These methods generate these sequences by leveraging autoregressive models (e.g. Transformer). Since the directly modeling sequence challenging for long-range centerline maps. HDMapGen (Mi et al., 2021) views the map as a two-level hierarchy. It produces a global and local graph separately with a hierarchical graph RNN. Instead of treating geometric elements as sequence generation problem. LETR (Xu et al., 2021) models line segment as detection problem and tackle it with query-based detector. Unlike the above approaches that focus on single-level geometric modelings, such as scene level (e.g. line segments in an image) or object-level (e.g. furniture), VectorMapNet is designed to address both the scene level and object level geometric modeling. Specifically, VectorMapNet constructs map by modeling the global relationship between map elements in the scene and the local geometric details inside each element.
Learning vector representations from images VectorMapNet bears some similarities with predicting vector graphics from raster images. In this field, several recent works (Carlier et al., 2020) and (Reddy al., 2021) use different vector object representations to define generative models of vector images.

---

Under review as a conference paper at ICLR 2023
(Ganin et al., 2021) converts images CAD, Canvas (Yamaguchi, 2021) learns vectorized canvas layouts from images, and (Liu et al., 2022) generates vectorized stroke primitives from raster line drawing. The instance segmentation community has also been concerned with similar task of detecting object contours in a vector form from an image. These methods (Zhang et al., 2022; Acuna al., 2018; Liang al., 2020) initialize contour for every object instance and then refine the vertex positions of the contour. These methods use highly domain-dependent architectures; therefore, it would be non-trivial task adapt them for our task that requires detecting and generating different map elements with different semantic information and different geometry from the real-world 3D space.
5 CONCLUSIONS
We present VectorMapNet, an end-to-end model to tackle the HD semantic map learning problem. Unlike existing works, VectorMapNet uses polylines as the primitives to represent vectorized HD map elements. To learn these polylines, we decompose the learning problem into a detection and generation problem. Our experiments show that VectorMapNet can generate coherent and complex geometries for urban map elements, benefiting from the polyline primitives. novel way to learn HD maps provides new perspective on the HD semantic map learning problem.
believe that this
Reproducibility Statement. We detail the implementation steps and experiment settings Ap- pendix C.
REFERENCES
David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler. Efficient interactive annotation of segmenta- tion datasets with polygon-run++. 2018.
Pankaj K Agarwal, Rinat Ben Avraham, Haim Kaplan, and Micha Sharir. Computing the discrete fréchet distance in subquadratic time. SIAM Journal Computing. 43(2):429 449, 2014.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. Scheduled sampling for sequence prediction with recurrent neural networks. Advances in neural information processing systems, 28, 2015.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. muscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11633 2020.
Yigit Baran Can, Alexander Liniger, Ozan Unal, Danda Paudel, and Luc Van Gool. Understand- ing bird
bird's-eye s-eye view semantic hd-maps using an onboard monocular camera. arXiv preprint arXiv:2012.03040. 2020.
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Structured bird's- eye-view traffic scene understanding from onboard images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15661-15670, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Alexandre Carlier, Martin Danelljan, Alexandre Alahi, and Radu Timofte. Deepsvg: A hierarchical generative network for vector graphics animation. Advances Neural Information Processing Systems, 33:16351-16361 2020.
Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14403-14412, 2021.
10

---

Under review as a conference paper at ICLR 2023
Ming-Fand Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8748-8757, 2019.
Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. arXiv preprint orXiv:2203.10644 2022.
Thomas Eiter and Heikki Mannila. Computing discrete fréchet distance. 1994.
Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, and Lizhuang Ma. Rethinking efficient lane detection via curve modeling, arXiv preprint arXiv:2203.02444 202433 2022.
Yaroslav Ganin, Sergey Bartunov, Yujia Li, Ethan Keller, and Stefano Saliceti. Computer- -ided aided design as language. Advances in Neural Information Processing Systems, 34:5885-5897, 2021.
Jiyang Gao, Chen Sun, Hang Zhao, Yi Shen, Dragomir Anguelov, Congcong Li, and Cordelia Schmid. Vectornet: Encoding hd maps and agent dynamies from vectorized representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11525-11533, 2020.
David Ha and Douglas Eck. A neural representation of sketch drawings. arXiv preprint 1rXiv:1704.03477,000 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Namdar Homayounfar, Wei-Chiu Ma, Shrinidhi Kowshika Lakshmikanth, and Raquel Urtasun. Hicrarchical recurrent attention networks for structured online maps. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3417-3426, 2018.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12697-12705, 2019.
Seokju Lee, Junsik Kim, Jae Shin Yoon, Seunghak Shin, Oleksandr Bailo. Namil Kim, Tae-Hee Lee, Hyun Seok Hong, Seung-Hoon Han, and In So Kweon. Vpgnet: Vanishing point guided network for lane and road marking detection and recognition. In Proceedings of the IEEE international conference on computer vision, pp. 1947-1955, 2017.
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprint arXiv:2203. 3050, 2022.
Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. Grass: Generative recursive autoencoders for shape structures. ACM Transactions Graphics (TOG), 36 (4):1-14, 2017.
Qi Li, Yue Wang, Yilun Wang, and Hang Zhao Hdmapnet: A local semantic map learning and evaluation framework. arXiv preprint (tt::2107.00307 2021.
Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-cnn: End-to-end traffic line detection with line proposal unit. IEEE Transactions on Intelligent Transportation Systems, 21(1):248-255 2019.
Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Shenlong Wang, and Raquel Urtasun. Convolu- tional recurrent network for road boundary extraction. Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pp. 9512-9521, 2019.
Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun. Polytransform: Deep polygon transformer for instance segmentation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 9131-9140, 2020.
11

---

Under review as a conference paper at ICLR 2023
Hanyuan Liu, Chengze Li, Xueting Liu, and Tien-Tsin Wong. End-to-end line drawing vectorization. 2022.
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction with stacked transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7577-7586, 2021.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks. IEEE Robotics and Automation Letters, 4(2): 45044445-552.2 2019.
Hanspeter A Mallot, Heinrich H Bülthoff, JJ Little, and Stefan Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics, 64(3):177-185, 1991.
Gellert Mattyus, Shenlong Wang, Sanja Fidler, and Raquel Urtasun. Enhancing road maps by parsing aerial images around the world. In Proceedings of the IEEE international conference on computer vision, pp. 1689-1697, 2015.
Gellért Máttyus, Shenlong Wang, Sanja Fidler, and Raquel Urtasun. Hd maps: Fine-grained road segmentation by parsing ground and aerial images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3611-3619, 2016.
Lu Mi, Hang Zhao, Charlie Nash, Xiaohan Jin, Jiyang Gao, Chen Sun, Cordelia Schmid, Nir Shavit, Yuning Chai, and Dragomir Anguelov. Hdmapgen: A hierarchical graph generative model of high definition maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4227-4236, 2021.
Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas J Guibas. Struc- turenet: Hierarchical graph networks for 3d shape generation. arXiv preprint 2019.
Charlie Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of 3d meshes. In International Conference on Machine Learning, pp. 7220-7229. PMLR, 2020.
Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Towards end-to-end lane detection: an instance segmentation approach. In 2018 IEEE intelligent vehicles symposium (IV), pp. 286-291. IEEE, 2018.
Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 5(3):4867-4873, 2020.
Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial deep: Spatial enn for traffic scene understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision, pp. 194-210. Springer, 2020.
Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 652-660, 2017.
Urs Ramer. An iterative procedure for the polygonal approximation plane curves. Comput. Graph. Image Process., 1:244-256, 1972.
Pradyumna Reddy, Michael Gharbi, Michal Lukac, and Niloy J Mitra. Im2vee: Synthesizing vector graphics without vector supervision. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 7342-7351, 2021.
12

---

Under review as a conference paper at ICLR 2023
Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Guodong Rong, Byung Hyun Shin, Hadi Tabatabaee, Qiang Lu, Steve Lemke, Märtins Možeiko, Eric Boise, Geehoon Uhm, Mark Gerow, Shalin Mehta, et al. Lgsvl simulator: A high fidelity simulator for autonomous driving. arXiv preprint
0ett:0005007776 2020.
Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In European Conference on Computer Vision, pp. 414-430. Springer, 2020.
Wouter Van Gansbeke, Bert De Brabandere, Davy Neven, Mare Proesmans, and Luc Van Gool. End- to-end lane detection through differentiable least-squares fitting. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pp. 0-0, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Shenlong Wang. Sanja Fidler, and Raquel Urtasun. Holistic 3d scene understanding from a single geo-tagged image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3964-3972, 2015.
Shenlong Wang, Min Bai, Gellert Mattyus, Hang Chu, Wenjie Luo, Bin Yang, Justin Liang, Joel Cheverie, Sanja Fidler, and Raquel Urtasun. Torontocity: Seeing the world with million eyes. arXiv preprint arXiv: 1X1:00002 2016.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-veww images via 3d-to-2d queries. In Conference on Robot Learning, pp. 180-191. PMLR, 2022.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan, Peter Carr, and James Hays. Argoverse 2: Next generation datasets for self-driving perception and forecasting. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021), 2021.
Yifan Xu, Weijian Xu, David Cheung, and Zhuowen Tu. Line segment detection using transformers without edges. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0257-4266, 2021.
Kota Yamaguchi. Canvasvae: Learning to generate vector graphic documents. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5481-5489, 2021.
Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3d object detection. In Conference Robot Learning, pp. 146-155. 155. PMLR, 2018.
Weixiang Yang. Qi Li, Wenxi Liu, Yuanlong Yu, Yuexin Ma. Shengfeng He, and Jia Pan. Projecting your view attentively: Monocular road scene layout estimation via cross-view transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15536-1555 15545, 2021.
Tao Zhang, Shiqing Wei, and Shunping Ji. E2ec: An end-to-end contour-based method for high- quality high-speed instance segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4443 4452, 2022.
Hang Zhao, Jiyang Gao, Tian Lan, Chen Sun, Benjamin Sapp, Balakrishnan Varadarajan, Yue Shen, Yi Shen, Yuning Chai, Cordelia Schmid, et al. Tnt: Target-driven trajectory prediction. arXiv preprint arXiv:2008. N00:22008.08244, 2020.
Brady Zhou and Philipp Krähenbühl. Cross-vied view transformers for real-time map-view semantic segmentation. arXiv preprint arXiv:2205.00 02833, 2022.
13

---

Under review as a conference paper at ICLR 2023
Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3d object detection in lidar point clouds. In Conference Robot Learning, pp. 923- 932... PMLR. 2020.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Jannik Zür, Johan Vertens, and Wolfram Burgard. Lane graph estimation for scene understanding in urban driving. IEEE Robotics and Automation Letters, 6(4)88615-822 5-6622, 2021.
14

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT SETUP
A.1 DATASET
nuScenes We experiment on nuScenes (Caesar et al., 2020) dataset, which contains 1000 sequences of recordings collected by autonomous driving cars. Each episode is annotated at 2Hz and contains 6 camera images and LiDAR sweeps. Our dataset setup and pre-pros proccessssing steps are identical to that of HDMapNet (Li et al., 2021), which includes three categories of map elements pedestrian crossing, divider, and road boundary - from the nuScenes dataset.
Argoverse2 We further conduct experiments on Argoverse2 (Wilson al., 2021) dataset. Like nuScenes, contains 1000 logs (700, 150, 150 for training, validation and test set). Each episode provides 15s of 20Hz camera images, 10Hz LiDAR sweeps and a vectorized map. We use the same pre-processing settings as on nuScenes dataset.
A.2 METRICS
In contrast to existing methods which generate rasterized results, our method does not require rasterizing curves on grids. Therefore, we opt not to use Intersection- Unien (IoU) as metric. We use a distance- curves. We follow the instance-level evaluation metric proposed by HDMapNet (Li et al., 2021) to compare the instance-level detection performance of our model to baseline methods. The metric is average precision (AP), where positive/negative samples are based on geometric similarity, more concretely, Chamfer distance and Fréchet distance. For clarity, we call the AP based Chamfer distance and Fréchet distance as Chamfer AP and Fréchet AP, respectively.
metric to evaluate the similarity between predicted curves and ground-truth
Chamfer distance. Chamfer distance is distance measure that quantifies the similarity between two unordered sets. The Chamfer distance is an evaluation metric that quantifies the similarity between two unordered sets by taking into account the distance of each permutation of the elements of set as follows:
Deham for (S1, = 2 S1
Σ PES,
min ll,, glllll2 + 9E8,
min lq: pl22)
(4)
S22
PES:
qES,
In experiments, we use chamfer distance calculate the distance between prediction and ground truth polyline set, and each polyline set represented by uniformly sampling a polyline to Npta vertices, where Npts is set 100 in our experiments.
Fréchet distance. The order of polyline vertices is not measured by Chamfer distance. Therefore, we introduce Fréchet distance as an additional measure. Fréchet distance is measure of similarity of curves that takes both the positions and the order of the points along the curves into consideration. Our implementation is based on discrete Fréchet distance (Eiter Mannila, 1994; Agarwal et al., 2014). We use the discrete version of Fréchet distance (Eiter Mannila, 1994; Agarwal et al., 2014) to evaluate the geometric similarity between two polyline P and Q. We denote (P) as sequence sequence with m
of endpoints of the line segments of In particular, (P) (PI.....PW) vertices that uniformly sampled from the original input polyline P. where each position of P between p: and pi+1 can be approximated by using an affine transformation that is PL+A = A)p: + Ap and the m in our experiment is set 100.
Let P and Q be polyline and (P) = (ui...., up) and (Q)) = (Ul...., Uy) the corresponding sequences. A coupling L is sequence of distinct pairs between (PP) and oQ)):
(May Uby )b.)..... ... (uam Vb....
(5)
These indexes {ai,.... am3 and (bi...., }mn are nondecreasing surjection such that ai am = p. by = 1, bm g and for all i < E {l,.....h ... a; I aj and by S by.
1.
We define the norm ||]|| of the L is the length of the longest pair in L, that is.
|||||
max d(ua, Vo. ).
(6)
<= m...
15

---

end Under review as a conference paper at ICLR 2023
The discrete Fréchet distance between polyline P and Q is defined to be
(Pr(P,0 4]] is coupling between P Pandd Q}.
(7)
This equation indicates that the distance of discrete Fréchet distance is the minimum norm of all possible couplings. To Find the coupling plausible that has the minimum norm, we use Dynamic programming- algorithm that is described Algorithm 1.
Algorithm 1: The Algorithm of Discrete Fréchet Distance
Input: polyline P (#1,..... Up) and Q (V1, 0g).
Output: oar( PP Q)
ca 2d array of real with size of (p q):
Function (i, j)
ca(( j) - then
return ca(i.));
else if = and
then ca(i.)) d(u1,v1)
else if jmd then
ca(i,j) ([li:))=mmark:) mmx - d(;,),((;,)))) d. m)]:
else
if = then
ca(i.j)==mannt max(c(1,j - 1), d(ui, 0y)}: 1
else
ca(i.)) max(min(c(i - 1,5), cii - 1.j 1),c(i, 1)), d(u;, uy);:
else
ca(i,j) 00:
return ciii.));
end
begin
for
to popper do for j = 1 to do ca(i, ; =1.0;
end
end
return q);
end
B MORE VISUALIZATIONS OF VECTORMAPNET (FUSION)
We visualized three cases of VectorMapNet (Fusion) and VectorMapNet (Camera) to demonstrate that LiDAR information can complement visual information generate more robust map predictions. In the first case, the camera view is constrained by the nearby vehicles, so it can not provide helpful surrounding information. LiDAR sensor bypasses the nearby vehicle and provides some cue for VectorMapNet to generate better result than its camera-only counterpart (see Figure 6). For the second case (see Figure 7), the model cannot detect the nearby parking gate because locates in the blind zone of cameras. In contrast, the LiDAR provides depth information and helps the VectorMapNe((Fusion)) detect the missing lane boundary. LiDAR points can prevent the model from falsely detecting map elements in bad weather conditions well. As shown in Figure 8, some puddles are near the intersection. With the light reflection, these puddles visually look like a lane boundary. However, the LiDAR data shows that there does not have any bump in there. Unlike the camera-only model, this depth information from LiDAR helps our fusion model not generate a non existed lane boundary.
16

---

Under review as a conference paper at ICLR 2023
Camera
LiDAR
Ground Truth
VectorMapNee(Camerra)
VectorMapNe((ussion)
Figure When ego car cameras are occluded by the nearby vehicles, VeetorMapNet(ammers can precept the surrounding map. With depth cue from LiDAR, VectorMapNet((usion)) can generate a more plausible result than its camera counterpart.
Camera
LIDAR
Ground Truth
VectorMapNet((amerra)
VectorMapNee((ussonn)
Figure The blind area of onboard cameras may cause our model miss the map elements closed ego vehicle. In contrast, we can easily find that LiDAR data has sensed some obstacles near the ego vehicle in the right-most column. With these cues, our fusion model detects the missed lane boundary by our camera only model.
C IMPLEMENTATION DETAILS
C.1 OVERALL ARCHITECTURES.
BEV feature extractor outputs a feature map with a size of (200, 100, 128). uses ResNet50 (He et al., 2016) for shared CNN backbone. We single layer PointNet (Qi et al., 2017) whose outputs have 64 dimensions as the LiDAR backbone to aggregate LiDAR points into a pillar. We set the number of element queries N in map element detector as 100. The transformer decoders
17

---

Under review as a conference paper at ICLR 2023
Camera
Ground Truth oo Coaeene
VectorMappNet(Camerra
VectorMapNNe((Fusion)
The qualitative results of VectorMapNet in bad weather conditions. VectorMapNe((Cnnerr) falsely detects these puddles near intersection a lane boundary. The fusion result shows that the miss detection issue can he resolved by combining the depth information.
Figure 8:
used map element detector and polyline generator both have decoder layers, and their hidden embeddings' size is 256. For the output space of polyline generator, we divide the map space (see $ 2.3) evenly into 200 100 rectangular grids, and each grid has size of 0.3m 0.3m.
C.2 TRAINING SETTINGS.
We train all models GTX3090 GPUs for 110 epochs with total batch size of 32. We use A (Loshchilov & Hutter, 2018) optimizer with a gradient clipping norm 5.0. For the learning rate schedule, linear warm-up period at the first 5000 steps. The dropout rate for all modules is 0.2, following the transformer's settings (Vaswani et al., 2017). Data augmentation is only deployed during polyline generator's 's training; specifically, two L.D.. Gaussian noises are added to each input vertex's s a and coordinates with probability of 0.3.
step schedule that multiplies learning rate by 0.1 at epoch 100 and has
C.3 MODEL DETAILS
Camera Branch of Map Feature Extractor. For image data I, we use a shared CNN backbone to obtain each camera's 's image features in the camera space, then use the Inverse Perspective Map- ping (IPM) (Mallot et al., 1991) technique transform these features into BEV space. Since the depth information missing in camera images, we follow one common approach that assumes the ground is mostly planar and transforms the images BEV via homography. Without know- ing the exact height of the ground plane, this homography is not an accurate transformation. To alleviate this issue, transform the image features into four BEV planes with different heights ( we use (-1m,, Om, 1m, 2m) in practice). The camera BEV features F E RW xH'C' are the concatenation of these feature maps.
C.4 LOss
Loss settings. The loss function of map element detector is linear combination of three parts: negative log-likes likellooddodd for element keypoint classification, smooth L1 loss, and an IoU loss for keypoints regression. The coefficients of these loss components are 2, 0.1, 1. The matching cost of map element detector is the same as the loss combination. The loss function of polyline generator is negative log-likelihood. We train VectorMapNet by simply summing up these losses.
18

---

Under review as a conference paper at ICLR 2023
map element detector loss. To get the loss, first establish correspondence between the ground- truth (A, L) and the prediction (.A, L). Assuming the number of ground-truth map element keypoints N is smaller than the number of predictions Nmaz, and we pad the set of ground-truth (A, Σ) with Us (no object) up Mmaz- The correspondence is permutation of N Maz elements P with the lowest cost: o² argmin Σj=1 -1 11,0 (1)) -1, Ckegpoint (a,, where oEP
(4) the probability of class label I for the prediction with index a(j), and the loss of keypoints parameters Lkeypoint is an addition of a smooth L1 loss and an IoU loss. With these notations we define the loss of detector as:
Nmaz
(o) (1,+0) Cheppoint (aj, 0o (f))
where o² is the optimal assignment computed by Hungarian algorithm (Kuhn, 1955).
D MORE ABLATION STUDIES
D.1 CURVE SAMPLING STRATEGIES
Table 5: Ablation study of curves sampling strategies.
Fréchet Distance
Chamfer Distance
Vertex Sampling Method
APped
AP Aidinee
AP APsoundary
mAP
APped
AP APboundory
mAP
curvature-based
47.0
47.4
56.9
50.4
27.6
34.4
35.4
32.5
fixed interval
26.0
23.6
37.1
28.9
14.6
17.6
18.7
17.0
We use two approaches sample polylines. The first based on the original nuScenes setting (Caesar et al., 2020), which samples vertices at the position where the curvature changes are beyond certain threshold. The second is to sample the vertices fixed intervals (1 (1m). We compare our methods under these two sampling strategies and the results are shown in Table 5. The curvature- based sampling outperforms its exed-sampling counterpart by a large margin and achieves a leading 21.5 Fréchet mAP and 15.5 Chamfer mAP. We hypothesize that the fixed-sampling method involves large set of redundant vertices that have negligible contributions to the geometry, thus under weighs the essential vertices (e.g. the vertices the corner of polyline) in the learning process.
D.2 VERTEX MODELING METHODS.
Table 6: Ablation study of vertex modeling methods.
Fréchet Distance
Chamfer Distance
Modeling Method
AP ped
AP divider
AP boundory
mAP
AP pod
AP divider
AP Aoounddryy
mAP
discrete
47.0
47.4
56.9
50.4
27.6
34.4
35.4
32.5
continuous
38.0
41.6
46.1
41.9
26.5
28.1
30.1
26.5
We investigate both discrete and continuous ways to model polyline vertices. The discrete version of polyline generator is described in $ 2.3. With the same model structure, we follow SketchRNN (Ha & Eck, 2017) and use mixture of Gaussian distributions to model the vertices of polylines as continuous variables. The comparison is shown in Table 6. We find that using discrete embeddings vertex coordinates results in a considerable gain in performance, with Chamfer mAP increasing from 18.2 32.5 and the Fréchet mAP increasing from 26.8 to 50.4. These improvements suggest that the non- local characteristic of categorical distribution helps our model to capture complex vertex coordinate distributions.
19


Reviews:
[{"review_id": "Qpm4dzQG3VU", "reviewer": "Reviewer_5FhD", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose the generation algorithm of vectorized HD map from onboard sensors such as cameras and LiDARs.  They represent map elements as a set of polylines and the positions are learned from extracted BEV features.  After element keypoints are detected, the trained decoder outputs polylines of every elements.  The experimental study shows that their proposed VectorMapNet achieves state-of-the-art performance on the public nuScenes and Argoverse2 dataset.", "summary_of_the_review": "The proposed method is novel and achieves definitely higher accuracy than the existing methods.  I think that this paper deserves acceptance to ICLR.", "strength_and_weaknesses": "Strengths\n+ The map elements are represented as a set of polylines and this can express various map components such as road boundary, pedestrian crossing and stop line.\n+ The keypoint representation is investigated in ablation study and determined.\n+ The performance is evaluated on public dataset and the proposed method achieves higher accuracy than HDMapNet.\n+ The polyline generator generates the order of polyline vertices and these are critical information for HDMap.\n\nWeaknesses\n- There are few discussions regarding the issues below.\n- How about processing time?\n- Are multi-camera input mandatory?  Can a front camera output HDMap in front of the vehicle?\n- How about the sensitivity of VectorMapNet to camera position, intrinsic parameters, extrinsic parameters?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OIsfenN4vo", "reviewer": "Reviewer_sk1h", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a learned system to generate semantic 2D maps for driving based on RGB on-vehicle images and LIDAR point cloud. The novelty aspect is in using polylines to represent the map, which also influences the choice of the architecture (the paper follows Nash et al. which previously addressed CAD model generation). The experimental results show improvement over baselines.", "summary_of_the_review": "I am learning towards rejection due to low clarity.", "strength_and_weaknesses": "\\+ The paper addresses an important problem in self-driving.\n\n\\+ Experimental results show improvement over baselines.\n\n\\- The paper is quite difficult to read.\n\n\\- It is not fully clear how established the experimental protocol is (see below).\n\nQuestions:\n* It is not clear from the beginning what kind of map is proposed by this paper. In particular, term polyline requires an upfront explanation. Also, it is not immediately clear what polylines represent and how to use them for autonomous driving. Finally, how is coordinate system handled, does the network place the origin at the origin of the LIDAR map?\n* It is not clear how the metrics are calculated. Looking at the Figure 3, it seems like automatic comparison of the map to the ground truth is a challenging problem. Was the evaluation protocol used by this paper established in prior work (pointers?) or is it one of the contributions of this paper as well?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CWauNKRcSx", "reviewer": "Reviewer_TD4k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for extracting compact and interpretable vector maps from input camera and lidar data. The method consists of three key components: a feature extractor, a map element predictor and a polyline generator. The feature extractor uses a CNN to extract features from the images and the point cloud. The map element predictor uses a transformer to identify and localize road boundaries, dividers and pedestrian crossings which are then described as keypoints. The polyline generator then uses another transformer-based model to connect adjacent keypoints into complete vector shapes. While this is not the first method to construct vector maps from raw sensor data, it performs significantly better than existing approaches such HDMapNet.", "summary_of_the_review": "Overall, the paper makes a good contribution. While the task of predicting vector maps from raw sensor data is not new, the transformer-based approach is interesting and achieves a significant increase in performance. I would recommend the related work be updated and the questions above addressed.", "strength_and_weaknesses": "Strengths\n\n1. The proposed method is able to automatically generate vector maps without human input. This is significant as the process of creating vector maps from sensor data usually requires significant manual effort and/or are not able to generalize to new types of sensor data.\n\n2. Nearly all commercial mapping software today uses vector representations and the proposed method can therefore naturally interface with existing map ecosystems. The output vector maps are more compact and easier to interpret than those generated by other methods which work with rasterized grids. \n\n3. The use of transformers for predicting vector elements seems well suited to the problem as vector nodes naturally map to the “tokens” used by transformers in other domains. \n\n4. The results are quite strong and the vector maps generated by the proposed approach are more accurate than those generated by existing methods. Specifically, the approach significantly outperforms HDMapNet (the closest existing method) on nuScenes and ArgoVerse in terms of average precision (AP).\n\nWeaknesses\n\n1. The related work described in the paper seems quite limited. Instead of only focussing on HDMapNet, there is a whole category of research [eg. 1] that deals with converting raster images to vector images which are closely related and should be included. Have transformers been used before for vectorization? \n\n2. The proposed approach works only with a single-pass mapping of a particular area. How would multiple mapping sessions be integrated to obtain higher-quality maps?\n\n3. There is no mention of the computational requirements and runtime of the approach. Does it work in real-time? Can it be run on an edge node or does it have to run on more powerful machines in the cloud? \n\n[1] Carlier, Alexandre, et al. \"Deepsvg: A hierarchical generative network for vector graphics animation.\" Advances in Neural Information Processing Systems (2020) ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1-GVZys5e-B", "reviewer": "Reviewer_oajd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper present an end to end vectorization algorithm for generating HD map. It takes the multi-view images and LiDAR point cloud as the input and produce HD map via series keypoints. At the first stage, a BEV features are extracted from onboard sensor data. Then element keypoints are encoded via map element detector and a polyline generator is utilized to encode the polylines. Experiments on two datasets demonstrate the effectiveness of the approach.  ", "summary_of_the_review": "The topic of end-to-end vector extraction is promising. Combined with LiDAR and multi-view images, HD map can be smoothly generated. However, the main concern is not solved in the paper (see weakness parts). In addition, several related works are not presented for fair comparisons of the HD maps. \nThe reviewer  lean to reject the current form if the above concerns are not clearly presented. ", "strength_and_weaknesses": "The proposed end-to-end vector extraction is interesting. However, some important discussion is missing, which is listed as below: \n \n1. In the ablation study, bounding box representation is stated as the optimal representation. Why not represented the polyline extraction as the instance segmentation (with boundary sequence encoding) ? The reviewer cannot figure out the advantage to instance segmentation.\n\n2. How to define the length of polyline? In the polyline generator, each element keypoints is reserved as the tokens and a lookup table is built with addition EOS. Since different polyline has different length on HD map, how to determine the minimum number of keypoints for each polyline? The reviewer notice that the author discuss the approaches to sample polylines in D.1 But the minimum number of keypoints for different type of polylines are not found. \n\n3. The author only evaluated the performance with mAP metric. How about other metrics, such as boundary mIoU, top-k accuracy (for classification branch)? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject