MapTR: Structured Modeling and Learning for Online Vectorized HD Map Construction
Authors: Bencheng Liao
Shaoyu Chen
Xinggang Wang
Tianheng Cheng
Qian Zhang
Wenyu Liu
Chang Huang
Venue: ICLR 2023 notable top 25% (2023)
Keywords: Autonomous Driving
Online Vectorized HD Map Construction
End-to-End

Abstract: High-definition (HD) map provides abundant and precise environmental information of the driving scene, serving as a fundamental and indispensable component for planning in autonomous driving system. We present MapTR, a structured end-to-end Transformer for efficient online vectorized HD map construction. We propose a unified permutation-equivalent modeling approach, i.e., modeling map element as a point set with a group of equivalent permutations, which accurately describes the shape of map element and stabilizes the learning process. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. MapTR achieves the best performance and efficiency with only camera input among existing vectorized map construction approaches on nuScenes dataset. In particular, MapTR-nano runs at real-time inference speed ($25.1$ FPS) on RTX 3090, $8\times$ faster than the existing state-of-the-art camera-based method while achieving $5.0$ higher mAP. Even compared with the existing state-of-the-art multi-modality method, MapTR-nano achieves $0.7$ higher mAP and $8\times$ faster inference speed, and MapTR-tiny achieves $13.5$ higher mAP and $3\times$ faster inference speed. Abundant qualitative results show that MapTR maintains stable and robust map construction quality in complex and various driving scenes. MapTR is of great application value in autonomous driving. Code and more demos are available at https://github.com/hustvl/MapTR.

Published as a conference paper at ICLR 2023
MAPTR: STRUCTURED MODELING AND LEARNING FOR ONLINE VECTORIZED HD MAP CONSTRUCTION
Bencheng Liao 1,2,3
Shaoyu Chen 23+ Xinggang Wang" 2
Tianheng Cheng 2,3
Qian Zhang Wenyu Liu Chang Huang
Institute of Artificial Intelligence, Huazhong University of Science & Technology
School of EIC, Huazhong University of Science & Technology
Horizon Robotics
{beliao, shaoyuchen, xgwang, thch, bachhh,,iwwy 1iwwy) @hust. edu, en {qian01. zhang, chang. nuang]@horizon
ABSTRACT
High-definitin (HD) map provides abundant and precise environmental informa- tion of the driving scene, serving as a fundamental and indispensable compo- nent for planning in autonomous driving system. We present MapTR, struc- tured end-to-end Transformer for efficient online vectorized HD map construc- tion. We propose unified permutation-equiaalee eling map element as a point set with a group of equivalent permutations, which accurately deseribes the shape of map element and stabilizes the learning pro- cess. We design a hierarchical query embedding scheme to flexibly encode structured map information and perform hierarchical bipartite matching for map element learning. MapTR achieves the best performance and efficiency with only camera input among existing vectorized map construction approaches on nuScenes dataset. In particular, MapTTR- runs at real-time inference speed (25.1FPS) on RTX 3090, 8x faster than the existing state-of-the-arr camera-based method while achieving 5.0 higher mAP. Even compared with the existing state- of-the-art multi-modality method, MapTR-nano achieves 0.7 higher mAP, and MapTR-tiny achieves 13.5 higher mAP and 3x faster inference speed. Abun- dant qualitative results show that MapTR maintains stable and robust map con- struction quality in complex and various driving scenes. MapTR is of great ap- plication value in autonomous driving. Code and more demos are available at attps: //gi trriitunn thub. http/hhustttaaaagg
modeling approach, i.e., mod-
INTRODUCTION
High-definition (HD) map is the high-precision map specifically designed for autonomous driving, composed of instance-level vectorized representation of map elements (pedestrian crossing, lane divider, road boundaries, etc.). HD map contains rich semantic information of road topology and traffic rules, which vital for the navigation of self self-driving vehicle.
Conventionally HD map is constructed offline with ILAM-based methods (Zhang & Singh, 2014; Shan & Englot, 2018; Shan et al., 2020), incurring complicated pipeline and high maintaining cost. Recently, online HD map construction has attracted ever-increasing interests, which constructs map around ego-vehicle at runtime with vehicle-mounted sensors, getting rid of offline human efforts.
Early works (Chen al., 2022a; Liu al., 2021a; Can al., 2021) leverage line-shape priors per- ceive open-shape lanes based on the front-view image. They are restricted to single-view perception and can not cope with other map elements with arbitrary shapes. With the development of bird's eye view (BEV) representation learning, recent works (Chen et al., 2022b; Zhou & Krähenbühl, 2022; Hu al., 2021: Li et al., 2022c) predict rasterized map by performing BEV semantic segmentation. However, the rasterized map lacks vectorized instance-level information, such as the lane structure,
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2023
Surrounding Views
Prediction
GT
Cloudy
5.00
Sunny
BACK
DROM PAE
Rainy
PRDNT 2211
Night REFCLIET
RAX RICAT
ped crossing
divider
boundary
Figure 1. MapTR maintains stable and robust vectorized HD map construction quality complex and various driving scenes.
which important for the downstream tasks (e.g., motion prediction and planning). To construct vectorized HD map. HDMapNet (Li et al., 2022a) groups pixel-wise segmentation results, which requires complicated and time-consuming post-processing. VectorMapNet (Liu al., 2022a) rep- resents each map element as a point sequence. It adopts a cascaded coarse-to- framework and utilizes rego-segressive decoder to predict points sequentially, leading to long inference time.
Current online vectorized HD map construction methods are restricted by the efficiency and not ap- plicable real-time scenarios. Recently, DETR (Carion al., 2020) employs simple and efficient encoder-decoder Transformer architecture and realizes end-to-end object detection.
natural ask a question: Can we design a DETR like paradigm for efficient end-to-end vec- torized HD map construction? We show that the answer is affirmative with our proposed Map TRansformer (MapTR).
Different from object detection in which objects can be easily geometrically abstracted as bounding box, vectorized map elements have more dynamic shapes. accurately describe map elements, we propose novel unified modeling method. We model each map element point with group of equivalent permutations. The point set determines the position the map element. And the permutation group includes all the possible organization sequences of the point set corresponding to the same geometrical shape, avoiding the ambiguity shape.
Based on the permutation-cquiialle modeling, design structured framework which takes as input images schicle-mountee cameras and outputs vectorized HD map. We streamline the on- line vectorized HD map construction parallel regression problem. Hierarchical query embed-
2

---

Published as a conference paper at ICLR 2023
dings proposed to flexibly encode instance-level and point-level information. All instances and all points of instance are simultaneously predicted with a unified Transformer structure. And the training pipeline is formulated as hierarchical set prediction task, where we perform hierarchical bipartite matching to assign instances and points in turn. And we supervise the geometrical shape in both point and edge levels with the proposed point2point loss and edge direction loss.
With all the proposed designs, we present MR, an efficient end-to-end online vectorized HD map construction method with unified modeling and architecture. MapTR achieves the best performance and efficiency among existing vectorized map construction approaches on nuScenes (Caesar et al., 2020) dataset. particular, MapTR- Rano runs real-time inference speed (25.1 FPS) on RTX 3090, 8x faster than the existing dfate-o-thee the-arte camera- based method while achieving 5.0 higher mAP. Even compared with the existing state-of-the multi-modality method, Map nano achieves 0.7 higher mAP and 8x faster inference speed, and MapTRR-tiyy achieves 13.5 higher mAP and 3x faster inference speed. As the visualization shows (Fig. 1), MapTR maintains stable and robust map construction quality in complex and various driving scenes.
Our contributions can be summarized as follows:
We propose unified commutatiio-equivallle modeling map element as a point set with a group of equivalent permutations, which accu- rately describes the shape of map element and stabilizes the learning process.
modeling approach for map elements, i.e.,
Based the novel modeling, we present MapTRR, structured end-to-end framework for efficient online vectorized HD map construction. We design hierarchical query embed- ding scheme to flexibly encode instance-level and point-level information, perform hierar- chical bipartite matching for map element learning, and supervise the geometrical shape in both point and edge levels with the proposed point2point loss and edge direction loss.
MapTR is the first real-time and SOTA vectorized HD map construction approach with stable and robust performance in complex and various driving scenes.
RELATED WORK
HD Map Construction. Recently, with the development of 2D-to-BEV methods (Ma et al., 2022), HD map construction is formulated as segmentation problem based on surround-view image data captured by vehicle-mounted cameras. Chen et al. (2022b); Zhou & Krähenbühl (2022); Hu et al. (2021); Li et al. (2022c); Philion & Fidler (2020); Liu et al. (2022b) generate rasterized map by performing BEV semantic segmentation. To build vectorized HD map, HDMapNet (Li et al., 2022a) groups pixel-wise semantic segmentation results with heuristic and time-consuming post-processing to generate instances. VectorMapNet (Liu et al., 2022a) serves as the first end-to-end framework, which adopts two-stage coarse-to toree-tto-fneeee framework and utilizes auto-regressive decoder to predict points scquentially, leading to long inference time and the ambiguity about permutation. Different from VectorMapNet, MapTR introduces novel and unified modeling for map element, solving the ambiguity and stabilizing the learning process. And MapTR builds a structured and parallel one- stage framework with much higher efficiency.
Lane Detection. Lane detection can be viewed as a sub task of HD map construction, which fo- cuses on detecting lane elements in the road scenes. Since most datasets of lane detection only provide single view annotations and focus on open-shape elements, related methods are restricted to single view. LaneATT (Tabelini al., 2021) utilizes machor-based deep lane detection model to achieve good off between accuracy and efficiency. LSTR (Liu et al., 2021a) adopts the Trans- former architecture to directly output parameters of a lane shape model. GANet (Wang et al., 2022) formulates lane detection design. Feng et al. (2022) proposes parametric Bezier curve-based method for lane detection. In- stead of detecting lane in the 2D image coordinate. Garnett et al. (2019) proposes 3D-LaneNet which performs 3D lane detection BEV. STSU (Can et al., 2021) represents lanes as a directed graph in BEV coordinates and adopts curve-based Bezier method predict lanes from monocular camera image. Persformer (Chen et al., 2022a) provides better BEV feature representation and optimizes anchor design to unify 2D and 3D lane detection simultaneously. Instead of only detecting lanes in the limited single view, MapTR can perceive various kinds of map elements of 360 horizontal FOV, with a unified modeling and learning framework.
keypoint estimation and association problem and takes bottom-up
3

---

Published as a conference paper at ICLR 2023
ped. crossing
divider
boundary
(a) Polyline
(b) Polygon
Figure 2. Typical cases for illustrating the amhiguity of map element about start point and direction. (a) Polyline: for the lane divider between two opposite lanes, defining its direction is difficult. Both endpoints of the lane divider can be regarded the start point and the point set be organized in directions. (b) Polygon: for the pedestrian crossing. each point polygon can be connected in two opposite directions (counter-clockwing and clockwise).
polygon can be regarded as the start point, and the
Contour- Instance Segmentation. Another line of work related to MapTR is contour-based 2D instance segmentation (Zhu et al., 2022; Xie et al., 2020; Xu et al., 2019; Liu et al., 2021c). These methods reformulate 2D instance segmentation object contour prediction task, and estimate the image coordinates of the contour vertices. CurveGCN (Ling et al., 2019) utilizes Graph Convolu- tion Networks to predict polygonal boundaries. Lazarow et al. (2022); Liang et al. (2020); Li et al. (2021); Peng et al. (2020) rely on intermediate representations and adopt two-stage paradigm, i.e., the first stage performs segmentation detection to generate vertices and the second stage converts vertices to polygons. These works model contours of 2D instance masks as polygons. Their model- ing methods cannot cope with line-shape map elements and are not applicable for map construction. Differently, MR is tailored for HD) map construction and models various kinds of map elements in a unified manner. Besides, MapTR does not rely on intermediate representations and has an efficient and compact pipeline.
MAPTR
3.1
PERMUTATION-EQUIVAII MODELING
MapTR aims at modeling and learning the HD map in a unified manner. HD map is a collection of vectorized static map elements, including pedestrian crossing, lane divider, road boundarie, For structured modeling, MapTR geometrically abstracts map elements as closed shape (like pedestrian crossing) and open shape (like lane divider). Through sampling points sequentially along the shape boundary, closed-shape element is discretized into polygon while open-shape element is discretized into polyline.
Preliminarily, both polygon and polyline can be represented as an ordered point set VF [vo. .,...., -1] (see Fig. 3 (Vanilla)). N, denotes the number of points. However, the per- mutation of the point set is not explicitly defined and not unique. There exist many equivalent permutations for polygon and polyline. For example, as illustrated in Fig. 2 (a), for the lane divider (polyline) between two opposite lanes, defining its direction is difficult. Both endpoints of the lane divider can be regarded as the start point and the point set can be organized two directions. In Fig. 2 (b), for the pedestrian crossing (polygon), the point set be organized in two opposite di- rections (counter-clockwis and clockwise). And circularly changing the permutation of point set has no influence on the geometrical shape of the polygon. Imposing a fixed permutation to the point set as supervision is not rational. The imposed fixed permutation contradicts with other equivalent permutations, hampering the learning process.
To bridge this gap, MapTTR models each map element with y V- V-
denotes the point set of the map element (N₂ is the number of points). r = { denotes a group of equivalent permutations of the point V. covering all the possible organization sequences.
Specifically, for polyline element (see Fig. (left)), r includes kinds of equivalent permutations:
?(i) = j mod No, (1) = (N. - -1) j mod N..
I Ppolyline = (77, -(7902
4

---

Published as a conference paper at ICLR 2023
point
point
connection Polygon
direction
Polyline
(1p H. r- (y (K.
Fixed Fixed_arder(()
r- y
Fixed_ order(())
(Pa Ps. Uz]
(V.C)
[Va Px]
Vanilla
MapTR: Vanilla
MRp
Figure 3. Illustration of vermulation-equivaaleee modeling of MapTR. Map elements are geometrically ab- stracted and discretized into polylines and polygons. MapTR models each map element with (V,1) (a point set and group equivalent permutations I'), avoiding the ambiguity and stabilizing the learning process.
For polygon element (see Fig. 3 (right)), I' includes 2 x N, kinds of equivalent permutations:
( j mod No.
>²) = (N, - 1) j mod No. >²() = ( mod No.
I' polyygon t79 ...., , 2x
70) -NN -j)- +1++) mod No.
(2)
2xN. -() = -j++ N, - 1) mod No, J 2x..-1 = -N, 1) - No 1) mod No.
By introducing the conception of equivalent permutations, MapTR models map elements in unified manner and addresses the ambiguity issue. MapTR further introduces hierarchical bipartite match- ing (see Sec. 3.2 and Sec. 3.3) for map element learning, and designs structured encoder-decoder Transformer architecture to efficiently predict map elements (see Sec. 3.4).
3.2 HIERARCHICAL MATCHING
MapTR parallelly infers fixed-size of N map elements in single pass, following the endto- end paradigm of query-based object detection and segmentation paradigm (Carion et al., 2020; Fang al., 2021a;b) N is set to be larger than the typical number of map elements in scene. Let's denote the set of N predicted map elements by y = (ύs N-1 The set of ground-truth truth (GT) map elements is padded with Ø (no object) to form a set with size N, denoted by Y = { N-1 y/s = (c,, Ve, Pt), where C₁ V, and H; respectively the target class label, point set and permutation group of GT map element ys- Ú: (p:, Vi), where 9), and V; are respectively the predicted classification score and predicted point set. To achieve structured map element modeling and learning, Map'TR introduces hierarchical bipartite matching, i.e., performing instance-level matching and point-level matching in order.
Instance-level Matching. First, we need to find an optimal instance-level label assignment be- tween predicted map elements { and GT map elements {3,). â is permutation of N elements ( IIx) with the lowest instance-level matching cost:
N-1
म
arg min Ell
Line Cine..matchhurn (Ur(it) V:).
(3)
1=0
Lins match (1) 3/) is pair-wise matching cost between prediction Un(s) and GT yt, which consid- ers both the class label of map element and the position of point set:
Lins match (D-()) 34) C (Pr(1): Ci) Epocition V=(). V.).
(4)
5

---

Published as a conference paper at ICLR 2023
Frocal Ci) is the class matching cost term, defined the Focal Loss (Lin al., 2017) be- tween predicted classification score Pr(() and target class label c. positioo (V=()) V2) the position matching cost term, which reflects the position correlation between the predicted point set Vz(1) and the GT point set V, (refer to Sec. B for more details). Hungarian algorithm is utilized to find the optimal instance-level assignment á following DETR.
Point-level Matching. After instance-level matching, each predicted map element (a( is as- signed with GT map element yi- Then for each predicted instance assigned with positive labels (c, # Ø), we perform point-level matching to find an optimal point2point assignment E I be- tween predicted point set (i))) and GT point set V,, is selected among the predefined permutation group I' and with the lowest point-level matching cost:
No-1
arg min TEr
Σ
DMonhattaa Ry()))
(5)
j=0
D Manhattan (0z, By(r) is the Manhattan distance between the j-th point of the predicted point set V and the 7())tth point of the GT point set V.
3.3 TRAINING LOSS
MapTR is trained based on the optimal instance-level and point-level assignment (/t and {% The loss function is composed of three parts, classification loss, point2point loss and edge direction loss:
L A ppp + BLdirr
where X, a and B are the weights for balancing different loss terms.
Classification Loss. With the instance- optimal matching result म, each predicted map ele- ment is assigned with a class label. The classification loss a Focal Loss term formulated as:
Cels
Σ
Frocal (Px().))
Point2point Loss. Point2point loss supervises the position of each predicted point. For each GT instance with index i, according to the point-level optimal matching result %, each predicted point (e(1). assigned with a GT point V1,4(() The point2point loss defined as the Manhattan distance computed between each assigned point pair:
N-1
N,
Cppp
Σ
1 (x
DMaubattan(Prt0.. (1
(8)
Edge Direction Loss. Point2point loss only supervises the node point of polyline and polygon, considering the edge (the connecting line between adjacent points). For accurately representing map elements, the direction of the edge is important. Thus, we further design edge direction loss supervise the geometrical shape the higher edge level. Specifically, we consider the cosine similarity of the paired predicted edge (x), and GT edge ei
N N-1
N -1
Can =
1 (c
cosine cosine_similarite www.siilarrttteeeesssss..ooooo
1=0
1=0
(9)
(+())
+(x().) 00()),--)mooN,,
01,4-
3.4 ARCHITECTURE
MapTR designs an encoder-decoder paradigm. The overall architecture is depicted in Fig. 4.

---

Published as a conference paper at ICLR 2023
pod. creesing
divider
boundary
Ou-board scuser
Map Fneeder
Map
2D
Hierarchical
Backhone
Transformer
bépariite matching
Profction
RFV
(%:]
(2t)
Vi = Car
Trunsfomett decoder layer
1-th bead
no object
9,
object
(0)
Instance-level matching
Point-level matching
Matching results
quint-ueryy query
instance query
forward
matching
prediction
ground-truh
Figure 4. The overall architecture MapTR. MapTR adopts an encoder-decoder paradigm. map encoder transforms sensor input a unified BEV representation. The map decnder adopts hierarchical query emhed- ding scheme to explicitly encode map elements and performs hicrarchical matching based on the permutation- equivalent modeling. MapTR is fully end-to-end. The pipeline is highly structured, compact and efficient.
Input Modality. MapTR takes murround-veww images of vehicle-mounter cameras input. MapTR is also compatible with other vehicle-mounted sensors (e.g. LiDAR and RADAR). Ex- tending MR multi-modality data is straightforward and trivial. And thanks to the rational permutation permotation-equuialenn equivalent modeling, even with only camera input, MapTR significantly outperforms other methods with multi-modality input.
Map Encoder. The map encoder of MapTR extracts features from images of multiple vehicle- mounted cameras and transforms the features into a unified feature representation, iee., BEV rep- resentation. Given multi-view images I = (11-- -..-... Tx). ) generate multi-view feature maps F = [i
leverage conventional backbone to FK Then 2D image features F are transformed BEV features B E Hxxx xx By default, we adopt GKT (Chen et al., 2022b) the basic 2D-to- BEV transformation module, considering its easy-to masy-to-deploy property and high efficiency. MapTR is compatible with other transformation methods and maintains stable performance, egg., CVT (Zhou & Krähenbühl, 2022), LSS (Philion & Fidler, 2020; Liu et al., 2022c; Li al., 2022b; Huang al., 2021), Deformable Attention (Li et al., 2022c; Zhu et al., 2021) and IPM (Mallot al., 1991). Ablation studies are presented in Tab. 4.
Map Decoder. We propose hierarchical query embedding scheme to explicitly encode each map N-1 and set of point-level element. Specifically, we define set of instance-level queries {q!" Ji=0
queries {q," J,N..1 shared by all instances. Each map element (with index i) corresponds to set
of hierarchical queries {ahje N. N.-1 The hierarchical query of j-th point of i-th map element is formulated as:
15=0
hic Ti5 -
ins
(10)
The map decoder contains several cascaded decoder layers which update the hierarchical queries it- eratively. each decoder layer, we adopt MHSA to make hierarchical queries exchange information with each other (both inter-instance and intra intra-nstaance. We then adopt Deformable Attention (Zhu et al., 2021) to make hierarchical queries interact with BEV features, inspired by BEVFormer (Li al., 2022c). Each query qhie predicts the 2-dimension normalized BEV coordinate (æaj. 3hj) of the reference point Pij. We then sample BEV features around the reference points and update queries.
Map elements usually with irregular shapes and require long-range context. Each map element corresponds of reference points (pg) N.-1 with flexible and dynamic distribution. The

---

Published as a conference paper at ICLR 2023
NN-11 can adapt to the arbitrary shape of map element and capture informative reference points {pij 1=ر context for map element learning.
The prediction head of MapTR is simple, consisting of classification branch and point regression branch. The classification branch predicts instance class score. The point regression branch predicts the positions of the point sets V. For each map element, outputs a dimeenion vector, which represents normalized BEV coordinates of the N,, points.
EXPERIMENTS
Dataset and Metric. We evaluate MapTR on the popular nuScenes (Caesar et al., 2020) dataset, which contains 1000 scenes of roughly 20s duration each. Key samples annotated at 2Hz. Each sample has RGB images from 6 cameras and covers 360 horizontal FOV of the ego-vehicle. Fol- lowing the previous methods (Li et al., 2022a; Liu et al., 2022a), three kinds of map elements chosen for fair evaluation pedestrian crossing, lane divider, and road boundary. The perception ranges are [-15.0m, 15.0m] for the X-axis and [-30.0m, 30.0m] for the Y-axis. And we adopt average precision (AP) to evaluate the map construction quality. Chamfer distance DCham is used determine whether the prediction and GT are matched or not. We calculate the AP, under several D Chomfer fer thresholds (T E T,T (0.5, 1.0, 1.5)), and then average across all thresholds as the final AP metric:
AP,- AP AAP T TET
(11)
Implementation Details. MapTR trained with 8 NVIDIA GeForce RTX 3090 GPUs. We adopt AdamW (Loshchilov & Hutter, 2019) optimizer and cosine annealing schedule. For ---ii- adopt ResNet50 (He al., 2016) as the backbone. We train MapTRRtiny with total batch size of 32 (containig 6 view images). All ablation studies are based on MapTR-tiny trained with 24 epochs. -nnno is designed for real-time applications. We adopt ResNet18 as the backbone. More details are provided in Appendix A.
4.1 COMPARISONS WITH STATE- -HE ART METHODS
In Tab. 1, we compare MapTR with state-of-the-ar methods. MapTR-nan nano runs at real-time in- ference speed (25.1 1 FPS) on RTX 3090, 8x faster than the existing statc-of-the art comera-based method (VectorMapNeet-C) while achieving 5.0 higher mAP. Even compared with the existing state- of-the-art multi modiittyd method, MapTR- achieves 0.7 higher mAP and 8x faster inference speed, and MapTR-tiny achieves 13.5 higher mAP and 3x faster inference speed. MapTR is also a fast converging method, which demonstrate advanced performance with 24-epoch schedule.
Method
Modality
Backbone
Epochs
AP PPd
AP Advvider
AP boundorr
mAP
FPS
HDMapNet
Effi-O PointPillars
30
14,4
21.7
33.0
23.0
0.8
HDMapNet
1.
30
10.4
24.1
37.9
24.1
1.0
HDMapNet
C& I.
Bffi-B & PointPillars
30
16.3
29.6
46.7
31.0
0.5
VectorMapNet
R50 PointPillars R50 & PointPillars
110
36.1
47.3
39.3
40.9
VectorMapNet
110
25.7
37.6
38.6
34.0
VectorMapNet
C & L
110
37.6
50.5
47.5
45.2
--nno-
R18
110
39.6
49.9
48.2
45.9
25.1
MapTR-
C
R50
24
46.3
51.5
53.1
50.3
11.2
MapTR-
R50
110
56.2
59.8
60.1
58.7
11.2
Table 1. Comparisons with state-of-the-art methods (Liu al., 2022a; et al., 2022a) on nuScenes val set. "C" and "L" respectively denotes camera and LiDAR. "Efti-BO -B" and "PointPillars" respectively correspond to Tan & Le (2019) and Lang et al. (2019). The APs of other methods are taken from the paper of VectorMapNet. The FPS of VectorMapNee-CC is provided by its authors and measured on RTX 3090. Other FPSs measured machine with RTX 3090. "." means that the corresponding results only camera input, MapTR-timg significantly outperforms multi-modality counterparts (+ 13.5 mAP). MRp nano achieves SOTA camera-based performance and runs at 25.1 FPS, realizing real-time vectorized map construction for the first time.
the
available. Even with
8

---

Published as a conference paper at ICLR 2023
4.2 ABLATION STUDY
To validate the effectiveness different designs, we conduct ablation experiments on nuScenes va set. More ablation studies are in Appendix B.
Effectiveness of Permutation--euuivalen Modeling. In Tab. 2, we provide ablation experi- ments validate the effectiveness of the proposed permutation-equiiaalnn modeling. Compared with vanilla modeling method which imposes a unique permutation to the point set, permutation- equivalent modeling solves the ambiguity of map element and brings an improvement of 5.9 mAP. For pedestrian crossing, the improvement even reaches 11.9 AP, proving the superiority in modeling polygon elements. We also visualize the learning process in Fig. show the stabilization of the proposed modeling.
Modeling method
APpel
APdivider
divider
AP bowndarry
mAP
Fixed-order V w/ ambiguity
34.4
48.1
50.7
44.4
Permuuation-eqquivalent (V,T) w/o ambiguity
46.3
51.5
53.1
50.3
Table 2. Ablations about modeling method. Vanilla modeling method imposes unique permutation the point set, leading to ambiguity. MR introduces permutation -euuivvaln--uvvlentteeee modeling avoid the ambiguity, which stabilizes the learning process and significantly improves performance ( (-5.9 mAP).
Effectiveness of Edge Direction Loss. Ablations about the weight of edge direction loss are pre- sented in Tab. 3. 3 = 0 means that we do not use edge direction loss. = 5e -3 corresponds to ap- propriate supervision and is adopted as the default setting.
AP
AP Aoondoy
mAP
41.4
51.3
51.9
48.2
3
44.8
52.1
5e-3
46.3
51.5
53.1
50.3
1e 41.9
50.9
52.0
48.3
Table 3. Ablations about the weight / of edge di- rection loss.
2D-to-BEV Transformation. In Tab. we ab- late on the 2D-to-BEV transformation methods (e.g., IPM (Mallot al., 1991), LSS (Liu et al.,
Method
mAP
FPS
Param.
IPM
46.2
11.7
35.7M
LSS
49.5
10.0
37.1M
2022c; Philion & Fidler, 2020), Deformable At- tention (Li al., 2022c) and GKT (Chen et al., 2022b)). We use optimized implementation of LSS (Liu et al., 2022c). And for fair comparison with IPM and LSS, GKT and Deformable Attention both adopt one-layer configuration. Experiments show MapTR is compatible with various 2D-to-BEV methods and achieves stable performance. We adopt GKT as the default configuration MapTR, considering its easy-to-deploy property and high efficiency.
Deform. Atten
49.7
11.2
36.0M
GKT
50.3
11.2
35.9M
Table 4. Ablations about 2D-to-BEV transforma- tion methods. MapTR is compatible with various 2D-to-BEV methods and achieves stable perfor- mance.
4.3 QUALITATIVE VISUALIZATION
We show the predicted vectorized HD map results complex and various driving scenes in Fig. 1. MapTR maintains stable and impressive results. More qualitative results are provided in Ap- pendix C. We also provide videos (in the supplementary materials) show the robustness.
CONCLUSION
MapTR a structured end-to-end framework for efficient online vectorized HD map construction, which adopts simple encoder-decoder Transformer architecture and hierarchical bipartite matching to perform map element learning based on the proposed permutation eeuiivaleenivvvnttteee modeling. Extensive experiments show that the proposed method can precisely perceive map elements of arbitrary shape in the challenging nuScenes dataset. We hope MapTR can serve as a basic module of self-driving system and boost the development of downstream tasks (e.g., motion prediction and planning).

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENT
This work was in part supported by NSFC (No. 6227072399). We would like to thank Yicheng Liu for his guidance evaluation and constructive discussions.
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Structured bird's bird's-eye- view traffic scene understanding from onboard images. In ICCV, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Con- ghui He, Jianping Shi, Yu Qiao, and Junchi Yan. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022a.
Shaoyu Chen, Tianheng Cheng, Xinggang Wang, Wenming Meng, Qian Zhang, and Wenyu Liu. Efficient and robust 2d-to-bev representation learning via coometry-guidee kernel transformer. arXiv preprint arXiv:2206.045555 2022b.
Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Rethinking transformer in vision through object detection. Advances in Neural Information Processing Systems, 34:26183-26199 2021a.
Yuxin Fang, Shusheng Yang, Xinggang Wang, Yu Li, Chen Fang, Ying Shan, Bin Feng, and Wenyu Liu. Instances as queries. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6910-6919, 2021b.
Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, and Lizhuang Ma. Rethinking efficient lane detection via curve modeling. In CVPR, 2022.
Noa Garnett, Raf Cohen, Tomer Pe'er, Roee Lahav, and Dan Levi. 3d-lanenet: end-to-end 3d multiple lane detection. In ICCV, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: Future instance segmentation in bird's s-ed' view from surround monocular cameras. In /CCV, 2021.
Junjie Huang, Guan Huang, Zheng Zhu, Ye Yun, and Dalong Du. Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11799 2021.
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
Justin Lazarow, Weijian Xu, and Zhuowen Tu. Instance segmentation with musk-super polyg- onal boundary transformers. In CVPR, 2022.
Qi Li, Yue Wang, Yilun Wang. and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In ICRA, 2022a.
Weijia Li, Wengian Zhao, Huaping Zhong, Conghui He, and Dahua Lin. Joint semantic-gcometric learning for polygonal building segmentation. In AAAI, 2021.
Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206. 10092, 2022b.
10

---

Published as a conference paper at ICLR 2023
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird s-eye-view representation from multi-camera images via spa- tiotemporal transformers. In ECCV, 2022c.
Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun. Polytransform: Deep polygon transformer for instance segmentation. In CVPR, 2020.
Tsung-Yi Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, 2017.
Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler. Fast interactive object annota- tion with curve-gen. In CVPR, 2019.
Ruijin Liu, Zejian Yuan, Tic Liu, and Zhiliang Xiong. End-to-end lane shape prediction with trans- formers. In WACV, 2021a.
Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao Vectormapnet: End-to-end vectorized hd map learning. arXiv preprint arXiv:2206.08990, 2022a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021b.
Zhi Liu, Shaoyu Chen, Xiaojie Guo, Xinggang Wang, Tianheng Cheng, Hongmei Zhu, Qian Zhang. Wenyu Liu, and Yi Zhang. Vision-basee uneven bev representation learning with polar rasteriza- tion and surface estimation. arXiv preprint arXiv:2207.01878 2022b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sense fusion with unified bird's s-yee view representation. arXiv preprint arXiv:2205.13522 2022c.
Zichen Liu, Jun Hao Liew, Xiangyu Chen, and Jiashi Feng. Dance: A deep attentive contour model for efficient instance segmentation. In WACVW, 2021c
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Yuexin Ma, Tai Wang, Xuyang Bai, Huitong Yang, Yuenan Hou, Yaming Wang, Y. Qiao, Ruigang Yang, Dinesh Manocha, and Xinge Zhu. Vision-centric bev perception: A survey. arXiv preprint arXiv:2208.00 02797, 2022.
Hanspeter A Mallot, Heinrich H Bülthoff, JJ Little, and Stefan Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics, 1991.
Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou. Deep snake for real-time instance segmentation. In CVPR, 2020.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
Tixiao Shan and Brendan Englot. Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. In IROS, 2018.
Tixiao Shan, Brendan J. Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Daniela Rus. LIO-SAM: tightly-coupled lidar inertial odometry via smoothing and mapping. In IROS, 2020.
Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto De Souza, and Thi- ago Oliveira-Santos. Keep your eyes the lane: Real-time attention-guided lane detection. In CVPR, 2021.
Mingxing Tan and Quoe V. Lee. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICMI., 2019.
Jinsheng Wang, Yinchao Ma, Shaofei Huang, Tianrui Hui, Fei Wang, Chen Qian, and Tianzhu Zhang. A keypoint-based global association network for lane detection. In CVPR, 2022.

---

Published as a conference paper at ICLR 2023
Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo. Polarmask: Single shot instance segmentation with polar representation. In CVPR, 2020.
Wenqiang Xu, Haiyang Wang, Fubo Qi, and Cewu Lu. Explicit shape encoding for real-time in- stance segmentation. In ICCV, 2019.
Zhang and Sanjiv Singh. LOAM: lidar odometry and mapping in real-time. In Robotics: Science and Systems X, University of California, 2014.
Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In CVPR, 2022.
Chenming Zhu, Xuanye Zhang, Yanran Li, Liangdong Qiu, Kai Han, and Xiaoguang Han. Sharp- contour: A contour-based boundary refinement approach for efficient and accurate instance seg- mentation. In CVPR, 2022.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In ICLR, 2021.
12

---

Published as a conference paper at ICLR 2023
Appendix
A IMPLEMENTATION DETAILS
This section provides more implementation details of the method and experiments.
The resolution of source images is 1600 x 900. For MapTRR-aam nano, we resize the source images with 0.2 ratio. For MapnTR- resize the source images with 0.5 ratio. Color jitter is used by default.
Data Augmentation.
Model Setting. For all experiments, À is set to 2, a is set to 5, and [ is set to 5e during training. For MaptR-, we set the number of instance-level queries and point-level queries to 50 and 20 respectively. And set the size of each BEV grid to 0.3m and stack transformer decoder layers. We train MapTR- with a total batch size of 32 (containig 6 view images), a learning rate of multiplier of the backbone is 0.1. All ablation studies based MapTRR-tin trained with 24 epochs. For MapTRR-aao,, we set the number of instance-level queries and point- level queries to 100 and 20 respectively. And we the size of each BEV grid to 0.75m and stack 2 transformer decoder layers. We train MapTR-nano with I 10 epochs, total batch size of 192, learning rate of 4e learning rate multiplier of the backbone is 0.1. We employ GKT (Chen al., 2022b) as the default ID-to-BEV module of MapTR.
Dataset Preprocessing. We process the map annotations following Liu et al. (2022a); Li et al. (2022a). Map elements in the perception ranges of ego-vehicle are extracted as ground-truth map elements. By default, The perception ranges are [-15.0m, 15.0m] for the X-axis and 1-3000m, 30.0m] for the -axxis.
B ABLATION STUDY
Permutation Permuutatoo-cquiiaalent
equivalent (V, w/o ambiguity
140
Fixed order w/ ambiguity
0.45
0.40
100
0.35
80
0.30
0.25
40
0.20
Permutation equivalent (V. ambiguity
Fixed order V w/ ambiguity
2500 5000 7500 10000 05000000010000000000000000
10
15
20
25
steps
epochs
Figure 5. Convergence curves of permutation modeling methods.
Point Number. Ablations about the number of points for modeling each map element are pre- sented in Tab. 5. Too few points can not describe the complex geometrical shape of the map element. Too many points affect the efficiency. We adopt points as the default setting of MapTRR
Element Number. Ablations about the number map elements are presented in Tab. 6. We adopt 50 the default number of map elements for MapTTRiiny.
13

---

Published as a conference paper at ICLR 2023
Pt. num.
AP ped
AP divider
AP houndary
mAP
FPS
10
42.5
51.3
50.1
48.0
12.3
20
46.3
51.5
53.1
50.3
11.2
40
44.7
52.4
52.9
50.0
10.8
Table 5, Ablations about the number of points for modeling each map element.
Ele. num.
APped ped
AP divider
AP boundary
mAP
FPS
25
36.3
43.8
44.7
41.6
11.4
50
46.3
51.5
53.1
50.3
11.2
75
48.2
53.1
55.3
52.2
11.1
Table 6. Ablations about the number of map elements.
Decoder Layer Number. Ablations about the layer number of map decoder are presented in Tab. 7. The map construction performance improves with more layers, but gets saturated when the layer number reaches 6.
Layer num.
AP APped ped
disivider
AP boundory
mAP
FPS
1
20.8
30.2
36.3
29.1
15.2
2
36.0
43.1
48.0
42.4
14.2
3
38.2
44.1
49.5
44.0
13.5
6
46.3
51.5
53.1
50.3
11.2
8 39.6
51.9
51.2
47.6
10.6
Table Ablations about number of decoder.
Position Matching Cost. As mentioned in Sec. 3.2, we adopt the position matching cost term Eposition Aoonittionnnna (Va()), Vi) instance-level matching, for reflecting the position correlation between the predicted point set V and the GT point set V,. In Tab. 8, we compare two kinds of cost design. i.e., Chamfer distance cost and point2point cost. Point2point cost is similar to the point-level matching cost. Specifically,
find the best point2point assignment, and sum the Manhattan distance of all point pairs as the position matching cost of two point sets. The experiments show point2point cost better than Chamfer distance cost.
Position matching cost
AP APped
AP Addivider
AP boundory
mAP
Chamfer distance cost Point2point cost
40.3
53.8
48.5
47.5
46.3
51.5
53.1
50.3
Table 8. Ablations about the position matching cost term.
Swin Transformer Backbones. 2021b) are presented in Tab. 9.
Ablations about the Swin Transformer backbones (Liu et al.,
Method
Backbone
AP ped
AP divider
AP boundary
mAP
FPS
Param.
MapTR-tiny
R50
46.3
51.5
53.1
50.3
11.2
35.9M
MapTR-tiny
Swin-tiny
45.2
52.7
52.3
50.1
9.1
39.9M
MapTR-small
Swin-small
50.2
55.4
57.3
54.3
7.3
61.2M
MapTR-
Swin-base
50.6
58.7
58.4
55.9
6.1
99.2M
Table Ablations about Swin Transformer backbones.
Modality. Multi-sensor perception is crucial for the safety of autonomous vehicles, and MapTR is compatible with other vehicle-moune sensors like LiDAR. As illustrated in Tab. 10, with the schedule of only 24 epochs, multi-modality MapTRR significantly outperform previous state-of-the- art result by 17.3 mAP while being 2x faster.
Robustness to the camera deviation. In real applications, the camera intrinsics are usually ac- curate and change little, but the camera extrinsics may be inaccurate due to the shift of camera position, calibration error, etc. To validate the robustness, we traverse the validation sets and ran- domly generate noise for each sample. We respectively add translation and rotation deviation of
14

---

Published as a conference paper at ICLR 2023
Method
Modality
Epochs
ped
AP divider
houndary
mAP
FPS
HDMapNet
C& L.
30
16.3
29.6
46.7
31.0
0.5
VectorMapNet
C&
110
37.6
50.5
47.5
45.2
<2.9
Map
C
24
46.3
51.5
53.1
50.3
11.2
MapTR-ttnn
L
24
48.5
53.7
64.7
55.6
7.2
MapTTR-ing
& L
24
55.9
62.3
69.3
62.5
5.8
Table 10.
Ablations about the modality.
(m)
Method
0.05
0.1
0.5
1.0
MapyTR-tiny
50.3
49.9
49.0
34.0
17.0
Table 11. Robustness to the translation deviation of camera. The metric is mAP. is the standard deviation of
Az, Ag. A..
02 (rad)
Method
0.005
0.01
0.02
0.05
MapTR-tiny 50.3
49.4
47.5
42.0
24.7
Table 12. Robustness to the rotation deviation of camera. The metric is mAP. da the standard deviation of O,, 0g O..
different degrees. Note that add noise all cameras and all coordinates. And the noise subject to normal distribution. There exists extremely large deviation in some samples, which affect the per- formance a lot. As illustrated in Tab. 11 and Tab. 12, when the standard deviation of A.. Ag. A is 0.1m or the standard deviation of 0z, fy. 02 is 0.02rad, MapTR still keeps comparable performance.
Detailed running time. have deeper understanding on the efficiency MapTR, we present the detailed running time of each component in Map with only multi-camera input in Tab. 13.
Component
Runtime (ms)
Proportion
Backbone
55.5
62.1 %
2D-to-BEV module (GKT)
12.3
13.8%
Map decoder
21.5
24.1% 1%
Total
89.3
100
Table 13. Detailed running time for each component in MapTR-tiny on a 3090.
C QUALITATIVE VISUALIZATION
We visualize map construction results of MapTR under various weather conditions and challenging road environment on nuScenes val set. As shown in Fig. 6, Fig. 7 and Fig. 8, MapTR maintains stable and impressive results. Video results are provided in the supplementary materials.
15

---

Published as a conference paper at ICLR 2023
SION
BICK
ERAA 1aa
I0I
FRONT
IUNI
SWS
RUCK RMCLA
BOON-BIGHH
Figure 6. Visualization under sunny and cloudy weather.
16

---

Published as a conference paper at ICLR 2023
SION
BACX
BUCA
EDAMT Fat
FIGHT-RIGHT
TRONI
ABON LIGHT
3W. LEFT
NON
TI
B4C4
Figure 7. Visualization under rainy weather.
17

---

Published as a conference paper at ICLR 2023
RONILLEP
FRONT
FRONI MONNRROO RIGHT
V
BACK.LEFT.
BACK
BUCKLAIGHT
FRONTLEFT
FRONT
FRONT_RIGHT
BNCKLLEFT
BACK_AI0HT
LEOT
BACK_LEFT
TOOTTLO
FRONT
RIGHT
SMCK.LEFT
BACK
RIGHTRIGHT
FRONTLEFT
FRONT
FRONT_RIGHT
B4C4
BACK
Figure 8, Visualization night.
18


Reviews:
[{"review_id": "hRMswm784y", "reviewer": "Reviewer_69Fv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose HD map construction algorithm from multi-view cameras.  They model each map element as a point set with a group of equivalent permutations.  The hierarchical matching both for instance-level and point-level is introduced and trained based on point2point loss and edge direction loss.  The experimental study shows that MapTR achieves higher extimation accuracy than the existing methods on public nuScenes dataset.  MapTR with light backbone can infer HD map at 25fps which is required for autonous driving.", "summary_of_the_review": "The proposed method is novel and achieves definitely higher accuracy than the existing methods.  I think that this paper deserves acceptance to ICLR.", "strength_and_weaknesses": "Strengths\n+ The map elements are represented as a set of points with a group of equivalent permutations which is suitable for expressing various shapes with edge directions.\n+ The proposed hierarchical matching and loss functions look technically sound.\n+ The performance is evaluated on public dataset and the proposed method achieves higher accuracy than the existing methods.\n+ The transformation from 2D to BEV in map encoder could be substituted by other modules such as conventional IPM.\n\nWeaknesses\n- The experimental studies are carried out only on nuScenes and not on Argoverse2 dataset like the existing method.  Are there any reasons for this?\n- More discussion on robustness towards real application is helpful.  For example, Are multi-camera input mandatory?  Can a front camera output HDMap in front of the vehicle?  How about the sensitivity of MapTR to camera position, intrinsic parameters, extrinsic parameters?  These robustness is important.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "EN-EBbkmU8j", "reviewer": "Reviewer_oVMw", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel pipeline for HD map learning, which is an essential component of autonomous driving. Traditional methods use pre-annotated HD maps for localization and mapping, preventing autonomous driving scaling up. To address that, recent works aim to predict the HD maps on-the-fly with machine learning models. These works, however, require some hand-crafted post-processing steps. This paper proposes a fully end-to-end approach to HD map learning, based on recent DETR architecture. In addition to the architecture, this paper also studies how to effectively model map elements and introduces a permutation equivalent modeling of map elements. Under all settings, this paper shows significant improvements over baselines (HDMapNet & VectorMapNet). ", "summary_of_the_review": "Overall, I like this paper and recommend weakly acceptance though with several concerns (I believe they can be addressed in the rebuttal). I think this paper proposes a simple and effective pipeline for this online map learning problem. It presents clear motivation and strong empirical performance. That said, I am looking forward to seeing authors' comments to my concerns. ", "strength_and_weaknesses": "Strength:\n\n1. This paper further streamlines recent learning-based map learning approaches and proposes a fully end-to-end solution to learning an online HD map from multi-camera images and point clouds. This pipeline is simple and effective.\n\n2. The proposed hierarchical modeling of map elements is technically sound. This follows a coarse-to-fine paradigm and model map elements in both instance level and point level. Such operation not only improves stability and performance of map learning as studied in this paper, but also has potentials to benefit other tasks that also use a Transformer-based architectures. \n\n3. This paper also introduced a permutation equivalent module to model map elements, which I think is novel. This permutation equivalent module removes the ambiguity introduced by primitives like polylines and polygons.\n\n4. This paper demonstrates strong empirical performance on several benchmarks, compared to existing methods. Also, this paper also shows that the proposed method can achieve real-time performance and be developed to cars. \n\nWeaknesses:\n\n1. My first concern is the reproducibility. The proposed method seems to be a complex Transformer model with several new modules. With current level of details, I am not sure if one can faithfully reproduce this work and achieve similar performance. \n\n2. It remains an open question to choose between polylines and polygons if we consider new map elements. Can this paper present some guidances of that? Can we unify the polyline representations and the polygon representations for all map elements? \n\n3. More ablation studies are needed. E.g., the choices of polylines and polygons as stated above. \n\n4. This paper (arguably) overclaims several contributions. It is *not* the first end-to-end approach as far as I know (e.g. VectorMapNet already introduced the Transformer-based end-to-end pipeline). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "D5HOxtuLgd", "reviewer": "Reviewer_1muv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a vectorization algorithm for map construction from surrounding views (RGB). \n\n\nIt acheives state-of-the art on the nuScenes dataset, surpasses multi-modality methods (RGB+LiDAR), for a final 58.7mAP (~ +8mAP over the previous method). A toned-down version of the algorithm (45.9mAP) reaches real time (~25FPS) on a high end GPU (RTX 3090).\n\nThe paper is essentially GKT (Chen et al., 2022b) + DETR (Carion et al., 2020) - style matching extended to point level + permutational encoding for vertex sequences. The speed gain is mostly due to GKT, the matching scheme is very similar to DETR, which leaves us with the main contribution: the permutational-equivalent encoding which adding all possible vertex sequences at training (~+6mAP gain).", "summary_of_the_review": "Solid performance improvements for map vectorization on nuScenes (~+8%mAP), improved speed over predecessors, but under-investigated gains source (e.g., is it the vectorization method? is semantic segmentation still a better start, or we can drop it and just use vectorization, similar to BoundaryFormer, but keep the speed?) and frugal ablation study. This paper could have been very good; given the limited set of experiments, I cannot vouch for its future impact.", "strength_and_weaknesses": "### Strengths:\n- state-of-the-art results on nuScenes (~58.7 mAP, +8mAP)\n- 8x more efficient processing compared to the previous contender (VectorMapNet)\n- introduces modeling map element as a point set with a group of equivalent permutations\n\n### Weaknesses:\n- the vertex-to-polygon problem has a number of modelling options available in the literature (e.g., [BoundaryFormer, 1*, 2*-- the permutation issue doe]); apart from HDMapNet / VertexMapNet, none of them have been discussed; BoundaryFormer is summarily dismissed due to different domain.\n- the speed claim can be largely attributed to GTK; given that GTK runs at 72.3 FPS on the 3090 (for the same dataset, probably no large setup differences), at least a proper timing table should be added (how much time is spent for encoder/decoder/matching)\n- the paper could use a logic check, there are a number of redundant claims / language errors (see below)\n\n\n[1*]Liang, J., Homayounfar, N., Ma, W. C., Xiong, Y., Hu, R., & Urtasun, R. (2020). Polytransform: Deep polygon transformer for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9131-9140).\n[2*]Li, W., Zhao, W., Zhong, H., He, C., & Lin, D. (2021, May). Joint semantic-geometric learning for polygonal building segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 35, No. 3, pp. 1958-1965).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%