Online Map Vectorization for Autonomous Driving: A Rasterization Perspective
Authors: Gongjie Zhang
Jiahao Lin
Shuang Wu
Yilin Song
Zhipeng Luo
Yang Xue
Shijian Lu
Zuoguan Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Online HD Map Construction
Map Vectorization
Autonomous Driving
Evaluation Metric
Rasterization
Differentiable Rasterization
Bird's-Eye-View Perception
TL;DR: Rasterization can offer complementary benefits to map vectorization. Motivated by this, we propose (i) MapVR, a precise map vectorization framework combining rasterization, and (ii) a highly sensitive rasterization-based metric for map vectorization.

Abstract: High-definition (HD) vectorized map is essential for autonomous driving, providing detailed and precise environmental information for advanced perception and planning. However, current map vectorization methods often exhibit deviations, and the existing evaluation metric for map vectorization lacks sufficient sensitivity to detect these deviations. To address these limitations, we propose integrating the philosophy of rasterization into map vectorization. Specifically, we introduce a new rasterization-based evaluation metric, which has superior sensitivity and is better suited to real-world autonomous driving scenarios. Furthermore, we propose MapVR (Map Vectorization via Rasterization), a novel framework that applies differentiable rasterization to vectorized outputs and then performs precise and geometry-aware supervision on rasterized HD maps. Notably, MapVR designs tailored rasterization strategies for various geometric shapes, enabling effective adaptation to a wide range of map elements. Experiments show that incorporating rasterization into map vectorization greatly enhances performance with no extra computational cost during inference, leading to more accurate map perception and ultimately promoting safer autonomous driving. Codes are available at https://github.com/ZhangGongjie/MapVR. A standalone map vectorization evaluation toolkit is available at https://github.com/jiahaoLjh/MapVectorizationEvalToolkit.

Online Map Vectorization for Autonomous Driving: A Rasterization Perspective
Gongjie Zhang
Jiahao Lin
Shuang Wu
Yilin Song
Zhipeng Luo
Yang Xue
Shijian Lu
Zuoguan Wang
Black Sesame Technologies
Nanyang Technological University, Singapore
equal contribution.
corresponding author.
Project Page: httpe: http://hhhh....../////oooo///aa/
(a) Map Rasterization
(b) Map Vectorization
MultiView Cameras
Rasterized Map
Niew Cameras
Vectorized Map
(c) MapVR (Map Vectorization via Rasterization)
Remove during inference
Map Vectorization
>>>>
>>>>
>>>>
>>>>
Model
Differentiable
Rasterizer
Multi View Cameras
Auxiliary Training
Supervisian
Vectorized Map
Training Supervision
Differentiably Rasterized Map
Figure 1: (a) Map rasterization produces HD semantic maps as output via semantic segmentation in bird's-eye view (BEV). (b) Map vectorization directly predicts compact and instance-level vectorized map elements that are better suited for autonomous driving systems. (c) Map employs differentiable rasterization to bridge vectorized and rasterized HD map representations, enabling more precise and accurate vectorized HD maps for reliable autonomous driving.
Abstract
Vectorized high-definition (HD) map is essential for autonomous driving, provid- ing detailed and precise environmental information for advanced perception and planning. However, current map vectorization methods often exhibit deviations, and the existing evaluation metric for map vectorization lacks sufficient sensitivity detect these deviations. To address these limitations, propose integrating the philosophy of rasterization into map vectorization. Specifically, we introduce a iisterization-hase evaluation metric, which has superior sensitivity and is better suited to real-world autonomous driving scenarios. Furthermore, we propose MapVR (Map Vectorization via Rasterization), novel framework that applies differentiable rasterization to vectorized outputs and then performs precise and geometry-aware supervision on rasterized HD maps. Notably, MapVR designs tailored rasterization strategies for various geometric shapes, enabling effective adaptation wide range of map elements. Experiments show that incorporating rasterization into map vectorization greatly enhances performance with no extra computational cost during inference, leading to more accurate map perception and ultimately promoting safer autonomous driving.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Online high-definition (HD) map construction is essential for autonomous driving systems, as supplies real-time and comprehensive information about the vehicle's s surroundings, such as lanes, curbsides, and crosswalks. It serves as the foundation for the vehicle's s navigation, planning, and decision-making processes, and is integral to the effective functioning of self-driving vehicles.
Existing online HD map construction methods fall into two classes: map rasterization and map vectorization. Map rasterization [37, 40, 12, 59, 19, 60, 36, 50] is straightforward as shown in Fig. 1 (a); models HD map construction as semantic segmentation task in bird's s-eye view (BEV), rasterizing the surroundings into semantic maps as output. However, rasterized maps are not ideal representations for autonomous driving, as they lack instance-level and structural information, and require extensive post-processing to be consumed by subsequent navigation and decision-making modules. To address these limitations, map vectorization (Fig. I (b)) emerges as a popular solution for constructing HD maps. HDMapNet [16] and SuperFusion [6] employ complex post-processing to group pixels from rasterized maps into vectors. The recent VectorMapNet [26] and MapTR [20] directly predict map elements vectorized point sets, achieving better accuracy with faster runtime.
Both VectorMapNet [26] and Map [20] utilize a sparse point set representation, where each map element is parameterized as a fixed-length vector of equidistantly sampled points, with LI loss applied to supervise regression predictions. While this approach is simple and intuitive, we empirically observe that it is often suboptimal due to several reasons. First, as shown in Fig. 2, the sparse point set representation is often lacking in precision, particularly when dealing with sharp bends complex details of map structures, resulting in significant parameterization errors.' Second, learning with equidistant points as regression targets causes ambigious supervision, because the intermediate points often lack clear visual clues. Third, relying solely the I.1 loss for regression supervision causes the model to overlook fine- grained geometric variations, yielding overly smooth predictions that are insensitive to local deviations. Likewise, the current evaluation metric, which relies on Chamfer distance among point sets, tends to overlook minor deviations and geometric details. For autonomous driving, where precision is matter of life and death, existing methods and metric for map vectorization are still inadequate.
Ground Truth
Equidistant- Serddisttnt-Poin-PPPe Point-Parrmeteriiee Line
Figure 2: Inaccurate map elements caused by the parameterization of sparse equidistant point sets.
To address these limitations, we reintroduce the philosophy of rasterization into map vectorization, to bring back the advantages of precision in HD map modeling while keeping the merits of vectorized outputs. We believe that rasterization can offer complementary benefits to map vectorization.
With the above motivation, we first design new rasterization- evaluation metric for map vectorization, which more sensitive to minor deviations and better suited for practical driving scenarios. Unlike existing metric that uses Chamfer distance to determine if a map element matches the ground truth, we rasterize both the predicted and ground truth map elements into HD maps, and then use mean intersection-over over-union (mloU) to decide whether they match. This metric aligns better with human perception, takes into account the actual shape and geometry of individual map elements, and offers increased sensitivity minor discrepancies.
We further present MapVR (Map Vectorization via Rasterization), a novel framework for precise HD map vectorization. MapVR can be integrated with any architecture that directly predicts vectorized map elements [16, 20J. Unlike existing map vectorization methods, our Map applies differentiable rasterization to vectorized output (ordered point sets) during training, transforms each vectorized map element into an HD map, and adds segmentation supervision on the rasterized HD maps. The proposed Map sharing the philosophy with our aforementioned evaluation metric, enables more precise and detailed supervision, thus significantly boosting precision. It also provides more reasonable supervision, it removes the ambiguity caused by equidistance. MRp can also adapt
While increasing the vector parameterization's 's dimensionality could resolve this issue theoretically, such an approach has been found to be not helpful in practice, observed in MapTR [20].

---

to wide range of map elements with specially designed geometry- aware differentiable rasterization strategies, showing strong scalability. At the inference stage, the additional differentiable rasterization can be simply removed, and the network's vectorized output can be employed as the final result. As our method does not introduce any additional computational overhead during inference, maintains high efficiency, while delivering more accurate and robust map construction results.
The contributions of this work are summarized as follows:
We propose a novel rasterization- bbsed evaluation metric for map vectorization that exhibits increased sensitivity to minor deviations, providing more accurate and reasonable assessment of map vectorization performance in real-world driving scenarios.
We propose MapVR (Map Vectorization via Rasterization), a novel framework that seamlessly combines differentiable rasterization with existing map vectorization approaches. Map VR substantially improves the precision for map vectorization, demonstrates robust scalability for diverse map elements, and incurs no extra computational overhead during inference.
The proposed Map VR framework and evaluation metric pave the way for future research and advancements in map vectorization for autonomous driving applications, demonstrating the complementary benefits of rasterization to map vectorization.
2 Related Work
HD Map Construction. Understanding the vehicle's surrounding environment, including lanes,
curbsides, crosswalks, and road topology, plays a central role in the navigation and decision-making of autonomous driving. Such driving scene information is usually provided by high-definition (HD) maps. Conventionally, HD maps are constructed offline using SL.AM-based methods [58, 32, 41, 42] with complex pipelines. Recently, with the emergence of the bird's- sard's-eye-view (BEV) perception [19, 27, 51, 36, 30, 48, 29], the focus has shifted towards online HD map construction, which generates maps around ego-vehicle from vehicle-mounted sensors (e.g., cameras) the fly.
Currently, there are two prevalent paradigms in online HD map construction: map rasterization and map vectorization. Rasterization methods [37, 40, 12, 59, 19, 60, 36, 50] generate HD maps via semantic segmentation BEV, which have good sensitivity instance-level information and lane topology limits the utility of rasterized maps in downstream tasks like navigation and planning. On the other hand, map vectorization addresses this limitation by producing vectorized map elements. HDMappNet [16] and SuperFusion [6] employ post-processing to group pixels from rasterized maps into vectorized elements. Moreover, VectorMapNet [26] proposes to directly predict map elements as vectorized point sets in auto-regressive manner, achieving superior performance. And MapTR [20] - the current state the art, further proposes unified permutation-eqqiialern modeling approach to model the HD map elements, achieving superior accuracy. Furthermore, MapTR [20] achieves real-time efficiency with its one-stage and parallel framework. However, despite the recent progresses. vectorized maps still often exhibit minor deviations that can be critical in autonomous driving, where safety is of utmost importance.
details. However, the lack of vital
Lane Detection. Lane detection, which can be seen as a sub-tasi task of online HD map construction, concentrates on map elements like lanes and curbsides. Most existing lane detection research targets 2D camera views, and can be categorized into several modes, including pixel-level segmentation [33, 39, 35, 52], machor-point-based regression [43, 18], and curve-prior fitting using polynomial [46, 44, 23] or Bezier curves [8]. With the recent progresses in BEV perception, some lane detection methods [10, 4, 47, 13] have also extended into 3D, perceiving lanes in BEV, aligning more closely with online HD map construction. Nonetheless, lane detection remains somewhat limited, perceiving only highly-regularizer line-shaped map elements. In contrast, map vectorization has better scalability and adaptability with fewer assumptions, making it better fit for real-world autonomous driving.
Differentiable Rasterization. Rasterization, a concept from computer graphics, refers to the process of rendering vector graphics representations (point coordinates or math formulas) into raster images (a series of pixels) for display on computer screens [45]. Typically, rasterization is non- differentiable [38, 9]. Fortunately, recent advances in graphics and vision [28, 24, 5, 25, 21, 17, 17,34, 15] have achieved differentiable rasterization, bridging the gap between vector graphics and raster image through backpropagation. In this work, we make the first attempt to adapt differentiable rasterization to the map vectorization task to bridge vectorized outputs and rasterized HD maps. enables more refined and comprehensive supervision and yields predictions with improved precision.
3

---

10.00, 6.12,
(0.5., 0.20, (1.1,,
(8.6).085
Rasterization
Dilation
[L 0.88)
Vactorized
threshold)
Truth
[8
matched
Rasterization
Dilation
0.81,
Vectorized
Prediction
Figure 3: Illustration of our proposed rasterization basod approach for determining the match between ground truth and predicted vectorized map elements.
3 A Rasterization-Based Evaluation Metric for Map Vectorization
3.1 Review of Existing
Evaluation Metric
Map vectorization requires instance-level evaluation, similar object detection [7, 22, 53, 54, 3, 55, 57, 56, 61]. Thus, current map vectorization works [16, 6, 26, 20] adopt Average Precision (AP) to evaluate the map construction accuracy, using Chamfer distance to determine whether the predicted map element and the ground truth map element match.
Specifically, Chamfer distance DChamfer (t: is measure of dissimilarity between two unordered point sets, which quantifies the average distance between each point in one set to the nearest point in the other set. It can be formulated as:
Dehamfer (P,Q) = 0.5 x
Σ min p g/2 + PEP q=Q
1 Q
Σ qEQ
min lp 4/2),
(1)
PEP
where and Q are the sets of points representing the predicted map element and the ground truth map element, respectively, |P] and [Q] are the cardinalities of point sets P and Q, and |p g/2
denotes the Euclidean distance between points p and q.
Despite its simplicity and ability to provide fair evaluation results, the following limitations of this metric make inadequate for highly demanding scenarios such as autonomous driving: /) is not scale-invariant; for smaller map elements such as stoplines, Chamfer distance error is consistently small, failing to provide meaningful assessment. 2) Chamfer distance solely relies on unordered point set distance, completely overlooking the shape and geometrical details of the map elements, thus yielding unreasonable results for many practical scenes, as shown in Fig. 4. These draw backs call for the development of more robust and accurate evaluation metric tailored to the stringent requirements autonomous driving map vectorization.
3.2 Proposed Rasterization-Ban -Based Evaluation Metric
To address the aforementioned limitations, we introduce a rasterization- seserizatiio-bbaeed evaluation metric that more sensitive minor deviations and better suited for real-world driving scenarios. While we still employ AP as our measurement, we adopt rasterization to precisely determine the matching between predicted and ground truth map elements.
As shown in Fig. we demonstrate our metric using line-shaped map elements (e.g.. lanes and curbsides). First, both ground truth and predicted elements are rasterized into a polyline in HD maps. In our setup, considering the perception range of =30m the y-axis and 15m on the x-axis, we set the spatial size of the HD map 480 x 240, such that each pixel represents 0. 0.125m, satisfying the sugh-precision requirement of autonomous driving. To better accommodate inaccuracies in predictions with thin and elongated geometry, we then dilate the rasterized polylines by 2 pixels on each side, thereby introducing an appropriate degree of tolerance. Finally, to determine whether the ground truth and predicted map elements match, we calculate the intersection--ver-umme their respective rasterized HD representations. Similar to MS- MS-COCO's 's metric [22], AP calculated at multiple IoU thresholds. For line-shaped elements, set the thresholds as 0.25 0.50 0.05.
(IoU) of

---

(a) Denowfor + 0.35 matched
(b) Dotenter 0.35 matched
(c)
Dawenfer* 1.12 not matched
(d)
Donawter - 0.75 matched
mloU matched
matched
mloU 0.56 matched
= not
Figure 4: Evaluation quality comparison between the Tlamfer-distance--aseee metric and our pro-
posed rasterization-be -raddd based metric on a few practical cases. Our metric is able to produce more
reasonable evaluation suitable for autonomous driving applications.
is worth noting that HD maps often contain elements other than lines, such as crosswalks, inter- sections, and carparks. These elements can be abstracted into polygons. To conduct appropriate evaluation for polygon-shaped map elements, we apply specially-designed polygon-shaped rasteriza- tion instead of line-shaped rasterization, and compute AP over 0.50 0.75 0.05.
3.3 Comparative Analysis and Discussion
Evaluation Quality. We examine the evaluation quality of the two metrics with a few practical examples. Fig. 4(a) displays a case involving a short stopline, where the prediction is perpendicular to the ground truth. The Chamfer distance metric judges a match, as it lacks scale-invariance, While the rasterization- baaedd metric successfully recognizes the discrepancy based on their low IoU. Fig. 4(b) presents scenario in which the predicted lane/curbside exhibits minor horizontal deviation from the ground truth. Such deviations, even if small, pose critical dangers in real driving scenes. The shamfee-distance-basedd metric considers the prediction as matched solely based on the small point- set distance. Conversely, our metric takes geometry into consideration, determining that they do not match. Fig. 4(c) illustrates a case with vertical deviation between the prediction and ground truth, typically arising from occlusion. This situation is generally non-critical, as the map updates continuously as the vehicle moves forward. By incorporating shape and geometry knowledge, the rasterization-base based metric evaluates more reasonably. Fig. 4(d) also verifies that our metric is more sensitive to small but critical errors. Collectively, these examples show that the rasterization-bat metric offers superior sensitivity and is better aligned with practical autonomous driving scenarios.
Computational Complexity. The rasterization-butt metric requires additional computation for rasterization but still runs acceptably fast. Empirically, the evaluation process nuScenes Map [2] validation set takes ~ 3 minutes on our server equipped with an Intel Xeon Gold 6226R CPU.
MR (Map Vectorization via Rasterization)
4.1 Framework Overview
As shown in Fig. I(c). MapVR a novel and generic learning framework for map vectorization, which combines rasterization to leverage the fine-grained supervisory signal from the rasterized HD maps while retaining the benefits of vectorized representation. MapVR is parameter-free and thus can be easily integrated with various network architectures for map vectorization (e.g., MapTR [20]).
Fig. illustrates the overall framework of MapVR. During training. the base map vectorization model first generates vectorized representation for each map element. Then, MapVR produces an HD map by rendering the vectorized element with a specially-designee differentiable rasterizer. Finally, begmentation-basel losses can be directly applied the rendered HD maps, providing more granular supervision on the shape and geometry of the map elements, which leads to more precise results.
4.2 Differentiable Rasterization: Bridging Vectorized Representation and HD Semantic Maps
Rasterization serves as a vital bridge between vectorized representation and HD maps. Generally, rasterization is not differentiable due to the binary assignment that decides whether a pixel covered by any shape primitive. Inspired by [24, 5, 17, 15], to enable fine-grained supervision signals directly from HD maps, we introduce soft version of rasterization, which renders each vectorized map element into an HD mask while preserving the whole framework's 's differentiaaiiityy
5

---

Banaa (226,026
eatl 11.82 D.Ki.
Dice
D380.
biffeenttibble Rastemser
Diflerentiable
Vectorited Ground Truth Type: Lane (Lina)
Vectorized Prediction
Rasterterr
12-4.a s8) It da. snl
11.2 6.15 8.15)
+
Map Vectorization
Lass
11.04 0-55)
>>>>
Differentable
Diforentiablc
Vectorized
Model
Vectorized
Rasterizer
Resteriaur
Ground Truth
Prediction
Type:
(Polygon)
Figure The learning pipeline of MapVR. MapVR utilizes base model for vectorized map generation, followed by a customized differentiable rasterizer to produce HD maps, which fine- grained, geometry-aware supervision is applied enhance the precision vectorized elements.
Concretely, for a line-shaped map element represented by an ordered point set P, we compute its softly-rendered mask Jine
[0, 1] x with
A (2, P) exp - -D(,,
where (æ, y; P) denotes the closest distance from pixel (I, 2) all segments of the polyline P, and the softness т controls the rasterization smoothness. A larger yields smoother transitions between the polyline and empty regions, while smaller leads sharper, more distinct line boundaries.
While for polygon-shaped map elements like intersections, the rendered mask [polygon computed as
polyyon (z, y; P) = [C(,,;; D(x, MR
where D(x, yú P) is the closest distance from pixel (x,, y) to any boundary segment of the polygon P. and C(z, y; P) E {-1,+1} indicates whether pixel (æ, y) falls inside (+1) or outside (-1) the polygon. a--) denotes the sigmoid function. Similarly, the softness controls the transition smoothness of the rasterized values at the polygon boundary areas.
Our differentiable rasterizer (Eq. & 3) transforms each vectorized map element into rasterized HD mask representation a parameter-fre free manner, which enables the learning of fine-grained shapes and geometric details through direct supervision on these rasterized HD masks.
4.3 Training and Inference Procedure
Training. Fig. illustrates how differentiable rasterization is incorporated into the map vectorization framework. First, we use a base map vectorization model (e.g.. MapTR [20]) to predict a set of vectorized map elements. Then, instead of relying on LI loss with equidistant points as targets as in [20, 26], we render both vectorized prediction and vectorized ground truth into rasterized HD masks, and apply supervision directly the masks using dice loss [3]]. Thanks the differentiability of our designed rasterization processes (Eq. 2 & 3), the segmentation loss is able to guide the learning of vectorized predictions. Notably, this supervision is geometry-aware (line-shaped or polygon-shaped rasterization) determined by the class of the target map element. The effectiveness of geometry- rendering is validated in Section 5.3. Moreover, the rasterization- based segmentation loss effectively weighs down the equidistance requirement (which is ill ill-posed due to the lack of clear visual clues), thus providing more reasonable learning target.
the rasterization procedure
In addition to the rendering-bann loss, we include direction regularization loss as an additional auxiliary loss. Specifically, we define the direction regularization loss the vectorized output as
N
P.P.4.,P41PP44
Cair
-PP.-il--PPP-PPP44) IP.1P
where P; denotes the th point in the predicted point set. It encourages the predictions to avoid unnecessary direction changes along adjacent segments. This effectively promotes smoother point set to avoid back-and-forte patterns that are not penalized by the rendering loss, and also facilitates the allocation of more points regions with higher curvature and fewer points in straight-line regions.
Efficient Inference. After training, the rasterization processes are no longer needed. Consequently, MapVR enhance map vectorization without adding any extra computational cost during inference.

---

Table 1: Comparison of various map vectorization methods on nuScenes Map (basic) validation set.
Method
Modality
Backbone
#Epochs
Chammerr hdry
AP raster
FPS
ped
avg.
ped
div
hdry
avg.
HDMapNet [16]
Effi-BO Effi-BO
30
14.4
21.7
33.0
23.0
0.8
HDMapNet [16]
&
30
16.3
29.6
46.7
31.0
0.5
VectorMapNet [26]
C
Rcs-50
110
36.1
47.3
39.3
40.9
26.2
12.7
6.1
15.0
2.9
VeetorMapNet [26]
C & L
Rcs-50
110
37.6
50.5
47.5
45.2
MapTR[P [20]
C
Rcs-50
24
46.3
51.5
53.1
50.3
32.4
23.5
17.1
24.3
18.4
MapTR [20]
C
Res-50
110
56.2
59.8
60.1
58.7
43.6
35.6
25.8
35.0
18.4
MapTR [20)
C & I.
Res-50
24
56.4
61.8
70.1
62.7
46.4
39.2
50.0
45.2
7.2
MapTR [20] + VR (Ours)
Res-50
24
47.7
54.4
51.4
51.2
37.5
33.1
23.0
31.2
18.4
MapTRR [20] VR (Ours)
Res-50
110
55.0
61.8
59.4
58.8
46.0
39.7
29.9
38.5
18.4
MapTR [20] MapVR (Ours)
C& L
Res-50
24
60.4
62.7
67.2
63.5
52.4
46.4
54.4
51.1
7.2
In modality 'C' denotes multi view camera iuput and &C denotes combined mubti view camera and LiDAR input.
When LJDAR data incorporated, PointPillars serves the backbone processing the LiDAR data.
The abbreviations 'ped', 'dv',, and 'bdry' correspond to the map elements of pedestrian crossing, divider, and boundary, respectively.
Table 2: Map vectorization performance nuScenes Map (extended) validation set.
AP Chamfee
AP roster
Method
FPS
ped
stp
int
cap
div
bdry
ped
stp
cap
div
bdry
avg.
MR [20]
34.3
29.9
21.5
37.9
44.9
45.1
35.6
22.5
12.1
38.4
23.4
18.3
12.1
21.1
18.4
TR [20] + MapVR (Ours)
39.5
31,6
21.9
42.4
45.8
45.9
37.9
30.8
13.9
43.3
32.8
27.0
18.8
27.8
18.4
competing mothods multi view camerus as input, ResNet-50 [11] as the buckbonee 'ped', 'sp,, 'int', 'cap', 'div', and ''dry' denote pedestrian srrian crossing, stopline, intersection, carpork orea, divider, and boundary, respectively.
trained for epochs.
5 Experiments
5.1 Experiment Setup
Dataset and Evaluation Metrics. MapVR is evaluated across multiple datasets, outlined below.
1. nuScenes Map (basic) [2], which consists of two line-shaped map classes (lane divider and road boundary) and one polygon-shaped map class (pedestrian crossing). This dataset setup aligns with prior works on map vectorization [16, 6, 26, 20].
2. nuScenes Map (extended) [2], extension of nuScenes Map (basic) that incorporates
more complex map elements, such as intersection, stopline area, and carpark area.
3. Argoverse2 [49], a large-scale dataset featuring the same classes as nuScenes Map (basic).
4. 6V-mini-v0.4 4, our proprictary large-scale commercial dataset for autonomous driving, cov- ering very complex driving scenes in real world. It includes three line-shaped classes (lane, curbside, and stopline) and two polygon-shaper classes (crosswalk and intersection).
Both Chamfer-distance hased metric (Section 3.1, denoted as AP Chamfer) and the newly proposed rasterization- based metric (Section 3.2, denoted as AP raster) are used for performance evaluation.
Implementation Details. All experiments, unless otherwise stated, conducted with 8x NVIDIA RTX 3090 GPUs As the proposed MR is a generic framework with no reliance on specific model architecture, adopt MapTR [20], the state-of-ther oh--the-rttttttto model for map vectorization, the base model. Our implementation aligns with MapTR [20]. Please refer to the appendix for more details.
5.2 Experiment Results
Results on nuScenes Map. Table I compares MapVR with existing map vectorization techniques on nuScenes Map (basic). Even under the less sensitive AhChmmerr metric, our proposed MapVR delivers superior overall performance across various settings. The advantage of MapVR becomes even more pronounced under the more precise and autonomous-drivingg-rriiee Specifically, MR provides notable 3.5% improvement over a fully-trained Map'TR. When working with multi-modality inputs, MapVR obtains an even larger margin of 5.9%. As shown in Table 2, on the more challenging nuScenes Map (extended) dataset that includes more complex elements, our MapVR achieves superior performance across all map elements under both metrics. These results validate the substantial improvements brought by Map and its exceptional capability of adapting to challenging scenarios. It noteworthy that these improvements are achieved without adding any additional computational burden during inference.
A.P raster metric.
7

---

Ego-Vehicle Surrounding Views
Ground Truth
MapTR
Ours
MMMM
Road Divider
Boundary
Ped. Crossing
Figure 6: Visualization of online HD map vectorization results. Our proposed MapVR demonstrates a superior ability in constructing more accurate maps, particularly for complex map elements and intricate details.
Results on Argoverse 2. Table presents the performance compari- son on the Argoverse2 dataset [49]. Note that in our setup, the height information for map elements is ig- nored. Our proposed MapVR method still achieves otate-of-the-art perfor- mance, which verifies its robustness across multiple scenarios.
Table 3: Comparison of various map vectorization methods Argoverse2 validation set.
AP Chamier div
AP Frater
Method
bdry
avg.
ped
div
bdry org.
HDMapNet [16]
13.1 5.7
37.6 18.8
VectorMapNet [26]
38.3
36.1
39.2
37.9
MapTR [20]
54.7
58.1
56.5
22.1
24.0 26.2 36.5
32.6
MapTR [20] (Ours)
54.6
60.0
58.0
57.5
23.5
30.2 30.1
''ped", 'div", and 'bdry" denote pedestrian crossing, divider, and boundary, respectively.
Results on 6V-mini-0..4 Finally, test MR 6V-mini-v0.4. our proprictary commercial dataset that features highly intricate real-world driving scenes. As Table 4 shows, MR greatly enhances the perfor- mance on all map elements, which validates its efficacy and robustness in complex and real-world contexts.
Table 4: Map vectorization performance on 6V-mini-v0.4 dataset (our proprietary commercial dataset).
AP puster
Method
stopline intersection
MapTR [20]
41.3
32.9
MapTR [20] MR (Ours) 50.8
37.3
11.8
14.3
44.0
Visualizations. Fig. 6 visualizes the results of HD map vectorization and compares our method with MapTR [20]. For fair comparison, both methods use the ResNet-50 [11] backbone and solely rely on multi-view camera images input. It can be observed that our method yields more accurate HD maps, particularly in capturing intricate details and accurately representing complex or eurved map elements. Conversely, while MapTR [20] produces generally correct vectorized maps, it inevitably exhibits deviations in finer details and struggles to precisely construct complex map elements. These observations reaffirm our motivation to incorporate the precise supervision from HD rasterization into map vectorization, which compensates for the inherent limitations caused by the sparse, equidistant point sets, thereby enhancing the precision of map vectorization.

---

Table 5: MapVR's ablation experiments on nuScenes Map (basic) validation set. All models employ ResNet-50 as backbones and trained for 24 epochs. MapVR's default setups are marked in gray
(a) Rasterization resolution. 'x' denotes rasterization.
(b) Line rasterization softness
resolution
64x32
128x64
180x90
256x 128
320x
line softness
0.5
1.0
2.0
4.0
6.0
m. Praster
24,3
21.5
29.8
30.4
31.2
30.9
mAP, "esteridiiides?
29.2
31.7
33.1
32.5
31.4
CCaamfrr
50.3
45.1
50.6
51.2
50.9
mA Chamfer(ddiviee)
48.0
50.3
54.4
53.3
52.8
(c) Regularization on point direction.
(d) Rasterization geometry
ecometry-wwareness. awareness.
(e) MapVR vs. parallel segm.
regularization
None
MapVR
parallel segm
GT
self
lines
polygons
mAPrrsee
29.5
29.3
31.2
mAP rastenped sing)
21.8
37.5
m.P PrAPee
31.2
26.7
mAPcttanter
48.5
51.2
mAP Chanterrpest sing)
34.9
47.7
Chammier
51.2
48.1
mAPchusfe added upon reviewers' kind suggestions, Entries marked with are unav-ailable due accidentally deleted checkpoints.
5.3 Ablation Study
Rasterization & Rasterization Resolution. As shown in Table 5a, incorporating rasterization enhances performance, following general trend where higher resolutions yield bet- ter results. However, an exception is observed at the 64x32 resolution, which degrades the performance due the lack of precise rasterization supervision at such coarse reso- lution. Fig. further presents the Precision- Recall curves, showing that MapVR leads to consistent performance gain under both metrics and, notably, a smaller gap under the two metrics. Conversely, the baseline exhibits large drop under the stricter raster. This proves the necessity of incorporating fine-grained supervision from rasterization.
0.8
Preciaion
0.6
0.4
Map [APoantee)
MapTR (APa
0.2
MapTR-MapVR (AP APPanneee
MapTR- MapVR (AP
0.00
0.05
Recall
Figure 7: Comparison of P-R curves. MapVR narrows the performance gap under the coarse and strict metrics.
Rasterization Softness т. 7 is a tricky aperrparameter. needs to be large enough to provide sufficient supervisory gradient while being small enough ensure precise supervision. Empirically, polygon-shapee map elements are robust against various т. while line-shaped elements are not, due to their thin and elongated shapes. Table 5b studies the effect of different for line, taking the 'divider' class (a line-shaped map element) as an example.
Auxiliary Regularization on Point Direction. Table 5c studies the effect of the direction regular- ization loss described in Eq. 4, and also compares it with MapTR's directional loss [20], which uses the directions of ground truth equidistant points as targets. Results show that our direction regulariza- tion loss (w/ self) improves performance, proving its effectiveness in regularizing vectorized smooth point sets and allocating more points higher curvature areas to improve precision. Conversely, the performance of our MapVR slightly degrades when using Map'RR's 's regularization (w/ GT). This is because our MapVR's supervision from rasterization longer requires the vectorized outputs be equidistant.
Geometry-Aware Rasterization. Table 5d shows that simply rendering all map elements into lines severely impairs performance. The performance drop mainly comes from polygon-shaped elements beed_crossing: 37.5% +21.8%). This verifies the necessity of geometry awareness in rasterization.
Why Not Introduce HD Supervisory Signals from an Auxiliary Segmentation Task? A simple alternative incorporate fine- mne-grained supervision from rasterization append an additional parallel segmentation branch
auxiliary task (dubbed as 'parallel segm'). This has been verified effective in many works [8, 4, 1]. Table 5e compares Map with this strategy. While 'paraliel segm' improves baseline performance by 2.4%, it still largely lags behind our VR The improved performance from 'parallel segm' supports our motivation to enhance map vectorization via rasterization. However, attribute its inferior performance compared to Map VR to the fact that, unlike in MapVR, the fine-grained supervisory signal is not directly applied to the vectorized output.
5.4 Computational Overhead During Training
With the CUDA-accelerane differenniablee rasterizer, proposed Map only brings marginal

---

Table 6: MapVR's computational overhead during the training stage. Results were obtained with 8x
NVIDIA A100 GPUs under the same training setups.
Method
Modality
Backhone
Training Time Iter
GPU Memory Usage
MapTR [20]
C
Res-50
0.82
14021 MB
MapTR [20]
C&T
Res-50
1.18
28557 MB
MapTR [20] MapVR (Ours)
C
Res-50
0.91
14169
MapTR [20] MR (Ours)
C
Res-50
1.37
28673 MB
In moddllyy donotes mudii-vies camera ispua
'C& combined mubi-view camera LIDAR input.
increase in memory footprint while maintaining training efficiency. Table 6 presents a detailed comparison between the training costs of our method, 'MapTR + MapVR', and its baseline, 'MapTR'.
5.5 Failure Case Analysis
While the proposed method greatly improves the quality of HD map vectorization, the nu- merical results suggest that the results are still far from per- fect. We provide visualization of few typical failure cases in Fig.8. 8.
From row 2. and 4 in Fig. 8, can be seen that occlusions, whether from vehicles, con- structions, or a limited field of view, hamper perception in the bird's-eye-view. Such occlu- sions often result in inaccura- cies the predicted vectorized maps. Yet, since road struc- tures typically follow regular patterns, current map vector- ization techniques may bene- fit from integrating road struc- ture priors, such as standard- definition maps (SDMap), to enhance their reasoning capa- bilities.
Row in Fig. shows that there is still room for improvement in nighttime driving.
Row 3 in Fig. 8 is caused by
DIIMS
ambiguity in annotation, where it is unclear whether the mid- dle crosswalk should connect to the adjacent ones or not.
Figure 8: Visualization of failure cases produced by our method.
6 Conclusion
In this paper, introduce new perspective on map vectorization: rasterization, through which we can learn and evaluate map vectorization more precisely. We demonstrate that, while vectorized representation is compact and easy use, it lacks representation capability, especially regarding fine- line-grained details: thus, it is necessary to incorporate rasterization as complement in both learning and evaluation. We hope our perspective can serve the comnersoon and spur further innovation map vectorization, and can eventually lead to safe and reliable autonomous driving.
10

---

Acknowledgement
The authors would like to thank the anonymous reviewers for their helpful suggestions.
References
[1] Yifeng Bai, Zhirong Chen, Zhangjie Fu, Lang Peng, Pengpeng Liang. and Erkang Cheng. CurveFormer: 3D lane detection by curve propagation with curve queries and attention. ICRA, 2023.
Holger Caesar, Varun Bankiti, Alex H Lang. Sourahh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset autonomous driving. In CVPR, 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with Transformers. In ECCV, 2020.
[4] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng. Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, and Junchi Yan, PersFormer: 3D lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022.
[5] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3D objects with an interpolation-bused differentiable renderer. In NeurlPS, 2019.
[6] Hao Dong, Xianjing Zhang, Jintao Xu, Rui Ai, Weihao Gu, Huimin Lu, Juho Kannala, and Xicyuanli Chen. SuperFusion: Multilevel LiDAR-Camera Fusion for Long-Range HD Map Generation. arXiv preprint arXiv:2211.15056. 2022.
[7] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88(2):003-338 2010.
[8] Zhengyang Feng, Shaohua Guo, Xin Tan, Ke Xu, Min Wang, and Lizhuang Ma. Rethinking efficient lane detection via curve modeling. In CVPR, 2022.
[9] Nader Gharachorloo, Satish Gupta, Robert F Sproull, and Ivan E Sutherland. A characterization of ten rasterization techniques. In SIGGRAPH, 1989.
[10] Yuliang Guo, Guang Chen, Peitao Zhao, Weide Zhang, Jinghao Miao, Jingao Wang. and Tae Eun Choe. Gen-LaneNet: A generalized and scalable approach for 3d lane detection. In ECCV, 2020.
[11] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[12] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: future instance prediction in bird's-eye view from surround monocular cameras. In ICCV, 2021.
[13] Shaofei Huang, Zhenwei Shen, Zehao Huang, Zi-han Ding, Jiao Dai, Jizhong Han, Naiyan Wang, and Si Liu. Anchor3DL Lane: Learning regress 3D anchors for monocular 3D lane detection. In CVPR, 2023.
[14] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. PointPillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[15] Justin Lazarow, Weijian Xu, and Zhuowen Tu. Instance segmentation with mask-supervised polygonal boundary transformers. In CVPR, 2022.
[16] Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. HDMapNet: An online HD map construction and evaluation framework. In ICRA, 2022.
[17] Tzu-Mao Li, Michal Lukáë, Michaël Gharbi, and Jonathan Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning. ACM Transactions on Graphics, 39(6):1-1) 15, 2020.
[18] Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-CNN: End-to-end traffic line detection with line proposal unit. IEEE Transactions on Intelligent Transportation Systems, 2 21()):248223 258, 2019.
[19] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: Learning bird's s-eee-iiw representation from multi-camera images via spatiotemporal transformers. In ECCV, 2022.
11

---

[20] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng. Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: Structured modeling and learning for online vectorized HDD map construction. In ICLR, 2023.
[21] Yiyi Liao, Katja Schwarz, Lars Mescheder, and Andreas Geiger. Towards unsupervised learning of generative models for 3D controllable image synthesis. In CVPR, 2020.
[22] Tsung- Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[23] Ruijin Liu, Zejian Yuan, Tie Liu, and Zhiliang Xiong. End-to-end lane shape prediction with transformers. In WACV, 2021.
[24] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for image-based reasoning. In /CCV, 2019.
[25] Shichen Liu, Tianye Li, Weikai Chen, and Hao A general differentiable mesh renderer for image-based 3D reasoning. IEEF Transactions on Pattern Analysis and Machine Intelligence, 44(1):50-62, 2020.
[26] Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao. VectorMapNet: End-to-end vectorized HDD map learning. arXiv preprint wrXiv:2206.089200 2022.
[27] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETR: Position embedding transformation for multi-view 3D object detection. In ECCV, 2022.
[28] Matthew M Loper and Michael Black. OpenDR: An approximate differentiable renderer. In ECCV, 2014.
[29] Zhipeng Luo, Changqing Zhou, Gongjie Zhang, and Shijian Lu. DETR4D: Direct multi-view 3D object detection with sparse attention, arXiv preprint arXiv:2212. 07849, 2022.
[30] Zhipeng Luo, Gongjie Zhang, Changqing Zhou, Tianrui Liu, Shijian Lu, and Liang Pan. TransPillars: Coarse-to-fine aggregation for multi-frame 3D object detection. In WACV, 2023.
[31] Fausto Milletari, Nassir Navab, and Seyed-Ahne -homad Ahmadi. V-Net: Fully convolutional neural networks for volumetric medical image segmentation. In 3DV, 2016.
[32] Raul Mur- Artal and Juan D Tardós. ORB-SLAM2: An open-source SLAM system for monocular, stereo, and RGB-D cameras. IEEE Transactions on Robotics, 3355):155-1126,, 2017.
[33] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Mare Proesmans, and Luc Van Gool. Towards end-to-end lane detection; an instance segmentation approach. In IEEE Intelligent Vehicles Symposium (IV), 2018
[34] Michael Niemeyer, Lars Mescheder, Michael Occhsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
[35] Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial CNN for traffic scene understanding. In AAAI, 2018
[36] Lang Peng. Zhirong Chen, Zhangjie Fu, Pengpeng Liang, and Erkang Cheng. BEVSegFormer: Bird's eye view semantic segmentation from arbitrary camera rigs. In WACV, 2023.
[37] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
[38] Juan Pineda. parallel algorithm for polygon rasterization. SIGGRAPH. 1988.
[39] Zequn Qin, Huanyu Wang, and Xi Li. Ultra fast structure-wware deep lane detection. In ECCV, 2020.
[40] Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. In CVPR, 2020.
[41] Tixiao Shan and Brendan Englot. LeGO-LOAM: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. In IROS, 2018
[42] Tixiao Shan, Brendan Englot, Drew Meyers, Wei Wang, Carlo Ratti, and Daniela Rus. LAO- Tightly-couped lidar inertial odometry via smoothing and mapping. In IROS, 2020.
12

---

[43] Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, and Thiago Oliveira-Santos. Keep your eyes the lane: Real-time atlention-guided lane detection. In CVPR, 2021.
[44] Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, and Thiago Oliveira-Santos. PolyLaneNet: Lane estimation via deep polynomial regression. In ICPR, 2021.
[45] Xingze Tian and Tobias Günther. A survey of smooth vector graphics: Recent advances in representation, creation, rasterization and image vectorization. IEEE Transactions on Visualization and Computer Graphics, 2022. doi: 10.1 N0.1109/VVG.2222..22222
[46] Wouter Van Gansbeke, Bert De Brabandere, Davy Neven, Mare Proesmans, and Lue Van Gool. End-to-end lane detection through differentiable least- squtre fitting. ICCV Workshops, 2019.
[47] Ruihao Wang, Jianbang Qin, Kai Li, Yaochen Li, Dongping Cao, and Jintao Xu. BEV-LaneDet: simple and effective 3d lane detection baseline. CVPR, 2023.
[48] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3D object detection from multi-view images via 3D-1o-2D queries. In CoRL., 2022.
[49] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan. Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, Deva Ramanan. Peter Cam, and James Hays. Argoverse 2: Nexi generation datasets for self-driving perception and forecasting. In NeurlPS (Dutasets and Benchmarks Truck), 2021.
[50] Xuan Xiong, Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao. Neural map prior for autonomous driving. In CVPR, 2023.
[51] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. BEVFormer v2: Adapting modern image backbones to bird's s-eye- view recognition via perspective supervision. arXiv preprint arXiv:2211 10439, 2022.
[52] Seungwoo Yoo. Hee Seok Lee, Heesoo Myeong, Sungrack Yun, Hyoungwoo Park, Janghoon Cho, and Duck Hoon Kim. End-to-end lane marker detection via row-wise classification. In CVPR Workshops, 2020.
[53] Gongjie Zhang, Shijian Lu, and Wei Zhang. CAD-Net: A context-aware detection network for objects in remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing, 57(12):10015 10024, 2019.
[54] Gongjie Zhang, Kaiwen Cui, Rongliang Wu. Shijian Lu, and Yonghong Tian. PNPDet: Efficient few-shot detection without forgetting via plug-and-play sub-networks. In WACV, 2021.
[55] Gongjie Zhang, Zhipeng Luo. Yingchen Yu, Kaiwen Cui, and Shijian Lu, Accelerating DETR convergence via semantic-aligned matching. In CVPR, 2022.
[56] Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, Shijian Lu, and Eric P. Xing. Meta-DETR: Image-level few-shot detection with inter-class correlation exploitation. IEEE Transactions on Pattern Analysis and Machine Intelligence, (6(11):12832 12843, 2023. doi: 10. 2000/TTAM.2202..31999999
[57] Gongjie Zhang, Zhipeng Luo, Zichen Tian, Jingyi Zhang, Xiaoqin Zhang, and Shijian Lu. Towards efficient use of multi-scale fcatures in transformer-basee object detectors. In CVPR, 2023.
[58] Ji Zhang and Sanjiv Singh. LOAM: LiDAR odometry and mapping in real-time. In Robotics: Science and Systems, 2014.
[59] Yumpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang. Guan Huang, Jie Zhou, and Jiwen Lu. BEVerse: Unified perception and prediction in sirds-eye-view for dision-centric autonomous driving. arXiv preprint arXiv:2205.09744 2022.
[60] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmenta- tion. In CVPR. 2022.
[61] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: Deformable transformers for end-to-ent object detection. In ICLR, 2021.
13


Reviews:
[{"review_id": "zqSzOyRAab", "reviewer": "Reviewer_ddVN", "summary": "The authors explore and analyze the existing HD map framework and its evaluation metrics, and propose to adopt rasterization on top of the existing vectorization-based HD map for more precision. Also, they propose a rasterization-based evaluation metric rather than the existing chamfer-distance one. The experimental results on a number of datasets are better compared to the existing methods.", "questions": "Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.", "limitations": "NA.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The task of HD map is very important in the 3D autonomous driving community. Interestingly, the authors propose to adopt rasterization on top of the existing vectorization-based HD map for more precision.\n\n2. The paper is easy to follow.\n\n3. The authors conduct the experiments on differnt outdoor datasets, including widely-used nuScenes, Argoverse2 and their own 6V-mini-v0.4.", "weaknesses": "1. Computation/memory footprint comparison. The authors didn't make a comparison of their work in terms of memory/speed with the existing 3D HD map methods. The time consumption might be not trivial since the memory/time might be heavy for the customized differentiable rasterizer from my point. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n&nbsp;\n\nThank you for acknowledging the significance, presentation, and thorough evaluation of our work. We also appreciate your kind suggestion on computation/memory footprint comparison.\n\nWe hope the information provided below will help address your concerns. And we will include them in our final version.\n\n&nbsp;\n\n&nbsp;\n\n## Computation/Memory Footprint (Evaluation Metric)\n\nOur empirical tests indicate that our rasterization-based evaluation metric $AP_\\text{raster}$, though more complex, runs at similar speeds to the Chamfer-distance-based metric $AP_\\text{chamfer}$. Specifically, our metric takes about 3 minutes to evaluate on the nuScenes Map validation set, while the Chamfer-distance-based metric takes around 2 minutes. Notably, our proposed evaluation metric can be further accelerated with multi-threading. \n\nBoth evaluation metrics are performed on CPU sequentially, therefore their memory consumption is negligible.\n\n&nbsp;\n\n&nbsp;\n\n## Computation/Memory Footprint (Neural-Network Model)\n\n\n### Training Stage\n\nWe concur with the reviewer that the runtime/memory consumption will be expensive if the customized differentiable rasterizer is implemented in pure PyTorch/TensorFlow. Given this, we implement the differentiable rasterizer (both forward and backward propagation) with CUDA. Please refer to our code implementation in the supplementary materials for details. Upon acceptance, we intend to release all our code implementations. This will allow our work to serve as a foundation for future research and development.\n\nWith the CUDA-accelerated differentiable rasterizer, our proposed MapVR ensures a marginal increase in memory footprint while maintaining computational efficiency during the training process. The following table presents a detailed comparison between the training costs of our method, `MapTR+MapVR`, and its baseline, `MapTR`. Experiments were conducted with 8x NVIDIA A100 GPUs under the training setups specified in our manuscript.\n\n&nbsp;\n\n| Methods              | Modality | Backbone       | Training Time / Iter | GPU Memory Usage |\n|:---------------------|:--------:|:--------------:|:--------------------:|:----------------:|\n| MapTR                | C        | Res-50         | 0.82 s               | 14021 MB         |\n| MapTR                | C & L    | Res-50         | 1.18 s               | 28557 MB         |\n| MapTR + MapVR (Ours) | C        | Res-50         | 0.91 s               | 14169 MB         |\n| MapTR + MapVR (Ours) | C & L    | Res-50         | 1.37 s               | 28673 MB         |\n\n&nbsp;\n### Inference Stage\n\nThe differentiable rasterization is not needed anymore once the training is complete. Hence, our method can improve map vectorization with no additional computation or memory usage during inference.\n\nIn Table 1 and Table 2 of our manuscript, we have compared the inference speed of different methods for map vectorization. The table below is an updated version of Table 1, which provides more details on inference time, memory consumption, etc. Results are obtained with 1x NVIDIA 3090 GPU.\n\n&nbsp;\n\n| Method             | Modality | Backbone | #Epochs | AP_chamfer_avg | AP_raster_avg | FPS  | GPU Memory |\n|:-------------------|:--------:|:--------:|:-------:|:--------------:|:-------------:|:----:|:----------:|\n| HDMapNet           | C        | Effi-B0  | 30      | 23.0           | -             | 0.8  | 3264 MB    |\n| HDMapNet           | C & L    | Effi-B0  | 30      | 31.0           | -             | 0.5  | -          |\n| VectorMapNet       | C        | Res-50   | 110     | 40.9           | 15.0          | 2.9  | 3232 MB    |\n| VectorMapNet       | C & L    | Res-50   | 110     | 45.2           | -             | -    | -          |\n| MapTR              | C        | Res-50   | 24      | 50.3           | 24.3          | 18.4 | 3115 MB    |\n| MapTR              | C        | Res-50   | 110     | 58.7           | 35.0          | 18.4 | 3115 MB    |\n| MapTR              | C & L    | Res-50   | 24      | 62.7           | 45.2          | 7.2  | 16760 MB   |\n| MapTR+MapVR  (Ours)| C        | Res-50   | 24      | 51.2           | 31.2          | 18.4 | 3115 MB    |\n| MapTR+MapVR  (Ours)| C        | Res-50   | 110     | 58.8           | 38.5          | 18.4 | 3115 MB    |\n| MapTR+MapVR  (Ours)| C & L    | Res-50   | 24      | 63.5           | 51.1          | 7.2  | 16760 MB   |\n\nNote: Entries marked with '-' indicate that official results are not reported, and codes or model checkpoints are unavailable.\n\n\n&nbsp;\n\n&nbsp;\n\nWe hope that our response has addressed your concern. And we welcome further dialogue to discuss any concerns and clear up any doubts that may still exist.\n\n&nbsp;\n"}, {"review_id": "C2FbK9dUoe", "reviewer": "Reviewer_yTjh", "summary": "--", "questions": "--", "limitations": "--", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "--", "weaknesses": "--", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2XGmOX68yN", "reviewer": "Reviewer_YAhf", "summary": "Online HD mapping is essential for autonomous driving because maps provide detailed and precise environmental information for perception and planning. Existing mapping literatures have limitations both in methods and metrics. To address them, the authors a new method called MapVR, integrating the philosophy of rasterization into map vectorization and a new metric, a rasterization-based evaluation metric. Experiments are conducted on four existing datasets (nuScenes Map (basic), nuScenes Map (extended), Argoverse2, and 6V-mini-v0.4) and show that incorporating rasterization into map vectorization greatly enhances performance with no extra computational cost during inference. ", "questions": "Please see the weakness section.", "limitations": "No. The authors do not provide adequate discussions on the limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Writing: Overall the paper is well-written and is easy to understand the goal of the paper.\n\n2. Motivation: Particularly, the authors provide a very introduction that explains the limitations of existing mapping algorithms and metrics. The work is highly valuable to the field of HD mapping.\n\n3. Proposed method: To address the limitations of existing mapping algorithms, the authors propose to combine the two camps (i.e., map rasterization and map vectorization) in a unified framework. In particular,  the proposed MapVR applies differentiable rasterization, inspired by recent advances in graphics and vision, to the map vectorization task to bridge vectorized outputs and rasterized HD maps. The proposed strategy enables more refined and comprehensive supervision and yields predictions with improved precision. The authors offer a new perspective on HD mapping. Moreover, the work could motivate relevant fields in autonomous vehicle research.\n\n4. Metric: the authors deliver a clear explanation of the proposed metric in Figures 3 and 4. Particularly, Figure 4 can immediately spot the effect of two different metrics. Importantly, the proposed metric can capture fine-grained differences for quality evaluation.\n\n5. Experimental results: the authors conduct experiments on multiple datasets, i.e., nuScenes Map (basic), nuScenes Map (extended), Argoverse2, and 6V-mini-v0.4. In Table 1-4, MapVR clearly demonstrates its superiority over existing baselines (i.e., HDMapNet, VectorMapNet, MapTR). The tables provide a clear benchmarking protocol for the community. It is worth noting that MapVR can enhance map vectorization without adding any extra computational cost during inference.\n\n6. Ablation study: the reviewer particularly finds the “1. **Why Not Introduce HD Supervisory Signals from an Auxiliary Segmentation Task?**” important. It shows that a simple auxiliary task can benefit rasterization prediction. The results are valuable for future research.", "weaknesses": "Overall, the paper is a really good shape. However, the reviewer has the following comments.\n\n1. Differentiable rasterization: In sec. 4.2, the authors discussed two rendered masks for line-shaped and polygon-shaped elements. Can the two rendered masks cover all elements in HD maps? On a relevant note, do the authors identify failure cases due to the conversion?\n\n2. Rasterization resolution: what is the implication of having 256x128 the best performance? Is it related to the HD map size? Do we have a systematic way to identify the proper rasterization resolution? \n\n3. While the proposed method achieves the best quality, the numerical results are still far from perfect. The reviewer strongly suggests the authors conduct a thorough failure case analysis that would shed light on future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "&nbsp;\n\nWe are immensely grateful for the comprehensive and positive acknowledgment received. And we deeply appreciate your insightful comments and suggestions. Below we would like to respond to your queries and address your concerns point-by-point.\n\n&nbsp;\n\n&nbsp;\n### Discussions on Failure Cases and Limitations\n\nWe greatly appreciate your constructive feedback. If our work is accepted for NeurIPS 2023, we will incorporate discussions on failure cases and limitations within the one additional page allowed in the camera-ready version.\n\n**_Please refer to `our global response to all reviewers` for (i) additional visualization and analysis on failure cases and (ii) additional discussions on limitations._**\n\n&nbsp;\n### Regarding Line-Shaped and Polygon-Shaped Differentiable Rasterization\n\nOur empirical findings indicate that the two proposed rasterization strategies can effectively cover all elements in HD maps.\n\nTo the best of our knowledge, the majority of HD map elements can be abstracted into either one of three primal shapes: points, lines, and polygons. As thoroughly explained in our manuscript, the most common HD map elements, such as lanes, curbsides, stoplines, crosswalks, intersections, and parking areas, can be categorized as lines and polygons, which can be well covered by the two designed rasterization strategies.\nOther elements like traffic lights, traffic signs, and cones are ideally abstracted as points within HD maps. These elements can be interpreted as small, circular-shaped polygons, thus integrating them seamlessly into our designed rasterization strategies. This strategy allows us to integrate our proposed method in an 'all-in-one' vectorized map perception framework without any complex network design.\n\nIt should be noted that the differentiable rasterization procedure is accomplished intuitively via Eq. 2 and Eq. 3 in our manuscript, rather than learned parametrically. We did not notice any error caused by the differentiable rasterizer.\n\n\n\n&nbsp;\n### Regarding Rasterization Resolution\n\nIn our understanding, this hyper-parameter is related to the perception range as well as the perception precision. \n\nTaking our nuScenes experiments as an example. The perception range is ±30m along the y-axis (vertically) and ±15m along the x-axis (horizontally), yielding a bird's-eye-view (BEV) perception range of 60m x 30m. With a rasterization resolution of 256x128 pixels, each rasterized pixel corresponds to a real-world area of 0.23m x 0.23m. This precision is generally considered sufficient for accurate fine-grained localization supervision.\n\nTo determine the appropriate rasterization resolution, it is crucial to proportionally align the rasterization resolution with the BEV perception range. For example, if we keep the per-pixel size unchanged, a perception range of 120m x 45m would correspond to a rasterization resolution of 480x180. In addition, fine-tuning the rasterization resolution within a small neighboring range should help for better performance.\n\nIt is also noteworthy that, as per Table 5a in our manuscript, the performance is robust against a certain range of rasterization resolutions (180x90 ~ 320x160).\n\n&nbsp;\n\n&nbsp;\n\nWe hope the above response addresses your concerns. Once again, we thank you for your insightful comments and strong recognition of our work, and we are more than happy to have further discussions with you.\n\n&nbsp;\n"}, {"review_id": "Y7tasNFcHF", "reviewer": "Reviewer_pGkQ", "summary": "This work targets online map creation for autonomous driving.  \nThe authors claim vector-based approaches exhibit artifacts due to lack of geometric supervision in current loss functions.  \nThe main idea is to add a differentiable rasterization layer to any model that predicts vectors and add an additional segmentation loss (dice loss) during training.  \nTheir method does not require any additional model parameters, and can be added to any existing vector-based model.  \nThey perform experiments on the nuScenes dataset using camera only, and camera + lidar,  \nand show their additional supervision can improve performance of the current state-of-the-art vector-based model. ", "questions": "* Table 1: It would be nice to have AP-raster numbers for HDMapNet and VectorMapNet. In the current draft these two methods do not provide much information as of now.\n* Should all future vector models be evaluated and assessed with AP-chamfer and AP-raster metrics? Is the proposed AP-raster superior?\n* Figure 2: the authors argue that equidistant parameterization causes inaccuracy in modeling map primitives. Couldn't the same be said for resolution of rasterization? Why not simply increase the number of points?", "limitations": "No limitations/societal impact section provided.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Simple idea, easy to execute and shows modest improvements for MapTR with camera+lidar.\n* The proposed method can be applied to any vector-based architecture.\n* Technical section described clearly, notation looks sound.\n* Convincing analysis and visual evidence for the need of the rasterization-based metric (Figure 4.)", "weaknesses": "* I am borderline on if the current contribution is significant enough for a paper. The results are sound, particularly for AP-raster since the baselines do not optimize for this. However, the story and writing needs more focus on the AP-raster metric if that is the real contribution (i.e. should all future map models focus on this particular metric?).\n* The main ideas presented are architecture agnostic but experiments are only done using MapTR.  \n* I found the direction regularization loss to be unrelated to the main idea of rasterization. How does the baseline (MapTR) perform with this extra loss, or how does MapVR perform without this? A majority of the main results in Table 1 are very close - this is my primary factor for my current rating.\n* I was hoping ablations included evaluation using both metrics (AP-raster+AP-chamfer). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n&nbsp;\n\nThank you for taking the time to review our manuscript. Your comments have provided us with valuable insights to improve our work. We have carefully considered each of your points, and we hope that our responses below could address your concerns.\n\n&nbsp;\n### Significance of $\\text{AP}_\\text{raster}$\n\nShort answer: **We believe $\\text{AP}_\\text{raster}$ is of great significance to the autonomous driving community, and we highly recommend future works to use** $\\text{AP}_\\text{raster}$ **for map vectorization evaluation.**\n\nActually, in our team, we already transitioned to $\\text{AP}_\\text{raster}$. And we believe it is also valuable for the emerging community, as it guides subsequent works toward the actual needs of autonomous driving.\n\nTo highlight its significance, we have dedicated the whole of Section 3 to introducing this evaluation metric. And we will amend the narratives to make our points clearer.\n\nWe further explain the significance of $\\text{AP}_\\text{raster}$ as follows.\n- First, as described in Section 3.1, the existing AP_chamfer metric is not scale-invariant and overlooks shape and geometrical details. Fig. 4 illustrates these issues with a few practical examples, where AP_chamfer fails to reasonably judge the prediction quality while AP_raster does clearly a better job.\n- Second, from the visualizations in Fig. 6, it can be observed that our proposed rasterization-based approach yields substantially better results, although it does not outperform MapTR by a large margin in terms of AP_chamfer. This also indicates that AP_chamfer falls short of providing a comprehensive and precise evaluation as compared to AP_raster.\n- Third, while the judgment of significance can be subjective, `Reviewers YAhf and U7tP` strongly support us on the significance of $\\text{AP}\\text{raster}$. Their comments reflect the value of this new metric to the community to some extent. Nevertheless, we fully appreciate your concern and welcome further conversations.\n\n\n&nbsp;\n### Compatibility of MapVR\n\nWe agree we should have integrated MapVR with more baselines. However, as map vectorization is an emerging field, existing studies are limited. \n\nBy the time of submission (May 2023), VectorMapNet and MapTR are the only two peer-reviewed works that directly use networks to predict the vectorized map elements. Since MapTR outperforms VectorMapNet significantly in both accuracy and efficiency, we selected MapTR as the baseline.\n\nBesides, as our work does not involve any assumption or modification in network architecture, it should be generalizable to other future map vectorization solutions.\n\n\n&nbsp;\n### Why Not Increase the Number of Equidistant Points to Avoid Parameterization Error?\n\nThe attempt was made in MapTR, but resulted in a decline in performance. You can find more information about this in Table 5 of MapTR's appendix [20]. This issue has also been mentioned in the footnote of Page 2 in our manuscript. \n\n\n&nbsp;\n### Regarding Direction Regularization Loss\n\nWe wish to clarify that **MapTR's official results were obtained with a similar direction loss**. Therefore the comparisons in Tables 1-3 are fair.\n\nMapTR's direction loss is computed between predicted points and ground-truth equidistant points; while our MapVR's direction loss is self-regularized along predicted adjacent segments. With the supervision via differentiable rasterization, MapVR largely diminishes the need for equidistant predicted points, which was a constraint in MapTR. As a result, our designed direction loss aids in allocating more points in areas with greater curvature and fewer points in straight-line regions, ultimately enhancing the precision.\n\nIn conclusion, the direction regularization loss is not unrelated to the rasterization idea, but serves as an important component that works in conjunction with it for better map vectorization.\n\nThe table below presents extra experiments regarding the direction loss.\n\n|  |MapTR   (AP-chamfer/AP-raster)|MapTR+MapVR(Ours)  (AP-chamfer/AP-raster)|\n|--|:--:|:--:|\n|w/o direction loss|48.2 / 23.7| 48.5 / 29.5 |\n|w/ MapTR's direction loss|50.3 / 24.3|---- / ----|\n|w/ our direction loss|---- / ----| 51.2 / 31.2 |\n\n\n&nbsp;\n### Regarding Blanks in Table 1\n\nNeither HDMapNet nor VectorMapNet provides source codes for the \"C & L\" modality. Besides, HDMapNet does not provide any checkpoint for evaluation. Therefore, we did not provide their performance under $\\text{AP}_\\text{raster}$.\n\nFortunately, VectorMapNet provides codes and checkpoint for the camera-only modality. The table below shows the additional results (highlighted in bold), which we will incorporate into future versions. \n\n|Method|Modality|Backbone|#Epochs|AP_chamfer_avg|AP_raster_avg|FPS|\n|--|:--:|:--:|:--:|:--:|:--:|:--:|\n|VectorMapNet|C|Res-50|110|40.9|**15.0**|2.9|\n|MapTR|C| Res-50 |110|58.7|35.0|18.4|\n|MapTR+MapVR (Ours)|C|Res-50|110|58.8|38.5|18.4|\n\nIt can be seen that: for methods that are inferior under $\\text{AP}\\text{chamfer}$, the gap under the stricter metric $\\text{AP}\\text{raster}$ will be even larger.\n\n\n&nbsp;\n### Ablations Under Both Metrics\n\n(a) Rasterization resolution\n\n| resolution | AP-chamfer/AP-raster |\n|--|:--:|\n|X | 50.3 / 24.3 |\n|64x32 | 45.1 / 21.5 |\n|180x90 | 50.6 / 30.4 |\n|256x128 | 51.2 / 31.2 |\n|320x160 | 50.9 / 30.9 |\n\n&nbsp;\n\n(b) Line rasterization softness\n\n| $\\tau$ | [Divider] AP-chamfer/AP-raster |\n|--|:--:|\n|0.5|48.0 / 29.2|\n|2.0|54.4 / 33.1|\n|6.0|52.8 / 31.4|\n\n&nbsp;\n\n(c) Direction loss.\n\n_Please refer to the response above ('Regarding Direction Regularization Loss')._\n\n&nbsp;\n\n(d) Geometry-aware rasterization\n\n| | [Ped_crossing] AP-chamfer/AP-raster |\n|--|:--:|\n|all as lines|34.9 / 21.8|\n|lines and polygons|47.7 / 37.5|\n\n&nbsp;\n\n(e) MapVR vs. parallel segm\n\n| |AP-chamfer/AP-raster|\n|--|:--:|\n|MapVR|51.2 / 31.2|\n|parallel segm|48.1 / 26.7|\n\n&nbsp;\n\n&nbsp;\n\nWe sincerely hope the above address your concerns. Please don't hesitate to continue the discussion if you have further queries.\n&nbsp;"}, {"review_id": "NF2buPZCPZ", "reviewer": "Reviewer_U7tP", "summary": "This paper tackles the emerging task of vision-based online HD mapping. It has 2 main contributions: (1) AP_{raster}, a new metric that is shown to be better suited to evaluate methods in this field, and (2) MapVR, a plug-in rasterization and loss module to any existing vector-based online mapping system. Several intuitive examples demonstrate the value of the new metric. Furthermore, MapVR is extensively validated on three different datasets, where it maintains or improves the existing metrics, while significantly boosting the proposed AP_{raster} metric.", "questions": "1. What is the computational overhead of MapVR during training?\n2. L30-32 and L96-106 have some text repetition. Would it be possible to rephrase the latter, focusing on the differences between different vector-based techniques (e.g. VectorMapNet and MapTR), instead of the motivation for MapVR?\n3. In Tables 1, 2 and 3, would it be possible to highlight the “avg.” columns with a grey background, as in Table 5?\n4. L243 “covering the most complex driving scenes in the real world” is a strong and unsubstantiated claim regarding this proprietary dataset. Please rephrase this.\n\nUpdate:\n\nThank you for the thorough response. I appreciate the effort taken to answer each question. My concerns have been addressed in the rebuttal, and I am maintaining my initial positive rating.", "limitations": "Limitations are not discussed in main text - the main limitation I see is that AP_{raster} may be sensitive to hyper-parameter choices, which could be mentioned in the conclusion.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Both contributions are simple and well-motivated. The empirical evaluation is extensive, covering 4 task settings over 3 datasets. In particular, the design choices of MapVR are carefully ablated. In addition, the presentation and organization of the draft are good. Overall, this paper makes a crucial contribution to a field that is beginning to rapidly expand by identifying limitations in the current metrics employed in the literature and proposing an alternative.", "weaknesses": "No major weaknesses stood out to me. The presentation of the draft could be slightly improved (see “Questions” section). AP_{raster} may be sensitive to hyper-parameter choices, but I believe a supplementary experiment may be sufficient to demonstrate that the chosen hyper-parameters are reasonable, since they are well-motivated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n&nbsp;\n\nWe sincerely thank you for appreciating our work as well as for your thoughtful and constructive comments, which we believe will significantly enhance the presentation of our manuscript.\n\n&nbsp;\n### Sensitivity of  $\\text{AP}_\\text{raster}$ to Hyper-Parameters\n\nWe agree with the reviewer that $\\text{AP}_\\text{raster}$ relies on certain hyper-parameters, more specifically, the rasterization resolution (480x240) and the dilation pixel number (2 pixels). However, we think it does not constitute a severe limitation, but instead offers better evaluation flexibility.\n\n- _First_, the two hyper-parameters have clear practical meanings and are easy to set. In our nuScenes experiments, the bird's-eye-view (BEV) perception range is 60m x 30m. Setting the evaluation rasterization resolution at 480x240 offers a localization precision of 0.125m, satisfying the requirement of autonomous driving. A dilation of 2 pixels can make the evaluation criteria tolerant to a deviation of at most 4 pixels, which equals 0.5m. Users can easily adapt these hyper-parameters to their own needs.\n\n- _Second_, we conducted sensitivity analyses in the table below, which demonstrate that $\\text{AP}_\\text{raster}$ is robust with respect to the two hyper-parameters. Therefore, the performance improvements brought by MapVR under $\\text{AP}\\text{raster}$ are genuine and significant.\n\n|  | MapTR | MapTR+MapVR (Ours) |\n|---|:---:|:--:|\n| AP-raster (320x160, dilation=2) | 34.9 | 41.6 |\n| **AP-raster (480x240, dilation=2, default metric)** | 24.3 | 31.2 |\n| AP-raster (480x240, dilation=1) | 15.1 | 20.0 |\n| AP-raster (640x320, dilation=2) | 18.7 | 24.8 |\n\n- _Finally_, it's worth noting that most computer vision evaluation metrics (e.g., COCO's AP, nuScenes' NDS, etc.) have inherent hyper-parameters, and their choices always involve a trade-off between different considerations. The proposed $\\text{AP}_\\text{raster}$ is not an exception and we believe it does provide a valuable and practical tool for assessing the performance of HD map vectorization methods.\n\n\n&nbsp;\n### Computational Overhead of MapVR During Training\n\nThe extra differentiable rasterizer is implemented with CUDA to maintain training efficiency. The table below summarizes the training overhead.\n\n| Methods              | Modality | Backbone       | Training Time / Iter | GPU Memory Usage |\n|:---------------------|:--------:|:--------------:|:--------------------:|:----------------:|\n| MapTR                | C        | Res-50         | 0.82 s               | 14021 MB         |\n| MapTR                | C & L    | Res-50         | 1.18 s               | 28557 MB         |\n| MapTR + MapVR (Ours) | C        | Res-50         | 0.91 s               | 14169 MB         |\n| MapTR + MapVR (Ours) | C & L    | Res-50         | 1.37 s               | 28673 MB         |\n\n\n&nbsp;\n### On the Presentation of the Manuscript\n\nWe cannot be more grateful for your thoughtful suggestions. We will incorporate the following changes into the camera-ready version of the manuscript or future submissions.\n\n&nbsp;\n\n**To rephrase Line #96-106, focusing on the differences between map vectorization techniques.**\n\nFollowing your advice, we will revise Line #96-106 as below:\n\n>  Rasterization methods [37, 40, 12, 58, 19, 59, 36, 50] generate HD maps via semantic segmentation in BEV, which have good sensitivity to details. However, the lack of vital instance-level information and lane topology limits the utility of rasterized maps in downstream tasks like navigation and planning. On the other hand, map vectorization addresses this limitation by producing vectorized map elements. HDMapNet [16] and SuperFusion [6] employ post-processing to group pixels from rasterized maps into vectorized elements. ~~Moreover, the latest approaches – VectorMapNet [26] and MapTR [20], directly predict map elements as vectorized point sets with neural networks, attaining superior performance.~~ _Moreover, VectorMapNet [26] proposes to directly predict map elements as vectorized point sets in an auto-regressive manner, achieving superior performance. And MapTR [20] - the current state of the art, further proposes a unified permutation-equivalent modeling approach to model the HD map elements, achieving superior accuracy. Furthermore, MapTR [20] achieves real-time efficiency with its one-stage and parallel framework._ However, _despite the recent progresses,_ vectorized maps still often exhibit minor deviations that can be critical in autonomous driving, where safety is of utmost importance. ~~We believe integrating HD rasterization into vectorization can improve precision while retaining vectorized representation.~~\n\n\n&nbsp;\n\n**To highlight the “avg.” columns in Table 1-3 with a grey background.**\n\nSure, we will do so in the updated version.\n\n\n&nbsp;\n\n**To rephrase Line #243.**\n\nThe suggested sentence will be rephrased into:\n> covering ~~the most~~ _very_ complex driving scenes in the real world.\n\n\n&nbsp;\n### Additional Discussions on Limitations\n\nDue to the space limitation, we did not discuss the limitations in the submitted manuscript. In the camera-ready version where one additional content page is allowed, we will incorporate a discussion on the potential limitations.\n\n**_Please refer to `our global response to all reviewers` for additional discussions on potential limitations._**\n\n\n&nbsp;\n\n&nbsp;\n\nWe hope that our response has addressed your concern. And we welcome further dialogue to discuss any concerns and clear up any doubts that may still exist.\n\n&nbsp;\n"}]

Author rebuttal:
&nbsp;

We are incredibly grateful for all the helpful comments received.

Here, we provide some additional contents that cannot fit into the nine-page manuscript. These additional contents will be included in the camera-ready version, in which one additional page of content is allowed.

&nbsp;

&nbsp;

## Potential Limitations

We found the potential limitations of our work primarily regarding the reliance on certain hyper-parameters.

&nbsp;

### Sensitivity of $\text{AP}_\text{raster}$ to Hyper-Parameters

Our designed $\text{AP}_\text{raster}$ relies on certain hyper-parameters, more specifically, the rasterization resolution (480x240) and the dilation pixel number (2 pixels). However, we think it does not constitute a severe limitation, but instead offers better evaluation flexibility.

- _First_, the two hyper-parameters have clear practical meanings and are easy to set. In our nuScenes experiments, the bird's-eye-view (BEV) perception range is 60m x 30m. Setting the evaluation rasterization resolution at 480x240 offers a localization precision of 0.125m, satisfying the requirement of autonomous driving. A dilation of 2 pixels can make the evaluation criteria tolerant to a deviation of at most 4 pixels, which equals 0.5m. Users can easily adapt these hyper-parameters to their own needs.

- _Second_, we conducted sensitivity analyses in the table below, which demonstrate that $\text{AP}_\text{raster}$ is robust with respect to the two hyper-parameters. Therefore, the performance improvements brought by MapVR under $\text{AP}\text{raster}$ are genuine and significant.

|  | MapTR | MapTR+MapVR (Ours) |
|---|:---:|:--:|
| AP-raster (320x160, dilation=2) | 34.9 | 41.6 |
| **AP-raster (480x240, dilation=2, default metric)** | 24.3 | 31.2 |
| AP-raster (480x240, dilation=1) | 15.1 | 20.0 |
| AP-raster (640x320, dilation=2) | 18.7 | 24.8 |

- _Finally_, it's worth noting that most computer vision evaluation metrics (e.g., COCO's AP, nuScenes' NDS, $\text{AP}\text{chamfer}$, etc.) have inherent hyper-parameters, and their choices always involve a trade-off between different considerations. The proposed $\text{AP}_\text{raster}$ is not an exception and we believe it does provide a valuable and practical tool for assessing the performance of HD map vectorization methods.


&nbsp;

### Sensitivity of MapVR to Hyper-Parameters

MapVR also introduces hyper-parameters that could affect performance. The major concern is on the rasterization resolution.

Similarly, this hyper-parameter is most related to the perception range as well as the perception precision.

Taking our nuScenes experiments as an example. The perception range is ±30m along the y-axis (vertically) and ±15m along the x-axis (horizontally), yielding a bird's-eye-view (BEV) perception range of 60m x 30m. With a rasterization resolution of 256x128 pixels, each rasterized pixel corresponds to a real-world area of 0.23m x 0.23m. This precision is generally considered sufficient for accurate fine-grained localization supervision.

It should be quite simple to customize the hyper-parameters under a particular setup. To determine the appropriate rasterization resolution, it is crucial to proportionally align the rasterization resolution with the BEV perception range. For example, if we keep the per-pixel size unchanged, a perception range of 120m x 45m would correspond to a rasterization resolution of 480x180. In addition, fine-tuning the rasterization resolution within a small neighboring range should help for better performance.

It is also noteworthy that, as per Table 5a in our manuscript, the performance is robust against a certain range of rasterization resolutions (180x90 ~ 320x160).




&nbsp;

&nbsp;

## Failure Case Analysis

Based on Fig. 6 in our manuscript as well as the visualization results in the appendix, it is evident that our method can perceive pretty accurately when the road structure is highly regularized and when there is no occlusion.

**In the `pdf` file attached in this 'global rebuttal' message, we present a few failure cases for analysis.**

These cases reveal challenges predominantly at complex road intersections. Here, occlusions, whether from vehicles, constructions, or a limited field of view, hamper our system's perception in the bird's-eye-biew (BEV). Such occlusions often result in inaccuracies in the predicted vectorized maps. Yet, since road structures typically follow regular patterns, there's an opportunity for improvement. Current map vectorization techniques may benefit from integrating road structure priors and enhancing their reasoning capabilities. Future research could explore merging perception frameworks with road prior information or leveraging the knowledge from the standard definition map (SDMap) to address these challenges. Moreover, enhancing map perception during nighttime remains an exciting direction for upcoming works.







&nbsp;


Meta-review: Existing online mapping works have limitations both in methods and metrics. To address them, this paper proposes a new method called MapVR, integrating the idea of rasterization into map vectorization and a new rasterization-based evaluation metric. It receives one borderline accept, two borderline rejects, one strong accept and one accept. Reviewer yTjh fails to provide detailed comments thus the rating is ignored by the AC. Authors provided detailed response and failure case analysis during the rebuttal. AC recommends acceptance for this paper.
Decision: Accept (poster)