SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison of various map vectorization methods on nuScenes Map (basic) validation set.
Method	Modality	Backbone	#Epochs	APChamfer	APraster	FPS
ped	div	bdry	avg.	ped	div	bdry	avg.
HDMapNet [16]	C	Effi-B0	30	14.4	21.7	33.0	23.0	-	-	-	-	0.8
HDMapNet [16]	C & L	Effi-B0	30	16.3	29.6	46.7	31.0	-	-	-	-	0.5
VectorMapNet [26]	C	Res-50	110	36.1	47.3	39.3	40.9	26.2	12.7	6.1	15.0	2.9
VectorMapNet [26]	C & L	Res-50	110	37.6	50.5	47.5	45.2	-	-	-	-	-
MapTR [20]	C	Res-50	24	46.3	51.5	53.1	50.3	32.4	23.5	17.1	24.3	18.4
MapTR [20]	C	Res-50	110	56.2	59.8	60.1	58.7	43.6	35.6	25.8	35.0	18.4
MapTR [20]	C & L	Res-50	24	56.4	61.8	70.1	62.7	46.4	39.2	50.0	45.2	7.2
MapTR [20] + MapVR (Ours)	C	Res-50	24	47.7	54.4	51.4	51.2	37.5	33.1	23.0	31.2	18.4
MapTR [20] + MapVR (Ours)	C	Res-50	110	55.0	61.8	59.4	58.8	46.0	39.7	29.9	38.5	18.4
MapTR [20] + MapVR (Ours)	C & L	Res-50	24	60.4	62.7	67.2	63.5	52.4	46.4	54.4	51.1	7.2
- In modality, 'C' denotes multi-view camera input and 'C & L' denotes combined multi-view camera and LiDAR input.
- When LiDAR data is incorporated, PointPillars [14] serves as the backbone for processing the LiDAR data.
- The abbreviations 'ped', 'div', and 'bdry' correspond to the map elements of pedestrian crossing, divider, and boundary, respectively.
Table 2: Map vectorization performance on nuScenes Map (extended) validation set.
Method	APChamfer	avg.	APraster	avg.
ped	stp	int	cap	div	bdry	ped	stp	int	cap	div	bdry
MapTR [20]	34.3	29.9	21.5	37.9	44.9	45.1	35.6	22.5	12.1	38.4	23.4	18.3	12.1	21.1
MapTR [20] + MapVR (Ours)	39.5	31.6	21.9	42.4	45.8	45.9	37.9	30.8	13.9	43.3	32.8	27.0	18.8	27.8
- All competing methods take multi-view cameras as input, use ResNet-50 [11] as the backbone, and are trained for 24 epochs.
- ped', 'sp', 'int', 'cap', 'div', and 'bdry' denote pedestrian crossing, stopline, intersection, carpark area, divider, and boundary, respectively.
Dataset and Evaluation Metrics. MapVR is evaluated across multiple datasets, as outlined below.
1. nuScenes Map (basic) [2], which consists of two line-shaped map classes (lane divider and road boundary) and one polygon-shaped map class (pedestrian crossing). This dataset setup aligns with prior works on map vectorization [16, 6, 26, 20].
2. nuScenes Map (extended) [2], an extension of nuScenes Map (basic) that incorporates more complex map elements, such as intersection, stopline area, and carpark area.
3. Argoverse2 [49], a large-scale dataset featuring the same classes as nuScenes Map (basic).
4. 6V-mini-v0.4, our proprietary large-scale commercial dataset for autonomous driving, covering very complex driving scenes in real world. It includes three line-shaped classes (lane, curbside, and stopline) and two polygon-shaped classes (crosswalk and intersection).
Both Chamfer-distance-based metric (Section 3.1, denoted as  $\mathrm{AP}_{\mathrm{Chamfer}}$ ) and the newly proposed rasterization-based metric (Section 3.2, denoted as  $\mathrm{AP}_{\mathrm{raster}}$ ) are used for performance evaluation.
Implementation Details. All experiments, unless otherwise stated, are conducted with 8x NVIDIA RTX 3090 GPUs. As the proposed MapVR is a generic framework with no reliance on specific model architecture, we adopt MapTR [20], the state-of-the-art model for map vectorization, as the base model. Our implementation aligns with MapTR [20]. Please refer to the appendix for more details.
Results on nuScenes Map. Table 1 compares MapVR with existing map vectorization techniques on nuScenes Map (basic). Even under the less sensitive  $\mathrm{AP}_{\mathrm{Chamfer}}$  metric, our proposed MapVR delivers superior overall performance across various settings. The advantage of MapVR becomes even more pronounced under the more precise and autonomous-driving-oriented  $\mathrm{AP}_{\mathrm{raster}}$  metric. Specifically, MapVR provides a notable  $3.5\%$  improvement over a fully-trained MapTR. When working with multi-modality inputs, MapVR obtains an even larger margin of  $5.9\%$ . As shown in Table 2, on the more challenging nuScenes Map (extended) dataset that includes more complex elements, our MapVR achieves superior performance across all map elements under both metrics. These results validate the substantial improvements brought by MapVR and its exceptional capability of adapting to challenging scenarios. It is noteworthy that these improvements are achieved without adding any additional computational burden during inference.
Figure 6: Visualization of online HD map vectorization results. Our proposed MapVR demonstrates a superior ability in constructing more accurate maps, particularly for complex map elements and intricate details.
Results on Argoverse 2. Table 3 presents the performance comparison on the Argoverse2 dataset [49]. Note that in our setup, the height information for map elements is ignored. Our proposed MapVR method still achieves state-of-the-art performance, which verifies its robustness across multiple scenarios.
Results on 6V-mini-v0.4. Finally, we test MapVR on 6V-mini-v0.4, our proprietary commercial dataset that features highly intricate real-world driving scenes. As Table 4 shows, MapVR greatly enhances the performance on all map elements, which validates its efficacy and robustness in complex and real-world contexts.
Table 3: Comparison of various map vectorization methods on Argoverse2 validation set.
Method	APChamfer	APraster
ped	div	bdry	avg.	ped	div	bdry	avg.
HDMapNet [16]	13.1	5.7	37.6	18.8	-	-	-	-
VectorMapNet [26]	38.3	36.1	39.2	37.9	-	-	-	-
MapTR [20]	54.7	58.1	56.7	56.5	22.1	32.6	24.0	26.2
MapTR [20] + MapVR (Ours)	54.6	60.0	58.0	57.5	23.5	36.5	30.2	30.1
- 'ped', 'div', and 'bdry' denote pedestrian crossing, divider, and boundary, respectively.
Table 4: Map vectorization performance on 6V-mini-v0.4 dataset (our proprietary commercial dataset).
Method	APraster
lane	curbside	stopline	crosswalk	intersection
MapTR [20]	41.3	32.9	7.6	13.3	43.6
MapTR [20] + MapVR (Ours)	50.8	37.3	11.8	14.3	44.0
Visualizations. Fig. 6 visualizes the results of HD map vectorization and compares our method with MapTR [20]. For a fair comparison, both methods use the ResNet-50 [11] backbone and solely rely on multi-view camera images as input. It can be observed that our method yields more accurate HD maps, particularly in capturing intricate details and accurately representing complex or curved map elements. Conversely, while MapTR [20] produces generally correct vectorized maps, it inevitably exhibits deviations in finer details and struggles to precisely construct complex map elements. These observations reaffirm our motivation to incorporate the precise supervision from HD rasterization into map vectorization, which compensates for the inherent limitations caused by the sparse, equidistant point sets, thereby enhancing the precision of map vectorization.
Table 5: MapVR's ablation experiments on nuScenes Map (basic) validation set. All models employ ResNet-50 as backbones and are trained for 24 epochs. MapVR's default setups are marked in gray .
(a) Rasterization resolution.  $\times$  denotes no rasterization.
resolution	×	64x32	128x64	180x90	256x128	320x160
\( \mathrm{mAP}_{\text{raster}} \)	24.3	21.5	29.8	30.4	31.2	30.9
\( \mathrm{mAP}_{\text{Chamfer}} \)	50.3	45.1	/	50.6	51.2	50.9
(b) Line rasterization softness  $\tau$
line softness τ	0.5	1.0	2.0	4.0	6.0
mAP_raster(divider)	29.2	31.7	33.1	32.5	31.4
mAP_Chamfer(divider)	48.0	50.3	54.4	53.3	52.8
(c) Regularization on point direction.
regularization	None	w/GT	w/self
\( \mathrm{mAP}_{\text{raster}} \)	29.5	29.3	31.2
\( \mathrm{mAP}_{\text{Chamfer}} \)	48.5	48.5	51.2
-  $mAP_{Chamfer}$  are added upon reviewers' kind suggestions. Entries marked with '/' are unavailable due to accidentally deleted checkpoints.
(d) Rasterization geometry-awareness.
all as lines	lines & polygons
mAPraster(ped xing)	21.8	37.5
mAPChamfer(ped xing)	34.9	47.7
(e) MapVR vs. parallel segm.
MapVR	parallel segm
\( mA P_{raster} \)	31.2	26.7
\( mA P_{Chamfer} \)	51.2	48.1
Rasterization & Rasterization Resolution. As shown in Table 5a, incorporating rasterization enhances performance, following a general trend where higher resolutions yield better results. However, an exception is observed at the  $64 \times 32$  resolution, which degrades the performance due to the lack of precise rasterization supervision at such a coarse resolution. Fig. 7 further presents the Precision-Recall curves, showing that MapVR leads to consistent performance gain under both metrics and, notably, a smaller gap under the two metrics. Conversely, the baseline exhibits a large drop under the stricter  $\mathrm{AP}_{\mathrm{raster}}$ . This proves the necessity of incorporating fine-grained supervision from rasterization.
Figure 7: Comparison of P-R curves. MapVR narrows the performance gap under the coarse and strict metrics.
Rasterization Softness  $\tau$ .  $\tau$  is a tricky hyper-parameter. It needs to be large enough to provide sufficient supervisory gradient while being small enough to ensure precise supervision. Empirically, polygon-shaped map elements are robust against various  $\tau$ , while line-shaped elements are not, due to their thin and elongated shapes. Table 5b studies the effect of different  $\tau$  for line, taking the 'divider' class (a line-shaped map element) as an example.
Auxiliary Regularization on Point Direction. Table 5c studies the effect of the direction regularization loss described in Eq. 4, and also compares it with MapTR's directional loss [20], which uses the directions of ground truth equidistant points as targets. Results show that our direction regularization loss (w/ self) improves performance, proving its effectiveness in regularizing vectorized smooth point sets and allocating more points on higher curvature areas to improve precision. Conversely, the performance of our MapVR slightly degrades when using MapTR's regularization (w/ GT). This is because our MapVR's supervision from rasterization no longer requires the vectorized outputs to be equidistant.
Geometry-Aware Rasterization. Table 5d shows that simply rendering all map elements into lines severely impairs performance. The performance drop mainly comes from polygon-shaped elements (ped CROSSing:  $37.5\% \rightarrow 21.8\%$ ). This verifies the necessity of geometry awareness in rasterization.
Why Not Introduce HD Supervisory Signals from an Auxiliary Segmentation Task? A simple alternative to incorporate fine-grained supervision from rasterization is to append an additional parallel segmentation branch as an auxiliary task (dubbed as 'parallel segm'). This has been verified effective in many works [8, 4, 1]. Table 5e compares MapVR with this strategy. While 'parallel segm' improves baseline performance by  $2.4\%$ , it still largely lags behind our MapVR. The improved performance from 'parallel segm' supports our motivation to enhance map vectorization via rasterization. However, we attribute its inferior performance compared to MapVR to the fact that, unlike in our MapVR, the fine-grained supervisory signal is not directly applied to the vectorized output.
5.4 Computational Overhead During Training
With the CUDA-accelerated differentiable rasterizer, our proposed MapVR only brings a marginal
Table 6: MapVR's computational overhead during the training stage. Results were obtained with 8x NVIDIA A100 GPUs under the same training setups.
Method	Modality	Backbone	Training Time / Iter	GPU Memory Usage
MapTR [20]	C	Res-50	0.82 s	14021 MB
MapTR [20]	C & L	Res-50	1.18 s	28557 MB
MapTR [20] + MapVR (Ours)	C	Res-50	0.91 s	14169 MB
MapTR [20] + MapVR (Ours)	C & L	Res-50	1.37 s	28673 MB
- In modality, 'C' denotes multi-view camera input and 'C & L' denotes combined multi-view camera and LiDAR input.
increase in memory footprint while maintaining training efficiency. Table 6 presents a detailed comparison between the training costs of our method, 'MapTR + MapVR', and its baseline, 'MapTR'.
5.5 Failure Case Analysis
While the proposed method greatly improves the quality of HD map vectorization, the numerical results suggest that the results are still far from perfect. We provide visualization of a few typical failure cases in Fig. 8.
From row 1, 2, and 4 in Fig. 8, it can be seen that occlusions, whether from vehicles, constructions, or a limited field of view, hamper perception in the bird's-eye-view. Such occlusions often result in inaccuracies in the predicted vectorized maps. Yet, since road structures typically follow regular patterns, current map vectorization techniques may benefit from integrating road structure priors, such as standard-definition maps (SDMap), to enhance their reasoning capabilities.
Row 5 in Fig. 8 shows that there is still room for improvement in nighttime driving.
Row 3 in Fig. 8 is caused by ambiguity in annotation, where it is unclear whether the middle crosswalk should connect to the adjacent ones or not.
Figure 8: Visualization of failure cases produced by our method.
In this paper, we introduce a new perspective on map vectorization: rasterization, through which we can learn and evaluate map vectorization more precisely. We demonstrate that, while vectorized representation is compact and easy to use, it lacks representation capability, especially regarding fine-grained details; thus, it is necessary to incorporate rasterization as a complement in both learning and evaluation. We hope our perspective can serve as the cornerstone and spur further innovation in map vectorization, and can eventually lead to safe and reliable autonomous driving.
The authors would like to thank the anonymous reviewers for their helpful suggestions.