MGMapNet: Multi-Granularity Representation Learning for End-to-End Vectorized HD Map Construction
Authors: Jing Yang
Minyue Jiang
Sen Yang
Xiao Tan
Yingying Li
Errui Ding
Jingdong Wang
Hanli Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Online HD map construction，vectorized representation，autonomous driving
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: The construction of vectorized high-definition map typically requires capturing both category and geometry information of map elements. Current state-of-the-art methods often adopt solely either point-level or instance-level representation, overlooking the strong intrinsic relationship between points and instances. In this work, we propose a simple yet efficient framework named MGMapNet (multi-granularity map network) to model map elements with multi-granularity representation, integrating both coarse-grained instance-level and fine-grained point-level queries. Specifically, these two granularities of queries are generated from the multi-scale bird's eye view features using a proposed multi-granularity aggregator. In this module, instance-level query aggregates features over the entire scope covered by an instance, and the point-level query aggregates features locally. Furthermore, a point-instance interaction module is designed to encourage information exchange between instance-level and point-level queries. Experimental results demonstrate that the proposed MGMapNet achieves state-of-the-art performances, surpassing MapTRv2 by 5.3 mAP on the nuScenes dataset and 4.4 mAP on the Argoverse2 dataset, respectively.

Published as a conference paper at ICLR 2025
MULTI-GRANULARITY REPRESENTA- TION LEARNING FOR END-TO-ENT VECTORIZED HD MAP CONSTRUCTION
MGMAPNET:
Jing Yang
Minyue Jiang 2+
Sen Yang
Xiao Tan 2
Yingying Li
Errui Ding
Jingdong Wang 2
Hanli Wang
1 College of Electronic and Information Engineering, Tongji University, China
Baidu Inc.
ABSTRACT
The construction of vectorized high-definition map typically requires capturing both category and geometry information of map elements. Current state-of-the- art methods often adopt solely either point-level or instance-level representation, overlooking the strong intrinsic relationship between points and instances. In this work, we propose simple yet efficient framework named MGMapNet (multi- granularity map network) to model map elements with multi-granularity repre- sentation, integrating both coarse-grained instance-level and fine-grained point- level queries. Specifically, these two granularities of queries are generated from the multi-scale bird's eye view features using proposed multi-granularity ag- gregator. In this module, instance-level query aggregates features over the entire scope covered by instance, and the point-level query aggregates features local- ly. Furthermore, a point-instance interaction module designed to encourage in- formation exchange between instance-level and point-level queries. Experimental results demonstrate that the proposed MGMapNet achieves state-of-the-ar perfor- mances, surpassing Map'TRv2 by 5.3 mAP on the nuScenes dataset and 4.4 mAP on the Argoverse2 dataset, respectively.
INTRODUCTION
It is crucial to perceive and understand road map elements for ensuring the safety in autonomous driving applications (Xiao et al., 2020; Xu et al., 2023; Prakash et al., 2021). High-definition (HD) maps provide category and geometry information about road elements, enabling autonomous vehi- cles to maintain lane position, anticipate intersections, and plan optimal routes to mitigate potential risks. However, constructing HD maps requires significant human effort for annotating and updat- ing, which limits scalability over large areas. Recent researches, such as (Li et al., 2022a; Liao et al., 2022; 2023; Ding al., 2023; Yuan et al., 2024; Hu et al., 2021), focus on learning-basse methods as alternatives to construct HD maps from onboard sensors. These methods can be mainly divided into two categories based on the representation in use: rasterized map-based representation (Li al., 2022a;b; Liu et al., 2023b; Xiong et al., 2023) and vectorized map-based representation (Ding et al., 2023; Li al., 2023; Liao et al., 2023).
Rasterized map-based methods often require complex post-processing to meet the need of down- stream modules, such as planning, and suboptimal results are usually obtained which are not en- tirely end-to-end optimized. Therefore, there has been increasing attention paid to end-to-end map construction methods (Shin et al., 2023; Qiao et al., 2023b; Zhang et al., 2024) using vectorized representations, which commonly employ bird's eye view (BEV) (Fadadu et al., 2022; Chen et al., 2017: Liang et al., 2019; You et al., 2019; Liang et al., 2018) space for end-to-end perception, effec- tively integrating various sensor information such surround-veww cameras and Lidar.
The state-of-the-art methods typically adopt DETR-like architectures (Carion et al., 2020), com- prising encoder and decoder components. The encoder extracts multi-sensor information into BEV
"Equal Contribution.
Corresponding author (H. Wang, E-mail: hanlivang@tongi.cddu....)

---

Published as a conference paper at ICLR 2025
Surrounding images
StreamMapNet
MapTRv2
Ours
GT
Inacourate point
missing
Figure 1: Comparison of visualization results. Visual comparison among StreamMapNer, MapTRv2, and MGMapNet, with vehicle-centric views fcaturing red boundaries, orange dividers, and blue pedestrian cross- ings. The green boxes denote the phenomenon of inaccurate point coordinates in instance-level queries, while the purple ellipses indicate the phenomenon of missing instances in point-level queries. StreamMapNet em- ploys MPA for single-frame results. Best viewd in color.
representation, while the decoder decodes the category and geometry information of each road el- ement through queries, and thus end-to-end vectorized representation of output map elements is achieved, eliminating the need for the complex post-processing steps involved in rasterized map rep- resentation. These methods use either point-level queries instance-level queries to generate map elements. Point-level queries are good at describing the geometric position of road elements. For instance, in MapTR (Liao et al., 2022) and MapTRv2 (Liao et al., 2023). a permutation eqqivvllent point expression accurately represents the location of map elements, ensuring stable training pro- cesses. However, these methods may lack an overall description of map elements, leading to defi- ciencies in representing lane relationships. For example, Map2 may miss lane lines in distant and merging scenarios, as the region illustrated in the purple ellipses of Fig. 1.
While instance-level queries excel at capturing the overall category information of road elements, they may struggle to accurately represent geometric details, especially for irregular or elongated map elements. For example, in StreamMapNet (Yuan et al., 2024), multi-point attention (MPA) is pro- posed to capture the overall information of road elements, allowing for longer attention ranges while maintaining computational efficiency. However, this method may encounter difficulties in perceiv- ing the geometry of irregular or elongated elements, leading to local disturbances. The green boxes in Fig. 1 highlight the issue of inaccurate point coordinates obtained from instance-level queries: although map elements are successfully detected, their positional accuracy is compromised.
The primary challenge lies in balancing detailed and comprehensive representations, which cur- rent researches and methods fail to adequately address. To integrate both fine-grained local po- sitions and cears--grained global classification information, we propose a multi- gaatiigranularity map network (MGMapNet) to represent map elements using multi-granularity queries. Within each de- coder layer, point-level queries and instance-level queries are simultaneously computed by querying multi-scale BEV features using multi-granularity aggregator. Subsequently, point-instance interac- tion, including point-to-poin attention and point-to-instance attention, is designed to enhance in- trinsic relationships. Ultimately, point-gramularity queries are utilized to localize point coordinates, while garaulceggranularity queries are employed to determine the categories of map elements.
The main contributions of this work are summarized follows. First, we propose robust multi- granularity representation, enabling the end-to-end construction of vectorized HD maps by employ- ing coarse-grained instance-level and fine-graine, point-level queries in one framework. Second, the multi-granularity aggregator, combined with point-instance interaction, facilitates an efficien- t interaction between point-level and instance-level queries, effectively exchanging category and geometry information. Third, we incorporate several strategy optimizations into model training, en- abling our proposed MGMapNet achieve state-of-the-art single-frame performances on both the nuScenes (Caesar et al., 2020) and Argoverse2 (Wilson et al., 2023) datasets.
2 RELATED WORK
Online HD Map Construction. In recent years, researchers have utilized onboard sensors in au- tonomous driving to construct HD maps. The previous works (Huang et al., 2023; Chen et al., 2022)
2

---

Published as a conference paper at ICLR 2025
focus on projecting and lifting map elements detected the perspective view plane into 3D space for map reconstruction. With the aim of integrating multiple sensors such as panoramic cameras and LiDAR. construction methods for online HD map are gradually transitioning to BEV represen- tation. The HD map construction methods can be broadly categorized into two types: rasterized map-based and vectorized map-based. The rasterized map-based methods, such as HDMapNet (Li et all,, 2022a), utilize BEV features for semantic segmentation, followed by post- post-processing step to obtain vectorized map instances. Similarly, BEV-LaneDet (Wang al., 2023) introduces an efficient key-point representation for 3D lanes, generating confidence scores, y-axis offsets, and heights for each BEV grid cell. While rasterized maps can provide detailed road information, the requirement of post-processing limits their applications. With the emergence of vectorized DETR-like (Carion et al., 2020) end-to-end methods, the need for post-processing eliminated. VectorMapNet (Liu et al., 2023a) is the first end-to-end map reconstruction model that utilizes transformers. Map'TR and Map2Rv2 (Liao et al., 2022; 2023) introduce novel and unified modeling methods for map el- ements, addressing ambiguity and ensuring stable learning processes. PivotNet (Ding et al., 2023) leverages pivot-based representations for map elements, organizing element point sets into pivotal and collinear point sequences achieve more precise map element learning.
However, these methods often exclusively use either point-level queries or instance-level queries, missing
the mutual advantages of both granularities. To address this limitation, introduce a multi-granularig mechanism to represent map elements, which adaptively derives features at both fine-graing tine-grained point granularity and goarse- instance granularity, thus preserving local details as well as global map information.
Lane Detection. Lane detection can be regarded subtask of HD map construction, focusing on the detection of lane elements within road scenes. Current methods (Li et al., 2019; Zheng et al., 2022; Tabelini et al., 2021b) predominantly engage in lane detection from a single perspective view image, and the majority of lane detection datasets provide annotations only from single perspective. LaneATT (Tabelini et al., 2021a) proposes an anchor-based attention mechanism to aggregate global information. Unlike lane detection, vectorized HD map construction involves more complex map elements within the vehicle's 's perception range, including lane markings, curbs, and sidewalks.
METHOD
3.1
OVERALL ARCHITECTURE
The overall architecture of MGMapNet is depicted in Fig. 2 (a). Similar to other DETR-like end-to- end HD map construction models, MGMapNet comprises a BEV feature encoder which is respon- sible extract multi-scale BEV features from perspective view images, and Transformer Decoder which stacks multiple layers of multi qulti-gramullrity attention (MGA) to generate predictions for map elements. The prediction from each layer encapsulates both category and geometry information within the perception range.
BEV Feature Encoder. The proposed MGMapNet takes surrounding-view RGB images as in- puts, expressing them as unified perceptual BEV feature representation for subsequent Transformer decoder. The unified BEV feature is denoted as Feeu E RCxN*W where C, H, W represent the number of feature channels, height, and width of the BEV feature, respectively. Given the diverse lengths of map elements, relying solely on a single-scale BEV feature fails to meet the requirements for detecting all elements with different lengths. Therefore, we employ downsample modules to reduce the spatial resolution of BEV features Feee by half, generating F Frre E R / W. More scales might be benificial, but it is observed that two scales are already good enough. As result, F Fis.bev& x +HxW) represents multi-scale BEV features, which are obtained by concate- nating the flattened tensors of Fee and
Decoder. Figure/ 2 (b) illustrates the 1-th MGA decoder layer, which is composed of self atten- tion, MGA, and ferd-forward network. The MGA consists of two components: multi multi-granuulaitty aggregator and point-instance interaction. The instance-level query is initialized using learnable pa- rameters and updated through interaction with BEV features, while the point query is dynamically generated by aggregating BEV features. Subsequently, the point-instance interaction facilitates the
3

---

Published as a conference paper at ICLR 2025
Surrounding images
Encoder
Decoder
Valkoreddrrty Attrrite
loes
1066
architecture MGMapNet
Point to-t-----nntttto point Atlention
Feed-forward Network
Multi-granularity Attention
Mult-granularity Aggregator
Point- Instance
Point-to-instance Attention
Self Attention
(b) MDA Decoder Layer
(e) Point-instance Interaction
Map Instance
Reference
Sampling
Instance
Point
Points
Points
Query
Query
Figure 2: Overview of MGMappNet. (a) MGMapNet takes multi-view images as inputs. processes them through encoder-decoder framework, and generates vectorized map representations. (b) The schematic diagram uulli-granularity attention (MGA) decoder layer. (c) Implementation details point- instance interaction, comprising point-to-poimt attention and point-to-instance to-inttance attention.
mutual interaction among local geometric information, global category information, and the point queries from the (1-1)-th layer.
3.2 MULTI-GRANULARITY ATTENTION
Instance-level queries effectively capture the overall categorical information of road elements but may struggle to represent geometric details, particularly for irregularly shaped elongated map features. Conversely, point-level queries provide detailed information, however, they only represent instances by aggregating multiple queries, resulting in lack of comprehensive descriptions of map elements. To simultaneously capture both detailed and comprehensive instance features, the multi- granularity attention (MGA) mechanism designed effectively maintain and update queries at various granularities. As illustrated in Fig. 2, MGA comprises two primary components: multi- granularity aggregator and point-instance interaction.
3.2.1 MULTI-GRANULARITY AGGREGATOR
In multi ceattiigranllrrity aggregator, instance-level queries interact with the multi-scale BEV features, and point-level queries are generated. Specifically, we improve the multi-head deformable atten- tion (Zhu et al., 2020) with multiple reference points for each query to aggregate long-range fea- tures from multi-scale BEV features. To improve readability, we omit the subscript m for the index of multiple heads M in the operator. More specifically, the multi-granularity aggregator takes as input the instance-level queries Qine E N,N in the first layer, along with the point-level queries Qpts R N, x,x,N N,, x and the reference points RF IR N,,NN, X,N, in the subsequent layers. N,, is the total number of instance-level queries, and N, is the total number of points belonging to an instance. Noted that the reference points in the first layer predicted by Qizs. and the reference points in
4

---

Published as a conference paper at ICLR 2025
subsequent layers updated by reference points from the previous layer
{
RF OL mns ) 0, (RF (RF MLP((
RF sigmoid (RF
where I represents the layer index, sigmoid(-) and sigmoid refer to the sigmoid and inverse sigmoid activation functions, and MLP(.) stands for multi- perceptron.
Since an instance is represented as a point sequence, position encoding added to the instance-level query. Given the location of reference point RF, we employ RF to generate positional encoding P as
PE ref MLP ref (RF
(2)
ref projection layer used to generate the positional embedding from reference points. We allocate Nrep sampling points to each reference point, aggregating features from these points to enhance the representation of the reference point. The location offset AS of sampling points w.r.t the reference point and the associated weights W are computed by combining the instance-level queries Qins and PEref
where MLP/-1
AS' = Saamplng-Offfeee Offieet Q /-1
ins + -PEE2))
R MoXNNxNNNxX
w Weight_ EEmeed( (-1 +P
IR NRN,
ins +
ref
E
S' (RF' + AS') E N
where Sampling ) and Weight Weight_ Embed(()) are MLP layers designed to generate the location offset AS and the attention weight W. respectively: RF is expanded uppropriately to match the dimension of AS'. By leveraging the sampling offset and the reference point, the sampling location S' is updated by adding RF and AS', Subsequently, Qins and Qpts are generated by the weighted sum of sampled features as
w
ins
softmax
(W (W',)) ERMoX(NoNN..),
(Ng.)E(N
w pts
softmax kENrep
(W's ERNxXNNXN..
No Neep
Q ins
w ins sampling(F | bev: s/,x) E ERN,xO
(4)
j=1 k=
Nrev
Q
w sampling bev. { NNNNN
pts
where j is the index of the N, points on instance, k is the index among the Nrep sampling points assigned the reference point, W ins and W pts denote the softmax- nofmaax-normalized weights across Np x Nrep and Nrep dimensions of W W while sampling(-) represents the bilinear sampling operator.
Through the cealiiigraularity aggregator, Qins and Opts are generated from multi-scale BEV fea- tures, capturing both global and local information for each map element. Compared with the multi- point attention in StreamMapNet (Yuan et al., 2024), our method incorporates point-level queries directly from multi-scale BEV features by sampling points and enhances the accuracy of predicted geometry points. In addition, compared with point-level representations such as MapTR (Liao et al., 2022) and MapTRv2 (Liao et al., 2023), our method updates instance-level queries with sampled point features and captures the overall category well as shape information of road elements.
3.2.2 POINT-INSTANCE INTERACTION
The point-instance interaction is designed with the intention of enhancing positional and categori- cal information interaction between these two granularities of queries. As illustrated in Fig. 2(c), point-instance interaction comprises two attention operators: point-to-point (P2P) attention and point-to-instance (P2I) attention. Concurrently, the sampling locations S and the attention weights
5

---

Published as a conference paper at ICLR 2025
W W inst pts obtained from the multi goltiigraaularity aggregator in the 1-th layer are flattened and con- catenated to encode positional information in P2P attention and P2I attention as
PE = MLP (S', W Wins). ins
(5)
PE pts MLP! ( w pt+),
where MLP (m2() and MLP
(te()) are MLP layers for instance-level queries and point-level queries respectively, PE and PEpts stand for the corresponding generated position embedding.
P2P Attention. As the coordinates of map elements are refined based on the point-level queries in previous MGA layer, these point-level queries play pivotal role in predicting coordinates in the current layer. Hence, the P2P attention module is devised to include point-level queries from both the current (-th layer and previous (1 1)-th layer as inputs of the attention layer
{ q SA(Q pts + PE pt.t) O, pts. Opto" + pts >=
pt= -- pto CA PE
(6)
Regarding the first MGA layer, since there is not previous decoder layer before it, the self attention operation (-(.) only conducts with the current point-level queries. For subsequent MGA layers, P2P attention is implemented through the cross attention operation CA(+) by combining the point-
level queries Q pts of the previous layer with the point-level queries Ope of the current layer.
P2I Attention. After P2P attention. P2I attention updates the point-level queries through cross- attention to capture instance-level geometric information, while leveraging position embeddings PE.. and PEpts to focus on the spatial distribution of polylines as
pts = CA(Q pt.
PE pts Q'n.. PE )E
The Np point-level queries belonging to the same instance-level query are aggregated through the aggregation layer MT.P agg- producing instance-level queries as
Nr (aeeg)
MLP agg
Output. Ultimately, point-level queries are utilized predict point location using regression head, while instance-level queries are employed to predict the categories of map elements using a classification head. In summary, multi-granularit aggregator and point-instance interaction are applied to generate and update multi-granularity queries, enabling perception of both the geometry and category of each map element.
EXPERIMENTS
4.1
EXPERIMENTAL SETTINGS
nuScenes Dataset. nuScenes (Caesar et al., 2020) is widely recognized dataset in the field of autonomous driving research, providing 1,000 scenes, each captured over a continuous 20-second interval. Each dataset sample incorporates data from six synchronized RGB cameras and includes detailed pose information. The perception ranges from -15.0m to 15.0m along the X-axis and from -3000m to 30.0m along the Y-axis. For experimental purposes, the dataset partitioned into 700 scenes comprising 28, 230 samples for training, and 150 scenes containing 6,019 samples for vali- dation.
Argoverse2 Dataset. The Argoverse2 dataset (Wilson et al., 2023) contains multimodal data from 1,000 sequences, including high-resolution images from seven ring cameras and two stereo cameras, well LiDAR point clouds and map-aligned 6-DoF pose data. All annotations sampled to facilitate the training and evaluation of 3D perception models. Results are reported on the validation set, with focus on the same three map categories identified in the nuScenes dataset.
densely

---

Published as a conference paper at ICLR 2025
Evaluation Metric. In alignment with MapTR (Liao et al., 2022), adopt the widely-acceppe metric of mean Average Precision (mAP) based on the Chamfer distance. The evaluation thresholds set at 0.5m, 1.0m, and 1.5m. We also utilize the IoU-based metric mAPrester as employed in MapVR (Zhang et al., 2024). Specifically, A Aeptt A APaiv. and A AP refer to the average preci- sion for pedestrians, dividers, and boundaries, respectively. Furthermore, real-time performance is measured in terms of frames per second (FPS), and the model's ss complexity is evaluated through the number of parameters (Params).
Training Loss. Our framework builds on Map (Liao et al., 2023), which serves as the prima- ry baseline. Therefore, align training loss with MapTRv2, including the point loss Cpts, the classification loss Lele: the edge direction loss Cair. and the dense prediction loss Ldense. To fur- ther stabilize training and improve perception, an auxiliary Loss Lauz is designed, which includes instance segmentation loss Cins seg and reference point loss Lres. The instance segmentation loss Lins_seg is defined as combination of cross-entropy loss and dice loss between the predicted in- stance segmentation map and the ground-truth instance mask, where the instance segmentation map generated through the dot product of instance-level queries and BEV features. Regarding the reference point loss Fref is concerned, it accelerates convergence and stabilizes training by super- vising reference points of each decoder layer with vectorized map instances. Specifically, Hungarian matching is first performed, followed by the calculation of the LI distance between each points pair, consistent with Cpts. The final loss defined as the weighted sum of the above losses as
L = Lps + B2Lals + ByLdir + BaLdense + Ag Baauu
where 8 represents the weight coefficient of the corresponding loss.
Implementation Details. Our model is trained on 8 A 100 GPUs with the batchsize of 2, utilizing the AdamW optimizer (Loshchilov et al., 2017) with the learning rate of 4 x 10-1 We adopt the ResNet50 (He et al., 2016) as the backbone and employ the LSS transformation (Philion & Fidler, 2020) with single encoder layer for feature extraction. The one-to-many training strategy (Liao al., 2023) is used, and the model trained for 24 epochs on the nuScenes dataset and epochs on the Argoverse2 dataset. The hyperparameters are configured as N₃ 100, Nrep 8, Np = 20,
31 = 5, 02 2, B3 0.005, 33 = 3, and B3-
4.2 COMPARISON WITH
ART METHODS
Results on nuScenes. Table 1 presents the results on the nuScenes validation dataset, utilizing multi-view RGB images as input. comparison to MapTRv22 (Liao al., 2023), the proposed MGMapNet has reached mAP of 66.8, exceeding by 5.3 with training duration of 24 epochs. After a prolonged training period of 110 epochs, the mAp obtained by MGMapNet 73.6, which is still significantly higher than 68.7 18.7 of MapTRv2 and 72.6 of MapQR (Liu et al., 2024b).
Table I: Comparison to the state-of state-of-thc-art methods on nuScenes val set.
Method
Epoch
APped
APas
AP Pou
mAP
FPS
Params (MB)
HDMapNet (Li et al., 2022a)
30
14.4
21.7
33.0
23.0
ReMapNet (Qian
2023a)
30
62.3
57.7
PivotNet (Ding al., 2023)
24
56.5
56.2
60.1
57.6
9.2
MapTRv2 (Liao et al., 2023)
24
59.8
62,4
62.4
61.5
14.1
40.3
MGMap (Liu et al., 2024a)
24
61.8
65.0
67.5
64.8
12
55,9
MapQR (Liu al., 2024b)
68.0
63.4
67.7
66.4
11.9
125.3
MGMapNet (Ours)
64.7
66.1
69.4
11.7
70.1
VectorMapNet (Liu al., 2023a)
110
42.5
$1.4
Mvp (Liao al., 2023)
110
68.1
68.3
69.7
68.7
14.1
40.3
MGMap (Liu et al.. 2024a)
110
64.4
67.6
67.7
66.5
12
55.9
MapQR (Liu et al., 2024b)
110
74.4
70.1
73.2
72.6
11.9
125.3
MGMapNet (Ours)
110
74.3
71.8
74.8
73.6
11.7
70.1
Moreover, the comparison between the proposed MGMapNet and two latest models in terms of loU-based metrics is performed, with the results shown in Table It is observed that the proposed MGMapNet generally achieves the best loU-based performances than MapVR and MGMap.

---

Published as a conference paper at ICLR 2025
Table 2: Performance comparison nuScenes val set in of loU-based AP.
Method
APraster APreater ped
APrester
m.APraster
MapVR (Zhang et al., 2024) (Nwu1S2113))
46.0
39.7
29.9
38.5
MGMap (Liu et ul., 2024a) (even20e))
54.5
42.1
37.4
44.7
MGMapNet (Ours)
54.0
42.7
44.1
46.9
Moreover, the qualitative results are depicted in Fig. 3, where there three complex scenarios: day- time vehicles with occlusion, nighttime low-light conditions, and low-light situations with occlusion. In the first case, MGMapNet exhibits more precise coordinate prediction compared to StreamMap- Net (Yuan et al., 2024) and preserves all road elements compared to MapTRv2. In the second case of nighttime low-light conditions, MapTRv2 struggles predict the divider on the right side of the vehicle due to its lack of instance-level perception. While StreamMapNet utilizes instance-level queries and identifies the divider, overall instance positioning accuracy remains inadequate. In contrast, MGMapNet accurately and completely detects the boundary in these challenging scenar- ios. In the third case involving nighttime conditions with occlusion, the results of StreamMapNet are nearly unusable; MapTRv2 incorrectly identifies the rear divider boundary, revealing its limi- tation in instance-level perception; by contrast, MGMapNet demonstrates remarkable robustness on accurately identifying the elements.
Sunsunding Images
StreomMapNet
MapTRv2
Occlusion
Night
Night Occlusion
Figure 3: Qualitative visualization on nuScenes set.
Results on Argoverse2. Table 3 presents the results on the Argoverse2 validation dataset for e- pochs. The Argoverse2 dataset provides two configurations for the representation of points: 2D and 3D point coordinates. We conduct experiments on both configurations, the proposed MGMap- Net outperforms the competing methods.
Efficiency Comparison. We conduct comprehensive efficiency analysis of the proposed MGMapNet and several open-source models. As demonstrated in the last two columns Ta- ble MGMapNet achieves the FPS of 11.7, which is comparable to the latest methods of MapQR and MGMap. The number of model parameters of MGMapNet 70.1 MB, which is lower than MapQR's 125.3 MB but slightly higher than MGMap's 55.9 MB.
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison to the state-of-the-art methads on Argoverse2 val
Method
Map dim.
A ped
APaso
A Aon
mAP
HDMapNet (Li et al., 2022a)
13.1
5.7
37.6
18.8
VectorMapNet (Liu al., 2023a)
38.3
36.1
39.2
37.9
MRp (Liao et al., 2023)
62.9
72.1
67.1
67.4
MapQR (Liu et al., 2024h)
64.3
72.3
68.1
68.2
HIMap (Zhou et al., 2024)
69.0
69.5
70.3
69.6
MGMapNet (Ours)
67.1
74.6
71.7
VectorMapNNet (Liu al., 2023a)
36.5
35.0
36.2
35.8
MRp (Liao et al., 2023)
60.7
68.9
64.5
MapQR (Liu et al., 2024b)
3
60.1
71.2
66.2
65.9
HIMap (Zhou ct al., 2024)
66.7
68.3
70.3
68.4
MGMapNet (Ours)
64.7
72.1
70.4
69.1
4.3 ABLATION STUDY
The proposed MGA consists of multi-granularity aggregator and point-instance interaction. order reveal the contributions of different modules the proposed model, ablation study is conducted on the nuScenes dataset with the results shown in Table 4, where the multi-point attention (MPA) from StreamMapNet (Yuan al., 2024) serves as the baseline. The multi- gulliigranullrityy aggregator is employed as comparative alternative in the decoder layer, leveraging multi iettiirrrnaaarttyy queries for prediction. We further validate the effectiveness of point-instance interaction by incrementally adding its components. There are several observations from the results. First, replacing MPA with proposed multi-gran granularity aggregator improves mAP from 59.6 62.7, showing the effective- ness of multi-granularit representations. Second, with multi-granularity aggregator, the separate inclusion P2P attention and P21 attention increases mAP from 62.7 to 64.8 and 65.0, respective- ly. This reveals that it is helpful to employ both geometric and category details to enhance queries. Third, by utilizing the complete point-instance interaction module, the model achieves the best mAP of 66.8, highlighting the importance of multi-granularity query interaction in boosting perception.
Table 4: Ablation study of the proposed modules.
Point-instance Interaction
Method
mAP
P2P Attention
P21 Attention
Multi-poine Attention (Yuan et al., 2024)
59.6
X
62.7
64.8
Multi-gramularity Aggregator
65,0
66.8
Table 5: Ablation study on the generality of the proposed modules transferred into MapTRRv2.
Experiment
Method
mAP
Multi Attention
55.9
(a)
Multi gulti-granullrity Attention
63.6 (+7.7)
+ Auxiliary Loss
64.4 (+0.8)
(C) Multi-scale BEV Feature
65.0 (+0.6)
(d)
Position Embeddings
66.2 (+1.2)
(e) + Increase Query Number
66.8 (+0.6)
To verify the generality of the proposed modules, we transfer them into the MapTRv2 (Liao et al., 2023) framework, with the ablation results shown in Table 5. The decoder with the multi-poin attention is used as the baseline, and we design the following experiments: (a) replace multi-point attention with MGA; (b) add the auxiliary loss; (c) utilize multi-scale BEV features; (d) incorporate

---

Published as a conference paper at ICLR 2025
position embeddings into the multi guantiigaanlarity aggregator; and (e) increase the number of queries. The experimental results demonstrate the effectiveness of these designed techniques.
5 CONCLUSION AND DISCUSSION
In this work, we propose struction. Specifically, the proposed multi-granularity attention leverages coarse-grained instance queries to represent fine-grained point queries. Point-instance interaction further captures category and geometrie information by facilitating interaction between features of different granularities. The proposed method enhances map construction performance by employing multi- multi-granularit queries to perceive map instance categories and polyline distributions. Extensive experiments are conducted on two benchmark datasets verify the superiority of the proposed method compared with other state-of-the-ar approaches. In the future, it is desired to extend the current work by considering the following two aspects: (1) exploring temporal methods to incorporate additional prior information, and (2) investigating more efficient representation mechanisms to enhance model efficiency.
map network for end-to-end vectorized HD map con-
ACKNOWLEDGMENTS
This work supported in part by National Natural Science Foundation of China un- der Grant 62371343 and Shanghai Municipal Science and Technology Major Project (No. 2021SHZDZX0100).
REFERENCES
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020.
Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. PersFormer: 3D lane detection via perspective transformer and the openlane benchmark. In European Conference on Computer Vision, pp. 550-567. Springer, 2022.
Xiaozhi Chen, Huimin Ma, Ji Wan, Li, and Tian Xia. Multi-view object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
Wenjie Ding, Limeng Qiao, Xi Qiu, and Chi Zhang. PivotNet: vectorized pivot learning for end-to- end HD map construction. In Proceedings of the IEFF/CVF International Conference on Com- puter Vision pp. 3172-366222 2023.
Sudeep Fadadu, Shreyash Pandey, Darshan Hegde, Yi Shi, Fang-Chich Chou, Nemanja Djuric, and Carlos Vallespi-Gonzalez. Multi-view fusion of sensor data for improved perception and predic- tion in autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2349-2355 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Rober- to Cipolla, and Alex Kendall. FIERY: future instance prediction in bird's s-eye view from surround monocular cameras. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15273-15282, 2021.
10

---

Published as a conference paper at ICLR 2025
Shaofei Huang, Zhenwei Shen, Zehao Huang, Zi-han Ding, Jiao Dai, Jizhong Han, Naiyan Wang, and Si Liu. Anchor3DLane: learning to regress 3D anchors for monocular 3D lane detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
17451-17460, 2023.
Qi Li. Yue Wang. Yilun Wang. and Hang Zhao. HDMapNet: an online HD map construction and evaluation framework. In 2022 International Conference on Robotics and Automation, pp. 4628- 4634, 2022a. doi: 10.1 119//C//46639..22.222..
Tianyu Li, Peijin Jia, Bangjun Wang, Li Chen, Kun Jiang, Junchi Yan, and Hongyang Li. Lane- SegNet: map learning with lane segment perception for autonomous driving. arXiv preprint 1IXi::3312.16108, 2023.
Xiang Li, Jun Li, Xiaolin Hu, and Jian Yang. Line-CNN: end-to-end traffic line detection with line proposal unit. IEEE Transactions on Intelligent Transportation Systems, 21(1):248 258, 2019.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. BEVFormer: learning birdseeye-view representation from cantra images via spatiotempo- ral transformers. In European conference computer vision. pp. 1-18. Springer, 2022b.
Ming Liang, Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi- sensor 3D object detection. In Proceedings of the European conference on computer vision, pp. 641-656, 2018.
Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-senson fusion for 3D object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 7345-7353, 2019.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. MapTR: structured modeling and learning for online vectorized HD map con- struction. arXiv preprint 2tt.:2208./44377, 2022.
Bencheng Liao. Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. MapTRv2: an end-to-end framework for online vectorized HD map con- struction. arXiv preprint
2023.
Xiaolu Liu, Song Wang, Wentong Li, Ruizi Yang. Junbo Chen, and Jianke Zhu. MGMap: mask- guided learning for online vectorized HD map construction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14812-14821, 2024a.
Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao. VectorMapNet: end-to-end vectorized HD map learning. In International Conference Machine Learning, pp. 22352- 22369, PMLR, 2023a.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela I. Rus, and Song Han. BEVFusion: Multi-task multi-sensor fusion with unified bird' bird's-eye view representation. In 2023 IEEE international conference on robotics and automation, pp. 2774-2781. IEEE, 2023b.
Zihao Liu, Xiaoyu Zhang, Guangwei Liu, Ji Zhao, and Ningyi Xu. Leveraging enhanced queries of point sets for vectorized map construction. arXiv preprint arXiv:2402. 174330 2024b.
Ilya Loshchilov, Frank Hutter, et al. Fixing weight decay regularization in Adam. arXiv preprint arXiv: 5, 2017.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: encoding images from arbitrary camera rigs by implicitly unprojecting to 3D. In Computer Eision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210. Springer, 2020.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to- end autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7077-7087, 2021.

---

Published as a conference paper at ICLR 2025
Limeng Qiao, Wenjie Ding, Xi Qiu, and Chi Zhang. End-to-end vectorized HD map construction with piecewise Bezier curve. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13218-122 13228, June 2023a.
Limeng Qiao, Yongchao Zheng, Peng Zhang, Wenjie Ding, Xi Qiu, Xing Wei, and Chi Zhang. MachMap: end-to-end vectorized solution for compact HD map construction. arXiv preprint arXiv:2306.10300 2023b.
Juyeb Shin, Francois Rameau, Hyeonjun Jeong, and Dongsuk Kum. InstaGraM: instance-level graph modeling for vectorized HD map learning, arXiv preprint arXiv:2301.0442 2023.
Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, and Thi- ago Oliveira-Santon Keep your eyes on the lane: real-time attention guided lane detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 294- 302, 2021a.
Lucas Tabelini, Rodrigo Berriel, Thiago M Paixao, Claudine Badue, Alberto F De Souza, and Thia- go Oliveira-Santos. PolyLaneNet: lane estimation via deep polynomial regression. In 2020 25th International Conference on Pattern Recognition, pp. 6150-611 61566 IEEE, 2021b.
Ruihao Wang, Jian Qin, Kaiying Li, Yaochen Li, Dong Cao, and Jintao Xu. BEV-LaneDet: an effi- cient 3D lane detection based on virtual camera via key-points. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1002-1011, June 2023.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:230J.00493 2023.
Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, and Antonio M López. Multimodal end-to-end autonomous driving. IEEE Transactions on Intelligent Transportation Systems, 23(1): 537-547, 2020.
Xuan Xiong, Yicheng Liu, Tianyuan Yuan, Yue Wang, Yilun Wang, and Hang Zhao Neural map prior for autonomous driving. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pp. 17535-17544, 2023.
Zhenhua Xu, Yujia Zhang, Enze Xie, Zhen Zhao, Yong Guo, Kenneth KY Wong, Zhenguo Li, and Hengshuang Zhao. DriveGPT4: interpretable end-to-end autonomous driving via large language model. arXiv preprint 0rXiv:2310.014:: 2023.
Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark accurate depth for 3D object detection
Campbell, and Kilian Q Weinberger. Pseudo-LIDARE in autonomous driving. arXiv preprint arXiv: 1XXXv:/9066006 9066063100 10, 2019.
Tianyuan Yuan, Yicheng Liu, Yue Wang, Yilun Wang, and Hang Zhao StreamMapNet:: streaming mapping network for vectorized online HD map construction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 7356-7365, 2024.
Gongjie Zhang, Jiahao Lin, Shuang Wu, Zhipeng Luo, Yang Xue, Shijian Lu, Zuoguan Wang, et al. Online map vectorization for autonomous driving: a rasterization perspective. Advances in Neural Information Processing Systems, 36, 2024.
Tu Zheng, Yifei Huang, Yang Liu, Wenjian Tang, Zheng Yang, Deng Cai, and Xiaofei He. CLRnet: cross layer refinement network for lane detection. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 898-907, 2022.
Yi Zhou, Hui Zhang, Jiaqian Yu, Yifan Yang, Sangil Jung, Seung-In Park, and ByungIn Yoo. HIMap: hybrid representation learning for end-to-end vectorized HD map construction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15406, 2024.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. arXiv preprint arxiv:2010.04119, 2020.
12


Reviews:
[{"review_id": "5alARrCs54", "reviewer": "Reviewer_hewp", "summary": "This paper presents MGMapNet, a multi-granularity map network for end-to-end vectorized HD map construction based on multi-scale bird’s eye view (BEV) images. Evaluations on four datasets show the effectiveness of MGMapNet over multiple baseline models.", "questions": "See the weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The contributions are highlighted. The novel contributions compared with previous approaches are also discussed properly.\n2. Both quantitative and qualitative results are shown and discussed. Ablation studies are conducted in a meaningful way.", "weaknesses": "1. The citations of the whole paper are wrong. It should be \\citep{} instead of \\cite{}. \n\n2. From Figure 1 and 3. we can see the advantages of MGMapNet over other models. However, I can still see that the extracted lanes by MGMapNet are sometimes zigzagged while the ground truth lines are straight lines. I wonder whether you can add some regularity or loss terms to avoid this. Maybe for those straight lines, resample their vertices along the straight lines every times during model training so that the model learns the linear feature instead of individual point locations?\n\n3. Can you describe each loss term in detail? Do you use the distance from each point to the ground truth points as the loss for L_{pts}?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ubFnemCDe0", "reviewer": "Reviewer_cU8c", "summary": "The paper introduces MGMapNet, designed to effectively model map elements through a multi-granularity representation by integrating both coarse-grained instance-level and fine-grained point-level queries to enhance map modeling. The framework employs a Multi-Granularity Aggregator. Besides, there is a Point Instance Interaction module, which facilitates the exchange of information between the instance-level and point-level queries, thereby improving the overall modeling capability of the network.", "questions": "1.\tIs it possible to consider instance-2-instance attention? Why not compare this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe problem studied in the paper is very important in practice and find applications in real world.\n2.\tThe paper is clearly written and easy to follow. \n3.\tExperiments are conducted to verify the performance of the proposed method.", "weaknesses": "1.\tThe challenges and contributions of the proposed techniques require further elaboration. What are the specific challenges to design these techniques in section 3? \n2.\tThe encoders and decoders are mostly MLP-based. It is difficult to understand the logic, rationale and difficulty to apply the techniques.\n3.\tSome evaluation metrics in experiments are not explained, e.g. AP_ped and AP_div, and AP_bou in table 1.\n4.\tHow are the proposed techniques related to High-Definition?\n5.\tQuality of figures and tables can be improved. For example, Table 4 has too big font size.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8J7K97ATwF", "reviewer": "Reviewer_pXZj", "summary": "The paper presents MGMapNet, a framework designed for end-to-end vectorized High-Definition map construction. MGMapNet introduces a multi-granularity representation that integrates both instance-level and point-level queries to effectively capture both category and geometric information of road elements. The proposed Multi-Granularity Aggregator and Point Instance Interaction modules allow information exchange between the two granularities, resulting in enhanced prediction accuracy. Experimental results demonstrate state-of-the-art performance on benchmark datasets such as nuScenes and Argoverse2, surpassing various baseline methods.", "questions": "Please clarify the comments for W1-W3.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper introduces a method that combines both coarse-grained instance-level and fine-grained point-level queries, effectively capturing both global category information and local geometric details of map elements. \n\nS2. The design of the Multi-Granularity Aggregator and Point Instance Interaction modules facilitates efficient and effective information sharing between instance-level and point-level queries.\n\nS3. The proposed MGMapNet framework outperforms several baseline models, achieving the state-of-the-art performance in HD map construction.", "weaknesses": "W1. The paper’s description can be overwhelming for readers who are not deeply familiar with the HD map construction topic (e.g., me). For example, it lacks a formal problem formulation, which would help in grounding the research context. Additionally, the method's explanation is a bit difficult to follow.\n\nW2. The paper could be strengthened by providing a detailed analysis of the time and space complexity of MGMapNet compared to baseline models. Given that efficiency is a key motivation, understanding how MGMapNet performs in terms of computational and memory resources would be beneficial.\n\nW3. It is not clear why the training epochs are set to have multiple values for various models, and why the long training schedule leads to fair comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)