VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization
Authors: Yiwei Zhang
Jin Gao
Fudong Ge
Guan Luo
Bing Li
Zhaoxiang Zhang
Haibin Ling
Weiming Hu
Venue: NeurIPS 2024 poster (2024)
Keywords: BEV map layout estimation
vector quantization
camera-only
Primary area: machine_vision

Abstract: Bird's-eye-view (BEV) map layout estimation requires an accurate and full understanding of the semantics for the environmental elements around the ego car to make the results coherent and realistic. Due to the challenges posed by occlusion, unfavourable imaging conditions and low resolution, \emph{generating} the BEV semantic maps corresponding to corrupted or invalid areas in the perspective view (PV) is appealing very recently. \emph{The question is how to align the PV features with the generative models to facilitate the map estimation}. In this paper, we propose to utilize a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to acquire prior knowledge for the high-level BEV semantics in the tokenized discrete space. Thanks to the obtained BEV tokens accompanied with a codebook embedding encapsulating the semantics for different BEV elements in the groundtruth maps, we are able to directly align the sparse backbone image features with the obtained BEV tokens from the discrete representation learning based on a specialized token decoder module, and finally generate high-quality BEV maps with the BEV codebook embedding serving as a bridge between PV and BEV. We evaluate the BEV map layout estimation performance of our model, termed VQ-Map, on both the nuScenes and Argoverse benchmarks, achieving 62.2/47.6 mean IoU for surround-view/monocular evaluation on nuScenes, as well as 73.4 IoU for monocular evaluation on Argoverse, which all set a new record for this map layout estimation task. The code and models are available on \url{https://github.com/Z1zyw/VQ-Map}.

VQ-Map: Bird's-Eye--iie View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization
Yiwei Zhang 1.2 , Jin Gao 1.2. Fudong Ge Guan Luo Bing Li
Zhaoxiang Zhang1.24 Haibin Ling", Weiming Hu State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA Schooo of Artificial Intelligence, University of Chinese Academy of Sciences School of Information Science and Technology, ShanghaiTech University *Center for Artificial Intelligence and Robotics, HKISI, CAS, PPoople AI, Inc, SStony Brook University Schangyiwei20233 gefudong2222 zhaoxiana zhang) ac. {jin gjin.gao, guanl, bli, wmhu) @nlpr. ia cn, alingdcs.stonybrook stonybrook.
Abstract
Bird's- sirdd'--ee-vveww (BEV) map layout estimation requires accurate and full under- standing of the semantics for the environmental elements around the ego car to make the results coherent and realistic. Due the challenges posed by occlusion, unfavourable imaging conditions and low resolution, generating the BEV semantic maps corresponding corrupted or invalid areas in the perspective view (PV) is appealing very recently. The question is how to align the PV features with the generative models to facilitate the map estimation. utilize generative model similar to the Vector Quantized- Varied--nai AutoEncoder TVQ-VABI to acquire prior knowledge for the high-level BEV semantics the tokenized discrete space. Thanks to the obtained BEV tokens accompanied with a codebook embedding encapsulating the semantics for different BEV elements in the groundtruth maps, features with the obtained BEV tokens from the discrete representation learning based on specialized token decoder module, and finally generate high-quality BEV maps with the BEV codebook embedding serving as a bridge between PV and BEV. We evaluate the BEV map layout estimation performance of model, termed VQ-Map, on both the nuScenes and Argoverse benchmarks, achieving 62.2/47.6 mean IoU for surround- wwwpoddiiiww//oollll.aa evaluation nuScenes, well as 73.4 IoU for monocular evaluation on Argoverse, which all set new record for this map layout estimation task. The code and models available on https://github. wwtp:/cwc///ocaeeeeee
this paper, we propose
able to directly align the sparse backbone image
Introduction
BEV layouts represent high-dimensional structured data that encompasses significant prior knowledge, particularly regarding road structures. While current methods for BEV map layout estimation mainly focus on constructing dense BEV features 22] 3 for semantic segmentation as map prediction, they often overlook the incorporation map prior knowledge. Additionally, occlusion and inherent challenges in depth estimation often lead to inaccuracies in dense features, especially in the areas that are corrupted or invalid in the PV. These factors contribute to incoherent and unrealistic BEV layout results, often with numerous artifacts (see Fig. [.. Yet, humans rely solely partial observations of a scene in the PV to imagine the entire coherent BEV layout elements. A natural
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

PREE
PRODY
ToT T
ROOK
FRINTLETT
PRONE
FRODEL FUSHT
BASKLFRENT
BPACK
EASKLEBT
Surround- View Images
Groundtruths
BEVFusion 1
VQ-Map (Ours)
We showcase the prediction results in various environmental conditions (day, rainy and night from top to bottom). Our VQ-Map produces more reasonable results, even for areas that are not directly visible, while significantly reducing artifacts. Color scheme is the same as in [
Figure 1:
approach to imitating the human imagination process is to leverage generative models to learn the prior knowledge from the groundtruth BEV map layouts. However, the question is how align the PV features with the generative models to facilitate BEV map estimation.
To this end, we propose novel pipeline called VQ-Map (see Fig. Vig.2). which aligns the generative models well in the spirit of discrete tokens. In specific, VQ-Map utilizes a generative model similar to VQ-VAE 4 to encode the groundtruth BEV semantic maps into tokenized. discrete and sparse BEV representations, termed BEV tokens, accompanied with a discrete embedding space (i.e., the codebook embedding). Each BEV token is the index of the nearest neighbor in the codebook embedding for an encoded BEV patch feature, representing the high-level semantics of BEV patch. BEV tokens serve as a new classification label to directly supervise the PV feature learning via a specialized token decoder in our pipeline. The training of the generative model and the token decoder module separated. By aligning with the sparse BEV tokens, our token decoder module is able to rely solely on sparse backbone features directly queried by token queries for BEV token prediction using an arbitrary transformer-lide architecture (577). Simultaneously, directly employing these sparse features for token prediction bypasses the challenges of building accurate dense BEV features in common practice. The predicted tokens can be integrated into BEV embeddings through the off-the-shelf codebook embedding for generating the final high-quality BEV semantic maps. This process is similar to the human brain's memory mechanism (8) where the targets (BEV map layouts) encoded into highly abstract, sparse representations (BEV embeddings) through memory neurons (BEV tokens) that can be activated by specific visual signals (generated based on token queries).
We evaluate our proposed VQ-Map on both the surround-view and monocular map estimation tasks, and our method sets new records in both tasks, achieving 62.2/47.6 6 mean IoU for surround- view/monocular evaluation on nuScenes [9, as well as 73.4 ToU for monocular evaluation on Argoverse [00.
In summary, our contributions are as follows: (1) We propose novel pipeline VQ-Map exploring a discrete codebook embedding to generate high-quality BEV semantic map layouts. The acquired prior knowledge subsequently helps to effectively align the sparse backbone image features with the generative models based on a specialized token decoder, leading more accurate BEV map layout estimation with generation. (2) By formulating map estimation as the alignment of perception and generation, our achieved BEV codebook embedding serves as bridge between PV and BEV, and can be used in the off-the-shelf manner. (3) Extensive experiments show that our VQ-Map establishes new state-of-ther -ftte--r-- performance on camera-based BEV semantic segmentation. Meanwhile, we confirm that PV-BEV alignment method, token classification is more effective than value regression.

---

2
Related Work
BEV Map Layout Estimation. Most existing approaches treat BEV map layout estimation as a semantic segmentation task in the BEV frame, where map elements are rasterized into pixels with each allocated multiple class labels. As pioneer of such technology, LSS B explicitly predicts discrete depth distributions image features, and then 'lifts' these 2D features to obtain pseudo 3D features, which are finally flatten into BEV features through a pooling operation. Building upon it, BEVFusion [ introduces LiDAR point clouds and implements multi-senso fusion within a unified BEV space, effectively maintaining semantic and geometric information. Other approaches [TH55), such VectorMapNet [12] and MIMap [15), tackle layout issues by incorporating vectorized prior maps. Besides, TaDe [16] utilizes a task decomposition strategy to improve monocular BEV semantic segmentation performance.
Recently, some methods utilize generative model-based technologies to enhance the performance of BEV map layout estimation. MapPrior [17] employs a generative map prior built on VQ-GAN [18]
architecture to capture the detailed structure of traffic scenarios on the basis of conventional dis- criminative models, achieving a unified advantage in precision, realism and uncertainty awareness. Furthermore, DDP [19] and DifiBEV 20 focus on integrating the denoising diffusion process (1 into contemporary perception frameworks, exhibiting outstanding performance.
The above mentioned work MapPrior [1 and TaDe [16] both approach the BEV map segmentation task through two stages: perceptual stage and a generative stage, which is relevant to our work. However, MapPrior aligns with the generative model by deriving complex BEV variables, which are constrained by the challenges of acquiring accurate dense BEV features. As for TaDe, training the generative model based on polar inverse inverse-projected BEV groundtruth maps results in the loss of certain prior knowledge embedded in conventional BEV maps, making it prone to artifacts. In contrast, our method aligns the generative model with tokenized discrete representations, which are more meaningful and easier to predict, while also preserving BEV map prior knowledge.
Tokenized Discrete Representation. VAE innovatively employs codebook mechanisms to establish an encoder-decoder architecture in a tokenized discrete latent space, capturing and representing richer and more complex data distributions. Following this approach, other generative models such as VQ-GAN 18), DALL-E 22 and VQ-Diffusion 23 also map inputs into discrete tokens corresponding to codebook entries to represent nigh-dimensional data. Meanwhile, some visual pre-training works 24 25 use tokens to represent image patches and treats the prediction of
masked tokens proxy task. Recently, UViM (6),, Unified-IO 27 and AiT (8) encode various outputs as tokens and predicts them through an auto-regressive modeling [29, modeling a wide range of visual tasks. this paper, we draw inspiration from the above work to predict BEV tokens for
generating high-quality BEV map layouts.
3 Methods
We herein summarize our VQ-Map perception framework in Fig. 2 as follows. Firstly, we create the discrete representations which encapsulate the high-level BEV semanties for different BEV elements in the groundtruth maps to serve as the prior knowledge (i.e., the codebook embedding) for map generation. Secondly, we conduct the PV-BEV alignment training with the specially designed token decoder module to predict the BEV tokens associated with the corresponding groundtruth maps. Finally, we directly combine the off-the-shie codebook embedding accompanied by the map generation decoder with the PV-BEV alignment module to predict the BEV map layouts.
3.1 Discrete Representation Learning for BEV Generation
we formulate the discrete representation learning as the task of BEV map reconstruction via sequence of discrete tokens to acquire prior knowledge for the high-level BEV semantics. We obtain this tokenized discrete space by employing the VQ- VAE architecture 4. which comprises three modules: BEV Patch Embedding E, Vector Quantization Q and BEV Map Generation Decoder DD Roughly speaking, & transforms local BEV semantic patches into more abstract high-level semantics; Q then clusters the semantics derived from patch embedding to create the discrete representations; and finally, D is attached to utilize these discrete representations for reconstruction of the corresponding groundtruth maps.
Similar some visual pre-training methods 24 25
3

---

Discrete Representation Learning
BEV Patch
Embedding
BEV Map Dacoder
Embedding
BEV Embeskings
Indea
31
BEV Map 67
ØEV Tokens G
Reconstruction
BEV Token Prediction for PV-BEV Alignment
Collbration
121
12
204
54
Backbone FPN
Tokenn Decoder
100
$1
Generation Decoder
BEY Tokens Pred
Sumound Vew Inages
Embesbings
Inference
Codebook Embodding
BEV Map Fred
Neerest Neighbor Search
Embedding Function
Frozen
Figure 2: VQ-Map employs generative model similar to the VQ-VAE framework to encode the BEV groundtruth maps into BEV tokens accompanied with codebook embedding. After the generative model training, BEV tokens serve as classification labels to supervise token decoder module. During inference, VQ-Map utilizes the predicted BEV tokens to generate high high-quality BEV map layouts based on the off off-the-We thellfl-shef codebook embedding and the BEV map generation decoder.
PV feature learning via specialized
BEV Patch Embedding E. BEV semantic maps significantly differ from the raw images with complex scenes. They inherently represent high-level semantics annotated by humans, which eliminates the need to aggregate extensive features using heavy encoders. Specifically, we initially patchify groundtruth BEV map M E x into a sequence non-overlapping BEV patches
(M' B Cx P1 N
map layouts and N i=1' where IB {0, 1}, is the patch size, C is the number of the groundtruth = HW/ p2 is the patch number. Our patch embedding
simple, aiming to abstract high-level semantics z E IR' from individual patches M', where is the embedded dimension. Fig. shows some BEV patch images visualize our discrete representation learning.
Vector Quantization Q. We define latent embedding space V R our codebook bedding, where K represents the maximum number of representations in the discrete latent space. We further denote using the {V1. V ......, vK). ). Our vector quantization Q receives the continuous latent vector ze from the patch embedding and outputs discrete latent Zy. termed BEV embedding, through the nearest neighbor search in the codebook. This is calculated as
7 = 2(z.) = arg min [[(2(ze) (v22)l
(2()))
where la means L2 normalization employed for codebook lookup based on cosine similarity, as described in ImprovedVQGAN 30 Each discrete latent can also be represented by its index in the codebook as the BEV token:
hy = min min [(22 E2 (vv)lll
BEV Map Generation Decoder D. We feed the BEV embeddings { (2 ) to our map generation decoder D by firstly reshaping them into a grid format and then reconstructing the original groundtruth BEV map following
M' D(Q(E(M)))
Training Loss. The overall training loss includes VQ loss Lug based on the codebook embedding due to the non-differentiable vector quantization operation besides the reconstruction loss Zre- Unlike the common practice, we additionally incorporate a loss term reflecting the patch-level data augmentations (such small-scale rotations, translations, and resizing) to aid clustering, The VQ

---

Figure 3: Visualization of the BEV codebook embedding by showing the BEV patch images corresponding to the specific BEV tokens. All BEV patch images in the same column correspond to the same token. The data is randomly sampled from the nuScenes validation dataset. Color scheme the same in
loss is defined as:
N
Nu
Dog =
1 N
1 sg((2(z!)))1 + ||=g(z)) - Eg 112 +
Σ [(|()))) j=1
e2
(4)
where sg denotes stop-gradient, Maug denotes the number of patch augmentations, and z; the continuous latent vector derived from the augmented patch. We also use the exponential moving average (EMA) to update the codebook embedding.
We utilize the class-specific weighted mean square error (MSE) for computing the reconstruction loss Cre, where the weight of each class for each sample inversely proportional to the number of groundtruth pixels in that class, M. E BB HW and M IR HW:
M.
M
Cre
(5)
1+
Mell
and the final loss C is defined as:
L Lre + frog
(6)
3.2 Token Prediction with Sparse Features for PV-BEV Alignment
Once the codebook embedding V is obtained from the vector quantization Q and BEV patching embedding ε, the dense prediction task of BEV semantic segmentation for the map layouts, repre- sented by M BC can be transformed into sparse token classification task represented by
grid M E A (W/P) consisting of BEV tokens (see Eq. 2 as illustrated in Fig. 2 where 21,2,3..... K).
Since each token represents a BEV patch instead of BEV pixel as mentioned in Sec. 1 the BEV tokens are significantly sparser compared the dense BEV features in common practice and the final BEV semantic maps. Each BEV semantic token can be recognized based on the semantics at the corresponding position the image features. So we propose transformer-like token decoder module based on deformable attention [6] to query image semantic features at the individual position predict the corresponding BEV token. This approach is inspired by the transformer based objection detection methods in DETR (7], Deformable-DETR
6 and BEVFormer [2].
Token Decoder Module. Our token decoder consists of multiple patch-level cross-attention layers, a convolutional layer and a head as shown in Fig. 4 It takes multi-scale image backbone features from the feature pyramid network (FPN) 31 and the camera calibration input, and outputs the predicted sparse BEV tokens for PV-BEV alignment. We use N learnable embeddings as our token queries, each associated with a specific 3D position determined via the LiDAR coordinate system. Specifically, the token reference points (or anchors in the following) are initially set in the LiDAR coordinates and subsequently projected to image space using the camera and LiDAR calibration.

---

For each patch-level cross-attention layer, follows the classical decoder layer ar- chitecture [ with self-attention, cross- attention and feed forward network (FFN). Initially, token queries engage in self- attention interactions, restricted to neigh- boring queries. Then, multiple anchors are set for each query, and the deformable cross-attention [6] samples features only from the images of surround view that the anchors can be projected onto. When the anchors of a query involve multiple images, the average of the sampled features from these images is taken (2). Given that each query aims capture patch-level seman- ties, the anchors are positioned at varying heights, widths, and depths before being projected onto the images. Finally, refined token queries are obtained through FFN.
Feature Sequence
reshape
Add Norm
FFN
Feature Grid
Surround-
Add Norm
Image Features
Conv
Attention
x M
Pos
Add & Norm
Taken Ref Refeeeece: Points
Multi-Head Sell-Altention
BEV Tokens
Pos
Camera Calibration
Token Query Embedding
Figure 4: Architecture of Our Token Decoder. Pos refers to the positional embedding, and M indicates the layer number.
After multiple patch-level cross-attention layer, a feature sequence is obtained and reshaped to a feature grid for predicting BEV tokens. We convolutional layer to integrate the neighboring sparse features in the feature grid. After the convolutional layer, there a multi-layer perceptron (MLP) serving the classification head to predict tokens. We utilize focal loss 32 to optimize this classification task.
Experiments
We evaluate the performance of our VQ-Map on both the surround-view and monocular BEV map layout estimation tasks and use the Intersection-over-nniiii (IoU) metric for evaluation.
We perform the surround- view experiments on nuScenes 9 involving 1000 on-road scenes from four locations in Boston and Singapore. Each scene has duration of approximately 20 seconds, resulting total of 40k samples/key-frane frammme... Each sample captured by six monocular cameras, covering 360" panoramic view around the ego vehicle. For BEV map estimation, nuScenes provides manual annotations encompassing
layout classes. According to the prior studies [3] M. we train and
validate VQ-Map on 700 scenes with 28130 samples and 150 scenes with 6019 samples respectively. In addition, we transform the original map layouts into the ego-centric frame through rasterization. We evaluate the IoU scores for 6 map layout classes, i.e., drivable area, pedestrian crossing, walkway, stop line, car- parking area, and lane divider. Additionally, we calculate the mean IoU (mloU) averaged over all of them.
We utilize both the nuScenes and Argoverse [10] datasets for the monocular task following the common practice. In particular, Argoverse employs seven surround-view cameras for data collection. It comprises 65 training sequences and validation sequences captured in Miami and Pittsburgh. Like the nuScenes dataset, Argoverse provides detailed and comprehensive semantic map layout annotations. We evalutate the loU scores for drivable area, pedestrian crossing, walkway and car-parking nuScenes while only the drivable area Argoverse.
4.1 Implementation Details for Surround- View Task
In this section, we primarily outline the experimental settings for the surround-viee task on nuScenes, while leaving the settings for the monocular task in Appendix due to the space limitation.
Data Preparation. Following BEVFusion [ we generate the BEV semantic segmentation map within a square area surrounding the ego car, which spans from -50 meters to 50 meters along both the x and y axes. We set the segmentation resolution 0.5 meters per pixel, which culminates in a final image output of 200 by 200 pixels in size. Considering the possibility of overlapping classes within the map, our model is designed to carry out binary segmentation across all classes. The input images resized to 256x704.

---

Table 1: State-of-the-an comparison for the surround-vier BEV map layout estimation on the nuScenes validation set. MapPrior [17 uses a fixed IoU threshold of 0.5, while other methods apply the threshold that maximizes loU according to their original settings. our method, we adopt constant IoU threshold of 0.5 to ensure fairer comparison across all existing approaches. We only evaluate different approaches in the camera-only setting.
IoU f (%)
Methods
Drivable
Ped, Cross.
Walkway
Stopline
Carpark
Divider
Mean
OFT
36
74.0
35.3
45.9
27.5
35.9
33,9
42.1
LSS
3
75.4
38.8
46.3
30.3
39.1
36.5
44.4
CVT 37
74.3
36.8
39.9
25.8
35.0
29.4
40.2
M R8
77.2
40.5
BEVFusion
81.7
54.8
58.4
47.4
50.7
46.4
56.6
MapPrior
17
81.7
58.3
53.3
45.1
56.7
X-Align 34
82.4
55.6
59.3
49.6
53.8
47.4
58.0
MetaBEV 35
83.3
56.7
61.4
50.8
55.5
48.0
59.3
DDP 19
83.6
58.3
61.6
52.4
51.4
49.2
59.4
VQ-Map
83.8
60.9
64.2
57.7
55.7
50.8
62.2
Discrete Representation Learning Details. We use a BEV patch size P 8 and the codebook embedding K = 256 with dimensionality of = 128. Each sample of the semantic groundtruth map can be divided into N = 25 25 625 patches. For BEV patch embedding, 3x3 convolutional modules without padding, with channels 64, 128, and 128 respectively. A 2x2 max pooling layer is inserted before the final convolutional module. Following VQGAN (18), construct the BEV map generation decoder using layers with each including residual blocks. Additionally, append sigmoid function at the end of the decoder. Regarding to the codebook embedding optimization, we set the EMA decay to 0.99 and the epsilon value to 1e-5.
employ three
Token Decoder Details. We adopt SwinT-tiny 33
backbone to maintain consistency with the prior work [ [9] 34 35). The token decoder comprises patch-level cross-attention layers and a convolutional module with kernel size of 5. Each layer has a dimension of 512, with 8 heads and 16 reference anchors sampled from 4 heights, depths and widths. Each anchor predicts 2 offsets for each scale of the features map. Self-attention is performed for each token interaction within 5 x 5 region. Additionally, we set the output channel number the FPN to 512 to align with the token decoder. During the inference stage, also use the soft token technology 28 to represent the BEV embeddings.
Training. For the discrete representation learning, we employ an initial learning rate of le-4 and batch size per GPU, and conduct training on 2 NVIDIA 100 (40G) GPUs, totaling 35 GPU hours. Subsequently, for training PV-BEV alignment, we adjust the initial learning rate to 5e-5 and batch size per GPU for training on 4 NVIDIA 100 (40G) GPUs, totaling 96 GPU hours. The training is conducted over 20 epochs, incorporating warm-up and cosine learning rate decay strategics. Additionally, we apply a weight decay factor of 0.01.
4.2 State-of-the-Art Comparison
We conduct the state-of-the-art comparison for the surround-view BEV map layout estimation task by comparing to several recent competitive methods as shown Tab. [ Among them, BEVFusion is widely regarded as standard framework that integrates three key components, i.e., backbone module, a view transformation and task decoder. X-Align [34] uses perspective supervision to integrate perspective predictions based on dense BEV feature for enhanced performance. Map- Prior [7 employs a generative model to enhance the perceptual outcomes derived from BEVFusion. Meanwhile, MetaBEV (35] introduces innovative coss-attention module subsequent to the dense BEV feature extraction, which instrumental in addressing sensor failures. Additionally, DDP 19 leverages a denoising diffusion process 21 within its decoder to achieve better precision.
Tab. shows that our VQ-Map sets a new mloU performance record 62.2 for the surround-view map estimation task nuScenes when comparing to the above methods. In particular, VQ-Map achieves a 5.5 mloU gain comparison to BEVFusion, with notable improvement of over in the Stopline layout class. VQ-Map also consistently outperforms all other methods by significant

---

Table State-of- the-ar comparison for the monocular BEV map layout estimation on the nuScenes
and Argoverse validation sets using the IoU (%) metric. Our VQ-Map uses the IoU threshold of 0.5 while other methods choose the best threshold following their original settings. During the evaluation process, grid cells that cannot be reached by LiDAR are ignored [39
nuScenes 9
Argoverse 10
Methods
Drivable
Crossing
Walkway
Carpark
Mean
Drivable
IPM
30
40.1
14.0
43.7
Depth Unpr.
39
27.1
14.1
33.0
VED 40
54.7
12.0
20.7
13.5
25.2
62.9
VPN
58.0
27.3
12.9
31.9
64.9
PON 39
60.4
28.0
31.0
18.4
34.5
65.4
DifBBE 20
65.4
41.3
41.1
28.4
44.1
GitNet
42
65.1
41.6
42.1
31.9
45.2
67.1
TaDe [16
65.9
40.9
42.3
30.7
45.0
68.3
VQ-Map
70.0
43.9
43.8
32.7
47.6
73.4
margins, with a minimum mloU gain of 2.8 without relying on any other supervisions like PV semantic segmentation or depth estimation.
For the monocular BEV map layout estimation task, we replace the backbone our VQ-Map
with ResNet50 43 align with the previous work (39 42 (Please refer Appendix A for more implementation details). As shown in Tab. 2 our VQ-Map also consistently outperforms all other competitive methods that publicly available the nuScenes and Argoverse datasets with significant margins. In comparison to the second best approach on nuScenes GitNet [42], we achieve 2.4 mloU performance gain, while a 5.1 IoU gain on Argoverse in comparison the second best TaDe 16). We provide visualization results in Fig. A3 It is noteworthy that our experiments on the Argoverse [T] dataset directly exploit the off-the-shelf codebook embedding and BEV map generation decoder trained on nuScenes, indicating good transferability of our discrete representation learning.
4.3 Ablations and Analysis
Table Ablation experiments on some key parameters of the token decoder. We perform ablations on the token decoder layer number M using layer dimension of 512, and ablations on different layer dimension by setting M 8.
(a) Ablation of token decoder.
(b) Ablation for the layer dimension of token decoder.
Layer Dimension
256
512
768
Drivable
81.1
82.7
83.6
83.8
Drivable
83.0
83.8
82.9
Ped. Cross.
55.9
58.2
60.1
60.9
Ped. Cross.
58.4
60.9
57.9
Walkway
59.2
61.7
63.5
64,2
Walkway
62.4
64.2
61.6
Stop Line
50.9
55.1
56.8
57.7
Stop Line
54.5
57.7
54.1
Carpark
49.9
52.2
56.2
55.7
Carpark
53.6
55.7
52.5
Divider
47.3
49.0
50.3
50.8
Divider
48.5
50.8
48.6
Mean
59.8
61.8
62.2
Mean
62.2
Token Decoder Parameters. The number of patch-level cross-attention layers M the token decoder has substantial impact on the final performance, evidenced in Tab. 3al We note that as M increases, the mloU value also increases. However, the mloU improvement becomes marginal when M increases from 6 to 8, and we thus use 8 layers by default. Additionally, of setting different dimensions for the token decoder layers in Tab, 3bb which shows that the optimal dimension is 512. Note that the following ablation studies all employ 6 layers and 512 dimension.
investigate the effects
Dense vs. Sparse BEV Features. the token decoder of our approach, the features before the classification MLP head can also be regarded as a kind of sparse BEV features with shape 25 x 25 512, obtained based on deformable attention. this ablation, we compare the utilization of dense BEV features sparse BEV features for predicting either the sparse tokens based on the off-the-shelf codebook embedding or directly the dense semantic maps, where the dense BEV feature obtained through LSS (3] following BEVFusion with shape 128 x 128 80.

---

Table 4: Ablation for various design choices. Comparing (a) (b) (c) (f) entails assessing different architecture designs, while (d) (e) (f) involves comparing different supervisions during the PV-BEV alignment training. The supervision signals shown columns (d), (e) and (f) are the explored three kinds of intermediate results in Sec. 3.1 The differences are: (d) uses latent variables that have been discretized by the codebook, (e) uses the latent variables that have been discretized by the codebook, and (f) uses the codebook indices. (g) sets N Naug 0 in Eq.
(a)
(b)
(d)
(e)
Sparse Feature
Arch.
Codebook Embedding
V
Supervision
M
[ig]t=1
M
1
< ft=1
[kg 1
[Mg 1.V
Drivable
81.5
80.4
83.9
82.5
82.5
83.6
83.5
Ped. Cross.
54.2
52.9
60.0
59.7
59.1
60.1
59,9
Walkway
58.1
58.2
63.5
62.2
62.1
63.5
63.4
Stop Line
46.1
47.2
53.2
55.1
54.9
56.8
56.8
Carpark
53.2
52.7
51.0
53.4
53.1
56.2
55.1
Divider
45.3
46.6
46.9
48,9
49.0
50,3
50.7
Mcan
56.4
56.3
59.8
60.3
60.1
61.8
61.6
Improvements
-0.1
3.9
3.7
5.4
Table 5: Computational overhead analysis. Training time is measured in GPU hours using NVIDIA A100 (40G). Our method, even in its tiny version, surpasses the previous SOTA DDP (3 steps). Additionally, the computational cost (MACs) of our tiny version significantly lower than previous methods. For the standard version of the model, it achieves substantial performance improvements while maintaining a relatively low computational cost.
Method
ml)U1(%)
(M))ms((M)
MACs(G)
Training Time
BEVFusion
56.6
50.1
155.5
100
MapPrior
56.7
719.1
396.0
>200
DDP(3 steps)
59.4
53.6
614.1
160
VQ-Map(tiny)
59.6
86.8
30474=104
VQ- Map(light)
60.1
81.9
137.3
35+80=115
VQ-Map
62.2
108.3
231.6
35+96=131
To predict dense maps using sparse BEV features in (c), we replace the classification MLP head in the token decoder with BEVFusion's map decoder and the groundtruth BEV map M is used for supervision. (a) can be seen as a variant of BEVFusion. Conversely, for predicting sparse tokens using dense BEV features in (b), we introduce deformable decoder [6] after the dense BEV features that initialized with weights from BEVFusion. VQ-Map (f) predicts sparse tokens using sparse BEV features. The results in Tab. 4 show that sparse BEV features are more effective for predicting sparse tokens than dense BEV features, and our achieved sparse features work better even the traditional end-to-end framework for the map estimation task.
Different PV-BEV Alignment. This part of the experiment explores which kind of intermediate results of our discrete representation learning based on VQ-VAE (see Sec. 3.1 is best for the second stage training of PV-BEV alignment. Instead of using the sparse token classification task for PV-BEV alignment supervision, we also test the case of utilizing the continuous latent variables directly derived from the patch embedding as a regression task for supervision in (d). In (e), we test an alternative strategy for supervision which involves predicting the latent BEV features and obtaining the BEV embeddings through nearest neighbor search based on the codebook embedding. In both (d) and (e), MSE is employed to train the token decoder. As shown in Tab. 4 the performance supervised by BEV tokens using focal loss surpasses that supervised by latent variables using MSE. These results indicate that BEV tokens serve superior abstract representation for guiding PV-BEV alignment.
Computational Overhead Analysis. We provide the computational overhead for our VQ-Map on the surround-vie view task as shown in Tab. [] including the number of parameters, computational cost (MACs), and model training time. Decreasing the layer dimension of the token decoder from 512 to 256 resulting in light version of our VQ-Map. Since the learned discrete representations also require a significant number of parameters, we further use smaller architecture with K 128, Q and D to achieve tiny version of our VQ-Map. It shows that even the tiny version of our approach can still achieve comparable results to the recent SOTA methods in Tab. [T] Our approach not only demonstrates strong performance, but also saves much computational cost comparison to the recent
in

---

SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion.
Conclusion
In this paper, present novel pipeline VQ-Map by aligning with the generative models using discrete BEV tokens, which efficiently enhances the performance of BEV map layout estimation. The core components of our method are the codebook embedding constructed via vector quantization as prior knowledge, serving as bridge between PV and BEV, and the specially designed BEV token decoder facilitate the PV-BEV alignment, both of which enable the generation of high-quality BEV semantic map layouts. We hope that our work will inspire further research on vector quantization, providing assistance not only for map estimation and its downstream tasks, but also for wide range of other applications.
Limitations and future work. A significant limitation of our approach is our inability to handle semantics that are position-sensitive and have small areas. It is challenging for patch-level semantics to effectively represent position information for small-scale semantics using the architecture similar to VQ-VAE. Tokenization in our approach is more robust against noise and geometric changes. However, it may lead to loss of some detailed spatial information. When the dangerous holes and some random obstacles appear in the realistic AD environment, an anomaly detection module may be needed to handle this situation. Our approach is a specialized method only for BEV map generation at present, but we believe the token- token-based multi-task modeling for autonomous driving is very promising. Additionally, tokenized intermediate results well-suited for combining with large language models.
Broader Impacts. Accurate BEV layout estimation helps drivers and autonomous driving systems better understand the surrounding environment, thereby reducing the occurrence of traffic accidents and improving traffic safety. However, it may be misused to capture detailed information about the environment, including sensitive data such as building layouts, infrastructure, and so on.
Acknowledgements. This work was supported in part by the Beijing Natural Science Foundation (Grant No. L223003, JQ22014), the Natural Science Foundation of China (Grant No. U22B2056, 62422317, 62192782, 62036011, U2033210, 62102417, 62222206, U2033210, 62172413), the Project Beijing Science and technology Committee (Project No. Z231 100005923046), Jin Gao and Bing Li were also supported in part by the Youth Innovation Promotion Association, CAS. Haibin Ling was not supported by any fund for this work.
10

---

References
[1] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's bird's-eye view representation. In ICRA, 2023.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning s-eee-view representation from multi-camera images via spatiotemporal trans- formers. In ECCV, 2022.
[3] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
[4] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. NewrIPS, 2017.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurlPS, 2017.
[6] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv:2010.04159, 2020.
[7] Nicolas Carion, Francisco Massa, Gabriel Synnacve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020.
[8] R Quian Quiroga, Leila Reddy, Gabriel Kreiman, Christof Koch, and Itzhak Fried. Invariant visual 102-111177 2005.
representation by single neurons in the human brain. Nature,
[9] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[10] Ming-Fan Fang Chang, John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, al. Argoverse: 3d tracking and forccasting with rich maps. In CVPR, 2019.
[11] Qi Li, Yue Wang. Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In ICRA, 2022.
[12] Yicheng Liu, Tianyuan Yuan, Yue Wang. Yilun Wang, and Hang Zhao. vectorized hd map learning. In ICML, 2023.
Vectormapnet: End-to-end
[13] Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. arXiv:2208. 14437, 2022.
[14] Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Maptrv2: An end-to-end framework for online vectorized hd map construction. arXiv:2338.057736 2023.
[15] Dhananjaya Wijerathne, Zhaoying Li, Anuj Pathania, Tulika Mitra, and Lothar Thiele. Himap: Fast and scalable high-quality mapping on egra via hicrarchical abstraction. IEEE Transactions on Computer-Aide Aided Design of Integrated Circuits and Systems, 41(10))3290-333 0303, 2021.
[16] Tianhao Zhao, Yongean Chen, Yu Wu, Tianyang Liu, Du Peilun Xiao, Shi Qiu. Hongda Yang. Guozhen Li, Yi Yang. et al. Improving bird's eye view semantic segmentation by task decomposition. arXiv:2404.01925, 2024.
[17] Xiyue Zhu, Vlas Zyrianov, Zhijian Liu, and Shenlong Wang. Mapprior: Bird's-eye view map layout estimation with generative models. In ICCV, 2023.
[18] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-rcsolution image synthesis. In CVPR, 2021.
[19] Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In ICCV, 2023.
[20] Jiayu Zou, Kun Tian, Zheng Zhu, Yun Ye, and Xingang Wang. Diffbev: Conditional difffusion model for bird's eye view perception. In AAAI, 2024.
[21] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NewrIPS, 2020.

---

[22] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[23]
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to-image synthesis. In CVPR, 2022.
[24] Hangbo Bao, Li Dong. Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. 1rXiv:2106.08254 2021.
[25] 7. Peng, I. Dong, H Bao, Q Ye, and F Wei. Beit v2: Masked image modeling with vector-quamtized visual tokenizers, 2rXiv22288.06366, 2022.
[26] Alexander Kolesnikov, André Susano Pinto, Lucas Beyer, Xiaohua Zhai, Jeremiah Harmsen, and Neil Houlshy. Uvim: A unified modeling approach for vision with learned guiding codes. NeurlPS, 2022.
[27] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unificd-io: A unified model for vision, language, and multi-modal tasks. In ICLR, 2022.
[28] Jia Ning, Chen Li, Zheng 7hang, Chunyu Wang, Zigang Geng, Qi Dai, Kun He, and Han Hu. All in tokens: Unifying output space of visual tasks via soft token. In ICCV, 2023.
[29] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, el. Improving language understanding by generative pre-training. 2018.
[30] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge. and Yonghui Wu. Vector-quantized image modeling with improved vqgan. 1rXiv:211000627. 2021.
[31] Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[32] Tsung-Yr Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. ICCV, 2017.
[33] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[34] Shubhankar Borse, Marvin Klingner. Varun Ravi Kumar, Hong Cai, Abdulaziz Almuzairee, Senthil Yogamani, and Fatih Porikli. X-align: Cross-modal cross-view alignment for bird s-eee-view segmentation. In WACV, 2023.
[35] Chongjian Ge, Junsong Chen, Enze Xie, Zhongdao Wang, Lanqing Hong, Huchuan Lu, Zhenguo Li, and Ping Luo. Metabev: Solving sensor failures for 3d detection and map segmentation. In /CCV, 2023.
[36] Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection, 181 1.08188, 2018.
[37] Brady Zhou and Philipp Krähenbühh. Cross-view transformers for real-time map-view semantic segmenta- tion. In CVPR, 2022.
[38] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. bebe Multi-camera joint 3d detection and segmentation with unificd birds-eye view representation. arXiv:2204.05088 2022.
[39] Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. In CVPR, 2020.
[40] Chenyang Lu, Marinus Jacobus Gerardus Van De Molengraft, and Gijs Dubbelman. Monocular semantic occupancy grid mapping with convolutional variational encoder-decoder networks. IEEE Robotics and Automation Letters, 4(2):445-452, 2019.
[41] Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung, Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 53))4867-48773 2020.
[42] Shi Gong, Xiaoqing Ye, Xiao Tan, Jingdong Wang, Errui Ding. Yu Zhou, and Xiang Bai. Gitnet: Geometric brior-based transformation for birds-eye-view segmentation. In ECCV, 2022.
[43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
12

---

Appendix
A
Implementation Details for the Monocular Experiments
Data Preparation. Following PON 39), we define square region in front of the given monocular camera, spanning [0, 50] [-25, 25] meters. Setting the resolution 0.25 meters per pixel results in final map size of 200 x 200 pixels during the evaluation. Additionally, the images are resized to 600 800 for nuScenes and 600 x 960 for Argoverse. We perform the monocular experiments using all available surrounding cameras with training/validation splits outlined in PON [39].
Details of the Monocular Layout Estimation Model Training. The groundtruth map is resized to 224 x 224 during our model training, and the size of the BEV patch is set to 16 x 16, resulting in a total of 14 14 196 patches. We ResNet50 the backbone align with the previous work 39 and employ a token decoder with 6 patch-level cross-attention layers and 256 layer dimension.
For the discrete representation learning, set the initial learning rate to le-4 with batch size of 32 per GPU, and conduct training on NVIDIA 100 (40G) GPUs for 50 epochs, totaling 28 GPU hours. Subsequently, for training PV-BEV alignment, the batch size to per GPU for training on 4 NVIDIA A100 (40G) GPUs for 40 epochs, totaling 42/44 GPU hours for muSeenes/Argoversee We also resize output map size from 224 224 back to 200 x 200 using a maxpooling operation to aligns with the previous work's 's results during the evaluation. Since our VQ-Map only predicts BEV tokens within the view frustum range, padding used for latent representations in other areas. When calculating the focal loss, the patches that LiDAR cannot reach will be ignored.
adjust the initial learning rate to 5e-4 and
B More Visualization Results
Bacer Stxdi
EARCLES
PRSEE
EADK ST
LIOT
FRAVE
BACKIETT
Surround-Viee Images
Groundtruths
BEVFusion
VQ-Map
Figure Al: More visualization results for sorround-view BEV map layout estimation nuScenes.
13

---

VAONI
BRCAC
PRODELAAS
PRICH
FOIDET FRIET
EACKLERENT
BREK
LAOKLEET
PAONY JPP
PROCRA
POONTY
BEAKL SEEET
BROK
BOOKLEE
Surround-Var View Images
Groundtruths
BEVFusion
VQ-Map
Figure A2: More visualization results for sorround-view BEV map layout estimation nuScenes.
Monocular Image
Groundtruths
VPN
PON
GitNet
TaDe 16
VQ-Map
Figure A3: Visualization results for monocular BEV map layout estimation on nuScenes. Unlike other methods, our VQ-Map only focuses on map layout classes. The color scheme the same in PON 33
14

---

PREE
PROKA
33
RAOK
FRINTLAET
FROME
FRONTL FUSHT
BASKLFIENT
BRACK
LASK
Surround- View Images
DDP
MapPrior
VQ-Map (Ours)
Figure A4: More visualization results for sorround-view BEV map layout estimation on nuScenes.
We showcase the comparison of our method with DDP and MapPrior in various environmental conditions (day, rainy and night from top to bottom), corresponding Fig.
C More Experiments
Table Al: Ablation experiments some key parameters of the codebook embedding. We perform ablations on the codebook size using dimension / of 128, and ablations on different dimension / by setting K to 256.
(a) Ablation for K' of codebook embedding.
(b) Ablation for D of codebook embedding.
K
128
256
512
64
128
256
Drivable
83.5
83.6
83.5
Drivable
83.7
83.6
83.6
Ped. Cross.
61.3
60.1
60.0
Ped. Cross.
59.9
60.1
59.9
Walkway
63.4
63.5
63.3
Walkway
63.4
63.5
63.2
Stop Line
57.9
56.8
57.2
Stop Line
56.6
56.8
56.3
Carpark
53.0
56.2
55.5
Carpark
55.6
56.2
54.5
Divider
50.2
50.3
50.5
Divider
50.2
50.3
49.9
Mean
61.5
61.8
61.7
Mean
61.6
61.8
61.2
Cdoebook Embedding Parameters. We conduct the experiments with different codebook size K D, and the results in Tab, AAla and Tab, show that our proposed method insensitive to the size of the codebook in wide range.
Table A2: More Metrics. Chamfer distance is used to evaluate the boundary quality of the drivable area, corresponding Drivable IoU. Normalized IoU and Mean IoU (test set) further illustrate the superiority of the method. MMD is metric capturing the realism in scene structures.
Metrie
BEV BEVFusion
DDP
VQ-Map
Cham. Dis. J (Pixels)
2.67
2.45
2,21
Drivable IoU f (%)
81.7
83.6
83,8
Normalized loU f (%)
70.6
72.9
74.2
Mean IoU (test set) 1 (%)
63.9
67.2
70.2
MMD (val. set) 4
39.6
23.7
24.4
MMD (test set) 1.
19.5
12.9
10.0
Boundary Quality Evaluation for Drivable Area. While IoU is widely used to assess the per- formance of BEV map predictions, primarily reflects the overall overlap between predicted and groundtruth areas. However, it does not fully capture the accuracy of the boundary regions, particu- larly in tasks like drivable area layout estimation, where boundary precision is crucial. We introduced an additional metric to specifically evaluate drivable area boundary quality.
15

---

We applied the Sobel operator to detect boundaries in the predicted and groundtruth BEV maps, followed by calculating the chamfer distance between them. The chamfer distance, measured in pixels. provides more precise evaluation of boundary alignment (lower values indicate better boundary matching). In Tab. A2 we compare our approach to BEVFusion and the previous SOTA method DDP for surround-vier view drivable area layout estimation. While our model achieves a modest 0.2% improvement in IoU over DDP, the Chamfer distance decreases by an average of 0.24 pixels. This improvement indicates that our method estimates the drivable area boundaries more accurately than the previous methods, further supporting the effectiveness of our approach for BEV map estimation.
Normalized IoU. Considering the ped. cross., walkway, stopline, carpark and divider is much fewer than the drivable area, we compute average performance based on normalization of the raw pixels. i.e. Le.Nommalieed IoU. The results in Tab. A2 shows our method still maintains an advantage.
Evaluation in test set. In earlier BEV semantic segmentation works 3 39], pixel-wise segmentation was applied to map both background static objects and foreground dynamic objects. However, because the nuScenes test set does not provide ground truth for dynamic objects, these works could not use it to evaluate their methods. To maintain consistency with prior work, we adopt the same dataset split as LSS K for the surround-view task, using the validation set for evaluation. For the monocular task, we follow the setup of the pioneering work PON [39] whose code provides a calibration set for tuning also tested the aff-the-shelf trained models from BEVFusion, DDP and our approach on the nuScenes test set to further show the superiority of our approach (although none of the previous works conduct the comparison on the nuScenes test set). Our approach in Tab. A2 consistently outperforms DDP and BEVFusion by large margins like in Tab. [
and uses a re-divided validation set for evaluation. We
A About the realism (MMD) results. MMD is a metric of distance between the generated layout predictions and the ground truth distribution, which is capture the realism in scene structures [17]. It is noteworthy that this realism metric and the common used precision metric are not closely coupled. It is possible to achieve higher IoU while generating non-realistic map layouts, of MapPrior in nuScenes validation set is 28.4 [17]. We provide the MMD comparison between BEVFusion, DDP and our approach under the camera-only setting in Tab. [22 which is conducted on both the nuScenes validation set and test set. It shows that the generative prior models can enjoy the better structure preservation ability, and our approach VQ-Map the best at pushing the limit of both the precision and realism simultaneously.
vice versa. MMD
About providing the uncertainty awareness (ECE) results. In MapPrior, its generative stage after the predictive stage during inference introduces a discrete latent code z' to guide the generative synthesis process through transformer-based controlled synthesis in the latent space, where multiple diverse (R) are obtained using nucleus sampling and then decoded into multiple output samples (see Figs. 2 and 5 in (17). So it is necessary to evaluate the uncertainty awareness (ECE) score for the generated multiple layout estimation samples. As for the discriminative BEV perception baselines that are trained end-to-end with cross-entropy loss, the predictions produced by the softmax function are assumed be the pseudo probabilities to facilitate the computation of ECE score.
However, during inference of our VQ-Map, the token decoder outputs the classification probabilities for each BEV token, and our approach enables us to use these probabilities as weights to perform a weighted sum of the elements in the codebook embeddings and use it as input to the decoder to achieve one final layout estimation output. This process is similar to the 1-step version of MapPrior, in which there is no need to evaluate the ECE score as shown in Tab. of [7].
Different Attention in Token Decoder. We also tested alternative design by replacing the deformable attention in (f) with standard attention. Its mean loU 56.5, lower than that of (f), which indicates that deformable attention is more suitable for this task. We designed the token decoder module based on deformable attention thanks to its ability to model the prior positional relationship between PV and BEV for feature alignment. Deformable attention can naturally use reference points to leverage these priors, whereas standard attention cannot.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims presented in the abstract and introduction provide accurate summary of the paper's primary contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of work are discussed in detail in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition machine may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed machines and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide detailed structure of our method under the surround-vies in 3 detailed model settings and training hyperparameters in Sec. 4.1 and the settings for the monocular method in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution primarily a new machine, the paper should make clear how reproduce that machine.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets utilized in this study all sourced from open-access repositories, accessible via the provided references. Comprehensive instructions for constructing the experiments be found in Sec 4.1 and Sec. A
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
mublic/guider CoodeSubmissioooollee
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips wttpccccccppppceeeeeeeeeeeeppeee
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, inediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed description of the implementation details in Sec. A
Aand
and
Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We focus qualitative analysis of data and assessing model performance rather than quantifying the statistical significance of specific results. Nonetheless, we provide detailed descriptions of experimental outcomes and offer comprehensive data and qualitative analyses to ensure readers fully grasp our research findings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide this information in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We carefully review the NeurIPS Code of Ethics and ensure that the research aligns with it in all aspects.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts of this paper in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic machine for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper concentrates on the BEV map layout estimation, which does not present such high risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately acknowledge the creators or original owners of all assets utilized in the paper and provide citations the relevant papers for each asset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular souree (e.g.. website), the copyright and of service of that source should be provided.
If assets are released, the license, copyright information, and s of use in the package should be provided. For popular datasets, ppaerswithcodee com/datasets has curated licenses for some datasets. Their licensing guide can help deine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not yet released new assets proposed in this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "KLcry3gcih", "reviewer": "Reviewer_1hbx", "summary": "The paper proposes a novel approach, VQ-Map, for BEV map layout estimation, addressing the challenges of occlusion and low-resolution images in perspective views. By leveraging a VQ-VAE-like generative model, the authors introduce BEV tokens to bridge the gap between sparse image features and dense BEV representations. The method demonstrates strong performance on both nuScenes and Argoverse benchmarks, setting new state-of-the-art results.", "questions": "How about the proposed method using other attention instead of the deformable attention? \n\n\n\nWhat is the effect of the codebook size?\n\n\n\nIn Table 1, what is the reason to make two different settings for the experiment results in nuScenes validation set?", "limitations": "The proposed methods are based on the previous methods like DDP or DiffBEV. The technical contributions are not very novel, like the use of the deformable attention.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper clearly identifies the challenges in BEV map layout estimation and provides a well-defined solution.\n\n\nThe proposed method is based on the use of generative models like VQ-VAE for BEV map layout estimation and aims to address the limitations of existing methods.\n\n\nThe proposed method achieves good results on nuScenes and Argoverse benchmarks.", "weaknesses": "The proposed idea of using generative model is based on the previous methods like DDP or DiffBEV. \n\n\nThe proposed method like toke-based decoder is not very novel for the feature alignment. For example, the deformable attention is used and explored by many other methods for feature alignment. \n\n\nThe paper focuses on two specific datasets. The comparison with more recent diffusion methods is not included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you recognized that our paper clearly identifies the challenges in BEV map layout estimation and provides a well-defined solution with a novel approach VQ-Map proposed. We provide responses to the specific points below:\n\n**About the idea of using generative model**: DDP and DiffBEV propose an end-to-end framework to generate a more comprehensive BEV representation and denoise noisy samples by *applying the diffusion model* to BEV perception. Although we share the similar idea of using generative model, our approach achieves better performance with much lower MACs (see Tab. R1 in the attached PDF for the comparison with DDP) as the diffusion process may introduce a significant computational overhead. We will clarify this point in the revision.\n\n**About the novelty with regard to our token-based decoder and using other attention**: In the main paper, we have pointed out (Lines 41-43) that our token decoder module can be an arbitrary transformer-like architecture to be compatible with our novel pipeline VQ-Map. Following the common practice, we design this token decoder module based on deformable attention thanks to its ability to model the prior positional relationship between PV (Perspective View) and BEV (Bird's Eye View) for feature alighment. Deformable attention can naturally use reference points to leverage these priors. Following your suggestion, we also consider other two kinds of attention in our token-based decoder for ablation comparison on the nuScenes validation set, i.e., replacing the deformable attention with (1) the standard attention and (2) the cross view attention proposed in CVT[37], respectively. We experimentally find that the deformable attention performs better than the above considered alternatives as shown in the table below. We will include this new ablation comparison in the revision.\n\n| **Attention** | Deformable | Standard (1) | Cross View (2) |\n| ------------- | ---------- | ------------ | -------------- |\n| **mIoU**      | 61.8       | 56.5         | 49.0           |\n\n\n**About the comparison with more recent diffusion methods**: In Sec. 4.2 of our main paper, we have conducted the comparison with the most recently published diffusion-based BEV map layout estimation works, i.e., DDP[19] and DiffBEV[20] published in ICCV'2023 and AAAI'2024 respectively. In the surround-view comparison in Tab. 1, we report the results of DDP[19] using its camera-only modality results published in the original paper. In the monocular comparison in Tab. 2, we report the published results of DiffBEV[20] based on its original paper. In Tab. R1 of the attached PDF, we have also presented a computational complexity comparison with DDP to better show the superiority of our approach. Note that we are not able to conduct the computational complexity comparison with DiffBEV due to the fact that its publicly available code is not complete. We will detail the analysis in comparison to DDP and DiffBEV in the revision.\n\n**About the effect of codebook size**: We have conducted the experiments with different codebook size $K\\times D$, and the results (shown in the table below) show that our proposed method is insensitive to the size of the codebook in a wide range. We will include this analysis in the revision.\n\n| $K~(D=128)$ | 128  | 256  | 512  |\n| ------------ | ---- | ---- | ---- |\n| **mIoU**     | 61.5 | 61.8 | 61.7 |\n\n| $D~(K=256)$ | 64  | 128  | 256  |\n| ------------ | ---- | ---- | ---- |\n| **mIoU**     | 61.6 | 61.8 | 61.2 |\n\n**About the different IoU threshold settings for different methods compared in Tab. 1**：Most of previous methods publish the results following the evaluation protocol in BEVFusion[1], where the threshold that maximizes IoU is used, while MapPrior[17] uses the same IoU threshold of 0.5 across all the background classes for the BEV map layout estimation. We believe using a constant IoU threshold for all the background classes in our approach can deliver a more fair comparison with the existing approaches without suppressing the performance of the methods that use the threshold maximizing IoU. Note that our approach using the threshold maximizing IoU achieves 62.3 mIoU in comparison to 62.2 in Tab. 1. We will clarify this point in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nqMY43Nend", "reviewer": "Reviewer_rd8m", "summary": "This paper proposes to use a generative model to encode BEV semantic maps into tokenized sparse BEV representations with codebooks.\nSpecifically, it consists a two stage training scheme. First, train a BEV generation VQ-VAE, then use the BEV Tokens as a ground truth to train the second stage network where it maps the surround view images into BEV map. The second stage reuse the first stage BEV generator. Experiments show the performance surpass the state-of-the-art by 0.2% - 3% in IoU, on average improving 2.8% IoU on nuScenes.", "questions": "as above", "limitations": "This paper cannot handle small regions but might be very important to realistic AD environment, where holes and some random obstacles can be dangerous to the vehicle. \nCompared to other method that can achieve multiple task at the same time, this is a specialized method only for BEV map generation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ This paper proposes a complicated pipeline to generate BEV maps, using a pre-trained VQ-VAE as a intermediate representation then trains a two stage network to firstly encode features from perspective view images into the BEV map.\n+ The idea of using a pre-trained network as codebook seems quite interesting in the BEV map generation domain", "weaknesses": "Q1. On average, is IoU a good metric for measuring the performance of BEV map? Considering the road can be a continuous structure, probably one of the most important aspect to measure the drivable area is its boundary, is this faithfully reflected in the IoU metric?\n\nQ2. I think the mean IoU increase is summing over the different settings, but not in terms of their raw pixel right? for example, in Table 1, there are 6 factors, f1 to f6, where each f is computed based on their raw pixels. Then the mean is computed by sum{f1, ..., f6} / 6. However, considering the Walkway, stopline, carpark and divider is much fewer than the drivable area, can you compute the average performance on based on normalization of  their raw pixels as well?\n\nQ3 Complexity of the cost\n\nCan the authors list their methods complexity? Since it consists of two stage training and might introduce many overhead. Though it is not a big deal, but it is also interesting to the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you find our idea of using a pre-trained codebook interesting in the BEV map generation domain. We provide responses to the specific points below:\n\n**About the IoU metric for measuring the performance of BEV map**: The common practice of using the IoU metric to reflect the overlap between the models' predictions and the groundtruths can indicate the overall accuracy of the predictions to a certain extent. However, we agree with you that the drivable area boundary is also one of the most important aspects to measure the accuracy of the predictions, and the IoU metric may not faithfully reflect the accuracy of the drivable area boundary. Inspired by your valuable comments, we thus propose to use the Sobel operator to detect the boundaries of the models' predictions and evaluate the boundary quality using the Chamfer distance metric in pixels (the smaller, the better). We compare our method with BEVFusion and the previous SOTA aproach DDP for the surround-view drivable area layout estimation in the table below. Compared to DDP, although our IoU score for the drivable area only achieves 0.2% performance gain, the Chamfer distance for our approach decreases by an average of 0.24 pixels, indicating that our approach can estimate the drivable area layout with more accurate boundary. We will include this new analysis in the revision.\n\n| Metric                          | BEVFusion | DDP  | VQ-Map   |\n| ------------------------------- | --------- | ---- | -------- |\n| Cham. Dis.$\\downarrow$ (Pixels) | 2.67      | 2.45 | **2.21** |\n| IoU$\\uparrow$ (%)               | 81.7      | 83.6 | **83.8** |\n\n**About computing the average performance based on normalization of the raw pixels**: For each background class in the map layout estimation, an IoU score is evaluated based on the raw pixels and reported in percentage. The common practice of computing mIoU is to take the average of the IoU scores across all the evaluated background classes as you have pointed out. Following your suggestion, we compute the average performance (Normalized IoU) over all the evaluated background classes based on normalization of the raw pixels as well and show the comparison with BEVFusion, MapPrior and DDP in the table below. We will include this new analysis in the revision.\n\n|                              | BEVFusion | MapPrior | DDP  | VQ-Map   |\n| ---------------------------- | --------- | -------- | ---- | -------- |\n| Normalized IoU$\\uparrow$ (%) | 70.6      | 70.6     | 72.9 | **74.2** |\n\n**About the complexity of the cost**: Following your suggestion, we have added a computational complexity comparison using the number of parameters, MACs, and the training time. Please refer to Tab. R1 in the attached PDF for “global” response. It clearly shows that our approach not only demonstrates strong performance (acknowledged by Reviewer `1hbx`), but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion. We will include this computational complexity analysis in the revision.\n\n**About handling small regions**: As acknowledged by Reviewer `xhZa`,  tokenization in our approach is more robust against noise and geometric changes, which however may lose some information or detailed spatial information. So our approach cannot handle small regions well. This may be the inherent limiation of our approach due to the No Free Lunch rule. When the dangerous holes and some random obstacles appear in the realistic AD environment, an anomaly detection module may be needed to handle this situation. We will detail this discussion in the limitation section.\n\n**About achieving multiple tasks at the same time**: Although our approach is a specialized method only for BEV map generation at present, we believe the token-based multi-task modeling for autonomous driving is very promising. Additionally, tokenized intermediate results are well-suited for combining with large language models. We will detail this discussion in the conclusion and leave the token-based multi-task modeling in the future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FlawVf4Ct3", "reviewer": "Reviewer_xhZa", "summary": "The authors propose to use a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to obtain prior knowledge for high-level Bird’s Eye View (BEV) semantics in a tokenized discrete space. By leveraging BEV tokens and a codebook embedding that encapsulates the semantics for different BEV elements, the proposed method aligns image features with the BEV tokens obtained from discrete representation learning in a token decoder. The experiments are conducted on two benchmarks, nuScenes and Argoverse.", "questions": "My biggest concern is the dataset split for this work. I expect the authors can explain this clearly.", "limitations": "N/A", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ The paper is well-written and very easy to follow. The figures are also very intuitive, significantly facilitating other researchers to follow and understand the gist.\n\n+ The visualization of the proposed method helps readers understand how the proposed method works. \n\n+ In fact, I quite like the tokenization idea since tokenization may be more robust against noise and geometric changes. However, tokenization may lose some information or detailed spatial information. Considering the output of this task, the output layout is much coarser than the input though. So tokenization may work well.", "weaknesses": "At a glance, the proposed method achieves very good performance. In the experimental part, the proposed method outperforms the competing method. However, there are two main issues in the experiments:\n\n**The competing methods are not state-of-the-art**\nIf we look at the reference [17] published in ICCV 2017, that is the state-of-the-art method (The same task and the same dataset). Moreover, the code of [17] is also publicly available. Comparison with reference [19] which is a generative model is not compelling. Based on the reported results of [17], the results of the proposed method are much worse than that of [17].\n\n**The dataset split of nuScene is not right**\nIn nuScene, there are 700 scenes for training, 150 scenes for validation and 150 scenes for testing. In this paper, (L197-198) there is no testing dataset. The authors mentioned to validate the performance of the proposed method. It is not clear whether the authors use the validation set or testing set. Moreover, how do the authors select the model weights if a validation set is not provided?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback! We are happy that you quite like our tokenization idea for the BEV map layout estimation task. We provide responses to the specific points below:\n\n**About the state-of-the-art comparison with MapPrior[17] and DDP[19]**: Both of the two references of MapPrior[17] and DDP[19] are published at ICCV'2023, and evaluate their approaches in both the camera-only setting and multi-modality setting (with LiDAR information combined). In both settings, the mIoU of DDP always surpasses MapPrior, i.e., 59.4 vs 56.7 in the camera-only setting and 70.6 vs 63.1 in the multi-modality setting. In Tab. 1 of our state-of-the-art comparison, we report the results for MapPrior[17] and DDP[19] only using their published results under the camera-only setting, because our state-of-the-art comparison is completely based on the camera-only setting. *We are very sorry for not clearly introducing the camera-only setting used in our main paper.* We will remendy this misleading issue in the revision. In Tab. 1, it clearly shows that our VQ-Map can achieve the best performance in comparison to other entries under the camera-only setting. We will also delve into how to incorporate data from other modalities (like the LiDAR) into our framework in the future work.\n\n**About the dataset split of nuScenes in the comparison**: In the much earlier BEV semantic segmentation works, e.g., LSS[3] published at ECCV'2020 and PON[39] at CVPR'2020, the pixel-wise segmentation is conducted for the layout of both the static objects in the background and dynamic objects in the foreground. However, the nuScenes *test* set does not provide the groundtruth for the dynamic objects in the foreground. So these works directly use the nuScenes *validation* set to validate the performance of their proposed approaches. To align with these pioneering works and make the fair comparison, the subsequent works shown in Tabs. 1 and 2 all keep using this dataset split setting by default, where the models are evaluated on the nuScenes *validation* set. More specifically, for our evaluation in the surround-view experiments in Tab. 1, only the *training* set is used to train our model and the number of training epochs and total iterations is aligned with the previous work BEVFusion. The model weights from the last epoch are used for final evaluation on the nuScenes *validation* set. For our evaluation in the monocular experiments in Tab. 2, we follow the practice of the pioneering work PON, which re-divides the (*training*, *validation*) split into (*training*, *calibration*, *validation*) split, where the *calibration* set is used to adjust the hyper-parameters, and the *validation* set is used for evaluation. We will clearly explain this dataset split setting in the revised version.\n\nFurthermore, to ease the concern with regard to the nuScenes *test* set, where the groundtruth for the layout of the static objects in the background is available, we directly tested the off-the-shelf trained models from BEVFusion, DDP and our approach on the nuScenes *test* set to further show the superiority of our approach (although none of the previous works conduct the comparison on the nuScenes *test* set). As shown in the table below, our approach consistently outperforms DDP and BEVFusion by large margins like in Tab. 1.\n\n|                    | BEVFusion | DDP  | VQ-Map   |\n| ------------------ | --------- | ---- | -------- |\n| mIoU$\\uparrow$ (%) | 63.9      | 67.2 | **70.2** |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aYIgU1DOiN", "reviewer": "Reviewer_1JdE", "summary": "To address challenges facing generating BEV maps posed by occlusion, unfavorable imaging conditions, and low resolution, this paper introduces a generative model to facilitate BEV estimation. A codebook embedding is used to encoder prior knowledge for the high-level BEV semantics in the tokenized discrete space. Compared with other methods leveraging dense features to supervise BEV maps, this paper takes this discrete semantic feature as a signal to supervise BEV tokens learned from PV views. In a word, the article is logical and clear in writing. Moreover,  it has excellent results and sufficient ablation experiments.", "questions": "1. The motivation of this paper is unclear. Why do sparse features work better than dense features, or do they work better together?\n2. In 3.2, the description is not clear. Does this part need supervision? If so, how do you implement the supervision method?\n3. The results of Table 3 show that 8-layer and 512-dimensional  are a better choice. Doubtly, compared to other methods (6-layer and 256-dimensional),  the proposed method has better results due to the increased number of layers and dimensions. The comparison is somewhat unfair. Would you consider discussing this? \n4. Would you consider presenting more computational complexity results like FLOPs/MACs, the number of parameters, and running time when comparing your proposed method against existing methods in the experiment tables, e.g., Table 1 and Table 2?\n5. Would you consider presenting a set of visual comparisons to directly compare your proposed method against MapPrior? This can better show the advantage of your proposed solution.", "limitations": "This paper has well analyzed the limitations and broad implications of the study.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is well-written and easy to understand. Moreover, the experimental data is rich. Both surround-view and monocular map estimation tasks are assessed. The proposed model has superior accuracy results compared to other methods. The visualization result of the BEV codebook is quite interesting.", "weaknesses": "In the experiment, the parameters of the ablation method and the table were not explained clearly. \n\nIn Table 4, the difference between (a), (b) and (c) is not clearly described. \n\nThe specific meaning of 'Supervision' in (d) - (f) is not described, which should be added.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and feedback!\n\n**About clearly explaining the parameters of the ablation method and the table**: We apologize for not providing more details for the ablation methods in Tab. 4. The parameters that vary across the different ablation methods mainly include the layer and dimension number settings that depend on whether our deformable attention-based architecture is used or not, and the feature spatial size and dimension that depend on whether the dense or sparse BEV features are used. The supervision signals for training PV-BEV alignment also vary across the different ablation methods. We have included more explainations for each of them when asking the subsequent questions. We will also integrate the clarification for them in the revision.\n\n**About clearly describing the difference between (a), (b) and (c)**: The specific structures of (a), (b) and (c) are briefly described in Lines 276-281. We are sorry for not describing them clearly. Columns (a) and (b) both use the dense BEV features generated from the BEVFusion method while not using the attention-like architecture like in our token decoder module. As (a) directly predicts the BEV maps, it can be seen as a variant of BEVFusion. (b) uses dense features to predict the sparse BEV tokens and finally generate the BEV maps. (c) uses the sparse features generated from our proposed deformable attention-based architecture to directly predict the BEV maps like the traditional end-to-end methods. We will include more details for them in the revision.\n\n**About clearly describing the specific meaning of 'Supervision' in (d), (e) and (f)**: This part of the experiment explores which kind of intermediate results of our discrete representation learning based on VQ-VAE (see Sec. 3.1) is best for the second stage training of PV-BEV alignment. The supervision signals shown in columns (d), (e) and (f) are the explored three kinds of intermediate results. The differences are: (d) uses latent variables that *have not* been discretized by the codebook, (e) uses the latent variables that *have* been discretized by the codebook, and (f) uses the codebook indices. It shows that token classification (f) is the most effective method for PV-BEV alignment. We will add this description into the caption of Tab. 4.\n\n**About the motivation of using sparse features in our token decoder**: In the second stage training for PV-BEV alignment in our framework, the BEV tokens generated from the BEV groundtruth maps are used as the supervision signals as illustrated in Fig. 2. Whether using the dense or sparse BEV features for token decoder can be experimentally determined. We first found that using dense features with 128 $\\times$ 128 $\\times$ 80 dimensions to predict the sparse BEV tokens in (b) even performs worse than the traditional end-to-end method in (a). Considering that using sparse features to align with our tokenization idea may be more robust against noise and geometric changes (as acknowledged by Reviewer `xhZa`), we thus tested using sparse features with 25 $\\times$ 25 $\\times$ 512 dimensions to predict the sparse BEV tokens based on our proposed deformable attention-based architecture, which significantly improves the overall performance (comparing (f) to (b)). We also surprisingly found that our achieved sparse features work better even in the traditional end-to-end framework (comparing (c) to (a)) for the BEV map estimation task.\n\nDue to the failure of using dense features in our experiments, it may be hard to get better performance when using sparse and dense features together (the performance degrades to 61.3 mIoU). However, we experimentally find that training the sparse features to predict tokens can improve the training of dense features which are used to directly predict the maps. Its performance increases from 56.4 to 57.7 mIoU.\n\n\n**About the description in 3.2**: This part is designed for predicting BEV tokens, and the supervision signals during the second stage training for PV-BEV alignment in our framework are the BEV tokens generated from the BEV groundtruth maps as illustrated in Fig. 2. We cast it as a classification task and use the focal loss (Lines 188-189) to train the backbone FPN and token decoder. This part does not involve any additional supervisions. We will clarify this point in the revision.\n\n**About the different layer and dimension number settings in the ablation experiment of Tab. 4**: In the ablation experiment of Tab. 4, columns (a) and (b) both use the dense BEV features generated from the BEVFusion method (not using the attention-like architecture like ours), so comparing our best result 62.2 mIoU using 8-layer and 512-dimensional settings in column (f) to them may be OK. However, columns (c) (d) (e) and (g) all use 6-layer and 512-dimensional settings, it may be not appropriate to compare (f) with them. We apologize for this confusion. We will replace the results for column (f) with 61.8 mIoU results using 6-layer and 512-dimensional settings (see Tab. 3) in the revision.\n\n**About more computational complexity results**: Following your suggestion, we have added a computational complexity comparison using the number of parameters, MACs, and the training time. Please refer to Tab. R1 in the attached PDF. It clearly shows that our approach not only demonstrates strong performance (also acknowledged by Reviewer `1hbx`), but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion. We will include them in the revision.\n\n\n**About more visualization comparisons against MapPrior**: Thank you for your suggestion. We show more visualization comparisons against MapPrior and DDP in Fig. R1 of the attached PDF, which can better show the advantage of our proposed solution. We will include them in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you for the valuable reviews pointing out that our *novel*  (`1hbx`) approach and the idea of a pre-trained codebook are particularly *interesting* (`rd8m`), and the visualization adds depth to the concept (`1JdE`). The tokenization approach we proposed appears to be *more robust against noise and geometric changes* (`xhZa`). We are also pleased to hear that the paper is *well-written* (`xhZa, 1JdE`). Moreover, the ablation experiments were found to be *sufficient* (`1JdE`), with excellent results that *set new state-of-the-art* (`rd8m, 1hbx`). Prompted by the insightful reviews, we mainly present the following additional experimental results and analyses for the common questions:

- Following the suggestion by Reviewer `rd8m`, we have rethought the evaluation metrics for the BEV map estimation task and additionally explored the boundary quality of the drivable area. The experimental results demonstrate that our method can achieve higher quality boundaries.

- Following the suggestion by Reviewers `rd8m, 1Jde, 1hbx`, we have added a computational complexity comparison using the number of parameters, MACs, and the training time. Please refer to Tab. R1 in the attached PDF. It clearly shows that our approach not only demonstrates strong performance (acknowledged by Reviewer `1hbx`), but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion.

- Following the suggestion by Reviewers `1hbx, 1Jde`, we have added additional visualization comparisons (please refer to Fig. R1 in the attached PDF), including comparisons with MapPrior and DDP. Visualization results demonstrate that our method has significant advantages under unfavorable imaging conditions, such as rainy and night scenes.

- Regarding the concern about dataset split of nuScenes raised by Reviewer `xhZa`, we further supplement our experiments under the surround-view settings to ease the concern with regard to the nuScenes *test* set. We directly tested the off-the-shelf trained models from BEVFusion, DDP and our approach on the nuScenes *test* set to further show the superiority of our approach (although none of the previous works conduct the comparison on the nuScenes *test* set). Our approach consistently outperforms DDP and BEVFusion by large margins.

Meta-review: This paper presents a novel approach for Bird's-Eye-View (BEV) map layout estimation using a generative model inspired by Vector Quantized-Variational AutoEncoder (VQ-VAE). The paper received 4 reviews with mixed paper ratings: 2 Weak Accepts, 1 Borderline Accept and 1 Reject. The reviewers pointed out the following advantages for this paper: novel approach, SOTA performance, clear presentation, and sufficient experimental justification. The authors provided a thorough and thoughtful rebuttal, addressing each of the reviewers' concerns. They also provided additional experiments in the rebuttal, including comparisons with other attention mechanisms, evaluations using different codebook sizes, and computational complexity analysis. Reviewer xhZa gave the Reject suggestion and the main concerns raised by the reviewer are the experimental comparison with SOTA methods and the data split of nuScene dataset in the experiment. The authors addressed these two issues well in the rebuttal, but the reviewer didn't respond to the rebuttal. In consideration of all reviewers suggestions after the rebuttal, the AC suggests to accept the paper.
Decision: Accept (poster)