VQ-Map: Bird's-Eye-View Map Layout Estimation in Tokenized Discrete Space via Vector Quantization

Authors: yiwei  zhang
jin  gao
  fudong ge
guan  luo
bing  li
zhaoxiang  zhang
haibin  ling
weiming  hu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 06:11:46

Abstract: Bird's-eye-view (BEV) map layout estimation requires an accurate and full understanding of the semantics for the environmental elements around the ego car to make the results coherent and realistic. Due to the challenges posed by occlusion, unfavourable imaging conditions and low resolution, \emph{generating} the BEV semantic maps corresponding to corrupted or invalid areas in the perspective view (PV) is appealing very recently. \emph{The question is how to align the PV features with the generative models to facilitate the map estimation}. In this paper, we propose to utilize a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to acquire prior knowledge for the high-level BEV semantics in the tokenized discrete space. Thanks to the obtained BEV tokens accompanied with a codebook embedding encapsulating the semantics for different BEV elements in the groundtruth maps, we are able to directly align the sparse backbone image features with the obtained BEV tokens from the discrete representation learning based on a specialized token decoder module, and finally generate high-quality BEV maps with the BEV codebook embedding serving as a bridge between PV and BEV. We evaluate the BEV map layout estimation performance of our model, termed VQ-Map, on both the nuScenes and Argoverse benchmarks, achieving 62.2/47.6 mean IoU for surround-view/monocular evaluation on nuScenes, as well as 73.4 IoU for monocular evaluation on Argoverse, which all set a new record for this map layout estimation task. The code and models are available on \url{https://github.com/Z1zyw/VQ-Map}.

Main paper: 
 introduction: 
 BEV layouts represent high-dimensional structured data that encompasses significant prior knowledge, particularly regarding road structures. While current methods for BEV map layout estimation mainly focus on constructing dense BEV features [2, 1, 3] for semantic segmentation as map prediction, they often overlook the incorporation of map prior knowledge. Additionally, occlusion and inherent challenges in depth estimation often lead to inaccuracies in dense features, especially in the areas that are corrupted or invalid in the PV. These factors contribute to incoherent and unrealistic BEV layout results, often with numerous artifacts (see Fig. 1 ). Yet, humans can rely solely on partial observations of a scene in the PV to imagine the entire coherent BEV layout elements. A natural Figure 1 : We showcase the prediction results in various environmental conditions (day, rainy and night from top to bottom). Our VQ-Map produces more reasonable results, even for areas that are not directly visible, while significantly reducing artifacts. Color scheme is the same as in [1] .approach to imitating the human imagination process is to leverage generative models to learn the prior knowledge from the groundtruth BEV map layouts. However, the question is how to align the PV features with the generative models to facilitate BEV map estimation.To this end, we propose a novel pipeline called VQ-Map (see Fig. 2 ), which aligns the generative models well in the spirit of discrete tokens. In specific, VQ-Map utilizes a generative model similar to VQ-VAE [4] to encode the groundtruth BEV semantic maps into tokenized, discrete and sparse BEV representations, termed BEV tokens, accompanied with a discrete embedding space (i.e., the codebook embedding). Each BEV token is the index of the nearest neighbor in the codebook embedding for an encoded BEV patch feature, representing the high-level semantics of a BEV patch. BEV tokens serve as a new classification label to directly supervise the PV feature learning via a specialized token decoder in our pipeline. The training of the generative model and the token decoder module is separated. By aligning with the sparse BEV tokens, our token decoder module is able to rely solely on sparse backbone features directly queried by token queries for BEV token prediction using an arbitrary transformer-like architecture [5] [6] [7] . Simultaneously, directly employing these sparse features for token prediction bypasses the challenges of building accurate dense BEV features in common practice. The predicted tokens can be integrated into BEV embeddings through the off-the-shelf codebook embedding for generating the final high-quality BEV semantic maps. This process is similar to the human brain's memory mechanism [8] , where the targets (BEV map layouts) are encoded into highly abstract, sparse representations (BEV embeddings) through memory neurons (BEV tokens) that can be activated by specific visual signals (generated based on token queries).We evaluate our proposed VQ-Map on both the surround-view and monocular map estimation tasks, and our method sets new records in both tasks, achieving 62.2/47.6 mean IoU for surroundview/monocular evaluation on nuScenes [9] , as well as 73. 4 IoU for monocular evaluation on Argoverse [10] .In summary, our contributions are as follows: (1) We propose a novel pipeline VQ-Map exploring a discrete codebook embedding to generate high-quality BEV semantic map layouts. The acquired prior knowledge subsequently helps to effectively align the sparse backbone image features with the generative models based on a specialized token decoder, leading to more accurate BEV map layout estimation with generation. (2) By formulating map estimation as the alignment of perception and generation, our achieved BEV codebook embedding serves as a bridge between PV and BEV, and can be used in the off-the-shelf manner. (3) Extensive experiments show that our VQ-Map establishes new state-of-the-art performance on camera-based BEV semantic segmentation. Meanwhile, we confirm that as a PV-BEV alignment method, token classification is more effective than value regression. 
related work: 
 BEV Map Layout Estimation. Most existing approaches treat BEV map layout estimation as a semantic segmentation task in the BEV frame, where map elements are rasterized into pixels with each allocated multiple class labels. As a pioneer of such technology, LSS [3] explicitly predicts discrete depth distributions on image features, and then 'lifts' these 2D features to obtain pseudo 3D features, which are finally flatten into BEV features through a pooling operation. Building upon it, BEVFusion [1] introduces LiDAR point clouds and implements multi-sensor fusion within a unified BEV space, effectively maintaining semantic and geometric information. Other approaches [11] [12] [13] [14] [15] , such as VectorMapNet [12] and HIMap [15] , tackle layout issues by incorporating vectorized prior maps. Besides, TaDe [16] utilizes a task decomposition strategy to improve monocular BEV semantic segmentation performance.Recently, some methods utilize generative model-based technologies to enhance the performance of BEV map layout estimation. MapPrior [17] employs a generative map prior built on VQ-GAN [18] architecture to capture the detailed structure of traffic scenarios on the basis of conventional discriminative models, achieving a unified advantage in precision, realism and uncertainty awareness. Furthermore, DDP [19] and DiffBEV [20] focus on integrating the denoising diffusion process [21] into contemporary perception frameworks, exhibiting outstanding performance.The above mentioned work MapPrior [17] and TaDe [16] both approach the BEV map segmentation task through two stages: a perceptual stage and a generative stage, which is relevant to our work. However, MapPrior aligns with the generative model by deriving complex BEV variables, which are constrained by the challenges of acquiring accurate dense BEV features. As for TaDe, training the generative model based on polar inverse-projected BEV groundtruth maps results in the loss of certain prior knowledge embedded in conventional BEV maps, making it prone to artifacts. In contrast, our method aligns the generative model with tokenized discrete representations, which are more meaningful and easier to predict, while also preserving BEV map prior knowledge.Tokenized Discrete Representation. VQ-VAE [4] innovatively employs codebook mechanisms to establish an encoder-decoder architecture in a tokenized discrete latent space, capturing and representing richer and more complex data distributions. Following this approach, other generative models such as VQ-GAN [18] , DALL-E [22] and VQ-Diffusion [23] also map inputs into discrete tokens corresponding to codebook entries to represent high-dimensional data. Meanwhile, some visual pre-training works [24, 25] use tokens to represent image patches and treats the prediction of masked tokens as a proxy task. Recently, UViM [26] , Unified-IO [27] and AiT [28] encode various outputs as tokens and predicts them through an auto-regressive modeling [29] , modeling a wide range of visual tasks. In this paper, we draw inspiration from the above work to predict BEV tokens for generating high-quality BEV map layouts.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of our work are discussed in detail in Sec. 5 .• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition machine may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed machines and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 We herein summarize our VQ-Map perception framework in Fig. 2 as follows. Firstly, we create the discrete representations which encapsulate the high-level BEV semantics for different BEV elements in the groundtruth maps to serve as the prior knowledge (i.e., the codebook embedding) for map generation. Secondly, we conduct the PV-BEV alignment training with the specially designed token decoder module to predict the BEV tokens associated with the corresponding groundtruth maps. Finally, we directly combine the off-the-shelf codebook embedding accompanied by the map generation decoder with the PV-BEV alignment module to predict the BEV map layouts.Similar to some visual pre-training methods [24, 25] , we formulate the discrete representation learning as the task of BEV map reconstruction via a sequence of discrete tokens to acquire prior knowledge for the high-level BEV semantics. We obtain this tokenized discrete space by employing the VQ-VAE architecture [4] , which comprises three modules: BEV Patch Embedding E, Vector Quantization Q and BEV Map Generation Decoder D. Roughly speaking, E transforms local BEV semantic patches into more abstract high-level semantics; Q then clusters the semantics derived from patch embedding to create the discrete representations; and finally, D is attached to utilize these discrete representations for reconstruction of the corresponding groundtruth maps. BEV Patch Embedding E. BEV semantic maps significantly differ from the raw images with complex scenes. They inherently represent high-level semantics annotated by humans, which eliminates the need to aggregate extensive features using heavy encoders. Specifically, we initially patchify a groundtruth BEV map M ∈ B C×H×W into a sequence of non-overlapping BEV patches M i ∈ B C×P ×P N i=1 , where B = {0, 1}, P is the patch size, C is the number of the groundtruth map layouts and N = HW/P 2 is the patch number. Our patch embedding E is simple, aiming to abstract high-level semantics z i ∈ R D from individual patches M i , where D is the embedded dimension. Fig. 3 shows some BEV patch images to visualize our discrete representation learning.Vector Quantization Q. We define a latent embedding space V ∈ R K×D as our codebook embedding, where K represents the maximum number of representations in the discrete latent space. We further denote it using the set {v 1 , v 2 , . . . , v k , . . . , v K }. Our vector quantization Q receives the continuous latent vector z c from the patch embedding and outputs discrete latent z q , termed BEV embedding, through the nearest neighbor search in the codebook. This is calculated asEQUATIONwhere ℓ 2 means L2 normalization employed for codebook lookup based on cosine similarity, as described in ImprovedVQGAN [30] . Each discrete latent can also be represented by its index in the codebook as the BEV token:EQUATION)BEV Map Generation Decoder D. We feed the BEV embeddings {z i q = ℓ 2 (v k i q )} N i=1 to our map generation decoder D by firstly reshaping them into a grid format and then reconstructing the original groundtruth BEV map followingEQUATIONTraining Loss. The overall training loss includes a VQ loss L vq based on the codebook embedding due to the non-differentiable vector quantization operation besides the reconstruction loss L re . Unlike the common practice, we additionally incorporate a loss term reflecting the patch-level data augmentations (such as small-scale rotations, translations, and resizing) to aid clustering. The VQ loss is defined as:EQUATIONwhere sg denotes stop-gradient, N aug = 3 denotes the number of patch augmentations, and zi c is the continuous latent vector derived from the augmented patch. We also use the exponential moving average (EMA) [4] to update the codebook embedding.We utilize the class-specific weighted mean square error (MSE) for computing the reconstruction loss L re , where the weight of each class for each sample is inversely proportional to the number of groundtruth pixels in that class, M c ∈ B HW and M ′ c ∈ R HW :EQUATIONand the final loss L is defined as:EQUATIONOnce the codebook embedding V is obtained from the vector quantization Q and BEV patching embedding E, the dense prediction task of BEV semantic segmentation for the map layouts, represented by M ∈ B C×H×W , can be transformed into a sparse token classification task represented by a grid M ∈ A (H/P )×(W/P ) consisting of BEV tokens (see Eq. ( 2)) as illustrated in Fig. 2 , where A = {1, 2, 3, . . . , K}.Since each token represents a BEV patch instead of a BEV pixel as mentioned in Sec. 1, the BEV tokens are significantly sparser compared to the dense BEV features in common practice and the final BEV semantic maps. Each BEV semantic token can be recognized based on the semantics at the corresponding position in the image features. So we propose a transformer-like token decoder module based on deformable attention [6] to query image semantic features at the individual position to predict the corresponding BEV token. This approach is inspired by the transformer-based objection detection methods in DETR [7] , Deformable-DETR [6] and BEVFormer [2] .Token Decoder Module. Our token decoder consists of multiple patch-level cross-attention layers, a convolutional layer and a head as shown in Fig. 4 . It takes multi-scale image backbone features from the feature pyramid network (FPN) [31] and the camera calibration as input, and outputs the predicted sparse BEV tokens for PV-BEV alignment. We use N learnable embeddings as our token queries, each associated with a specific 3D position determined via the LiDAR coordinate system. Specifically, the token reference points (or anchors in the following) are initially set in the LiDAR coordinates and subsequently projected to image space using the camera and LiDAR calibration. For each patch-level cross-attention layer, it follows the classical decoder layer architecture [5] with self-attention, crossattention and feed forward network (FFN). Initially, token queries engage in selfattention interactions, restricted to neighboring queries. Then, multiple anchors are set for each query, and the deformable cross-attention [6] samples features only from the images of surround view that the anchors can be projected onto. When the anchors of a query involve multiple images, the average of the sampled features from these images is taken [2] . Given that each query aims to capture patch-level semantics, the anchors are positioned at varying heights, widths, and depths before being projected onto the images. Finally, refined token queries are obtained through a FFN.After multiple patch-level cross-attention layer, a feature sequence is obtained and reshaped to a feature grid for predicting BEV tokens. We use a convolutional layer to integrate the neighboring sparse features in the feature grid. After the convolutional layer, there is a multi-layer perceptron (MLP) serving as the classification head to predict tokens. We utilize focal loss [32] to optimize this classification task.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 We evaluate the performance of our VQ-Map on both the surround-view and monocular BEV map layout estimation tasks and use the Intersection-over-Union (IoU) metric for evaluation.We perform the surround-view experiments on nuScenes [9] involving 1000 on-road scenes from four locations in Boston and Singapore. Each scene has a duration of approximately 20 seconds, resulting in a total of 40k samples/key-frames. Each sample is captured by six monocular cameras, covering a 360°panoramic view around the ego vehicle. For BEV map estimation, nuScenes provides manual annotations encompassing 11 layout classes. According to the prior studies [3, 1] , we train and validate VQ-Map on 700 scenes with 28130 samples and 150 scenes with 6019 samples respectively.In addition, we transform the original map layouts into the ego-centric frame through rasterization. We evaluate the IoU scores for 6 map layout classes, i.e., drivable area, pedestrian crossing, walkway, stop line, car-parking area, and lane divider. Additionally, we calculate the mean IoU (mIoU) averaged over all of them.We utilize both the nuScenes [9] and Argoverse [10] datasets for the monocular task following the common practice. In particular, Argoverse employs seven surround-view cameras for data collection. It comprises 65 training sequences and 24 validation sequences captured in Miami and Pittsburgh. Like the nuScenes dataset, Argoverse provides detailed and comprehensive semantic map layout annotations. We evalutate the IoU scores for drivable area, pedestrian crossing, walkway and car-parking area on nuScenes while only the drivable area on Argoverse.In this section, we primarily outline the experimental settings for the surround-view task on nuScenes, while leaving the settings for the monocular task in Appendix A due to the space limitation.Data Preparation. Following BEVFusion [1] , we generate the BEV semantic segmentation map within a square area surrounding the ego car, which spans from -50 meters to 50 meters along both the x and y axes. We set the segmentation resolution at 0.5 meters per pixel, which culminates in a final image output of 200 by 200 pixels in size. Considering the possibility of overlapping classes within the map, our model is designed to carry out binary segmentation across all classes. The input images are resized to 256×704. Discrete Representation Learning Details. We use a BEV patch size P = 8 and the codebook embedding of K = 256 with a dimensionality of D = 128. Each sample of the semantic groundtruth map can be divided into N = 25 × 25 = 625 patches. For BEV patch embedding, we employ three 3×3 convolutional modules without padding, with channels of 64, 128, and 128 respectively. A 2×2 max pooling layer is inserted before the final convolutional module. Following VQGAN [18] , we construct the BEV map generation decoder using 5 layers with each including 3 residual blocks. Additionally, we append a sigmoid function at the end of the decoder. Regarding to the codebook embedding optimization, we set the EMA decay to 0.99 and the epsilon value to 1e-5.Token Decoder Details. We adopt SwinT-tiny [33] as our backbone to maintain consistency with the prior work [1, 17, 19, 34, 35] . The token decoder comprises 8 patch-level cross-attention layers and a convolutional module with a kernel size of 5. Each layer has a dimension of 512, with 8 heads and 16 reference anchors sampled from 4 heights, 2 depths and 2 widths. Each anchor predicts 2 offsets for each scale of the features map. Self-attention is performed for each token interaction within a 5 × 5 region. Additionally, we set the output channel number of the FPN to 512 to align with the token decoder. During the inference stage, we also use the soft token technology [28] to represent the BEV embeddings.Training. For the discrete representation learning, we employ an initial learning rate of 1e-4 and a batch size of 16 per GPU, and conduct training on 2 NVIDIA A100 (40G) GPUs, totaling 35 GPU hours. Subsequently, for training PV-BEV alignment, we adjust the initial learning rate to 5e-5 and the batch size to 8 per GPU for training on 4 NVIDIA A100 (40G) GPUs, totaling 96 GPU hours.The training is conducted over 20 epochs, incorporating warm-up and cosine learning rate decay strategies. Additionally, we apply a weight decay factor of 0.01.We conduct the state-of-the-art comparison for the surround-view BEV map layout estimation task by comparing to several recent competitive methods as shown in Tab. 1. Among them, BEVFusion [1] is widely regarded as a standard framework that integrates three key components, i.e., a backbone module, a view transformation and a task decoder. X-Align [34] uses perspective supervision to integrate perspective predictions based on dense BEV feature for enhanced performance. Map-Prior [17] employs a generative model to enhance the perceptual outcomes derived from BEVFusion. Meanwhile, MetaBEV [35] introduces an innovative cross-attention module subsequent to the dense BEV feature extraction, which is instrumental in addressing sensor failures. Additionally, DDP [19] leverages a denoising diffusion process [21] within its decoder to achieve better precision.Tab. 1 shows that our VQ-Map sets a new mIoU performance record of 62.2 for the surround-view map estimation task on nuScenes when comparing to the above methods. In particular, VQ-Map achieves a 5.5 mIoU gain in comparison to BEVFusion, with a notable improvement of over 10 in the Stopline layout class. VQ-Map also consistently outperforms all other methods by significant margins, with a minimum mIoU gain of 2.8 without relying on any other supervisions like PV semantic segmentation or depth estimation. We provide visualization results in Fig. A3 .For the monocular BEV map layout estimation task, we replace the backbone of our VQ-Map with ResNet50 [43] to align with the previous work [39, 42] (Please refer to Appendix A for more implementation details). As shown in Tab. 2, our VQ-Map also consistently outperforms all other competitive methods that are publicly available on the nuScenes and Argoverse datasets with significant margins. In comparison to the second best approach on nuScenes GitNet [42] , we achieve a 2.4 mIoU performance gain, while a 5.1 IoU gain on Argoverse in comparison to the second best TaDe [16] . It is noteworthy that our experiments on the Argoverse [10] dataset directly exploit the off-the-shelf codebook embedding and BEV map generation decoder trained on nuScenes, indicating good transferability of our discrete representation learning. Token Decoder Parameters. The number of patch-level cross-attention layers M in the token decoder has a substantial impact on the final performance, as evidenced in Tab. 3a. We note that as M increases, the mIoU value also increases. However, the mIoU improvement becomes marginal when M increases from 6 to 8, and we thus use 8 layers by default. Additionally, we investigate the effects of setting different dimensions for the token decoder layers in Tab. 3b, which shows that the optimal dimension is 512. Note that the following ablation studies all employ 6 layers and 512 dimension.Dense vs. Sparse BEV Features. In the token decoder of our approach, the features before the classification MLP head can also be regarded as a kind of sparse BEV features with shape 25 × 25 × 512, obtained based on deformable attention. In this ablation, we compare the utilization of dense BEV features vs. sparse BEV features for predicting either the sparse tokens based on the off-the-shelf codebook embedding or directly the dense semantic maps, where the dense BEV feature is obtained through LSS [3] following BEVFusion with shape 128 × 128 × 80. 4). To predict dense maps using sparse BEV features in (c), we replace the classification MLP head in the token decoder with BEVFusion's map decoder and the groundtruth BEV map M is used for supervision. (a) can be seen as a variant of BEVFusion. Conversely, for predicting sparse tokens using dense BEV features in (b), we introduce a deformable decoder [6] after the dense BEV features that initialized with weights from BEVFusion. VQ-Map (f) predicts sparse tokens using sparse BEV features. The results in Tab. 4 show that sparse BEV features are more effective for predicting sparse tokens than dense BEV features, and our achieved sparse features work better even in the traditional end-to-end framework for the map estimation task.(a) (b) (c) (d) (e) (f) (g) Arch. Sparse Feature ✓ ✓ ✓ ✓ ✓ Codebook Embedding ✓ ✓ ✓ ✓ ✓ Supervision M {k i q } N i=1 M {z i c } N i=1 {z i q } N i=1 {k i q } N i=1 {k i q } NDifferent PV-BEV Alignment. This part of the experiment explores which kind of intermediate results of our discrete representation learning based on VQ-VAE (see Sec. 3.1) is best for the second stage training of PV-BEV alignment. Instead of using the sparse token classification task for PV-BEV alignment supervision, we also test the case of utilizing the continuous latent variables directly derived from the patch embedding as a regression task for supervision in (d). In (e), we test an alternative strategy for supervision which involves predicting the latent BEV features and obtaining the BEV embeddings through nearest neighbor search based on the codebook embedding. In both (d) and (e), MSE is employed to train the token decoder. As shown in Tab. 4, the performance supervised by BEV tokens using focal loss surpasses that supervised by latent variables using MSE. These results indicate that BEV tokens serve as a superior abstract representation for guiding PV-BEV alignment.Computational Overhead Analysis. We provide the computational overhead for our VQ-Map on the surround-view task as shown in Tab. 5, including the number of parameters, computational cost (MACs), and model training time. Decreasing the layer dimension of the token decoder from 512 to 256 resulting in a light version of our VQ-Map. Since the learned discrete representations also require a significant number of parameters, we further use a smaller architecture with K = 128, D = 64 in Q and D to achieve a tiny version of our VQ-Map. It shows that even the tiny version of our approach can still achieve comparable results to the recent SOTA methods in Tab. 1. Our approach not only demonstrates strong performance, but also saves much computational cost in comparison to the recent SOTA methods MapPrior and DDP, in both the training and testing phases. In addition, the two-stage training of our approach introduces some additional training overhead in comparison to BEVFusion. 
conclusion: 
 In this paper, we present a novel pipeline VQ-Map by aligning with the generative models using discrete BEV tokens, which efficiently enhances the performance of BEV map layout estimation. The core components of our method are the codebook embedding constructed via vector quantization as prior knowledge, serving as a bridge between PV and BEV, and the specially designed BEV token decoder to facilitate the PV-BEV alignment, both of which enable the generation of high-quality BEV semantic map layouts. We hope that our work will inspire further research on vector quantization, providing assistance not only for map estimation and its downstream tasks, but also for a wide range of other applications.Limitations and future work. A significant limitation of our approach is our inability to handle semantics that are position-sensitive and have small areas. It is challenging for patch-level semantics to effectively represent position information for small-scale semantics using the architecture similar to VQ-VAE. Tokenization in our approach is more robust against noise and geometric changes. However, it may lead to a loss of some detailed spatial information. When the dangerous holes and some random obstacles appear in the realistic AD environment, an anomaly detection module may be needed to handle this situation. Our approach is a specialized method only for BEV map generation at present, but we believe the token-based multi-task modeling for autonomous driving is very promising. Additionally, tokenized intermediate results are well-suited for combining with large language models.Broader Impact. Accurate BEV layout estimation helps drivers and autonomous driving systems better understand the surrounding environment, thereby reducing the occurrence of traffic accidents and improving traffic safety. However, it may be misused to capture detailed information about the environment, including sensitive data such as building layouts, infrastructure, and so on. Monocular Image Groundtruths VPN [41] PON [39] GitNet [42] TaDe [16] VQ-Map Figure A3 : Visualization results for monocular BEV map layout estimation on nuScenes. Unlike other methods, our VQ-Map only focuses on map layout classes. The color scheme is the same as in PON [39] .We applied the Sobel operator to detect boundaries in the predicted and groundtruth BEV maps, followed by calculating the chamfer distance between them. The chamfer distance, measured in pixels, provides a more precise evaluation of boundary alignment (lower values indicate better boundary matching). In Tab. A2 we compare our approach to BEVFusion and the previous SOTA method DDP for surround-view drivable area layout estimation. While our model achieves a modest 0.2% improvement in IoU over DDP, the Chamfer distance decreases by an average of 0.24 pixels. This improvement indicates that our method estimates the drivable area boundaries more accurately than the previous methods, further supporting the effectiveness of our approach for BEV map estimation.Normalized IoU. Considering the ped. cross., walkway, stopline, carpark and divider is much fewer than the drivable area, we compute average performance based on normalization of the raw pixels, i.e.Normalized IoU. The results in Tab. A2 shows our method still maintains an advantage.Evaluation in test set. In earlier BEV semantic segmentation works [3, 39] , pixel-wise segmentation was applied to map both background static objects and foreground dynamic objects. However, because the nuScenes test set does not provide ground truth for dynamic objects, these works could not use it to evaluate their methods. To maintain consistency with prior work, we adopt the same dataset split as LSS [3] for the surround-view task, using the validation set for evaluation. For the monocular task, we follow the setup of the pioneering work PON [39] , whose code provides a calibration set for tuning hyper-parameters and uses a re-divided validation set for evaluation. We also tested the off-the-shelf trained models from BEVFusion, DDP and our approach on the nuScenes test set to further show the superiority of our approach (although none of the previous works conduct the comparison on the nuScenes test set). Our approach in Tab. A2 consistently outperforms DDP and BEVFusion by large margins like in Tab. 1.About the realism (MMD) results. MMD is a metric of distance between the generated layout predictions and the ground truth distribution, which is to capture the realism in scene structures [17] .It is noteworthy that this realism metric and the common used precision metric are not closely coupled. It is possible to achieve higher IoU while generating non-realistic map layouts, or vice versa. MMD of MapPrior in nuScenes validation set is 28.4 [17] . We provide the MMD comparison between BEVFusion, DDP and our approach under the camera-only setting in Tab. A2, which is conducted on both the nuScenes validation set and test set. It shows that the generative prior models can enjoy the better structure preservation ability, and our approach VQ-Map is the best at pushing the limit of both the precision and realism simultaneously.About providing the uncertainty awareness (ECE) results. In MapPrior, its generative stage after the predictive stage during inference introduces a discrete latent code z ′ to guide the generative synthesis process through a transformer-based controlled synthesis in the latent space, where multiple diverse z (k) are obtained using nucleus sampling and then decoded into multiple output samples (see Figs. 2 and 5 in [17] ). So it is necessary to evaluate the uncertainty awareness (ECE) score for the generated multiple layout estimation samples. As for the discriminative BEV perception baselines that are trained end-to-end with cross-entropy loss, the predictions produced by the softmax function are assumed to be the pseudo probabilities to facilitate the computation of ECE score.However, during inference of our VQ-Map, the token decoder outputs the classification probabilities for each BEV token, and our approach enables us to use these probabilities as weights to perform a weighted sum of the elements in the codebook embeddings and use it as input to the decoder to achieve one final layout estimation output. This process is similar to the 1-step version of MapPrior, in which there is no need to evaluate the ECE score as shown in Tab. 1 of [17] .Different Attention in Token Decoder. We also tested alternative design by replacing the deformable attention in (f) with standard attention. Its mean IoU is 56.5, lower than that of (f), which indicates that deformable attention is more suitable for this task. We designed the token decoder module based on deformable attention thanks to its ability to model the prior positional relationship between PV and BEV for feature alignment. Deformable attention can naturally use reference points to leverage these priors, whereas standard attention cannot. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the broader impacts of this paper in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic machine for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper concentrates on the BEV map layout estimation, which does not present such high risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and s of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We appropriately acknowledge the creators or original owners of all assets utilized in the paper and provide citations to the relevant papers for each asset.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and s of service of that source should be provided. • If assets are released, the license, copyright information, and s of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help deine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA] Justification: We have not yet released new assets proposed in this paper.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claims presented in the abstract and introduction provide an accurate summary of the paper's primary contributions and scope. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper presents a novel approach for Bird's-Eye-View (BEV) map layout estimation using a generative model inspired by Vector Quantized-Variational AutoEncoder (VQ-VAE). The paper received 4 reviews with mixed paper ratings: 2 Weak Accepts, 1 Borderline Accept and 1 Reject. The reviewers pointed out the following advantages for this paper: novel approach, SOTA performance, clear presentation, and sufficient experimental justification. The authors provided a thorough and thoughtful rebuttal, addressing each of the reviewers' concerns. They also provided additional experiments in the rebuttal, including comparisons with other attention mechanisms, evaluations using different codebook sizes, and computational complexity analysis. Reviewer xhZa gave the Reject suggestion and the main concerns raised by the reviewer are the experimental comparison with SOTA methods and the data split of nuScene dataset in the experiment. The authors addressed these two issues well in the rebuttal, but the reviewer didn't respond to the rebuttal. In consideration of all reviewers suggestions after the rebuttal, the AC suggests to accept the paper.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel approach, VQ-Map, for BEV map layout estimation, addressing the challenges of occlusion and low-resolution images in perspective views. By leveraging a VQ-VAE-like generative model, the authors introduce BEV tokens to bridge the gap between sparse image features and dense BEV representations. The method demonstrates strong performance on both nuScenes and Argoverse benchmarks, setting new state-of-the-art results.
strength_weakness: strength_weakness: The proposed idea of using generative model is based on the previous methods like DDP or DiffBEV. 


The proposed method like toke-based decoder is not very novel for the feature alignment. For example, the deformable attention is used and explored by many other methods for feature alignment. 


The paper focuses on two specific datasets. The comparison with more recent diffusion methods is not included.
questions: questions: How about the proposed method using other attention instead of the deformable attention? 



What is the effect of the codebook size?



In Table 1, what is the reason to make two different settings for the experiment results in nuScenes validation set?
limitations: limitations: The proposed methods are based on the previous methods like DDP or DiffBEV. The technical contributions are not very novel, like the use of the deformable attention.
paper_summary: paper_summary: This paper proposes to use a generative model to encode BEV semantic maps into tokenized sparse BEV representations with codebooks.
Specifically, it consists a two stage training scheme. First, train a BEV generation VQ-VAE, then use the BEV Tokens as a ground truth to train the second stage network where it maps the surround view images into BEV map. The second stage reuse the first stage BEV generator. Experiments show the performance surpass the state-of-the-art by 0.2% - 3% in IoU, on average improving 2.8% IoU on nuScenes.
strength_weakness: strength_weakness: Q1. On average, is IoU a good metric for measuring the performance of BEV map? Considering the road can be a continuous structure, probably one of the most important aspect to measure the drivable area is its boundary, is this faithfully reflected in the IoU metric?

Q2. I think the mean IoU increase is summing over the different settings, but not in terms of their raw pixel right? for example, in Table 1, there are 6 factors, f1 to f6, where each f is computed based on their raw pixels. Then the mean is computed by sum{f1, ..., f6} / 6. However, considering the Walkway, stopline, carpark and divider is much fewer than the drivable area, can you compute the average performance on based on normalization of  their raw pixels as well?

Q3 Complexity of the cost

Can the authors list their methods complexity? Since it consists of two stage training and might introduce many overhead. Though it is not a big deal, but it is also interesting to the community.
questions: questions: as above
limitations: limitations: This paper cannot handle small regions but might be very important to realistic AD environment, where holes and some random obstacles can be dangerous to the vehicle. 
Compared to other method that can achieve multiple task at the same time, this is a specialized method only for BEV map generation.
paper_summary: paper_summary: The authors propose to use a generative model similar to the Vector Quantized-Variational AutoEncoder (VQ-VAE) to obtain prior knowledge for high-level Bird’s Eye View (BEV) semantics in a tokenized discrete space. By leveraging BEV tokens and a codebook embedding that encapsulates the semantics for different BEV elements, the proposed method aligns image features with the BEV tokens obtained from discrete representation learning in a token decoder. The experiments are conducted on two benchmarks, nuScenes and Argoverse.
strength_weakness: strength_weakness: At a glance, the proposed method achieves very good performance. In the experimental part, the proposed method outperforms the competing method. However, there are two main issues in the experiments:

**The competing methods are not state-of-the-art**
If we look at the reference [17] published in ICCV 2017, that is the state-of-the-art method (The same task and the same dataset). Moreover, the code of [17] is also publicly available. Comparison with reference [19] which is a generative model is not compelling. Based on the reported results of [17], the results of the proposed method are much worse than that of [17].

**The dataset split of nuScene is not right**
In nuScene, there are 700 scenes for training, 150 scenes for validation and 150 scenes for testing. In this paper, (L197-198) there is no testing dataset. The authors mentioned to validate the performance of the proposed method. It is not clear whether the authors use the validation set or testing set. Moreover, how do the authors select the model weights if a validation set is not provided?
questions: questions: My biggest concern is the dataset split for this work. I expect the authors can explain this clearly.
limitations: limitations: N/A
paper_summary: paper_summary: To address challenges facing generating BEV maps posed by occlusion, unfavorable imaging conditions, and low resolution, this paper introduces a generative model to facilitate BEV estimation. A codebook embedding is used to encoder prior knowledge for the high-level BEV semantics in the tokenized discrete space. Compared with other methods leveraging dense features to supervise BEV maps, this paper takes this discrete semantic feature as a signal to supervise BEV tokens learned from PV views. In a word, the article is logical and clear in writing. Moreover,  it has excellent results and sufficient ablation experiments.
strength_weakness: strength_weakness: In the experiment, the parameters of the ablation method and the table were not explained clearly. 

In Table 4, the difference between (a), (b) and (c) is not clearly described. 

The specific meaning of 'Supervision' in (d) - (f) is not described, which should be added.
questions: questions: 1. The motivation of this paper is unclear. Why do sparse features work better than dense features, or do they work better together?
2. In 3.2, the description is not clear. Does this part need supervision? If so, how do you implement the supervision method?
3. The results of Table 3 show that 8-layer and 512-dimensional  are a better choice. Doubtly, compared to other methods (6-layer and 256-dimensional),  the proposed method has better results due to the increased number of layers and dimensions. The comparison is somewhat unfair. Would you consider discussing this? 
4. Would you consider presenting more computational complexity results like FLOPs/MACs, the number of parameters, and running time when comparing your proposed method against existing methods in the experiment tables, e.g., Table 1 and Table 2?
5. Would you consider presenting a set of visual comparisons to directly compare your proposed method against MapPrior? This can better show the advantage of your proposed solution.
limitations: limitations: This paper has well analyzed the limitations and broad implications of the study.

Comments: title: Thanks
Thank Reviewer 1hbx for the reply
Comment
Questions
Thank Reviewer 1JdE for the reply
Comment
comment: I have no further question and raised my score.
Thank you for your careful review and thoughtful reply, and we are sorry for not clearly describing the specific structures and meaning of the ablation methods in Tab. 4, though they are briefly described in Lines 276-281.

**About the deformable attention in the ablation study of Tab. 4**: The supervision signals during the second stage training for PV-BEV alignment in our novel pipeline are the BEV tokens generated from the BEV groundtruth maps as illustrated in Fig. 2. It shows that token classification (f) using the codebook indices is the most effective method for PV-BEV alignment in comparison to columns (d) and (e), which are corresponding to using latent variables that *have not* and *have* been discretized by the codebook respectively. Column (g) aims to show the effectiveness of $N_{aug}$ in Eq. (4). They all use the sparse features generated from our proposed deformable attention-based architecture to predict the supervision signals. 

Whether using the dense or sparse BEV features for token decoder in our pipeline can be experimentally determined. Columns (a) and (b) both use the dense BEV features generated from the BEVFusion method while not using the attention-like architecture like in our token decoder module. We first found that using dense features with 128$\times$128$\times$80 dimensions to predict the sparse BEV tokens in (b) even performs worse than the traditional end-to-end method in (a). Note that (a) directly predicts the BEV maps and it can be seen as a variant of BEVFusion. Considering that using sparse features to align with our tokenization idea may be more robust against noise and geometric changes (as acknowledged by Reviewer `xhZa`), we thus tested using sparse features with 25$\times$25$\times$512 dimensions to predict the sparse BEV tokens based on our proposed deformable attention-based architecture, which significantly improves the overall performance (comparing (f) to (b)). We also surprisingly found that our achieved sparse features work better even in the traditional end-to-end framework (comparing (c) to (a)) for the BEV map estimation task.

Based on the above analyses, we can see that only (a) and (b) do not have the deformable attention included. If (a) uses our achieved sparse features based on the proposed deformable attention-based architecture, it becomes (c) with mIoU improved from 56.4 to 59.8. Due to the failure of using dense features (see column (b) with 56.3 mIoU) or the standard attention-based sparse features (56.5 mIoU), our novel pipeline really enjoys the benefit of deformable attention to achieve the new performance record of 62.2 mIoU, which significantly surpasses the SOTA method DDP (59.4 mIoU). Note that deformable attention has always been a common practice for PV-BEV alignment in the discriminative BEV perception approaches that are trained end-to-end with cross-entropy loss. We will include more details for them in the revision.

**About the respective improvements from different proposed components**: Our generative pipeline mainly includes two components, i.e., the deformable attention-based token decoder using sparse features and the codebook embedding for BEV generation, where some parameter settings include the layer and dimension number settings for the deformable attention-based architecture, and the supervision signals for training PV-BEV alignment after discrete representation learning. We list the respective improvements for them below:
- The deformable attention-based sparse features employing 6 layers and 512 dimensions improves over the baseline (a) to achieve (c) with 3.4 mIoU performance gain. We also tested the deformable attention-based dense features with 128$\times$128$\times$80 dimensions, which ahcieves 1.6 mIoU performance gain over (a).
- The codebook embedding using the latent variables as the supervision signals further improves over (c) to achieve (d) or (e) with 0.3-0.5 mIoU performance gains, while using the codebook indices achieves 61.8 mIoU with 2.0 mIoU performance gain over (c).
- Further changing the layer and dimension number settings to 8 layers and 512 dimensions finally achieves (f) with an additional mIou performance gain of 0.4.

We will add these respective improvements into Tab. 4 for more clear presentation.
The reviewer would like to thank the authors for supplementing these analyses, which could be added to the revised version.

Sincerely,
Thanks for the response. The rebuttal has addressed many of my concerns. 

An interesting point is that using deformable attention to replace standard attention can lead to such a large improvemence. Which of the other methods (in Tab. 4 ablation study) has or has not included deformable attention? How about if include? Apart from that, is it possible to list the respective improvements from different proposed components?
Thank you for your careful review and thoughtful reply, and we are happy to continue the discussion. The added computation complexity analysis and qualitative visualization comparison results will be integrated into the final version.

**About providing the realism (MMD) results**: MMD is a metric of distance between the generated layout predictions and the ground truth distribution, which is to capture the realism in scene structures (e.g., the lane and sidewalks that have gaps and are not straight may result in topological changes and drastically impact downstream modules). It is noteworthy that this realism metric and the common used precision metric are not closely coupled. It is possible to achieve higher IoU while generating non-realistic map layouts, or vice versa. We provide the MMD comparison between BEVFusion, MapPrior, DDP and our approach under the camera-only setting in the table below, which is conducted on both the nuScenes *validation* set and *test* set. It shows that the generative prior models can enjoy the better structure preservation ability, and our approach VQ-Map is the best at pushing the limit of both the precision and realism simultaneously. We will include this MMD-based analysis in the revision.

|            | BEVFusion | MapPrior | DDP | VQ-Map |
| --------- | --------- | --------- | --------- | --------- |  
| MMD$\downarrow$ on *validation* set | 39.6 | 28.4 | 23.7 | 24.4 |
| MMD$\downarrow$ on *test* set | 19.5 | - | 12.9 | 10.0 |


**About providing the uncertainty awareness (ECE) results**: In MapPrior[17], its generative stage after the predictive stage during inference introduces a discrete latent code $\textbf{z}’$ to guide the generative synthesis process through a transformer-based controlled synthesis in the latent space, where multiple diverse $\textbf{z}^{(k)}$ are obtained using nucleus sampling and then decoded into multiple output samples (see Figs. 2 and 5 in [17]). So it is necessary to evaluate the uncertainty awareness (ECE) score for the generated multiple layout estimation samples. As for the discriminative BEV perception baselines that are trained end-to-end with cross-entropy loss, the predictions produced by the softmax function are assumed to be the pseudo probabilities to facilitate the computation of ECE score.

However, during inference of our VQ-Map, the token decoder outputs the classification probabilities for each BEV token, and our approach enables us to use these probabilities as weights to perform a weighted sum of the elements in the codebook embeddings and use it as input to the decoder to achieve **one** final layout estimation output. This process is similar to the 1-step version of MapPrior, in which there is no need to evaluate the ECE score as shown in Tab. 1 of [17]. We will clarify this point in the revision.
The rebuttal helps to address many of the concerns. The added computation complexity analysis and qualitative visualization comparison results should be integrated into the final version. 

MapPrior also provides realism (MMD) and uncertainty awareness (ECE) results. Would it be possible to provide such results for analysis as well? This could better verify the superiority of your approach over the baseline method.

Sincerely,