MaskBEV: Towards A Unified Framework for BEV Detection and Map Segmentation

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 16:36:41

Abstract: Accurate and robust multimodal multi-task perception is crucial for modern autonomous driving systems. However, current multimodal perception research follows independent paradigms designed for specific perception tasks, leading to a lack of complementary learning among tasks and decreased performance in multi-task learning (MTL) due to joint training. In this paper, we propose MaskBEV, a masked attention-based MTL paradigm that unifies 3D object detection and bird's eye view (BEV) map segmentation. MaskBEV introduces a task-agnostic Transformer decoder to process these diverse tasks, enabling MTL to be completed in a unified decoder without requiring additional design of specific task heads. To fully exploit the complementary information between BEV map segmentation and 3D object detection tasks in BEV space, we propose spatial modulation and scene-level context aggregation strategies. These strategies consider the inherent dependencies between BEV segmentation and 3D detection, naturally boosting MTL performance. Extensive experiments on nuScenes dataset show that compared with previous state-of-the-art MTL methods, MaskBEV achieves 1.3 NDS improvement in 3D object detection and 2.7 mIoU improvement in BEV map segmentation, while also demonstrating slightly leading inference speed.

Main paper: 
 introduction: 
 Perceiving the 3D environment around a vehicle is crucial for autonomous driving systems. Lidar and cameras are widely used in autonomous driving fusion perception due to their complementary characteristics. Some object-centric methods [1, 7, 38, 40, 41, 52] have carefully designed multimodal fusion perception modules to enhance the performance of 3D object detection. However, they are difficult to adapt to multi-task requirements and lack flexibility in generalizing to other tasks. These shortcomings limit their practical [11, 28, 35] adopt independent task head design. (b) The unified multi-task head design fully exploits the complementary advantages between multiple tasks, and uses one decoder to perform MTL in the unified BEV features.application. The traditional single-task perception paradigm is gradually shifting towards multi-task learning (MTL), such as sparse 3D detection tasks and dense BEV map segmentation tasks. Based on dense bird's eye view (BEV) representations, a feasible solution is provided, which has received widespread attention due to its natural support for multi-task perception. However, experiments by [11, 28] have found that current MTL paradigms are affected by the negative transfer problem of multitasking.BEVFusion [28] proposed that joint training with a shared BEV encoder led to a decrease in MTL performance, and then mitigated the negative transfer of MTL by separating the BEV encoder during training, as shown in Fig. 1(a) . MetaBEV [11] adopted the routing multi-task mixture-of-experts technology of natural language processing (NLP) and separated BEV features to improve MTL, but its MTL accuracy is still much lower than that of single tasks. The powerful UniTR unified the image and LiDAR encoder backbones, but more importantly, these state-of-the-art (SOTA) works [11, 28, 35] still employed independent prediction head designs, such as the Transformer head for 3D detection [1, 36] and the CNN head [28, 49] for map segmentation, as shown in Fig. 1(a) . Then, MTL is achieved through a simple combination of 3D detection and BEV segmentation task heads. The design of these multitask methods leads to unnecessary increases in computational costs and performance degradation, with complementary features between tasks not being utilized. In this paper, we aim to extend the current multimodal fusion framework by designing a multi-task complementary learning decoder to construct a unified multi-task perception framework.In this paper, we introduce MaskBEV, a unified multi-task outdoor 3D perception framework. As shown in Fig. 1 (b), unlike previous task-specific perception heads, our MaskBEV is the first to achieve simultaneous perception of 3D object detection and BEV map segmentation in one decoder head. To achieve this aim, we adopt the advanced Mask2Former [9] paradigm, leveraging the complementary nature of the BEV map segmentation task and the 3D object detection task to construct a unified multi-task decoder head. Masked attention focuses attention on local features centered around potential queries. We utilize the union of multitask masks in BEV space to guide query-based feature learning. To maximize the coverage of masks over potential regions of interest while excluding the entire BEV space, we introduce a spatial modulation strategy that fully considers the geometric relationships of detection and the semantic principles of segmentation. Moreover, we propose a powerful scene-level feature aggregation module to aggregate multi-granular contextual features to serve the BEV map segmentation task better. Specifically, the module consists of two BEV feature aggregation blocks. The multi-window window-attention (MWWA) adjusts window sizes on different attention heads to aggregate multi-granularity contextual features. ASPP [5] implements scene-level global feature extraction from BEV feature maps in a convolution-based manner. Structurally, Transformer-based MWWA and convolution-optimized ASPP are complementary, and the performance gain also demonstrates the effectiveness of this module.The query-based decoding paradigm naturally fits the current 3D object detection, the mask decoder structure achieves the segmentation of the BEV map, and the query focus on foreground regions allows for better updating of queries. In summary, our main contributions are as follows:• We propose MaskBEV which is a unified perception framework for 3D object detection and BEV map segmentation tasks for the first time. The proposed multi-task decoder based on masked attention can achieve high-performance joint training. • We propose a spatial modulation strategy to assist in obtaining multi-task reliable masks and a new scene-level feature aggregation module to capture multi-granularity and even scene-level BEV contextual features. • Our MaskBEV achieves state-of-the-art performance on multitask learning (3D object detection and BEV map segmentation) on nuScenes dataset. Multiple multimodal feature encoder networks and sensor robustness analyses are also provided for a comprehensive evaluation of MaskBEV. 
related work 2.1 3d object detection: 
 3D object detection is one of the key tasks in autonomous driving perception. The performance of Lidar-only methods [18, 39, 42] and camera-only methods [14, 15, 22, 37] is limited by the deficiencies of their respective sensors. Multimodal fusion methods [1, 6, 8, 11, 21, 28, 35, 38] recently show significant effectiveness in 3D object detection. Object-centric detection methods [7, 16, 38, 41, 52] improve detection accuracy by carefully designing potential object query proposal generation and generation modules. CMT [38] enriches multimodal 3D features with coordinate encoding and designs a 3D object detection decoder head through the original Transformer decoder in DETR. SparseFusion [52] extracts sparse instance features from the multimodal and fuses them directly to obtain the final sparse instance features for detection. In addition, some methods [11, 17, 23, 28, 35] mainly use BEV representation to fuse the two modalities. BEVFusion [28] applies lift-splat-shoot (LSS) [31] operations to project image features onto BEV space and fuses Lidar BEV features in that space. Then, the improved TransFusion [1] decoder head is used for 3D detection. Current SOTA methods [11, 17, 35] focus on generating BEV features and applying this detection head. UniTR [35] achieves unified feature encoding of images and Lidar through the modality-independent Transformer encoder. It is worth noting that object-centric method [7, 38, 40, 41] is difficult to extend to the BEV map segmentation task.BEV map segmentation is the task of performing dense semantic segmentation in a bird's eye view. Influenced by the development of 3D object detection in BEV representation, BEV map segmentation [22, 31, 42] has recently received considerable attention. Such as LSS [31] achieves BEV map segmentation through ResNet-18 [13] and a multi-scale feature fusion network. Some works [20, 37] transform images into BEV views through feature projection. BEV-Former [22] , CVT [49] , BEVSegFormer [30] , and MetaBEV [11] construct BEV representations in a learnable manner [54] and they adopt a convolution-based segmentation head similar to the head of LSS. Convolution-based segmentation heads [31, 49] are widely used in current SOTA BEV map segmentation methods [11, 28, 35] . Additionally, PETR V2 [26] proposes a query-based segmentation head from the vanilla DETR [4] . These perception methods aim to transform each sensor feature into BEV space to achieve multi-task prediction, including BEV segmentation.MTL has garnered widespread attention and mutual reinforcement in both computer vision and NLP fields. Previous multitask research can be roughly divided into camera-only methods [22, 26, 37] , Lidar-only methods [18, 42] , and cross-modal fusion methods [11, 23, 28, 33, 35, 43] . Camera-only methods mostly convert multi-view cameras into BEV feature maps, and perform 3D object detection or BEV segmentation based on BEV map combined with specific task heads. Lidar-only methods extract features through the point cloud network [18, 51] and compress them in the 𝑍 -axis direction to obtain BEV representation. Some general MTL works [11, 22, 28, 35, 42, 43, 48] design unified BEV representations to achieve multi-task perception, including sparse detection tasks and dense semantic tasks. However, due to the adoption of independent task heads in each architecture, MTL performance is adversely affected by task conflicts [28] , resulting in poorer performance. Some works [11, 28] adopt separating BEV encoders to mitigate the negative transfer of multi-task joint training on each single task. In  
overall architecture: 
 In this paper, we introduce a new unified multi-task learning decoder to address the performance degradation issues of 3D object detection and BEV map segmentation in joint training. Fig. 2 illustrates the architecture of MaskBEV. Given multimodal inputs, they are encoded into tokens using a multimodal feature encoder, and then fused into a BEV space through a BEV encoder [28] . Finally, a decoder based on advanced Mask2Former [9] is used to perform various 3D perception tasks. Our main innovation focuses on the decoder module. Convert the multi-task perception results into a binary mask in masked attention, allowing the query to focus on the local region of the entire BEV map (Section 3.3). The decoder decodes segmentation predictions as Transformer-based mask classification and detection predictions into basic classification and regression. Scene-level feature aggregation fuses multi-scale features to facilitate the BEV map segmentation task (Section 3.4).BEV features can be sourced from most SOTA feature encoding backbone networks [11, 28, 35] . In our research, we take UniTR [35] as an example to process multimodal inputs to generate BEV features. Specifically, modality-specific tokenizers [10, 50] process multimodal signals to generate input token sequences for subsequent Transformer encoders. Image and Lidar tokens learn complementary features through modality-agnostic Transformer blocks based on DSVT blocks [34] . Camera and Lidar feature tokens are fused into a unified BEV space through a convolution-based BEV encoder [28] . We use the function 𝑓 (•) to represent the multimodal feature encoding process:EQUATIONwhere 𝐹 is the BEV feature, 𝐹 ∈ R 𝐶 ×𝐻 ×𝑊 , 𝐶 is the channel dimension, 𝐻 and 𝑊 are the BEV feature map sizes. 𝐹 𝐶 is the camera feature, and 𝐹 𝐿 is the Lidar feature. The BEV features are fed to the decoder for multi-task predictions.Previous SOTA works [11, 28, 35] adopt a Transformer-based 3D detection head [1] and a CNN-based segmentation head [49] to implement MTL via a simple union as shown in Fig. 1 (a). However, these methods often rely on task-specific decoders and do not consider the unified modeling and complementary effects of multiple tasks, which can enhance the performance of any single task. To this end, inspired by the advanced Mask2Former [9] decoder design, we propose the unified MTL framework MaskBEV. As shown in Fig. 2 , the region of interest for MaskBEV is only a small part of the entire BEV map. We attempt to focus the cross-attention between object queries and the BEV feature on the masks of potential tasks rather than focusing on the whole BEV. The predicted results can explicitly guide the update of query features. The self-attention between object queries in the decoder infers the pairwise relationships between different queries. Specifically, with the input BEV features 𝐹 ∈ R 𝐶 ×𝐻 ×𝑊 and a set of parameterized query features 𝑄 ∈ R 𝐶 ×𝑁 , 𝑁 is the query number. Define the anchor for each query 𝐴 𝑞 = (𝑥 𝑞 , 𝑦 𝑞 , 𝑧 𝑞 , 𝑙 𝑞 , 𝑤 𝑞 , ℎ 𝑞 , 𝜃 𝑞 ), where 𝑥 𝑞 , 𝑦 𝑞 is the center point, 𝑧 𝑞 is bounding box height, 𝑙 𝑞 , 𝑤 𝑞 , and ℎ 𝑞 is length, width and height, 𝜃 𝑞 is yaw angle. We encode the anchor of each query through a multi-layer perception (MLP) to obtain the position embedding 𝑃 𝑞 :EQUATIONwherePE(𝐴 𝑞 ) ∈ R 2𝐶 , MLP implements 𝑅 2𝐶 → 𝑅 𝐶 , 𝑃 𝑞 ∈ R 𝐶 , Cat means concatenate function.The query consists of position encoding 𝑃 𝑞 and learnable content query 𝐶 𝑞 , 𝑄 = 𝑃 𝑞 + 𝐶 𝑞 [4, 25] . This allows the network to learn context and location features simultaneously.The transformer decoder performs an iterative updating of the query features toward the desired 3D object detection and BEV map segmentation. Specifically, in each iteration layer 𝑙, the query 𝑄 𝑙 focus on their corresponding regions through masked attention:EQUATIONwhere 𝐾 𝑙 , 𝑉 𝑙 = 𝐹𝑊 𝑘 , 𝐹𝑊 𝑣 ,𝑊 𝑘 and 𝑊 𝑣 are parameters of linear projection. M 𝑙 -1 is the multi-task union mask of the attention mask from the previous layer. Specifically, the attention mask is the union of 3D objects in BEV and BEV map segmentation. However, during the training process, predictions of potential objects and regions are inaccurate, and predicted boxes cannot effectively represent the precise location of objects in BEV space. To address this, we propose a spatial modulation strategy to ensure that the attention mask covers as many objects and semantic regions as possible. Firstly, we use BEV segmentation prediction results with a threshold greater than 0.1 as the segmentation masks. For 3D objects, we use the top 200 box prediction results since the max number of objects in one frame is 142, and draw circular regions of interest with the predicted center point of the box as the center and 1.3 times the length of the box as the diameter to create object masks. Fig. 3 shows a visual example of attention masks.After each iteration, on the one hand, the feed-forward network (FFN) independently decodes 𝑁 object queries containing instance information into 3D boxes and class labels. The FFN predicts 𝛿𝑥, 𝛿𝑦, 𝑧, 𝑙𝑜𝑔(𝑙), 𝑙𝑜𝑔(𝑤), 𝑙𝑜𝑔(ℎ), 𝑠𝑖𝑛(𝜃 ), 𝑐𝑜𝑠 (𝜃 ) of 3D anchor box. And predict the per class probability (𝑝 ∈ [0, 1] 𝐾 ) of 𝐾 object semantic classes. More details of FNN are consistent with previous 3D detection paper [1] . On the other hand, each query 𝑞 𝑖 is projected to predict its semantic logits 𝑆 𝑖 and the mask embedding 𝐸 𝑚𝑎𝑠𝑘,𝑖 , 𝐸 𝑚𝑎𝑠𝑘,𝑖 ∈ R 𝐶 . Then do the dot product of 𝐸 𝑚𝑎𝑠𝑘,𝑖 with the BEV features, 𝐹 𝑎 ∈ R 𝐶 ×𝐻 ×𝑊 , aggregated by scene-level feature aggregation (see Section 3.4). Finally, the binary BEV mask is obtained through a sigmoid function.EQUATIONwhere 𝛿 (•)is a sigmoid function, 𝑀𝑎𝑠𝑘 𝑏,𝑖 ∈ R 𝐻 ×𝑊 . BEV map semantic segmentation prediction results 𝑀𝑎𝑠𝑘 𝑠 are as follows:EQUATIONWe propose scene-level feature aggregation to capture multi-granular contextual BEV features. Inspired by recent progress in introducing windows into Transformer [19, 27, 47] , we design scene-level feature aggregation as a hybrid structure, as shown in Fig. 4 . For the input fused BEV features, multi-window windowed attention (MWWA) performs window attention with windows of different sizes. MWWA aggregates multi-granular contextual semantic features on each attention head. In addition, we apply ASPP [5] and the bottleneck structure [12] to reduce the channel number by 4× to capture the global context. Finally, the dual outputs are weighted and fused through lightweight split-attention [46] and skip-connected with the original BEV feature map. Specifically, as shown in Fig. 4 (a), MWWA uses a pyramid window setting. Each window has a different attention range to capture multi-granular contextual information. It is worth noting that, our MWWA has different scale windows for each head, unlike standard multi-head window attention. Specifically, set the window size of 𝑖-th head ℎ 𝑖 to 𝑃 𝑖 × 𝑃 𝑖 . Given the input BEV features 𝐹 ∈ R 𝐶 ×𝐻 ×𝑊 , and split it into 𝑀 sub-BEV features, where 𝐷 is the sub-BEV feature dimension. We assign each sub-BEV feature an attention head for a specific window 𝑃 𝑖 × 𝑃 𝑖 . To dynamically focus on the remote region, we also use the shifted window operation following [27] . MWWA can be expressed as:𝐹 𝑖 ∈ R 𝐷 ×𝐻 ×𝑊 , 1, 2, • • • , 𝑀,EQUATIONEQUATION)where 𝑙 is the number of heads, Cat means concatenate function, 𝑓 WA (•) is the window attention. 
experiments 4.1 dataset: 
 We evaluate the performance of MaskBEV by comparing it with exiting SOTA methods on nuScenes [3] dataset. nuScenes is a very challenging large-scale autonomous driving dataset, containing 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. It provides point clouds collected with 32-beam Lidar and six cameras with complete 360 environment coverage. The annotated data can be widely used in tasks such as 3D object detection, object tracking, and BEV map segmentation. Following [28, 35] , we set the detection range to [-51.2𝑚, 51.2𝑚] for the 𝑋 and 𝑌 axes, and [-5𝑚, 3𝑚] for the 𝑍 axis and the segmentation range to [-50𝑚, 50𝑚] for the 𝑋 and 𝑌 axes. Our evaluation metrics align with [3, 28] . For 3D detection, we utilize the standard nuScenes detection score (NDS) and mean average precision (mAP). For BEV map segmentation, we follow [28, 49] to calculate the mean intersection over union (mIoU) on the overall six categories (drivable space, pedestrian crossing, walkway, stop line, car-parking area, and lane divider). The input camera and Lidar size depend on the specific BEV encoding backbone network.Model. Multimodal feature encoder can be BEVFusion [28] or UniTR [35] . More configuration details can be found in the previous paper [28, 35] . The number of perception queries is set to 𝑁 =300. The multi-task decoder adopts 𝐿=3. Scene-level feature aggregation loops twice. For MWWA, the sub-BEV number 𝑀 is set to 4, 8 heads are used in the attention, and each 2 heads focus on the same scale. The window sizes 𝑃 𝑖 are set to 3 × 3, 6 × 6, 9 × 9, and 18 × 18 respectively. Loss. For the loss of 3D object detection 𝐿 3𝐷 , we adopt the focal loss [24] for classification and 𝐿1 loss for 3D bounding box regression, and the loss weights of the two are set to 2.0 and 0.25 respectively. We use the standard focal loss 𝐿 𝑠𝑒𝑔 for BEV map segmentation. The losses of the two tasks are simply added with weights to form the overall loss 𝐿 𝑡𝑜𝑡𝑎𝑙 = 𝛼𝐿 3𝐷 + 𝛽𝐿 𝑠𝑒𝑔 . To balance multiple training tasks, we set the weights 𝛼 and 𝛽 to 3 and 1 respectively.Training. Different from the training strategy of the general architecture [11, 28] that separates the BEV encoders for different tasks, our MaskBEV jointly trains 3D detection and BEV map segmentation tasks in the unified encoder-decoder framework. We trained our model on 8 NVIDIA A800 GPUs by AdamW optimizer [29] . We used a batch size of 8 and 24 and trained for 20 epochs for BEVFusion [28] and UniTR [35] encoder backbone. Both encoder backbone with the once-cycle learning policy [32] and a maximum learning rate of 2e-3. We follow [28, 35] using the CBGS [53] strategy and the multi-modal data augmentation.Our MaskBEV is designed for MTL (3D object detection and BEV map segmentation), and we mainly focus on MTL methods on nuScenes. We use UniTR [35] as the feature encoding backbone network in experiments. As shown in Table 1 , despite the negative transfer [28] of MTL, our MaskBEV achieves SOTA performance on MTL with 72.9 NDS and 73.9 mIoU and outperforms previous SOTA UniTR by +1.3 NDS and +2.7 mIoU. UniTR, like BEVFusion [28] and MetaBEV [11] , adopts a separate BEV encoders strategy to perform MTL. MaskBEV outperforms MetaBEV by +3.1 NDS and +7.0 mIoU, where MetaBEV adopts an MTL optimization module. Furthermore, MaskBEV achieves comparable results to UniTR trained with single-task learning (STL) in 3D detection (72.9 NDS 𝑣.𝑠. 73.3 NDS) and map segmentation (73.9 mIoU 𝑣.𝑠. 74.7 mIoU).The results show that exploiting the complementary characteristics between multiple tasks improves the performance of MTL. Fig. 5 shows some qualitative results. The yellow and green marks show that our MaskBEV performance on MTL is far better than the UniTR performance on MTL, and is close to the UniTR performance on STL. See Fig. 6 for more visualizations and a video in Appendix.To demonstrate robustness, we evaluate our MTL framework on different BEV feature encoder backbone networks [28, 35] . Since MetaBEV [11] is not open source, we do not use it as a baseline. All experiments are performed on the nuScenes val set. The results in Table 2 show that the performance of each method is impaired on MTL, but our MaskBEV can bring consistent improvements to them, which proves the effectiveness of MaskBEV on MTL. Specifically, for BEVFusion [28] , our proposed multi-task head can obtain +1.2 NDS and +3.4 mIoU improvements compared to the separate BEV encoders strategy. The separation strategy proposed by BEVFusion can improve MTL performance. On the stronger baseline UniTR [35] , our MaskBEV achieves improvements of +1.4 NDS and +2.7 mIoU. The results show that MaskBEV can be used as a general MTL framework. Moreover, we compare inference latency with open source methods in Table 3 . Taking BEVFusion [28] and UniTR [35] as baselines respectively, our methods both achieve slightly leading inference speed, but greatly improve the performance.Network configurations. In Table 4 , we analyze the impact of different segmentation mask thresholds. We observe that lower thresholds help increase mask coverage and improve performance, as we analyzed in Sec 3.3. Table 5 suggests that using only boxes as masks limits the performance. We aim to expand the potential regions of interest appropriately, and the 1.3 times enlargement of circular regions validates this idea. This brings a performance improvement of +0.6 mAP and validates our motivation. Scene-level feature aggregation. In Table 6 , we ablate the impact of scene-level feature aggregation. Both MWWA and ASPP positively contribute to the final performance. The 1 𝑠𝑡 and 2 𝑛𝑑 rows show that extracting multi-scale features with windowed attention brings performance improvements to multi-task perception. Their complementary improvements in the 3 𝑟𝑑 row are understandable since MWWA focuses on multi-granularity features and ASPP focuses on the global contexts. We follow the same evaluation protocols adopted in UniTR [35] to demonstrate the robustness of our MaskBEV for Lidar and camera malfunctioning. We refer readers to [23, 44] for more implementation details. As shown in Table 7 and 8, under certain camera and Lidar failure conditions, our MTL method shows comparable results with STL variant of UniTR, and outperforms MTL variant of UniTR. which proves the robustness of MaskBEV to sensor failure conditions.  
 
 Appendix:  


Decision: This paper was reviewed by six experts.

The initial ratings were consistently borderline accept, indicating that all reviewers' first impressions were generally supportive.

After the rebuttal, four reviewers finalized their ratings as borderline accept (one reviewer posted their final rating in the discussion channel).

The AC read the paper, the reviews, and the rebuttal, and would like to second all reviewers' "borderline accept" ratings.

Reviews: review: paper_summary: paper_summary: The paper introduces MaskBEV, a novel masked attention multi-task learning (MTL) framework designed for 3D object detection on bird's eye view (BEV) map segmentation. MaskBEV integrates 3D object detection and BEV map segmentation into a single, task-agnostic Transformer decoder. Compared to previous works of separated BEV Encoders, this unified approach has better overall performance and learning efficiency by using the complementary nature of multi tasks. MaskBEV employs a masked attention mechanism, which selectively focuses on relevant regions in the BEV space, this improves computational efficiency and task accuracy. Also, spatial modulation and scene-level feature aggregation techniques are used to further enhance the model's ability to capture both geometric and semantic relationships in 3D object detection and BEV map segmentation.

The main contribution of this paper include its novel unified framework, utilizing multi-task complementary features. The framework begins by encoding camera and LiDAR data into feature tokens, which are then fused into a unified BEV feature map. A unified multi-task Transformer decoder, employing masked attention, processes these features to simultaneously generate 3D object detection and BEV map segmentation predictions, with scene-level feature aggregation, enhancing segmentation accuracy and making the approach theoretically sound and practically efficient.
strength_weakness: strength_weakness: 1. Simplifies the architecture by using a single decoder for multiple tasks, reducing computational complexity and enhancing performance. Achieves state-of-the-art results in both 3D object detection and BEV map segmentation.
2. The use of masked attention is theoretically sound as it allows the model to selectively attend to relevant features, reducing noise and improving the efficiency of the attention mechanism. The paper provides a detailed explanation of the implementation of this mechanism, ensuring technical correctness.
3. Introduces spatial modulation and scene-level feature aggregation strategies to exploit complementary information between tasks. The theoretical basis for these methods is well-explained, and their integration on the MaskBEV framework is coherent and logical.
4. The clarity of the methodology ensures that the framework can be easily understood. The use of diagrams and detailed explanations helps in conveying complex concepts effectively.
5. The evaluation includes comparisons with several state-of-the-art methods, demonstrating MaskBEV's advantages in both 3D object detection and BEV map segmentation performance, providing strong empirical evidence for the efficacy of the proposed framework. The ablation experiment supports the importance of the masked attention mechanism, spatial modulation, and context aggregation.
limitations: limitations: 1. English gramma issues. E.g. line 148 and 793.
2. Evaluating the inference speed on various hardware configurations (e.g., different types of GPUs) would help in understanding the practical deployment implications of MaskBEV.
3. The only autonomous driving dataset used in this paper is nuScenes. However, the paper would be benefit from comparing various datasets.
4. There might be specific scenarios, such as varying environment conditions, where separate BEV encoders could perform better. These scenarios are not thoroughly evaluated, and the potential advantages of separate encoders are not fully explored. Detailed performance metrics comparing unified and separate BEV encoders across different tasks and conditions would provide a more comprehensive understanding.
5. As shown in Table 7 and Table 8, the results of the experiments on both single-task learning(STL) and multi-task learning(MTL) indicates that the STL variant slightly outperforms MTL variant, there are still space to improve the model to outperform STL.
6. Evaluate and compare the performance metrics (e.g., NDS, mIoU) of MaskBEV with masked attention versus unmasked attention on the same dataset would be more comprehensive.
paper_summary: paper_summary: This paper proposed MaskBEV, a novel method for unified multi-task learning in autonomous driving systems, focusing on adapting the fused camera and lidar features for 3D object detection and BEV map segmentation into a single Transformer-based framework. MaskBEV innovatively addresses the limitations of traditional approaches that handle detection and segmentation tasks separately by employing a spatial modulation strategy that capitalises on the integration of both tasks. This approach enhances how well the system understands and processes environmental data while also improves its efficiency. By testing this framework on the nuScenes dataset, MaskBEV demonstrates improvements in performance metrics and processing speed.
strength_weakness: strength_weakness: 1. The MaskBEV framework demonstrated a novel approach using cross-modality features for multi-task learning which achieved state-of-the-art performance. 
2. The evaluation displayed the efficiency of the proposed framework compared to other state-of-the-art methods in terms of computational complexity.
3. The ablation study showed the robustness of the framework against unexpected situations like sensor failures, where the method still maintained good performance.
limitations: limitations: 1. While the method achieves state-of-the-art performance in multi-task learning approaches on the nuScenes dataset, it appears that all experiments were conducted only on this one dataset. Despite the size of the dataset and its variety of semantic settings, conducting more experiments on different datasets could further demonstrate the generalisation ability and performance of the model.
2. The evaluation demonstrated the efficiency of the approach by comparing the latency during processing. However, other measures, such as memory usage, could be take into account to further prove the advantages of the framework in terms of efficiency.
3. The ablation study section showed the robustness against sensor failure, this robustness could be further proven by testing under unexpected semantic settings such as heavy snow, or it could be tested on different camera settings such as lower resolution.
4. Despite claiming to mitigate effect of negative transfer in multi-task learning problems, the paper does not provide detailed evidence specifically showing how negative transfer is resolved compared to other methods.
paper_summary: paper_summary: This paper introduces MaskBEV, a unified multi-task learning (MTL) framework for 3D object detection and bird’s eye view (BEV) map segmentation in autonomous driving. MaskBEV utilizes a masked attention-based Transformer decoder that processes both tasks simultaneously without requiring specific task heads. The proposed framework incorporates spatial modulation and scene-level context aggregation strategies to enhance performance. Experiments on the nuScenes dataset demonstrate that MaskBEV achieves significant improvements in both tasks compared to state-of-the-art MTL methods.
strength_weakness: strength_weakness: 1. Extensive experiments on the nuScenes dataset demonstrate the effectiveness of MaskBEV. 
2. Detection accuracy, sensor failures, and latency are evaluated, showing a clear overview of the method.
limitations: limitations: 1. Based on my knowledge, most of current work use unified detetcion and segmentation frameworks. The claimed contribution of unified framework with shared encoder lacks novelty.
2. Missing citations: 
[1] Planning-oriented autonomous driving. CVPR 2023
[2] Vad: Vectorized scene representation for efficient autonomous driving. ICCV 2023
[3] VLP: Vision Language Planning for Autonomous Driving. CVPR 2024
[4] Clip-bevformer: Enhancing multi-view image-based bev detector with ground truth flow. CVPR 2024
paper_summary: paper_summary: This paper proposes a shared Transformer decoder like Mask2Former for unifying 3D object detection and bird’s eye view (BEV) map segmentation.
strength_weakness: strength_weakness: This paper proposes a shared Transformer decoder for unifying 3D object detection and bird’s eye view (BEV) map segmentation and achieves state-of-the-art performance on multi-task learning.

This paper proposes a spatial modulation strategy to obtain reliable multi-task masks and a scene-level feature aggregation module to capture multi-granularity and scene-level BEV contextual features.
limitations: limitations: 1. The authors claim that the proposed framework fully exploits the complementary advantages between multiple tasks. It is intriguing to investigate whether the information from complementary multi-modal tasks outperforms single-modal tasks. The paper lacks a comparison between the single-task results of Maskbev and the multi-task results to illustrate this point.
2. The paper's ablation analysis is not comprehensive enough as it fails to include the overall ablation of the proposed spatial modulation strategy and feature aggregation module. For instance, it would be beneficial to compare the results of the baseline model against the baseline model with module 1, the baseline model with module 2, and the Maskbev model (baseline model with both module 1 and module 2).
paper_summary: paper_summary: The paper introduces a novel multi-task learning framework, MaskBEV, designed to improve autonomous driving systems by unifying 3D object detection and BEV map segmentation through a single Transformer decoder. This approach utilizes advanced techniques like spatial modulation and scene-level feature aggregation to enhance performance. MaskBEV outperforms current state-of-the-art methods on the nuScenes dataset, demonstrating improvements in detection accuracy and segmentation.
strength_weakness: strength_weakness: 1. Unlike existing methods that use separate task-specific decoders, MaskBEV employs a unified Transformer decoder.
2. The paper introduces technically sound methods such as spatial modulation to manage attention masks and scene-level feature aggregation to enhance feature representation. These techniques enhance overall system performance.
3. The evaluation of MaskBEV is thorough. The paper compares MaskBEV's performance against state-of-the-art MTL methods, demonstrating significant improvements in both 3D object detection and BEV map segmentation.
limitations: limitations: 1. MaskBEV integrates 3D object detection and BEV map segmentation using a unified Transformer decoder, while may introduce complexity in terms of model architecture. A detailed analysis of the trade-offs between performance gains and computational overhead compared with existing MTL frameworks would be beneficial.
2. The integration of advanced techniques like spatial modulation and scene-level feature aggregation requires careful tuning of numerous parameters and design choices. This can make the optimization process more challenging.
3. The paper primarily evaluates the performance of MaskBEV on the nuScenes dataset. To further validate the effectiveness and robustness of the MaskBEV framework, it is recommended that the authors conduct experiments on additional datasets.
4. The paper demonstrates the effectiveness of MaskBEV for two specific tasks. However, its scalability to incorporate additional perception tasks (like motion prediction) is not discussed.
paper_summary: paper_summary: The authors propose a task-agnostic Transformer decoder to process 3D object detection and BEV map segmentation via a unified decoder, which fully exploits the complementary information between BEV map segmentation and 3D object detection tasks in BEV space.
strength_weakness: strength_weakness: 1) The author claims they are the first to unify transformer encoders to achieve simultaneous perception of 3D object detection and BEV map segmentation. 
2) The proposed approach outperforms SOTA multi-task learning methods in both 3D object detection and BEV map segmentation tasks on nuScenes dataset.
limitations: limitations: I had two main concerns regarding their proposed unified transformer decoder below, I would like to increase the score if the authors can clearly explain it. 
1) The architecture (Figure.2 ) of the unified transformer decoder in your paper is  the same as the one (Figure.2) in the previous publication Mask2Former [9], it is ok if the authors of both papers are the same, but it may break the double-blind review policy. 
2) It is understood that multi-modal Camera and Lidar possess complementary information, the former more on color and texture, while the latter more on depth, but it is not clear how the complementary nature of the BEV map segmentation task and the 3D object detection task comes from? or just because your experimental results shows better results?