CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers
Authors: Runsheng Xu
Zhengzhong Tu
Hao Xiang
Wei Shao
Bolei Zhou
Jiaqi Ma
Venue: CoRL 2022 Poster (2022)
Keywords: Autonomous driving
BEV map understanding
Vehicle-to-Vehicle (V2V) application

Abstract: Bird’s eye view (BEV) semantic segmentation plays a crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress on BEV map understanding, they are all based on single-agent camera-based systems. These solutions sometimes have difficulty handling occlusions or detecting distant objects in complex traffic scenes. Vehicle-to-Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state-of-the-art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmentation with single-agent multi-camera and 2) 3D object detection with multi-agent LiDAR systems, achieving state-of-the-art performance with real-time inference speed. The code is available at https://github.com/DerrickXuNu/CoBEVT.

CoBEVT: Cooperative Bird's Eye View Semantic Segmentation with Sparse Transformers
Runsheng Xu"; Zhengzhong Tu Hao Xiang', Wei Shao Bolei Zhou', Jiaqi Malt University of California, Los Angeles, 2 University of Texas at Austin 3 University of California, Davis
Abstract: Bird's eye view (BEV) semantic segmentation plays crucial role in spatial sensing for autonomous driving. Although recent literature has made significant progress BEV map understanding, they are all based on single- agent camera-based systems. These solutions sometimes have difficulty handling occlusions detecting distant objects in complex traffic scenes. Vehicle-to- Vehicle (V2V) communication technologies have enabled autonomous vehicles to share sensing information, dramatically improving the perception performance and range compared to single-agent systems. In this paper, we propose CoBEVT, the first generic multi-agent multi-camera perception framework that can cooper- atively generate BEV map predictions. To efficiently fuse camera features from multi-view and multi-agent data in an underlying Transformer architecture, we design a fused axial attention module (FAX), which captures sparsely local and global spatial interactions across views and agents. The extensive experiments on the V2V perception dataset, OPV2V, demonstrate that CoBEVT achieves state- of-the art performance for cooperative BEV semantic segmentation. Moreover, CoBEVT is shown to be generalizable to other tasks, including 1) BEV segmen- tation with single-agent multi-cament and 3D object detection with multi-agent LiDAR systems, achieving state-of-the-arr performance with real-time inference speed. The code is available at https: / / wtg.ithto/eeerrrr.... www./eooe/eeeeeeeeeeeeeeeeeeeeee Dit.erricckuuu/oBBVV....
Keywords: Autonomous driving, BEV map understanding, Vehiele-to-Vehille (V2V) application
Introduction
Autonomous vehicles (AVs) need accurate surrounding perception and robust online mapping capa- bilities for robust and safe autonomy [1]. AVs are normally located on the ground plane, so it nat- ural represent semantic and geometric information of surroundings in the bird's eye view (BEV) maps. Projecting multi-camera views onto the holistic BEV space brings clear strengths in preserv- ing the location and scale of road elements both spatially and temporally, which is critical for various autonomous driving tasks, including scene understanding and planning [2, 3]. It also presents a scal- able vision-based solution for real-world deployment without relying on costly LiDAR sensors.
Map-view (or BEV) semantic segmentation is fundamental task that aims to predict road segments from single- multi-calibrater camera inputs. Significant efforts have been made toward precise camera-based BEV semantic segmentation. One of the most popular techniques is to leverage depth information to infer the correspondences between camera views and the canonical maps [4, 5, 6, 7]. Another family works directly learns the camera-to-BEl
space transformation, either implicitly explicitly, using attention- based models [3, 8, 10]. Despite the promising results, vision-based perception systems have intrinsic limitations camera sensors are known to be sensitive to object occlusions and limited depth-of ofeld, which can lead to inferior performance in areas that are heavily occluded or far from the camera lens [3, 11].
Recent advancements in Vehicle-to- Vehicle (V2V) communication technologies have made it pos- sible to overcome the limitations of single-agent line-of-sight sensing. That is, multiple connected
Equal contribution. 'Corresponding author: jiaqima@ucla.edu.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

BEV Fectures
SinBEVT
Sparial
Warping
AV
Multi view RGB Images
Camera Encoder
Multi-view Features
Shared weights
Compressed Feature Sharing
RISTBEVT Transformer
Aggregated RFV Features
SinBEVT Transformer
EsoAV
Mulli-view RGB Images
Camera Multiview Features
SimBEVT Decoder
BEV Fealures
CoBEVT
E
EguAV
dop-down view mup
SiaBEVT Prediction
CoBEVT Prediction
Figure 1: The overall framework of CoBEVT
AVs can share their sensory information with each other through broadcasting, thereby providing multiple viewpoints of the same scene. Several prior works have demonstrated the efficacy of coop- erative perception utilizing LiDAR sensors [12, 13, 14, 15]. Nevertheless, whether, when, and how this V2V cooperation can benefit camera-based perception systems has not been explored yet.
In this paper, we present CoBEVT, the first-of-its-kin framework that employs multi-age multi-agent multi- camera sensors to generate BEV segmented maps via sparse vision transformers cooperatively. Fig. illustrates the proposed framework. Each AV computes its own BEV representation from its camera rigs with the SinBEVT Transformer and then transmits it to others after compression. The receiver (i.e. other AVs) transforms the received BEV features onto its coordinate system, and employs the proposed FuseBEVT for BEV-level aggregation. The novel fused axial attention (FAX) module, which can search over the whole BEV camera image space across all agents camera views via local and global spatial sparsity. FAX contains global attention to model long-distance dependencies, and local attention to aggregate regional detailed features, with low computational complexity. Our extensive experiments on the V2V perception dataset [12] show that CoBEVT achieves performance gains of 22.7% and 6.9% over single-agent baseline and leading multi-agent fusion models, respectively.
ingredient of these
two transformers
Furthermore, we demonstrate the generalizability the proposed framework in two additional tasks. First, evaluate SinBEVT alone for single-agent multi-view BEV segmentation. Second, validate the attention fusion on a different sensor modality multi-agent LiDAR fusion. Our ex- periments the nuScenes dataset [16] and the LiDAR-track of OPV2V [12] show that CoBEVT exhibits outstanding performance and capably generalize to many other tasks. Our contributions are:
We present the generic Transformer framework (CoBEVT) for cooperative camera-based BEV semantic segmentation. CoBEVT delivers superior performance and flexibility, achieving state-of-the-art results on multi-agent camera- camera-based, single-vehicle multi-view BEV semantic segmentation, and multi-agent LiDAR- 3D detection.
We propose a novel sparse attention module called fused axial (FAX) attention, which can efficiently capture both local and global relationships between different agents or cameras. We build two instantiations self-attention (FAX-SA) and cross-attention (FAX-CA) to accommodate different application scenarios.
We construct a large-scale benchmark study on the cooperative BEV map segmentation task with total of eight strong baseline models. Extensive experimental results and abla- tion studies show the strong performance and efficiency of the proposed model. All code, baselines, and pre-trained models will be released.

---

2 Related Work
2.1 V2V Perception
V2V perception leverages communication technologies to enable AVs to share their sensing in- formation to enhance the perception. Previous works mainly focus cooperative 3D object de- tection with LiDAR. A straightforward sharing strategy is to transmit raw point cloud (i.e. early fusion) [17, 18] detection outputs (i.e. late fusion) [19]. However, they either require large bandwidth or ignore the context information. Recently, V2VNet [20] proposes to circulate the in- termediate features extracted from 3D backbones (i.e. intermediate fusion), then utilize spatial- aware graph neural network for multi-agent feature aggregation. Following similar transmission paradigm, OPV2V [12] employs simple agent-wise single-head attention fuse all features. F- Cooper [21] uses a simple maxout operation to fuse features. DiscoNet [14] explores knowledge distillation by constraining intermediate feature maps to match the correspondences in the early- fusion teacher model.
2.2 BEV Semantic Segmentation
BEV semantic segmentation aims to take camera views as input and predict a rasterized map with surrounding semantics under the BEV view. A common approach for this task is to use inverse per- spective mapping (IPM) [22] to learn the homography matrix for view transformation [23, 24, 25]. As camera images lack explicit 3D information, another family of models includes depth estimation to inject auxiliary 3D information [4, 5, 2, 26]. Recently, researchers start to directly model the image-to- map correspondence using transformers or MLPs. VPN [27] learns map-view transforma- tion in spatial MLP module on flattened camera-view image features. CVT [3] develops positional embedding for each individual camera depending on their intrinsic and extrinsic calibrations. BEV- Former [8] exploits the camera intrinsic and extrinsic explicitly to compute the spatial features in the regions of interest of the BEV grid across camera views using deformable transformer [28].
sange-t--map&oo
2.3 Transformers in Vision
Transformers are originally proposed for natural language processing [29]. VIT [30] has demon- strated for the first time that, pure Transformer that simply regards image patches as visual words, is sufficient for vision tasks by large-scale pre-training. Swin Transformer [31] further improves the generality and flexibility of pure Transformers via restricting attention fields in local (shifted) windows. For high dimensional data, video Swin Transformer [32] extends the Swin approach onto shifted 3D space-tima windows, achieving high performance with low complexity. Recent works have been focused on improving the architectures of attention models, including sparse atten- tion [33, 34, 35, 36, 37, 38, 39], enlarged receptive fields [40, 41], pyramidal designs [42, 43, 44], efficient alternatives [45, 46, 47], etc.
3 Methodology
We consider V2V communication system where all AVs exchange sensing information with others. Assuming the poses all the agents are accurate and transmitted messages are synchronized, propose robust cooperative framework that can exploit the shared information across multiple agents to obtain a holistic BEV segmentation map. The overall architecture of CoBEVT is illustrated in Fig. which consists of: SinBEVT for BEV feature computation (Sec. 3.2), feature compres- sion and sharing (Sec. 3.3), and FuseBEVT for multi-agent BEV fusion (Sec. 3.3). We propose a novel 3D attention mechanism called fused axial attention (FAX, Sec. 3.1) the core component of SinBEVT and FUS that can efficiently aggregate features across agents or camera views both locally and globally. We will later show that this FAX attention has great generality, showing effi- cacy on different modalities for multiple perception tasks, including cooperative/single segmentation based on multi-view cameras and cooperative 3D LiDAR object detection.
BEV
3.1 Fused Axial Attention (FAX)
Fusing BEV features from multiple agents requires both local and global interactions across all agents' spatial positions. On the one hand, neighboring AVs often have different occlusion levels
3

---

DWQ
Faod
(a) 3D FAX attention for multi-agent BEV fusion.
(h) 3D FAX attention for multi-view fusion.
Figure 2: Illustrated examples of fused axial attention (FAX) in two use cases (a) multi-agent BEV fusion and (b) multi-view camera fusion. FAX attends to both 3D local windows (red) and sparse global tokens (blue) to attain location-wise and contextual-aware aggregation. In (b), for example, the white van is torn apart in three views (front-right, back, and back-left), our sparse global attention can capture long-distance relationships across parts in different views attain global contextual understanding.
on the same object; hence, local attention, which cares more about details, can help construct pixel- to-pixel correspondence on that object. Take the scene in Fig. 2(a) as an example. The ego vehicle should aggregate all the BEV features per location from nearby AVs to obtain reliable estimates. On the other hand, long-tern seng-term global contextual awareness can also assist in understanding the road topological semantics or traffic states the road topology and traffic density ahead of the vehicle are often highly correlated with the one behind. This global reasoning is also beneficial for views understanding. In Fig. 2(b), for instance, the same vehicle is torn apart into multi-views, and global attention highly capable of connecting them for semantic reasoning.
To attain such local-global properties efficiently, we propose a sparse 3D attention model called fused axial attention (FAX), which performs both local window-based attention and sparse global interactions, inspired by [32, 48, 49]. Formally, let X E RN XxXxHxXxx WC' be the stacked BEV features with spatial dimension H x W from N agents. In the local branch, we partition the feature map into 3D overlaaaaa windows, each of size N x P x P. The partitioned tensor of shape ( x W. P. N p2, C) is then fed into the self-attention model, representing mixing information along the second axis i.c., within local 3D windows [32]. Likewise, the global branch, feature is divided using uniform 3D grid N x X G into the shape (N x G² H. x W C). Employing attention the first axis of this tensor representing attending to sparsely sampled tokens [48, 49]. Fig. 2 illustrates the attended regions using red and blue colored boxes for local and global branches, respectively.
Combining this 3D local and global attention with typical designs of Transformers [30, 31, 32], including Layer Normalization (LN) [50]. MLPs [30], and skip-connections, forms our proposed FAX attention block, as shown in Fig. 3b. Our 3D FAX attention only requires () HWC) complexity assuming P ~ G (typically N <= 5, P.G' E {8, 16)), significantly cheaper than the full attention O ((NN)W)*C) Still, it enjoys non-local 3D interactions by seeing through all the agents, which is more expressive than local attention approaches [31, 32]. The 3D FAX self- attention (FAX-SA) block can be expressed as:
z = 3DLLLAt f )) + + 2 ++1 3DG-Att((L(22)) +z', +z',
z = MLP(LN(22)) 2
(1)
z f+1 MLP(LN((2)) -2
(2)
where 2 and z denote the output features of the 3DL(G)-Attn module and MLP module for block l. The 3DL-Attn and 3DG-Attn represent the above-defined 3D local and global attention, respectively.
3.2 SinBEVT for Single-agent BEV Feature Computation
Given monocular views from m cameras on the i-th agent (Ti, Ki. Ri, 1 )m denoting input im- ages I, E hxxxx3 camera intrinsic K₃ E R3x3 rotation extrinsic RA E R3x3 and translation to 33, every agent needs to compute BEV feature representation F, E "xxCC (height H,
width W, and channels C) before any cross- agent collaboration. F, can be either fed into a decoder to perform single-agent predictions or shared the ego vehicle for multi-agent feature fusion.
We take BEV processing architecture similar to CVT [3], wherein learnable BEV embedding is initialized as the query interact with encoded multi-view camera features, as shown in Fig. 3a. We

---

Carn, Feats
Cam, Feats
SiaBEVT
Pos.
(144)
Pos.
(1/8)
Embh
Embh.
MLP
MLP
Altention
FAX FAN Corss
Atteution
FAX FAXCCoss Corss
Conv
Come
BEV Embedding
BEV Features
FT
3D L-Amn
3D G.Am
Attention
FAX Self
FAX Attention
Q K
Y.K.V
FFN
Aggr. BEV Feats.
Fused BEV Feats
(a) SinBEVT and Vu Architecture
FAX SA and FAX-CA block
Figure 3: Architectures of (a) SinBEVT and FuseBEVT, and (b) the FAX-SA and FAX-CA block.
have observed that CVT uses low-resolution BEV query that fully cross-attends to image features, which leads to degraded performance on small objects, despite being efficient. Thus, CoBEVT learns a high-resolution BEV embedding instead, then uses hierarchical structure refine the BEV features with reduced resolution. To efficiently query features from camera encoders at high resolution, the FAX-SA module further extended build FAX cross-attention (FAX-CA) mod- ule (Fig. 3b), in which the query vector is obtained using the BEV embedding. whereas the key/value vectors are projected by multi-view camera features. Before applying cross-attention, we add camera-aware positional encoding derived from camera intrinsics and extrinsics, learn implicit geometric reasoning from individual camera views to a canonical map-view representation, follow- ing CVT. This rather simple, implicit approach demonstrates good balance of performance and efficiency, and our FAX attention allows for global interactions hierarchical network, showing better accuracy against low-rcsolution isotropic approaches such CVT.
3.3 FuseBEVT for Multi-agent BEV Feature Fusion
Feature Compression and Sharing. Transmission data size is critical to V2V applications, as large bandwidth requirements will likely cause severe communication delays. Therefore, it is necessary to compress the BEV features before broadcasting. Similar to [13, 14], apply simple 1x1 convolutional auto-encoder [51] to compress and decompress the BEV features. Once receiving the broadcasted messages that contain intermediate BEV representations, and the pose of the sender, the ego vehicle applies a differentiable spatial transformation operator Tg to geometrically warp the received features [52] onto the ego's coordinate system: H, Ig (F,) E HxW
We design customized 3D vision Transformer called FuseBEVT that can atten- tively fuse information of the received BEV features from multiple agents. The ego vehicle first stacks the received and projected BEV features H,, i 1..., N into a high dimensional tensor E N xC then feeds them into the FuseBEVT encoder which consists of multiple layers of FA blocks (Fig. 3a). Benefiting from the linear complexity of FAX attention (Sec. 3.1), this agent-wise fusion Transformer is also efficient. Each FAX-SA block conducts a 3D global and local BEV feature transformation via Eqs. 1-2. As exemplified in Fig. 2(a), the 3D FAX-SA can attend to the same region of estimations (red boxes) drawn from multiple agents to derive the final aggregated representations. Moreover, the sparsely sampled tokens (blue boxes) interact globally to attain a contextual understanding of the map semantics such as road, traffic, etc.
Feature Fusion.
Decoder. We apply series of lightweight convolutional layers and bi-linear upsampling operations on the aggregated BEV representation and generate the final segmentation output.
Experiments
We evaluate the effectiveness of the proposed CoBEVT on the camera track of the V2V percep- tion dataset OPV2V [12]. To show the flexibility and generality of our CoBEVT, we also conduct experiments the LiDAR track of OPV2V and the autonomous driving dataset nuScenes [16].
5

---

Table 1: Map-view segmenta-
Table 2: 3D detection results on the OPV2V LiDAR-track. All methods employ PointPil- lar [61] backbone. (C) de- notes using 64x feature com- pression.
Table Vehicle map-view segmentation on nuScenes. All models use only a sin- gle time-stamp. * denotes our reproduced result with EfficientNet-b4 backbone.
tion on OPV2V camera-track.
We report loU for all classes. All fusion methods employs CVT [3] backbone, except for CoBEVT which uses SinBEVT backbone.
Method
Vch.
Dr.Area
Lane
Method
AP0.7
APO.7(C)
Method
Vch. Par(M) FPS
No Fusion
37.7
57.8
43.7
No Fusion
60.2
60.2
VPN [27]
29.3
4.
31
Map Fusion
45.1
60.0
44.1
Late Fusion
78.1
78.1
OFT [59]
30.1
F-Cooper [21]
52.5
60.4
46.5
Early Fusion
80.0
Lift-Splat
32.1
14
25
AttFuse [12]
51.9
60.5
46.2
F-Cooper
79.0
78.8
FIERY
35.8
7
8
V2VNet [20]
53.5
60.2
47.5
AttFuse
81.5
81.0
CVT [3]
36.0
1.2
35
DiscoNet [14]
52.9
60.7
45.8
V2VNet
82.2
81.4
FuseBEVT
59.0
62,1
49.2
DiscoNet
83.6
83.1
SinBEVT
37.1
1.6
35
CoBEVT
60.4
63.0
53.0
FuseBEVT
85.2
84.9
4.1 Datasets and Evaluations
OPV2V is large-scale V2V perception dataset that is collected in CARLA [53] and the cooperative driving automation tool OpenCDA [54]. It contains 73 diverse scenarios, which have an average of 25 seconds duration. In each scenario, various numbers each one is equipped with one LiDAR sensor and 4 cameras towards different directions to cover 360 horizontal field-of-view. Our main experiment only utilizes the camera rigs of the dataset, and we use Intersection over Union (IoU) between map prediction and ground truth map-view labels as the performance metric. Since OPV2V has multiple AVs in the same scene, we select fixed one as the ego vehicle during testing and evaluate 100mx I 100m area around it with a 39cm map resolution.
7) of AVs show up simultaneously, and
To demonstrate its generality, also evaluated our proposed CoBEVT the OPV2V LiDAR-trac' 3D detection task. We use the same evaluation range in [12, 55], and the detection performance is measured by Average Precisions (AP) at an loU threshold of 0.7. For both camera and LiDAR track, there are 6764/1981/2719 frames for train/ cain/validation/tess set, respectively.
The nuScenes dataset contains 1000 diverse scenes, each of around 20 seconds long. In total, there are 40K sampled frames in this dataset, and the dumped data captures a 360 view of surroundings using 6 cameras. We use the groundtruth in [3]. The evaluation ranges are [-50m, 50m] for the X and Y axis, and the resolution of the BEV grid is 0.5m.
4.2 Experiments Setup
Implementation details. We assume all the AVs have a 70m communication range following [20], and all the vehicles out of this broadcasting radius of ego vehicle will not have any collaboration. For the OPV2V camera-track, we choose ResNet34 [56] as the image feature extractor in SinBEVT. The transmitted BEV intermediate representation has resolution of 32 32 x 128. For the multi- agent fusion, our FuseBEVT component has 3 encoded layers and a window size of for both local and global attention. We train the whole model end-to-end with Adam [57] optimizer and cosine annealing learning rate scheduler [58]. We use weighted cross entropy loss and train all models with 60 epochs, with batch size of per GPU. Please refer to the supplementary materials for more details, as well as the configurations on nuScenes and OPV2V LiDAR-track.
Compared methods. For multi-agent perception task, consider single-agent perception system No Fusion as the baseline. We compare with the state-of-the-art multi-agent perception algorithms: F-Cooper [21], AutFuse [12]. V2VNet [20], and DiscoNet [14]. We also implement straightforward fusion strategy Map Fusion, which transmits the segmentation map instead of BEV features and fuses all maps by selecting the closest agent's prediction for each pixel.
For the nuScenes dataset, we compare against state-of-the-ar models including CVT [3], FIERY [4], View Parsing Network (VPN) [27], Orthographic Feature Transform (OFT) [59], and Lift-Splat- Shoot [5]. All models only utilize single-step timestamp data for fair comparisons. We intentionally use the same image feature extractor Efficient-Be B4 [60] and decoder as CVT and FIERY.

---

(a) ego
(b) av1
(c) av2
(d) groundtruth
(c) prediction
Figure 4: Qualitative results of COBEVT. From left to right: the front camera image of (a) ego, (b) 1 (c) av2, (d) groundtruth and (e) prediction. The green bounding boxes represent ego vehicle, while the white boxes denote the segmented vehicles. CoBEVT demonstrates robust performance under various traffic situations and road types.
4.3 Quantitative Evaluation
OPV2V camera-track results. To make a fair comparison, we first employ CVT [3] to extract the BEV feature from camera rigs for methods and only use the fusion component (i.e. FuseBEVT) of CoBEVT to compare with other fusion models. Then we compare with our complete CoBEVT to show the effectiveness of SinBEVT as well. shown Tab. 1, all cooperative methods perform better than No Fusion, which proves the benefits from multi-agent perception system. Among all fusion models, our FUs achieves the best IoU for all classes, outperforming the sccond-best method by 5.5%. 1.4%, and 3.4% on vehicle, drivable area, and lane, respectively. More importantly, by replacing the CVT with our SinBEVT for feature extraction, our CoBEVT can further increase the accuracy by 1.4%, 0.9%, and 3.8% the three classes compared using FuseBEVT only.
OPV2V LiDAR-track results. As Tab. reveals, our FuseBEVT also has the best performance on the LiDAR-track task, which improves the single-agent system by 25.0% and outperforms the leading algorithm DiscoNet by 1.7%. Furthermore, our method exhibits great robustness against LiDAR feature compression, with only 0.3% drop with the 64x compression rate.
nuScenes vehicle map-view segmentation. Our SinBEVT can run 35 FPS on RTX2080 with 37.1 IoU score and 1.6 M parameters, achieving the best accuracy with real-time performance. Compared to the SOTA method CVT, we are 1.1% higher with similar parame- ters and latency.
Table 4: Compression ef- feet on OPV2V Camera.
rat-rate
Size (KB)
loU
Ox
524
60.4
66
60.1
Effect of compression rate. Data transmission size is a critical factor
16x
33
58.9
32x
16
56.2
in V2V applications. Here we study the effect of different compres- sion rates on our CoBEVT by adjusting the x convolution. Tab. shows that CoBEVT insensitive compression, and it can still beat other fusion methods with large compression rate of 64.
64x
8
54.8
4.4 Qualitative Analysis
Fig. shows the qualitative results CoBEVT on scenes containing AVs. In each row, we draw the front camera image of each AV along with the ground truth and prediction pairs. Our framework
7

---

40
55
16U
50
HPS
20
45
16x16
CupSEVI
SnBEVT
40
50
64654
(a) Number of cameras dropped
(b) Number of agents
(c) Number of agents
Figure 5: Ablation studies. (a) IoU number of dropped cameras (b) IoU vs. number of agents.
(c) FPS vs. number of agents. The channel dimension of BEV feature map is fixed as 128 for (c).
can overcome most of the occlusions and perceive distant objects accurately, benefiting from our Transformer design that learns from all agents and views.
4.5 Ablation Study
Component analysis. Tab. 5 shows the importance of local and global attention. Both attention blocks significantly contribute the final performance.
Table 5: Component ablation.
Local Global Veh./Dr.ArewLLane
52.6/57 .9/42.0
Robustness to camera dropout. Sensor failure during driving can lead to fatal accidents. Therefore, here we investigate how well our CoBEVT handles it. We random drop E [1, 4] cam- cras of the ego vehicle, and demonstrate the performance de- crease for both SinBEVT (no collaboration) and CoBEVT in Fig. 5a. It can be that by intro- ducing sensing cooperation, driving safety can be significantly improved, as even all ego cameras break down, CoBEVT can still reach loU score of 44.3.
57,8/61 .549.2
57.9/60. 0/1600888/86666 6
60.4/ 63.0 /53.0
Number of agents. Here we study the influence brought by the number of collaborators on CoBEVT. As Fig. 5b describes, increasing the collaborators generally bring performance im- provement, whereas such gain will marginal when the agent number is greater than 4.
Inference speed of FusTBEVT. Real-time multi-agent feature fusion is critical for real-world de- ployment. Here we examine the inference speed of FuseBEVT with different BEV feature map spatial resolution (from 16 to 64) and the number of agents on RTX3090. Fig. 5c shows that our fusion algorithm can achieve real-time performance under distinct collaboration scenarios.
Conclusion and Limitations
In this paper, propose a holistic vision Transformer dubbed CoBEVT for multi-view cooperative semantic segmentation. We propose a fused axial attention (FAX) mechanism that allows for local and global interactions across all views and agents. Extensive experiments on both simulated and real-world datasets show that CoBEVT achieves superior performance on BEV segmentation.
Limitations. The proposed approach does not explicitly model realistic V2V challenges such as asynchronization and position errors, which may impair robustness under these noises. Address- ing these limitations needs future research real-world, realistic, and diverse cooperative datasets and benchmarks.

---

Acknowledgments
This material is supported in part by the Federal Highway Administration Exploratory Advanced Research (EAR) Program, and by the US National Science Foundation through Grants CMMI 1901998. We thank Xiaoyu Dong for her insightful discussions.
References
[1] W. Liu, K. Quijano, and M. M. Crawford. Yolov5-tassel: Detecting tassels in rgb uav imagery with improved yolov5 based transfer learning. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15:8085-8094, 2022.
[2] M. H. Ng, K. Radia, J. Chen, D. Wang. Gog, and J. E. Gonzalez. BEV-Seg: Bird's eye view semantic segmentation using geometry and semantic point cloud. arXiv preprint arXiv:2006. / 1436, 2020.
[3] B. Zhou and P. Krähenbühl. Cross-view transformers for real-time map-view semantic seg- mentation. arXiv preprint arXiv:2205.02833 2022.
[4] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall. Fiery: Future instance prediction in bird's-eye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15273-15282, 2021.
[5] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference Computer Vision, pages 194-210. Springer, 2020.
[6] S. Ammar Abbas and A. Zisserman. A geometric approach to obtain a bird's eye view from image. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshops, pages 0-0, 2019.
[7] Y. Kim and D. Kum. Deep learning based vehicle position and orientation estimation via inverse perspective mapping image. In 2019 IEEE Intelligent Vehicles Symposium, pages 317- 323, 2019.
[8] Z. Li, W. Wang. H. Li, E. Xie, C. Sima, T. Lu, Q. Yu, and J. Dai. Bevformer: Learning bird's s- eye-view representation from multi-cames images via spatiotemporal transformers. arXiv preprint arXiv: 17270, 2022.
Li, [9] W. Yang, Q. Li, W. Liu, Y. Yu, Y. Ma, S. He, and J. Pan. Projecting your view attentively:
Monocular road scene layout estimation via cross-view transformation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15545,, 2021.
[10] A. Saha, O. M. Maldonado, C. Russell, and R. Bowden. Translating images into maps. arXiv preprint arXiv:21 2021.
[11] W. Liu, L. Xiong, X. Xia, Y. Lu, L. Gao, and S. Song. Vision-aided intelligent vehicle sideslip angle estimation based on dynamic model. IET Intelligent Transport Systems, 14(10): 1833 1189, 2020.
[12] R. Xu, H. Xiang, X. Xia, X. Han. J. Liu, and Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vchick. communication. arXiv preprint arXiv:2109.07644, 2021.
[13] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and Ma. V2x-vit: Vehicle-to-everryhhing cooperative perception with vision transformer. arXiv preprint 223.0:2003. 2022.
[14] Y. Li, S. Ren, P. Wu. S. Chen, C. Feng, and W. Zhang, Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34, 2021.
[15] Y. Yuan, Cheng, and M. Sester. Keypoints- based deep feature fusion for cooperative vehicle detection of autonomous driving. IEÉE Robotics and Automation Letters, 2022.

---

[16] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong. Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1 11621-11631, 2020.
[17] Q. Chen, S. Tang, Q Yang, and S. Fu. Cooper: Cooperative perception for connected au- tonomous vehicles based on 3d point clouds. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages 514-524. OPTorganization, 2019.
[18] H. Hu, Z. Liu, S. Chitlangia, A. Agnihotri, and Zhao. Investigating the impact of multi- lidar placement on object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2550-2558 2022.
[19] Z. Y. Rawashdeh and Z. Wang. Collaborative automated driving: A machine learning- based method to enhance the accuracy of shared information. 2018 2 International Conference on Intelligent T Transportation Systems (ITSC), pages 3961-3966. OPTorganization, 2018.
[20] T.-H. Wang. S. Manivasagam, M. Liang, B. Yang, W. Zeng, and R. Urtasun. V2vnet: Vehicle- to io-vehicle communication for joint perception and prediction. In European Conference on Computer Vision, pages 605-621. Springer, 2020.
[21] Q Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu. F-cooper: Feature based cooperative per- ception for autonomous vehicle edge computing system using 3d point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88-100, 2019.
[22] H. A. Mallot, H. H. Bülthoff, Little, and S. Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics, 64(3):177-185, 1991.
[23] L. Reiher, B. Lampe, and L. Eckstein. sim2real deep learning approach for the transfor- mation of images from multiple vehicle- cameras to semantically segmented image bird's eye view. 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pages 1-7. IEEE, 2020.
[24] T. Bruls, H. Porav, L. Kunze, and P. Newman. The right (angled) perspective: Improving the understanding of road scenes using boosted inverse perspective mapping. In 2019 IEEE Intelligent Vehicles Symposium (IV), pages 302-309. IEEE, 2019.
[25] M. Zhu, S. Zhang, Y. Zhong, P. Lu, H. Peng, and J. Lenneman. Monocular 3d vehicle detec- tion using uncalibrated traffic cameras through homography. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 3814-3821. IEEE, 2021.
[26] H. Hu, B. Yang, Z. Qiao, D. Zhao, and H. Wang. Seasondepth: Cross-season monoeu- lar depth prediction dataset and benchmark under multiple environments. arXiv preprint arXiv:201 1. 04408, 2020.
[27] B. Pan, J. Sun, H. T. Leung, A. Andonian, and B. Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 5(3):4867-4873, 2020.
[28] X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 4rXiv:2010.04159, 2020.
[29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. Gomez, L. Kaiser, and I. Polo- sukhin. Attention all you need. In Advances in neural information processing systems, pages 5998 2017.
[30] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transform- ers for image recognition at scale, arXiv preprint orXiv:2010. I 1929, 2020.
[31] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEFF/CVE ternational Conference Computer Vision, pages 10012-10022, 2021.
10

---

[32] Z. Liu, J. Ning, Y. Cao. Y. Wei, Z. Zhang, S. Lin, and H. Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[33] x. Dong, J. Bao, D. Chen, W. Zhang, N. Yu, L. Yuan, D. Chen, and B. Guo. Cswin trans- former: A general vision transformer backbone with cross-shaped windows. arXiv preprint 2021.
arXiv:2107.0000
[34] J. Yang, C. Li. P. Zhang, X. Dai, B. Xiao, L. Yuan, and J. Gao. Focal self-attention for local- global interactions in vision transformers. arXiv preprint arXiv:2107.006// 2021.
[35] Y. Rao, W. Zhao, B. Liu, J. Lu, J. Zhou, and C.-J. Hsieh, Dynamievit: Efficient vision trans- formers with dynamic token sparsification. Advances in neural information processing sys- lems, 34, 2021.
[36] H. Wang. Y. Zhu, B. Green, H. Adam, A. Yuille, and L.-C. Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. In European Conference on Computer Vision, pages 108-126 126. Springer, 2020.
[37] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid. Vivit: video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6836-6846, 2021.
[38] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li. Maxim: Multi-axis mlp for image processing. arXiv preprint arXiv:200.00297,, 2022.
[39] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong, J. Malik, and C. Feichtenhofer. Im- proved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2112.01526, 2021.
[40] Min, Y. Zhao, C. Luo, and M. Cho. Peripheral vision transformer. arXiv preprint arXiv:2206.066 06801, 2022.
[41] A. Hatamizadeh, H. Yin, J. Kautz, and P. Molchanov. Global context vision transformers. arXiv preprint 2rXvv:2006.09959, 2022.
[42] W. Wang, E. Xie, X. Li, D.-P Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyra- mid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 568-578, 2021.
[43] H. Fan, B. Xiong, K. Mangalam, Y. Li, Z. Yan, J. Malik, and C. Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6824 6835, 2021.
[44] Y. Xu, Q Zhang, J. Zhang, and D. Tao. Vitae: Vision transformer advanced by exploring intrinsic inductive bias. Advances in Neural Information Processing Systems, 34:25522-28535, 2021.
[45] A. Ali, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek, et al. Xeit: Cross-covariance image transformers. Advances in neural information processing systems, 34, 2021.
[46] H. Liu, Z. Dai, D. So, and Q. Le. Pay attention to mlps. Advances in Neural Information Processing Systems, 34, 2021.
[47] J. Lee-Thorp, J. Ainslie, I. Eckstein, and S. Ontanon. Fnet: Mixing tokens with fourier trans- forms. arXiv preprint arXiv:2105.03888 2021.
[48] Z. Huang, Y. Ben, G. Luo, P. Cheng. G. Yu, and B. Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint arXiv: (tl::2006.3360 006.03650, 2021.
[49] Z. Tu, H. Talebi, Zhang, Yang, Milanfar, Bovik, and Y. Li. Maxvit: Multi-axis vision transformer. arXiv preprint arXiv:2204.01697, 2022.

---

[50] L. Ba, R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: arXiv:1607.06450, 2016.
[51] J. Masci, Meier, D. Cireşan, and J. Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extraction. In International conference on artificial neural networks, pages 52-59. Springer, 2011.
[52] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28:2017-2025, 2015.
[53] A. Dosovitskiy, Ros, F. Codevilla, Lopez, and V. Koltun. CARLA: An open urban driving simulator. In Proceedings of the 1st Annual Conference on Robot Learning, pages 1-16, 2017.
[54] R. Xu. Y. Guo, X. Han, X. Xia, H. Xiang, and J. Ma. Opencda: an open cooperative driving automation framework integrated with co-simulation. 202/ IEFE International Intelligent Transportation Systems Conference (FTSC), pages 1155-1162. OPTorganization, 2021.
[55] W. Chen, R. Xu, H. Xiang, L. Liu, and J. Ma. Model-agnostie multi-agent perception frame- work. arXiv preprint 2rXiv:2203.13166 2022.
[56] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[57] I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
[58] Loshchilov and Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint
2016.
[59] T. Roddick, A. Kendall, and R. Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint arXiv:1811.08188. 2018
[60] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning, pages 6105-61 14. PMLR, 2019.
[61] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast en- coders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22697-12705 2019.
12


Reviews:
[{"review_id": "irAxXOckvxJ", "reviewer": "Reviewer_gxHq", "summary": "The paper presents a solution to attention-based multi-sensor multi-agent camera data fusion for the task of bird's eye view (BEV) semantic segmentation. The proposed framework CoBEVT depends on a novel fused axial attention (FAX) mechanism at both the feature extraction and fusion phases as part of Transformer architectures. The FAX mechanism conducts attention at both the local and global scales, while remaining tractable. CoBEVT is empirically validated as part of a simulated (in CARLA) vehicle-to-vehicle benchmark and on the real-world NuScenes dataset. CoBEVT outperforms state-of-the-art baselines across cooperative BEV semantic segmentation, singe-agent multi-camera BEV semantic segmentation, and 3D object detection with multi-agent LiDAR sensors tasks with real-time inference speed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The biggest strength of the paper is its empirical validation. The authors do their due diligence in thoughtfully constructing experiments that showcase the advantages of the CoBEVT framework as compared to a significant number of baselines across several tasks. The ablation studies clearly show the benefits of both local and global attention, robustness to sensor failure, multi-agent sensor fusion, and BEV feature map compression on inference speed.\n* The camera-based multi-sensor multi-view CoBEVT framework appears to be novel given a quick search of existing literature. The authors do a good job contextualizing their approach in prior work.\n* The limitations section is honest and thorough providing helpful context and opportunities for future work.\n* Fig. 1 is an effective illustration of the CoBEVT framework.\n\nWeaknesses:\n\nMy biggest concerns with the paper center around the presentation and polish.\n* In Sec. 2.2 and 2.3 of the literature review, it would be helpful to include a sentence in each section relating how the described literature relates to the presented approach.   \n* Fig. 2 lacks clarity and polish. It is unclear how the BEV image on the left in Fig. 2(a) corresponds to the predicted scenes on the right. Occluded vehicles should be highlighted in the figure. The captions for the two subfigures should be better spaced out. In Fig. 2(b), the different views (side, back, front) should be labeled. From the caption and the text in the body of the paper, it is not entirely clear what is the takeaway from Fig. 2(b). For example, which car is torn apart into multiple views? Is it the one in the lower row? How does the illustrated attention combat this?\n* In methods lines 128-137, it was not really clear the difference between local and global attention. The notation makes it seem like they are the same thing but transposed. Further illustrating the difference between the approaches would be helpful here. Additionally, since the sparsity in the attention is critical to the proposed CoBEVT approach, a description of how sparsity is achieved should be included either in the main paper or in the appendix.\n* In methods lines 143-145, it is not clear why the equations for local and global attention have notation that differs by one index value or why they are sequential. More context and explanation is necessary here. The 'LN' acronym is not defined in the paper.\n* Table 1, 2, and 3 captions are too close together and have spelling errors (PointPillar, nuScene - both should be plural).\n* In Fig. 4, the white boxes (vehicles) were not explained in the caption. The occluded and far-away vehicles should be indicated in the figure to showcase the performance of CoBEVT. The merging of vehicles in the predictions should be listed as a limitation.\n* In Table 5, it is unclear what is the structure of the no local and global attention FAX architecture (first table row).\n\nThe paper currently contains many typos and requires polishing. The following is a non-exhaustive list of typos as well as grammatical and consistency issues I found:\n\n1. Lines 3-5: incorrect grammar in 'they are all based on single-agent camera-based systems which are difficult to handle occlusions and detect distant objects in complex traffic scenes'.\n2. Fig. 1: white blocks are not indicated to be cars.\n3. Line 51-52: 'FAX contains global attention to reason high-level contextual information' should read 'about high-level contextual information'.\n4. Line 59: 'show that CoBEVT exhibit' should be 'exhibits'.\n5. Line 112: 'We propose a novel 3D attention mechanism ... as the core components' should be 'component'. \n6. Line 113: there should be no comma before 'that'.\n7. Line 115: 'sing-agent' should be 'single-agent'.\n8. Line 124: 'ahead the vehicle' should be 'ahead of the vehicle'.\n9. Lines 140-143: 'Our 3D FAX ... more expressive than local approaches [27, 28].' sentence is too long and has grammatical errors.\n10. Lines 176-177: 'The ego vehicle first stacks the received and projected BEV features ..., then feed into' should be 'then feeds them into'.\n11. Line 182: 'sparsely sampled tokens are interacted globally' should be 'tokens interact globally'.\n12. Line 192: '4 cameras towards distinction to cover' is incorrect grammar.\n13. Line 201: 'the detection performance is measured with Average Precisions (AP) at IoU thresholds 0.7' should be 'Average Precision (AP) at an IoU threshold of 0.7'.\n14. Line 206: 'the size of resolutions of the BEV grid is 0.5m' should be 'the resolution of the BEV grid is 0.5 m'.\n15. Line 212: 'layer3' should be 'layers'.\n16. Line 213: 'Adaw' should be 'Adam'.\n17. Line 243: 'stat-of-the-art' should be 'state-of-the-art'.\n18. Line 246: Missing space between sentences.\n19. Line 247: Missing space between 'the' and '$1 \\times 1$'.\n20. The references have typos and consistency issues. For example, 'Bev' in [1] should be capitalized. Conference acronyms are sometimes included (e.g., IV in [6]) and sometimes not (e.g., [3]). When possible, the conference venue should be cited instead of ArXiv (e.g., [1]). The references should in general be proofread (e.g., [6] lists the year and IEEE twice).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper presents a novel approach to multi-agent multi-sensor BEV semantic segmentation, which is of interest to the autonomous driving community. A new scalable attention mechanism is introduced for the task. The experiments extensively validate the proposed approach. However, the current presentation lacks polish and clarity. As it stands, I am leaning towards rejecting the paper in its current state. I am open to increasing my score based on the rebuttal discussion and updates made to the paper during the rebuttal phase. That being said, I think this is a promising paper, and with some improvements to the presentation it will make a good contribution.\n\n**Post-Rebuttal**: Upon going through the authors' rebuttal, I am inclined to raise my score to a Weak Accept. The authors put in significant effort in terms of clarifications and further exposition to address my concerns. Although I still have some concerns surrounding clarity and polish, I believe that the proposed method is novel and interesting with extensive and promising empirical results.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "In addition to the weaknesses listed above, I am hoping the authors can address the following clarifications and discussion points during the rebuttal period.\n\n1. Are the stacked BEV features from $N$ agents (line 130) already transformed into the same spatial area?\n2. How could object disappearance and merging of multiple vehicles in the predictions be addressed (e.g., in the appendix Fig. 3, third row example). \n3. Are there ways to enforce known road geometry to avoid deformation in the predictions?\n4. In Fig. 5, is $32 \\times 32$ the spatial resolution used in the earlier experiments?\n5. It would be helpful to include the ground truth in the appendix Figs. 4 and 5. What is the difference between CoBEVT and DiscoNet qualitative results in these two figures?"}}, {"review_id": "Ufn0Kji1gzB", "reviewer": "Reviewer_Kibb", "summary": "The authors propose CoBEVT for bird's eye view (BEV) semantic segmentation from multiple cameras in a multi-agent environment in this paper. CoBERT is a Transformer-based method, in which the BEV representation is extracted by the SinBEVT Trasnformer in each vehicle, and then aggregated by FuseBEVT. These Transformers include a fused axial (FAX) module with both global and local attention, which reduces the computational cost and shows high performance in this problem setting. The proposed method has shown high performance in a variety of different settings at various metrics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths:\n  - The paper is well organized and readable.\n  - The proposed method, FAX, is less expensive than full attention, yet very effective in the current problem setting, and could be useful in this research area in the future.\n  - The experiments in this paper have been evaluated with various metrics (e.g., segmentation and processing speed) and settings (e.g., number of agents, camera dropout) on different methods and datasets, which are highly reliable.\n\n- Weakness:\n  - The authors assume that \"poses of all the agents are accurate and transmitted messages are synchronized\" but as the authors state in the limitation section, this is not necessarily the case in a real vehicle. Since this conference is also focused on the feasibility of hardware robots, the authors need more discussion on the extent to which these assumptions may not hold in reality, the extent to which these difficulties affect the current problem setting, and whether the proposed method has the potential to address these difficulties.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although I am not an expert in BEV map understanding, I believe that the proposed method is novel and will be useful in this community in the future. I would like to accept this paper, although I have some concerns that it has not been sufficiently discussed for real-world applications.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "As pointed out in \"weakness\", I would like the authors to discuss more how well the hypotheses in this setting are valid in the real world and how well the proposed method can be used in the real world."}}, {"review_id": "TqTz7pqLlzF7", "reviewer": "Reviewer_dUwG", "summary": "The paper proposes CoBEVT, the first generic multi-agent multi-camera perception framework that can cooperatively generate BEV map predictions. Each view is first passed through an image encoding to extract image images, which are then in turn used to form BEV features. The features are concatenated and fused together to make BEV predictions. The method is tested on OPV2V and nuScenes.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The papers utilize the latest advance in transformer architectures to tackle the V2V communication problem.\n\nWeaknesses\n\n- It is not clear whether those proposed transformer structures are necessary for the problem setup or not. Although it makes sense, but there is no comparison with the proposed structure with the existing methods. For example, we can compare the BEV features produced by single camera with the existing models through map segmentation and 3D object detection on nuScenes to justify the model designs to extract features from single cameras.\n\n- The experiment detail on nuScenses is really sparse, which leads to confusion. On Line 242, it claims that \"Compared to the stat-of-the-art method CVT, we are 1.1% higher with similar parameters and latency.\" This statement is too vague, especially compared to the results on OPV2V. The supplementary material only provides quantitative results.\n\n- It is strange to use ResNet34 as the image encoder while the authors try to put latest transformer models on top of it, instead of designing an end-to-end transformer for image encoding to BEV features.\n\n- It is better to know where the multi-agent fusion can improve the prediction accuracies, and whether the improvements depend on the distances of the vehicles.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am on the borderline for the paper, but leaning towards rejection now. While fusing multiple cameras for 3D predictions is an interesting topic, the present paper leaves too many details out to understand and justify the model designs. The experiment detail on nuScenes is too little for a comprehensive understanding.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Similar to the weaknesses above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "3puuz61Ot2p", "reviewer": "Reviewer_KVMN", "summary": "The paper proposes a method for RGB-alone BEV perception across multiple agents and views. Each agent uses a perspective-to-BEV attention mechanism to construct a BEV feature map, and  sends its BEV feature maps to other agents. Then, each agent stacks all BEV feature maps and computes a fused BEV feature map.  The paper proposes as its contribution a hierarchical transformer in the BEV space, where attention is first performed within local windows, and then across far away locations on the map. They evaluate their model on both single agent and multi-agent RGB alone BEV segmentation and LiDAR 3D detection.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is overall clearly written and addresses an important problem\n\n-The main contribution of the paper seems to be the sparse attention module (FAX). There are no ablations where everything is kept the same in the model, and instead a generic transformer is used to fuse information in the BEV map, or a SWIN transformer or a 2D CNN. the ablations consider previous fusion mechanisms of other methods (Table 1), and the reader does not know neither what fusion method the previous methods used, nor whether there are more changes between the present model and the models of the previous paper. The proposes sparse  transformer is never evaluated in apples-to-apples comparison with other choices.\n\n- recent papers on arxiv can obtain way higher numbers than the present paper without any transformer  for fusion of information and without even any multi agent collaboration, at least in the tasks of BEV segmentation, e.g., https://arxiv.org/pdf/2206.07959.pdf table 2. This casts some doubt on the validity of the conclusions of the present paper, since hyperparameters such as resolution or batchsize are not investigated, and the reader does not know if the gains survive such hyperparameter changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper addresses an important problem but does not provide convincing ablations of its main contribution, which is the FAX module and its numbers are significantly lower than recent papers in the literature. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Could you use your model architecture and compare FAX with a generic 2D transformer in the BEV space, a SWIN transformer and a 2D CNN with varying number of layers in terms of accuracy and computational cost? Could you repeat these comparisons with double the resolution of the BEV map?\n\n\nWhy building a BEV per agent is important? What if the multiview images across all agents are broadcasted, and a transformer directly attends into them to build the complete BEV map of the scene?\n\nI am happy to update my score if the authors address the questions above.\n\n\nPost rebuttal: I greatly appreciate the effort  of the authors for providing additional experiments. \n\nThe authors argue that https://arxiv.org/pdf/2206.07959.pdf showed up on arxiv post CoRL submission which is correct. I was pointing though  to the Table 2 of that paper, where you can see *earlier* papers, such as  BEVformer having *much* higher IoU in Nuscenes than the proposed model. Essentially this means the present paper is much below the SOTA. (BEVformer was submitted on arxiv on 31 Mar 2022)\n\nFro the comparison against other transformer variant and number of layers of those that the authors provide,  we see that the proposed model has marginally better performance (2%) with slightly less number of FLOPS . I think 2% difference is small given that changing the resolution had a 7% boost (as they report), and the best proposed method's variant provided at rebuttal time is already 5% below the SOTA in Nuscenes in comparison to BEVformer. \n\nBy broadcasting images i meant broadcasting perspective features instead of BEV features, apologies if this was confusing. \n\nOverall, this is a reasonable paper that is simply not moving as fast as the SOTA in the field of  multi-view BEV fusion. I deeply appreciate the effort of the authors, and I think their additional experiments have elucidated what really matters for performance in BEV. For the reasons outlined above, I maintain my original rating.\n\n"}}]

Author rebuttal:

Decision: Accept (poster)