BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities
Authors: Binyu Zhao
Wei ZHANG
Zhaonian Zou
Venue: CoRL 2023 Poster (2023)
Keywords: Multi-Agent Perception
Multi-Modal Fusion
Vehicle-to-Everything (V2X) Application

Abstract: Collaborative perception enables agents to share complementary perceptual information with nearby agents. This can significantly benefit the perception performance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most proposed approaches mainly focus on single modality (especially LiDAR), and not fully exploit the superiority of multi-modal perception. We propose an collaborative perception paradigm, BM2CP, which employs LiDAR and camera to achieve efficient multi-modal perception. BM2CP utilizes LiDAR-guided modal fusion, cooperative depth generation and modality-guided intermediate fusion to acquire deep interactions between modalities and agents. Moreover, it is capable to cope with the special case that one of the sensors is unavailable. Extensive experiments validate that it outperforms the state-of-the-art methods with 50X lower communication volumes in real-world autonomous driving scenarios. Our code is available at supplementary materials.

BM2CP: Efficient Collaborative Perception with LiDAR-Camera Modalities
Binyu Zhao, Wei Zhang", Zhaonian Zou School of Computer Science and Technology
Harbin Institute of Technology, China byzhao@stu. www.hiit..uc... (weizhang, znzou @hit. cn
Collaborative perception enables agents to share complementary per- ceptual information with nearby agents. This would improve the perception per- formance and alleviate the issues of single-view perception, such as occlusion and sparsity. Most existing approaches mainly focus on single modality (especially Li- DAR), and not fully exploit the superiority of multi-modal perception. We propose collaborative perception paradigm, BM2CP, which employs LiDAR and cam- era to achieve efficient multi-modal perception. It utilizes LiDAR-guide modal fusion, cooperative depth generation and modality guided intermediate fusion to acquire deep interactions among modalities of different agents, Moreover, it is ca- pable to cope with the special case where one the sensors, same or different type, of any agent is missing. Extensive experiments validate that our approach outperforms the statc-of-the-ee ofettheart methods with 50x lower communication volumes in both simulated and real-world autonomous driving scenarios. Our code is avail- able at https: // thub. com/byyzhaoAIBBMCC.
Abstract:
Keywords: Multi-Agent Perception, Multi-Modal Fusion, Vehicle-to-Evverything (V2X) Application
Introduction
Collaborative perception enables agents to share complementary perceptual information with their nearby agents. This would fundamentally alleviate the issues of single-agent perception, such as occlusion and sparsity in raw observations. Recently, different strategies have been proposed to implement collaborative perception. These approaches
be divided into three categories: LiDAR based collaboration [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], camera based collaboration [16, 17, 18. 19] and multi-modal based collaboration [20].
Intuitively, different types of sensors can provide heterogeneous perceptual information at different levels, and thus, more accurate perception could be achieved through multi-moda analysis. How- ever, most existing approaches are not multi-modal based methods, and present better performance only using LiDAR. Take camera and LiDAR as example, fusing the two modalities straightforwardly will bring negative impacts on perception performance, which is demonstrated by experiments in Ta- ble 3a. Camera captures rich semantics and contexts in a fixed view, but lacks the information of distance. Thus distance information, i.e. depth, will be estimated at first generally, which could help lift the camera representations from 2D to 3D to align with LiDAR representations. But the estimation brings uncertainty and have negative effect modal fusion and subsequent collaborative feature fusion.
Therefore, it poses challenges to build well-behaved collaborative perception method with LiDAR- camera modalities: (a) How to generate depth information? (b) How to fuse the LiDAR data and camera data effectively? (c) How to collaborate between agents with multi-modal data?
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

In this paper, we answer the aforementioned questions by proposing a Biased Multi-Modal Col- laborative Perception (BM2CP) method including three components: (a) cooperative depth gener- ation. A hybrid strategy is applied to provide more reliable depth distribution, which combines projection from ego and nearby LiDARs and prediction from ego camera; (b) biased multi-modal fusion. A preferable modal fusion is obtained by LiDAR-guided feature selection, which approve the importance of LiDAR representations and utilize it to select useful camera representations; (c) modality-guided collaborative fusion. A preference threshold mask is generated to filter bird's s-rde-- view (BEV) features, which achieves multi-views multi-modal critical feature sharing. Meanwhile, a flexible workflow makes BM2CP capable to deal with the case that one of the sensors, which is same or different type, of any agent is missing.
In summary, main contributions are threefold:
We propose novel framework for multi-modal collaborative perception. where modal fusion is guided by LiDAR and collaborative fusion is guided by modality. Besides, it can handle the case where modality data is incomplete.
We design LiDAR-guided depth generation and biased modal fusion, which achieves deeper interactions between LiDAR and camera modalities. The message containing in- formation of depths and features is exchanged among agents, which achieves better modal feature learning and efficient communication. These designs encourage sufficient feature fusion in both modality aspect and collaboration aspect.
Extensive experimental results and ablation studies on both simulated and real-world datasets demonstrate the performance and efficiency of BM2CP. It achieves superior per- formances with 83.72%/63 117%% 17% and 64.03 44.03% 9.9% AP at loU0.5/0.7 on OPV2V [9] and DAIR-V23 V2X [21]. When all camera sensors are missing, the performance is still comparable to other state-of- oo-the--rtt LiDAR based methods.
Related Works
Collaborative perception. As aforementioned stated, Most collaborative methods are LiDAR based, focusing on different issues such as performance [1, 8, 12], bandwidth trade-off [11, 18), communication interruption [7], latency [13] and pose error correction [5, 15]. For camera based methods, Xu et al. [18] propose the first attention-based multi-view cooperative framework, which shares features in BEV with Transformer [22]. Hu et al. [19] first conduct depth estimation and share it to reduce the impact of erroneous depth estimates. However, the depth ground truth is required to be collected in advance, which limits the generalization of the method. For LiDAR Racmmaa based methods, Zhang et al. [20] use limited number of image pixels, which are in 2D predicted bounding box, to generate virtual 3D point clouds. Whereas the modality for collaboration is still LiDAR. Xiang et al. [23] focus on constructing attention- based collaborative network. The method requires that different agents provide heterogeneous modalities.
Absence of sensor data. To the best of our knowledge, none of multi-agent collaborative perception approach addresses the absence of sensor data issue. Recently, Li et al. [24] propose a solution with RADAR and LiDAR for single-agent perception that fills the missing sensor data with zero value, and uses the teacher-student model with exponential moving average (EMA) to learn equally from both modalities.
BM2CP constructs flexible and straightforward workflow to overcome the absence of sensor data with available modalities. The proposed networks do not need any fine-tuning, Meanwhile, it is worth emphasizing that camera cannot accomplish the perception task independently, especially when each agent is only equipped with one camera. In this situation, reliable depth generation would become impossible and inferior perceptual information might be produced. This can be inferred from the research of Hu et al. [19] well. Their experiments on DAIR-V2X dataset [21] show that the perception generated through combining cameras and depth ground truth is much worse than that generated based on LiDAR data.

---

LiDAR
Foxel feuture
BEV feuture
Updated BEV Jeature
Biased Modai
Modality-guided
Voneláze
Fasion
Collaboration
Collapsc
Cooperative Depch
Decoder
Generation
intermediutee collaboration
Depik distrbuttion
exchange message
Encoder
Camera(s)
feutore
Vanel featore
Prediction in BEV
The framework of BM2CP. Colors indicate different modal voxels: orange for camera, which is obtained only from camera; blue for LiDAR, which is obtained only from LiDAR; green for LiDAR-camera hybrid, which is obtained from both modalities; and gray for normal, which receives no modal information from each modality and is filled with 0. In the object detection task example, green boxes ground truths and red boxes detected vehicles.
Figure 1:
Efficient Collaborative Perception with LiDAR-Camera Modalities
Our designs include (a) cooperative depth generation, which is guided by LiDAR and shares depth information among agents; (b) biased mulli-modal fusion, which achieves sufficient local feature learning; and (c) modality-guicn guided collaborative fusion, which shares critical BEV detection features to improve representation and performance. The overall framework is illustrated in Fig 1.
3.1 Problem formulation
To the best of our knowledge, there does not exist any problem formulation about multi-agent col- laborative perception with LiDAR-camera modalities. Here, we try to provide feasible definition of LiDAR and camera perception fusion in voxel space. Considering N agents in the scenario, let T, E RRx' xH the RGB image collected by the camera of the i-th agent, where W and are the height and width of image. Let P: E R. N
the point cloud collected by LiDAR the i-th agent, where N is the number of point clouds. Y; is the corresponding ground truth of detection. Agents exchange their positions and relative poses build communication graph.
In order to fuse modal features in voxel space, the point cloud and the image need to voxelized.
f Jvozelize (P.), Vi = Simg (I: @ S
(1)
where fvorelize (.) series of operations obtain voxel features from point cloud, which are similar the operation defines in Lang et al. [25]. f(mggeze(() and (deepsst(-) the feature extractor and depth estimator based on raw image. Plane features can be obtained through fing (-). The & operation produces the voxel features, which expands the plane features new dimension in Z-axis.
Then, we fuse point cloud features v E RXxxx and image features V E XXXXX by cell and collapse the fused voxel features v XXxxx to BEV.
fmodol Juee ( V.), F, feotlop (V!)
where fmodal.fase (..) and fcollap(.) denote the modality fuse operation and collapse operation, respectively. Collapse operation integrates the dimension of Z-axis to the channel dimension [25]. After collapsing, BEV feature F, R XXY will be packed and transmitted to nearby agents based on communication graph.
3

---

Each agent aggregates the received features with its own BEV feature and finally conduct prediction for a specific task, such as object detection or scenario segmentation.
= -feat fuse (Fi, (Fz-e)jeN,); Fidteeee (F:)
(3)
where F, is the aggregated feature. Fj-ii the warped feature based on relative poses of the j- th agent and the i-th agent. N₂ is the nearby agent set that the i-th agent can communicate with. ffatt_fsse() denotes the feature fuse operation and fdecoder denotes the decoder for prediction.
The objective of BM2CP is to minimize the distance between predicted and the ground truth detec- tion Σ, 9 9Y,, Y.), where ((:-) is the evaluation metric.
3.2 Cooperative depth generation
is applied to reduce the erroneous of estimation and obtain a reliable depth distribution. Prediction is used to predict pixel-wise depth using convolutional blocks. In order to reduce the number of model parameters, image feature extractor fimg.ent (+) and depth estimator ftep_est (-) utilize shared encoder and independent heads, i.e. depth head and image head. Each head is composed of several convolutional layers and training from scratch. Similar to the method in Reading et al. [26], the predicted depths are classified to a series of discrete values. The number of classes is the same as that of discretized depth bins. Projection is applied to transform LiDAR point clouds to RGB image coordinates. Let P; = [(T), Vi, 2i, 1)} be the homogeneous coordinates of point cloud, where (Ii, Ye. 24) is the 3D coordinate. Let Tidr2cam be the mapping from LiDAR sensor to camera sensor, and Team2img represent the mapping from camera sensor to the RGB image. The overall mapping from LiDAR RGB image is
hybrid peaaleeyppreiiciion&pppppp
1. Tidr2img P. Tidr2cam T Temm2img
where T, = {u,, v/ d)) is the projected image with depth information, and (u,, v4) is the 2D coordinate, d; is the corresponding depth of each pixel. Tidr2com and Team2ing equal to the extrinsic and intrinsic matrices of camera, respectively. The projected depths are also mapped to discrete values corresponding to the discretized depth bins.
Considering that some pixels may not have any projection depth, while some pixels have multiple depths to map. the hybrid strategy is implemented as follows: a) For pixel with no projection depth, it obtains the depth through prediction strategy; b) For pixel with only one projection depth, extra operation is required; c) For pixel with multiple projection depths, it selects the minimum depth. According to the principle of imaging, each pixel only presents the attribute of nearest object in reflected lights while the other objects are occluded. Therefore, is more reasonable to select the minimized depth which is the closest to camera.
On the other hand, point clouds from nearby agents contain different depth information. Intuitively, the 3D location of correct depth candidate is spatially consistent through viewpoints of multiple agents. Therefore, more reliable depth distribution could be obtained through communications. Since the depths projected from ego agent is more accurate, the depths from nearby agents are only used to replace predicted depths.
3.3 Biased multi-modal fusion
Motivated by the fact that LiDAR-based detectors usually surpass camera- counterparts, we take LiDAR as the guiding modality to achieve multi-modal fusion and generate fused voxel fea- tures v!. The illustration presented in Fig 2a. The voxels are grouped into four categories: LiDAR voxels, camera voxels, LiDAR- camera hybrid voxels, and normal voxels. LiDAR voxels and camera voxels are features only obtained from LiDAR branch and camera branch, respectively. Hybrid voxels are features obtained from both branches. Normal voxels denote it receives no modal information from each modalities, which are filled with 0.

---

LIDAR
Fassedl votel feature
Preferance Map
vorel feature
galle
decompese
identity mapping
collepee
ikenliy mupping:
Collapood
Confullnce Mask
Fizeed watel feature
BEV Jesture
BEY
global atfcuttion
Canera
noxel
To-be-shaared feutune
(a)
(b)
Figure 2: (a) MIDAR-guided modal fusion. (b) Modality-guided preference map and confidence mask generation. Colors indicate different modal voxels: orange for camera, blue for LiDAR, green for LiDAR-camers hybrid, and gray for normal.
For LiDAR-camera hybrid voxels, the fusion result üy is conditioned by LiDAR, which formu- lated as üн Conv( (Conv(vx.) ec, vr.)), where vr. and oe denote cells from the same position of LiDAR and camera, respectively. [t denotes concatenation.
For camera only voxels, LiDAR information is not contained in the same cell. Therefore, we conduct global attention to filter camera voxel features V The guidance matrix Ag,y, comes from overall LiDAR voxel features V
A.g.22 {
o MHA(VI V V threshold MHA(V) V threshold
(5)
where M MHA(: (s is the multi-head attention [22] which outputs the scaled dot-product attention weight. The threshold is empirically set 0.5. The final camera voxel sets {ūc} is formulated as (ūc] Ar.y, x Ve 2. which collects the features that Az,g.z is not O.
Since LiDAR voxels and normal voxels are not affected by camera voxel features, their fusion results can acquired through identity mapping. Finally, biased modal fusion is achieved and the fine-grained voxel features V are collapsed BEV feature.
3.4 Multi-agent collaborative perception
We design a modality-guidee collaboration to select the most critical spatial features and promote efficient communication, as Fig illustrated.
First, plane preference map T XXY is generated based the fused voxel features v X A threshold is assigned to each cell Tm.n of the preference map based set of voxels that can be collapsed to it in Z-axis. From the set of voxels, we select a preferred voxel to decide the threshold of corresponding cell. The preferred order is hybrid >LiDAR >camera >normal. Examples and more cases can be found at Appendix A.3. Since the hybrid voxels is guided by LiDAR and contains sufficient multi-modal information, the hybrid threshold is as For better performance- bandwidth eerdowmance-bandiidth trade-off, the threshold for other types cell is set 0.5.
Then, binary confidence mask M RXXY is generated based on the preference map T and the collapsed BEV feature F, E RXXY First, classification head fes(-) is used to evaluate the importance of each cell BEV feature, and fata E [0, 1]. On the other hand, preference map provides threshold Tm.n for each cell in BEV feature at the same location (m, n). Then we compare the value of importance the position (m, n) in fats fats(F)) and the threshold T Tm,.. When the value of importance greater than threshold, Mm.. and the cell the same location (m, n) in BEV feature F, is regarded as critical, and will be broadcast to nearby agents. When the value of importance is smaller, M 0.
5

---

Table 1: 3D detection quantitative results OPV2V dataset and DAIR-V22 V2X dataset. Comm is the communication volumes in log-scale.
Dataset
OPV2V
DAIR-V2N
Method
Comm
AP@0.5
AP@0.7
Comm
A.@0.5
A.@0.7
No Fusion
61.65
43.26
0
52.75
45.63
Late Fusion
18.43
74.27
57.45
18.62
53.88
38.12
When2com (CVPR'20)
20.17
69.12
53.76
20.31
51.88
37.05
V2VNet (ECCV'20)
22.56
78.54
59.42
22.90
58.80
43.75
DiscoNet (NeurIPS'21)
21.61
77.01
58.50
21.85
55.17
39.84
CoBEVT (CoRL'222)
21.07
81.37
61.32
21.31
50.70
39.39
V2X-ViT (ECCV'22)
20.21
80.92
61.23
20.45
56.75
39.90
Where2comm (NeurIPS' 22)
15.64
79.67
60.15
16.54
60.85
46.48
BM2CP
11.13
83.72
63.17
11.01
64.03
48.99
After collaboration, multi-head attention AH v) implemented aggregate these criti- cal BEV features from agents and generates updated BEV feature F,, where g k Fr, {F, je [
is concatenation operation and {-} denotes the feature set from nearby agent set N, The updated BEV feature is finally used to predict the detection with task-specific decoder.
3.5 Robustness against missing sensor
BM2CP is capable to deal with the case when one of the sensors is missing through flexible and straightforward workflow. Suppose that the camera sensor is now absent and RGB image is not available. In modal fusion step, LiDAR voxels and normal voxels remain, and they are used as the fusion results through identity mapping. In collaborative fusion step, since hybrid voxels exist, the threshold of cells is uniformly set as 0.5. The paradigm is similar when LiDAR sensor is missing. By conducting this, BM2CP adapts well to the missing sensor can be found at Appendix A.4. The experimental results are collected in Sec 4.2.
cases. More discussion about
Experimental Results
4.1 Experimental setup
Dataset. We conduct experiments of 3D object detection task on OPV2V dataset [9] and DAIR- V2X dataset [21]. OPV2V dataset is a vehicle-to-vehicle (V2V) collaborative perception dataset, which is co-simulated by OpenCDA and Carla [27]. The perception range is 40m 40m. DAIR- V2X dataset the first public real-world collaborative perception dataset. Each sample contains vehicle and infrastructure, and they equipped with LiDAR and front-view camera. The perception range is 201.6m x 80m.
Compared methods. We consider comparisons with following LiDAR-based methods: No Fu- sion, Late fusion, When2com [4), V2VNer (3), DiscoNer [18], V2X- VI7 [12] and Where2comm [11]. Among these methods, No Fusion is considered as the baseline which only uses individual observa- tion. Late fusion shares the detected 3D bounding boxes with nearby agents. Rest state-of-the-an (SOTA) LiDAR-bars -based collaborative perception algorithms.
Implementation details. We re-implement all methods based on the pyToorhh [28] framework and OpenCOOD [9] codebase with Adam [29] optimizer and multi-step learning rate scheduler. In order to compare communication volumes fairly, all compared methods use the same architectures and follows PointPillar [25] with feature compression. Weighted cross entropy loss used. The detection results are evaluated by Average Precision (AP) at Intersection-over-- Union (IoU) threshold of 0.50 and 0.70.

---

DicoNett
47000
20
14
(MMcI1-mbuut
0.5
tocaliaaion (n)
oml)
OPV2V.
DAIR-V2X.
Figure 3: Robustness to localization error. Gaussian noise with 0 mean and varying std is introduced.
Table 2: Evaluation results against missing sensor on DAIR-V2D V2X dataset. Camera or LiDAR denotes the sensor that the agent equipped with.
Ego agent
Nearby agent(s)
AP@0.5
AP@0.7
Camera
LiDAR
19.21
5.74
Camera
Both
20.41
6.05
LiDAR
LiDAR
61.59
47.30
LiDAR
Both
62.71
47.64
Both
Camera
53.31
43.86
Both
LiDAR
63.54
48.38
Both
Both
64.03
48.99
4.2 Quantitative Evaluation
Comparison with baseline and SOTA methods. Table shows the comparisons with recent meth- ods in terms of communication volumes and detection performance. Mathematically, the communi- cation volume is calculated in log-scale by (og2([F,-j])) where | is the LO norm which counts the non-zero elements in BEV features. It is observed that BM2CP: (a) achieves superior perception- communication trade-off; (b) achieves significant improvements over previous SOTA methods on both datasets; (c) achieves a better performance with extremely less communication volume, which 105 times less than baseline and 46 times less than Where2comm.
Robustness against the absence of modality. Table 2 shows the results with the setting of missing sensor. It is observed that: (a) the performance degrades severely when one of the agents is only equipped with camera sensor. This is consistent with the conclusion that the performance is poor when only camera sensor works; (b) The performance (61.59% on AP@ loU0.5 and 47.30% on AP@ (P@I)U0.7) comparable with SOTA methods, when agents only use LiDAR for perception; (c) The performance drop slightly when one of the agents miss the camera sensor.
Robustness to localization noise. We also evaluate the robustness to localization noise following the setting in V2VNet [3] and V2X-V77 [12]. Gaussian noise with mean of Om and standard deviation of Omn - 0.6m is used, and the results are shown in Fig 3. Unfortunately, performance degrading happens when localization noise increases. The reason comes from the cooperative depth generation. The shared depth distributions are broadcast based on the localization and relative pose of each agent. Therefore, it intensifies the errors and provides wrong depth information for images. To solve this problem, we correct relative pose before depth and feature communication and name BM2CP- Detailed process can be found Appendix A.5. Comparing the results of BM2CP. robust with BM2CP, the performance degrading gets alleviated evidently.

---

Figure 4: Qualitative results on DAIR-V2X dataset. Ground truths are colored in green and predic- tions colored in red. BM2CP detects objects than V2X-VIT and Where2comm in the first row, and detects less false positive objects in the second row.
Table 3: Component ablation studies DAIR-V2X dataset.
(a) Modal fusion strategy.
(b) Depth projection.
(c) Collaborative strategy.
Strategy No
AP@0.5/0.7
Projection
AP@00./007
Strategy
AP@0.5/0.7
50.17/37.33
No
52.84/38.07
Max
61.33/46.02
Equal
58.33/42.13
Ego
60.31/44.42
Concat
62.25/46.46
Biased
60.31/44.42
All
62.25/46.46
Attention
64.03/48.99
4.3
Qualitative Evaluation
Fig shows the comparison with No Fusion, V2X-VIT and Where2comm. BM2CP achieves more complete and less false negative detection. The reason that BM2CP leverages multi-modas feature fusion in critical voxels and employs modality-guided cell-level confidence mask to achieve more comprehensive fusion. We also visualize the projected depths, which be found at Appendix B.3.
4.4 Ablation Study
Ablation study is conducted to investigate the effectiveness of the main components in our method. The results are presented in Table 3. We also conduct ablation studies on the number of agents cameras. They can be found at Appendix B.4.
LiDdR- modal fusion. As shown in Table 3a, equally multi-modal fusion result in minor drop in performance. It proves the importance of the guidance by LiDAR.
LiseR- depth generation. We investigate the effect of using projected depths from ego and nearbys. Results Table 3b indicate LiDAR-base cooperative depth generation essential.
-oddlity-guided collaborative fusion. We compare of max fusion and concatenate fusion. Table 3c show that both max fusion and concatenate fusion lead to performance degradation and increase the demand of bandwidth.
masking-attentiin strategy with strategies
5 Conclusion
We propose a novel framework, termed BM2CP, for multi-agent collaborative perception, which fo- cuses on fusion with LiDAR-camera modalities. We adopt LiDAR-guided modal fusion to achieve reasonable and comprehensive modal feature learning, apply cooperative depth generation to en- hance the modal fusion with more reliable depth information, and propose modality-guicke collab- orative fusion for more efficient and critical feature fusion. Extensive experiments demonstrate the superior performance and the effectiveness of designed components.
Limitation and Future Works. Although BM2CP-robust significantly alleviates performance de- grading, it increases the runtime during training and test. Further efforts is needed to reduce the complexity of overall workflow and computation. Besides, validations on more datasets are critical prove its generalization.

---

Acknowledgments
We would like thank all the reviewers for their helpful and valuable comments.
References
[1] Q. Chen, S. Tang, Q. Yang, and S. Fu. Cooper: Cooperative perception for connected au- tonomous vehicles based on 3d point clouds. In 2019 JEEE 39th International Conference on Distributed Computing Systems (ICDCS), pages
IEEE, 2019.
[2] Q. Chen, X. Ma, S. Tang, J. Guo, Q. Yang, and S. Fu. F-cooper: Feature based cooperative per- ception for autonomous vehicle edge computing system using 3d point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88-100, 2019.
[3] T.-H. Wang. S. Manivasagam, M. Liang, B. Yang. W. Zeng, and R. Urtasun. V2vnet: Vehicle- to- vehicle communication for joint perception and prediction. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 605-621. Springer, 2020.
[4] Y.C.. Liu, J. Tian, N. Glaser, and 7. Kira. When2com: Multi-agent perception via communi- cation graph grouping. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106-4115, 2020.
[5] N. Vadivelu, M. Ren, J. Tu, J. Wang, and R. Urtasun. Learning to communicate and correct pose errors. In Conference on Robot Leurning, pages 1195-1210, PMLR, 2021.
[6] Y. Li, S. Ren, P. Wu, S. Chen, C. Feng, and W. Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541- 29552, 2021.
[7] S. Ren, Z. Lei, Z. Wang, S. Chen, and W. Zhang, Robust collaborative perception against com- munication interruption. the 2nd IJCAI Workshop on Artificial Intelligence for Autonomous Driving. 2022.
[8] Y. Yuan, Cheng, and M. Sester. Keypoints- deep feature fusion for cooperative vehicle detection of autonomous driving. IEEE Robotics and Automation Letters, 7(2):3054-361,, 2022.
[9] R. Xu, H. Xiang, X. Xia, X. Han, J. Li, and J. Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In 2022 International Conference on Robotics and Automation (ICRA), pages 2583-2589. IEEE, 2022.
[10] Cui, H. Qiu, D. Chen, Stone, and Y. Zhu. Coopernaut: end-to-end driving with cooperative perception for networked vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 17252-17262, 2022.
[11] Y. Hu, Fang, Lei, Y. Zhong, and S. Chen. Where2comm: Communication-efficci collab- orative perception via spatial confidence maps. In Advances in Neural Information Processing Systems, volume 35, pages 4874 4886, 2022.
[12] R. Xu, H. Xiang, Z. Tu, X. Xia, M.-H. Yang, and J. Ma. V2x-vit: Vehicle-to-everything coop- erative perception with vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 107-124. Springer, 2022.
[13] Z. Lei, S. Ren, Hu, W. Zhang, and Chen. Latency-aware collaborative perception. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel. October 23-27, 2022, Proceedings, Part XXXII, pages 316-332. Springer, 2022.

---

[14] Y. Li, J. Zhang, D. Ma, Y. Wang, and C. Feng. Multi-robot scene completion: Towards task- agnostic collaborative perception. In Proceedings of The 6th Conference on Robot Learning, pages 2062-2072. PMLR, 2022.
[15] Y. Lu, Q. Li, B. Liu, M. Dianati, C. Feng, S. Chen, and Wang. Robust collaborative 3d object detection in presence of pose errors. IEEE International Conference on Robotics and Automation (ICRA). May 2023.
[16] N. Glaser, Y.-C. Liu, J. Tian, and Z. Kira. Overcoming obstructions via bandwidth-limite multi-agent spatial handshaking. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
IEEE, 2021.
[17] Y. Zhou, J. Xiao, Y. Zhou, and G. Loianno. Multi-robot collaborative perception with graph neural networks. IEEE Robotics and Automation Letters, 7(2):2289-2296, 2022.
[18] R. Xu, Z. Tu, H. Xiang, W. Shao, B. Zhou, and J. Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. In Proceedings of The 6th Annual Conference on Robot Learning, pages 989-1000. PMLR, 2022.
[19] Y. Hu, Y. Lu, R. Xu, W. Xie, S. Chen, and Y. Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9243-9255 9252, 2023.
[20] H. Zhang, Luo, Y. Cao, Y. Jin, and Y. Li. Multi-modal virtual-real fusion based trans- former for collaborative perception. 2022 IEEE 13th International Symposium on Parallel Architectures, Algorithms and Programming (PAAP), pages 1-6, 2022.
[21] H. Yu, Y. Luo, M. Shu, Y. Huo, Z. Yang, Y. Shi, 7. Guo, H. Li, X. Hu, J. Yuan, et al. Dair-v2x: A large-scale dataset for vehiole-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21361- 21370, 2022.
[22] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[23] H. Xiang, R. Xu, and J. Ma. Hm-vit: Hetero-modal vehicle-to-vehicle cooperative percep- tion with vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 284-295, October 2023.
[24] Y.-J. Li. Park, M. O'Toole, and K. Kitani. -odality- learning for aadar-lidar fusion in vehicle detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 918-927, 2022.
[25] A. H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast en- coders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 12697-12705, 2019.
[26] C. Reading, A. Harakeh, J. Chae, and S. L. Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8555-8564, 2021.
[27] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[28] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, al. Pytorch: An imperative style, iigh-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[29] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint axXiv:171.05101,
2017. 2017.
10

---

Appendix
A Method
A.1 Motivation
Our motivation comes from two aspects. Firstly, depth is the major gap for RGB image to lift up voxels, but part of correct depths could be acquired from LiDARs of agent or nearby agents. Thus, depth information could be transmitted among modalities of agents. This allows compensation from LiDAR in different views, and reduces the uncertainty of the infinite depth prediction. Secondly, the transmitted data should include detection clues to provide refined and complementary information, which can fundamentally overcome inevitable limitations detected by single modality or single agent.
A.2 Coordinate transformation among LiDAR, camera and RGB image
If the LiDAR coordinate system is regarded as the world coordinate system, the 3D coordinate of point W could be:
Tworld
W Wworld
Dworld
(6)
zworld
We also have the camera coordinate and image coordinate of point W as
Zeom
W car
Yeam
Wing
Timg
Yimn
Zeam
Its world homogeneous coordinate in camera coordinate system and its camera homogeneous coor- dinate image coordinate system are
Eworld
Vingg
W world.h
Yworrd
W
Wing.h
Vang
Zworld
Suppose that É is the transformation matrix from LiDAR coordinate system camera coordinate system and I is the transformation matrix from camera coordinate system to image coordinate sys- tem, The inverse matrix of E and matrix I are
Ty1
t
fz
u
E4
Tx2
Ty2
Te2
ty
Tax3
0
fy
Ty3
tz
0
1
Generally, and are the extrinsic matrix and intrinsic matrix of camera, which are given by dataset [21].
Then we have
W Weam = EAx4 W Wworld h. Wsmg
I8x3 * W Weam
(10)
Zeam
A.3 Generate preference map
For example, when the voxel set is compased of {Ulybbrdd Veamera, Unormat }. the Uhgbrid is the pre- ferred voxel and the threshold of corresponding cell in preference map is assigned as hybrid thresh- old; When the voxel set composed of [VLADAR. 11 "orrmat 122 )ormat)) the VlidAR is the preferred voxel and the threshold corresponding cell preference map assigned as LiDAR threshold.
We also visualize four typical cases when generating cell of preference map, which are shown in Fig 5.

---

Fused
Voxel
Feature
Preference
Map
Figure 5: Four typical cases when generating one cell preference map.
A.4 Robustness against missing sensor
There mainly 2 cases of missing sensor. Case is shown in Fig 6a. Suppose that the camera sen- sor is now absent and RGB image is not available. In modal fusion step, LiDAR voxels and normal voxels remain, and they are used as the fusion results through identity mapping. In collaborative fusion step, since no hybrid voxels exist, the threshold of cells uniformly set as 0.5. The paradigm similar case 2 when LiDAR sensor missing. The modal fusion step shown in Fig 6b. By conducting this, BM2CP adapts well the modality-unan unavaalale cases.
decompose
gate
decompose
gale
identity mapping
identity mapping
identity mupping
identity mapping
global attention
identity mapping
(b)
Figure 6: The workflow of biased multi-moda fusion when camera (a) LiDAR (b) is missing. Existing LiDAR voxels and normal voxels in case (a), or camera voxels and normal voxels in case (b) will conduct identity mapping the final fusion results. Colors indicate different modal voxels: orange for camera, blue for LiDAR, green for LiDAR-camera hybrid, and gray for normal. Trans- parent cells and dashed arrows indicate the corresponding type of voxel features do not exist.
A.5 Robust BM2CP
BM2CP-robust corrects relative pose before depth and feature communication. Specifically, single- agent 3D object detection is first conducted to estimate local bounding boxes and their uncertainty for each agent with LiDAR voxel features. Then we conduct internal agent-object pose graph opti- mization [15] for each agent correct relative pose Ez-+i, where and j are ego agent and nearby agent, respectively. The corrected relative pose used correct shared depth maps and BEV fea- tures.
12

---

B Experiments
B.1 Detailed settings of architecture
follow the default settings in OpenCOOD [9] codebase, which also shown in Tab
Table 4: Details of unified network architecture.
Blocks
Settings
Voxel Feature Encoder (VFE)
use normalization and absolute 3D coordinates, 64 filters 64-channel output
PointPillar Scatter
ResNet backbone:
layers= 4, 5]
strides= 2,
BEV backbone
filters: 128, 256
upsample_strids=(1122
gpsample.filters(1225 128, 128)
Shrink Header
shrink from 384 channels to 256 channels with stride 3
Detect Head
256-channel output with anchors
B.2
Detailed settings of experiments
Table Details of unified network architecture.
Method
optimizer
Ir schedule
initial Ir
No Fusion
Adam
multistep
le-3
Late Fusion
Adam
multistep
le-3
When2com (CVPR'20) 20)
Adam
multistep
le-3
V2VNet (ECCV'20)
Adam
multistep
le-3
DiscoNet (NeurlPS'21)
Adam
multistep
2e-3
CoBEVT (CoRL'222)
Adam
multistep
2e-3
ViT-VIT (ECCV'22)
Adam
multistep
2e-3
Where2comm (NeurIPS' 22)
Adam
multistep
2e-3
BM2CP
Adam
multistep
le-3
B.3 More Visualizations
Visualization of projected depths. Fig shows how depth distribution is empowered by LiDAR in projection design. In the scene, objects show evident contour against the background with lighter coloring. And more depths from nearby agents will further fill the depth in empty (white pixel in image).
Figure 7: Two visualizations of projected depths from LiDAR coordinates to image coordinates. Paired arrows in colors indicate the same objects including car and sign in LiDAR data (left), camera data (right) and projected depth map (middle), respectively.
13

---

More visualizations of comparison with SOTA methods. Fig shows more comparisons with No Fusion, V2X-Vi7' and Where2comm.
0:00
1:000
D:A
(a) No Fusion
(b) V2X-ViT
(c) Where2comm
(d) BM2CP
Figure 8: More qualitative results in DAIR-V -AIR-V2X dataset. Ground truths are colored in green and predictions are colored in red.
B.4 More Ablation Studies
Number of agents. We study the influence brought by the number of collaborative agents. As shown in Table 6a, increasing the number of collaborative agents generally bring performance improvement on OPV2V dataset, whereas such gain will be more marginal when the number is greater than 4.
Robustness camera dropout. We demonstrate performance when the ego agent carries [1, 4] cameras in Table 6b. be seen that by the performance decreases but still maintain in acceptable level.
Table 6: Ablation studies of the number of agents and cameras on OPV2V dataset.
(a) The number of agents.
(b) The number of cameras.
Number
AP@0.5
A.@
Number
AP@0.5
AP@0.7
58.05
28.78
79,84
59.07
67.83
34.66
80.02
60.55
3 75.73
48.57
82.57
61.68
79.29
57.85
83.72
63.17
5
83.72
63.17
14


Reviews:
[{"review_id": "vePaivyqZEJ", "reviewer": "Reviewer_oA8p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new fusion-based representation learning framework under the multi-agent and multi-modality (camera, LiDAR) setting. The fusion module takes LiDAR as input and decides which (camera/LiDAR) feature from which agent should be used. The authors conduct experiments on and compare with multiple strong baselines to show that their 3D object detection performance is better. They also show how their framework can handle the scenario where some sensors are missing.", "summary_of_recommendation": "I think the paper has a certain value in proposing a new framework for multi-agent multi-modal fusion for perception representation learning. Though they show strong results compared to many baselines, the paper is not written in a very clear way, the algorithm is only tested on one dataset and some important ablation studies are missing. Thus my decision is weak reject.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "It will be great if the authors could address the questions/content from the **Weaknesses** section above. Besides:\n\n1. What are the training time and the inference runtime for the proposed algorithm and the baselines? What is the hardware needed to run the algorithm?\n2. In the \"Cooperative depth generation\" procedure, does the depth estimation network module use any pre-trained model, or entirely train from scratch?\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The quality of the paper is okay. The fusion idea is a bit incremental, but I believe this framework might still be valuable in scenarios like autonomous driving and multi-robot collaboration tasks.\n\n**Strengths**:\n1. Strong detection results: the authors compare with two simple baselines and five SOTA approaches and consistently outperform other methods in the average precision measure. \n\n**Weaknesses**:\n1. Clarity: The problem formulation and methodology part is not very clear to me. Specifically, equation 1 provides little information: what is the $\\otimes$ operator, and how is it formulated? What are normal voxels mentioned in line 139? In equation 7, what are the mappings $T_{lidar2cam}$ and $T_{cam2img}$, and why formulating $P_i$ like a 4d vector? My best guess is that those T are extrinsic/intrinsic matrices and P is like homogeneous coordinates, but the authors should make it clear in the paper. The categorical depth distribution in line 122 is not clear at first glance - I guess it might be related to the discretized depth bins mentioned in line 158, but again the authors should make it clear.\n2. Generalizability: The authors only use one dataset (DAIR-V2X) throughout the paper. But there are also other datasets like OPV2V [1] and V2X-Sim [2], which have more agents (the authors mention that in DAIR-V2C only 2 agents are considered). Without conducting experiments on different datasets, it is hard to say whether this framework can be generalized to different datasets.\n3. Ablation studies: there are many thresholds (line 147, line 186, line 188) and gate structures (ReLU, Sigmoid, Attention) in this framework. The authors should do experiments to provide the rationale they choose the one shown in the paper.\n\n**References**\n\n[1] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Liu, and Jiaqi Ma. OPV2V: An open benchmark\ndataset and fusion pipeline for perception with vehicle-to-vehicle communication. ICRA, 2022.\n\n[2] Yiming Li, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. V2X-Sim: A virtual\ncollaborative perception dataset for autonomous driving. IEEE Robotics and Automation Letters, 7, 2022.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "pSWt4hwhHQ", "reviewer": "Reviewer_bD4g", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a multi agent collaborative perception framework with LiDAR-Camera modalities.\n1, A LiDAR guided modal fusion strategy is proposed to fuse the LiDAR and camera feature.\n2, Modality guided  collaborative fusion is proposed to achieve efficient and effective multi-agent communication.\n3, The author reports superior performance on public datasets and robustness in handling missing camera sensor.", "summary_of_recommendation": "This paper has some novel and interesting ideas and is well written and presented.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Some Camera-LiDAR related works could be included and compared in the \"related works\" section:\n\n[1] \"CamLiFlow: Bidirectional camera-LiDAR fusion for joint optical flow and scene flow estimation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2]  \"Perception-aware multi-sensor fusion for 3d lidar semantic segmentation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3] \"Advancing self-supervised monocular depth learning with sparse liDAR.\" Conference on Robot Learning. PMLR, 2022.\n\nLine 174: LIDAR -> LiDAR", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths:\n* The problem this paper solving is interesting and important.\n* Novel multi-modal feature fusion methods are proposed and evaluated in detail.\n* Detailed illustration of method and experiment.\n* Interesting analyze of the robustness against missing sensor\n\n# Weaknesses:\n* Analyze of the computational cost is missing\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "1zhFn8gvYI", "reviewer": "Reviewer_UhGW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a novel problem formulation and a new framework for multi-agent collaborative perception given both LiDAR and camera inputs. Experiments are done using the DAIR-V2X dataset and the proposed method is experimentally shown to be superior than several baseline methods.", "summary_of_recommendation": "I have no experience in this field of study and my review should only be regarded as an educated guess. Overall, the proposed new task and method have merits, but maybe not very significant. The paper writing should be improved as well.\n\nPost rebuttal: Thanks for authors' rebuttal. I've raised my score to Weak Accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "see weaknesses", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- the paper is the first to present the new task of multi-agent collaborative perception given both LiDAR and camera inputs\n- the proposed method is reasonable and novel in its cooperate depth generation and bias multi-model fusion techniques\n- the comparison to baselines is quite comprehensive and the proposed method achieves the best performance \n\nWeaknesses:\n- the paper writing can be improved, especially on the equations and figures. The figures are usually not well explained by the captions. Also, many operations in equations are not explained very clearly in the text.\n- the novelty of the new problem is low as there are plenty works on studying how to combine lidar and camera inputs and also many works investigating multi-agent collaboration. The proposed problem is new, but not significantly different from other related tasks.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "0qiS6x0giFc", "reviewer": "Reviewer_TJuY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The manuscript presents a new approach called BM2CP for multi-modal collaboration between multiple vehicles to address perception tasks in BEV space. The example given in this paper is 3D object detection.\n\nThe main contributions of this work are the following: First, a new LiDAR-camera fusion scheme to generate voxel features in BEV space, where the fusion is primarily guided by the LiDAR measurements. This fusion scheme is capable of handling the absence of either the LiDAR or the camera sensor. Second, a data communication approach to efficiently share information between agents in a local environment on a feature level.\n\nThe proposed approach is evaluated on a single dataset, namely DAIR-V2X, and demonstrates promising results given ground truth poses of the agents.", "summary_of_recommendation": "While I do appreciate the general idea of the submitted work, the manuscript in its current form requires major rewriting on a language level to clarify open questions. I would encourage the authors to address both of the listed limitations as they do not point to future research but are mostly related to a more in-depth evaluation of the developed method.\n\nPOST-REBUTTAL:\nThe authors extensively replied to all of my questions eliminating the majority of my concerns. Based on the rebuttal, I changed my recommendation from \"weak reject\" to \"weak accept\" assuming that the authors will update their submission with the provided revision.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Why is the minimum depth selected for pixels with more than one projected depth values? What is the intuition behind this, e.g., as opposed to averaging?\n- In line 172, what is meant by \"part pixels\"?\n- Is the confidence mask generated only based on the values of the preference map?\n- What are the numbers mentioned in Sec 4.2.1 (b) based on? Taking the results from Table 1 does not yield improvements of 5.2% and 5.4% for IoU0.5 and IoU0.7, respectively.\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors address an important aspect of the large-scale deployment of robots. They showcase a key advantage of leveraging sensor information across agents to extend the field of view and overcome sensor failures by bootstrapping measurements from nearby agents.\n\nThe general structure of the manuscript is adequate and manages to guide the reader through the approach by introducing the components step-by-step.\n\nThe biggest problem of the manuscript in its submitted form is that it suffers from several language (mostly grammar) errors that render certain sections difficult to read, e.g., the last paragraph in Sec. 2, and/or make it extremely hard to understand the technical method, e.g., Sec. 3.5.\n\nIn some sections, the manuscript fails to provide sufficient details to fully comprehend and evaluate the proposed method. Examples include the generation of depth prediction if no LiDAR data is available. The manuscript vaguely state \"using neural network\", more technical details would be appreciated.\nSimilarly, how are the cells of the voxel features collapsed along the z-axis? The classification into hybrid, LiDAR, etc. is clear but I do not understand which mathematical operation is used to aggregate the original voxels.\n\nA major downside of the proposed method is the fact that it relies on ground truth poses. While this is mentioned in the limitations section, it should be communicated more clearly and earlier in the manuscript. In particular, as the authors state that the performance drastically drops if they add artificial noise to the poses. A more detailed ablation study including a quantitative analysis would help to better evaluate the performance of the method and whether publication at the current stage can be justified.\n\nSimilarly, the evaluation of the method is not very detailed given that the authors only use a single dataset. While it may be difficult to assess other datasets due to the limited availability of collaborative data, a fairly simple solution would be to leverage simulators such as Carla.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for collaborative perception with LiDAR and camera modalities. All the reviewers find the problem being addressed very interesting and the proposed solution is novel. In the initial review, the reviewers raised several concerns, most of which were addressed in the rebuttal. The major concern was the poor writing, concerns on generalization ability since the approach was only evaluated on one dataset, and clarity on the method requiring ablations. All the reviewers are satisfied with the responses and clarifications provided by the authors.
Decision: Accept (Poster)