Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper presents a collaborative perception model that specifically focuses on reducing communication using spatial confidence maps. These maps allow agents to decide which areas of the scene they need further information from, as well as which areas of the scene they have relevant information. The authors use attention to allow the agents to attend their, and other, confidence maps to create a communication graph. The authors then use a Transformer Architecture with Multi-Head Attention to fuse the features from multiple agents. The fused features will act as the agents input features in the next iteration, and they are also fed to a decoder that can regress the relevant information (class and bounding box). Overall, the approach achieves significantly lower communication rates, while enabling the network to also use similar communication Volumes as the state of the art.
 
strengths_and_weaknesses: The paper is well presented, generally reads well and provides a good case for the authors Spatial Confidence maps as a way to reduce communication and increase performance in collaborative perception. The quantitative data suggest the model can outperform the state of the art on the evaluated datasets. It shows a strong ability to reduce the amount of data, while still outperforming all the benchmarks. There is a massive reduction in the amount of data used, which should be commended. The qualitative data shows much improved performance in the detection. Furthermore, the ablation study cleary shows improvements introduced by each part of the architecture.

In terms of weaknesses, the main limitation is that the models is only evaluated on synthetic datasets with favourable conditions. A discussion of limitations including things like time synchronisation, noise, availability of data and other considerations are important to present on an approach that relies on collaborative perception. Additionally, the paper has various spelling and grammar mistakes that should be rectified.

questions: 1. You mention using distance between the sensor and the feature to perform positional encoding. How does this work for a monocular camera? Do you assume known depth?
2. Can you elaborate in the warping function from iage to BEV? How is this performed? This is an active area of research, see e.g. Saha, Avishkar, et al. "Translating images into maps." arXiv preprint arXiv:2110.00966 (2021). and Roddick, Thomas, and Roberto Cipolla. "Predicting semantic map representations from images using pyramid occupancy networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
3. Why BEV? Have you looked at other ways of unifying the data? Could this be a learned transformation applied to the data?
4. For BEV, how robust is your system to distortions introduced by ground projection?
limitations: The work is only evaluated in simulation. This presents important challengs that must be solved. The authors should include limitations in terms of time alignment, assumptions about sensor modalities (is depth required?) and other consequences of moving into the real world. The authors mention expanding their idea into the temporal dimension, which seems like a very good idea. Although another possible limitation of their approach is that as the dimensionality of the problem gets bigger, the attention component might not be capable of attending to all the relevant information, resulting in sub-standard communication.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Multi-agent collaborative perception could avoid accidents due to an agent's limited field of view via sensing information from other agents. However, the communication bandwidth is a bottleneck for multi-agent collaborative perception systems. The critical question is what to send and whom to send it. The authors propose a novel framework called Where2comm for 3D detection to address the challenges. A spatial confidence map is proposed to advance the performance of multi-agent collaborative perception. The paper benchmarks the framework on three datasets, OPV2V, V2X-Sim, and our original CoPerception-UAVs. The framework achieves state-of-the-art performance on these benchmarks. Moreover, the authors show that the proposed method can significantly reduce communication bandwidth while maintaining favorable performance compared with STOA.
strengths_and_weaknesses: Strengths:
1. To reduce the number of traffic fatalities, multi-agent collaborative perception for 3D object detection is a promising direction, and it is a topic of interest to machine learning, computer vision, and intelligent transportation system researchers.
2. To the best knowledge of the reviewer, the concept of spatial confidence map has not been leveraged for multi-agent collaborative perception. The authors proposed an architecture to realize the concept and used the encoded information to send critical information for 3D object detection. The reviewer appreciates the proposed architecture as it improves interpretability. In addition, because of the design of a spatial confidence map, the framework can send sparse yet critical information to other agents and show improvement in 3D object detection.
3. Promising performance in 3D object detection AP and communication volume usage is demonstrated on three datasets, OPV2V, V2X-Sim, and CoPerception-UAVs. The experiments empirically demonstrate the effectiveness of the proposed framework.
4. Qualitative evaluations show that the proposed method **exhibits** promising explainability. Toward trustworthy multi-agent collaborative perception, such attributes would be much appreciated.


Weaknesses:
1. Related work: The reviewer found that the proposed spatial confidence map is highly relevant to the probabilistic occupancy map and the concept of a local grid map for robotics applications. The reviewer encourages the authors to acknowledge the existence of these works.
    1. H. Li, M. Tsukada, F. Nashashibi, and M. Parent, "Multivehicle Cooperative Local Mapping: A Methodology Based on Occupancy Grid Map Merging," in *IEEE Transactions on Intelligent Transportation Systems*, vol. 15, no. 5, pp. 2089-2100, Oct. 2014
    2. N. Deo and M. M. Trivedi, "Convolutional Social Pooling for Vehicle Trajectory Prediction," CVPR 2018
    3. C. Chen, Y. Liu, S. Kreiss, and A. Alahi, "Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning," *ICRA 2019*
2. Perfect localization: According to the reviewer's understanding, the proposed framework relies on the accurate localization of all agents in a scene. Without the assumption, a spatial confidence map cannot be applied easily. The reviewer felt that noisy localization could lead to significant performance degradation for 3D object detection. Toward real-world applications, it is inevitable to encounter such issues. It would be an informative study to analyze the sensitivity of the proposed method on noisy localization. Moreover, what is the corresponding performance compared with the existing STOA?
3. Communication volume: The reviewer has difficulty understanding how the authors adjust the communication volume shown in Figures 3 and 4. The reviewer's best guess is relevant to Figure 2, shown in the supplementary material. However, it is hard to reproduce the results without a clear explanation.
4. Ablative studies:
    1. What is the baseline implementation without using SCM? Is it simply using BEV features?
    2. Spatial confidence-aware message fusion module: What if we use max-pooling/concatenation for fusion?
5. Can the proposed algorithm solve pedestrian or other traffic participant detection? What could be the potential limitations?
questions: The paper presents a promising framework for Multi-agent collaborative perception for 3D object detection. The authors motivate the readers and propose a spatial confidence map that can be used to communicate critical information with other agents and reduce communication bandwidth. The experiments empirically demonstrate the effectiveness of the proposed framework. The reviewer recommends accepting the paper. The current rating is weak accept (6). The reviewer's main concern is the sensitivity of the framework to noisy localization, which might significantly degrade the contribution of a spatial confidence map in a real-world setting. 
limitations: The authors did mention limitations of the work on the lack of considering the temporal aspect of critical timestamps.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework, where2comm, to enable multi-agent collaborative perception via a more improved information sharing through communications among agents. The core idea of where2comm is the use of a spatial confidence map, enabling each agent to decide which area needs to be shared and communicated about with other agents. Where2comm is composed of (1) spatial confidence generator (2) spatial confidence-aware communication module, (3) spatial confidence-aware message fusion module. Figure 2 well elaborates the overall structure of the system; the camera/lidar input goes through the encoder first, and the output of the encoder goes through the spatial confidence generator, which generates spatial confidence map that shows which areas in the map is perceptually critical level. The use of spatial confidence map is leveraged during the communication phase, which selects where to communicate about. 
strengths_and_weaknesses: Strength of this paper comes from the originality of the framework, especially novel modules like message packing and communication graph construction. Message packing includes request map, which indicates where (which areas) needs to be further known, and is computed from negative of the confidence score. Also, there is spatially sparse map, which are computed from requests from other agents. 

Communication graph construction identifies when and who to communicate by measuring overlap between information i-th agent has and information j-th agent needs. 

I think the overall strength of this paper comes with sound problem formulation, the need for a fraemwork for multi-agent collaborative perception, and original/novel framework proposals. Weak points (or the points that need to be clarified) will be asked in the Questions section.

Another strength is that the where2comm achieves thousand times less communication volume and still outperforms other baselines, (Figure 3).
questions: Q1. The select function in the message packing module: to me, select function is not well described in the section. Selection function takes the element-wise  multiplication of confidence map of i-th agent C_i and request map of j-th agent R_j as inputs, and returns the output by selecting the largest elements of the given input. What does largest elements mean? Under which criteria the inputs of the C_i*R_j survive? 

Q2. Communication graph is constructed from the adjacency matrix A_ij^k, which is the result of the maximum value of the binary selection matrix M_ij^k. It seems that A_ij^k is matrix is binary (0 or 1) matrix; what is the form of M_ij^k matrix? Are A and M both binary matrix? Can you visualize M matrix and A matrix?

Q3. Figure 5 (visualization) is very helpful. (b) is the confidence map, which indicates that the drone1’s view is more confidence about the non-dark area in the (a). How should I interpret (g)? It seems that this is the request map from the drone2 view. What does the red spots in the (g) request map represent? 
limitations: The authors mentioned about the limitations, which is that this paper only focuses on perceptually critical spatial areas, and doesn't care about temporal dimension. I think this is reasonable limitations and suggestions for the future work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the task of multi-agent collaborative perception with the purpose of achieving better trade-off between performance and bandwidth consumption. To reduce the communication cost across the agents, the authors propose a new approach to generate and transmit the spatial confidence map instead of the traditional whole feature map.  The framework proposed in this paper consists of five modules, including the observation encoder, spatial confidence generator, spatial confidence-aware communication module, communication graph construction, spatial confidence-aware message fusion and detection decoder. The authors have conducted experiments on three datasets to verify the effectiveness of their new approach. Compared with five baselines, the new approach has shown higher detection accuracy and smaller communication cost.
strengths_and_weaknesses: Strengths:
1. It is a good idea to transmit part of the feature map across the agents to reduce the communication volume, and potentially improve detection accuracy. 
2. The authors have conducted extensive experiments on three different datasets to validate the new approach. The experiments cover two types of agents (cars and drones), and two types of sensors (LiDAR and camera). They also present many experimental results to support the conclusion, including quantitative comparison and visualization.

Weaknesses:
1. It’s a very complex model with 5 modules. Although the authors present the high-level ideas of each module, a lot of details on the architectures and optimization techniques are missing. For example, how the constrained optimization presented in Sec. 3 is optimized? How are binary variables in M (Eq. 2) and A optimized? Are there any optimization issues due to hard constrains and non-differentialability of binary variables? 

2. The authors compare their proposed method against five baselines. I have checked the original papers of the baselines, and found that only DiscoNet reports the results on V2X-Sim, which is one of three datasets in this paper. Since most results of five baselines reported in this work are not directly from the original papers, a solid comparison between the proposed approach and the baselines is required. Apparently, Fig. 3(c) shows that the performance gap between DiscoNet and Where2Com on V2X-Sim is much smaller than the other two datasets, while V2X-Sim is the only dataset that DiscoNet reported its performance.

3. The performance of communication volume varies much on three different datasets, 5128 times less on CoPerception-UAVs, 10,000 times less on OPV2V, and 55 times less on V2X-Sim. It seems that the authors do not introduce the details of the metric for communication volume. I am not sure whether the baselines used the same metric before. The comparison also has the potential fairness issue.

4. Overall, given the presentation of the paper and a large amount of details are missing, I don’t think the algorithm can be easily reproduced by readers.

questions: 1. Where are the results of the five baselines from?  Did the authors run the public source codes of the baselines on the three datasets?  Could the authors provide some details in the paper?

2. In the original paper, DiscoNet has reported detection accuracy of 60.3% and 58.5% in two different versions on dataset V2X-Sim, without and with 16 times compression, respectively. However, the accuracy of DiscoNet is around 58.0% in this paper. Why the results in two papers have the difference?

3. Where is the dataset CoPerception-UAVs from? It seems it's not a public dataset. If so, more details on the dataset would be helpful. For example, collecting data from drones is much trickier than from self-driving cars since drones can have any arbitrary poses. So more details are required. Also, how is training/val/test conducted? The appendix doesn’t contain dataset details even though the main text claims so.

4. How are the constrained loss function and binary variables optimized? 

5. Can you elaborate more on "sensor positional encoding"? Why should it represent the physical distance between each agent’s sensor and its observation? There is neither justifications nor references provided.

6. Given the complexity of the proposed method, it would be great if the authors could provide the code (the same as the baseline methods). So people check for details that aren't provided from the paper. 

limitations: Yes
review_title: null

Meta-review: metareview: This paper proposes a multi-agent collaborative perception algorithm where agents exchange perceived sensors (e.g., LIDAR) and share their observations with other agents sparsily by maintaining spatial confidence maps that determine the communication connectivity matrix. Communication happens over multiple rounds and incoming messages are fused using multi-head attention. The method is evaluated on synthetic drone and car data from popular simulators, where it achieves superior results with significantly lower communication volume.

Reviewers praised the experiements and the performance of the model (cKVM, VaH1, Jgi5), the large reduction in communication overhead (cKVM, YUZ9), the novelty of the confidences-map based communication framework (VaH1, YUZ9, Jgi5).

Reviewer cKVM noted that the model was evaluated only on synthetic data (cKVM) and a missing section on data synchronisation and availability; in the rebuttal, authors argued about limited availability of simulated and real-world data, and explained how depth was extracted from RGB observations. Reviewer VaH1 suggested existing literature on local grid map pooling of probabilistic occupancy maps, questioned performance in case of noisy observations (the authors replied with additional results in those conditions), and doing ablation experiments. Reviewer YUZ9 had some questions, answered in the rebuttal. Reviewer Jgi5 had several questions on the complexity of the model, the fairness of some comparisons and the reproducibility of the method - points addressed in the rebuttal.

The reviewers agree on the score (6) and on the fact that the paper should be accepted, and the AC concurs.

Sincerely,
Area Chair

confidence: Certain
Decision: Accept