Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The submission proposes a new multi-agent communication algorithm to simultaneously maximize what information is shared while trying to reduce how much data is transferred between agents. It calls this algorithm DiscoGraph and showcases its utility for a 3D object detection task built on top of CARLA and SUMO. 

There are two contributions within the DiscoGraph algorithm. First, to maximize information sharing between agents, they use a teacher-student framework. The teacher is only used during training and discarded during inference. The teacher receives raw sensory LIDAR data and aggregates all the data across all the agents. It then uses this combined data to do 3D object detection. The features generated by the teacher therefore contain information seen by all the agents. DiscoGraph distills this knowledge to the individual agents by ensuring that the features generated by the student networks match the one generated by the teacher. The student network for a single agent does not get to see the raw sensory data from all other agents. Instead, it encodes the features into a lower dimensional representation and shares that information through a graph-attention network formulation. The resulting features, if the graph-attention network is trained well, encodes relevant information from the features encoded by the other agents. This is enforced using a KL loss with the teacher network’s representations. In other words, the graph-attention network must learn to share appropriate information such that the features encoded by each agent’s student networks is the same as the features encoded by the teacher that has all the raw data.

The second contribution within the DiscoGraph algorithm is the per spatial location attention generated within the graph-attention network. This allowed the graph updates to attend over specific spatial locations when sharing information.

The paper ablates these two contributions and shows that both are in fact useful for the main task of 3D object detection. 

main_review: 
I actually like the method, the experimental setup, the ablations, and the results are compelling. However, there are two key problems that have prevented me from giving this submission a higher score. First, the paper could be written more clearly. Second, the paper doesn’t properly explain why it develops a new benchmark/task and doesn’t use existing ones proposed by the baselines [17, 18, 31].

**Writing: The abstract and introduction dives into details about the model before even explaining why we need such a model. 3D multi-agent object detection isn’t as established a problem space as object recognition; there needs to be more text explaining the motivation, ecological validity and utility of such a task. This is especially important since the submission introduces a new benchmark. What design options were considered when creating the dataset? For example, what does it mean to perform at XX average precision? What are the realistic communication bandwidth limitations that we would expect agents to have in these ecologically valid tasks? The answers to these decisions would have convinced me that the task / problem space is worth exploring.

Some of the arguments and sentences could be restructured to get the main message across with less reading effort. For example, in line 26: “LiDAR-based perception system can hardly perceive the target in the occluded or long-range areas. Intuitively, with an appropriate collaboration strategy, multi-agent perception could fundamentally upgrade the perception ability over single-agent perception.” This sentence is trying to say that collectively the agents perceive more than any one given agent. And if they can pool their perception together, they would be able to make better decisions. There are quite a few sentences like this that could be simplified.

The submission uses collaboration and communication interchangeably. I suggest differentiating between them and making the writing a bit more consistent. 

**New benchmark: First, why is a new task required? Why not use the V2V-Sim [31] or use AirSim-MAP or MobileNet 40 [17, 18]? Since this method directly compares against these other existing methods on similar task setup, why not evaluate on the benchmarks that have already been introduced?

The second problem is the lack of rigorous comparison to previous work. Why can’t we use the previous methods [17, 18, 31] and combine it with the spatial matrix attention contribution of this submission? I might have missed it but I didn’t see such a baseline. Similarly, why not use the knowledge distillation component with the previous methods? I know that table 3 removed both the knowledge distillation component and the maxtrix weight component from the current model. But those components can also be added to the previous methods. If it is not trivially to incorporate these contributions to existing work, perhaps add a justification for that in the text?

limitations_and_societal_impact: The submission could do a better job articulating the societal impacts of this work. It is clear to me that being able to communicate effectively about what each agent sees is a valuable improvement to multi-agent performance. But it’s not clear to me exactly what kinds of tasks (other than multi-agent 3D object detection) would benefit from such effective communication. 

It’s also hard to understand the limitations of the method. The dataset appears to be limited to a particular town. There needs to be a longer discussion around the dataset -- what kinds of scenarios does it test for? Are there common types of errors the agents make? How consistent are the agents when they share an observation space? Are there situations that are not evaluated by the dataset?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a distilled collaboration graph (DiscoGraph) to promote better performance-bandwidth trade-off for multi-agent perception. The contribution lies in two folds:  (1) the paper raised a collaborative teacher-student framework to train DiscoGraph, (2)  Matrix-valued edge weight is introduced to DiscoGraph to allow each agent to adaptively highlight the informative regions.

main_review: Pros:
1. In general, the paper is well organized and easy to follow.
2. The proposed intermediate-collaboration method sounds somewhat interesting and makes sense.
3. The figures and tables in this paper are well designed. All figures are not only informative but also beautiful.
4. Sufficient experiments on several datasets are conducted to verify the effectiveness of the proposed method. 

Cons: 
The related works are not well discussed in this paper. As distillation has become a hot area in recent years, there are extensive new methods investigating its underlying power. However, the authors utilized the distillation method without analyzing the relevance of previous methods.  

limitations_and_societal_impact: Add discussion of differences between their proposed method and the SOTA distillation method.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a learning algorithm for multi-agent perception tasks using ideas from knowledge distillation ideas. They model the relationship between each agent as a collaboration graph to be learned and use richer data in the teacher model to distill its knowledge to the student model. Through extensive analysis, they have shown that the approach can outperform other baseline methods in the multi-agent and single-agent perception tasks of object detection. They also combined two datasets, in order to introduce their multi-agent dataset for object detection.
main_review: In this paper, the authors proposed a new learning algorithm for multi-agent perception tasks, which is interesting in its own right. The proposed task is similar to the combination of ideas from privileged information with knowledge distillation for this specific application. The teacher has access to more information while the student model process single-view data and aggregate them using a collaboration graph. The usage of the knowledge distillation in this task is nontrivial and novel, which seems to be useful in practice. The organization of the paper is clear and the related baselines are discussed and compared with. I have some concerns regarding this paper:

 - How the structure of the collaboration graph is decided? is it a full graph or there might be some links missing? Is the graph bidirectional or directed? If it is bidirectional, are the weights for both directions shared or separate? I think the description of this graph is missing from the text.

- In Figure 7.d the weights of the link between agent 2 and 1 is presented, when there is no teacher. This is confusing since even without the teacher, these parameters should be learned using the loss on different views. Then, how is that this matrix is converging to a single value? 

- Since this algorithm has some overhead computation during the training, it is beneficial to compare training times with and without KD as well.
limitations_and_societal_impact: It does not apply to this paper, nor do they discussed this in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper tackles the problem of 3D detection from LiDAR in the multi-agent setting. It has two main contributions. The first contribution is the new method, which uses knowledge distillation from a teacher network using early (sensor) fusion to individual student networks that use intermediate (feature) fusion. Student networks fuse features from other student networks using a learnable spatial attention while being trained using both a standard detection loss as well as a regularization to be close to the teacher networks’ intermediate features. The method outperforms baselines in both AP and communication bandwidth
The second contribution is the release of the large scale synthetic dataset constructed with (SUMO and CARLA) used in the experiments. 

main_review: The paper presents two nice contributions, 1) a novel method and 2) a benchmark dataset for multi-agent perception. However, I think some more in depth analysis is needed.

Regarding the dataset, can the authors comment on why the AP and IOU values seem to be quite lower than other comparable datasets? For example, V2V 93.1/89.9 AP at 0.5/0.7 IOU on the V2V dataset, but only achieves 57.0/49.1 on this dataset. The lower bound model (single agent perception) achieves only 45.8/40.6 which to my understanding is extremely low (e.g. compared to baselines for real world single agent settings, e.g. KITTI). This makes me more skeptical about where the gains DiscoGraph are actually coming from, and how the well baselines were tuned, etc. I believe such analysis is crucial for a paper which contributes a new dataset (which has the potential to be a widely used benchmark in the future!)

Regarding the method, one of the major sellings points is the decrease in communication bandwidth. However it seems like vanilla DiscoNet is not much better than the baselines, and it's largely the feature compression using 1x1 convolution that brings the gain. This makes me wonder how other baselines would perform using the same trick in feature compression. For example, it would be interesting to see if there is some aspect specific to the novelties in the Disco method that allows for maintaining AP with such aggressive compression, or if other baselines benefit from this easy trick as well*. I am also confused as to why the vanilla version of Disco actually has higher memory bandwidth than the ‘upper bound’ baseline. Finally, since it seems that DISCO maintains its AP with compression, I’d be interested in a simple baseline using some lidar compression (e.g. Draco as in V2V) for the ‘upper bound’ baseline.

In summary, I’d be more confident in the AP gains if the authors could shed light on why the numbers are so low across the board and share more details regarding tuning of baselines. I’d be more confident in the communication bandwidth gains if the feature compression trick was similarly applied to baselines.

Minor notes
Figure 2 caption
Since using Agent 1 as example, update notation F_i -> F_1 
Give a sentence on where collaboration graph comes from
Line 129 - How is the original feature map F_i used to make H_i? I see in Figure 2 there is the self edge F_i->i. Maybe mention in text to be clear? 
Section 2.3: What is the decoded feature map M? EDIT: I see it is mentioned later on in the paper, but I think it would be good to introduce it here first.
Figure 3: Why are RGB images included? Are they used at all in the method?
Figure 4: Units for x axis

*While the 1x1 convolution feature compression is simple and well known, as far as I know it has not been studied or used in the multi-agent perception setting, so I believe it is definitely a useful finding (due to how effective it is despite its simplicity as shown in this paper). However I am advocating to understand whether it is effective because of other things introduced in the Disco framework, or if it can be applied to many other scenarios. This of course affects how valuable the other architectural contributions are.

---

**EDIT:** I have updated my score, please see the response to the authors for details

limitations_and_societal_impact: Perhaps authors could discuss security and safety implications of multiagent communication in general, and whether or not the specific proposed method has any differences in this regard compared to baselines.
review_title: null

Meta-review: This paper proposes a new architecture for collaborative multi-agent perception problem. The core idea is to train a (privileged) teacher which receives a holistic-view point cloud from all agents and distill the teacher's feature map to individual agents to encourage each agent to incorporate the other agents' information without direct access to their information. The paper also introduces a large-scale multi-agent 3D object detection dataset and show that the proposed method outperforms the baselines with much less communication. 

All of the reviewers agreed that the proposed architecture is reasonable and novel, and the overall results look good as well. In addition, the new dataset introduced by the paper would be valuable to the multi-agent perception community. In the meantime, there was a concern that the main compression benefit comes from 1x1 conv rather than the main idea of the paper. But, the authors promised to revise the paper so that the contribution of each idea becomes clear, and the reviewers acknowledged that 1x1 conv is also a part of the proposed architecture. Another concern was that it was not crystal clear why the proposed method is not evaluated on 2D datasets, where the baselines were evaluated. The authors clarified that the proposed architecture is specifically designed for 3D perception, though some of the ideas could be generally applicable. I encourage the authors to clarify this in the camera-ready version. 

Assuming that the authors will reflect these comments, I'd recommend to accept the paper.
Decision: Accept (Poster)