Learning Distilled Collaboration Graph for Multi-Agent Perception
Authors: Yiming Li
Shunli Ren
Pengxiang Wu
Siheng Chen
Chen Feng
Wenjun Zhang
Venue: NeurIPS 2021 Poster (2021)
Keywords: knowledge distillation
matrix-valued weighted graph
multi-agent perception
3d object detection

Abstract: To promote better performance-bandwidth trade-off for multi-agent perception, we propose a novel distilled collaboration graph (DiscoGraph) to model trainable, pose-aware, and adaptive collaboration among agents. Our key novelties lie in two aspects. First, we propose a teacher-student framework to train DiscoGraph via knowledge distillation. The teacher model employs an early collaboration with holistic-view inputs; the student model is based on intermediate collaboration with single-view inputs. Our framework trains DiscoGraph by constraining post-collaboration feature maps in the student model to match the correspondences in the teacher model. Second, we propose a matrix-valued edge weight in DiscoGraph. In such a matrix, each element reflects the inter-agent attention at a specific spatial region, allowing an agent to adaptively highlight the informative regions. During inference, we only need to use the student model named as the distilled collaboration network (DiscoNet). Attributed to the teacher-student framework, multiple agents with the shared DiscoNet could collaboratively approach the performance of a hypothetical teacher model with a holistic view. Our approach is validated on V2X-Sim 1.0, a large-scale multi-agent perception dataset that we synthesized using CARLA and SUMO co-simulation. Our quantitative and qualitative experiments in multi-agent 3D object detection show that DiscoNet could not only achieve a better performance-bandwidth trade-off than the state-of-the-art collaborative perception methods, but also bring more straightforward design rationale. Our code is available on https://github.com/ai4ce/DiscoNet.

Learning Distilled Collaboration Graph for Multi-Agent Perception
Yiming Li New York University y yimingli@nyu. edu
Shunli Ren Shanghai Jiao Tong University
Pengxiang Wu Rutgers University pxiangwue gmail com
renshunliesjtu.co edu.ccedd.com
Siheng Chen
Chen Feng" New York University cfengEnyu. edu
Wenjun Zhang
Shanghai Jiao Tong University sihengces jtu. edu.cn
Shanghai Jiao Tong University
juangwenjunés jtu. edu. en
Abstract
To promote better performance-bandvivn trade-off for multi-agent perception, we propose a novel distilled collaboration graph (DiscoGraph) to model trainable, pose-aware, and adaptive collaboration among agents. Our key novelties lie two aspects. First, we propose teacher-student framework to train DiscoGraph via knowledge distillation. The teacher model employs an early collaboration with holistic-vier view inputs; the student model is based on intermediate collaboration with single-view inputs. Our framework trains DiscoGraph by constraining post- collaboration feature maps in the student model to match the correspondences in the teacher model. Second, we propose matrix-valued edge weight DiscoGraph. In such matrix, each element reflects the inter-agent attention at specific spatial region, allowing an agent to adaptively highlight the informative regions. During in- ference, only need to the student model named the distilled collaboration network (DiscoNet). Attributed to the teacher-student framework, multiple agents with the shared DiscoNet could collaboratively approach the performance of hypothetical teacher model with a holistic view. Our approach is validated on V2X- Sim 1.0, large-scale multi-agent perception dataset that synthesized using CARLA and SUMO co-simulation. Our quantitative and qualitative experiments in multi-agen 3D object detection show that DiscoNet could not only achieve a better performance-bandvidt trade-off than the state-of-the-art collaborative perception methods, but also bring more straightforward design rationale. Our code is available on https:://// htttp::///thhbbbbbett thb..com a i 4cee Di coNet.
Introduction
Perception, which involves organizing, identifying and interpreting sensory information, a crucial ability for intelligent agents to understand the environment. Single-agent perception [4] has been studied extensively in recent years, e.g., 2D/3D object detection [18, 27], tracking [23, 22] and segmentation [25, 16], etc. Despite its great progress, single-agent perception suffers from number of shortcomings stemmed from individual perspective. For example, in autonomous driving [8], the LiDAR-based perception system can hardly perceive the target in the occluded or long-range areas. Intuitively, with an appropriate collaboration strategy, multi-agent perception could fundamentally upgrade the perception ability over single-agent perception.
To design collaboration strategy, current approaches mainly include www-measurement-be early collaboration, output-based late collaboration and feature-based intermediate collaboration.
*Corespondiigg authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

data
Raw data
data
Raw data
Enooder
Enooder
Collaboration
Features
Featuee
Collaboration
Collaboration
Collaboration
+ Encoder
|
Encoder
Guidancee
Festures
Features
1 Results
Decoder
| Results
Dacoder
Decoder
Only appeer durin training
Results
Discarded infingineereeee
Early
(b) Intermediate
(c) Ours
Figure 1: Scheme comparison. (a) Early collaboration requires expensive bandwidth for raw data transmission. (b) Intermediate collaboration needs an appropriate collaboration strategy. (c) The proposed method incorporates both early and intermediate collaboration into a knowledge distillation framework, enabling the knowledge of early collaboration to guide the training of an intermediate collaboration strategy, leading better trade-off between performance and bandwidth.
Early collaboration [3] aggregates the raw measurements from all the agents, promoting holistic perspective; see Fig. (a). It can fundamentally solve the occlusion and long-range issues occurring in the single-agent perception; however, it requires lot of communication bandwidth. Contrarily, late collaboration aggregates each agent's perception outputs. Although it is bandwidth-effiiient each individual perception output could be noisy and incomplete, causing unsatisfying fusion results. trade-off, intermediate collaboration [19, 34, 20] has
To deal with the performance-bandvee been proposed aggregate intermediate features across agents; Fig. 1 (b). Since can squeeze representative information to compact features, this approach can potentially both achieve communication bandwidth efficiency and upgrade perception ability; however, bad design of collaboration strategy might cause information loss during feature abstraction and fusion, leading to limited improvement of the perception ability.
To achieve an effective design of intermediate collaboration, we propose a distilled collaboration graph (DiscoGraph) to model the collaboration among agents. In DiscoGraph, cach node is an agent with real-time pose information and each edge reflects the pair-wise collaboration between two agents. The proposed DiscoGraph is trainable, pose-aware, and adaptive to real-time measurements, reflecting dynamic collaboration among agents. It is novel from two aspects, First, from the training aspect, we propose seacher-studen framework to train DiscoGraph through knowledge distillation [1, 10, 26]; see Fig. (c). Here the teacher model is based on early collaboration with holistic-view inputs and the student model is based intermediate collaboration with single-view inputs. The wwwllege-diisillatioon
framework enhances the training of DiscoGraph by constraining the post- collaboration feature maps in the student model to match the correspondences in the teacher model. With the guidance of both output-level supervision from perception and feature-level supervision from knowledge distillation, the distilled collaboration graph promotes better feature abstraction and aggregation, improving the performance- bandwidth trade-off. Second, from the modeling aspect, we propose a matrix-valued edge weight in DiscoGraph reflect the collaboration strength with high spatial resolution. In the matrix, each element represents the inter-agent attention at a specific spatial region. This design allows the agents adaptively highlight the informative regions and strategically select appropriate partners to request supplementary information.
During inference, we only need component, we call the student model as the distilled collaboration network (DiscoNet). Multiple agents with the shared DiscoNet could collaboratively approach the performance of hypothetical teacher model with the holistic view.
the student model. Since leverages DiscoGraph as the key
To validate the proposed method, we build V2X-Sim 1.0, new large-scale multi-agent 3D object detection dataset in autonomous driving scenarios based on CARLA and SUMO co-simulation platform [6]. Comprehensive experiments conducted in 3D object detection [37, 28, 29, 17] have shown that the proposed DiscoNet achieves better communication latency than the state-of the-ttherttt intermediate collaboration methods.
trade-off and lower

---

Aggnesatim 6. Mo
Teacher
Aiu
Crop
Krowledgee Distilationn
Graph I. Leaming
Classification
BEV
Collaboration Camph (u)
Regresssion
D2
Clessiiccation
BEV
6 Clurab
7
$
Regression
Wa--n
(Shored
Xy
Bar
Wis
Classilicatio
Regrassion
(1))) Mgn
Student
Figure 2: Overall teacher-stur stuuent training framework. In the student model, for Agent (in red), its input single-view point cloud can be converted to a BEV map, and then be consumed by a shared encoder @a to obtain the feature map FÝ. Based on the collaboration graph 9m. Agent aggregates the neural messages from other agents and obtains the updated feature map H. The shared header following the shared decoder outputs the 3D detection results. In the teacher model, we aggregate the points collected from all agents to obtain a holistic-view point cloud. We crop the region and align the pose to ensure the BEV maps in both teacher and student models are synchronized. We constrain all the post-collaboration feature maps in the student model to match the correspondences in the teacher model through knowledge distillation, resulting in collaborative student model.
2 Multi-Agent Perception System with Distilled Collaboration Graph
This work considers collaborative perception system to perceive a scene, where multiple agents can perceive and collaborate with each other through a broadcast communication channel. We assume each agent is provided with an accurate pose and the perceived measurements are well synchronized. Then, given certain communication bandwidth, each agent through optimizing a collaboration strategy. To design such a strategy, we consider teacher-studes training framework to integrate the strengths from both early and intermediate collaboration. During training, we leverage an early- corly-collaboration model (teacher) to teach an ceeemediate-collaborttie
aim to maximize the perception ability of
model (student) how to collaborate through knowledge distillation. Here the teacher model and the student model are shared by all the agents, but each agent would input raw measurements from its own view either model; see Fig. 2. During inference, we only need the student model, where multiple agents with the shared student model could collaboratively approach the performance of the teacher model with the hypothetical holistic view.
In this work, we focus on the perception task of LiDAR-based 3D object detection because the unifying 3D space naturally allows the aggregation of multiple LiDAR scans. Note that in principle the proposed method is generally-applicatl to aggregate raw data across multiple agents.
collaborative perception if there exists a unified space
2.1 Student: Intermediate collaboration via graphs
Feature encoder. The functionality is to extract informative features from raw measurements for each agent. Let X be the 3D point cloud collected by the ith agent (M agents in total), the feature of the ith agent is obtained as F <- (x)),, where (-(-) is the feature encoder shared by all the agents and the superscript s reflects the student mode. To implement the encoder, we convert a 3D point cloud to bird's hird's-eye-view (BEV) map. which is amenable to classic 2D convolutions. Specifically, we quantize the 3D points into regular voxels and represent the 3D voxel lattice 2D pseudo-image. with the height dimension corresponding to image channels. Such a 2D image is virtually BEV map, whose basic element cell that associated with binary vector along the vertical axis. Let B, E {0, x be the BEV map of the ith agent associated with X² With this map, we can apply four blocks composed of 2D convolutions, batch normalization and ReLU activation to gradually reduce the spatial dimension and increase the number of channels for the BEV, to obtain the feature map F. E 12 R RxC with K x K the spatial resolution and C' the number of channels.
3

---

Feature compression. To save the communication bandwidth, each agent could compress its feature map prior to transmission. Here we consider a convolutional autoencoder [24] to compres//éconpress the feature maps along the channel dimension. The autoencoder is trained together with the whole system, making the system work with limited collaboration information.
Collaboration graph process. The functionality is to update the feature map through data transmis- sion among the agents. The core component here is a collaboration graph' GI ET),, where V is the fixed node set and Eς is the trainable edge set. Each node in V is an agent with the real-time pose information; for instance, &, E se(3) is the ith agent's 's pose in the global coordinate system; and each edge in Eι is trainable and models the collaboration between two agents, with II an edge-weight en- coder, reflecting the trainable collaboration strength between agents. Let Men (+) be the collaboration process defined on the collaboration graph Gn. The feature maps of all the agents after collaboration M This process has three stages: neural message transmission (S1), neural message attention (S2) and neural message aggregation (S3).
are { < Man ((F:),
In the neural message transmission stage (S1), each agent transmits its BEV-based feature map to the other agents. Since the BEV-based feature map summarizes the information of each agent, we consider it as neural message. In the neural message attention stage (S2), each agent receives others' neural messages and determines the matrix-valued edge weights, which reflect the importance of the neural message from one agent to another at each individual cell. Since each agent has its unique pose, we leverage the collaboration graph to achieve feature transformation across agents. For the ith agent, the transformed BEV-based feature map from the jth agent is then F = j- (F;) E R R where the transformation r (-)) is based on two ego poses E and &, Now F J-ر and F. are supported in the same coordinate system. To determine the edge weights, we use the edge encoder II
to correlate the ego feature map and the feature map from another agent; that is, the matrix-valued
edge weight from the jth agent to the ith agent is W j-vi 11(F)> Fū) E R x where II concatenates two feature maps along the channel dimension and then uses four 1 X 1 convolutional layers to gradually reduce the number of channels from 2C to 1. Meanwhile, there is a softmax operation applied at each cell in the feature map to normalize the edge weights across multiple agents. Note that previous works [19, 20, 34] generally consider scalar-valued edge weight to reflect the overall collaboration strength between two agents; while we consider a matrix-valueee edge weight Wj j which models the collaboration strength from the jth agent to the ith agent with a K K spatial resolution. In this matrix, each element corresponds to a cell in the BEV map. indicating a specific spatial region; thus, this matrix reflects the spatial attention a cell-level resolution. In the neural message aggregation stage (S3), each agent aggregates the feature maps from all the agents based on the normalized matrix-valued edge weights. The updated feature map of the agent is H = Σÿ Wj-x @ F where @ denotes the dot product with channel-wise broadcasting.
Remark. The proposed collaboration graph is trainable because each matrix-valued edge weight is a trainable matrix to reflect the agent-to-agent attention in a cell-level spatial resolution; it is pose- aware, empowering all the agents work with the synchronized coordinate system; furthermore, it is dynamic at each timestamp as each edge weight would adapt to the real-time neural messages. Ac- cording to the proposed collaboration graph, the agents can discover the region requiring collaboration on the fly, and strategically select appropriate partners to request supplementary information.
Decoder and header. After collaboration, each agent decodes the updated BEV-based feature map. The decoded feature map is M + (²(H;) To implement the decoder (.). we progressively up-sample HE with four layers, where each layer first concatenates the previous feature map with the corresponding feature map in the encoder and then uses a 1 x convolutional operation to halve the number of channels. Finally, we use output header to generate the final detection outputs, Y + (M;). To implement the header y=*(), we use two branches of convolutional layers to classify the foreground forgrrundd-background categories and regress the bounding boxes.
2.2 Teacher: Early collaboration
During the training phase, an early-collaboration model, as the teacher, is introduced to guide the citemedditte-olllaoortoooo model, which is a student. Similar to the student model, the teacher's pipeline has the feature encoder O², feature decoder T and the output header tt. Note that all the agents share the same teacher model to guide one student model; however, each agent provides the inputs with its own pose, and its inputs to both the teacher and student models should be well aligned.
We consider connecco bidirectional graph. and the weights for both directions are distinct.

---

Feature encoder. Z'= A(&1 X1. o ,2,,... ÊM o AAx) be holistic-view 3D point cloud that aggregates all the points from all Mf agents the global coordinate system, where A(......) the aggregation operator of multiple 3D point clouds, and &, and X; are the pose and the 3D point cloud of the ith agent, respectively. To ensure the inputs to the teacher model and the student model are aligned, we transform the holistic-view point cloud X' to an agent's 's own coordinate based on the pose information. Now, for the ith agent, the input to the teacher model & о X' and the input to the student model X; are in the same coordinate system. Similarly to the feature encoder in the student model, we convert the 3D point cloud &, X' to BEV map and use 2D convolutions obtain the feature map of the ith agent in the teacher model, H E IK XxXxx Rx Here we crop the BEV map to ensure it has the same spatial range and resolution with the BEV map in the student model.
Decoder and header. Similarly the decoder and header in the student model, we adopt M <- Ve (HHH)) to obtain the decoded BEV-based feature map and Y + x6 (M:) to obtain the predicted foreground-backgrouudd categories and regressed bounding boxes.
Teacher training scheme. As common seacher-student separately. We employ the binary cross-entropy loss to supervise
frameworks, we train the teacher model
classifi- cation and the smooth L1 loss to supervise the bounding-bon regression. Overall, we minimize the loss function L = > i== Liet (Y;,
da Y)), where classification and regression losses are collectively denoted as Laet and Y; Y is the ground-truth detection in the perception region of the ith agent.
2.3 System training with knowledge distillation
Given a well-trained teacher model, we use both detection loss and knowledge distillation loss to supervise the training of the student model. We consider minimizing the following loss
M
C - (Coel(X; Yt + Sed (H), H!) + kdd (M,, M;)
The detection loss Cdet is similar to that of the teacher, including both brreground-backgroone classification loss and the bounding box regression loss, pushing the detection result of each agent to be close to its local ground-truth. The second and third terms form a knowledge distillation loss, regularizing the student model to generate similar feature maps with the teacher model. The hyperparameter Akd controls the weight of the knowledge distillation loss Lkd defined as follows
Rx
Lxd HH, H
Σ
DKI. (or ((H)),, l|o ((H)).))
where D (p(x) denotes the Kullback-Leibler (KL) divergence of distribution (xx) from distribution p(x), a()) indicates the softmax operation of the feature vector along the channel dimension, and (HT),, and (H)),, denote the feature vectors at the nth cell of the ith agent's 's feature map in the student model and teacher model, respectively. Similarly, the loss on decoded feature maps Lea (M., M)) can be introduced to enhance the regularization.
Remark. As mentioned in Section 2.1, the feature maps output by the collaboration graph process M + MMn ((F:) Intuitively, the feature map of in the student model is computed by (H;
M the teacher model (H)) i=1 would be the desired output of the collaboration graph process Mça (.). Therefore, we constrain all the post-collaboration feature maps the student model to match the correspondences in the teacher model through knowledge distillation. This constraint would further regularize the upfront trainable components: i) the distilled student encoder O², which abstracts the features from raw measurements and produces the input to MM (-), and ii) the weight encoder II in the distilled collaboration graph. Consequently, through knowledge distillation and back- propagatp,,,,,,, the distilled student encoder would learn to abstract informative features from raw data for better collaboration; and the distilled blight encoder would learn how to control the collaboration based agents' features. In word, our distilled collaboration network (DiscoNet) can comprehend feature abstraction and fusion via the proposed knowledge distillation framework.
3 Related Work
Multi-agent communication. As core components in a multi-agent system, communication strat- egy among agents has been actively studied in previous works [30, 7, 31J. For example, Comm- Net [32] adopted the averaging operation to achieve continuous communication in multi-agent system; VAIN [11] considered an attention mechanism to determine which agents would share information;
5

---

(a) Point clouds
(b) Simulation
(c) The map of Town0S
Figure 3: V2X-Sim 1.0 dataset. (a) Point clouds from multiple agents in top-down view (each color indicates an individual agent). (b) Snapshot of the rendered CARLA-SUMO co-simulation. (c) The map of Town05 which is squared-grid town with cross junctions and multiple lanes per direction.
ATOC [12] exploited an attention umit to determine what information to share with other agents; and TarMAC [5] implicitly learned a signature-based soft attention mechanism let agents actively select which agents should receive the messages. In this work, we focus on the 3D perception task and propose a trainable, dynamic collaboration graph to control the communication among agents.
Collaborative perception. Collaborative perception is an application of multi-agent communication system to perception tasks [9, 36, 14]. Who2com [20] exploited handshake communication mecha- nism to determine which two agents should communicate for image segmentation; When2com [19] introduced an asymmetric attention mechanism decide when to communicate and how create communication groups for image segmentation; V2VNet [34] proposed multiple rounds of message passing on a spatial-awar graph neural network for joint perception and prediction in autonomous driving: and [33] proposed a pose error regression module to learn to correct pose errors when the pose information from other agents is noisy.
Knowledge distillation. Knowledge distillation (KD) is a widely used technique to compress a larger teacher network smaller student network by pushing the student approach the teacher in either output or intermediate feature space [10], and it has been applied to various tasks such as semantic segmentation [21], single-agent 3D detection [35], and object re-identification [13]. In this work, we apply the KD technique to a new application scenario: multi-agen agent graph learning. The teacher is an collacoollaooation model with privileged information, and the student an intermediate- collaboration model with limited viewpoint. The distilled collaboration graph could enable effective and efficient inference for the student model without teacher's supervision.
Advantages and limitation of DiscoNet. First, previous collaborative perception only rely on the final detection supervision; while DiscoNet leverages both detection supervision and intermediate feature supervision, acquiring more explicit guidance. Second, the collaboration attention in previous methods is scalar, which cannot reflect the significance of each region; while DiscoNet uses matrix, leading to more flexible collaboration at various spatial regions. Third, previous methods uses multiple-round collaboration: When2com needs at least one more round of communication after the first handshake, and V2VNet claims three rounds to ensure reliable performance: while DiscoNet only requires one round, suffering less from latency. Admittedly. DiscoNet has limitation by assuming accurate pose for each agent, which could be improved by method like [33].
4 Experiment
4.1
V2X-Sim 1.0: A multi-agent 3D object detection dataset
Since there is no public dataset to support the research multi-agent 3D object detection in self- driving scenarios, we build such a dataset named as V2X-Sim 1.0", based on both SUMO [15], a micro-traffic simulation, and CARLA A [6], widely-used open-source simulator for autonomous driving research. SUMO is firstly used to produce numerically-realistic traffic flow and CARLA is
FV2X (vehicle-to (vehicle-t--everything denotes the collaboration between a vehicle and other entities such as vehicle (V2V) and roadside infrastructure (V21). V2X- Sim dataset is maintained on ht nttps: / / / we.githubl i V2X- sim/, and the first version of V2X- used in this work includes the LLDAR-based V2V scenario.

---

Table I: Detection comparison. Our method in bold and indicates the pose-aware version. The proposed DiscoNet is the best among intermediate collaboration. Even with 16 times compression, DiscoNet( 16) still out- performs V2 without compression.
Table 2: Detection performance by regularizing various layers via knowledge distillation (KD). ( = 1, 3, 3, 4) denotes different layers in the decoder. The performance significantly boots once the KD regularization is applied. Further regularization has slight effect the detection performance.
Method
Collahoration Approach Early Insermediate
IoU=05
Precision
loL-:07
KD Regularization
Average Precision (AP)
Uppes-houdd
X
59.7
63.3
558
6022
H
MF(3] MP(2)
[P IoU=0.5
IoU=0.7
Whenlcom" 1191
x
X
45.7
41.7
X
X
X
X
K
57.2
52.3
Whendeen [19]
x
X
45.7
418
X
$7.5
52.7
Whozeoms" [20]
Wholcom (20)
X
64.
44.3
40.4
4013
X
X
58.2
52.7
V2VNe; [34]
56.8
50.7
V
X
X
59.7
53.5
X
60.3
53.9
DiscoNiet(16)
58.5
5300
53.8
Lower bound
45.8
576
423
54.3
V
58.3
52.6
employed to get realistic sensory streams including LiDAR and RGB data, from multiple vehicles located in the same geographical area. The simulated LiDAR is rotated frequency of 20Hz and has 32 channels; the range which denotes the maximum distance to capture is 70 meters. We use Town05 with multiple lanes and cross junctions for dense traffic simulation; see Fig. 3.
V2X-Sim 1.0 follows the same storage format of nuScenes [2], a widely-used autonomous driving dataset. nuScenes collected real-world single-agent data; while simulate multi-agent scenarios. Each scene includes a 20 second traffic flow at a certain intersection of Town05, and the LiDAR streams recorded every 0.2 second, each scene consists of 100 frames. We generate 100 scenes with total of 10,000 frames, and in each scene, we randomly select 2-5 vehicles as the collaboration agents. We use 8,000/900/1, 100 frames for ceaiiinn/vvlidationn/eeeee samples, and there are 23,500 samples in the training set and 3, 100 samples in the test set.
Each frame has multiple
4.2 Quantitative evaluation
Implementation and evaluation. We crop the points located in the region of [-32, 32] x [-32, 32) x [-3, 2] meters defined in the ego-vehicle XY Z coordinate. We set the width/length of each voxel as 0.25 meter, and the height as 0.4 meter, therefore the BEV map input to the student/teacher encoder has a dimension of 256 x 256 x 13. The intermediate feature output by the encoder has a dimension of 32 x 32 x 256, and we use an autoencoder (AE) for compression to save bandwidth: the sent message is the embedding output by AE. After the agents receive the embedding, they will decode to the original resolution 32 32 256 for intermediate collaboration. The hyperparameter Ata is set as 10 We train all the models using NVIDIA GeForce RTX 3090 GPU. We employ the generic BEV detection evaluation metric: Average Precision (AP) at Intersection-oveee -oee---ovvo-nonnnn (IoU) threshold of 0.5 and 0.7. We target the detection of the car category and report the results on the test set.
Baselines and existing methods. We consider the single-agent perception model; called Lower- bound, which consumes a single-view point cloud. The teacher model based on early collaboration with holistic view is considered as the Upper-bounde Both lower-bound and upper- bound can involve late collaboration, which aggregates all the boxes and applies a global non-maximum suppression (NMS). We also consider three peermediate-collaborato methods, Who2com [20], When2com [19] and V2VNet [34]. Since the original Who2com and When2com do not consider pose information, we consider both pose-aware and pose-agnostic versions to achieve fair comparisons. All the methods use the same detection architecture and conduct collaboration at the same intermediate feature layer.
Results and comparisons. Table I shows the comparisons in terms of AP (@IoU= = 0.5/0.7). We see that i) early collaboration (first row) achieves the best detection performance, and there is an obvious improvement over no collaboration (last row), i.e., AP@0.5 and AP@0.7 are increased by 38.2% and 42.3% respectively, reflecting the significance of collaboration; ii) late collaboration improves the lower-bound (second last row), but hurts the upper- bound (second row), reflecting the unreliability of late collaboration. This is because the final, global NMS can remove lot of noisy boxes for the lower-bound, but would also remove many useful boxes for the upper-bound; and iii) among the intermediate collaboration- Comparing to the pose-aware When2com, DiscoNet improves by 31.9% in A.@0.5 and 29.3% in
methods, the proposed DiscoNet achieves the best performance.
7

---

Table 3: Comparison of various intermediate collaboration strategies. KD indicates knowledge distillation. With KD and matrix-valued edge weights, our DiscoGraph outperforms the others.
Scalar Weight Average
Matrix Weight Cat
Collahoration Strategy
No
Sum
Weighted Average
Max
DiscoGraph
w/o KD (AP@ 0.5/0.7)
45.8/42.3
55.7/50.9
55.7/50.4
1/51/
56.7/51
55.0/50.2
57.2/52.3
w/ KD TAU 0.570.7)
46.5/42.9
$4,4/46.3
56.4/51.1
56.7/50.9
56.7/5
57.5/52.6
60.353.9
Gain
0.7/0.6
-1.3/-4.6
0.7.0.7
0.6/-0.5
0.0N0.4
2.5/2.4
3,1/1.6
Gain (%)
1.5/1.4
-2.3/9.0
1.3114
1.1/10
0.0N0.7
3.3/4.6
5.4/3.1
70
65
Upperbound: Early collaboration
Upperbound: Early collaboration
DiscoNet(64)
DiscoNet
DiscoNet(32)
55
DiscoNet(16)
DiscoNet
DiscoNet(16)
V2VNet
AP AP@loU=0..
AP@IoU=01
DiscoNet(322
$6
192 times less!
50
(6s4
V2VNet
192 times less!
90
45
Lowerbound: No collaboration
Lowerbound: No collaboration
WhenZcom
When2com
Who2com
40
Who2com
35
Communication volume in log scale
Communication volume in log scale
(a) Scatterplot in AP@IoU 0.5
(b) Scatterplot in A.@1oU
Figure 4: Eiffrmmance-bandwidhh trade-off. DiscoNet(64) achieves 192 times less communication volume and still outperforms V2VNet [34].
AP@0.7. Comparing to V2VNet, DiscoNet improves by 6.2 % in AP@0.5 and 6.3 % in AP@0.7. Even with 16-times compression by the autoencoder, DiscoNet(16 (16) still outperforms V2VNet.
Performance-bur aerformance-bandiichh bandwidth trade-off analysis. Fig. 4 thoroughly compares the proposed DiscoNet with the baseline methods in terms of the trade-off between detection performance and communication bandwidth. In Plots (a) and (b), the lower-bound with no collaboration is shown as dashed line since the communication volume is zero, and the other method shown as one dot in the plot. We see that i) DiscoNet with feature-may compression by an autoencoder achieves far-more superior trade-ofi than the other methods; ii) DiscoNet(64), which DiscoNet with 64-times feature compression by autoencoder, achieves 192 times less communication volume and still outperforms V2VNet for both AP@0.5 and 0.7; and iii) feature compression does not significantly hurt the detection performance.
Comparison of training times with and without KD. Incorporating KD will increase the training time little bit, from 1200s to -5500s per epoch (each epoch consists of 2,000 iterations). However, during inference, DiscoNet does not need the teacher model anymore and can work alone without extra computations. Note that our key advantage push the computation burden into the training stage in pursuit of efficient and effective inference which is crucial in the real-world deployment.
4.3 Qualitative evaluation
Visualization of edge weight. To understand the working mechanism of the proposed collaboration graph, visualize the detection results and the corresponding edge weights; see Fig. and Fig. Note that the proposed edge weight is a matrix, reflecting the collaboration attention in a cell-level resolution, which is shown as a heat map. Fig. 5 (a), (b) and (c) show three exemplar detection results of Agent based on the lower-bound, upper-bound and the proposed DiscoNet, respectively. We see that with the collaboration, DiscoNet is able detect boxes in those occluded and long-range regions. To understand why. Fig. 5 (d) and (e) provide the corresponding ego edge weight and the edge weight from agent to agent 1, respectively. We clearly see that with the proposed collaboration graph, Agent is able to receive complementary information from the other agents. Take the first row Fig. 5 as an example, we see that in Plot (d), the bottom-left spatial region has much darker color, indicating that Agent has less confidence about this region; while in Plot (c), the bottom-left spatial

---

00
Na mesemmeent
Appunt
Spare mosmmnt
(a) Lower-bound
(b) Upper-bound
(c) DiscoNet
(d) W1-11
(e) W2-1
Figure 5: Detection and matrix-valuee edge weight for Agent Green and red boxes denote ground- truth (GT) and predictions, respectively. (a-c) shows the outputs of lower-bound upper- bound and DiscoNet compared to GT. (d) Ego matrix-valued edge weight for Agent 1. Attention to the region with sparse/no measurement is suppressed. (e) Matrix-valued edge weight from Agent 2 to Agent 1. Attention to the region with complementary information from Agent 2 is enhanced.
27
(a) When2com
(b) V2VNet
(c) DiscoNet
(d) W1-1
(c) W
W Wa-11
Figure 6: DiscoNet qualitatively outperforms the state-of-the art methods. Green and red boxes denote ground-truth and detection, respectively. (a) Output of when2com. (b) Output of V2VNet. (c) Output DiscoNet. (d)-(f) Matrix-valued edge weights. (Ego agent: neighbour agents: 2 and 3.)
region has a much brighter color, indicating that Agent has much stronger demands request information from Agent 2.
Comparison with When2com and V2VNet. Fig. 6 shows two examples compare the proposed DiscoNet with When2com, V2VNet. We see that DiscoNet is able to detect more objects. The reason that both V2VNet and when2com employ scalar to denote the agent-to-agent attention, which cannot distinguish which region is more informative; while DiscoNet can adaptively find beneficial region cell-level resolution; see the visualization of -aauue edge weights in Fig. (d)-
4.4 Ablation study
Effect of collaboration strategy. Table compares the proposed method with five baselines using different intermediate collaboration strategies. Sum simply sums all the intermediate feature maps. Average calculates the mean of all the feature maps. Max selects the maximum value of all the feature maps at each cell to produce the final feature. Cat concatenates the mean of others' features with its own feature, and then uses a convolution layer to halve the channel number. Weighted Average appends one convolutional layer to our edge encoder II, to generate scalar value the agent-wiss wise edge weight. We use matrix-valued edge weight to model the collaboration among agents. Without knowledge distillation, Max has achieved the second best performance probably because it ignores

---

89
Agus
8
18
88
B8
2
B
BR
A8:8
(a) Upper-bound
(b) DiscoNet
(c) W1-11
(d) W2-1
Figure 7: Detection and matrix-valued edge weight for Agent 1. Green/red boxes denote GT/predictions. (a)(b) show the outputs of upper-bound and DiscoNet. (c) Ego edge weight for Agent 1. (d) Edge weight from Agent 2 to Agent 1. The spatial regions containing complementary information for Agent are highlighted by green circles in (d) and red circles in (a).
the noisy information with lower value. We see that with the proposed -aauue edge weight and knowledge distillation, DiscoGraph significantly outperforms all the other collaboration choices.
Effeet of knowledge distillation. In Table we investigate the versions with and without knowledge distillation, We see that i) for our method, the knowledge distillation can guide the learning of collaboration graph, and the agents can work collaboratively to approach the teacher's performance; ii) for max without learnable module during collaboration, knowledge distillation has no impact; and iii) for cal and averuge, their performances are improved a little bit, as knowledge distillation can influence the feature abstraction process. Table 2 further shows the detection performances when regularization at various network layers. We see that once we apply knowledge distillation to regularize the feature map, the proposed method starts to achieve improvement; and ii) applying regularization on four layers has the best performance.
we apply knowledge-distillan
Conclusion
propose a novel internediate-collabrriii method, called distilled collaboration network (Dis- coNet), for multi-agent perception. Its core component is a distilled collaboration graph (Disco- Graph), which novel in both the training paradigm and the edge weight setting. DiscoGraph is also pose-aware and adaptive to perception measurements, allowing multiple agents with the shared DiscoNet to collaboratively approach the performance of the teacher model. To validate, we build V2X-Sim 1.0, large-scale multi-agent 3D object detection dataset based on CARLA and SUMO. Comprehensive quantitative and qualitative experiments show that DiscoNet achieves appealing berfowance-bandwiithh trade-off with more straightforward design rationale.
Acknowledgment. This research partially supported by the NSF CPS program under CMMI- 1932187, 17,, the National Natural Science Foundation of China under Grant 6217010074, and the Science and Technology Commission of Shanghai Municipal under Grant 2151 100900. The authors gratefully acknowledge the useful comments and suggestions from anonymous reviewers.
10

---

References
[1] Rohan Anil, Gabriel Pereyra, Alexandre Passos, Robert Ormandi, George E Dahl, and Geoffre Hinton. Large scale distributed neural network training through online distillation. In International Conference on Learning Representations, 2018.
[2] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 11621-1 1631, 2020. 7
[3] Qi Chen, Sihai Tang, Qing Yang, and Song Fu. Cooper: Cooperative perception for connected autonomous vehicles based on point clouds. In IEEE International Conference on Distributed Compuring Systems (ICDCS), pages 514-524, 2019.
[4] Siheng Chen, Baoan Liu, Chen Feng, Carlos Vallespi-Gonzalee and Carl Wellington. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEEE Signal Processing Magazine, 38:68-86, 2021.
[5] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pincau. Tarmac: Targeted multi-agent communication. In International Conference Machine Learning. pages 1538 1546, 2019. 6
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. Proceedings of the 1st Anmual Conference on Robot Learning, pages 1-16, 2017. 2,
[7] Jakob N. Foerster, Yannis M. Assael, N. D. Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In NeurlPS, 201n/ 5
[8] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE Conference Computer Vision and Pattern Recognition, pages 3354-3361, 2012.
Ramyad Hadidi, Jiashen Cao, Matthew Woodward, Michael S Ryoo, and Hyesoon Kim. Distributed perception by collaborative robots. IEEE Robotics and Automation Letters, 3(4):370933766, 2018.
[10] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint (50.::1503.0331,, 2015. 2,
[11] Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. In NeurIPS, 2017.
[12] Jicchuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In NewrIPS, 2018.
[13] Xin Jin, Cuiling Lan, Wenjun Zeng, and Zhibo Chen. Uncertainty-awan multi-shot knowledge distillation for image-based based object re- re-identification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11165 11112.111111111111 2020.
[14] Saifullah Khan, Franz Andert, Nicolai Wojke, Julian Schindler, Alejandro Correa, and Anton Wijbenga. Towards collaborative perception automated vehicles in heterogencous traffic. In International Forum on Advanced Microsystems for Automotive Applications, pages 31 42, 2018.
[15] Daniel Krajzewiez, Jakoh Erdmann, Michael Behrisch, and I Laura Bieker. Recent development and applications of sumo-simulation of urban mobility. International journal on advances in systems and meaxurements, 5(3&4). 2012. 6
[16] Loic Landricu and Martin Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings the IEEE Conference Computer Vision and Pattern Recognition, pages 4558 4567, 2018.
[17] Yiming Li, Congcong Wen, Felix Jucfei-Xu, and Chen Feng. Fooling lidar perception via adversarial trajectory perturbation. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 7898-7907, October 2021. 2
[18] Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietiklinen. Deep learning for generic object detection: A survey. International journal of computer vision, 128(2):261-2
2020. I

---

[19] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4106-4115, 2020. 2, 2,4, 6.7
[20] Yen-Cheng Liu, Junjiao Tian, Chih-Yao Ma, Nathan Glaser, Chia-Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA). pages 6876-6883, 2020. 2, 2,4, 6.
[21] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2604-2613, 2019. 6
[22] Wenjie Luo, Bin Yang. and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3569-3577, 2018.
[23] Seyed Mojtaba Marvasti-Zadee Li Cheng. Hossein Ghanei- Yakhdan, and Shohrch Kasaei. Deep learning for visual tracking: A comprehensive survey, IEEE Transactions on Intelligent Transportation Systems. 2021. 1
[24] Jonathan Masci, Ueli Meier, Dan Ciregan, and Jürgen Schmidhuber. Stacked convolutional auto-encoders for hierarchical feature extraction. In International conference on artificial neural networks, pages 52-59,
2011. 4
[25] Shervin Minace, Yuri Baykov, E. Porikli, A. Plaza, N. Kehtarnavaz, and Demetri Terzopoulos. Image seg- mentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence, PP, 2021.
[26] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. CoRR, abs/1412.6550, 2015. 2
[27] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7770-779.00 2019. 1
[28] Shaoshuai Shi, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE transactions on pattern analysis and machine intelligence, 2020. 2
[29] Weijing Shi and Raj Rajkumar. Point-gnn gnn: Graph neural network for 3d object detection in point cloud. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1711-1719
2020. 2
[30] Yoav Shoham and Kevin Scyton-Brown Multiagent systems: Algorithmic, ghme-the and logical foundations. Cambridge University Press, 2008. 5
[31] Amanpreet Singh. Tushar Jain, and Sainbayar Sukhbaatar. Learning when communicate scale in multiagent cooperative and competitive tasks. In International Conference on Learning Representations, 2018. 5
[32] Sainbayar Sukhbaatar, Arthur D. Szlam, and Rob Fergus. Learning multiagent communication with backpropagation. In NeurlPS, 2016. 5
[33] Nicholas Vadivelu, Mengye Ren, James Tu, Jingkang Wang, and Raquel Urtasun. Learning to communicate and correct pose crrors. In 4th Conference on Robot Learning (CoRL), 2020. 6
[34] Tsun-Hsuan Wang. Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehice communication for joint perception and prediction. In Proceedings of the European Conference on Computer Vision (ECCV). pages 605- 2020. 2, 6,
[35] Yue Wang, Alireza Fathi, Jiajun Wu, Thomas Funkhouser, and Justin Solomon. Multi-frame to single- frame: Knowledge distillation for 3d object detection. In Proceedings of the European Conference on Computer Vision Workshops, 2020.
[36] Yufeng Yue, Chunyang Zhao, Mingxing Wen. Zhenyu Wu, and Danwei Wang. Collaborative semantic perception and relative localization based on map matching. In IEFF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 6188-6193, 2020. 6
[37] Yin Zhou and Oncel Tuzel. Voxelnet: End-to End-to-end learning for point cloud based 3d object detection. In Pr Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4490 4499, 2018.
12


Reviews:
[{"review_id": "vY2HQ94ehvl", "reviewer": "Reviewer_AgpJ", "summary": "The submission proposes a new multi-agent communication algorithm to simultaneously maximize what information is shared while trying to reduce how much data is transferred between agents. It calls this algorithm DiscoGraph and showcases its utility for a 3D object detection task built on top of CARLA and SUMO. \n\nThere are two contributions within the DiscoGraph algorithm. First, to maximize information sharing between agents, they use a teacher-student framework. The teacher is only used during training and discarded during inference. The teacher receives raw sensory LIDAR data and aggregates all the data across all the agents. It then uses this combined data to do 3D object detection. The features generated by the teacher therefore contain information seen by all the agents. DiscoGraph distills this knowledge to the individual agents by ensuring that the features generated by the student networks match the one generated by the teacher. The student network for a single agent does not get to see the raw sensory data from all other agents. Instead, it encodes the features into a lower dimensional representation and shares that information through a graph-attention network formulation. The resulting features, if the graph-attention network is trained well, encodes relevant information from the features encoded by the other agents. This is enforced using a KL loss with the teacher network’s representations. In other words, the graph-attention network must learn to share appropriate information such that the features encoded by each agent’s student networks is the same as the features encoded by the teacher that has all the raw data.\n\nThe second contribution within the DiscoGraph algorithm is the per spatial location attention generated within the graph-attention network. This allowed the graph updates to attend over specific spatial locations when sharing information.\n\nThe paper ablates these two contributions and shows that both are in fact useful for the main task of 3D object detection. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nI actually like the method, the experimental setup, the ablations, and the results are compelling. However, there are two key problems that have prevented me from giving this submission a higher score. First, the paper could be written more clearly. Second, the paper doesn’t properly explain why it develops a new benchmark/task and doesn’t use existing ones proposed by the baselines [17, 18, 31].\n\n**Writing: The abstract and introduction dives into details about the model before even explaining why we need such a model. 3D multi-agent object detection isn’t as established a problem space as object recognition; there needs to be more text explaining the motivation, ecological validity and utility of such a task. This is especially important since the submission introduces a new benchmark. What design options were considered when creating the dataset? For example, what does it mean to perform at XX average precision? What are the realistic communication bandwidth limitations that we would expect agents to have in these ecologically valid tasks? The answers to these decisions would have convinced me that the task / problem space is worth exploring.\n\nSome of the arguments and sentences could be restructured to get the main message across with less reading effort. For example, in line 26: “LiDAR-based perception system can hardly perceive the target in the occluded or long-range areas. Intuitively, with an appropriate collaboration strategy, multi-agent perception could fundamentally upgrade the perception ability over single-agent perception.” This sentence is trying to say that collectively the agents perceive more than any one given agent. And if they can pool their perception together, they would be able to make better decisions. There are quite a few sentences like this that could be simplified.\n\nThe submission uses collaboration and communication interchangeably. I suggest differentiating between them and making the writing a bit more consistent. \n\n**New benchmark: First, why is a new task required? Why not use the V2V-Sim [31] or use AirSim-MAP or MobileNet 40 [17, 18]? Since this method directly compares against these other existing methods on similar task setup, why not evaluate on the benchmarks that have already been introduced?\n\nThe second problem is the lack of rigorous comparison to previous work. Why can’t we use the previous methods [17, 18, 31] and combine it with the spatial matrix attention contribution of this submission? I might have missed it but I didn’t see such a baseline. Similarly, why not use the knowledge distillation component with the previous methods? I know that table 3 removed both the knowledge distillation component and the maxtrix weight component from the current model. But those components can also be added to the previous methods. If it is not trivially to incorporate these contributions to existing work, perhaps add a justification for that in the text?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The submission could do a better job articulating the societal impacts of this work. It is clear to me that being able to communicate effectively about what each agent sees is a valuable improvement to multi-agent performance. But it’s not clear to me exactly what kinds of tasks (other than multi-agent 3D object detection) would benefit from such effective communication. \n\nIt’s also hard to understand the limitations of the method. The dataset appears to be limited to a particular town. There needs to be a longer discussion around the dataset -- what kinds of scenarios does it test for? Are there common types of errors the agents make? How consistent are the agents when they share an observation space? Are there situations that are not evaluated by the dataset?\n", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "gJ57fPFZfMz", "reviewer": "Reviewer_rKjq", "summary": "This paper tackles the problem of 3D detection from LiDAR in the multi-agent setting. It has two main contributions. The first contribution is the new method, which uses knowledge distillation from a teacher network using early (sensor) fusion to individual student networks that use intermediate (feature) fusion. Student networks fuse features from other student networks using a learnable spatial attention while being trained using both a standard detection loss as well as a regularization to be close to the teacher networks’ intermediate features. The method outperforms baselines in both AP and communication bandwidth\nThe second contribution is the release of the large scale synthetic dataset constructed with (SUMO and CARLA) used in the experiments. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents two nice contributions, 1) a novel method and 2) a benchmark dataset for multi-agent perception. However, I think some more in depth analysis is needed.\n\nRegarding the dataset, can the authors comment on why the AP and IOU values seem to be quite lower than other comparable datasets? For example, V2V 93.1/89.9 AP at 0.5/0.7 IOU on the V2V dataset, but only achieves 57.0/49.1 on this dataset. The lower bound model (single agent perception) achieves only 45.8/40.6 which to my understanding is extremely low (e.g. compared to baselines for real world single agent settings, e.g. KITTI). This makes me more skeptical about where the gains DiscoGraph are actually coming from, and how the well baselines were tuned, etc. I believe such analysis is crucial for a paper which contributes a new dataset (which has the potential to be a widely used benchmark in the future!)\n\nRegarding the method, one of the major sellings points is the decrease in communication bandwidth. However it seems like vanilla DiscoNet is not much better than the baselines, and it's largely the feature compression using 1x1 convolution that brings the gain. This makes me wonder how other baselines would perform using the same trick in feature compression. For example, it would be interesting to see if there is some aspect specific to the novelties in the Disco method that allows for maintaining AP with such aggressive compression, or if other baselines benefit from this easy trick as well*. I am also confused as to why the vanilla version of Disco actually has higher memory bandwidth than the ‘upper bound’ baseline. Finally, since it seems that DISCO maintains its AP with compression, I’d be interested in a simple baseline using some lidar compression (e.g. Draco as in V2V) for the ‘upper bound’ baseline.\n\nIn summary, I’d be more confident in the AP gains if the authors could shed light on why the numbers are so low across the board and share more details regarding tuning of baselines. I’d be more confident in the communication bandwidth gains if the feature compression trick was similarly applied to baselines.\n\nMinor notes\nFigure 2 caption\nSince using Agent 1 as example, update notation F_i -> F_1 \nGive a sentence on where collaboration graph comes from\nLine 129 - How is the original feature map F_i used to make H_i? I see in Figure 2 there is the self edge F_i->i. Maybe mention in text to be clear? \nSection 2.3: What is the decoded feature map M? EDIT: I see it is mentioned later on in the paper, but I think it would be good to introduce it here first.\nFigure 3: Why are RGB images included? Are they used at all in the method?\nFigure 4: Units for x axis\n\n*While the 1x1 convolution feature compression is simple and well known, as far as I know it has not been studied or used in the multi-agent perception setting, so I believe it is definitely a useful finding (due to how effective it is despite its simplicity as shown in this paper). However I am advocating to understand whether it is effective because of other things introduced in the Disco framework, or if it can be applied to many other scenarios. This of course affects how valuable the other architectural contributions are.\n\n---\n\n**EDIT:** I have updated my score, please see the response to the authors for details\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Perhaps authors could discuss security and safety implications of multiagent communication in general, and whether or not the specific proposed method has any differences in this regard compared to baselines.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "fR5jNDjXgM", "reviewer": "Reviewer_4Gjd", "summary": "This paper proposes a learning algorithm for multi-agent perception tasks using ideas from knowledge distillation ideas. They model the relationship between each agent as a collaboration graph to be learned and use richer data in the teacher model to distill its knowledge to the student model. Through extensive analysis, they have shown that the approach can outperform other baseline methods in the multi-agent and single-agent perception tasks of object detection. They also combined two datasets, in order to introduce their multi-agent dataset for object detection.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors proposed a new learning algorithm for multi-agent perception tasks, which is interesting in its own right. The proposed task is similar to the combination of ideas from privileged information with knowledge distillation for this specific application. The teacher has access to more information while the student model process single-view data and aggregate them using a collaboration graph. The usage of the knowledge distillation in this task is nontrivial and novel, which seems to be useful in practice. The organization of the paper is clear and the related baselines are discussed and compared with. I have some concerns regarding this paper:\n\n - How the structure of the collaboration graph is decided? is it a full graph or there might be some links missing? Is the graph bidirectional or directed? If it is bidirectional, are the weights for both directions shared or separate? I think the description of this graph is missing from the text.\n\n- In Figure 7.d the weights of the link between agent 2 and 1 is presented, when there is no teacher. This is confusing since even without the teacher, these parameters should be learned using the loss on different views. Then, how is that this matrix is converging to a single value? \n\n- Since this algorithm has some overhead computation during the training, it is beneficial to compare training times with and without KD as well.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It does not apply to this paper, nor do they discussed this in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "VJrrZkwST-Y", "reviewer": "Reviewer_EHoG", "summary": "This paper proposed a distilled collaboration graph (DiscoGraph) to promote better performance-bandwidth trade-off for multi-agent perception. The contribution lies in two folds:  (1) the paper raised a collaborative teacher-student framework to train DiscoGraph, (2)  Matrix-valued edge weight is introduced to DiscoGraph to allow each agent to adaptively highlight the informative regions.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. In general, the paper is well organized and easy to follow.\n2. The proposed intermediate-collaboration method sounds somewhat interesting and makes sense.\n3. The figures and tables in this paper are well designed. All figures are not only informative but also beautiful.\n4. Sufficient experiments on several datasets are conducted to verify the effectiveness of the proposed method. \n\nCons: \nThe related works are not well discussed in this paper. As distillation has become a hot area in recent years, there are extensive new methods investigating its underlying power. However, the authors utilized the distillation method without analyzing the relevance of previous methods.  \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Add discussion of differences between their proposed method and the SOTA distillation method.", "review_text": "", "extra_scores": {"ethical_concerns": "Not any", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new architecture for collaborative multi-agent perception problem. The core idea is to train a (privileged) teacher which receives a holistic-view point cloud from all agents and distill the teacher's feature map to individual agents to encourage each agent to incorporate the other agents' information without direct access to their information. The paper also introduces a large-scale multi-agent 3D object detection dataset and show that the proposed method outperforms the baselines with much less communication. 

All of the reviewers agreed that the proposed architecture is reasonable and novel, and the overall results look good as well. In addition, the new dataset introduced by the paper would be valuable to the multi-agent perception community. In the meantime, there was a concern that the main compression benefit comes from 1x1 conv rather than the main idea of the paper. But, the authors promised to revise the paper so that the contribution of each idea becomes clear, and the reviewers acknowledged that 1x1 conv is also a part of the proposed architecture. Another concern was that it was not crystal clear why the proposed method is not evaluated on 2D datasets, where the baselines were evaluated. The authors clarified that the proposed architecture is specifically designed for 3D perception, though some of the ideas could be generally applicable. I encourage the authors to clarify this in the camera-ready version. 

Assuming that the authors will reflect these comments, I'd recommend to accept the paper.
Decision: Accept (Poster)