Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps
Authors: Yue Hu
Shaoheng Fang
Zixing Lei
Yiqi Zhong
Siheng Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Collaborative perception
3D object detection

Abstract: Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication. It inevitably results in a fundamental trade-off between perception performance and communication bandwidth. To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information. It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to communicate. Based on this novel spatial confidence map, we propose Where2comm, a communication-efficient collaborative perception framework. Where2comm has two distinct advantages: i) it considers pragmatic compression and uses less communication to achieve higher perception performance by focusing on perceptually critical areas; and ii) it can handle varying communication bandwidth by dynamically adjusting spatial areas involved in communication. To evaluate Where2comm, we consider 3D object detection in both real-world and simulation scenarios with two modalities (camera/LiDAR) and two agent types (cars/drones) on four datasets: OPV2V, V2X-Sim, DAIR-V2X, and our original CoPerception-UAVs. Where2comm consistently outperforms previous methods; for example, it achieves more than $100,000 \times$ lower communication volume and still outperforms DiscoNet and V2X-ViT on OPV2V. Our code is available at~\url{https://github.com/MediaBrain-SJTU/where2comm}.

Where2comm: Communication-Effiienn Collaborative Perception via Spatial Confidence Maps
Yue Hu
Shaoheng Fang
Zixing Lei
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
(18671129361, shfang, chezacarss) edu.cn
Yiqi Zhong
Siheng Chen
University of Southern California yi yigizhon@usc.edi
Shanghai Jiao Tong University, Shanghai Al Laboratory sihengc@sjtu. edu. cn
Abstract
Multi-agent collaborative perception could significantly upgrade the perception performance by enabling agents to share complementary information with each other through communication. between perception performance and communication bandwidth. To tackle this bottleneck issue, we propose a spatial confidence map, which reflects the spatial heterogeneity of perceptual information. It empowers agents to only share spatially sparse, yet perceptually critical information, contributing to where to commu- nicate. Based this novel spatial confidence map, we propose Where2comm, a communication-effiiiie
inevitably results in a fundamental trade-off
collaborative perception framework. Where2comm has two distinct advantages: i) it considers pragmatic compression and uses less communication to achieve higher perception performance by focusing on perceptually critical areas; and ii) it can handle varying communication band- width by dynamically adjusting spatial areas involved in communication. To evaluate Where2comm,
consider 3D object detection in both real-world and simulation scenarios with two modalities (camera/LiDARR and two agent types (cars/drones) four datasets: OPV2V, V2X-Sim, DAIR-V2X. and origi- nal Co
Where2comm consistently outperforms previous meth- ods; for example, it achieves more than 100, 000 lower communication volume and still outperforms DiscoNet and VIX-ViT on OPV2V, Our code is available at htttt:://// htttps://github..com http:///dddaaaaannn-nSS/t/eeeee
Introduction
Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome number of inevitable limitations of single-agent perception, such occlusion and long- range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everythine corpinngcoommmintttiioi.ttttie robot warchouse automation system 4 5 and multi-UAVs (unmanned acrial vehicles) for search and rescue 6 To realize collaborative perception, recent works have contributed high-quality
autonomous driving 31. multi-
datasets 9
and effective collaboration methods 12 13 14
In this emerging field, the current biggest challenge is how optimize the trade-off between perception performance and communication bandwidth. Communication systems in real-world scenarios always constrained that they can hardly afford huge communication consumption in real-time, such as passing complete raw observations or large volume of features. Therefore,
*Corresponiigg author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

According to received message
Synchronized coordinate
Occluded
D
(e) Detection BEV Figure 1: Collaborative perception could contribute to safety-critical scenarios, where the white and the red car may collide due to occlusion. This collision could be avoided when the blue car can share a message about the red car's position. Such message is spatially sparse, yet perceptually critical. Considering the precious communication bandwidth, each agent needs to speak to the point!
(a) Scene in BEV
(b) Image view
(c) LiDAR view (d) Confidence in BEV
we cannot solely promote the perception performance without evaluating the expense of every bit of precious communication bandwidth. To achieve a better performance and bandwidth trade-off, previous works put forth solutions from several perspectives. For example, When2com 2 considers handshake mechanism which selects the most relevant collaborators; V2VNet | considers end- go-end-learminn bareeng-bbsed based source coding; and DiscoNet [2] uses ID convolution to compress message. However, all previous works make a plausible assumption: once two agents collaborate, they are obligated to share perceptual information of all spatial areas equally. This unnecessary assumption can hugely waste the bandwidth as a large proportion spatial areas may contain irrelevant information for perception task. Figure [
illustratts such spatial heterogeneity of perceptual information.
To fill this gap, we consider novel ppatial-confidence-awaaa communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse. yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration.
Following this strategy, we propose Where2comm, novel communication--ffiiiee multi-agent collab- orative perception framework with the guidance spatial confidence maps; see Fig. 2 Where2comm includes three key modules: i) spatial confidence generator, which produces spatial confidence map to indicate perceptually critical areas; ii) a spatial confidence-awarr communication module, which leverages the spatial confidence map decide where to communicate via novel message packing, and who to communicate via novel communication graph construction; and iii) a spatial confidence-ave aware message fusion module, which uses novel confidence-awee aware multi-hean attention to fuse all messages received from other agents, upgrading the feature map for each agent.
Where2comm has two distinct advantages. First, it promotes pragmatic compression at the feature level and uses less communication to achieve higher perception performance by focusing on perceptually critical areas. Second, it adapts to various communication bandwidths and communication rounds, while previous models only handle one predefined communication bandwidth and fixed number of communication rounds. To evaluate Where2comm. we consider the collaborative 3D object detection task on four datasets: DAIR-V2X [1I. V2X-Sim 99, OPV2V (10) and our original dataset CoPerception-Ure UAVs. Our experiments cover both real-world and simulation scenarios, two types of agents (cars and drones) and sensors (LiDAR and cameras). Results show that i) the proposed Where2comm consistently and significantly outperforms previous works in the performance- bandwidth trade-off across multiple datasets and modalities; and ii) Where2comm achieves better trade-off when the communication round increases.
Related Works
Multi-agent communication. The communication strategy in multi-agent systems has been widely studied 20 Early works 231 often use predefined protocols or heuristics to decide how agents communicate with each other. However, is difficult to generalize those methods to complex tasks. Recent works, thus, explore marning-basel methods for complex scenarios. For example,

---

Table 1: Major components comparisons of collaborative perception systems.
Method
Venue
Message packing
Commmnicaatin graph construction
Messgge fusion
WhenZcom
CVPR 2020
Full Ieature map
Handsstako-base spuss graph
Attention per-agent
V2VNet !
BCCV 2020
Full feature mup
Fully connected graph
Averuge per-agent
DissoNet
NeurTPS 2021
Full feature map
Fully connected graph
Mi.P-hused attention pec-location
VIX-VIT
20
ECCV 2022
Full feature map
Fully connected graph
Self-attention per-location
Where2oomm
NeurIPS 2022
Corfidence-aware sparse
Confidence-swarc sparse graph
Confddece-awarc multt-heod
Teature map
request map
atlention per- per-location
CommNet [4
learns continuous communication in the multi-agent system. Vain (25) adopts the attention mechanism to help agents selectively fuse the information from others. Most of these previous works consider decision-making tasks and adopt reinforcement learning due to the lack of explicit supervision. In this work, we focus on the perception task. Based on direct perception supervision, we apply supervised learning optimize the communication strategy both trade-off' perception ability and communication cost.
Collaborative perception. As recent application of multi- aaget communication systems per- ception tasks, collaborative perception is still immature. To support this area of research, there is a surge of high-quality datasets (e.g.. V2X-Sim (.. Open [IO]. Comap 27 and DAIR- V2X as well as collaboration methods aimed for better performance-banddvidn trade-off (see comparisons in Table When2com 12 proposes a handshake communication mechanism decide when to communicate and create sparse communication graph. V2VNet []] proposes multi-round message passing based on graph neural networks achieve better perception and prediction performance. DiscoNet [2] adopts knowledge distillation to take the advantage of both early and intermediate collaboration. OPV2V [TO] proposes graph-based attentive intermediate fusion to improve percep- tion performances. V2X-ViT (26 introduces a novel heterogeneous multi-agent attention module to fuse information across heterogeneous agents. In this work, we leverage the proposed spatial confidence map to promote more compact messages, more sparse communication graphs, and more comprehensive fusion, resulting in efficient and effective collaboration.
Problem Formulation
Consider N agents in the scene. Let X; and Vi be the observation and the perception supervision of the ith agent, respectively. The objective of collaborative perception is to achieve the maxi- mized perception performance of all agents function of the total communication budge B and communication round K; that is,
N
K
N
(x = arg max U.P
(K) Eg(5 (x. ..)) s.t.
S B,
where 9(.,-) -) is the perception evaluation metric, & is the perception network with trainable parameter 0, and p(k) is the message transmitted from the ith agent the jth agent at the leth communication round. Note that i) when B K O. there is no collaboration and & (0, 0) reflects the single- agent perception performance; ii) through optimizing the communication strategy and the network parameter, collaborative perception should perform well consistently at any communication bandwidth or round; and iii) we consider multi-roun communication, where each agent serves as both a supporter (offering message help others) and a requester (requesting messages from others).
In this work, we consider the perception task of 3D object detection and present three contributions: i) we make communication more efficient by designing compact messages and sparse communication graphs; ii) we boost the perception performance by implementing more comprehensive message fusion; iii) we enable the overall system to adapt to varying communication conditions by dynamically adjusting where and who to communicate.
4 Where2comm: Spatial Confidence-Aware Collaborative Perception System
This section presents Where2comm, a multi-round, multi-modality, multi-agent collaborative per- ception framework based on a spatial-confidee cettiil-coofieene-avweee communication strategy; see the overview in Fig. 2 Where2comm includes an observation encoder, a spatial confidence generator, the spatial confidence-awarr communication module, the spatial confidence-awer message fusion module and a detection decoder. Among five modules, the proposed spatial confidence generator generates the spatial confidence map. Based on this spatial confidence map, the proposed spatial confidence-aware communication generates compact messages and sparse communication graphs to save commu- nication bandwidth; and the proposed spatial confidence-awee message fusion module leverages
3

---

Requis MACH
x communicatinn rounds
Spatial confidance- communication
Receised
Chservatio
The STANA
Hessag packing
Erchange Haasages
Sport contider oo
[R]
Biitry select mustts
Haxsge
Sparial confidence grownater
Communication graph cttin
X - Encodar
Message
Decoder
fuston
Figure 2: System overview. In Where2comm, spatial confidence generator enables the awareness of spatial heterogeneous of perceptual information, spatial confidence aware communication enables efficient communication, and spatial confidence-awars message fusion boosts the performance.
informative spatial confidence priors achieve better aggregation; also see an algorithmic summary in Algorithm 1 and the optinization-orrenteed design rationale in Section 7.3 in Appendix.
4.1 Observation encoder
The observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multi- modality inputs, such as RGB images and 3D point clouds. This work adopts the feature represen- tations in bird's eye view (BEV), where all agents project their individual perceptual information the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration. For the ith agent, given its input X; the feature map is
,: 2 enne (X) H xD, where (-ne(-) )) is the encoder, the superscript 0 reflects that the feature is obtained before communication and H, W, D are its height, weight and channel. All agents share the same BEV coordinate system. For the image input, & Pene() (+) is followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as BEV map and & Pene (-) extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module.
4.2 Spatial confidence generator
The spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas. Intuitively, for object detection task, the areas that contain objects are more critical than background areas. During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth. So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with high confidence score.
To implement, we use a detection decoder structure to produce the detection confidence map. Given
the feature map at the keth communication round, (A)), the corresponding spatial confidence map is
geuerator (()) E [0, 1]
(1)
where the generator @generstor (-) follows a detection decoder. Since we consider multi-round collaboration, Where2comm iteratively updates the feature map by aggregating information from
other agents. Once [(()) is obtained, [ is triggered to reflect the perceptually critical level each spatial location. The proposed spatial confidence map answers crucial question that was ignored by previous works: for each agent, information at which spatial area worth sharing with others. By answering this, it provides solid base for efficient communication and effective message fusion.
4.3 Spatial confidence-aware communication
With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connene communication graph. Most existing collaboration perception systems 2 26 considers full feature maps the messages and fully-connected communication graphs. To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most

---

informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate).
Message packing. Message packing determines what information should be included in the to-be- sent message. The proposed message includes: i) request map that indicates at which spatial areas the agent needs to know more; and ii) spatially sparse, yet perceptually critical feature map.
The request map of the ith agent is R (8) = - E I "FxW, .negttively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent's detection accuracy.
The spatially sparse feature map are selected based on each agent's spatial confidence map and the received request maps from others. Specifically, binary selection matrix is used to represent each location selected or not, where 1 denotes selected, and 0 elsewhere. For the message sent from the ith agent to the jth agent at the kth communication round, the binary selection matrix is
(k) M 1
{ 4 Pselect select (C,t)) @ R (k- ( ) [0,1)
(C, (C(()) E (0, 10,11
= O; k 0:
(2)
(k is the request map from the jth agent received at the previous round, spetect() (-) is the selection function which targets to select the most critical areas conditioned on the input matrix, which represents the critical level at the certain spatial location. We implement sselect()) by selecting the locations where the largest elements at in the given input matrix conditioned the bandwidth limit; optionally, Gaussian filter could be applied to filter the outliers and introduce some context. In the initial communication round, each agent selects the most critical areas from its own perspective as the request maps from other agents are not available yet; in the subsequent rounds, each agent also takes the partner's 's request into account, enabling more targeted communication. Then, the selected feature map is obtained as 2 M which provides spatially sparse, yet perceptually critical information.
where @ is the element-wiss multiplication, R
F(() E RR HxXD
Overall, the message sent from the ith agent to the jth agent at the kth communication round
P!L)) (R!*) (R) 2. Note that i) R (8) provides spatial priors to request complementary
information for the ith agent's need in the next round; the feature map 2 provides supportive information for the ith agent's need in the this round. They together enable mutually beneficial collaboration; ii) since 2 is sparse, we only transmit non-zero features and corresponding indices,
leading to low communication cost; and iii) the sparsity of 2,4h), is determined by the binary selection matrix, which dynamically allocates the communication budget at various spatial areas based on their perceptual critical level, adapting to various communication conditions.
Communication graph construction. Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth. Most previous works [1 consider fully-connected communication graphs. When2com [12] proposes handshake mechanism, which uses similar global features to match partners. This is hard to interpret because two agents, which have similar global features, do not necessarily need information from each other. Different from all previous works, we provide an explicit design rationale: the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jthh agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph.
For the initial communication round, every agent in the system is not aware of other agents yet. To activate the collaboration, we construct a fully-connected communication graph. Every agent will broadcast its message the rest of the system. For the subsequent communication rounds, examine if the communication between agent i and agent j is necessary based on the maximum value of the binary selection matrix (4) j= i.e. if there is at least one patch is activated, then we regard the connection is necessary. Formally, let A (8) be the adjacency matrix of the communication graph at the ktth communication round, whose (i, 1tt element is
k = 0; k > 0:
(O.,.........- WE (0,1..., (M h.w0
E 10,
5

---

where h, w index the spatial area, reflecting message passing from the ith agent to the jth agent. Given this sparse communication graph, agents can exchange messages with selected partners.
4.4 Spatial confidence-aware message fusion
Spatial confidence-aware message fusion targets to augment the feature of cach agent by aggregating the received messages from the other agents. To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location. The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning. The intuition is that, the spatial confidence map could explicitly reflect the perceptually critical level, providing a useful prior for attention learning.
Specifically, for the ith agent, after receiving the jth agent's message p(k) j- it could unpack to retrieve
the feature map j-o and the spatial confidence map 0
R (k: We also include the ego
feature map in fusion and denote _!!² (k) to make the formulation simple and consistent, where 2 might not be sparse. To fuse the features from the jth agent the kth communication round, the cross-agent/ego attention weight for the ith agent is
(k) j-i = MHAw (A
12, 2/-1 2(1)) O C.A
E xxw
(3)
where MHAw( (+) is multi-head attention applied at each individual spatial location, which outputs the scaled dot-product attention weight. Note that i) the proposed spatial confidence maps contributes to the attention weight, as the features with higher perceptually critical level are more preferred in the feature aggregation; ii) the cross-agent attention weight models the collaboration strength with H W spatial resolution, leading to more flexible information fusion various spatial regions. Then, the feature map of the ith agent after fusing the messages in the kth communication round is
F((+D) FFN
Σ JEA
w (k) joHi Z
HXWxD
where (-N is the feed-forward network and Nr is the neighbors of the ith agent defined in the communication graph A (k), The fused feature F((++) would serve as the ith agent's feature in the (k + 1)th round. In the final round, we output F F((++) to the detection decoder to generate detections.
Sensor positional encoding. Sensor positional encoding represents the physical distance between each agent's 's sensor and its observation. It adopts a standard positional encoding function conditioned on the sensing distance and feature dimension. The features are summed up with the positional encoding of each location before inputting to the transformer.
Compared to existing fusion modules that do not use attention mechanism 1 or only agent-level attentions [2], the per-location attention mechanism adopted by the proposed fusion emphasizes the recation-specifi, feature interactions. It makes the feature fusion more targeted. Compared to the methods that also use the per-location attention-bann
fusion module [2 10 26), the proposed fusion module leverages muat attention with two extra priors, including spatial confidence map and sensing distances. Both assist attention learning to prefer high quality and critical features.
4.5 Detection decoder
The detection decoder decodes features into objects, including class and regression output. Given the feature map at the kth communication round (r(A), the detection decoder Pete (.) generate the
detections of ith agent by ((k) ae (F:(t)) #xx7 where each location of O(th) represents rotated box with class (c, I, h, w, cos a, sin a),, denoting class confidence, position, size and angle.
The objects are the final output of the proposed collaborative perception system. Note that ((0)
denotes the detections without collaboration.
4.6 Training details and loss functions
To train the overall system, we supervise two tasks: spatial confidence generation and object detection at each round. As mentioned before, the functionality of the spatial confidence generator is the same as the classification in the detection decoder. To promote parameter efficiency, our spatial confidence generator reuses the parameters of the detection decoder. For the multi-round settings, each round is

---

66
Where2comm
Where2comm
V2X-VIT
64
V2X-VIT
40
V2VNet
AP@0.55 0.50
AP@0:50 0.0.50
DiscoNet
62
30
When2com
NoCollaboration
60
V2VNet
20
DiscoNet
When2com
58
NoCollaboration
LateFusion
10
LateFusion
20
30
20
25
Communictioovolumello))
communicatiovoouue((()))
(a) CoPereeption-UAVs
(b) OPV2V
60
Where2comm
Where2comm
58
DiscoNet VIT
AP@0.5 0.50
56
AP@0.50
54
V2VNet
52
V2VNet DiscoNet
50
LateFusion
48
When2com
V2X-VIT
LateFusion
46
NoCollaboration
50
NoCollaboration
When2com
44
9
12
15
18
21
5
10
15
20
25
Communicationvolume(lo00
Communicatonvolumelll)))
(c) V2X-Sim
(d) -2X
Figure 3: Where2comm achieves consistently superior performance-bandviie trade-off on all the three collaborative perception datasets, e.g. Where2comm achieves 5, 5,000 times less communication volume and still outperforms When2com on CoPerception-UAY UAVs dataset. The entire red curve comes from single Where2comm model evaluated at varying bandwidths.
supervised with one detection loss, the overall loss is L = Ek Σ Laet (o/h). .O.) where O is the ith agent's ground-truth objects, Laet. is the detection loss
Training strategy for multi-round setting. To adapt to multi-round communication and dynamic bandwidth, we train the model under various communication settings with curriculum learning strategy [29]. We first gradually increase the communication bandwidth and round: and then, randomly sample bandwidth and round promote robustness. Through this training strategy, single model can perform well at various communication conditions.
5 Experimental Results
Our experiments covers four datasets, both real-world and simulation scenarios. two types of agents (cars and drones) and two types of sensors (LiDAR and cameras). Specifically, we con- duct camera-only 3D object detection in the setting of V2X-communication aided autonomous driving on OPV2V dataset [10], camera-only 3D object detection the setting of drone swarm on the proposed CoPercepttion-UAY UsVs dataset, and LiDAR-basee based 3D object detection DAIR-V2X dataset [II] and V2X-Sim dataset [9]. The detection results are evaluated by Average Precision (AP) at Intersection-over-UU Unver-Union (IoU) threshold of 0.50 and 0.70. The communication results count the message size by byte in log scale with base 2. To compare communication results straightforward and fair, do not consider any extra data/feature/model compression.
5.1 Datasets and experimental settings
OPV2V. OPV2V [10] is vehicle-to-vehicle collaborative perception dataset, co-simulated by OpenCDA [0] and Carla (30]. includes 12K frames of 3D point clouds and RGB images with 230K annotated 3D boxes. The perception range is 40m x 40m. For camera-only 3D object detection task on OPV2V, we implement the detector following CADDN [T. The input front-view image size (416, 160). The front-view input feature map transformed to BEV with resolution 0.5m/pixel.
V2X-Sim. V2X-Sim [9] is tohiele-t--evrrythingg collaborative perception dataset, co-simulated by SUMO 32 and Carla, including 10K frames of 3D LiDAR point clouds and 501K 3D boxes.
7

---

30
64-
APg
03
Rounds
Rounds
Rounds
Rounds
Rounds
Round
Round
Round
17.5 20,0 22.5
25.0 27.5
36
Communicaioonvluuelleez)
Conmonicaatioveeuueeteeee
(enmenicatiovveuumeltt))
(a) CoPerception UAVs
(b) OPV2V
(c) V2X-Sim
Figure 4: More communication rounds continuously improve iefffrmmncee-baddvie trade-off.
The perception range is 64mx64m. For LiDAR- -based 3D object detection task, our detector follows
MotionNet [33]. We discretize 3D points into a BEV map with size (256, 256, 13) and the resolution is 0.4m/pixel in length and width, 0.25m in height.
CoPerception-UAVS. To enrich the collaborative perception datasets, we consider the swarm of unmanned aerial vehicles (UAV) and propose UAV-swarm-baat CoPerception- UAVs, oo-simulated by AirSim [34] and Carla [30], including 131.9K aerial images and 1.94M 3D boxes. The perception range is 200m x 350m. For the camera-only 3D object detection task on CoPerception-UAV
collaborative perception dataset:
our detector follows DVDET [8] The input aerial image size (800, 450). The aerwal-view input feature map is transformed to BEV with the resolution of 0.25m/pixel, and the size is (192, 352); see more details in Appendix.
DAIR- DAIR-V2L [T]] is the only public real-world collaborative perception dataset. Each sample contains two agents: vehicle and an infrastructure, with 3D annotations. The perception range is 201. 101.6mx 80m. Originally
AAX--V2X does not label objects outside the camera's 's view, we relabel all objects to cover 360-degree detection range. We complement several intermediate fusion- basdd baselines on DAIR- V2X to comprehensively validate our method on real data. For iiDAR-based 3D object detection task, our detector follows PointPillar [35]. We represent the field of view into a BEV map with size (200, 504, 64) and the resolution is 0.4m/pixel in length and width.
5.2 Quantitative evaluation
Benchmark comparison. Fig. compares the proposed Where2comm with the previous methods in terms the trade-off between detection performance (AP loU=0.50) and communication bandwidth; also see exact values in Table 3 of Appendix. We consider single-agent detection without collaboration
( When2com [12. V2VNet M. DiscoNet (2) V2X- V2T-ViT (26 and late fusion, where agents directly exchange the detected 3D boxes. The red curve comes from a single Where2comm model evaluated at varying bandwidths. We see that the proposed Where2comm: i) achieves a far-more superior perception-communiaatioo trade-off across all the communication bandwidth choices and various collaborative perception tasks, including camera-only 3D object detection from aerial view and car front view, and LiDAR- basee 3D object detection; ii) achieves significant improvements previous of on both real-world (DAIR- V2X) and simulation scenarios, improves the SOTA performance by 7.7% DAIR-V2X. 6.62% CoPerception UAVs, 25.81% % on OPV2V, 1.9% on V2X-Sim; iii) achieves the same detection performance of previous state-of-the-ards with extremely less communication volume: 5128 times less on CoPerception-UAN UAVs, more than 100K times less on OPV2V, 55 times less on V2X-Sim, 105 times less on DAIR-V2X.
Multi-round evaluation. Fig. presents the performances of Where2comm communication rounds ranging from 3. Each curve comes from single Where2comm model with a certain communica- tion round evaluated at varying bandwidths. Results show that 1 communication round is good, more rounds are even better. Multi-round communication stcadily improves the serformaacc-baaddiine trade-off across all three datasets, reflecting its effectiveness and robustness. This encourages the agents to actively collaborate without worrying the performance degradation. This also validates that Where2comm can well work at various communication bandwidths and rounds.
Robustness to localization noise. We follow the localization noise setting in V2VNet and V2X-VIT (Gaussian noise with a mean of Om and a standard deviation of 0m-0.6m) and conduct experiments on all the three datasets to validate the robustness against realistic localization noise. Where2comm more robust to the localization noise than previous SOTAs. Fig- 5 shows the detection performances as a function of localization noise level in CoPerception UAVs, OPV2V and V2X-Sim datasets, respectively We see: i) overall the collaborative perception performance degrades with the increasing
8

---

0.50
AP8 50
60
Wheradca
Wherdzamm
WrenZcom
25
2hor2com
Wher2com
Discollrr
Discoflet
Disco
V/Viit
V2Vhet
ViVitt
NoCollsboration
NoCollbboetinn
NoCalabormsinn
0.5
0.1
0.3
Std localization im)
Sid of Incalization error (mr)
Std Ior aeralizztionn evtar (mm)
(a) CoPereeption-UASs
(b) OPV2V
(c) V2X-Sim
Figure Robustness to localization error. Gaussian noise with zero mean and varying std is
introduced. Where2comm consistently outperforms previous SOTAs and No Collaboration.
(a) 21 in BEV
(b) c
(c) M
1-22
(d) W
1-1
(e)
1.0
06
04
0.
X₂ in BEV
(g) (0)
(h) 2
W
(i) 1
0.0
2-11
Figure 6: Visualization of collaboration between Drone and Drone 2 on CoPerception-Ul UAVs (0)
dataset, including spatial confidence map (C) selection matrix (M 22 message ((R)) (0) -
the communication module, attention weight in the fusion module (W(") ((()) 2 and Drone I's
1->1 1->1.
detection results before (o()) and after (Ö(²) collaboration. Green and red boxes denote ground-
truth and detection, respectively. The objects occluded by tall building can be detected through
transmitting spatially sparse, yet perceptually critical message.
localization noise, while where2comm outperforms previous SOTAs (When2com, DisooNe)))) under all the localization noise. ii) where2comm keeps being superior to No Collaboration while V2VNet fails when noise over 0.4m and DiscoNet fails when noise is over 0.5m on CoPerception- UAVs. The reasons are: i) the powerful transformer architecture in fusion module attentively select the most suitable collaborative feature; ii) the spatial confidence map helps filter out noisy features, these two designs work together to mitigate noise localization distortion effects.
5.3 Qualitative evaluation
Visualization of spatial confidence map. Fig. iillustrates how Where2comm is empowered by the proposed spatial confidence map. In the scene, Drone I's view occluded by a tall building. With Drone 2 help, Drone 1 is able to detect through occlusion. Fig. 6 (a-d) shows Drone I's observation, spatial confidence map [ binary selection matrix ! and ego attention weight 3 Fig. 6 (f-h)
shows Drone 2's observation and message sent to Drone 1, including the request map (opposite of confidence map) and the sparse feature map, achieving efficient communication. Fig. 6 (i) shows the attention weight for Drone to fuse Drone 2's messages, which is sparse, yet highlights the objects' positions. Fig. 6 (c) and (j) compares the detection results before and after the collaboration with Drone 2. We see that the proposed spatial confidence map contributes spatially sparse, yet perceptually critical message, which effectively helps Drone detect occluded objects.
Visualization of detection results. Fig. shows that compared to No Collaboration, When2com and DiscoNet, Where2comm able to achieves more complete and accurate detection results. The reason is that When2com employs a scalar to denote the agent-to-agen agent attention, which cannot distinguish which spatial area is more informative: DiscoNel employs a MLP-based fusion weight learning, which cannot well capture the complex collaboration attention; while Where2comm can zoom to critical spatial areas a cell-level resolution and leverage the spatial confidence map and sensing distances priors achieve more comprehensive fusion.

---

00
00
(d) Where2comm Figure 7: Where2comm qualitatively outperforms When2com and DiscoNet in DAIR-V22 V2X dataset. Green and red boxes denote ground-trute truth and detection, respectively. Yellow and blue denote the point clouds collected from vehicle and infrastructure, respectively.
(a) No Collaboration
(b) When2com
(c) DiscoNet
Table 2: Fusion component ablation study. Multi-head atten- tion (MHA), sensor positional encoding (SPE) and spatial confidence map (SCM) all improves the performances. Re- sults are reported in AP@0.50/AP@0.70.
AP@0.50
64-
Gaussian NoGaussian
MHA SPE SCM
OPV2V
CoPerception UAVs V2X-Sim
18
Communicationollumclioz2
34.96/13.92
63.48/44.23
51.2/45.7
Figure 8: Selection matrix ablation study. Applying Gaussian filter im- proves performance.
38.75/13.28
63, 63.99/44.46
57.3/50.8
39.82/16.43
64,34/46.80
59.1/52.0
47.30/19.30
64.83/47.62
59.1/52.2
5.4 Ablation studies
Effect of Gaussian filter in perceptually critical area selection. Fig. 8 compares two versions of the selection matrix (2) with and without Gaussian filter. We see that applying Gaussian filter improves the overall performance. The reason that: i) Gaussian filter could help filter the outliers in the input map, selecting more robust critical regions; ii) it considers the context, benefiting the independent feature selection at each certain location by providing more information.
Effect of components in spatial confidence-aware message fusion. Tab, 2 assesses the effectiveness of the proposed fusion with priors. We see that: i) per-location multi-head attention (MHA) outperforms the vanilla attention by 10.84% on OPV2V on 0.50.50 because MHA leverages information from multiple heads, better capturing cross-agent attention; and ii) As two informative priors, both sensing position encoding (SPE) and spatial confidence map (SCM) can consistently improve the performance. Especially, the version with all three designs improves the detection performance by 22.06% on OPV2V on AP@0.50.
6 Conclusion and limitation
We propose Where2comm, novel core idea to exploit a spatial confidence map at each agent to promote pragmatic compression, assisting agents to decide what to communicate with whom, and whose information to aggregate. Each agent offers spatially sparse, yet perceptually critical features to support other agents; meanwhile, requests complementary information from others in multi-round communication. Comprehensive experiments covering multi-type agents and multi-modall modality inputs show that Where2comm achieves far superior trade-off between perception performance and communication bandwidth.
www.mmiiiaaooonffffiieee collaborative perception framework. The
Limitation and future work. The current work focuses on perceptually critical spatial areas. In future, we plan to expand a similar idea to the temporal dimension and determine critical time stamps. More cost will be reduced by exploring when to communicate. We also expect that more methods on pragmatic compression and emergent communication could be applied to collaborative perception.
Acknowledgment. This research is partially supported by the National Key R&D Program of China under Grant 2021 2021ZD01 12801, National Natural Science Foundation of China under Grant 62171276. the Science and Technology Commission of Shanghai Municipal under Grant 2151 21511100000 1009000, CCF- DiDi GAIA Research Collaboration Plan 2021 12 and CALT Grant 2021-01.
10

---

References
[1] Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang. Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehiele communication for joint perception and prediction. In European Conference on Computer Vixion, pages 605-621. Springer, 2020.
[2] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng. and Wenjun Zhang, Learning distilled collahoration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34, 2021.
[3] Siheng Chen, Baoan Liu, Chen Feng. Carlos Vallespi Gonzalez, and Carl K. Wellington. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEEE Signal Processing Magazine, 38:68-B8 86, 2021.
Zhi Li, Ali Vatankhah Barenji, Jiazhi Jiang, Ray Y Zhong, and Gangyan Xu. mechanism for scheduling multi robot intelligent warehouse system face with dynamic demand. Journal of Intelligent Manufacturing. 31 3142)4499480, 2020.
Michela Zaccaria, Mikhail Giorgini, Riccardo Monica, and Jacopo Alcotti. Multi-robot multiple camera people detection and tracking in automated warchouses. In 2021 IEEE 19th International Conference on Industrial Informatics (INDIN), pages 1-6. IEEE, 2021.
[6] Jürgen Scherer, Saeed Yahyanejad. Samira Hayat, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovic, Christian Bettstetter, Hermann Hellwagner, and Bemhard Rinner. An autonomous multi-uav system for search and rescue. In Proceedings of the First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use, pages 33-38, 2015.
[7] Ebtehal Turki Alotaibi, Shahad Saleh Alqefari, and Anis Koubaa. Lsar: Multi-uav collaboration for search and rescue missions. IEEE Access, 7:55817-55832 2019.
[8] Yue Hu, Shaoheng Fang, Weidi Xie, and Siheng Chen, Aerial monocular 3d object detection. arXiv preprint 22xx:2208.03974, 2022.
[9] Yiming Li, Ziyan An, Zixun Wang, Yiqi Zhong, Siheng Chen, and Chen Feng. V2X-Sim: virtual collaborative pereeption dataset for autonomous driving. IEEE Robotics and Automation Letters, 7, 2022.
[10] Runsheng Xu, Hao Xiang. Xin Xia. Xu Han, Jinlong Liu, and Jiaqi Ma. OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to- vehicll communication. ICRA, 2022.
[11] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang. Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, al. DAIR-V2X: A large-scale dataset for rehicle-infrastrucuur cooperative 3d object detection. In Proceedings of the IEFF/CVF Conference on computer vision and pattern recognition (CVPR), 2022.
[12] Ven-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agemt perception via communication graph grouping. Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 4106-4115 15, 2020.
[13] Yen- Ben-Cheng Liu, Junjiao Tian, Chih- Yao Ma, Nathan Glaser, Chia- Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via Icarnable handshake communication. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 6876-6883, IEEE, 2020.
[14] Yang Zhou, Jiuhong Xiao. Yue Zhou, and Giuseppe Loianno. Multi-robot collaborative perception with graph neural networks. IEEE Robotics and Automation Letters, 2022.
[15] Eduardo Amold, Mchrdad Dianati, and Robert de Temple. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. IEFE Transactions on Intelligent Transportation Systems, 23:1852-1864. 2022.
[16] Zixing Lei. Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. ECCV, 2022.
[17] Yiming Li, Juexiao Zhang, Dekun Ma. Yue Wang, and Chen Feng. Multi-robot scene completion: Towards task agnostii collaborative perception. In Conference on Robot Learning (CoRL). PMLR, 2022.
[18] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. CoBEVT: Cooperative bird's eye view semantic segmentation with sparse transformers. CoRL, 2022.
[19] Sanbao Su, Yiming Li, Sihong He, Songyang Han, Chen Feng, Caiwen Ding, and Fei Miao. Uncertainty quantification of collaborative detection for self-driving, 2022.
11

---

[20] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. ICLR, 2019.
[21] Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceedings of the tenth international conference on machine learning, pages 330-337, 1993.
[22] Faisal Qureshi and Demetri Terzopoulos. Smart camera networks in virtual reality. Proceedings of the IEEE, (6(1 1646-1666,,,, 2008.
[23] Yiming Li, Bir Bhanu, and Wei Lin. Auction protocol for camera active control. In 2010 JEEF International Conference on Image Processing, pages 4325-4322 IEEE, 2010.
[24] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in neural information processing systems, 29, 2016.
[25] Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. Advances in Neural Information Processing Systems, 30, 2017.
[26] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2X-ViT: Vehiccl-to-everyyhingg cooperative perception with vision transformer. ECCV, 2022.
[27] Y Yuan and M Sester. Comap: A synthetic datasct for collective multi- agent perception of autonomous driving. The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences, 43:255-263, 2021.
[28] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühh. Objects as points. In arXiv preprint arXiv: 1904.0990000 904.000000 2019.
[29] Yoshua Bengio, Jérôme Louradour, Ronan Collohert, and Jason Weston. Curriculum learning. In Proceedings of 26th annual international conference on machine learning, pages 41-48, 2009.
[30] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[31] Cody Reading, Ali Harakeh, Julia Chac, and Steven L.. Waslander. Categorical depth distrihution network for monocular object detection. CVPR, 2021.
[32] Daniel Krajzewicz, Jakob Erdmann, Michael Behrisch, and Laura Bieker. Recent development and applications sumo-simulation urban mobility. International journal on advances in systems and measurements, 5(3&4), 2012.
[33] Pengxiang Wu, Siheng Chen, and Dimitris N. Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based on bird's сye view maps. 2020 /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11132-1139,, 2020.
[34] Shital Shah, Dehadeepta Dey, Chris Lovett, and Ashish Kapnor. Airsim: High-fidelity visual and physical simulation for autonomous vehicles. In Field and service robotics, pages 621-635. Springer, 2018 8.
[35] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12689-12697, 2019.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See Section [.] and the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5.1 and the supplemental material.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We have not repeated experiments many times get error bars since experiments of 3d object detection on large scale datasets is time-consuming.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 5.1 and the supplemental material.
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 5.1
(b) Did you mention the license of the assets? [Yes] See Section 5.1
(c) Did you include any assets either in the supplemental material
URL? [Yes]
See Section 5.1 and the supplemental material.
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A] We generate the data using the open- tool and the owners consent to all the public use for research purposes.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] Our data synthesized using the open-sourced tool, there is no real-world personally identifiable information, nor offensive content.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "xAF8-sgM_I1", "reviewer": "Reviewer_YUZ9", "summary": "This paper proposes a framework, where2comm, to enable multi-agent collaborative perception via a more improved information sharing through communications among agents. The core idea of where2comm is the use of a spatial confidence map, enabling each agent to decide which area needs to be shared and communicated about with other agents. Where2comm is composed of (1) spatial confidence generator (2) spatial confidence-aware communication module, (3) spatial confidence-aware message fusion module. Figure 2 well elaborates the overall structure of the system; the camera/lidar input goes through the encoder first, and the output of the encoder goes through the spatial confidence generator, which generates spatial confidence map that shows which areas in the map is perceptually critical level. The use of spatial confidence map is leveraged during the communication phase, which selects where to communicate about. ", "questions": "Q1. The select function in the message packing module: to me, select function is not well described in the section. Selection function takes the element-wise  multiplication of confidence map of i-th agent C_i and request map of j-th agent R_j as inputs, and returns the output by selecting the largest elements of the given input. What does largest elements mean? Under which criteria the inputs of the C_i*R_j survive? \n\nQ2. Communication graph is constructed from the adjacency matrix A_ij^k, which is the result of the maximum value of the binary selection matrix M_ij^k. It seems that A_ij^k is matrix is binary (0 or 1) matrix; what is the form of M_ij^k matrix? Are A and M both binary matrix? Can you visualize M matrix and A matrix?\n\nQ3. Figure 5 (visualization) is very helpful. (b) is the confidence map, which indicates that the drone1’s view is more confidence about the non-dark area in the (a). How should I interpret (g)? It seems that this is the request map from the drone2 view. What does the red spots in the (g) request map represent? ", "limitations": "The authors mentioned about the limitations, which is that this paper only focuses on perceptually critical spatial areas, and doesn't care about temporal dimension. I think this is reasonable limitations and suggestions for the future work. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength of this paper comes from the originality of the framework, especially novel modules like message packing and communication graph construction. Message packing includes request map, which indicates where (which areas) needs to be further known, and is computed from negative of the confidence score. Also, there is spatially sparse map, which are computed from requests from other agents. \n\nCommunication graph construction identifies when and who to communicate by measuring overlap between information i-th agent has and information j-th agent needs. \n\nI think the overall strength of this paper comes with sound problem formulation, the need for a fraemwork for multi-agent collaborative perception, and original/novel framework proposals. Weak points (or the points that need to be clarified) will be asked in the Questions section.\n\nAnother strength is that the where2comm achieves thousand times less communication volume and still outperforms other baselines, (Figure 3).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "j36VTWhSf3u", "reviewer": "Reviewer_VaH1", "summary": "Multi-agent collaborative perception could avoid accidents due to an agent's limited field of view via sensing information from other agents. However, the communication bandwidth is a bottleneck for multi-agent collaborative perception systems. The critical question is what to send and whom to send it. The authors propose a novel framework called Where2comm for 3D detection to address the challenges. A spatial confidence map is proposed to advance the performance of multi-agent collaborative perception. The paper benchmarks the framework on three datasets, OPV2V, V2X-Sim, and our original CoPerception-UAVs. The framework achieves state-of-the-art performance on these benchmarks. Moreover, the authors show that the proposed method can significantly reduce communication bandwidth while maintaining favorable performance compared with STOA.", "questions": "The paper presents a promising framework for Multi-agent collaborative perception for 3D object detection. The authors motivate the readers and propose a spatial confidence map that can be used to communicate critical information with other agents and reduce communication bandwidth. The experiments empirically demonstrate the effectiveness of the proposed framework. The reviewer recommends accepting the paper. The current rating is weak accept (6). The reviewer's main concern is the sensitivity of the framework to noisy localization, which might significantly degrade the contribution of a spatial confidence map in a real-world setting. ", "limitations": "The authors did mention limitations of the work on the lack of considering the temporal aspect of critical timestamps.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. To reduce the number of traffic fatalities, multi-agent collaborative perception for 3D object detection is a promising direction, and it is a topic of interest to machine learning, computer vision, and intelligent transportation system researchers.\n2. To the best knowledge of the reviewer, the concept of spatial confidence map has not been leveraged for multi-agent collaborative perception. The authors proposed an architecture to realize the concept and used the encoded information to send critical information for 3D object detection. The reviewer appreciates the proposed architecture as it improves interpretability. In addition, because of the design of a spatial confidence map, the framework can send sparse yet critical information to other agents and show improvement in 3D object detection.\n3. Promising performance in 3D object detection AP and communication volume usage is demonstrated on three datasets, OPV2V, V2X-Sim, and CoPerception-UAVs. The experiments empirically demonstrate the effectiveness of the proposed framework.\n4. Qualitative evaluations show that the proposed method **exhibits** promising explainability. Toward trustworthy multi-agent collaborative perception, such attributes would be much appreciated.\n\n\nWeaknesses:\n1. Related work: The reviewer found that the proposed spatial confidence map is highly relevant to the probabilistic occupancy map and the concept of a local grid map for robotics applications. The reviewer encourages the authors to acknowledge the existence of these works.\n    1. H. Li, M. Tsukada, F. Nashashibi, and M. Parent, \"Multivehicle Cooperative Local Mapping: A Methodology Based on Occupancy Grid Map Merging,\" in *IEEE Transactions on Intelligent Transportation Systems*, vol. 15, no. 5, pp. 2089-2100, Oct. 2014\n    2. N. Deo and M. M. Trivedi, \"Convolutional Social Pooling for Vehicle Trajectory Prediction,\" CVPR 2018\n    3. C. Chen, Y. Liu, S. Kreiss, and A. Alahi, \"Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning,\" *ICRA 2019*\n2. Perfect localization: According to the reviewer's understanding, the proposed framework relies on the accurate localization of all agents in a scene. Without the assumption, a spatial confidence map cannot be applied easily. The reviewer felt that noisy localization could lead to significant performance degradation for 3D object detection. Toward real-world applications, it is inevitable to encounter such issues. It would be an informative study to analyze the sensitivity of the proposed method on noisy localization. Moreover, what is the corresponding performance compared with the existing STOA?\n3. Communication volume: The reviewer has difficulty understanding how the authors adjust the communication volume shown in Figures 3 and 4. The reviewer's best guess is relevant to Figure 2, shown in the supplementary material. However, it is hard to reproduce the results without a clear explanation.\n4. Ablative studies:\n    1. What is the baseline implementation without using SCM? Is it simply using BEV features?\n    2. Spatial confidence-aware message fusion module: What if we use max-pooling/concatenation for fusion?\n5. Can the proposed algorithm solve pedestrian or other traffic participant detection? What could be the potential limitations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "UaKsG38hkZ", "reviewer": "Reviewer_cKVM", "summary": "The paper presents a collaborative perception model that specifically focuses on reducing communication using spatial confidence maps. These maps allow agents to decide which areas of the scene they need further information from, as well as which areas of the scene they have relevant information. The authors use attention to allow the agents to attend their, and other, confidence maps to create a communication graph. The authors then use a Transformer Architecture with Multi-Head Attention to fuse the features from multiple agents. The fused features will act as the agents input features in the next iteration, and they are also fed to a decoder that can regress the relevant information (class and bounding box). Overall, the approach achieves significantly lower communication rates, while enabling the network to also use similar communication Volumes as the state of the art.\n ", "questions": "1. You mention using distance between the sensor and the feature to perform positional encoding. How does this work for a monocular camera? Do you assume known depth?\n2. Can you elaborate in the warping function from iage to BEV? How is this performed? This is an active area of research, see e.g. Saha, Avishkar, et al. \"Translating images into maps.\" arXiv preprint arXiv:2110.00966 (2021). and Roddick, Thomas, and Roberto Cipolla. \"Predicting semantic map representations from images using pyramid occupancy networks.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n3. Why BEV? Have you looked at other ways of unifying the data? Could this be a learned transformation applied to the data?\n4. For BEV, how robust is your system to distortions introduced by ground projection?", "limitations": "The work is only evaluated in simulation. This presents important challengs that must be solved. The authors should include limitations in terms of time alignment, assumptions about sensor modalities (is depth required?) and other consequences of moving into the real world. The authors mention expanding their idea into the temporal dimension, which seems like a very good idea. Although another possible limitation of their approach is that as the dimensionality of the problem gets bigger, the attention component might not be capable of attending to all the relevant information, resulting in sub-standard communication.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well presented, generally reads well and provides a good case for the authors Spatial Confidence maps as a way to reduce communication and increase performance in collaborative perception. The quantitative data suggest the model can outperform the state of the art on the evaluated datasets. It shows a strong ability to reduce the amount of data, while still outperforming all the benchmarks. There is a massive reduction in the amount of data used, which should be commended. The qualitative data shows much improved performance in the detection. Furthermore, the ablation study cleary shows improvements introduced by each part of the architecture.\n\nIn terms of weaknesses, the main limitation is that the models is only evaluated on synthetic datasets with favourable conditions. A discussion of limitations including things like time synchronisation, noise, availability of data and other considerations are important to present on an approach that relies on collaborative perception. Additionally, the paper has various spelling and grammar mistakes that should be rectified.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "R4gA9S1R7m", "reviewer": "Reviewer_Jgi5", "summary": "This paper studies the task of multi-agent collaborative perception with the purpose of achieving better trade-off between performance and bandwidth consumption. To reduce the communication cost across the agents, the authors propose a new approach to generate and transmit the spatial confidence map instead of the traditional whole feature map.  The framework proposed in this paper consists of five modules, including the observation encoder, spatial confidence generator, spatial confidence-aware communication module, communication graph construction, spatial confidence-aware message fusion and detection decoder. The authors have conducted experiments on three datasets to verify the effectiveness of their new approach. Compared with five baselines, the new approach has shown higher detection accuracy and smaller communication cost.", "questions": "1. Where are the results of the five baselines from?  Did the authors run the public source codes of the baselines on the three datasets?  Could the authors provide some details in the paper?\n\n2. In the original paper, DiscoNet has reported detection accuracy of 60.3% and 58.5% in two different versions on dataset V2X-Sim, without and with 16 times compression, respectively. However, the accuracy of DiscoNet is around 58.0% in this paper. Why the results in two papers have the difference?\n\n3. Where is the dataset CoPerception-UAVs from? It seems it's not a public dataset. If so, more details on the dataset would be helpful. For example, collecting data from drones is much trickier than from self-driving cars since drones can have any arbitrary poses. So more details are required. Also, how is training/val/test conducted? The appendix doesn’t contain dataset details even though the main text claims so.\n\n4. How are the constrained loss function and binary variables optimized? \n\n5. Can you elaborate more on \"sensor positional encoding\"? Why should it represent the physical distance between each agent’s sensor and its observation? There is neither justifications nor references provided.\n\n6. Given the complexity of the proposed method, it would be great if the authors could provide the code (the same as the baseline methods). So people check for details that aren't provided from the paper. \n", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. It is a good idea to transmit part of the feature map across the agents to reduce the communication volume, and potentially improve detection accuracy. \n2. The authors have conducted extensive experiments on three different datasets to validate the new approach. The experiments cover two types of agents (cars and drones), and two types of sensors (LiDAR and camera). They also present many experimental results to support the conclusion, including quantitative comparison and visualization.\n\nWeaknesses:\n1. It’s a very complex model with 5 modules. Although the authors present the high-level ideas of each module, a lot of details on the architectures and optimization techniques are missing. For example, how the constrained optimization presented in Sec. 3 is optimized? How are binary variables in M (Eq. 2) and A optimized? Are there any optimization issues due to hard constrains and non-differentialability of binary variables? \n\n2. The authors compare their proposed method against five baselines. I have checked the original papers of the baselines, and found that only DiscoNet reports the results on V2X-Sim, which is one of three datasets in this paper. Since most results of five baselines reported in this work are not directly from the original papers, a solid comparison between the proposed approach and the baselines is required. Apparently, Fig. 3(c) shows that the performance gap between DiscoNet and Where2Com on V2X-Sim is much smaller than the other two datasets, while V2X-Sim is the only dataset that DiscoNet reported its performance.\n\n3. The performance of communication volume varies much on three different datasets, 5128 times less on CoPerception-UAVs, 10,000 times less on OPV2V, and 55 times less on V2X-Sim. It seems that the authors do not introduce the details of the metric for communication volume. I am not sure whether the baselines used the same metric before. The comparison also has the potential fairness issue.\n\n4. Overall, given the presentation of the paper and a large amount of details are missing, I don’t think the algorithm can be easily reproduced by readers.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept