How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perception
Authors: Dingkang Yang
Kun Yang
Yuzheng Wang
Jing Liu
Zhi Xu
Rongbin Yin
Peng Zhai
Lihua Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Collaborative perception
multi-agent communication
TL;DR: We propose a multi-agent collaborative perception framework for autonomous driving scenarios to achieve a trade-off between perception performance and communication bandwidth.

Abstract: Multi-agent collaborative perception has recently received widespread attention as an emerging application in driving scenarios. Despite the advancements in previous efforts, challenges remain due to various noises in the perception procedure, including communication redundancy, transmission delay, and collaboration heterogeneity. To tackle these issues, we propose \textit{How2comm}, a collaborative perception framework that seeks a trade-off between perception performance and communication bandwidth. Our novelties lie in three aspects. First, we devise a mutual information-aware communication mechanism to maximally sustain the informative features shared by collaborators. The spatial-channel filtering is adopted to perform effective feature sparsification for efficient communication. Second, we present a flow-guided delay compensation strategy to predict future characteristics from collaborators and eliminate feature misalignment due to temporal asynchrony. Ultimately, a pragmatic collaboration transformer is introduced to integrate holistic spatial semantics and temporal context clues among agents. Our framework is thoroughly evaluated on several LiDAR-based collaborative detection datasets in real-world and simulated scenarios. Comprehensive experiments demonstrate the superiority of How2comm and the effectiveness of all its vital components. The code will be released at https://github.com/ydk122024/How2comm.

How2comm: Conmunication-lfiiient and Dollaboration-Pragmattii Multi-Agent Perception
Dingkang Yang 1.2t Zhi Xu
Kun Yanglt
Yuzheng Wang'
Jing Liu' Lihua 122,3,4
Rongbin Yin
Peng 2.ai
AAcademy for Engineering and Technology, Fudan University "Cognition and Intelligent Technology Laboratory (CIT Lab)
Engineering Research Center of Al and Rohotics, Ministry of Education AT and Unmanned Systems Engineering Research Center of Jilin Province
FFAW (Nanjing) Technology Development Company Ltd { [kkyang20, kungyang20, pzhai, lihuazhang / @fudan edu.
Abstract
Multi-agent collaborative perception has recently received widespread attention as emerging application driving scenarios. Despite the advancements in previous efforts, challenges remain due to various dilemmas in the perception procedure, including communication redundancy, transmission delay, and collaboration hetero- geneity. To tackle these issues, we propose How2comm, collaborative perception framework that seeks a trade-off between perception performance and communi- cation bandwidth. Our novelties lie in three aspects. First, we devise a mutual information-aware communication mechanism to maximally sustain the informa- tive features shared by collaborators. The spatial-channel filtering is adopted perform effective feature sparsífication for efficient communication. Second, we present flow-guided delay compensation strategy predict future characteristics from collaborators and climinate feature misalignment due to temporal asynchrony. Ultimately, pragmatic collaboration transformer introduced to integrate holistic spatial semantics and temporal context clues among agents. Our framework is thoroughly evaluated several LiDAR- collaborative detection datasets in real-world and simulated scenarios. Comprehensive experiments demonstrate the superiority of How2comm and the effectiveness of all its vital components. The code will be released at https: htpps:///itaan /ggtthu..... com/ydk cam//dk122024 111/222222Hoooooommm Hoom/com...
Introduction
Precise perception of complex and changeable driving environments is essential to ensure the safety and reliability of intelligent agents [25, 46), e.g., autonomous vehicles (AVs). With the emergence of learning-bast based technologies, remarkable single-agen perception systems are extensively explored for several in-vehicle tasks, such instance segmentation [19, 58] and object detection [26, 49]. Nevertheless, single-agent perception suffers from various shortcomings due to the isolated view, such unavoidable occlusions [55], restricted detection ranges [56], and sparse sensor observa- tions [57]- Recently, multi-agent collaborative perception [39, 54) has provided promising solutions emerging application for vehicle-to-vee shhiccee-o-vehiclleeeeethhhiii (V2V/X) communications, The im- pressive studies [7, 14, 16, 17, 31, 32, 33, 38, 40, 51] have progressively presented to aggregate valuable information and complementary perspectives among on-road agents, resulting in a more
'Equal contributions * The two first authors thank Runsheng Xu for providing constructive suggestions. "Correponniig authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) Aligned point cloud fusion Non-aligned point cloud fusion (c) Infrastructure side image
(d) Fused point cloud
Figure I: (a) and (b) show the point cloud fusion results in the absence and presence of transmission delay, respectively. (c) and (d) show the vanilla image and fused point cloud of the collaborative perception scene containing ego vehicle (red circle) and infrastructure, respectively.
precise perception. Despite recent advancements, challenges remain due to unpreventable dilemmas, including communication redundancy, transmission delay, and collaboration heterogeneity.
Communication Redundancy, The dominant patterns for reducing communication overhead are summarized as feature compression [14, 33, 40] and spatial filtering [7, 8). The former assumes that agents share all spatial areas indiscriminately, which dramatically wastes bandwidth. The latter overly relies confidence maps to highlight gullible locations and fails to consider spatially holistic information. Moreover, these methods invariably cause losses of transmitted valuable information.
Transmission Delay. Figures ((a)((b) present the point cloud fusion results from an ego vehicle and an infrastructure in the sime-synchooouu and time-asye seee-aayccccoouus cases, respectively. The in- evitable transmission delay causes position misalignment of fast-moving objects within the green circles, potentially harming subsequent collaboration performance. Although several delay-aware strategies [13, 40, 53] proposed tackle this issue, they either suffer from performance bottle- necks [13, 40] or introduce massive computation costs [53], leading to sub-optimal solutions.
Collaboration Heterogeneity. Figures (d))&(d) show the typical collaboration scenario involving two agents and the fused point cloud. Intuitively, LiDAR configuration discrepancies (e.g.. different LiDAR densities, distributions, reflectivities, and noise interference) across agents potentially cause collaboration heterogeneity within the feature space. In this case, the orange box contains the common perception region of both agents, which facilitates bridging the feature-level gap caused by sensor configuration discrepancies [37, 40J. The magenta box contains the exclusive perception region of the infrastructure, which provides complementary information for the ego vehiele and compensates for the occluded view. Fusing valuable spatial semantics from these two perception regions facilitates comprehensive and pragmatic perception. However, most previous methods [14, 16, 32, 33, 40] integrate collaborator-shared features via per-agen/loocttion message fusion to enhance ego representations, whose collaboration processes could be vulnerable since the advantages of distinct perception regions from heterogeneous agents are not considered holistically. Moreover, the current single-frame perception paradigm faces the challenges of 3D point cloud sparsity and localization errors, increasing the difficulty of building robust multi-agent perception system.
Motivated by the above observations, we propose How2comm, end-to-end collaborative perception framework to address the existing issues jointly. Through three novel components, How2comm advances towards reasonable trade-off between perception performance and communication band- width. Specifically, (i) we first design a mutual information aware communication mechanism to maximally preserve the beneficial semanties from vanilla characteristics in the transmitted messages of collaborators. In this case, spatial-channel message filtering is introduced to determine how to use less bandwidth for efficient communication. (ii) Second, present flow-guided delay compensation strategy to predict the future features of collaborators by mining contextual dependencies in sequential frames. Our ingenious strategy determines how dynamically compensate for the delay's 's impact and explicitly accomplish temporal alignment. (iii) Furthermore, we construct a spatio-temporal collaboration transformer (STCFormer) module to integrate perceptually comprehensive information from collaborators and temporally valuable clues among agents. Our unified transformer structure determines how to achieve pragmatic collaboration, contributing more robust collaborative per- ception against localization errors and feature discrepancies. How2comm is systematically evaluated on various collaborative 3D object detection datasets, including DAIR- V2X [52], V2XSet [40], and OPV2V [41]. Quantitative experiments demonstrate that our framework significantly outperforms previous state-of-the-art (SOTA) methods under the handwidth-limite noisy setting. Systematic analyses confirm the robustness of How2comm against distinct collaboration noises.

---

Indinstructurm
Flow G Generator
Spatio-Temporr
JF
Collaboration
5%
F7
Transformer
XL.
(ST
Projected
Message Filtering
Detection Decoders
Stared
R. R.c.
Regueet
Message Packing
I
Zi
M Msssage Delivery
Proflct
LIDAR Projected
F F
R. Broadcast
Z;
Z
(1)
Importance G Generator
Ego Vohidl
X!
Map
Temporal Connect
Figure 2: How2comm overview. The projected features of all agents obtained via a shared encoder. Upon receiving requests {R. R. from the ego vehicle, the collaborator (i.e., infrastructure) shares the sparse feature f feature flow OJ and scale matrix S/ via the message filtering and flow generator. After that, the ego vehicle predicts the future feature Z and adopts the importance maps {I; to get the exclusive and common maps, which decouple Z into Zj.e and Zj.. Finally, our STCFormer fuses the temporal context and decoupled spatial features to output F; used for detection.
2 Related Work
Multi-Agent Communication. Benefiting from rapid advances in learning baasid technolo- gies [12, 21, 22, 23, 24, 42, 43, 44, 45, 47, 48, 50], communication has played an essential role in constructing robust and stable multi-agent systems. Although early works provided heuristic insights into information sharing among different agents through predefined protocols and centralized paradigms [4, 28, 29), these efforts are typically difficult to generalize into challenging scenarios. Recently, several lcarning-driven communication strategies have been proposed to accommodate diverse scenario applications. For instance, Vain [6] utilized an attentional neural structure specify what information needs to be shared in agent interactions. ATOC [9] introduced the recurrent unit to decide whom the agent communicates with by receiving local observations and action intentions from other agents. TarMAC [2] designed a reinforcement bearning-oriente communication from task-specific rewards. In comparison, 3D object detection tasks. For more challenging driving scenarios, we design the mutual information supervision and intention-guiding mechanism achieve efficient communication across agents.
architecture to learn focus LiDAR -assed collaborative
Collaborative Perception. Collaborative perception only in infancy as a promising application of multi-agent systems. Several impressive approaches have been designed to facilitate the overall perception performance of AVs. The mainstream works [7, 14, 16, 33, 37, 38, 40] followed the intermediate collaboration pattern to balance average precision and bandwidth overhead. Specifically, When2com [16] introduced a handshake mechanism to determine when to communicate with collab- orators. V2VNet [33] employed fully connected graph network to aggregate feature representations shared by agents. After that, DiscoNet [14] proposed a knowledge distillation framework to supervise the intermediate collaboration through early collaboration of full views. V2X-ViT [40] designed distinct attention interactions to facilitate adaptive information fusion among heterogeneous agents. Where2comm [7] aimed transmit perceptually critical information via sparse spatial confidence maps. However, these methods invariably ignore valuable historical clues and lead to sub-optimal solutions. In this paper, we propose a novel collaboration transformer to jointly capture spatial semantics and temporal dynamics among agents, resulting in a more pragmatic collaboration.
3 Methodology
3.1 Problem Formulation
In this paper, we seek to develop communication-efficee and collaboration-pragmaii multi-agent system to enhance the perception ability of the ego agent. Figure 2 illustrates the proposed system framework, which accommodates different agents (e.g., AVs and infrastructures). Consider N agents in driving scene, let X be the local point cloud observation of the i-th agent (i.c., ego agent) and

---

}/; be the corresponding ground ground-truth truth supervision. The objective of How2comm is to maximize the LiDAR-based 3D detection performance (.) under a total Commumntt Connicatioon budget B:
R () arg max 0.F;
(F) 8 Vo X ( ).))) s.t. Σ F B,
(1)
where Ö(- :) denotes the perception evaluation metric and V is the perception system parameztan by 0. F is the message transmitted from the j-th agent the i-th agent at time delay 7-aware moment to. where to = t - T. The remainder of Section 3 details the major components.
3.2 Metadata Conversion and Feature Extraction
In the initial stage of collaboration, we build a communication graph [14, 40] where one agent is selected as the ego agent and the other connected agents act as collaborators. Upon receiving the
broadcast metadata (e.g., poses and extrinsic) from the ego agent, the collaborators project their local observations to the ego agent's coordinate system. Moreover, ego-motion compensation [35] synchronizes each agent's historical frames. The shared PointPillar [11] encoder seno(-) converts the point cloud of the i-th agent at timestamp t into the bird's-eye- s-eee-vieww (BEV) features as F. = Senc(A (!) E IR. HxxWxX, where H, W, C denote height, width, and channel, respectively.
3.3 Mutual Information-awan Communication
Previous attempts to reduce the required transmission bandwidth relied heavily autoencoders [14, 33, 40] or confidence maps [7, 8). which are one-sided as they only consider information compression over spatial positions channels. To this end, design mutual information-aww aware communicaioos (MIC) mechanism to select the most informative messages from space and channels to save precious bandwidth. MIC consists of two core parts follows.
Spatial-channel Message Filttriigg Stemming from solid evidence in signal picking, we first introduce the CBAM [34]-like spatial-channel attention queries to assist each agent in sharing their salient features. The spatial query A a (wssslyaa (F( ): Pm (F F ))) E R W x reflects what
spatial locations on delayed feature F are informative, where [+; .] is the concatenation, is the sigmoid activation, (Pa/m() (-) denote average and max pooling functionn, and w3.3()) is the 2D 3 x convolution operation. The channel query A ( (F/1) + who1 w (Fo))) ))) E RRIIIX
reflects which channels in F semantically meaningful. Pa/m (-) in the spatial and channel queries are applied to the channel and spatial dimensions, respectively. Then, the ego agent indicates the supplementary messages required to improve local perception performance by broadcasting request queries / = A / The j-th collaborator then aggregates the requests with its attention queries to obtain spatial-chanae binary message filtering matrix as follows:
M (w1 IR. 4%% @ Wi.1 [R s,ct
A (,c))) H)
(2)
where [sa(()) is threshold-based selection function and @ is the element-wise multiplication. Ultimately, the selected feature map is obtained as Fifo channel sparse, yet perceptually critical information.
F MM which provides spatial-
Mutual Information Maximization Supervision. Most existing works ignore the potential loss of valuable information due to feature compression, To overcome this dilemma, we maximally sustain the local critical semantics in the correspondigg vanilla feature E/ the selected regions of the
rransmine
transmitted features f by mutual information estimation. Since we only focus on maximizing the mutual information rather than getting its precise value, stable estimator [5] is utilized to build the objective supervision based on the Jensen-Shannon divergence. Formally, the mutual information e
between random variables X' and 2 is estimated follows:
F(JD)) (X, (x: 2)
E Epp(..) logg(1 + e TT((.)))] (ppe))(ee + eToor.)))))
where Te X'x x Z + R is a statistics network parameterized by g. In our case, the mutual information supervision of all collaborators within the communication link is defined as follows:
Lmal
Σ
f(SD) (F F
(4)
N-1
BE(I....NNA$44

---

(a)
(b)
(c)
N,50
Adaptive Fusion
2D Comv
Filling
inear
F7
saylmas
Cress- Atentionn
AW +1
Foed Forward
Feed Forward
Query
N.XAXLX
Linnar
Attaddnng Feature
Add
Linear
Exclusive Spatial Atention
Common Spatial Attnntion
Com Tixan
ConY
Predtion Emblidiing
Global Pooling
Global Pooling
L Linear
Target Query
Temporal C Tross- ttention
20 Com
2D Conv
1x xac
2D Conv
Sampling
Officts
R. SC
#NWNNG
F
F
F
F
H
R
Zi
Embedding
Figure 3: (a) The overall architecture of the proposed STCFormer. (b) and (c) show the structure of the temporal cross-attention (TCA) and exclusive spatial attention (ESA) modules, respectively, which contain the computational flow and the major dimensional transformations of features.
3.4 Flow-guided Delay Compensation
We present a flow-guide now-guided delay compensation (FDC) strategy to eliminate the two-sided fusion error in feature-level collaboration due to temporal asynchrony. Existing solutions relied on received historical features [13] and produced large errors under severe delay [53], leading performance bottlenecks. To tackle the issues, we adopt the philosophy of feature flow to predict the collaborators' future features for temporal alignment with the ego representation. The details are as follows.
Flow Generation and Warping. Due to the uncertain delay between the ego agent and collaborators, FDC predicts the feature flow J0 for fixed time interval and scale matrix Sh based on the j-th agent's historical frames. Specifically, Figure shows, features {F ..... F are concatenated in channel dimension and entered to a generator (-) to output O¹ R #xxx x2 and S R xH Then the j-th agent sends {O S with prediction ability and sparse feature F to the ego agent. The ego agent estimates predicted collaborator features as Z = fwarp F
to)
Ote @ Sho where Swarp :) is the bilinear warping function applied all positions and channels [60, 61). and is the scalar multiplication. The temporally aligned features are passed to the STCFormer.
Self-supervised Training Pattern. Self-supervised learning is employed to train the flow generator fro (-) since the existing datasets [40, 41, 52] lack the motion annotations. Concretely, we first form -
the training group {F ..... F } where (F :... F is continuous feature sequence,
and F is considered as the ground truth feature. Subsequently, predict the feature Z as Z [warp (t to) O000 s S Since the optimization objective of flow (-) is increase the similarity between F and Zj, we formulate the self-supervised loss function L flow based on the cosine similarity [53] as follows:
F
Z
flone N
Σ
(1
(5)
1 12
Je(l...,NN.jzii
where |1 is the squared Frobenius norm.
3.5 Spatio-Temporal Collaboration Transformer
To efficiently mitigate collaboration heterogeneity, we propose a spatio-temporal collaboration trans- former (STCFormer) jointly integrate the decoupled spatial semanties and temporal dynamics among agents. From Figure 3(a), the core contributions of STCFormer lie the following three cus- tomized modules, where the other basic components follow the choice of the vanilla transformer [30].
Temporal Cross-Attention. To bridge the detection gap regarding fast-moving objects due to point cloud sparsity, we capture historical context clues across agents to reinforce the current
5

---

representation via temporal cross cross-attention (TCA) module. The core is to perform Query-Key- Value-like attention operations by projecting the ego feature F and merged temporal features E [F F. ...., F into different subspaces via three 2D convolutional layers Ws-a(-). In Figure 3(b), the branches of K Value Ex/D + wi(ga (W((a(W))(()) @ W3.3 (E) share the same structure but separate weights, where a ID temporal convolution wi w1(+) with global average pooling (Pa()) provides temporal correlations. (-)-) is applied to the spatial dimension to shrink the feature map. The computation of TSA can be shown as:
H w3.3 (sffmmoofta
Ex) BXXXX
(6)
Decoupled Spatial Attention. To comprehensively integrate the distinct spatial semantics from the collaborators, we facilitate pragmatic message fusion with a feature decoupling perspective inspired by the observation in Figure ((d). Formally, an importance generator fgen (-) is employed to generate the importance maps of the ego feature F/ and the estimated collaborator feature Z / (J)) E [),1,xxW The importance maps reflect the perceptually critical level of each pixel in the features. Then, the j-th agent spatially decouples the feature Zj via candidate maps E = (1 IG o Ij and C = I @ Ij. Intuitively, E and с₁ depict the collaborators' exclusive and common perception regions relative to the ego agent, respectively. The exclusive and common collaborator features are obtained as jj.- (E)) Zj and Z (C)) @ Zj.
as I = a (fpm (Joen
Then, we present two spatial attention modules based on deformable cross-attention [59] to aggregate the decoupled exclusive and common features, which share the same structure but different weights. Here the exclusive spatial attention (ESA) is taken as an example (see Figure 3(c)), and its input comprises H and Z An importance-aware query initialization is first designed to guide ESA to focus on the potential foreground objects. Specifically, we obtain the element- wise summation of the importance maps as I' Σラ I and extract N₂ target queries from the salient locations in Z', The attention scores are learned from the initial queries via a linear layer and the softmax function. Subsequently, linear layer learns offset map for each input feature, providing the 2D spatial offset (Age S S N,} for each query q. We sample the keypoints based on the learned offset maps and extract these keypoints' features to form the attending feature. The cross-attention layer aggregates multiple collaborators' features to output the enhanced feature for each query g as:
N
ESA(q) = Σ W
(W) Aq.]l,
==1
where u indexs the attention head, and Wu/f denotes the learnable parameters. Then, the filling operation fills ESA(q) into H based on the initial positions of the queries and outputs F Similarly, the enhanced common feature Ft_ is obtained via the common spatial attention (CSA).
Adaptive Late Fusion. The adaptive late fusion (ALF) module is presented to effectively fuse the exclusive and common representations {Fi Ft } for incorporating their perceptual advantages. Formally, we obtain two weight maps as Gi = Win1 ( eF Ff,.), and apply the softmax function to produce the normalized weight maps as //,.//Gt = coftmax(G,, ltt.) The learned G and G reflect the complementary perception contributions of (F Ft. at each spatial location. Therefore, we adaptively activate the perceptually critical information of each representation by a weighted summation. The refined feature map is obtained as F = G f.c @ F.. + G Fi.-
s,e/
3.6 Detection Decoders and Objective Optimization
Two detection decoders (ffāee(). f)) are employed convert the output fused representation Ft into the prediction results. The regression result represents the position, size, and yaw angle of the
predefined box at each location, which y!! (F) E "xxx7 The classification result is
(i(!) = (ā{c E Hx x2, revealing the confidence value of each bounding box to be an object. For objective optimization, we leverage the smooth absolute error loss for regression (denoted as Lregy) and the focal loss [15] for classification (denoted as Lota). In total, we formulate the overall objective function as follows: Lall Lreg + Leta + Lmas + L

---

AP0.7 cn DAIR-V2 XXX
APD V2X5et
65
72.5
66
65.0
50
62.5
36
55.0
12
14
16
16
12
14
13
20
12
14
13
20
Communication Volume (log2)
Communication Volume (log2)
Communication Volume (log2)
Wrandccom
CuBEVT
WoenZcom
Figure 4: Collaborative perception performance comparison of How2comm and Where2comm [7]
the DAIR-V2X, V2XSet, and OPV2V datasets with varying communication volumes.
AP0.7 on DAIR-V2X
30
70
APO on OPV2V
45
56
40
APD
16
0.
0.3 Std (m)
Error (m)
Lacolization
(m)
How2comm (Durs)
Wherobcomm
Discohet
...-
VIX
V2VNet
WhenZcom
No Fuston
Figure 5: Robustness to the localization error on the DAIR-V2N V2XSet, and OPV2V datasets.
4
Experiments
4.1 Datasets and Implementation Details
Multi-Agent 3D Detection Datasets. To evaluate the performance of How2comm on the collabora- tive perception task, we conduct extensive experiments on three multi-agent datasets, including DAIR- V2X [52]. V2XSet [40], and OPV2V [41]. DAIR-V2X [52] is a real-world Phticle-toinffasttrece perception dataset containing 100 realistic scenarios and 18,000 data samples. Each sample collects the labeled LiDAR point clouds of vehicle and an infrastructure. The training/validationeessi sets are split in ratio of 5:2:3. V2XSet [40] is a simulated dataset supporting V2X perception, co-simulated by Carla [3] and OpenCDA [36]. It includes 73 representative scenes with 2 to 5 connected agents and 11 11,447 3D annotated LiDAR point cloud frames. The araining/validaaiinnnesttgg sets are 6,694, 1,920, and 2,833 frames, respectively. OPV2V [41] is a large-scale simulated dataset for multi-agent V2V perception, comprising 10,914 LiDAR point cloud frames with 3D annotation. The suininn/validaaion//esingg splits include 6,764, 1,981, and 2,169 framess respectively.
Evaluation Metrics. We adopt the Average Precision (AP) Intersection-over-UU Union (IoU) thresh- olds of 0.5 and 0.7 to evaluate the 3D object detection performance. Also, the calculation format of communicatin volume in [7] used count the message size by byte in the log scale with base
4.2 Quantitative Evaluation
Experimental Settings. We build all the mod- els using the Pytorch toolbox [18] and train them Tesla V100 GPUs with the Adam opti- mizer [10]. The learning rate is set to 2e-3 and decays exponentially by 0.1 every 15 epochs. The training settings on the DAIR-V2X [52], V2XSet [40], and OPV2V [41] datasets include: the training epochs are (30, 40, 40), and batch sizes are (2, 1, I). The height and width lution of the feature encoder (.) is 0.4 m. The selection function (-) has threshold of 0.01. The flow generator (-) leverages the multi-scale backbone to extract multi-grainet representations and an extra encoder to produce Oh and Sho. We implement the importance gen- erator fgen() (-) and statistical network (.).) with the classification decoder in [11] and following [27],
Table 1: Performance comparison the DAIR-
V2X [52], V2XSet [40], and OPV2V [41] datasets. The results are reported in A.50.5.
DAIR- V2X
V2XSet
OPV2V
Models
AF.U.
AF@U.300.
AF@0.300
No Fusion
50.03/43.57
60.60/40.20
68.71/48.66 1164
Late Fusion
48.93/34.06
54.92/30.75
79.62/59.46
When2com [16]
46.64/32.49
65.06/41.88
70.64/53.73
F-Cooper
49.77/35.21
71.48/46.92
75.27/63.05
AttFuse [41]
50.86/38.30
70.85/48.66
1/.14/64.52
V2VNet [33]
52.18/38.6 62
79.09/49.25 25
77.45/62.1 10
DiscoNet [14]
51.44/40.01
79.83/54.06
81.08/64.85
V2X-VIT [40]
51.68/39.97
83.64/61.41
80.61/66.42
CoBEVT [38]
56.08/41.4
81.07/54.33
81.59/67.55
WhereZcomm [7]
59.34/43.53
82.02/53.38
82.75/67.27
How2comm (ours)
62.36/47.18
84.05/67.01
85.42/72.24
respectively. The keypoint number N., 9, and the attention head is 8. Two 1 X convolutional layers
7

---

AP0.7 on DAIR-V2X
75
50-
65
46
$22
65
40
%6
0.2
0.4
0.8
0.3
1.0
0.0
0.2
04
0.8
0.6
0.0
022
0.4
0.6
0.8
1.0
Heading Error Std
Heading Emor Std (*)
Heading
Emror Sid
Hoa&com (Gurs)
Discoliel
VIX-IT
Wosr/.com
Figure 6: Robustness to the heading error the DAIR- -2X, v2XSet, and OPV2V datasets.
APOT on DAIR V2X
APO on OPV2V
50
60
46
on
50
40
%6
30
0.0
0.1
622
0.3
0.4
0.0
0.1
02
6.3
0.4
0.0
0.1
0.3
0.4
Transmission Dealy (8)
Transmiasion Dealy (8)
Transmission Dealy (8)
Hon2Coom (Dura)
Wher #ccomm
Diecohiw
V2X-NT
VIVNet
Whwrdramm
No Fusion
Figure 7: Robustness to the transmission delay on the PAIR-V2X, V2XSet, and OPV2V datasets.
are utilized to implement the detection decoders. Under the default noise settings, the transmission delay 7 is set to 100 ms, and the localization and heading errors of the collaborators are sampled from a Gaussian distribution with standard deviations of 0.2 m and 0.2%, respectively. All experiments are constrained to x 1 MB bandwidth consumption to reflect the narrow communication channels in real V2V/X scenarios [20].
Detection Performance Comparison. Table 1 compares the detection performance of the proposed How2comm with various models on three datasets under default noise settings. We consider two typical baselines. No Fusion is a single-agent perception pattern that only uses the local observations. Late Fusion integrates predicted boxes across agents and produces results with non-maximum sup- pression. Moreover, the existing SOTAs are comprehensively considered, including When2com [16], F-Cooper [1], AttFuse [41], V2VNet [33], DiscoNet [14], V2X-ViT [40], CoBEVT [38], and Where2comm [7]. Intuitively, How2comm outperforms previous methods in the real-world (DAIR- V2X [52]) and simulated datasets, demonstrating the superiority of our model and its robustness to various realistic noises. In particular, the SOTA performance of AP@0.7 on the DAIR- V2X and OPV2V is improved by 8.4% and 7.0%, respectively. Compared to previous per-agent/location message fusion efforts [14, 16, 33, 40], How2comm simultaneously considers the decoupled spatial semantics and temporal dynamics among agents, resulting more precise perception.
Comparison of Communication Volume. Figure 4 presents the performance comparison results with distinct bandwidth consumptions. Coneretely, the orange and blue curves denote the detection precision of our How2comm and Where2comm under varying communication volumes, respec- tively. (i) How2comm keeps superior Where2comm across all the communication choices, i.e., How2comm achieves a better performance- bandwidth trade-off with -patial-channel filtering than Where2comm. (ii) Moreover, our framework seeks comparable performances as the SOTAs by con- suming less bandwidth. The noteworthy improvements demonstrate that the proposed communication mechanism filters invalid semantics and maintains performance by mutual information maximization.
Robustness to Localization and Heading Errors. We verify the detection performance of How2comm under varying pose errors of collaborators in Figures and following the noise settings in [40]. Specifically, the localization and heading errors are sampled from Gaussian distri- butions with a standard deviation of a Ozyz E [0, 0.5) m and [0 1.0%], respectively. As shown in the figures, the performance of all intermediate collaboration models consistently deteriorates due to feature map misalignment as the pose errors increase. Noticeably, How2comm is superior to the previous SOTA models and No Fusion across three datasets under all error levels, while some models (e.g., V2VNet and When2com) are even weaker than No Fusion when the error exceeds 0.2 m and 0.2" on the DAIR-V2X dataset. This comparison demonstrates the robustness of How2comm against collaboration pose noises. One reasonable explanation is that our framework captures perceptually critical and holistic information across heterogeneous agents via the tailored STCFormer.
8

---

Robustness to Transmission Delay. As noted in Figure 1(b), temporal asynchrony due to transmis- sion delay results in two-sided fusion errors and harms the collaboration procedure. We analyze the sensitivity of existing models to varying delays (i.e., from 0 to 400 ms) in Figure 7. Noticeably, all the intermediate fusion methods inevitably degrade with increasing transmission delay due to misleading feature matching. Nevertheless, How2comm maintains higher precision than SOTAs at all delay levels across all three datasets and improves A.70.7 by 7.5% than No Fusion on DAIR- -2X under a severe delay (400 ms). This robustness to the transmission delay proves that How2comm accomplishes temporal alignment of two-sided features by predicting the collaborators' future features.
4.3 Ablation Studies
We perform thorough ablation studies on all datasets to understand the necessity of the different designs and strategies in How2comm. Table 2 shows the following vital observations.
Rationality of Communication Mechanism. (i) The spatial and channel attention queries are first removed separately to perform incom- plete message filtering. The decreased perfor- mance implies that both query patterns con- tribute to sharing sparse yet salient features among agents. (ii) There is a significant degra- dation in the detection results on all datasets when the communication mechanism lacks mu- tual information supervision. A plausible de- duction that our supervision mitigates the
Table 2: Ablation study results of the proposed components and strategies on all the datasets. "w/o" stands for the without.
Components/trategies
DAIR-V2X
VIXSet
OPV2V
AP APGU.50.7
Full Framework
62.36/47.18
84.05/67.01 85,42/72.24
Rationality of Comminication Mechanism
wlo Spatial Query
61.25/46.33 83.14/66.15 84.52/1
web Channel Query
61.76/46.52 83.7066.55 85.23/71.76
w/o Mutual Information
60.61/46.04 82.53/65775 84.06/70.69
Effect of Delay Compensation Strategy
loss of valuable information due to spatial- channel feature filtering.
w/o Scale Matrix
62. 02.17/46.6 83.61/66.63 85/08/71 54
Self-supervised Training 60.53/45.77 82.84/65.78 84.20/70.36
Importance of STC.oreer
Effect of Delay Compensation Strategy. (i) Here, we remove the scale matrix to verify its
wra Temporal Cross-Allention 60.15/45 92 83.09/65 88 84,27/70.85
w/o Exclusive Spatial Altention 59.46/45 .06 82.65/65.0 70 83,81/71.19
who Commnn Sputial Attention 61 61.24/46.63 83.48/66.41 84.39/71 .44
effect. The poor results show that appropriate scaling of predicted features promotes effective temporal alignment. (ii) Furthermore, the per- formance drop caused by the self-supervised training removal shows the importance of im-
w/o Adaptive Late Fusion 61.93/46.77 83.72/66.69 84,86/71.64
Impact of Keypoint Number N.
N.
61.68/46.00 05 82 82/66.16 84.67/71.8 .38
N, 9 (Default)
62.36/47 18 84.05/67.0 00 85.42/72.24
N. =
13/46.77 84.12/66.59 85.28/72.01
posing motion representation supervision for
Necessity Decoupled Design
collaborator-share features.
w/o Decoupled Design
60.08/45.36 36 82.41/65 38 84:02/71.55
Importance of STCFormer. STCFormer is evaluated in three dimensions. (i) We first find that temporal cross-attention provides beneficial gains due to performance deterioration when discarded. It is because the meaningful temporal clues in historical frames bridge the single-frame detection gap. (ii) Then, exclusive and common spatial attention modules are removed separately to explore the impact on performance. The consistently decreased results on each dataset suggest that integrating distinct spatial semantics is indispensable for pragmatic collaboration. (iii) Finally, the adaptive late fusion is replaced by pixel-wise addition. The gain decline suggests that our fusion paradigm provides new insights for aggregating the perceptual advantages of distinct spatial semantics.
Impact of Keypoint Number N,.. Empirically, we set the variable number of keypoints for the decoupled spatial attention modules to perform experiments and find that 9 keypoints achieve the most competitive detection performance. Conversely, too few keypoints may cause valuable semantics loss and too many keypoints may cause performance bottlenecks due to accumulated errors. The above finding inspires us to determine the appropriate keypoint number that samples more rich visual clues and captures more pragmatic collaboration information from collaborators.
Necessity of Decoupled Design. Ultimately, we set E and C to while maintaining two spatial attention branches in the STCFormer to justify the decoupled design. Without explicitly defining exclusive and common features, spatial attention would indiscriminately sample the entire region due to the lack of guidance from prior perception information, which may introduce excessive collaborator noises and impede bridging to collaboration heterogeneity.
4.4 Qualitative Evaluation
Visualization of Detection Results. To illustrate the perception performance of different models, Figure 8 shows the detection visualizations of two challenging scenarios from the DAIR-V2X

---

000
007
(a) V2X-VI
(b) CoBEVT
(c) Where2oomm
(d) How2comm (ours)
Figure 8: Detection visualization comparison real-world scenarios from the DAIR-V22 dataset. Green and red boxes denote the ground truths and detection results, respectively.
(a) GT bounding boxes
(b) Exclusive spatial attention
(c) Common spatial attention
Figure 9: Visualization of learned exclusive and common spatial attention maps. The red and blue point clouds are derived from the ego vehicle and infrastructure, respectively. In (b)&(c), cyan cross marker denotes the target query in the deformable cross-attention. The 9 sampled keypoints represented by the forestgreen dots whose colors reflect the corresponding attention weights.
dataset [52] under default noise settings. How2comm axiomatically achieves more robust and accurate detection compared previous SOTA models, including V2X-VIT [40], CoBEVT [38], and Where2comm [7]. Concretely, our method produces more predicted bounding boxes well aligned with the ground truths. The merits may lie in two aspects. (i) Our delay compensation strategy mitigates feature misalignment due to temporal asynchrony. improving detection performance. (fi) The proposed STCFormer provides effective temporal context clues for detecting fast-moving objects and fuses meaningful information from nearby agents to compensate for the occluded perspective.
Visualization of Spatial Attention Maps. We show visualizations of exclusive and common spatial attention (ESA/CSA) A) Figure to justify the effectiveness of our feature decoupling philosophy. (i) Intuitively, from Figure 9b), ESA effectively samples keypoints at the exclusive perception region from the infrastructure, providing complementary information for the ego agent to promote perception ability. (ii) In Figure (c). CSA mitigates the detection gap due collaboration heterogeneity by sampling keypoints that reasonably focus on the common perception region between two agents.
5 Conclusion and Limitation
This paper presents How2comm, novel collaborative perception framework tackle existing issues jointly. How2comm maximizes the beneficial semantics in filtered features and accomplishes temporal aligmment via the feature flow estimation. Moreover, our STCFormer holistically aggregates the spatial semantics and temporal dynamics among agents. Extensive experiments on several multi-agent datasets show the effectiveness of How2comm and the necessity of all its components.
Limitation and Future Work. The current work only exploits the short-term historical frames. In future work, plan to expand the utilization of temporal information to long-term point cloud sequences. Also, we will explore optimizing the feature flow prediction with uncertainty estimation.
Acknowledgment. This work is supported in part by the National Key R&D Program of China under Grant 20212D01 13503 and in part by the Shanghai Municipal Science and Technology Major Project under Grant 002ISH2DZX0103.
10

---

References
[1] Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pages 88-100, 2019. 7,
[2] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning (ICML), pages 1538-1546. PMLR, 2019. 3
[3] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning, pages 1-16. PMLR, 2017.
[4] Jakob Foerster, loannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016. 3
[5] R Devon Hjelm, Alex Fedorov, Samuel Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: /808 06670, 2018. 4
Karan Grewal, Phil Bachman,
[6] Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. Advances in Neural Informa-
fion Processing Systems (NeurIPS), 30, 2017. 3
[7] Yue Hu, Shaoheng Fang. Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: communicationneeffccen collaborative perception via spatial confidence maps. Advances in Neural Information Processing Systems (NeurIPS), 35:4874-48866 2022. 1, 2, 4, 7, 10
[8] Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages
2023. 2,4
[9] Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent coopera- tion. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[10] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015. 7
[11] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12697- 12705, 2019. 4,7
[12] Yuxuan Lei, Dingkang Yang, Mingcheng Li, Shunli Wang, Jiawei Chen, and Lihua Zhang. Text- oriented modality reinforcement network for multimodal sentiment analysis from unaligned multimodal sequences. arXiv preprint arXiv:2307. 13205 2023.
[13] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency- collaborative perception. Proceedings of the European Conference on Computer Vision (ECCV), pages 316-332. Springer, 2022. 2.5
[14] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems (NeurIPS), 34:29541-29552 2021. 1, 2, 3, 7.8
[15] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 2980-2988, 2017. 6
[16] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4106-4115, 2020. 2,
11

---

[17] Guiyang Luo, Hui Zhang, Quan Yuan, and Jinglin Li. Complementarity-enaane and redundancy-minin minimized collaboration network for multi-agent perception. In Proceedings of the ACM International Conference on Multimedia (ACM MM), pages 3578-3586, 2022. 1
[18] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019. 7
[19] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 652-660, 2017. 1
[20] Hang Qiu, Pohan Huang. Namo Asavisanu, Xiaochen Liu, Konstantinos Psounis, and Ramesh Govindan. Autocast: Scalable infrastructurr--ees cooperative perception for distributed collabo- rative driving. 2022. 8
[21] Linhao Qu, Shaolei Liu, Manning Wang. and Zhijian Song. Transmef: A transformer- multi-exposure image fusion framework using self supervied multi-task learning. In Proceed- ings of the AAAI Conference on Artificial Intelligence, pages 2126 2264,, 2022. 3
[22] Linhao Qu, Siyu Liu, Xiaoyu Liu, Manning Wang, and Zhijian Song. Towards label-efficient automatic diagnosis and analysis: comprehensive survey of advanced deep learning-basd based weakly-supervised, supervipe and self supervised techniques in histopathological image analysis. Physics in Medicine & Biology, 2022. 3
[23] Linhao Qu, Xiaoyuan Luo, Shaolei Liu, Manning Wang, and Zhijian Song. Dgmil: Distribution guided multiple instance learning for whole slide image classification. In Medical Image Computing and Computer-Asiisted Intervention (MICAAI), pages 24-34. Springer, 2022. 3
[24] Linhao Qu, Manning Wang, Zhijian Song, et al. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Advances in Neural Information Processing Systems (NeurIPS), 35:15368-155 15381, 2022. 3
[25] Tika Ram and Khem Chand. Effect of drivers' risk perception and perception of driving tasks on road safety attitude. Transportation Research Part F: Traffic Psychology and Behaviour, 42:162 176, 2016.
[26] Joseph Redmon, Santosh Divvala, Ross Girshick. and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 779-788, 2016.
[27] Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled repre- sentations via mutual information estimation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 205-221. Springer 2020.
[28] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks, arXiv preprint arXiv:1812.09755, 2018. 3
[29] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropa- gation. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016. 3
[30] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017. 5
[31] Binglu Wang, Lei Zhang, Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Cooper- ative reconstruction for multi-agent perception. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 8710-8720, October 2023. 1
[32] Tianhang Wang, Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. Ume: A unified bandwidth-efffiient and multi-resolution based collaborative perception framework. In Proceedings of the IEFFF/CVF International Conference on Computer Vision (ICCV), pages 8187-8196 October 2023. 1.2
12

---

[33] Suun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 605-621, Springer,
2020. 1, 3, 7,8
[34] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV),
pages 3- 2018. 4
[35] Pengxiang Wu, Siheng Chen, and Dimitris N Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based on bird's eye view maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11385-11395, 2020. 4
[36] Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Openeda: an open coopera- tive driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 1155-1162. IEEE, 2021. 7
[37] Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma. Bridging the domain gap for multi-agent perception. In IEEE International Conference on Robotics and Automation (ICRA), pages 6035-6042. IEFE, 2023. 2,
[38] Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. In Conference on Robot Learning (CoRL), 2022. 3, 8, 10
[39] Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang. Xiaoyu Dong, Rui Song, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13712-13722, 2023.
[40] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang. and Jiaqi Ma. V2x-vit: Vehiele-to-everythin cooperative perception with vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV), pages 107-124. Springer, 2022. 1, 2, 3, 4, 7, 8, 10
[41] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In Proceedings of the International Conference Robotics and Automation (ICRA), pages 2583-2589. IEEE, 2022. 2, 5, 7,8
[42] Dingkang Yang, Zhaoyu Chen, Yuzheng Wang, Shunli Wang, Mingcheng Li, Siao Liu, Xiao Zhao, Shuai Huang, Zhiyan Dong, Peng Zhai, and Lihua Zhang, Context de-confounded emotion recognition. Proceedings of the IEFE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), pages 19005-19015, June 2023. 3
[43] Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), page 1642-1651, 2022. 3
[44] Dingkang Yang, Shuai Huang, Yang Liu, and Lihua Zhang. Contextual and cross-modal interaction for multi-modal speech emotion recognition. IEEE Signal Processing Letters, 29:2093-2097, 2022. 3
[45] Dingkang Yang, Shuai Huang, Shunli Wang, Yang Liu, Peng Zhai, Liuzhen Su, Mingcheng Li, and Lihua Zhang. Emotion recognition for multiple context awareness. In Proceedings of the European Conference on Computer Vision (ECCV), volume 13697, pages 144-162. Springer, 2022. 3
[46] Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, Yan Wang, Jing Liu, Peixuan Zhang, Peng Zhai, and Lihua Zhang. Aide: A vision-driven milti-view, multi-modal, multi-tasking dataset for assistive driving perception. In Proceedings of the IEFF/CVF International Conference on Computer Vision (ICCV), pages 20459-20470, October 2023. 1
13

---

[47] Dingkang Yang. Haopeng Kuang. Shuai Huang, and Lihua Zhang. Learning moddlity-speciine and -agnostic representations for asynchronous multimodal language sequences. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), page
2022.
3
[48] Dingkang Yang, Yang Liu, Can Huang. Mingeheng Li, Xiao Zhao, Yuzheng Wang, Kun Yang. Yan Wang, Peng Zhai, and Lihua Zhang. Target and source modality co-reinforcement for emotion understanding from asynchronous multimodal sequences. Knowledge- Basedd Systems, page 110370, 2023. 3
[49] Kun Yang, Jing Liu, Dingkang Yang, Hanqi Wang. Peng Sun, Yanni Zhang, Yan Liu, and Liang Song. A novel efficient multi-view traffic-related object detection framework. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, 2023.
[50] Kun Yang, Peng Sun, Jieyu Lin, Azzedine Boukerche, and Liang Song. A novel distributed task scheduling framework for supporting vehicular edge intelligence. In IEEE International Conference on Distributed Computing Systems (ICDCS), pages 972-988,222 2022.
[51] Kun Yang. Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang. Peng Sun, and Liang Song. Spatio-temporal domain awareness for multi-agent collaborative perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 23383-23392, October 2023. 1
[52] Haibao Yu. Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li. Xing Hu, Jirui Yuan, al. Dair-v2x: A large-scale dataset for shhicle-ifraastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 21361-21370, 2022. 2, 7,
[53] Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Jirui Yuan, Ping Luo, and Zaiqing Nie. vehicle-iffrastruetur cooperative 3d object detection via feature flow prediction. arXiv preprint arXiv:2303. 10552, 2023. 2,5
[54] Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle-infrratructuuee cooperative perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5486-5495, 2023. 1
[55] Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 11141-11150, 2021.
[56] Zhenxun Yuan, Xiao Song, Lei Bai, Zhe Wang, and Wanli Ouyang. Temporal-channel trans- former for 3d lidar-based video object detection for autonomous driving. IEEE Transactions on Circuits and Systems for Video Technology, 32(4):2068-2078, 2021.
[57] Zixu Zhang and Jaime F Fisac. Safe wwwlluioon- autonomous driving via same-theoretic active perception. arXiv preprint arXiv:2/05.08//6, 2021. 1
[58] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2881-2890, 2017. 1
[59] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2021. 6
[60] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 408-417, 2017. 5
[61] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2017.
14


Reviews:
[{"review_id": "3qpJIWMbmb", "reviewer": "Reviewer_fcX9", "summary": "The paper introduces three key contributions in tackling the aforementioned challenges. Firstly, the authors devise a mutual information-aware communication mechanism that maximizes the sharing of informative features among collaborators. They employ spatial-channel filtering for efficient feature sparsification and effective communication. Secondly, a flow-guided delay compensation strategy is presented to predict future characteristics from collaborators, addressing feature misalignment caused by temporal asynchrony. Lastly, a pragmatic collaboration transformer is introduced to integrate spatial semantics and temporal context clues, enhancing the overall perception capabilities among agents. The proposed framework, How2comm, is thoroughly evaluated on three LiDAR-based collaborative detection datasets in both real-world and simulated scenarios. The experiments conducted are comprehensive and demonstrate the superiority of How2comm over existing methods. The effectiveness of all vital components within the framework is also validated, providing solid evidence for the contributions made.", "questions": "The authors are suggested to provide a video demonstration of the results.", "limitations": "N.A.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel Approach: How2comm addresses the challenges faced in multi-agent collaborative perception by introducing innovative techniques, such as mutual information-aware communication, flow-guided delay compensation, and pragmatic collaboration transformer.\n\n2. Comprehensive Evaluation: The framework is rigorously evaluated on various datasets, including real-world and simulated scenarios, establishing its superiority over existing methods.\n\n3. Clear Presentation: The paper effectively explains the proposed framework and its components, making it accessible to readers with varying levels of expertise.", "weaknesses": "Here are some suggestions: \n\n1. It would be beneficial for the authors to discuss any limitations or potential areas for improvement in the How2comm framework, providing avenues for future research.\n\n2. The authors are suggested to add some missing literature to be more comprehensive.\n\n[1] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S. and Feng, C., 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4), pp.10914-10921.\n\n[2] Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J. and Nie, Z., 2022. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21361-21370).\n\n[3] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R. and Yu, H., 2023. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13712-13722).\n\n[4] Yu, H., Yang, W., Ruan, H., Yang, Z., Tang, Y., Gao, X., Hao, X., Shi, Y., Pan, Y., Sun, N. and Song, J., 2023. V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5486-5495).\n\n[5] Li, Y., Zhang, J., Ma, D., Wang, Y. and Feng, C., 2023, March. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning (pp. 2062-2072). PMLR.\n\n[6] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023. Uncertainty quantification of collaborative detection for self-driving. ICRA.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our contribution of novel approach, comprehensive evaluation, and clear presentation. We present our responses below.\n\n**Q1**: Discussion of  any limitations or potential areas for improvement.\n\n**A1**: 1. Thanks for your constructive comments! We have discussed the limitations and future work of the proposed framework in lines 340-342 of the main manuscript. Concretely, \n*  The current work only exploits the short-term historical frames. In future work, we plan to expand the utilization of temporal information to long-term point cloud sequences. \n*  We will explore optimizing the feature flow prediction with uncertainty estimation.\n2. Moreover, we have discussed the **Broader Impacts** of our How2comm in Section 5 (lines 356-367) of the **Supplementary Material**. Concretely,\n*  The positive impact of this work is that How2comm can leverage vehicular communication\ntechnology to facilitate information sharing and enhance the vehicles' perception ability of the\nsurrounding environment, reducing the incidence of traffic accidents.\n *  The collaborative perception ability provided by How2comm can be used to monitor real-time traffic conditions and improve traffic efficiency.\n*  The illegal personnel may invade the vehicle communication channel to attack the collaborative perception system, leading to the vehicles' misjudgment of the surrounding environment or malicious remote control.\n*  How2comm facilitates Vehicle-to-Vehicle/Everything (V2V/X)-oriented multi-agent communication applications in autonomous driving.\n3. Further, we have discussed other **Realistic Limitations** and indicated future research directions to optimize our system in Section 6 (lines 368-387) of the **Supplementary Material**. Concretely,\n*  For the attack issue, How2comm could improve the robustness against potential adversarial attacks by introducing an adversarial training paradigm. In addition, by focusing on decoupled spatial regions and discovering perceptually critical information, How2comm is relatively less likely to be attacked.\n*  For the data bias issue, How2comm has demonstrated its preliminary generalization on real-world and simulated datasets with different configurations and sensor facilities. In future work, How2comm could disentangle causal relationships among variables in the collaborative perception pipeline with the assistance of causal inference techniques, further enhancing its ability to generalize to more complex real-world scenarios.\n*  For the domain gap issue, How2comm could hopefully achieve sim2real adaptation and inter-agent adaptation by introducing an unsupervised sim2real domain bridging mechanism.\n***\n**Q2**: Addition of missing literature.\n\n**A2**: Many thanks for the insightful suggestions! We promise to add all the literature you listed in the revision for a more comprehensive overview and presentation.\n***\n**Q3**: Provision of a video demonstration of the results.\n\n**A3**:  Valuable proposal! In our submitted **Supplementary Material**, we have provided a video demonstration of an intuitive performance comparison between How2comm and the previous SOTA models. You can freely browse the *video.mp4* file. The video demonstration provides the following observations. \n\nHow2comm has fewer regression misalignments and fewer undetected objects, resulting in a more robust performance. In comparison, the previous methods fail to tackle fast-moving objects and perceive remotely occluded regions. The indisputable evidence is that some ground truth bounding boxes are not well matched with their detection results."}, {"review_id": "4NiaR7EcKw", "reviewer": "Reviewer_aK4W", "summary": "This paper proposes How2comm, a collaborative perception framework, which seeks a trade-off between perception performance and communication bandwidth. This framework leverages a mutual information-aware communication to select the most informative messages to transmit and save bandwidth, where ego agents compute and send spatial and channel queries to neighbor agents. And a flow-guided delay compensation is applied to reduce temporal alignment. This paper designs Spatio-Temporal Collaboration Transformer to generate comprehensive features from spatial and temporal information. Experiments are conducted on three datasets Dair-V2X, V2XSet and OPV2V to validate the proposed methods.", "questions": "1. What is the meaning or definition of Collaboration Heterogeneity mentioned in section 1? What are the difficulties of fusing the information from different agents?\n2. In Spatial-channel Message Filtering of section3.3, which dimension is the pooling functions applied to? The authors need to clarify it as well as all pooling functions in this paper.\n3. Since the goal is to improve the perception performance of the ego agent with as little communication cost as possible, why maximizing the mutual information between the compressed and the vanilla feature  is necessary? If some information is useless to ego agent, it is unnecessary to keep them.\n4. Does the communication cost include the queries and the indices of non-zero pixels? The authors need to clarify it to ensure a fair comparison.", "limitations": "The paper integrates a good collaboration perception system by overlaying various more or less known techniques. The core technical novelty is not clear. The experiments are also insufficient.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This work considers the key problems in collaborative perception, including communication cost, transmission delay, localization error, designs a proper pipeline of collaboration and information processing  and get superior experiment results.\n2. The logic and structure of the paper is clear.\n3. The review and analysis of the field of study is comprehensive.", "weaknesses": "1. Many technical contributions claimed in this paper has been proposed in previous works, including requesting complementary information from neighbor agents to decrease communication costs [1], compensating latency with historical information [2] and fusing features with transformer [3]. This work appears to involve straightforward modifications of previous techniques and a system ensemble. So what is the essential technical novelty of the proposed methods? The authors need to clarify this point.\n\n2. The main experiment results in Table.1 are significantly lower and inconsistent with previous work[4,5], which use the same Pointpillars backbone. The results seem not convincing.\n\n3. The proposed framework includes more networks (transformer-based network), computations (importance map), and data (historical frames) than general collaborative perception methods, which may increase training costs and inference time. However, the performance seems not significant. There is a lack of evaluations of training cost, inference time and a trade-off between them and perception performance. \n\n4. OPV2V and V2XSet are very similar datasets, the experiments should be conducted on datasets with different characteristics, like the V2X dataset V2X-Sim[5], the real-world V2V dataset V2V4real[6].\n\n5. It lacks comprehensive comparison with previous methods that handle transmission delay. Previous work, like SyncNet [2] and V2X-ViT [3], has proposed to compensate for the latency with historical information. What are the advantages of the proposed flow-guided delay compensation?\n\n[1] Hu Y, Fang S, Lei Z, et al. Where2comm: Communication-efficient collaborative perception via spatial confidence maps, Advances in neural information processing systems 2022.\n[2] Lei Z, Ren S, Hu Y, et al. Latency-aware collaborative perception, European Conference on Computer Vision 2022.\n[3] Xu R, Xiang H, Tu Z, Xia X, Yang M-H, Ma J, V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer, European Conference on Computer Vision 2022.\n[4] Liu Y C, Tian J, Glaser N, et al. When2com: Multi-agent perception via communication graph grouping, Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2020.\n[5]  Li Y, Ma D, An Z, et al. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving, IEEE Robotics and Automation Letters, 2022.\n[6] Xu R, Xia X, Li J, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Insightful comments! Below are some specific responses. The common questions can be found in the global response.\n\n**Q1**: About technical novelties.\n\n**A1**: Our three contributions fundamentally differ from previous works regarding design philosophy and structure modeling. We have highlighted contributions in lines 53-69 of Main Paper and Sec.1 of Supplementary. \n1. Mutual information-aware communication (MIC)\n* Philosophy: Where2comm focuses on spatial sparsification without considering the redundant information in the channel dimension. In contrast, our MIC considers spatial-channel sparsification and the preservation of local critical semantics.\n* Structure: Where2comm is built on spatial confidence maps to perform spatial sparse communication. Our MIC performs space-channel sparse communication via customized attention queries. Also, MIC introduces mutual information maximization supervision to mitigate the loss of locally valuable semantics, which has been neglected in previous works.\n2.  Flow-guided delay compensation (FDC)\n* Philosophy: SyncNet aims to predict future features with the received historical features, which may lead to large prediction errors since compressed features cause excessive information loss. In contrast, our FDC directly utilizes the complete local features within the collaborators to capture feature-level motion patterns.\n* Structure: SyncNet uses multi-scale LSTM structures to capture temporal cues for predicting the whole feature. Our FDC utilizes a simple flow generator to output feature flows and scale matrix, which can speed up convergence and achieves more accurate predictions at various delay levels.\n3.  STCFormer\n* Philosophy: V2X-ViT exploits the vision transformer to aggregate collaborator features, ignoring contextual cues. Our STCFormer uses a unified structure to integrate spatio-temporal information and achieves more holistic feature fusion with decoupled design.\n* Structure: V2X-ViT uses per-location attention to learn pixel-wise attention weights among features. In our STCFormer, we design the novel temporal cross-attention to capture historical information, which V2X-ViT ignores. Further, we propose the importance-aware query initialization, keypoint sampling mechanism, and adaptive late fusion to achieve more effective collaboration.\n***\n**Q2**: About main results.\n\n**A2**: We want to clarify that the inconsistent results are since our experimental setting has significant differences in bandwidth limitations from previous works. As stated in lines 241-242 of Main Paper and 245-250 of Supplementary, all experiments are constrained to $\\approx$ 1 MB bandwidth consumption to reflect the narrow communication channels in more realistic and challenging V2V/X scenarios.\n\nIn this case, we reproduce the baselines in Tab. 1 under bandwidth-limited noise settings based on the public codebase. Compared to the original reports, the decreased results of previous works prove that restricted communication resources can severely impact collaboration performance. Notably, How2comm outperforms existing models and improves the SOTA performance of AP\\@0.7 by 7.4% on the real-world dataset DAIR-V2X. \n\nWe will open the complete codebase for community progress if the paper is accepted.\n***\n**Q3**: About transmission delay.\n\n**A3**: SyncNet utilizes received historical features to predict future features, which greatly consumes storage resources within the ego agent. Also, extracting the temporal semantics from the compressed features is difficult since the transmitted features have lost too much rich information. V2X-ViT uses delay-aware positional encoding to capture temporal information. This implicit compensation strategy would cause excessive accumulated errors under severe delay due to not utilizing historical features.\n\nIn contrast,  our FDC strategy generates feature flows and the scale matrix with prediction ability at the collaborators' side and sends them to the ego agent. This strategy captures rich temporal information using complete local features within collaborators for more accurate prediction without consuming the ego agent's computation resources. From Fig. 6 in Main Paper, How2comm achieves SOTA performance at all delay levels of three datasets and maintains precision under severe delay, proving the FDC's advantages over existing methods.\n***\n**Q4**: About pooling functions.\n\n**A4**: \n*  In line 122 of Sec. 3.3, average and max poolings in the spatial query are applied to the channel dimension. In line 125 of Sec. 3.3, average and max poolings are applied to the spatial dimension in the channel query. We provide more details in Fig.1 of Supplementary.\n*   In line 175, global average pooling is applied to the spatial dimension to compress the feature map. \n*   In line 181, max pooling is applied to the channel dimension to generate two-dimensional importance maps.\n\nWe promise to add these minor explanations to the revision.\n***\n**Q5**: About mutual information.\n\n**A5**: From lines 120-132, after the ego agent broadcasts request queries, collaborators eliminate unnecessary information for the ego agent via the spatial-channel message filtering matrix. In this case, the goal of mutual information supervision is not to maintain global information between vanilla features and compressed features, but to maximally preserve local critical semantics in the selected regions after message filtering, which avoids the potential loss of locally valuable information.\n***\n**Q6**: About communication cost. \n\n**A6**: The ego agent broadcasts two request queries with sizes $(H \\times W \\times 1)$ and $(1 \\times 1\\times C)$. Since queries and non-zero-pixel indices are tiny compared to the communication volume of transmitted features, we do not consider their costs following Where2comm. Where2comm similarly ignores the overhead caused by the request map with size $(H \\times W \\times 1)$ and non-zero-pixel indices of transmitted features. "}, {"review_id": "7ehy3wWkY5", "reviewer": "Reviewer_ToHv", "summary": "This paper presents a collaborative perception system called How2comm, which aims to address three common challenges encountered in the collaborative perception setting:\n\n1) Communication redundancy: To mitigate communication redundancy, the paper introduces a Mutual Information-aware communication approach. This approach incorporates a spatial-channel message-filtering strategy and a supervision signal that seeks to maximize the mutual information between the transmitted features and the original features.\n\n2) Transmission delay: To tackle transmission delay, the paper proposes the utilization of flow generation and warping techniques for effective delay compensation.\n\n3) Collaboration heterogeneity: To handle collaboration heterogeneity, the paper employs a carefully designed decoupled spatial attention mechanism in the STC transformer. This mechanism enables the network to learn exclusive and common collaborator features. Additionally, the paper introduces adaptive late fusion to combine these features in a meaningful manner.\n\nBy implementing these approaches, the proposed system aims to enhance collaboration in the perception setting by addressing communication redundancy, transmission delay, and collaboration heterogeneity.", "questions": "The questions are almost all related to the [Weaknesses] section:\n1. A clear definition of Collaboration Heterogeneity is needed;\n\n2. The design logic of the common feature attention module, and why it can tackle the difficulty mentioned in the introduction;\n\n3. Missing ablation study for STCFormer.\n\n4. Since it is a comparatively complex system and latency is one of the biggest issues in collaborative perception systems, can authors provide some inference time comparison with other SOTA methods? such as where2comm. \n\nMinor questions: \n\n1. In the ablation study (Table 2), it seems that the dynamic late fusion only has a very marginal improvement in performance. Is any explanation for that? \n\n2. For the dots in Figure 8 (b)(c), does the lighter dot mean that the weight is lower or higher?\n\n3. In L165-L167, the expression of \"the decoupled spatial semantics and temporal dynamics among agents\" is a little bit confusing. The STCFormer seems only to decouple the spatial semantics into common and exclusive, then why \"temporal dynamics\" is here? \n\n", "limitations": "Another limitation of this paper is that it assumes the features from different agents' sensors are readily available for fusion when designing the STCFormer, despite acknowledging the presence of collaboration heterogeneity. This assumption may oversimplify the challenges associated with fusing heterogeneous sensor data and overlook the potential difficulties in achieving seamless integration of diverse sensor modalities. The paper could further explore and address the practical issues and complexities related to the fusion of sensor data with varying characteristics and formats, providing a more comprehensive and robust solution for collaboration heterogeneity in the context of collaborative perception systems.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) This paper presents a system that addresses three significant challenges in collaborative perception simultaneously, making it a relatively complex undertaking. However, it is worth noting that the paper is exceptionally well-organized, and the writing surpasses the average standard in terms of clarity and coherence.\n\n2) Regarding the first two challenges, namely communication redundancy and transmission delay, the proposed designs exhibit both originality and effectiveness. Particularly noteworthy is the intuitive flow-based compensation method, which proves to be highly effective in practice.\n\n3) The experimental results presented in the paper are promising, and the authors provide a clear and comprehensive demonstration of their findings. This aspect enhances the overall credibility of the research and its potential practical implications.", "weaknesses": "The third challenge, collaboration heterogeneity, raises some concerns and requires further clarification and evidence of the authors' approach to addressing it.\n\n1) From the reviewer's perspective, collaboration heterogeneity primarily refers to the heterogeneity of sensors, which may result in heterogeneity within the feature space. For instance, when one agent uses a camera while another employs a LiDAR sensor, additional operations are needed to ensure proper fusion of the information. Even in Figure 1(d), the dominant cause of collaboration heterogeneity is the differing beam numbers of the two LiDAR sensors. It is mentioned that there might be a misunderstanding on the reviewer's part, as a clear definition of collaboration heterogeneity is not evident from lines 42 to 52 of the paper.\n\n2) In Section 3.5, it is observed that the network architectures for the feature fusion module of exclusive and common features appear to be identical. However, in lines 48 to 50, the authors indicate that the fusion of the common area is the more challenging aspect. This raises the expectation among readers for specialized designs addressing the fusion of common features. Have the authors considered the heterogeneity in the features of different collaborators?\n\n3) Table 2 displays the ablation study of excluding Exclusive Spatial Attention and Common Spatial Attention. It would be helpful to know if the authors have also conducted experiments without explicitly defining exclusive or common features, using the same attention module structure twice. In this scenario, both $\\mathcal{E}$ and $\\mathcal{C}$ would be set to 1 throughout the dataset. This question arises due to the concerns expressed in the second point regarding the efficiency of the decoupled design.\n\nGiven these uncertainties, it is crucial to provide additional explanations and experimental results to enhance the understanding of collaboration heterogeneity as perceived by the authors and the logical efficiency of the STCFormer in overcoming this challenge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our effective designs and comprehensive experiments. Below are some specific responses. The common questions regarding the definition of collaboration heterogeneity, ablation studies, and inference time can be found in the global response.\n\n**Q1**:  Design logic of the common spatial attention (CSA).\n\n**A1**:  We want to clarify that in lines 46-50, we use Fig. 1(d) to introduce the exclusive and common perception regions, and to explain that it is essential to jointly fuse the valuable spatial semantics of these two types of regions. The original intent of the description was not to imply that the fusion of common area features is more challenging. We apologize for the misunderstanding and will improve the presentation in the revision.\n\nNext, we introduce the design logic of the CSA module in three parts: \n*  As stated in lines 182-185, we get the common candidate map following $\\mathcal{C}_j^t = \\mathcal{I}_i^t \\odot \\mathcal{I}_j^t$. Since the importance map reflects the perceptual critical levels of features at each spatial location, $\\mathcal{C}_j^t$ reflects the regions where both the ego agent and the collaborator have high importance, i.e., common perception regions. \n*  We use type-dependent linear layers to project agent features with distribution discrepancies to the same feature space based on the agent types. Such type-dependent operations would help capture the agent heterogeneity and optimize feature fusion. Since the common perception region reflects the heterogeneous characterizations of the same spatial semantics from the ego agent and collaborator, the type-dependent linear layers would learn the data characteristics of distinct agents and thus bridge the data discrepancies in the CSA module.\n*  As stated in lines 193-195, we introduce a learnable sampling method to enhance the target queries' feature. This method can adaptively select keypoints and assign attention weights based on feature characteristics, which can learn to set reasonable weights based on agent feature distribution.\n\nFor the challenges in the introduction:\n*  For the collaboration heterogeneity, the CSA module eliminates the agent feature heterogeneity with type-dependent projection and a learnable sampling mechanism that dynamically allocates the attention weights based on the feature distribution. The above operations enable the CSA module to distinguish the learning process of diverse agents to alleviate collaboration heterogeneity.\n*  For the feature misalignment, the CSA module enhances the target queries' features with local context and long-distance visual cues through an adaptive sampling method, improving the robustness against feature misalignment.\n***\n**Q2**: Ablation of adaptive late fusion (ALF).\n\n**A2**: Insightful comments! As shown in lines 315-316, we replace the ALF module in Tab. 2 of the Main Paper with the pixel-wise addition strategy to perform the ablation study. The exclusive and common features are spatially complementary due to the previous decoupling operation. Therefore, the pixel-wise addition operation is able to partially combine the advantages of both features and present fine detection performance. The reason is that pixel-wise addition achieves the partial contribution of the ALF module since ALF is essentially a weighted version of the addition operation, as stated in lines 200-206. In this case, our ALF module adaptively integrates the perceptual contributions of the two features at each spatial location by adjusting the pixel-wise weights and, therefore, can further enhance the perception performance compared to the heuristic addition operation.\n\nNotably, we compare the other two fusion strategies in Tab. 1 of the Supplementary. The declining performance demonstrates the significant performance improvement of the proposed ALF module and its advantages in integrating perception contributions of different features.\n***\n**Q3**:  About the dots in Fig. 8(b)&(c).\n\n**A3**:  The darker the colour of the forestgreen dots, the higher the attention weights of the corresponding sampled keypoints. From Fig. 8, the proposed exclusive and common spatial attention modules sample keypoints around the target query to extract object-related meaningful perception cues and assign larger attention weights to these keypoints. Moreover, they also reasonably sample long-range keypoints to extend the perception range of the ego agent. We will refine this description in the revision.\n***\n**Q4**: About temporal dynamics in STCFormer.\n\n**A4**: As stated in lines 169-171, the ability of STCFormer to integrate temporal dynamics derives from our proposed Temporal Cross-Attention (TCA) module. From Fig. 3(a)&(b), the TCA module captures the temporal context from the ego agent's historical features and collaborator features to obtain the enhanced representation $H_{i}^{t}$, which is then passed to the subsequent Decoupled Spatial Attention component. Overall, STCFormer integrates both spatial semantic and temporal information, enabling more accurate perception.\n***\n**Q5**: About limitations.\n\n**A5**: Constructive comments! We want to clarify that the assumption that features from different agents' LiDAR sensors are available follows previous works and consistent dataset protocols for a fair comparison. We agree that exploring the integration across sensor modalities with different characteristics via STCFormer is an essential research direction. However, the accessible dataset formats cause unavoidable limitations. For instance, current dataloaders do not support heterogeneous sensor data loading, and the labels required for image- and LiDAR-based detection paradigms are mismatched. We have done the best we can with the current scope. Moreover, we have further discussed the broad impacts and realistic limitations of How2comm in Sec. 5 and 6 of Supplementary, while indicating future directions for optimizing our system."}, {"review_id": "dqenyQgVNq", "reviewer": "Reviewer_kdrk", "summary": "This article proposes a communication-efficient and collaboration-pragmatic framework that effectively reduces object detection errors caused by transmission delays in multi-agent collaborative perception tasks, while achieving a balance between bandwidth utilization and detection accuracy. This paper introduces the spatial-channel message filter to assist collaborators in selecting important feature information. Additionally, to tackle the issue of information transmission delays, it employs flow-guided delay compensation operations to predict the future state of collaborators, thereby reducing errors caused by information transmission latency. Finally, this article integrates feature information into the STCFomer framework and leverages two attention mechanism modules to effectively improve the accuracy of target detection after feature fusion.", "questions": "1. In Section 3.4 \"Flow-guided Delay Compensation\", in line 150, should the referenced image be Figure 2 rather than Figure 1?\n2. In Section 3.3 \"Spatial-channel message filtering\", in line 128, Why is the feature sent to other agents required to undergo the \"1-A\" operation?\n3. If the range and content of point clouds collected by different agents may overlap but are not entirely consistent, how can the method ensure that the queries sent by the ego agent effectively associate with the same vehicles in the feature map of collaborators?\n4. Because the chosen dataset for this article does not provide point cloud flows, I would like to ask if the authors, when performing the \"flow-guided delay compensation\" operation, treated the continuous frame sequences in the dataset as a whole and re-segmented it? It would be helpful if the authors could explain the data processing steps involved.\n", "limitations": "The authors have acknowledged the limitation of their work in only being able to handle short-term historical frames. However, this is influenced by various factors such as the dataset. The authors have made a comprehensive effort within the existing scope of their work.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "\n1. The article proposes an effective solution to one of the primary issues in multi-agent collaborative perception tasks, achieving a reduction in detection errors after feature fusion while maintaining a balance between bandwidth utilization and detection accuracy. The proposed method demonstrates improved detection accuracy on three main datasets.\n2. The designed STCFormer architecture in this article effectively integrates the information transmitted to the ego agent. By leveraging temporal cross-attention and exclusive spatial attention modules, it enhances information in both the temporal and spatial dimensions.\n3. The article exhibits fluency in language, completeness in experiments, and reliable comparative results. It has the potential to contribute to the development of the research field.", "weaknesses": "The paper's ablation experiments are expected to provide more detailed analysis and discussion, including jointly eliminating several modules. For example, simultaneously removing the delay compensation strategy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing our effective solution, complete experiments, and reliable results. Below are some specific responses. The common ablation experiments can be found in the global response.\n\n**Q1**:  Reference problem for the figure in line 150.\n\n**A1**:  Thank you for the conscientious checking! We will correct this typo in the revision.\n***\n**Q2**:  Explanation of the $1-A_{i, s/c}^{t_0}$ operation.\n\n**A2**:  As stated in lines 122-126 of the Main Paper, the obtained spatial and channel queries $A_{i, s/c}^{t_0}$ reflect salient spatial locations and semantically meaningful channels in the feature $F_{i}^{t_0}$. As stated in lines 90-91 of the Supplementary, the two request queries $R_{i, s/c}^{t_0}$ obtained via operation $1-A_{i, s/c}^{t_0}$ reflect potentially unconfident positions in the ego agent due to practical difficulties such as occlusion, which require complementary information to improve detection accuracy. To this end, collaborators can produce a spatial-channel binary filtering matrix to assist in transmitting spatial-channel sparse, yet perceptually critical features based on the needs reflected in the request queries and their own strengths.\n***\n**Q3**:  Discussion of the position association.\n\n**A3**:  We follow the existing works using point cloud projection and coordinate system transformation to synchronize the collaborator feature maps to the ego agent's coordinate frame. Based on this, the queries learned based on the ego agent feature are spatially aligned with the collaborator feature maps to support subsequent cross-agent interaction and collaboration. Concretely, as stated in lines 108-110 of the Main Paper, we assume that the collaborators project their own point cloud data to the ego agent's coordinate system upon receiving the ego agent's metadata (e.g., poses and extrinsic) before feature extraction. Also, we utilize the differential transformation and sampling operator provided by V2X-ViT [1] to transform the collaborator feature maps to the current ego agent's pose.\n\n[1] Xu R, Xiang H, Tu Z, Xia X, Yang M-H, Ma J, V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer, European Conference on Computer Vision 2022.\n***\n**Q4**: Discussion of the data processing steps.\n\n**A4**: In practice, we optimize the data processing steps of existing codebases to return consecutive frame sequences with configurable lengths.\n\nThe complete raw data in existing datasets consists of multiple scenes, where each scene has a continuous frame sequence with a fixed length. The current collaborative perception codebase treats sensor data from multiple agents at the same timestamp as one data sample and assigns a unique sample index. In addition, the scene index is also provided to reflect the scene in which the data sample is located. When loading data, the training script provides the sample indexes in a disrupted order and loads the corresponding data samples. This data-loading step limits the usage of historical samples. \n\nTo this end, we provide a new data processing step: when loading a target data sample with sample index $n$, we simultaneously load multiple previous historical data samples, such as $( n-2, n-1 )$, constituting a sequence of consecutive frames. It is worth noting that we utilize the scene index to determine whether the loaded samples belong to the same scene to avoid data loading errors. As described in lines 150-152 of the Main Paper, the loaded sequence of consecutive frames is passed into the generator to extract temporal information and output the feature flow and scale matrix.\n\nIf this paper is accepted, we will release the complete codebase for community development.\n***\n**Q5**: About limitations.\n\n**A5**: The authors are very grateful to the reviewer for helping us clarify that many practical limitations are indeed impacted by diverse factors, especially the current dataset configuration and format. As a complement, we have further discussed the broad impacts and realistic limitations of How2comm in Sec. 5 and 6 of the Supplementary, indicating future directions for optimizing our system."}]

Author rebuttal:
We thank all reviewers for their time and effort. Here we provide detailed responses to common questions.

**Q1**: Trade-off between training cost, inference time, and perception performance.

**A1**: 
* We compare the evaluated models regarding space-time complexity, inference time, and detection performance across the five datasets in Tab. 1 of the **response pdf**. Results of the OPV2V Culver City and V2V4Real datasets are presented in Tab. 2 in Supplementary and Tab. 3 of the response pdf. We adopt three mainstream metrics: Parameter number, FLOPs, and Fps, and use the same hyperparameter setting as the Main Paper. Intuitively, How2comm outperforms previous SOTAs on five datasets. In particular, How2comm improves the SOTA performance of AP\@0.7 by 7.4% and 6.5% on DAIR-V2X and V2V4Real, respectively, showing significant improvements.
* Concretely, How2comm outperforms the complex transformer-based V2X-ViT in terms of Parameter number, FLOPs, and Fps on five datasets. Moreover, How2comm reduces the Parameter number and FLOPs compared to CoBEVT and improves the Fps on most datasets compared to Where2comm. The above results indicate that How2comm is competitive and comparable in space-time complexity and inference time compared to the SOTAs. Overall, the above results prove that How2comm can achieve a better tradeoff among training cost, inference time, and perception performance than most SOTA methods.
* The reasonable explanations are:
1. From lines 189-190, we introduce importance-aware query initialization in STCFormer to guide decoupled spatial attention modules to focus on the potential foreground objects. This approach uses the sparsity of point cloud data to reduce query number and computing complexity while maintaining perception performance.
2. From lines 179-181, the importance maps are obtained by simple sigmoid activation, max pooling, and $f_{gen}$ implemented by 1*1 convolution. The complexity introduced by these operations is negligible.
***
**Q2**:  Discussion of datasets.

**A2**: We want to clarify that OPV2V and V2XSet datasets differ greatly regarding agent type, agent count, and scenario characteristics, which V2X-ViT confirms. Also, Tab. 2 of the Supplementary indicates the excellent performance of How2comm on the OPV2V Culver City validation set, which provides challenging driving scenarios that are close to the real world.

Further, we provide the results on the V2V4Real dataset in Tab. 3 of the **response pdf** for a more holistic comparison. We employ the same delay settings and bandwidth constraints as the Main Paper, and localization noises are not added since this dataset has natural GPS errors. Intuitively, How2comm outperforms existing SOTA models and improves the SOTA performance of AP\@0.7 by 6.5%. Overall, How2comm improves SOTA performance on all five datasets, showing our method's superiority in real-world and simulated scenarios. Limited rebuttal time did not allow us to conduct experiments on the V2X-Sim; we will add comparative results in the revision.
***
**Q3**: Definition of collaboration heterogeneity and difficulties for fusing information from different agents.

**A3**: Current heterogeneity in collaborative perception systems may come from two aspects: heterogeneous sensor modalities and the same type of sensor's configuration discrepancies. We consider the latter due to some practical constraints. In this paper, collaboration heterogeneity means the heterogeneity in feature-level collaboration due to LiDAR configuration discrepancies from distinct agents (e.g., vehicles and infrastructures), including different LiDAR densities, distributions, reflectivities, and noise interference.

In this case, the following difficulties exist for feature fusion in the collaboration process:
* How to integrate object-related representations from complementary regions of heterogeneous agents to extend the perception range.
* How to mitigate the feature map gap due to sensor configuration differences in the common perception regions.
* How to overcome feature map misalignment caused by collaborative noises. e.g., localization errors.

Motivated by the above difficulties, we propose the novel STCFormer to tackle the existing challenges.
***
**Q4**: Discussion of ablation studies.

**A4**: The ablation results on key hyperparameters and structure choices are presented in Tab. 1 of the Supplementary. Moreover, we provide more ablation results for decoupled designs and complete modules in Tab. 2 of the **response pdf**.

First, we set both $\mathcal{E,C}$ to 1 and use the same attention module in both branches. Intuitively, Full Model achieves better detection performance on all five datasets than the version without Decoupled Design, proving that How2comm effectively integrates the perception advantages from exclusive and common regions via two decoupled branches and yields more robust visual representations against collaboration heterogeneity. Without explicitly defining exclusive and common features, spatial attention will indiscriminately sample the entire region due to the lack of guidance from prior information, which may introduce excessive collaborator noises and impede bridging to collaboration heterogeneity. 

Meanwhile, we show the ablation results for the complete modules. We directly remove the FDC module and replace MIC and STCFormer with a compression-based communication method and a 1*1 convolution-based fusion, respectively. The degraded performance provides the following observations:
 
1. Our MIC module can facilitate detection performance by retaining more critical semantic information under limited bandwidth.
2. When FDC is removed, the feature misalignment caused by temporal asynchrony significantly impact the feature fusion process and harm the precision.
3. STCFormer enables more valuable spatio-temporal semantics to be extracted from collaborator-shared features and historical frames than 1*1 convolution-based fusion.

Meta-review: The paper proposes an efficient cooperative perception method with robust modules designed to accommodate a variety of real-world factors. Reviewers fcX9, ToHv, and kdrk are enthusiastic about the concept, affirming that the paper is well-written, sufficiently novel, and addresses potential real-world challenges effectively. Reviewer aK4W expressed reservations about the novelty and experimental sections of the paper. While the ACs partially agree with reviewer aK4W's concerns about similarities between certain portions of the paper and previous works, they appreciate the innovative elements within the paper. After reading the comments and the draft, the ACs recommend to accept this paper.
Decision: Accept (poster)