Title: How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perception
Venue: NeurIPS 2023
Accepted: yes
Keywords: Collaborative perception

multi-agent communication
Abstract: Multi-agent collaborative perception has recently received widespread attention as an emerging application in driving scenarios. Despite the advancements in previous efforts, challenges remain due to various noises in the perception procedure, including communication redundancy, transmission delay, and collaboration heterogeneity. To tackle these issues, we propose \textit{How2comm}, a collaborative perception framework that seeks a trade-off between perception performance and communication bandwidth. Our novelties lie in three aspects. First, we devise a mutual information-aware communication mechanism to maximally sustain the informative features shared by collaborators. The spatial-channel filtering is adopted to perform effective feature sparsification for efficient communication. Second, we present a flow-guided delay compensation strategy to predict future characteristics from collaborators and eliminate feature misalignment due to temporal asynchrony. Ultimately, a pragmatic collaboration transformer is introduced to integrate holistic spatial semantics and temporal context clues among agents. Our framework is thoroughly evaluated on several LiDAR-based collaborative detection datasets in real-world and simulated scenarios. Comprehensive experiments demonstrate the superiority of How2comm and the effectiveness of all its vital components. The code will be released at https://github.com/ydk122024/How2comm.

How2comm: Communication-Efficient and Collaboration-Pragmatic Multi-Agent Perception

Abstract

Multi-agent collaborative perception has recently received widespread attention as an emerging application in driving scenarios. Despite the advancements in previous efforts, challenges remain due to various dilemmas in the perception procedure, including communication redundancy, transmission delay, and collaboration heterogeneity. To tackle these issues, we propose How2comm , a collaborative perception framework that seeks a trade-off between perception performance and communication bandwidth. Our novelties lie in three aspects. First, we devise a mutual information-aware communication mechanism to maximally sustain the informative features shared by collaborators. The spatial-channel filtering is adopted to perform effective feature sparsification for efficient communication. Second, we present a flow-guided delay compensation strategy to predict future characteristics from collaborators and eliminate feature misalignment due to temporal asynchrony. Ultimately, a pragmatic collaboration transformer is introduced to integrate holistic spatial semantics and temporal context clues among agents. Our framework is thoroughly evaluated on several LiDAR-based collaborative detection datasets in real-world and simulated scenarios. Comprehensive experiments demonstrate the superiority of How2comm and the effectiveness of all its vital components. The code will be released at https://github.com/ydk122024/How2comm .

1 Introduction

Precise perception of complex and changeable driving environments is essential to ensure the safety and reliability of intelligent agents [25, 46], e.g. , autonomous vehicles (AVs). With the emergence of learning-based technologies, remarkable single-agent perception systems are extensively explored for several in-vehicle tasks, such as instance segmentation [19, 58] and object detection [26, 49]. Nevertheless, single-agent perception suffers from various shortcomings due to the isolated view, such as unavoidable occlusions [55], restricted detection ranges [56], and sparse sensor observations [57]. Recently, multi-agent collaborative perception [39, 54] has provided promising solutions as an emerging application for vehicle-to-vehicle/everything (V2V/X) communications. The impressive studies [7, 14, 16, 17, 31, 32, 33, 38, 40, 51] have progressively presented to aggregate valuable information and complementary perspectives among on-road agents, resulting in a more

(a) Aligned point cloud fusion

(c) Infrastructure side image

(d) Fused point cloud

(b) Non-aligned point cloud fusion

precise perception. Despite recent advancements, challenges remain due to unpreventable dilemmas, including communication redundancy, transmission delay, and collaboration heterogeneity.

Communication Redundancy. The dominant patterns for reducing communication overhead are summarized as feature compression [14, 33, 40] and spatial filtering [7, 8]. The former assumes that agents share all spatial areas indiscriminately, which dramatically wastes bandwidth. The latter overly relies on confidence maps to highlight gullible locations and fails to consider spatially holistic information. Moreover, these methods invariably cause losses of transmitted valuable information.

Transmission Delay. Figures 1(a)&(b) present the point cloud fusion results from an ego vehicle and an infrastructure in the time-synchronous and time-asynchronous cases, respectively. The inevitable transmission delay causes position misalignment of fast-moving objects within the green circles, potentially harming subsequent collaboration performance. Although several delay-aware strategies [13, 40, 53] are proposed to tackle this issue, they either suffer from performance bottlenecks [13, 40] or introduce massive computation costs [53], leading to sub-optimal solutions.

Collaboration Heterogeneity. Figures 1(c)&(d) show the typical collaboration scenario involving two agents and the fused point cloud. Intuitively, LiDAR configuration discrepancies ( e.g. , different LiDAR densities, distributions, reflectivities, and noise interference) across agents potentially cause collaboration heterogeneity within the feature space. In this case, the orange box contains the common perception region of both agents, which facilitates bridging the feature-level gap caused by sensor configuration discrepancies [37, 40]. The magenta box contains the exclusive perception region of the infrastructure, which provides complementary information for the ego vehicle and compensates for the occluded view. Fusing valuable spatial semantics from these two perception regions facilitates comprehensive and pragmatic perception. However, most previous methods [14, 16, 32, 33, 40] integrate collaborator-shared features via per-agent/location message fusion to enhance ego representations, whose collaboration processes could be vulnerable since the advantages of distinct perception regions from heterogeneous agents are not considered holistically. Moreover, the current single-frame perception paradigm faces the challenges of 3D point cloud sparsity and localization errors, increasing the difficulty of building a robust multi-agent perception system.

Motivated by the above observations, we propose How2comm , an end-to-end collaborative perception framework to address the existing issues jointly. Through three novel components, How2comm advances towards a reasonable trade-off between perception performance and communication bandwidth. Specifically, (i) we first design a mutual information-aware communication mechanism to maximally preserve the beneficial semantics from vanilla characteristics in the transmitted messages of collaborators. In this case, spatial-channel message filtering is introduced to determine how to use less bandwidth for efficient communication. (ii) Second, we present a flow-guided delay compensation strategy to predict the future features of collaborators by mining contextual dependencies in sequential frames. Our ingenious strategy determines how to dynamically compensate for the delay's impact and explicitly accomplish temporal alignment. (iii) Furthermore, we construct a spatio-temporal collaboration transformer (STCFormer) module to integrate perceptually comprehensive information from collaborators and temporally valuable clues among agents. Our unified transformer structure determines how to achieve pragmatic collaboration, contributing to a more robust collaborative perception against localization errors and feature discrepancies. How2comm is systematically evaluated on various collaborative 3D object detection datasets, including DAIR-V2X [52], V2XSet [40], and OPV2V [41]. Quantitative experiments demonstrate that our framework significantly outperforms previous state-of-the-art (SOTA) methods under the bandwidth-limited noisy setting. Systematic analyses confirm the robustness of How2comm against distinct collaboration noises.

Shared

Encoder

Importance

Generator

Message

Filtering

Flow

Generator

Message Packing

Message

Delivery

Predict

Exclusive

Common

Detection

Decoders

Spatio-Temporal

Collaboration

Transformer

(STCFormer)

Request

Broadcast

1-

Infrastructure

Temporal Context

Projected

LiDAR

Projected

LiDAR

22t

22t

Xg-

...X

Stg

J

t

t

J

Ct

3

X...X

,Rt,c

(-)

2 Related Work

Multi-Agent Communication. Benefiting from rapid advances in learning-based technologies [12, 21, 22, 23, 24, 42, 43, 44, 45, 47, 48, 50], communication has played an essential role in constructing robust and stable multi-agent systems. Although early works provided heuristic insights into information sharing among different agents through predefined protocols and centralized paradigms [4, 28, 29], these efforts are typically difficult to generalize into challenging scenarios. Recently, several learning-driven communication strategies have been proposed to accommodate diverse scenario applications. For instance, Vain [6] utilized an attentional neural structure to specify what information needs to be shared in agent interactions. ATOC [9] introduced the recurrent unit to decide whom the agent communicates with by receiving local observations and action intentions from other agents. TarMAC [2] designed a reinforcement learning-oriented architecture to learn communication from task-specific rewards. In comparison, we focus on LiDAR-based collaborative 3D object detection tasks. For more challenging driving scenarios, we design the mutual information supervision and attention-guiding mechanism to achieve efficient communication across agents.

Collaborative Perception. Collaborative perception is only in its infancy as a promising application of multi-agent systems. Several impressive approaches have been designed to facilitate the overall perception performance of AVs. The mainstream works [7, 14, 16, 33, 37, 38, 40] followed the intermediate collaboration pattern to balance average precision and bandwidth overhead. Specifically, When2com [16] introduced a handshake mechanism to determine when to communicate with collaborators. V2VNet [33] employed a fully connected graph network to aggregate feature representations shared by agents. After that, DiscoNet [14] proposed a knowledge distillation framework to supervise the intermediate collaboration through an early collaboration of full views. V2X-ViT [40] designed distinct attention interactions to facilitate adaptive information fusion among heterogeneous agents. Where2comm [7] aimed to transmit perceptually critical information via sparse spatial confidence maps. However, these methods invariably ignore valuable historical clues and lead to sub-optimal solutions. In this paper, we propose a novel collaboration transformer to jointly capture spatial semantics and temporal dynamics among agents, resulting in a more pragmatic collaboration.

3 Methodology

3.1 Problem Formulation

In this paper, we seek to develop a communication-efficient and collaboration-pragmatic multi-agent system to enhance the perception ability of the ego agent. Figure 2 illustrates the proposed system framework, which accommodates different agents ( e.g. , AVs and infrastructures). Consider N agents in a driving scene, let X t i be the local point cloud observation of the i -th agent ( i.e. , ego agent) and

Y i be the corresponding ground-truth supervision. The objective of How2comm is to maximize the LiDAR-based 3D detection performance ℜ ( · ) under a total communication budget B :

where ð ( · , · ) denotes the perception evaluation metric and Ψ θ is the perception system parameterized by θ . ˜ F t 0 j is the message transmitted from the j -th agent to the i -th agent at time delay τ -aware moment t 0 , where t 0 = t -τ . The remainder of Section 3 details the major components.

3.2 Metadata Conversion and Feature Extraction

In the initial stage of collaboration, we build a communication graph [14, 40] where one agent is selected as the ego agent and the other connected agents act as collaborators. Upon receiving the broadcast metadata ( e.g. , poses and extrinsic) from the ego agent, the collaborators project their local observations to the ego agent's coordinate system. Moreover, ego-motion compensation [35] synchronizes each agent's historical frames. The shared PointPillar [11] encoder f enc ( · ) converts the point cloud of the i -th agent at timestamp t into the bird's-eye-view (BEV) features as F t i = f enc ( X t i ) ∈ R H × W × C , where H,W,C denote height, width, and channel, respectively.

3.3 Mutual Information-aware Communication

Previous attempts to reduce the required transmission bandwidth relied heavily on autoencoders [14, 33, 40] or confidence maps [7, 8], which are one-sided as they only consider information compression over spatial positions or channels. To this end, we design a mutual information-aware communication (MIC) mechanism to select the most informative messages from space and channels to save precious bandwidth. MIC consists of two core parts as follows.

Spatial-channel Message Filtering. Stemming from solid evidence in signal picking, we first introduce the CBAM [34]-like spatial-channel attention queries to assist each agent in sharing their salient features. The spatial query A t 0 i,s = σ ( ω 3 ∗ 3 [ φ a ( F t 0 i ); φ m ( F t 0 i )]) ∈ R H × W × 1 reflects what spatial locations on delayed feature F t 0 i are informative, where [ · ; · ] is the concatenation, σ is the sigmoid activation, φ a/m ( · ) denote average and max pooling functions, and ω 3 ∗ 3 ( · ) is the 2D 3 × 3 convolution operation. The channel query A t 0 i,c = σ ( ω 1 ∗ 1 ( φ a ( F t 0 i )) + ω 1 ∗ 1 ( φ m ( F t 0 i ))) ∈ R 1 × 1 × C reflects which channels in F t 0 i are semantically meaningful. φ a/m ( · ) in the spatial and channel queries are applied to the channel and spatial dimensions, respectively. Then, the ego agent indicates the supplementary messages required to improve local perception performance by broadcasting request queries R t 0 i,s / R t 0 i,c = 1 -A t 0 i,s / A t 0 i,c . The j -th collaborator then aggregates the requests with its attention queries to obtain a spatial-channel binary message filtering matrix as follows:

where f sel ( · ) is a threshold-based selection function and ⊙ is the element-wise multiplication. Ultimately, the selected feature map is obtained as ˜ F t 0 j = F t 0 j ⊙ M t 0 j , which provides spatialchannel sparse, yet perceptually critical information.

Mutual Information Maximization Supervision. Most existing works ignore the potential loss of valuable information due to feature compression. To overcome this dilemma, we maximally sustain the local critical semantics in the corresponding vanilla feature F t 0 j on the selected regions of the transmitted features ˜ F t 0 j by mutual information estimation. Since we only focus on maximizing the mutual information rather than getting its precise value, a stable estimator [5] is utilized to build the objective supervision based on the Jensen-Shannon divergence. Formally, the mutual information between two random variables X and Z is estimated as follows:

where T ϱ : X×Z → R is a statistics network parameterized by ϱ . In our case, the mutual information supervision of all collaborators within the communication link is defined as follows:

̸

Time

Embedding

Temporal Cross-Attention

Feed Forward

Feed Forward

Adaptive Late Fusion

Exclusive

Spatial

Attention

Common

Spatial

Attention

Add & Norm

Add & Norm

Add & Norm

Add & Norm

2D Conv

2D Conv

2D Conv

Global

Pooling

Target

Query

Linear

Attending

Feature

Cross-

Attention

Filling

Sampling

Offsets

Position

Embedding

1D Conv

Global

Pooling

1D Conv

2D Conv

Linear

Linear

Linear

(a)

(b)

(c)

3.4 Flow-guided Delay Compensation

We present a flow-guided delay compensation (FDC) strategy to eliminate the two-sided fusion error in feature-level collaboration due to temporal asynchrony. Existing solutions relied on received historical features [13] and produced large errors under severe delay [53], leading to performance bottlenecks. To tackle the issues, we adopt the philosophy of feature flow to predict the collaborators' future features for temporal alignment with the ego representation. The details are as follows.

Flow Generation and Warping. Due to the uncertain delay between the ego agent and collaborators, FDC predicts the feature flow O t 0 j for a fixed time interval and scale matrix S t 0 j based on the j -th agent's historical frames. Specifically, as Figure 2 shows, features { F t 0 -k j , ..., F t 0 j } are concatenated in channel dimension and entered to a generator f flow ( · ) to output O t 0 j ∈ R H × W × 2 and S t 0 j ∈ R H × W × 1 . Then the j -th agent sends {O t 0 j , S t 0 j } with prediction ability and sparse feature ˜ F t 0 j to the ego agent. The ego agent estimates predicted collaborator features as Z t j = f warp ( ˜ F t 0 j , ( t -t 0 ) · O t 0 j ) ⊙S t 0 j , where f warp ( · ) is the bilinear warping function applied to all positions and channels [60, 61], and · is the scalar multiplication. The temporally aligned features are passed to the STCFormer.

Self-supervised Training Pattern. Self-supervised learning is employed to train the flow generator f flow ( · ) since the existing datasets [40, 41, 52] lack the motion annotations. Concretely, we first form the training group { F t 0 -k j , ..., F t 0 j , F t j } , where { F t 0 -k j , ..., F t 0 j } is a continuous feature sequence, and F t j is considered as the ground truth feature. Subsequently, we predict the feature Z t j as Z t j = f warp ( F t 0 j , ( t -t 0 ) · O t 0 j ) ⊙S t 0 j . Since the optimization objective of f flow ( · ) is to increase the similarity between F t j and Z t j , we formulate the self-supervised loss function L flow based on the cosine similarity [53] as follows:

̸

where ∥ · ∥ 2 F is the squared Frobenius norm.

3.5 Spatio-Temporal Collaboration Transformer

To efficiently mitigate collaboration heterogeneity, we propose a spatio-temporal collaboration transformer (STCFormer) to jointly integrate the decoupled spatial semantics and temporal dynamics among agents. From Figure 3(a), the core contributions of STCFormer lie in the following three customized modules, where the other basic components follow the choice of the vanilla transformer [30].

Temporal Cross-Attention. To bridge the detection gap regarding fast-moving objects due to point cloud sparsity, we capture historical context clues across agents to reinforce the current

representation via a temporal cross-attention (TCA) module. The core is to perform Query-KeyValue-like attention operations by projecting the ego feature F t i and merged temporal features E = [ ˜ F t 0 j , F t 0 i , ..., F t i ] into different subspaces via three 2D convolutional layers ω 3 ∗ 3 ( · ) . In Figure 3(b), the branches of Key&Value E k/v ← ω 1 ( φ a ( ω 3 ∗ 3 ( E ))) ⊙ ω 3 ∗ 3 ( E ) share the same structure but separate weights, where a 1D temporal convolution ω 1 ( · ) with global average pooling φ a ( · ) provides temporal correlations. φ a ( · ) is applied to the spatial dimension to shrink the feature map. The computation of TSA can be shown as:

Decoupled Spatial Attention. To comprehensively integrate the distinct spatial semantics from the collaborators, we facilitate pragmatic message fusion with a feature decoupling perspective inspired by the observation in Figure 1(d). Formally, an importance generator f gen ( · ) is employed to generate the importance maps of the ego feature F t i and the estimated collaborator feature Z t j as I t i / I t j = σ ( φ m ( f gen ( F t i / Z t j ))) ∈ [0 , 1] H × W . The importance maps reflect the perceptually critical level of each pixel in the features. Then, the j -th agent spatially decouples the feature Z t j via candidate maps E t j = (1 -I t i ) ⊙I t j and C t j = I t i ⊙I t j . Intuitively, E t j and C t j depict the collaborators' exclusive and common perception regions relative to the ego agent, respectively. The exclusive and common collaborator features are obtained as Z t j,e = f sel ( E t j ) ⊙ Z t j and Z t j,c = f sel ( C t j ) ⊙ Z t j .

Then, we present two spatial attention modules based on deformable cross-attention [59] to aggregate the decoupled exclusive and common features, which share the same structure but different weights. Here the exclusive spatial attention (ESA) is taken as an example (see Figure 3(c)), and its input comprises H t i and Z t j,e . An importance-aware query initialization is first designed to guide ESA to focus on the potential foreground objects. Specifically, we obtain the element-wise summation of the importance maps as I t = ∑ N j =1 I t j and extract N q target queries from the salient locations in I t . The attention scores are learned from the initial queries via a linear layer and the softmax function. Subsequently, a linear layer learns an offset map for each input feature, providing the 2D spatial offset { ∆ q v | 1 ≤ v ≤ N v } for each query q . We sample the keypoints based on the learned offset maps and extract these keypoints' features to form the attending feature. The cross-attention layer aggregates multiple collaborators' features to output the enhanced feature for each query q as:

where u indexs the attention head, and W u/f denotes the learnable parameters. Then, the filling operation fills ESA ( q ) into H t i based on the initial positions of the queries and outputs F t i,e . Similarly, the enhanced common feature F t i,c is obtained via the common spatial attention (CSA).

Adaptive Late Fusion. The adaptive late fusion (ALF) module is presented to effectively fuse the exclusive and common representations { F t i,e , F t i,c } for incorporating their perceptual advantages. Formally, we obtain two weight maps as G t i,e / G t i,c = ω 1 ∗ 1 ( F t i,e / F t i,c ) , and apply the softmax function to produce the normalized weight maps as G t i,e / G t i,c = softmax( G t i,e / G t i,c ) . The learned G t i,e and G t i,c reflect the complementary perception contributions of { F t i,e , F t i,c } at each spatial location. Therefore, we adaptively activate the perceptually critical information of each representation by a weighted summation. The refined feature map is obtained as F t i = G t i,e ⊙ F t i,e + G t i,c ⊙ F t i,c .

3.6 Detection Decoders and Objective Optimization

Two detection decoders { f r dec ( · ) , f c dec ( · ) } are employed to convert the output fused representation F t i into the prediction results. The regression result represents the position, size, and yaw angle of the predefined box at each location, which is Y ( t ) i,r = f r dec ( F t i ) ∈ R H × W × 7 . The classification result is Y ( t ) i,c = f c dec ( F t i ) ∈ R H × W × 2 , revealing the confidence value of each bounding box to be an object. For objective optimization, we leverage the smooth absolute error loss for regression (denoted as L reg ) and the focal loss [15] for classification (denoted as L cla ). In total, we formulate the overall objective function as follows: L all = L reg + L cla + L mul + L flow .













































































































































































































4 Experiments

4.1 Datasets and Implementation Details

Multi-Agent 3D Detection Datasets. To evaluate the performance of How2comm on the collaborative perception task, we conduct extensive experiments on three multi-agent datasets, including DAIRV2X [52], V2XSet [40], and OPV2V [41]. DAIR-V2X [52] is a real-world vehicle-to-infrastructure perception dataset containing 100 realistic scenarios and 18,000 data samples. Each sample collects the labeled LiDAR point clouds of a vehicle and an infrastructure. The training/validation/testing sets are split in a ratio of 5:2:3. V2XSet [40] is a simulated dataset supporting V2X perception, co-simulated by Carla [3] and OpenCDA [36]. It includes 73 representative scenes with 2 to 5 connected agents and 11,447 3D annotated LiDAR point cloud frames. The training/validation/testing sets are 6,694, 1,920, and 2,833 frames, respectively. OPV2V [41] is a large-scale simulated dataset for multi-agent V2V perception, comprising 10,914 LiDAR point cloud frames with 3D annotation. The training/validation/testing splits include 6,764, 1,981, and 2,169 frames, respectively.

Evaluation Metrics. We adopt the Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.5 and 0.7 to evaluate the 3D object detection performance. Also, the calculation format of communication volume in [7] is used to count the message size by byte in the log scale with base 2.

4.2 Quantitative Evaluation

Experimental Settings. We build all the models using the Pytorch toolbox [18] and train them on Tesla V100 GPUs with the Adam optimizer [10]. The learning rate is set to 2e-3 and decays exponentially by 0.1 every 15 epochs. The training settings on the DAIR-V2X [52], V2XSet [40], and OPV2V [41] datasets include: the training epochs are { 30 , 40 , 40 } , and batch sizes are { 2 , 1 , 1 } . The height and width resolution of the feature encoder f enc ( · ) is 0.4 m. The selection function f sel ( · ) has a threshold of 0.01. The flow generator f flow ( · ) leverages the multi-scale backbone to extract multi-grained representations and an extra encoder to produce O t 0 j and S t 0 j . We implement the importance gen-

Table 1: Performance comparison on the DAIRV2X [52], V2XSet [40], and OPV2V [41] datasets. The results are reported in AP@0.5/0.7.

erator f gen ( · ) and statistical network T ϱ ( · ) with the classification decoder in [11] and following [27], respectively. The keypoint number N v is 9, and the attention head is 8. Two 1 × 1 convolutional layers





















































































































































































are utilized to implement the detection decoders. Under the default noise settings, the transmission delay τ is set to 100 ms, and the localization and heading errors of the collaborators are sampled from a Gaussian distribution with standard deviations of 0.2 m and 0.2 ◦ , respectively. All experiments are constrained to ≈ 1 MB bandwidth consumption to reflect the narrow communication channels in real V2V/X scenarios [20].

Detection Performance Comparison. Table 1 compares the detection performance of the proposed How2comm with various models on three datasets under default noise settings. We consider two typical baselines. No Fusion is a single-agent perception pattern that only uses the local observations. Late Fusion integrates predicted boxes across agents and produces results with non-maximum suppression. Moreover, the existing SOTAs are comprehensively considered, including When2com [16], F-Cooper [1], AttFuse [41], V2VNet [33], DiscoNet [14], V2X-ViT [40], CoBEVT [38], and Where2comm [7]. Intuitively, How2comm outperforms previous methods in the real-world (DAIRV2X [52]) and simulated datasets, demonstrating the superiority of our model and its robustness to various realistic noises. In particular, the SOTA performance of AP@0.7 on the DAIR-V2X and OPV2V is improved by 8.4% and 7.0%, respectively. Compared to previous per-agent/location message fusion efforts [14, 16, 33, 40], How2comm simultaneously considers the decoupled spatial semantics and temporal dynamics among agents, resulting in a more precise perception.

Comparison of Communication Volume. Figure 4 presents the performance comparison results with distinct bandwidth consumptions. Concretely, the orange and blue curves denote the detection precision of our How2comm and Where2comm under varying communication volumes, respectively. (i) How2comm keeps superior to Where2comm across all the communication choices, i.e. , How2comm achieves a better performance-bandwidth trade-off with spatial-channel filtering than Where2comm. (ii) Moreover, our framework seeks comparable performances as the SOTAs by consuming less bandwidth. The noteworthy improvements demonstrate that the proposed communication mechanism filters invalid semantics and maintains performance by mutual information maximization.

Robustness to Localization and Heading Errors. We verify the detection performance of How2comm under varying pose errors of collaborators in Figures 5 and 6 following the noise settings in [40]. Specifically, the localization and heading errors are sampled from Gaussian distributions with a standard deviation of σ xyz ∈ [0 , 0 . 5] mand [0 ◦ , 1 . 0 ◦ ] , respectively. As shown in the figures, the performance of all intermediate collaboration models consistently deteriorates due to feature map misalignment as the pose errors increase. Noticeably, How2comm is superior to the previous SOTA models and No Fusion across three datasets under all error levels, while some models ( e.g. , V2VNet and When2com) are even weaker than No Fusion when the error exceeds 0.2 m and 0 . 2 ◦ on the DAIR-V2X dataset. This comparison demonstrates the robustness of How2comm against collaboration pose noises. One reasonable explanation is that our framework captures perceptually critical and holistic information across heterogeneous agents via the tailored STCFormer.

Robustness to Transmission Delay. As noted in Figure 1(b), temporal asynchrony due to transmission delay results in two-sided fusion errors and harms the collaboration procedure. We analyze the sensitivity of existing models to varying delays ( i.e. , from 0 to 400 ms) in Figure 7. Noticeably, all the intermediate fusion methods inevitably degrade with increasing transmission delay due to misleading feature matching. Nevertheless, How2comm maintains higher precision than SOTAs at all delay levels across all three datasets and improves AP@0.7 by 7.5% than No Fusion on DAIR-V2X under a severe delay (400 ms). This robustness to the transmission delay proves that How2comm accomplishes temporal alignment of two-sided features by predicting the collaborators' future features.

4.3 Ablation Studies

We perform thorough ablation studies on all datasets to understand the necessity of the different designs and strategies in How2comm. Table 2 shows the following vital observations.

Rationality of Communication Mechanism.

(i) The spatial and channel attention queries are first removed separately to perform incomplete message filtering. The decreased performance implies that both query patterns contribute to sharing sparse yet salient features among agents. (ii) There is a significant degradation in the detection results on all datasets when the communication mechanism lacks mutual information supervision. A plausible deduction is that our supervision mitigates the loss of valuable information due to spatialchannel feature filtering.

Effect of Delay Compensation Strategy. (i) Here, we remove the scale matrix to verify its effect. The poor results show that appropriate scaling of predicted features promotes effective temporal alignment. (ii) Furthermore, the performance drop caused by the self-supervised training removal shows the importance of imposing motion representation supervision for collaborator-shared features.

Importance of STCFormer. STCFormer is evaluated in three dimensions. (i) We first find that temporal cross-attention provides beneficial gains due to performance deterioration when discarded. It is because the meaningful temporal clues in historical frames bridge the single-frame detection gap. (ii) Then, exclusive and common spatial attention modules are removed separately to explore the impact on performance. The consistently decreased results on each dataset suggest that integrating distinct spatial semantics is indispensable for pragmatic collaboration. (iii) Finally, the adaptive late fusion is replaced by pixel-wise addition. The gain decline suggests that our fusion paradigm provides new insights for aggregating the perceptual advantages of distinct spatial semantics.

Impact of Keypoint Number N v . Empirically, we set the variable number of keypoints for the decoupled spatial attention modules to perform experiments and find that 9 keypoints achieve the most competitive detection performance. Conversely, too few keypoints may cause valuable semantics loss and too many keypoints may cause performance bottlenecks due to accumulated errors. The above finding inspires us to determine the appropriate keypoint number that samples more rich visual clues and captures more pragmatic collaboration information from collaborators.

Necessity of Decoupled Design. Ultimately, we set E t j and C t j to 1 while maintaining two spatial attention branches in the STCFormer to justify the decoupled design. Without explicitly defining exclusive and common features, spatial attention would indiscriminately sample the entire region due to the lack of guidance from prior perception information, which may introduce excessive collaborator noises and impede bridging to collaboration heterogeneity.

4.4 Qualitative Evaluation

Visualization of Detection Results. To illustrate the perception performance of different models, Figure 8 shows the detection visualizations of two challenging scenarios from the DAIR-V2X

(a) V2X-ViT

(b) CoBEVT

(c) Where2comm

(d) How2comm (ours)

D

D

吕

吕

0

吕

D

0

号

0

D

00

0

00

D

D

1

D

叫吧

(a) GT bounding boxes

(b) Exclusive spatial attention

(c) Common spatial attention

0600

8

dataset [52] under default noise settings. How2comm axiomatically achieves more robust and accurate detection compared to previous SOTA models, including V2X-ViT [40], CoBEVT [38], and Where2comm [7]. Concretely, our method produces more predicted bounding boxes well aligned with the ground truths. The merits may lie in two aspects. (i) Our delay compensation strategy mitigates feature misalignment due to temporal asynchrony, improving detection performance. (ii) The proposed STCFormer provides effective temporal context clues for detecting fast-moving objects and fuses meaningful information from nearby agents to compensate for the occluded perspective.

Visualization of Spatial Attention Maps. We show visualizations of exclusive and common spatial attention (ESA/CSA) in Figure 9 to justify the effectiveness of our feature decoupling philosophy. (i) Intuitively, from Figure 9(b), ESA effectively samples keypoints at the exclusive perception region from the infrastructure, providing complementary information for the ego agent to promote perception ability. (ii) In Figure 9(c), CSA mitigates the detection gap due to collaboration heterogeneity by sampling keypoints that reasonably focus on the common perception region between two agents.

5 Conclusion and Limitation

This paper presents How2comm, a novel collaborative perception framework to tackle existing issues jointly. How2comm maximizes the beneficial semantics in filtered features and accomplishes temporal alignment via the feature flow estimation. Moreover, our STCFormer holistically aggregates the spatial semantics and temporal dynamics among agents. Extensive experiments on several multi-agent datasets show the effectiveness of How2comm and the necessity of all its components.

Limitation and Future Work. The current work only exploits the short-term historical frames. In future work, we plan to expand the utilization of temporal information to long-term point cloud sequences. Also, we will explore optimizing the feature flow prediction with uncertainty estimation.

Acknowledgment. This work is supported in part by the National Key R&D Program of China under Grant 2021ZD0113503 and in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0103.

References

Qi Chen, Xu Ma, Sihai Tang, Jingda Guo, Qing Yang, and Song Fu. F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In Proceedings of the 4th ACM/IEEE Symposium on Edge Computing , pages 88-100, 2019. 7, 8

Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In International Conference on Machine Learning (ICML) , pages 1538-1546. PMLR, 2019. 3

Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on Robot Learning , pages 1-16. PMLR, 2017. 7

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in Neural Information Processing Systems (NeurIPS) , 29, 2016. 3

R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv:1808.06670 , 2018. 4

Yedid Hoshen. Vain: Attentional multi-agent predictive modeling. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017. 3

Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication-efficient collaborative perception via spatial confidence maps. Advances in Neural Information Processing Systems (NeurIPS) , 35:4874-4886, 2022. 1, 2, 3, 4, 7, 8, 10

Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9243-9252, 2023. 2, 4

Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018. 3

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR) , 2015. 7

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 1269712705, 2019. 4, 7

Yuxuan Lei, Dingkang Yang, Mingcheng Li, Shunli Wang, Jiawei Chen, and Lihua Zhang. Textoriented modality reinforcement network for multimodal sentiment analysis from unaligned multimodal sequences. arXiv preprint arXiv:2307.13205 , 2023. 3

Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 316-332. Springer, 2022. 2, 5

Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems (NeurIPS) , 34:29541-29552, 2021. 1, 2, 3, 4, 7, 8

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 2980-2988, 2017. 6

Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 4106-4115, 2020. 1, 2, 3, 7, 8

Guiyang Luo, Hui Zhang, Quan Yuan, and Jinglin Li. Complementarity-enhanced and redundancy-minimized collaboration network for multi-agent perception. In Proceedings of the ACM International Conference on Multimedia (ACM MM) , pages 3578-3586, 2022. 1

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems (NeurIPS) , 32, 2019. 7

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 652-660, 2017. 1

Hang Qiu, Pohan Huang, Namo Asavisanu, Xiaochen Liu, Konstantinos Psounis, and Ramesh Govindan. Autocast: Scalable infrastructure-less cooperative perception for distributed collaborative driving. 2022. 8

Linhao Qu, Shaolei Liu, Manning Wang, and Zhijian Song. Transmef: A transformer-based multi-exposure image fusion framework using self-supervised multi-task learning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 2126-2134, 2022. 3

Linhao Qu, Siyu Liu, Xiaoyu Liu, Manning Wang, and Zhijian Song. Towards label-efficient automatic diagnosis and analysis: a comprehensive survey of advanced deep learning-based weakly-supervised, semi-supervised and self-supervised techniques in histopathological image analysis. Physics in Medicine & Biology , 2022. 3

Linhao Qu, Xiaoyuan Luo, Shaolei Liu, Manning Wang, and Zhijian Song. Dgmil: Distribution guided multiple instance learning for whole slide image classification. In Medical Image Computing and Computer-Assisted Intervention (MICAAI) , pages 24-34. Springer, 2022. 3

Linhao Qu, Manning Wang, Zhijian Song, et al. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Advances in Neural Information Processing Systems (NeurIPS) , 35:15368-15381, 2022. 3

Tika Ram and Khem Chand. Effect of drivers' risk perception and perception of driving tasks on road safety attitude. Transportation Research Part F: Traffic Psychology and Behaviour , 42:162-176, 2016. 1

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 779-788, 2016. 1

Eduardo Hugo Sanchez, Mathieu Serrurier, and Mathias Ortner. Learning disentangled representations via mutual information estimation. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 205-221. Springer, 2020. 7

Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. arXiv preprint arXiv:1812.09755 , 2018. 3

Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropagation. Advances in Neural Information Processing Systems (NeurIPS) , 29, 2016. 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017. 5

Binglu Wang, Lei Zhang, Zhaozhong Wang, Yongqiang Zhao, and Tianfei Zhou. Core: Cooperative reconstruction for multi-agent perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8710-8720, October 2023. 1

Tianhang Wang, Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. Umc: A unified bandwidth-efficient and multi-resolution based collaborative perception framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 8187-8196, October 2023. 1, 2

Tsun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 605-621. Springer, 2020. 1, 2, 3, 4, 7, 8

Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cbam: Convolutional block attention module. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 3-19, 2018. 4

Pengxiang Wu, Siheng Chen, and Dimitris N Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based on bird's eye view maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11385-11395, 2020. 4

Runsheng Xu, Yi Guo, Xu Han, Xin Xia, Hao Xiang, and Jiaqi Ma. Opencda: an open cooperative driving automation framework integrated with co-simulation. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC) , pages 1155-1162. IEEE, 2021. 7

Runsheng Xu, Jinlong Li, Xiaoyu Dong, Hongkai Yu, and Jiaqi Ma. Bridging the domain gap for multi-agent perception. In IEEE International Conference on Robotics and Automation (ICRA) , pages 6035-6042. IEEE, 2023. 2, 3

Runsheng Xu, Zhengzhong Tu, Hao Xiang, Wei Shao, Bolei Zhou, and Jiaqi Ma. Cobevt: Cooperative bird's eye view semantic segmentation with sparse transformers. In Conference on Robot Learning (CoRL) , 2022. 1, 3, 7, 8, 10

Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang, Zhengzhong Tu, Zonglin Meng, Hao Xiang, Xiaoyu Dong, Rui Song, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13712-13722, 2023. 1

Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 107-124. Springer, 2022. 1, 2, 3, 4, 5, 7, 8, 10

Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In Proceedings of the International Conference on Robotics and Automation (ICRA) , pages 2583-2589. IEEE, 2022. 2, 5, 7, 8

Dingkang Yang, Zhaoyu Chen, Yuzheng Wang, Shunli Wang, Mingcheng Li, Siao Liu, Xiao Zhao, Shuai Huang, Zhiyan Dong, Peng Zhai, and Lihua Zhang. Context de-confounded emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 19005-19015, June 2023. 3

Dingkang Yang, Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM) , page 1642-1651, 2022. 3

Dingkang Yang, Shuai Huang, Yang Liu, and Lihua Zhang. Contextual and cross-modal interaction for multi-modal speech emotion recognition. IEEE Signal Processing Letters , 29:2093-2097, 2022. 3

Dingkang Yang, Shuai Huang, Shunli Wang, Yang Liu, Peng Zhai, Liuzhen Su, Mingcheng Li, and Lihua Zhang. Emotion recognition for multiple context awareness. In Proceedings of the European Conference on Computer Vision (ECCV) , volume 13697, pages 144-162. Springer, 2022. 3

Dingkang Yang, Shuai Huang, Zhi Xu, Zhenpeng Li, Shunli Wang, Mingcheng Li, Yuzheng Wang, Yang Liu, Kun Yang, Zhaoyu Chen, Yan Wang, Jing Liu, Peixuan Zhang, Peng Zhai, and Lihua Zhang. Aide: A vision-driven multi-view, multi-modal, multi-tasking dataset for assistive driving perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 20459-20470, October 2023. 1

Dingkang Yang, Haopeng Kuang, Shuai Huang, and Lihua Zhang. Learning modality-specific and -agnostic representations for asynchronous multimodal language sequences. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM) , page 1708-1717, 2022. 3

Dingkang Yang, Yang Liu, Can Huang, Mingcheng Li, Xiao Zhao, Yuzheng Wang, Kun Yang, Yan Wang, Peng Zhai, and Lihua Zhang. Target and source modality co-reinforcement for emotion understanding from asynchronous multimodal sequences. Knowledge-Based Systems , page 110370, 2023. 3

Kun Yang, Jing Liu, Dingkang Yang, Hanqi Wang, Peng Sun, Yanni Zhang, Yan Liu, and Liang Song. A novel efficient multi-view traffic-related object detection framework. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5, 2023. 1

Kun Yang, Peng Sun, Jieyu Lin, Azzedine Boukerche, and Liang Song. A novel distributed task scheduling framework for supporting vehicular edge intelligence. In IEEE International Conference on Distributed Computing Systems (ICDCS) , pages 972-982, 2022. 3

Kun Yang, Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang, Peng Sun, and Liang Song. Spatio-temporal domain awareness for multi-agent collaborative perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 23383-23392, October 2023. 1

Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, et al. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 21361-21370, 2022. 2, 5, 7, 8, 10

Haibao Yu, Yingjuan Tang, Enze Xie, Jilei Mao, Jirui Yuan, Ping Luo, and Zaiqing Nie. Vehicle-infrastructure cooperative 3d object detection via feature flow prediction. arXiv preprint arXiv:2303.10552 , 2023. 2, 5

Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, et al. V2x-seq: A large-scale sequential dataset for vehicle-infrastructure cooperative perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5486-5495, 2023. 1

Xiaoding Yuan, Adam Kortylewski, Yihong Sun, and Alan Yuille. Robust instance segmentation through reasoning about multi-object occlusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11141-11150, 2021. 1

Zhenxun Yuan, Xiao Song, Lei Bai, Zhe Wang, and Wanli Ouyang. Temporal-channel transformer for 3d lidar-based video object detection for autonomous driving. IEEE Transactions on Circuits and Systems for Video Technology , 32(4):2068-2078, 2021. 1

Zixu Zhang and Jaime F Fisac. Safe occlusion-aware autonomous driving via game-theoretic active perception. arXiv preprint arXiv:2105.08169 , 2021. 1

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2881-2890, 2017. 1

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR) , 2021. 6

Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pages 408-417, 2017. 5

Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2349-2358, 2017. 5

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article proposes a communication-efficient and collaboration-pragmatic framework that effectively reduces object detection errors caused by transmission delays in multi-agent collaborative perception tasks, while achieving a balance between bandwidth utilization and detection accuracy. This paper introduces the spatial-channel message filter to assist collaborators in selecting important feature information. Additionally, to tackle the issue of information transmission delays, it employs flow-guided delay compensation operations to predict the future state of collaborators, thereby reducing errors caused by information transmission latency. Finally, this article integrates feature information into the STCFomer framework and leverages two attention mechanism modules to effectively improve the accuracy of target detection after feature fusion.\n\nSTRENGTHS:\n1. The article proposes an effective solution to one of the primary issues in multi-agent collaborative perception tasks, achieving a reduction in detection errors after feature fusion while maintaining a balance between bandwidth utilization and detection accuracy. The proposed method demonstrates improved detection accuracy on three main datasets.\n2. The designed STCFormer architecture in this article effectively integrates the information transmitted to the ego agent. By leveraging temporal cross-attention and exclusive spatial attention modules, it enhances information in both the temporal and spatial dimensions.\n3. The article exhibits fluency in language, completeness in experiments, and reliable comparative results. It has the potential to contribute to the development of the research field.\n\nWEAKNESSES:\nThe paper's ablation experiments are expected to provide more detailed analysis and discussion, including jointly eliminating several modules. For example, simultaneously removing the delay compensation strategy.\n\nQUESTIONS:\n1. In Section 3.4 \"Flow-guided Delay Compensation\", in line 150, should the referenced image be Figure 2 rather than Figure 1?\n2. In Section 3.3 \"Spatial-channel message filtering\", in line 128, Why is the feature sent to other agents required to undergo the \"1-A\" operation?\n3. If the range and content of point clouds collected by different agents may overlap but are not entirely consistent, how can the method ensure that the queries sent by the ego agent effectively associate with the same vehicles in the feature map of collaborators?\n4. Because the chosen dataset for this article does not provide point cloud flows, I would like to ask if the authors, when performing the \"flow-guided delay compensation\" operation, treated the continuous frame sequences in the dataset as a whole and re-segmented it? It would be helpful if the authors could explain the data processing steps involved.\n\nLIMITATIONS:\nThe authors have acknowledged the limitation of their work in only being able to handle short-term historical frames. However, this is influenced by various factors such as the dataset. The authors have made a comprehensive effort within the existing scope of their work.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a collaborative perception system called How2comm, which aims to address three common challenges encountered in the collaborative perception setting:\n\n1) Communication redundancy: To mitigate communication redundancy, the paper introduces a Mutual Information-aware communication approach. This approach incorporates a spatial-channel message-filtering strategy and a supervision signal that seeks to maximize the mutual information between the transmitted features and the original features.\n\n2) Transmission delay: To tackle transmission delay, the paper proposes the utilization of flow generation and warping techniques for effective delay compensation.\n\n3) Collaboration heterogeneity: To handle collaboration heterogeneity, the paper employs a carefully designed decoupled spatial attention mechanism in the STC transformer. This mechanism enables the network to learn exclusive and common collaborator features. Additionally, the paper introduces adaptive late fusion to combine these features in a meaningful manner.\n\nBy implementing these approaches, the proposed system aims to enhance collaboration in the perception setting by addressing communication redundancy, transmission delay, and collaboration heterogeneity.\n\nSTRENGTHS:\n1) This paper presents a system that addresses three significant challenges in collaborative perception simultaneously, making it a relatively complex undertaking. However, it is worth noting that the paper is exceptionally well-organized, and the writing surpasses the average standard in terms of clarity and coherence.\n\n2) Regarding the first two challenges, namely communication redundancy and transmission delay, the proposed designs exhibit both originality and effectiveness. Particularly noteworthy is the intuitive flow-based compensation method, which proves to be highly effective in practice.\n\n3) The experimental results presented in the paper are promising, and the authors provide a clear and comprehensive demonstration of their findings. This aspect enhances the overall credibility of the research and its potential practical implications.\n\nWEAKNESSES:\nThe third challenge, collaboration heterogeneity, raises some concerns and requires further clarification and evidence of the authors' approach to addressing it.\n\n1) From the reviewer's perspective, collaboration heterogeneity primarily refers to the heterogeneity of sensors, which may result in heterogeneity within the feature space. For instance, when one agent uses a camera while another employs a LiDAR sensor, additional operations are needed to ensure proper fusion of the information. Even in Figure 1(d), the dominant cause of collaboration heterogeneity is the differing beam numbers of the two LiDAR sensors. It is mentioned that there might be a misunderstanding on the reviewer's part, as a clear definition of collaboration heterogeneity is not evident from lines 42 to 52 of the paper.\n\n2) In Section 3.5, it is observed that the network architectures for the feature fusion module of exclusive and common features appear to be identical. However, in lines 48 to 50, the authors indicate that the fusion of the common area is the more challenging aspect. This raises the expectation among readers for specialized designs addressing the fusion of common features. Have the authors considered the heterogeneity in the features of different collaborators?\n\n3) Table 2 displays the ablation study of excluding Exclusive Spatial Attention and Common Spatial Attention. It would be helpful to know if the authors have also conducted experiments without explicitly defining exclusive or common features, using the same attention module structure twice. In this scenario, both $\\mathcal{E}$ and $\\mathcal{C}$ would be set to 1 throughout the dataset. This question arises due to the concerns expressed in the second point regarding the efficiency of the decoupled design.\n\nGiven these uncertainties, it is crucial to provide additional explanations and experimental results to enhance the understanding of collaboration heterogeneity as perceived by the authors and the logical efficiency of the STCFormer in overcoming this challenge.\n\nQUESTIONS:\nThe questions are almost all related to the [Weaknesses] section:\n1. A clear definition of Collaboration Heterogeneity is needed;\n\n2. The design logic of the common feature attention module, and why it can tackle the difficulty mentioned in the introduction;\n\n3. Missing ablation study for STCFormer.\n\n4. Since it is a comparatively complex system and latency is one of the biggest issues in collaborative perception systems, can authors provide some inference time comparison with other SOTA methods? such as where2comm. \n\nMinor questions: \n\n1. In the ablation study (Table 2), it seems that the dynamic late fusion only has a very marginal improvement in performance. Is any explanation for that? \n\n2. For the dots in Figure 8 (b)(c), does the lighter dot mean that the weight is lower or higher?\n\n3. In L165-L167, the expression of \"the decoupled spatial semantics and temporal dynamics among agents\" is a little bit confusing. The STCFormer seems only to decouple the spatial semantics into common and exclusive, then why \"temporal dynamics\" is here?\n\nLIMITATIONS:\nAnother limitation of this paper is that it assumes the features from different agents' sensors are readily available for fusion when designing the STCFormer, despite acknowledging the presence of collaboration heterogeneity. This assumption may oversimplify the challenges associated with fusing heterogeneous sensor data and overlook the potential difficulties in achieving seamless integration of diverse sensor modalities. The paper could further explore and address the practical issues and complexities related to the fusion of sensor data with varying characteristics and formats, providing a more comprehensive and robust solution for collaboration heterogeneity in the context of collaborative perception systems.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes How2comm, a collaborative perception framework, which seeks a trade-off between perception performance and communication bandwidth. This framework leverages a mutual information-aware communication to select the most informative messages to transmit and save bandwidth, where ego agents compute and send spatial and channel queries to neighbor agents. And a flow-guided delay compensation is applied to reduce temporal alignment. This paper designs Spatio-Temporal Collaboration Transformer to generate comprehensive features from spatial and temporal information. Experiments are conducted on three datasets Dair-V2X, V2XSet and OPV2V to validate the proposed methods.\n\nSTRENGTHS:\n1. This work considers the key problems in collaborative perception, including communication cost, transmission delay, localization error, designs a proper pipeline of collaboration and information processing  and get superior experiment results.\n2. The logic and structure of the paper is clear.\n3. The review and analysis of the field of study is comprehensive.\n\nWEAKNESSES:\n1. Many technical contributions claimed in this paper has been proposed in previous works, including requesting complementary information from neighbor agents to decrease communication costs [1], compensating latency with historical information [2] and fusing features with transformer [3]. This work appears to involve straightforward modifications of previous techniques and a system ensemble. So what is the essential technical novelty of the proposed methods? The authors need to clarify this point.\n\n2. The main experiment results in Table.1 are significantly lower and inconsistent with previous work[4,5], which use the same Pointpillars backbone. The results seem not convincing.\n\n3. The proposed framework includes more networks (transformer-based network), computations (importance map), and data (historical frames) than general collaborative perception methods, which may increase training costs and inference time. However, the performance seems not significant. There is a lack of evaluations of training cost, inference time and a trade-off between them and perception performance. \n\n4. OPV2V and V2XSet are very similar datasets, the experiments should be conducted on datasets with different characteristics, like the V2X dataset V2X-Sim[5], the real-world V2V dataset V2V4real[6].\n\n5. It lacks comprehensive comparison with previous methods that handle transmission delay. Previous work, like SyncNet [2] and V2X-ViT [3], has proposed to compensate for the latency with historical information. What are the advantages of the proposed flow-guided delay compensation?\n\n[1] Hu Y, Fang S, Lei Z, et al. Where2comm: Communication-efficient collaborative perception via spatial confidence maps, Advances in neural information processing systems 2022.\n[2] Lei Z, Ren S, Hu Y, et al. Latency-aware collaborative perception, European Conference on Computer Vision 2022.\n[3] Xu R, Xiang H, Tu Z, Xia X, Yang M-H, Ma J, V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer, European Conference on Computer Vision 2022.\n[4] Liu Y C, Tian J, Glaser N, et al. When2com: Multi-agent perception via communication graph grouping, Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition. 2020.\n[5]  Li Y, Ma D, An Z, et al. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving, IEEE Robotics and Automation Letters, 2022.\n[6] Xu R, Xia X, Li J, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nQUESTIONS:\n1. What is the meaning or definition of Collaboration Heterogeneity mentioned in section 1? What are the difficulties of fusing the information from different agents?\n2. In Spatial-channel Message Filtering of section3.3, which dimension is the pooling functions applied to? The authors need to clarify it as well as all pooling functions in this paper.\n3. Since the goal is to improve the perception performance of the ego agent with as little communication cost as possible, why maximizing the mutual information between the compressed and the vanilla feature  is necessary? If some information is useless to ego agent, it is unnecessary to keep them.\n4. Does the communication cost include the queries and the indices of non-zero pixels? The authors need to clarify it to ensure a fair comparison.\n\nLIMITATIONS:\nThe paper integrates a good collaboration perception system by overlaying various more or less known techniques. The core technical novelty is not clear. The experiments are also insufficient.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces three key contributions in tackling the aforementioned challenges. Firstly, the authors devise a mutual information-aware communication mechanism that maximizes the sharing of informative features among collaborators. They employ spatial-channel filtering for efficient feature sparsification and effective communication. Secondly, a flow-guided delay compensation strategy is presented to predict future characteristics from collaborators, addressing feature misalignment caused by temporal asynchrony. Lastly, a pragmatic collaboration transformer is introduced to integrate spatial semantics and temporal context clues, enhancing the overall perception capabilities among agents. The proposed framework, How2comm, is thoroughly evaluated on three LiDAR-based collaborative detection datasets in both real-world and simulated scenarios. The experiments conducted are comprehensive and demonstrate the superiority of How2comm over existing methods. The effectiveness of all vital components within the framework is also validated, providing solid evidence for the contributions made.\n\nSTRENGTHS:\n1. Novel Approach: How2comm addresses the challenges faced in multi-agent collaborative perception by introducing innovative techniques, such as mutual information-aware communication, flow-guided delay compensation, and pragmatic collaboration transformer.\n\n2. Comprehensive Evaluation: The framework is rigorously evaluated on various datasets, including real-world and simulated scenarios, establishing its superiority over existing methods.\n\n3. Clear Presentation: The paper effectively explains the proposed framework and its components, making it accessible to readers with varying levels of expertise.\n\nWEAKNESSES:\nHere are some suggestions: \n\n1. It would be beneficial for the authors to discuss any limitations or potential areas for improvement in the How2comm framework, providing avenues for future research.\n\n2. The authors are suggested to add some missing literature to be more comprehensive.\n\n[1] Li, Y., Ma, D., An, Z., Wang, Z., Zhong, Y., Chen, S. and Feng, C., 2022. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4), pp.10914-10921.\n\n[2] Yu, H., Luo, Y., Shu, M., Huo, Y., Yang, Z., Shi, Y., Guo, Z., Li, H., Hu, X., Yuan, J. and Nie, Z., 2022. Dair-v2x: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 21361-21370).\n\n[3] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R. and Yu, H., 2023. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13712-13722).\n\n[4] Yu, H., Yang, W., Ruan, H., Yang, Z., Tang, Y., Gao, X., Hao, X., Shi, Y., Pan, Y., Sun, N. and Song, J., 2023. V2X-Seq: A Large-Scale Sequential Dataset for Vehicle-Infrastructure Cooperative Perception and Forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 5486-5495).\n\n[5] Li, Y., Zhang, J., Ma, D., Wang, Y. and Feng, C., 2023, March. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning (pp. 2062-2072). PMLR.\n\n[6] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023. Uncertainty quantification of collaborative detection for self-driving. ICRA.\n\nQUESTIONS:\nThe authors are suggested to provide a video demonstration of the results.\n\nLIMITATIONS:\nN.A.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}