SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then, we present two spatial attention modules based on deformable cross-attention [59] to aggregate the decoupled exclusive and common features, which share the same structure but different weights. Here the exclusive spatial attention (ESA) is taken as an example (see Figure 3(c)), and its input comprises  $H_{i}^{t}$  and  $Z_{j,e}^{t}$ . An importance-aware query initialization is first designed to guide ESA to focus on the potential foreground objects. Specifically, we obtain the element-wise summation of the importance maps as  $\mathcal{I}^t = \sum_{j=1}^N \mathcal{I}_j^t$  and extract  $N_{q}$  target queries from the salient locations in  $\mathcal{I}^t$ . The attention scores are learned from the initial queries via a linear layer and the softmax function. Subsequently, a linear layer learns an offset map for each input feature, providing the 2D spatial offset  $\{\Delta q_v | 1 \leq v \leq N_v\}$  for each query  $q$ . We sample the keypoints based on the learned offset maps and extract these keypoints' features to form the attending feature. The cross-attention layer aggregates multiple collaborators' features to output the enhanced feature for each query  $q$  as:
$$
E S A (q) = \sum_ {u = 1} ^ {U} \mathcal {W} _ {u} \left[ \sum_ {j = 1} ^ {N} \sum_ {v = 1} ^ {N _ {v}} \operatorname {s o f t m a x} \left(\mathcal {W} _ {f} \boldsymbol {H} _ {i} ^ {t} (q)\right) \boldsymbol {Z} _ {j, e} ^ {t} \left(q + \Delta q _ {v}\right) \right], \tag {7}
$$
where  $u$  indexes the attention head, and  $\mathcal{W}_{u / f}$  denotes the learnable parameters. Then, the filling operation fills  $ESA(q)$  into  $\pmb{H}_i^t$  based on the initial positions of the queries and outputs  $\pmb{F}_{i,e}^{t}$ . Similarly, the enhanced common feature  $\pmb{F}_{i,c}^{t}$  is obtained via the common spatial attention (CSA).
Adaptive Late Fusion. The adaptive late fusion (ALF) module is presented to effectively fuse the exclusive and common representations  $\{\pmb{F}_{i,e}^{t},\pmb{F}_{i,c}^{t}\}$  for incorporating their perceptual advantages. Formally, we obtain two weight maps as  $\mathcal{G}_{i,e}^{t} / \mathcal{G}_{i,c}^{t} = \omega_{1*1}(\pmb{F}_{i,e}^{t} / \pmb{F}_{i,c}^{t})$ , and apply the softmax function to produce the normalized weight maps as  $\pmb{G}_{i,e}^{t} / \pmb{G}_{i,c}^{t} = \mathrm{softmax}(\mathcal{G}_{i,e}^{t} / \mathcal{G}_{i,c}^{t})$ . The learned  $\pmb{G}_{i,e}^{t}$  and  $\pmb{G}_{i,c}^{t}$  reflect the complementary perception contributions of  $\{\pmb{F}_{i,e}^{t},\pmb{F}_{i,c}^{t}\}$  at each spatial location. Therefore, we adaptively activate the perceptually critical information of each representation by a weighted summation. The refined feature map is obtained as  $\mathcal{F}_i^t = \pmb{G}_{i,e}^t\odot \pmb{F}_{i,e}^t +\pmb{G}_{i,c}^t\odot \pmb{F}_{i,c}^t$
3.6 Detection Decoders and Objective Optimization
Two detection decoders  $\{f_{dec}^r (\cdot),f_{dec}^c (\cdot)\}$  are employed to convert the output fused representation  $\mathcal{F}_i^t$  into the prediction results. The regression result represents the position, size, and yaw angle of the predefined box at each location, which is  $\mathcal{Y}_{i,r}^{(t)} = f_{dec}^{r}(\mathcal{F}_{i}^{t})\in \mathbb{R}^{H\times W\times 7}$ . The classification result is  $\mathcal{Y}_{i,c}^{(t)} = f_{dec}^{c}(\mathcal{F}_{i}^{t})\in \mathbb{R}^{H\times W\times 2}$ , revealing the confidence value of each bounding box to be an object. For objective optimization, we leverage the smooth absolute error loss for regression (denoted as  $\mathcal{L}_{reg}$ ) and the focal loss [15] for classification (denoted as  $\mathcal{L}_{cla}$ ). In total, we formulate the overall objective function as follows:  $\mathcal{L}_{all} = \mathcal{L}_{reg} + \mathcal{L}_{cla} + \mathcal{L}_{mul} + \mathcal{L}_{flow}$ .
Figure 4: Collaborative perception performance comparison of How2comm and Where2comm [7] on the DAIR-V2X, V2XSet, and OPV2V datasets with varying communication volumes.
Figure 5: Robustness to the localization error on the DAIR-V2X, V2XSet, and OPV2V datasets.
Multi-Agent 3D Detection Datasets. To evaluate the performance of How2comm on the collaborative perception task, we conduct extensive experiments on three multi-agent datasets, including DAIR-V2X [52], V2XSet [40], and OPV2V [41]. DAIR-V2X [52] is a real-world vehicle-to-infrastructure perception dataset containing 100 realistic scenarios and 18,000 data samples. Each sample collects the labeled LiDAR point clouds of a vehicle and an infrastructure. The training/validation/testing sets are split in a ratio of 5:2:3. V2XSet [40] is a simulated dataset supporting V2X perception, co-simulated by Carla [3] and OpenCDA [36]. It includes 73 representative scenes with 2 to 5 connected agents and 11,447 3D annotated LiDAR point cloud frames. The training/validation/testing sets are 6,694, 1,920, and 2,833 frames, respectively. OPV2V [41] is a large-scale simulated dataset for multi-agent V2V perception, comprising 10,914 LiDAR point cloud frames with 3D annotation. The training/validation/testing splits include 6,764, 1,981, and 2,169 frames, respectively.
Evaluation Metrics. We adopt the Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.5 and 0.7 to evaluate the 3D object detection performance. Also, the calculation format of communication volume in [7] is used to count the message size by byte in the log scale with base 2.
4.2 Quantitative Evaluation
Experimental Settings. We build all the models using the Pytorch toolbox [18] and train them on Tesla V100 GPUs with the Adam optimizer [10]. The learning rate is set to 2e-3 and decays exponentially by 0.1 every 15 epochs. The training settings on the DAIR-V2X [52], V2XSet [40], and OPV2V [41] datasets include: the training epochs are \{30, 40, 40\}, and batch sizes are \{2, 1, 1\}. The height and width resolution of the feature encoder  $f_{enc}(\cdot)$  is  $0.4 \mathrm{~m}$ . The selection function  $f_{sel}(\cdot)$  has a threshold of 0.01. The flow generator  $f_{flow}(\cdot)$  leverages the multi-scale backbone to extract multi-grained representations and an extra encoder to produce  $\mathcal{O}_j^{t_0}$  and  $S_j^{t_0}$ . We implement the importance gen
Table 1: Performance comparison on the DAIR-V2X [52], V2XSet [40], and OPV2V [41] datasets. The results are reported in AP@0.5/0.7.
Models	DAIR-V2X	V2XSet	OPV2V
AP@0.5/0.7	AP@0.5/0.7	AP@0.5/0.7
No Fusion	50.03/43.57	60.60/40.20	68.71/48.66
Late Fusion	48.93/34.06	54.92/30.75	79.62/59.48
When2com [16]	46.64/32.49	65.06/41.87	70.64/53.73
F-Cooper [1]	49.77/35.21	71.48/46.92	75.27/63.05
AttFuse [41]	50.86/38.30	70.85/48.66	79.14/64.52
V2VNet [33]	52.18/38.62	79.09/49.25	77.45/62.10
DiscoNet [14]	51.44/40.01	79.83/54.06	81.08/64.85
V2X-ViT [40]	51.68/39.97	83.64/61.41	80.61/66.42
CoBEVT [38]	56.08/41.45	81.07/54.33	81.59/67.50
Where2comm [7]	59.34/43.53	82.02/53.38	82.75/67.29
How2comm (ours)	62.36/47.18	84.05/67.01	85.42/72.24
erator  $f_{gen}(\cdot)$  and statistical network  $T_{\varrho}(\cdot)$  with the classification decoder in [11] and following [27], respectively. The keypoint number  $N_v$  is 9, and the attention head is 8. Two  $1 \times 1$  convolutional layers
Figure 6: Robustness to the heading error on the DAIR-V2X, V2XSet, and OPV2V datasets.
Figure 7: Robustness to the transmission delay on the DAIR-V2X, V2XSet, and OPV2V datasets.
are utilized to implement the detection decoders. Under the default noise settings, the transmission delay  $\tau$  is set to  $100\mathrm{ms}$ , and the localization and heading errors of the collaborators are sampled from a Gaussian distribution with standard deviations of  $0.2\mathrm{m}$  and  $0.2^{\circ}$ , respectively. All experiments are constrained to  $\approx 1$  MB bandwidth consumption to reflect the narrow communication channels in real V2V/X scenarios [20].
Detection Performance Comparison. Table 1 compares the detection performance of the proposed How2comm with various models on three datasets under default noise settings. We consider two typical baselines. No Fusion is a single-agent perception pattern that only uses the local observations. Late Fusion integrates predicted boxes across agents and produces results with non-maximum suppression. Moreover, the existing SOTAs are comprehensively considered, including When2com [16], F-Cooper [1], AttFuse [41], V2VNet [33], DiscoNet [14], V2X-ViT [40], CoBEVT [38], and Where2comm [7]. Intuitively, How2comm outperforms previous methods in the real-world (DAIR-V2X [52]) and simulated datasets, demonstrating the superiority of our model and its robustness to various realistic noises. In particular, the SOTA performance of AP@0.7 on the DAIR-V2X and OPV2V is improved by  $8.4\%$  and  $7.0\%$ , respectively. Compared to previous per-agent/location message fusion efforts [14, 16, 33, 40], How2comm simultaneously considers the decoupled spatial semantics and temporal dynamics among agents, resulting in a more precise perception.
Comparison of Communication Volume. Figure 4 presents the performance comparison results with distinct bandwidth consumptions. Concretely, the orange and blue curves denote the detection precision of our How2comm and Where2comm under varying communication volumes, respectively. (i) How2comm keeps superior to Where2comm across all the communication choices, i.e., How2comm achieves a better performance-bandwidth trade-off with spatial-channel filtering than Where2comm. (ii) Moreover, our framework seeks comparable performances as the SOTAs by consuming less bandwidth. The noteworthy improvements demonstrate that the proposed communication mechanism filters invalid semantics and maintains performance by mutual information maximization.
Robustness to Localization and Heading Errors. We verify the detection performance of How2comm under varying pose errors of collaborators in Figures 5 and 6 following the noise settings in [40]. Specifically, the localization and heading errors are sampled from Gaussian distributions with a standard deviation of  $\sigma_{xyz} \in [0, 0.5]$  m and  $[0^\circ, 1.0^\circ]$ , respectively. As shown in the figures, the performance of all intermediate collaboration models consistently deteriorates due to feature map misalignment as the pose errors increase. Noticeably, How2comm is superior to the previous SOTA models and No Fusion across three datasets under all error levels, while some models (e.g., V2VNet and When2com) are even weaker than No Fusion when the error exceeds  $0.2\mathrm{m}$  and  $0.2^\circ$  on the DAIR-V2X dataset. This comparison demonstrates the robustness of How2comm against collaboration pose noises. One reasonable explanation is that our framework captures perceptually critical and holistic information across heterogeneous agents via the tailored STCFoer.
Robustness to Transmission Delay. As noted in Figure 1(b), temporal asynchrony due to transmission delay results in two-sided fusion errors and harms the collaboration procedure. We analyze the sensitivity of existing models to varying delays (i.e., from 0 to  $400\mathrm{ms}$ ) in Figure 7. Noticeably, all the intermediate fusion methods inevitably degrade with increasing transmission delay due to misleading feature matching. Nevertheless, How2comm maintains higher precision than SOTAs at all delay levels across all three datasets and improves AP@0.7 by  $7.5\%$  than No Fusion on DAIR-V2X under a severe delay  $(400\mathrm{ms})$ . This robustness to the transmission delay proves that How2comm accomplishes temporal alignment of two-sided features by predicting the collaborators' future features.
We perform thorough ablation studies on all datasets to understand the necessity of the different designs and strategies in How2comm. Table 2 shows the following vital observations.
Rationality of Communication Mechanism. (i) The spatial and channel attention queries are first removed separately to perform incomplete message filtering. The decreased performance implies that both query patterns contribute to sharing sparse yet salient features among agents. (ii) There is a significant degradation in the detection results on all datasets when the communication mechanism lacks mutual information supervision. A plausible deduction is that our supervision mitigates the loss of valuable information due to spatial-channel feature filtering.
Effect of Delay Compensation Strategy. (i) Here, we remove the scale matrix to verify its effect. The poor results show that appropriate scaling of predicted features promotes effective temporal alignment. (ii) Furthermore, the performance drop caused by the self-supervised training removal shows the importance of imposing motion representation supervision for collaborator-shared features.
Table 2: Ablation study results of the proposed components and strategies on all the datasets. "w/o" stands for the without.
Components/Strategies	DAIR-V2X	V2XSet	OPV2V
AP@0.5/0.7	AP@0.5/0.7	AP@0.5/0.7
Full Framework	62.36/47.18	84.05/67.01	85.42/72.24
Rationality of Communication Mechanism
w/o Spatial Query	61.25/46.33	83.14/66.15	84.52/71.40
w/o Channel Query	61.76/46.52	83.70/66.58	85.23/71.76
w/o Mutual Information	60.61/46.04	82.53/65.75	84.06/70.69
Effect of Delay Compensation Strategy
w/o Scale Matrix	62.17/46.86	83.61/66.63	85.08/71.54
w/o Self-supervised Training	60.55/45.77	82.84/65.78	84.20/70.36
Importance of STCFormer
w/o Temporal Cross-Attention	60.13/45.92	83.09/65.88	84.27/70.85
w/o Exclusive Spatial Attention	59.46/45.06	82.65/65.70	83.81/71.19
w/o Common Spatial Attention	61.24/46.63	83.48/66.41	84.39/71.44
w/o Adaptive Late Fusion	61.93/46.77	83.72/66.69	84.86/71.64
Impact of Keypoint Number Nv
Nv=6	61.68/46.05	82.82/66.15	84.67/71.38
Nv=9 (Default)	62.36/47.18	84.05/67.01	85.42/72.24
Nv=12	62.13/46.57	84.12/66.59	85.28/72.01
Necessity of Decoupled Design
w/o Decoupled Design	60.08/45.36	82.41/65.38	84.02/71.15
Importance of STCFormer. STCFormer is evaluated in three dimensions. (i) We first find that temporal cross-attention provides beneficial gains due to performance deterioration when discarded. It is because the meaningful temporal clues in historical frames bridge the single-frame detection gap. (ii) Then, exclusive and common spatial attention modules are removed separately to explore the impact on performance. The consistently decreased results on each dataset suggest that integrating distinct spatial semantics is indispensable for pragmatic collaboration. (iii) Finally, the adaptive late fusion is replaced by pixel-wise addition. The gain decline suggests that our fusion paradigm provides new insights for aggregating the perceptual advantages of distinct spatial semantics.
Impact of Keypoint Number  $N_{v}$ . Empirically, we set the variable number of keypoints for the decoupled spatial attention modules to perform experiments and find that 9 keypoints achieve the most competitive detection performance. Conversely, too few keypoints may cause valuable semantics loss and too many keypoints may cause performance bottlenecks due to accumulated errors. The above finding inspires us to determine the appropriateKeypoint number that samples more rich visual clues and captures more pragmatic collaboration information from collaborators.
Necessity of Decoupled Design. Ultimately, we set  $\mathcal{E}_j^t$  and  $\mathcal{C}_j^t$  to 1 while maintaining two spatial attention branches in the STCFormer to justify the decoupled design. Without explicitly defining exclusive and common features, spatial attention would indiscriminately sample the entire region due to the lack of guidance from prior perception information, which may introduce excessive collaborator noises and impede bridging to collaboration heterogeneity.
Visualization of Detection Results. To illustrate the perception performance of different models, Figure 8 shows the detection visualizations of two challenging scenarios from the DAIR-V2X
(a) V2X-ViT
(b) CoBEVT
(c) Where2comm
(d) How2comm (ours)
Figure 8: Detection visualization comparison in real-world scenarios from the DAIR-V2X dataset. Green and red boxes denote the ground truths and detection results, respectively.
(a) GT bounding boxes
Figure 9: Visualization of learned exclusive and common spatial attention maps. The red and blue point clouds are derived from the ego vehicle and infrastructure, respectively. In (b)&(c), cyan cross marker denotes the target query in the deformable cross-attention. The 9 sampled keypoints are represented by the forestgreen dots whose colors reflect the corresponding attention weights.
(b) Exclusive spatial attention
(c) Common spatial attention
dataset [52] under default noise settings. How2comm axiomatically achieves more robust and accurate detection compared to previous SOTA models, including V2X-ViT [40], CoBEVT [38], and Where2comm [7]. Concretely, our method produces more predicted bounding boxes well aligned with the ground truths. The merits may lie in two aspects. (i) Our delay compensation strategy mitigates feature misalignment due to temporal asynchrony, improving detection performance. (ii) The proposed STCFformer provides effective temporal context clues for detecting fast-moving objects and fuses meaningful information from nearby agents to compensate for the occluded perspective.
Visualization of Spatial Attention Maps. We show visualizations of exclusive and common spatial attention (ESA/CSA) in Figure 9 to justify the effectiveness of our feature decoupling philosophy. (i) Intuitively, from Figure 9(b), ESA effectively samples keypoints at the exclusive perception region from the infrastructure, providing complementary information for the ego agent to promote perception ability. (ii) In Figure 9(c), CSA mitigates the detection gap due to collaboration heterogeneity by sampling keypoints that reasonably focus on the common perception region between two agents.
This paper presents How2comm, a novel collaborative perception framework to tackle existing issues jointly. How2comm maximizes the beneficial semantics in filtered features and accomplishes temporal alignment via the feature flow estimation. Moreover, our STCFoer holistically aggregates the spatial semantics and temporal dynamics among agents. Extensive experiments on several multi-agent datasets show the effectiveness of How2comm and the necessity of all its components.
Limitation and Future Work. The current work only exploits the short-term historical frames. In future work, we plan to expand the utilization of temporal information to long-term point cloud sequences. Also, we will explore optimizing the feature flow prediction with uncertainty estimation.
Acknowledgment. This work is supported in part by the National Key R&D Program of China under Grant 2021ZD0113503 and in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDZX0103.