SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Dingkang Yang $^{1,2\dagger}$  Kun Yang $^{1\dagger}$  Yuzheng Wang $^{1}$  Jing Liu $^{1}$  Zhi Xu $^{1}$  Rongbin Yin $^{5}$  Peng Zhai $^{1,2*}$  Lihua Zhang $^{1,2,3,4*}$
$^{1}$ Academy for Engineering and Technology, Fudan University
$^{2}$ Cognition and Intelligent Technology Laboratory (CIT Lab)
<sup>3</sup>Engineering Research Center of AI and Robotics, Ministry of Education
$^{4}$  AI and Unmanned Systems Engineering Research Center of Jilin Province
$^{5}$ FAW (Nanjing) Technology Development Company Ltd
{dkyang20, kungyang20, pzhai, lihuazhang}@fudan.edu.cn
Multi-agent collaborative perception has recently received widespread attention as an emerging application in driving scenarios. Despite the advancements in previous efforts, challenges remain due to various dilemmas in the perception procedure, including communication redundancy, transmission delay, and collaboration heterogeneity. To tackle these issues, we propose How2comm, a collaborative perception framework that seeks a trade-off between perception performance and communication bandwidth. Our novelties lie in three aspects. First, we devise a mutual information-aware communication mechanism to maximally sustain the informative features shared by collaborators. The spatial-channel filtering is adopted to perform effective feature sparsification for efficient communication. Second, we present a flow-guided delay compensation strategy to predict future characteristics from collaborators and eliminate feature misalignment due to temporal asynchrony. Ultimately, a pragmatic collaboration transformer is introduced to integrate holistic spatial semantics and temporal context clues among agents. Our framework is thoroughly evaluated on several LiDAR-based collaborative detection datasets in real-world and simulated scenarios. Comprehensive experiments demonstrate the superiority of How2comm and the effectiveness of all its vital components. The code will be released at https://github.com/ydk122024/How2comm.
Precise perception of complex and changeable driving environments is essential to ensure the safety and reliability of intelligent agents [25, 46], e.g., autonomous vehicles (AVs). With the emergence of learning-based technologies, remarkable single-agent perception systems are extensively explored for several in-vehicle tasks, such as instance segmentation [19, 58] and object detection [26, 49]. Nevertheless, single-agent perception suffers from various shortcomings due to the isolated view, such as unavoidable occlusions [55], restricted detection ranges [56], and sparse sensor observations [57]. Recently, multi-agent collaborative perception [39, 54] has provided promising solutions as an emerging application for vehicle-to-vehicle/everything (V2V/X) communications. The impressive studies [7, 14, 16, 17, 31, 32, 33, 38, 40, 51] have progressively presented to aggregate valuable information and complementary perspectives among on-road agents, resulting in a more
$^{\dagger}$ Equal contributions  $\mathfrak{X}$ . The two first authors thank Runsheng Xu for providing constructive suggestions.
(a) Aligned point cloud fusion
(b) Non-aligned point cloud fusion
Figure 1: (a) and (b) show the point cloud fusion results in the absence and presence of transmission delay, respectively. (c) and (d) show the vanilla image and fused point cloud of the collaborative perception scene containing an ego vehicle (red circle) and an infrastructure, respectively.
(c) Infrastructure side image
(d) Fused point cloud
precise perception. Despite recent advancements, challenges remain due to unpreventable dilemmas, including communication redundancy, transmission delay, and collaboration heterogeneity.
Communication Redundancy. The dominant patterns for reducing communication overhead are summarized as feature compression [14, 33, 40] and spatial filtering [7, 8]. The former assumes that agents share all spatial areas indiscriminately, which dramatically wastes bandwidth. The latter overly relies on confidence maps to highlight gullible locations and fails to consider spatially holistic information. Moreover, these methods invariably cause losses of transmitted valuable information.
Transmission Delay. Figures 1(a)&(b) present the point cloud fusion results from an ego vehicle and an infrastructure in the time-synchronous and time-asynchronous cases, respectively. The inevitable transmission delay causes position misalignment of fast-moving objects within the green circles, potentially harming subsequent collaboration performance. Although several delay-aware strategies [13, 40, 53] are proposed to tackle this issue, they either suffer from performance bottlenecks [13, 40] or introduce massive computation costs [53], leading to sub-optimal solutions.
Collaboration Heterogeneity. Figures 1(c)&(d) show the typical collaboration scenario involving two agents and the fused point cloud. Intuitively, LiDAR configuration discrepancies (e.g., different LiDAR densities, distributions, reflectivities, and noise interference) across agents potentially cause collaboration heterogeneity within the feature space. In this case, the orange box contains the common perception region of both agents, which facilitates bridging the feature-level gap caused by sensor configuration discrepancies [37, 40]. The magenta box contains the exclusive perception region of the infrastructure, which provides complementary information for the ego vehicle and compensates for the occluded view. Fusing valuable spatial semantics from these two perception regions facilitates comprehensive and pragmatic perception. However, most previous methods [14, 16, 32, 33, 40] integrate collaborator-shared features via per-agent/location message fusion to enhance ego representations, whose collaboration processes could be vulnerable since the advantages of distinct perception regions from heterogeneous agents are not considered holistically. Moreover, the current single-frame perception paradigm faces the challenges of 3D point cloud sparsity and localization errors, increasing the difficulty of building a robust multi-agent perception system.
Motivated by the above observations, we propose How2comm, an end-to-end collaborative perception framework to address the existing issues jointly. Through three novel components, How2comm advances towards a reasonable trade-off between perception performance and communication bandwidth. Specifically, (i) we first design a mutual information-aware communication mechanism to maximally preserve the beneficial semantics from vanilla characteristics in the transmitted messages of collaborators. In this case, spatial-channel message filtering is introduced to determine how to use less bandwidth for efficient communication. (ii) Second, we present a flow-guided delay compensation strategy to predict the future features of collaborators by mining contextual dependencies in sequential frames. Our ingenious strategy determines how to dynamically compensate for the delay's impact and explicitly accomplish temporal alignment. (iii) Furthermore, we construct a spatio-temporal collaboration transformer (STCFormer) module to integrate perceptually comprehensive information from collaborators and temporally valuable clues among agents. Our unified transformer structure determines how to achieve pragmatic collaboration, contributing to a more robust collaborative perception against localization errors and feature discrepancies. How2comm is systematically evaluated on various collaborative 3D object detection datasets, including DAIR-V2X [52], V2XSet [40], and OPV2V [41]. Quantitative experiments demonstrate that our framework significantly outperforms previous state-of-the-art (SOTA) methods under the bandwidth-limited noisy setting. Systematic analyses confirm the robustness of How2comm against distinct collaboration noises.
Figure 2: How2comm overview. The projected features of all agents are obtained via a shared encoder. Upon receiving requests  $\{\mathcal{R}_{i,s}^{t_0},\mathcal{R}_{i,c}^{t_0}\}$  from the ego vehicle, the collaborator (i.e., infrastructure) shares the sparse feature  $\tilde{\pmb{F}}_j^{t_0}$ , feature flow  $\mathcal{O}_j^{t_0}$ , and scale matrix  $\mathcal{S}_j^{t_0}$  via the message filtering and flow generator. After that, the ego vehicle predicts the future feature  $\pmb{Z}_j^t$  and adopts the importance maps  $\{\mathcal{I}_i^t,\mathcal{I}_j^t\}$  to get the exclusive and common maps, which decouple  $\pmb{Z}_j^t$  into  $\pmb{Z}_{j,e}^{t}$  and  $\pmb{Z}_{j,c}^{t}$ . Finally, our STCFoer m fuses the temporal context and decoupled spatial features to output  $\mathcal{F}_i^t$  used for detection.
Multi-Agent Communication. Benefiting from rapid advances in learning-based technologies [12, 21, 22, 23, 24, 42, 43, 44, 45, 47, 48, 50], communication has played an essential role in constructing robust and stable multi-agent systems. Although early works provided heuristic insights into information sharing among different agents through predefined protocols and centralized paradigms [4, 28, 29], these efforts are typically difficult to generalize into challenging scenarios. Recently, several learning-driven communication strategies have been proposed to accommodate diverse scenario applications. For instance, Vain [6] utilized an attentional neural structure to specify what information needs to be shared in agent interactions. ATOC [9] introduced the recurrent unit to decide whom the agent communicates with by receiving local observations and action intentions from other agents. TarMAC [2] designed a reinforcement learning-oriented architecture to learn communication from task-specific rewards. In comparison, we focus on LiDAR-based collaborative 3D object detection tasks. For more challenging driving scenarios, we design the mutual information supervision and attention-guiding mechanism to achieve efficient communication across agents.
Collaborative Perception. Collaborative perception is only in its infancy as a promising application of multi-agent systems. Several impressive approaches have been designed to facilitate the overall perception performance of AVs. The mainstream works [7, 14, 16, 33, 37, 38, 40] followed the intermediate collaboration pattern to balance average precision and bandwidth overhead. Specifically, When2com [16] introduced a handshake mechanism to determine when to communicate with collaborators. V2VNet [33] employed a fully connected graph network to aggregate feature representations shared by agents. After that, DiscoNet [14] proposed a knowledge distillation framework to supervise the intermediate collaboration through an early collaboration of full views. V2X-ViT [40] designed distinct attention interactions to facilitate adaptive information fusion among heterogeneous agents. Where2comm [7] aimed to transmit perceptually critical information via sparse spatial confidence maps. However, these methods invariably ignore valuable historical clues and lead to sub-optimal solutions. In this paper, we propose a novel collaboration transformer to jointly capture spatial semantics and temporal dynamics among agents, resulting in a more pragmatic collaboration.
In this paper, we seek to develop a communication-efficient and collaboration-pragmatic multi-agent system to enhance the perception ability of the ego agent. Figure 2 illustrates the proposed system framework, which accommodates different agents (e.g., AVs and infrastructures). Consider  $N$  agents in a driving scene, let  $\pmb{X}_i^t$  be the local point cloud observation of the  $i$ -th agent (i.e., ego agent) and
$\mathcal{V}_i$  be the corresponding ground-truth supervision. The objective of How2comm is to maximize the LiDAR-based 3D detection performance  $\Re(\cdot)$  under a total communication budget  $B$ :
$$
\Re (B) = \underset {\theta , \tilde {\boldsymbol {F}} _ {j}} {\arg \max } \sum_ {i} ^ {N} \tilde {\partial} \left(\Psi_ {\theta} \left(\boldsymbol {X} _ {i} ^ {t}, \left\{\tilde {\boldsymbol {F}} _ {j} ^ {t _ {0}} \right\} _ {j = 1} ^ {N}, \mathcal {Y} _ {i}\right), \quad \text {s . t .} \sum_ {j} \left| \tilde {\boldsymbol {F}} _ {j} ^ {t _ {0}} \right| \leq B, \right. \tag {1}
$$
where  $\tilde{\partial} (\cdot ,\cdot)$  denotes the perception evaluation metric and  $\Psi_{\theta}$  is the perception system parameterized by  $\theta$ .  $\tilde{F}_j^{t_0}$  is the message transmitted from the  $j$ -th agent to the  $i$ -th agent at time delay  $\tau$ -aware moment  $t_0$ , where  $t_0 = t - \tau$ . The remainder of Section 3 details the major components.
3.2 Metadata Conversion and Feature Extraction
In the initial stage of collaboration, we build a communication graph [14, 40] where one agent is selected as the ego agent and the other connected agents act as collaborators. Upon receiving the broadcast metadata (e.g., poses and extrinsic) from the ego agent, the collaborators project their local observations to the ego agent's coordinate system. Moreover, ego-motion compensation [35] synchronizes each agent's historical frames. The shared PointPillar [11] encoder  $f_{enc}(\cdot)$  converts the point cloud of the  $i$ -th agent at timestamp  $t$  into the bird's-eye-view (BEV) features as  $\pmb{F}_i^t = f_{enc}(\pmb{X}_i^t) \in \mathbb{R}^{H \times W \times C}$ , where  $H, W, C$  denote height, width, and channel, respectively.
3.3 Mutual Information-aware Communication
Previous attempts to reduce the required transmission bandwidth relied heavily on autoencoders [14, 33, 40] or confidence maps [7, 8], which are one-sided as they only consider information compression over spatial positions or channels. To this end, we design a mutual information-aware communication (MIC) mechanism to select the most informative messages from space and channels to save precious bandwidth. MIC consists of two core parts as follows.
Spatial-channel Message Filtering. Stemming from solid evidence in signal picking, we first introduce the CBAM [34]-like spatial-channel attention queries to assist each agent in sharing their salient features. The spatial query  $\mathcal{A}_{i,s}^{t_0} = \sigma (\omega_{3*3}[\varphi_a(\pmb {F}_i^{t_0});\varphi_m(\pmb {F}_i^{t_0})])\in \mathbb{R}^{H\times W\times 1}$  reflects what spatial locations on delayed feature  $F_{i}^{t_{0}}$  are informative, where  $[\cdot ;\cdot ]$  is the concatenation,  $\sigma$  is the sigmoid activation,  $\varphi_{a / m}(\cdot)$  denote average and max pooling functions, and  $\omega_{3*3}(\cdot)$  is the 2D  $3\times 3$  convolution operation. The channel query  $\mathcal{A}_{i,c}^{t_0} = \sigma (\omega_{1*1}(\varphi_a(\pmb {F}_i^{t_0})) + \omega_{1*1}(\varphi_m(\pmb {F}_i^{t_0})))\in \mathbb{R}^{1\times 1\times C}$  reflects which channels in  $F_{i}^{t_{0}}$  are semantically meaningful.  $\varphi_{a / m}(\cdot)$  in the spatial and channel queries are applied to the channel and spatial dimensions, respectively. Then, the ego agent indicates the supplementary messages required to improve local perception performance by broadcasting request queries  $\mathcal{R}_{i,s}^{t_0} / \mathcal{R}_{i,c}^{t_0} = 1 - \mathcal{A}_{i,s}^{t_0} / \mathcal{A}_{i,c}^{t_0}$ . The  $j$ -th collaborator then aggregates the requests with its attention queries to obtain a spatial-channel binary message filtering matrix as follows:
$$
\mathcal {M} _ {j} ^ {t _ {0}} = f _ {s e l} \left(\omega_ {1 * 1} \left[ \mathcal {R} _ {i, s} ^ {t _ {0}}; \mathcal {A} _ {j, s} ^ {t _ {0}} \right] \odot \omega_ {1 * 1} \left[ \mathcal {R} _ {i, c} ^ {t _ {0}}; \mathcal {A} _ {j, c} ^ {t _ {0}} \right]\right) \in \{0, 1 \} ^ {H \times W \times C}, \tag {2}
$$
where  $f_{sel}(\cdot)$  is a threshold-based selection function and  $\odot$  is the element-wise multiplication. Ultimately, the selected feature map is obtained as  $\tilde{\pmb{F}}_j^{t_0} = \pmb{F}_j^{t_0} \odot \mathcal{M}_j^{t_0}$ , which provides spatial-channel sparse, yet perceptually critical information.
Mutual Information Maximization Supervision. Most existing works ignore the potential loss of valuable information due to feature compression. To overcome this dilemma, we maximally sustain the local critical semantics in the corresponding vanilla feature  $F_{j}^{t_{0}}$  on the selected regions of the transmitted features  $\tilde{F}_{j}^{t_{0}}$  by mutual information estimation. Since we only focus on maximizing the mutual information rather than getting its precise value, a stable estimator [5] is utilized to build the objective supervision based on the Jensen-Shannon divergence. Formally, the mutual information between two random variables  $\mathcal{X}$  and  $\mathcal{Z}$  is estimated as follows:
$$
\hat {\boldsymbol {I}} _ {\varrho} ^ {(J S D)} (\mathcal {X}, \mathcal {Z}) = \mathbb {E} _ {p (x, z)} [ - l o g (1 + e ^ {- T _ {\varrho} (x, z)}) ] - \mathbb {E} _ {p (x) p (z)} [ l o g (1 + e ^ {T _ {\varrho} (x, z)}) ], \tag {3}
$$
where  $T_{\varrho}:\mathcal{X}\times \mathcal{Z}\to \mathbb{R}$  is a statistics network parameterized by  $\varrho$ . In our case, the mutual information supervision of all collaborators within the communication link is defined as follows:
$$
\mathcal {L} _ {m u l} = \frac {1}{N - 1} \sum_ {j \in \{1, \dots , N \}, j \neq i} \hat {\boldsymbol {I}} _ {\varrho} ^ {(J S D)} \left(\boldsymbol {F} _ {j} ^ {t _ {0}}, \tilde {\boldsymbol {F}} _ {j} ^ {t _ {0}}\right). \tag {4}
$$
Figure 3: (a) The overall architecture of the proposed STCFoermer. (b) and (c) show the structure of the temporal cross-attention (TCA) and exclusive spatial attention (ESA) modules, respectively, which contain the computational flow and the major dimensional transformations of features.
3.4 Flow-guided Delay Compensation
We present a flow-guided delay compensation (FDC) strategy to eliminate the two-sided fusion error in feature-level collaboration due to temporal asynchrony. Existing solutions relied on received historical features [13] and produced large errors under severe delay [53], leading to performance bottlenecks. To tackle the issues, we adopt the philosophy of feature flow to predict the collaborators' future features for temporal alignment with the ego representation. The details are as follows.
Flow Generation and Warping. Due to the uncertain delay between the ego agent and collaborators, FDC predicts the feature flow  $\mathcal{O}_j^{t_0}$  for a fixed time interval and scale matrix  $S_{j}^{t_{0}}$  based on the  $j$ -th agent's historical frames. Specifically, as Figure 2 shows, features  $\{\pmb {F}_j^{t_0 - k},\dots,\pmb {F}_j^{t_0}\}$  are concatenated in channel dimension and entered to a generator  $f_{flow}(\cdot)$  to output  $\mathcal{O}_j^{t_0}\in \mathbb{R}^{H\times W\times 2}$  and  $S_j^{t_0}\in$ $\mathbb{R}^{H\times W\times 1}$ . Then the  $j$ -th agent sends  $\{\mathcal{O}_j^{t_0},S_j^{t_0}\}$  with prediction ability and sparse feature  $\tilde{\pmb{F}}_j^{t_0}$  to the ego agent. The ego agent estimates predicted collaborator features as  $Z_{j}^{t} = f_{warp}(\tilde{F}_{j}^{t_{0}},(t - t_{0})\cdot$ $\mathcal{O}_j^{t_0})\odot S_j^{t_0}$ , where  $f_{warp}(\cdot)$  is the bilinear warping function applied to all positions and channels [60, 61], and  $\cdot$  is the scalar multiplication. The temporally aligned features are passed to the STCFoermer.
Self-supervised Training Pattern. Self-supervised learning is employed to train the flow generator  $f_{flow}(\cdot)$  since the existing datasets [40, 41, 52] lack the motion annotations. Concretely, we first form the training group  $\{F_j^{t_0 - k},\dots ,F_j^{t_0},F_j^t\}$ , where  $\{F_j^{t_0 - k},\dots ,F_j^{t_0}\}$  is a continuous feature sequence, and  $F_{j}^{t}$  is considered as the ground truth feature. Subsequently, we predict the feature  $Z_{j}^{t}$  as  $Z_{j}^{t} = f_{warp}(F_{j}^{t_{0}},(t - t_{0})\cdot \mathcal{O}_{j}^{t_{0}})\odot S_{j}^{t_{0}}$ . Since the optimization objective of  $f_{flow}(\cdot)$  is to increase the similarity between  $F_{j}^{t}$  and  $Z_{j}^{t}$ , we formulate the self-supervised loss function  $\mathcal{L}_{flow}$  based on the cosine similarity [53] as follows:
$$
\mathcal {L} _ {\text {f l o w}} = \frac {1}{N - 1} \sum_ {j \in \{1, \dots , N \}, j \neq i} (1 - \frac {\boldsymbol {F} _ {j} ^ {t} \odot \boldsymbol {Z} _ {j} ^ {t}}{\| \boldsymbol {F} _ {j} ^ {t} \| _ {F} ^ {2} \cdot \| \boldsymbol {Z} _ {j} ^ {t} \| _ {F} ^ {2}}), \tag {5}
$$
where  $\| \cdot \| _F^2$  is the squared Frobenius norm.
3.5 Spatio-Temporal Collaboration Transformer
To efficiently mitigate collaboration heterogeneity, we propose a spatio-temporal collaboration transformer (STCFormer) to jointly integrate the decoupled spatial semantics and temporal dynamics among agents. From Figure 3(a), the core contributions of STCFormer lie in the following three customized modules, where the other basic components follow the choice of the vanilla transformer [30].
Temporal Cross-Attention. To bridge the detection gap regarding fast-moving objects due to point cloud sparsity, we capture historical context clues across agents to reinforce the current
representation via a temporal cross-attention (TCA) module. The core is to perform Query-Key-Value-like attention operations by projecting the ego feature  $\pmb{F}_i^t$  and merged temporal features  $\pmb{E} = [\tilde{\pmb{F}}_j^{t_0},\pmb {F}_i^{t_0},\dots,\pmb {F}_i^t]$  into different subspaces via three 2D convolutional layers  $\omega_{3*3}(\cdot)$ . In Figure 3(b), the branches of Key&Value  $E_{k / v}\gets \omega_{1}(\varphi_{a}(\omega_{3*3}(\pmb {E})))\odot \omega_{3*3}(\pmb {E})$  share the same structure but separate weights, where a 1D temporal convolution  $\omega_{1}(\cdot)$  with global average pooling  $\varphi_{a}(\cdot)$  provides temporal correlations.  $\varphi_{a}(\cdot)$  is applied to the spatial dimension to shrink the feature map. The computation of TSA can be shown as:
$$
\boldsymbol {H} _ {i} ^ {t} = \omega_ {3 * 3} \left(\operatorname {s o f t m a x} \left(\omega_ {3 * 3} \left(\boldsymbol {F} _ {i} ^ {t}\right) \otimes \boldsymbol {E} _ {k} ^ {T}\right) \otimes \boldsymbol {E} _ {v}\right) \in \mathbb {R} ^ {H \times W \times C}. \tag {6}
$$
Decoupled Spatial Attention. To comprehensively integrate the distinct spatial semantics from the collaborators, we facilitate pragmatic message fusion with a feature decoupling perspective inspired by the observation in Figure 1(d). Formally, an importance generator  $f_{gen}(\cdot)$  is employed to generate the importance maps of the ego feature  $\pmb{F}_i^t$  and the estimated collaborator feature  $Z_j^t$  as  $\mathcal{I}_i^t / \mathcal{I}_j^t = \sigma(\varphi_m(f_{gen}(F_i^t / Z_j^t))) \in [0,1]^{H \times W}$ . The importance maps reflect the perceptually critical level of each pixel in the features. Then, the  $j$ -th agent spatially decouples the feature  $Z_j^t$  via candidate maps  $\mathcal{E}_j^t = (1 - \mathcal{I}_i^t) \odot \mathcal{I}_j^t$  and  $\mathcal{C}_j^t = \mathcal{I}_i^t \odot \mathcal{I}_j^t$ . Intuitively,  $\mathcal{E}_j^t$  and  $\mathcal{C}_j^t$  depict the collaborators' exclusive and common perception regions relative to the ego agent, respectively. The exclusive and common collaborator features are obtained as  $Z_{j,e}^t = f_{sel}(\mathcal{E}_j^t) \odot Z_j^t$  and  $Z_{j,c}^t = f_{sel}(\mathcal{C}_j^t) \odot Z_j^t$ .