SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sizhe Wei $^{1\dagger}$  Yuxi Wei $^{1\dagger}$  Yue Hu $^{1}$  Yifan Lu $^{1}$  Yiqi Zhong $^{2}$  Siheng Chen $^{1,3*}$  Ya Zhang $^{1,3*}$
$^{1}$  Cooperative Medianet Innovation Center, Shanghai Jiao Tong University  $^{2}$  University of Southern California  $^{3}$  Shanghai AI Laboratory  $^{1}$  {sizhewei, wyx3590236732, 18671129361, yifan_lu}@sjtu.edu.cn {sihengc, ya_zhang}@sjtu.edu.cn  $^{2}$  yiqizhon@usc.edu
Collaborative perception can substantially boost each agent's perception ability by facilitating communication among multiple agents. However, temporal asynchrony among agents is inevitable in the real world due to communication delays, interruptions, and clock misalignments. This issue causes information mismatch during multi-agent fusion, seriously shaking the foundation of collaboration. To address this issue, we propose CoBEVFlow, an asynchronous-robust collaborative perception system based on bird's eye view (BEV) flow. The key intuition of CoBEVFlow is to compensate motions to align asynchronous collaboration messages sent by multiple agents. To model the motion in a scene, we propose BEV flow, which is a collection of the motion vector corresponding to each spatial location. Based on BEV flow, asynchronous perceptual features can be reassigned to appropriate positions, mitigating the impact of asynchrony. CoBEVFlow has two advantages: (i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous time stamps without discretization; and (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, avoiding additional noises. To validate CoBEVFlow's efficacy, we create IRregular V2V(IRV2V), the first synthetic collaborative perception dataset with various temporal asynchronies that simulate different real-world scenarios. Extensive experiments conducted on both IRV2V and the real-world dataset DAIR-V2X show that CoBEVFlow consistently outperforms other baselines and is robust in extremely asynchronous settings. The code is available at https://github.com/MediaBrain-SJTU/CoBEVFlow.
Multi-agent collaborative perception allows agents to exchange complementary perceptual information through communication. This can overcome inherent limitations of single-agent perception, such as occlusion and long-range issues. Recent studies have shown that collaborative perception can substantially boost the performance of perception systems [1-5] and has great potential for a wide range of real-world applications, such as multi-robot automation system [6, 7], vehicle-to-everything-communication-aided autonomous driving[8, 9] and multi-UAVs (unmanned aerial vehicles) [10-12]. As an emerging area, the study of collaborative perception has many challenges to be tackled, such as high-quality datasets [13-15], model-agnostic and task-agnostic formulation [16] and the robustness to pose errors [17] and adversarial attacks [18, 19].
$^{\dagger}$ Equal contribution. \*Corresponding author.
However, a vast majority of existing works do not seriously account for the harsh realities of real-world communication among agents, such as congestion, heavy computation, interruptions, and the lack of calibration. These factors introduce delays or misalignments that severely impact the reliability and quality of information exchange among agents. Some prior works have touched upon the issue of communication latency. For instance, V2VNet [4] and V2X-ViT [5] incorporated the delay time as an input for feature compensation. However, they only account for a single frame without leveraging historical frames, making them inadequate for high-speed scenarios (above  $20\mathrm{m / s}$ ) or high-latency scenarios (above 0.3s) scenarios. Meanwhile, SyncNet [20] uses historical features to predict the complete feature map at the current timestamp [21]. Nevertheless, this RNN-based
method assumes equal time intervals for its input, causing failures when delays are irregular. Overall, previous works have not addressed the issues raised by common irregular time delays, rendering existing collaborative perception systems can never reach their full potential in real-world scenarios.
To fill the research gap, we specifically formulate a setting of asynchronous collaborative perception; see Fig. 1 for a visual demonstration. Here asynchrony indicates that the time stamps of the collaboration messages from other agents are not aligned and the time interval of two consecutive messages from the same agent is irregular. Due to the universality and inevitability of temporal asynchrony in practical applications, handling this setting is critical to the further development of collaborative perception. To address this, we propose CoBEVFlow, an asynchrony-robust collaborative perception system based on bird's eye view (BEV) flow. The key idea is to align perceptual information from other agents by compensating relative motions. Specifically, CoBEVFlow uses historical frames to estimate a BEV flow map, which encodes the motion information in each grid cell. Using the BEV flow map, CoBEVFlow can reassign asynchronous perceptual features to appropriate spatial locations, which aligns perceptual features on the time dimension, mitigating the impact caused by asynchrony. The proposed CoBEVFlow has two major advantages: i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous timestamps without discretization; ii) CoBEVFlow better adheres to the essence of compensation, which is to move features to the designated spatiotemporal positions. Not like SyncNet[20] that regenerates features, this motion-guided position adjustment fundamentally prevents introducing extra noises to the features.
When validating the effectiveness of CoBEVFlow, we noticed that there is no appropriate collaborative perception dataset that contains asynchronous samples. To facilitate research on asynchronous collaborative perception, we create IRregular V2V(IRV2V), the first synthetic asynchronous collaborative perception dataset with irregular time delays, simulating various real-world scenarios. Sec. 5 show the experiment results and analysis on both IRV2V and a real-world dataset DARI-V2X[14]. Results show that CoBEVFlow consistently achieves the best compensation performance across various latencies. When the expected latency on the IRV2V dataset is set to  $500\mathrm{ms}$ , CoBEVFlow outperforms other methods by more than  $18.9\%$ . In the case of a  $300\mathrm{ms}$  latency with an additional  $200\mathrm{ms}$  disturbance, the decrease in AP@0.50 is only  $0.25\%$ .
2.1 Collaborative Perception
Factors such as limited sensor fields of view and physical environmental occlusions can negatively impact perception tasks for individual agents[22, 23]. To address the aforementioned challenges, collaborative perception based on multi-agent systems has emerged as a promising solution [2, 3, 5, 1, 24-26]. It can enhance the performance of perception tasks by leveraging the exchange of information among different agents within the same scenario. V2VNet uses multi-round message passing via graph neural networks to achieve better perception and prediction performance[4]; DiscoNet adopts knowledge distillation to take advantage of both early and intermediate collaboration[9]; V2X-ViT proposes a heterogeneous multi-agent attention module to aggregate information from heterogeneous
Asynchronous communication
Perception results under synchronous communication
Figure 1: Illustration of asynchronous collaborative perception and the perception result w.o./w. CoBEVFlow. Red boxes are detection results and green boxes are the ground-truth.
, causing failures when delays are irregular. Overall, raised by common irregular time delays, rendering over reach their full potential in real-world scenarios.
agents; Where2comm[1] introduces a spatial confidence map which achieves pragmatic compression and improves perception performance with limited communication bandwidth.
As unideal communication is an inevitable issue that negatively impacts the performance and application of collaborative perception, some methods have been studied for robust collaborative perception. V2VNet[4] utilizes a convolutional neural network to learn how to compensate for communication delay by taking time information and relative pose as input; V2X-ViT[5] designs a Delay-aware Positional Encoding module to learn the influence caused by latency, but these methods do not consider the historical temporal information for compensation. SyncNet[20] uses historical multi-frame information and compensates for the current time by Conv-LSTM[21], but its compensation for the whole feature map leads noises to the feature channels, and RNN-based framework can not handle temporal irregular inputs. This work formulates asynchronous collaborative perception, which considers real-world communication asynchrony.
2.2 Time-Series Forecasting
Time series analysis aims to extract temporal information of variables, giving rise to numerous downstream tasks. Time series forecasting leverages historical sequences to predict observations of variables in future time periods. Classical methods for time series forecasting include the Fourier transform[27], autoregressive models[28], and Kalman filters[29]. In the era of deep learning, a plethora of RNN-based and attention-based methods have emerged to model sequence input and address time series prediction problems[30-32]. The issue of sampling irregularity, which may arise due to physical interference or device issues, calls for irregularity-robust time series analysis systems. Approaches such as mTAND[33], IP-Net[34], and  $\mathrm{DGM}^2$  [35] employ imputation-based techniques to tackle irregular sampling, which estimate the observation or hidden embedding of series at regular timestamps and then apply the methods designed for regular time series analysis, while SeFT[36] and Raindrop[37] utilize operations that are insensitive to sampling intervals for processing irregularly sampled data. In this work, we consider each collaboration message as one irregular sample and use irregular time series forecasting to estimate the BEV flow map for asynchronous feature alignment.
Consider  $N$  agents in a scene, where each agent can send and receive collaboration messages from other agents, and store  $k$  historical frames of messages. For the  $n$ th agent, let  $\mathcal{X}_n^{t_n^i}$  and  $\mathcal{Y}_n^{t_n^i}$  be the raw observation and the perception ground-truth at time current  $t_n^i$ , respectively, where  $t_n^i$  is the  $i$ -th timestamp of agent  $n$ , and  $\mathcal{P}_{m\to n}^{t_j^i}$  be the collaboration message sent from agent  $m$  at time  $t_m^j$ . The key of the asynchronous setting is that the timestamp of each collaboration message  $t_n^i \in \mathbb{R}$  is a continuous value, those messages from other agents are not aligned,  $t_m^i \neq t_n^i$ , and the time interval between two consecutive timestamps  $t_n^{i-1} - t_n^i$  is irregular. Therefore, each agent has to encounter collaboration messages from other agents sent at arbitrary times. Then, the task of asynchronous collaborative perception is formulated as:
$$
\max  _ {\theta , \mathcal {P}} \sum_ {n = 1} ^ {N} g \left(\widehat {\mathbf {Y}} _ {n} ^ {t _ {n} ^ {i}}, \mathbf {Y} _ {n} ^ {t _ {n} ^ {i}}\right) \tag {1}
$$
$$
\text {s u b j e c t} \hat {\mathbf {Y}} _ {n} ^ {t _ {n} ^ {i}} = \mathbf {c} _ {\theta} \left(\mathcal {X} _ {n} ^ {t _ {n} ^ {i}}, \left\{\mathcal {P} _ {m \rightarrow n} ^ {t _ {m} ^ {j}}, \mathcal {P} _ {m \rightarrow n} ^ {t _ {m} ^ {j - 1}}, \dots , \mathcal {P} _ {m \rightarrow n} ^ {t _ {m} ^ {j - k + 1}} \right\} _ {m = 1} ^ {N}\right),
$$
where  $g(\cdot, \cdot)$  is the perception evaluation metric,  $\widehat{\mathbf{Y}}_n^{t_n^i}$  is the perception result of agent  $n$  at time  $t_n^i$ ,  $\mathbf{c}_{\theta}(\cdot)$  is the collaborative perception network with trainable parameters  $\theta$ , and  $t_m^{j-k+1} < t_m^{j-k+2} < \dots < t_m^j \leq t_n^i$ . Note that: i) when the collaboration messages from other agents are all aligned and the time interval between two consecutive timestamps is regular; that is,  $t_m^i = t_n^i$  for all agent's pairs  $m, n$ , and  $t_n^i - t_n^{i-1}$  is a constant for all agents  $n$ , the task degenerates to standard well-synchronized collaborative perception; and ii) when the collaboration messages from other the agents are not aligned, yet the time interval between two consecutive timestamps is regular; that is,  $t_m^i \neq t_n^i$  and  $t_n^i - t_n^{i-1}$  is a constant, the task degenerates to the setting in SyncNet [20].
Given such irregular asynchrony, the performances of collaborative perception systems would be significantly degraded since features from asynchronized timestamps differ from the actual current features, and using asynchronized features may contain erroneous information during the perception process. In the next section, we will introduce CoBEVFlow to address this critical issue.
Figure 2: System overview. Message packing process prepares ROI and sparse features as the message for efficient communication and BEV flow map generation. Message fusion process generates and applies BEV flow map for compensation, and fuses the features at the current timestamp from all agents.
4 CoBEVFlow: Asynchrony-Robust Collaborative Perception System
This section proposes an asynchrony-robust collaborative perception system, CoBEVFlow. Figure 2 overviews the overall scheme of CoBEVFlow. We introduce the overall framework of the CoBEVFlow system in Sec. 4.1. The details of three key modules of CoBEVFlow can be found in Sec. 4.2-4.4. Sec. 4.5 demonstrates the training details and loss functions of the whole system.
4.1 Overall architecture
The problem of asynchrony results in the misplacements of moving objects in the collaboration messages. That is, the collaboration messages from multiple agents would record various positions for the same moving object. The proposed CoBEVFlow addresses this issue with two key ideas: i) we use a BEV flow map to capture the motion in a scene, enabling motion-guided reassigning asynchronous perceptual features to appropriate positions; and ii) we generate the region of interest(ROI) to make sure that the reassignment only happens to the areas that potentially contain objects. By following these two ideas, we eliminate direct modification of the features and keep the background feature unaltered, effectively avoiding unnecessary noise in the learned features.
Mathematically, let the  $n$ -th agent be the ego agent and  $\mathbf{X}_n^{t_n^i}$  be its raw observation at the  $i$ -th timestamp of agent  $n$ , denoted as  $t_n^i$ . The proposed asynchrony-robust collaborative perception system CoBEVFlow is formulated as follows:
$$
\mathbf {F} _ {n} ^ {t _ {n} ^ {i}} = f _ {\mathrm {e n c}} \left(\mathbf {X} _ {n} ^ {t _ {n} ^ {i}}\right), \tag {2a}
$$
$$
\widetilde {\mathbf {F}} _ {n} ^ {t _ {n} ^ {i}}, \mathcal {R} _ {n} ^ {t _ {n} ^ {i}} = f _ {\text {r o i ＿ g e n}} \left(\mathbf {F} _ {n} ^ {t _ {n} ^ {i}}\right), \tag {2b}
$$
$$
\mathbf {M} _ {m} ^ {t _ {m} ^ {j} \rightarrow t _ {n} ^ {i}} = f _ {\text {f l o w} - \text {g e n}} \left(t _ {n} ^ {i}, \left\{\mathcal {R} _ {m} ^ {t _ {m} ^ {q}} \right\} _ {q = j - k + 1, j - k + 2, \dots , j}\right), \tag {2c}
$$
$$
\widehat {\mathbf {F}} _ {m} ^ {t _ {n} ^ {i}} = f _ {\text {w a r p}} \left(\widetilde {\mathbf {F}} _ {m} ^ {t _ {m} ^ {j}}, \mathbf {M} _ {m} ^ {t _ {m} ^ {j} \rightarrow t _ {n} ^ {i}}\right), \tag {2d}
$$
$$
\widehat {\mathbf {H}} _ {n} ^ {t _ {n} ^ {i}} = f _ {\text {a g g}} \left(\widetilde {\mathbf {F}} _ {n} ^ {t _ {n} ^ {i}}, \left\{\widehat {\mathbf {F}} _ {m} ^ {t _ {n} ^ {i}} \right\} _ {m \in \mathcal {N} _ {n}}\right), \tag {2e}
$$
$$
\widehat {\mathbf {Y}} _ {n} ^ {t _ {n} ^ {i}} = f _ {\mathrm {d e c}} \left(\widehat {\mathbf {H}} _ {n} ^ {t _ {n} ^ {i}}\right), \tag {2f}
$$
where  $\mathbf{F}_n^{t_n^i} \in \mathbb{R}^{H \times W \times D}$  is the BEV perceptual feature map of agent  $n$  at timestamp  $t_n^i$  with  $H, W$  the size of BEV map and  $D$  the number of channels;  $\mathcal{R}_n^{t_n^i}$  is the set of region of interest (ROI);  $\widetilde{\mathbf{F}}_n^{t_n^i} \in \mathbb{R}^{H \times W \times D}$  is the sparse version of  $\mathbf{F}_n^{t_n^i}$ , which only contains features inside  $\mathcal{R}_n^{t_n^i}$  and zeropadding outside;  $\mathbf{M}_m^{t_m^j \rightarrow t_m^i} \in \mathbb{R}^{H \times W \times 2}$  is the  $m$ -th agent's the BEV flow map that reflects each grid cell's movement from timestamp  $t_m^j$  to timestamp  $t_n^i$ ,  $\{\mathcal{R}_m^{t_m^q}\}_{q = j-k+1,j-k+2,\dots,j}$  indicates historical  $k$  ROI sets sent by agent  $m$ ;  $\widehat{\mathbf{F}}_m^{t_n^i} \in \mathbb{R}^{H \times W \times D}$  is the realigned feature map from the  $m$ -th agent's at timestamp  $t_n^i$  after motion compensation;  $\widehat{\mathbf{H}}_n^{t_n^i} \in \mathbb{R}^{H \times W \times D}$  is the aggregated features from all of the agents;  $\mathcal{N}_n$  is the collaboration neighbors of the  $n$ -th agent; and  $\widehat{\mathbf{Y}}_n^{t_n^i}$  is the final output of the system. Step 2a extracts BEV perceptual feature from observation data. Step 2b generates the ROIs for each feature map, enabling BEV flow generation in Step 2c. Now all the agents exchange their messages,
including  $\widetilde{\mathbf{F}}_n^{t_n^i}$  and the  $\mathcal{R}_n^{t_n^i}$ . Step 2c generate the BEV flow map  $\mathbf{M}_m^{t_m^j\to t_n^i}$  by leveraging historical ROIs from the same agent. Step 2d gets the estimated feature map by applying the BEV flow map to reassign the asynchronized features. Step 2e aggregates the feature maps of all agents. Finally, Step 2f outputs the final perceptual results.
Note that i) Steps 2a-2b are done before communication. Steps 2c-2f are performed after receiving the message from others. During the communication process, both sparse perceptual features and the ROI set are sent to other agents, which is communication bandwidth friendly; and ii) CoBEVFlow adopts the feature representations in bird's eye view (BEV), where the feature maps of all agents are projected to the same global coordinate system, avoiding complex coordinate transformations and supporting easier cross-agent collaboration.
The proposed asynchronous collaborative perception system has three advantages: i) CoBEVFlow can deal with asynchronous collaboration messages sent at irregular, continuous timestamps without discretization; (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, causing additional noises; and (iii) CoBEVFlow promotes robustness to asynchrony by introducing minor communication cost (ROI set).
We now elaborate on the details of Steps 2b-2e in the following subsections.
4.2 ROI generation
Given the perceptual feature map of an agent, Step 2b aims to generate a set of spatial regions of interest (ROIs) for the areas that possibly contain objects. Each ROI indicates one potential object's region in the scene. The intuition is that foreground objects are the only ones that move, while the background remains static. Therefore, using ROI can enable the subsequent BEV flow map to concentrate on critical regions and simplify the computation of the BEV flow map.
To implement, we use the structure of the object detection decoder to produce the ROIs. Given agent  $m$ 's perceptual feature map at timestamp  $t_m^j$ ,  $\mathbf{F}_m^{t_m^j}$ , the corresponding detection result is obtained as:
$$
\mathbf {O} _ {m} ^ {t _ {m} ^ {j}} = \Phi_ {\text {r o i ＿ g e n}} \left(\mathbf {F} _ {m} ^ {t _ {m} ^ {j}}\right) \in \mathbb {R} ^ {H \times W \times 7}, \tag {3}
$$
where  $\Phi_{\mathrm{roi\_gen}}(\cdot)$  is the ROI generation network with detection decoder structure, and each element  $(\mathbf{O}_m^{t_m})_{h,w} = (c,x,y,h,w,\cos \alpha ,\sin \alpha)$  represents one detected ROI with its class confidence, position, size, and orientation. We threshold the class confidence, apply non-max suppression, and obtain a set of detected boxes, whose occupied spaces form the set of ROIs,  $\mathcal{R}_m^{t_m}$ .
Based on this ROI set, we can also get a binary mask  $\mathbf{H} \in \mathbb{R}^{H \times W}$ , whose values inside ROIs are 1 and the others are 0. We then get the sparse feature map  $\widetilde{\mathbf{F}}_m^{t_m^j} = \mathbf{F}_m^{t_m^j} \odot \mathbf{H}$ , which only contains the features within ROIs. Then, agent  $m$  packs its sparse feature map  $\widetilde{\mathbf{F}}_m^{t_m^j}$  and the ROI set  $\mathcal{R}_m^{t_m^j}$  as the message and sends out for collaboration.
4.3 BEV flow map generation
After receiving collaboration messages from other agents at various timestamps, Step 2c aims to generate the BEV flow map to correct feature misalignment due to asynchrony. The proposed BEV flow map encodes the motion vector of each spatial location. The main idea of obtaining this BEV flow is to associate correlated ROIs based on a sequence of messages sent by the same collaborator. In this step, each ROI is regarded as an instance that has its own attributes generated by Step 2b. After the ROI association, we are able to compute the motion vector and further estimate the positions where the corresponding object would appear at a certain timestamp. The generation of the BEV flow map includes two key steps: adjacent timestamp' ROI matching and BEV flow estimation.
Adjacent frames' ROI matching. The purpose of adjacent frames' ROI matching is to match the ROIs in two consecutive messages sent by the same agent. The matched ROIs are essentially the same instance perceived at different timestamps. This module contains three processes: cost matrix construction, greedy matching, and post-processing. We first construct a cost matrix  $\mathbf{C} \in \mathbb{R}^{o_1 \times o_2}$ , where  $o_1$  and  $o_2$  are the numbers of ROIs in the two frames to be matched. Each element  $\mathbf{C}_{p,q}$  is the matching cost between ROI  $p$  in the earlier frame and ROI  $q$  in the later frame. To determine the value of  $\mathbf{C}_{p,q}$ , we define the vicinity of the front and rear directions as a feasible angle range for matching. We set  $\mathbf{C}_{p,q} = d_{p,q}$  when  $q$  is within the feasible angle range of  $p$ , otherwise  $\mathbf{C}_{p,q} = +\infty$ , where  $d_{p,q}$  is the Euclidean distance between the center of ROI  $p$  and  $q$ . We then use the greedy matching
strategy to search the paired ROIs. For each row  $p$ , we search the  $q$  with minimum  $\mathbf{C}_{p,q}$ , and match  $p, q$  as a pair. To avoid invalid matching, we further post-process matched pairs by removing those with excessively large values of  $C_{p,q}$ . Through these processes, we can get the matched ROI pairs in adjacent frames. For a sequence of frames, we can track each ROI's multiple locations across frames.
Figure 3: The process of the BEV flow estimation.
BEV flow estimation. We now retrieve each ROI's historical locations at a series of irregular timestamps. In this module, we use those irregular tracklets to predict the location and orientation of ROIs at the ego agent's current timestamp  $t_{n}^{i}$  and generate the corresponding BEV flow map  $\mathbf{M}_{m}^{t_{m}^{j}\rightarrow t_{n}^{i}}$ . To formulate the  $r$ -th ROI's irregular tracklet perceived by  $m$ -th agent, we extract the motion-related attributes (i.e. location and orientation) from the  $r$ -th ROI's attributes set at each timestamp. Let  $\mathbf{V}_{m,r} = \{\mathbf{v}_{r}^{t_{m}^{j}},\mathbf{v}_{r}^{t_{m}^{j - 1}},\dots ,\mathbf{v}_{r}^{t_{m}^{j - k + 1}}\}$  be a historical sequence of the  $r$ -th ROI's attributes sent by the  $m$ -th agent, where  $\mathbf{v}_{r}^{t_{m}^{j}} = (x_{r}^{t_{m}^{j}},y_{r}^{t_{m}^{j}},\alpha_{r}^{t_{m}^{j}})$ , with  $(x_{r}^{t_{m}^{j}},y_{r}^{t_{m}^{j}})$  is the 2D BEV center position and  $\alpha_{r}^{t_{m}^{j}}$  is the orientation. Note that  $\mathbf{V}_{m,r}$  is an irregularly sampled sequence due to time asynchrony. Fig 3 shows this process.