SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on  $\mathbf{V}_{m,r}$ , we now predict  $\mathbf{v}_r^{t_n}$ , which is the location and orientation of the  $r$ -th ROI at the ego agent's current timestamp. Unlike the common motion estimation, we need to handle an irregularly sampled sequence. To enable the irregularity-compatible motion estimation method, the information of the timestamp should be taken into account. Here we propose to use traditional trigonometric functions [38] for timestamp encoding, by which we map the continuous-valued timestamp  $t$  into its corresponding time code  $\mathbf{u}(t)$  through:
$$
(\mathbf {u} (t)) _ {2 e} = \sin \left(\frac {t}{1 0 0 0 0 ^ {2 e / d}}\right), \quad (\mathbf {u} (t)) _ {2 e + 1} = \cos \left(\frac {t}{1 0 0 0 0 ^ {2 e / d}}\right), \tag {4}
$$
where  $e$  is the index of temporal encoding. The timestamp information now can be input into the estimation process along with the irregularly sampled sequence and make the estimation process capable of irregularity-compatible motion estimation. We implement the estimation process using multi-head attention(MHA). The query of MHA is the time code of the target timestamp  $t_{n}^{i}$ , and the key and value both are the sum of the features of the irregularly sampled sequence and its corresponding time code set  $\mathbf{U}_k$ :
$$
\widehat {\mathbf {v}} _ {r} ^ {t _ {n} ^ {i}} = \operatorname {M H A} \left(\mathbf {u} \left(t _ {n} ^ {i}\right), \operatorname {M L P} \left(\mathbf {V} _ {m, r}\right) + \mathbf {U} _ {k}, \operatorname {M L P} \left(\mathbf {V} _ {m, r}\right) + \mathbf {U} _ {k}\right), \tag {5}
$$
where  $\hat{v}_r^{t_n}$  is the estimation of ROI's location and orientation  $v_r^{t_n}$ , MLP( $\cdot$ ) is the encoding function of the irregular historical series  $\mathbf{V}_m^r$ , and MHA( $\cdot$ ) is the multi-head attention for temporal estimation.
With the estimated location and orientation of ROIs in the ego agent's current timestamp along with the ROIs' sizes predicted by Step 2b, we calculate the motion vector at each grid cell by an affine transformation of the associated ROI's motion, constituting the whole BEV flow map  $\mathbf{M}_{m}^{t_{m}^{j}\rightarrow t_{n}^{i}}\in \mathbb{R}^{H\times W\times 2}$ . Note that the grid cells outside ROIs are zero-padded.
Compared to Syncnet [20] that uses RNNs to handle regular communication latency, the generated BEV flow map has two benefits: i) it handles irregular asynchrony via the attention-based estimation with appropriate time encoding; and ii) it facilitates the motion-guided feature warping, which avoids regenerating the entire feature channels.
4.4 Feature warp and aggregation
The BEV flow map  $\mathbf{M}_m^{t_j^i\to t_n^i}$  is applied on the sparse feature map  $\widetilde{\mathbf{F}}_m^{t_j^i}$ , which implements Step 2d. The features at each grid cell are moved to the estimated position based on  $\mathbf{M}_m^{t_j^i\to t_n^i}$ . The warping
(a) IRV2V
Figure 4: Comparison of the performance of CoBEVFlow and other baseline methods under the expectation of time interval from 0 to  $500\mathrm{ms}$ . CoBEVFlow outperforms all the baseline methods and shows great robustness under any level of asynchrony on both two datasets.
(b) DAIR-V2X
process is:  $\widetilde{\mathbf{F}}_m^{t_n^i}\left[h + \mathbf{M}_m^{t_m^j\to t_n^i}[h,w,0],w + \mathbf{M}_m^{t_m^j\to t_n^i}[h,w,1]\right] = \mathbf{F}_m^{t_m^j}[h,w]$ . After adjusting each non-ego agent's feature map, these estimated feature maps and ego feature map are aggregated together by an aggregation function  $f_{\mathrm{agg}}(\cdot)$ , which implements Step 2e. The fusing function can be any common fusion operation. All our experiments adopt Multi-scale Max-fusion.
4.5 Training details and loss function
To train the overall system, we supervise three tasks: ROI generation, flow estimation, and the final fusion detector. As mentioned before, the functionality of the ROI generator and final fusion detector share the same architecture but do not share the parameters. During the training process, the ROI generator and flow estimation module are trained separately, and later the final fusion detector is trained with the pre-trained two modules. Common loss functions in detection tasks: cross entropy loss and weighted smooth L1 loss are used for the classification and regression of ROI generation and final fusion detector, and MSE loss is used for flow estimation.
We propose the first asynchronous collaborative perception dataset and conduct extensive experiments on both simulated and real-world scenarios. The task of the experiments on the two datasets is pointcloud-based object detection. The detection performance was evaluated using Average Precision (AP) at Intersection-over-Union (IoU) thresholds of 0.50 and 0.70.
5.1 Datasets
IRregular V2V(IRV2V). To facilitate research on asynchrony for collaborative perception, we simulate the first collaborative perception dataset with different temporal asynchronies based on CARLA [39], named IRregular V2V(IRV2V). We set 100ms as ideal sampling time interval and simulate various asynchronies in real-world scenarios from two main aspects: i) considering that agents are unsynchronized with the unified global clock, we uniformly sample a time shift  $\delta_s \sim \mathcal{U}(-50, 50)\mathrm{ms}$  for each agent in the same scene, and ii) considering the trigger noise of the sensors, we uniformly sample a time turbulence  $\delta_d \sim \mathcal{U}(-10, 10)\mathrm{ms}$  for each sampling timestamp. The final asynchronous time interval between adjacent timestamps is the summation of the time shift and time turbulence. In experiments, we also sample the frame intervals to achieve large-scale and diverse asynchrony. Each scene includes multiple collaborative agents ranging from 2 to 5. Each agent is equipped with 4 cameras with a resolution of  $600 \times 800$  and a 32-channel LiDAR. The detection range is  $281.6\mathrm{m} \times 80\mathrm{m}$ . It results in 34K images and 8.5K LiDAR sweeps. See more details in the Appendix.
DAIR-V2X. DAIR-V2X [14] is a real-world collaborative perception dataset. There is one ego agent and one roadside unit in each frame. All frames are captured from real scenarios at  $10\mathrm{Hz}$  with 3D annotations. Lu et al. [17] complemented the missing annotations outside the camera view on the vehicle side to cover a 360-degree view detection. We adopt the complemented annotations[17] and set the perceptual range to  $x \in [-100.8\mathrm{m}, +100.8\mathrm{m}]$ ,  $y \in [-40\mathrm{m}, +40\mathrm{m}]$ .
5.2 Quantitative evaluation
Benchmark comparison. The baseline methods include late fusion, DiscoNet[9], V2VNet[4], V2X-ViT[5] and Where2comm[1]. The red dashed line represents single-agent detection without
Table 1: Detection performance on IRV2V and DAIR-V2X[14] dataset with pose noises following Gaussian distribution in the testing phase.
Dataset	IRV2V	DAIR-V2X
Noise Level σt/σr(m/∞)	0.0/0.0	0.1/0.1	0.2/0.2	0.3/0.3	0.4/0.4	0.0/0.0	0.1/0.1	0.2/0.2	0.3/0.3	0.4/0.4
Model / Metric	AP@0.50 ↑
V2X-ViT	0.641	0.626	0.627	0.625	0.619	0.693	0.692	0.545	0.685	0.681
Where2comm	0.510	0.411	0.411	0.411	0.411	0.702	0.693	0.679	0.658	0.643
Where2comm+SyncNet	0.654	0.653	0.652	0.651	0.648	0.711	0.692	0.583	0.579	0.671
CoBEVFlow (ours)	0.831	0.820	0.815	0.802	0.781	0.738	0.743	0.732	0.723	0.703
Model / Metric	AP@0.70 ↑
V2X-ViT	0.511	0.504	0.502	0.504	0.501	0.545	0.545	0.545	0.685	0.543
Where2comm	0.388	0.323	0.312	0.302	0.293	0.577	0.577	0.561	0.658	0.543
Where2comm+SyncNet	0.549	0.550	0.545	0.538	0.527	0.587	0.583	0.579	0.570	0.567
CoBEVFlow (ours)	0.757	0.730	0.686	0.628	0.570	0.599	0.593	0.579	0.571	0.560
collaboration. We also consider the integration of SyncNet[20] with Where2comm[1], which presents the SOTA method Where2comm[1] with resistance to time delay. All methods use the same feature encoder based on PointPillars[40]. To simulate temporal asynchrony, we sample the frame intervals of received messages with binomial distribution to get random irregular time intervals. Fig. 4 shows the detection performances (AP@IoU=0.50/0.70) of the proposed CoBEVFlow and the baseline methods under varying levels of temporal asynchrony on both IRV2V and DAIR-V2X, where the  $x$ -axis is the expectation of the time interval of delay of the latest received information and interval between adjacent frames and  $y$ -axis the detection performance. Note that, when the  $x$ -axis is at 0, it represents standard collaborative perception without any asynchrony. We see that i) the proposed CoBEVFlow achieves the best performance in both simulation and real-world datasets at all asynchronous settings. On the IRV2V dataset, CoBEVFlow outperforms the best methods by  $23.3\%$  and  $35.3\%$  in terms of AP@0.50 and AP@0.70, respectively, under a 300ms interval expectation. Similarly, under a 500ms interval expectation, we achieve  $30.3\%$  and  $28.2\%$  improvements, respectively. On the DAIR-V2X dataset, CoBEVFlow still performs best. ii) CoBEVFlow demonstrates remarkable robustness to asynchrony. As shown by the red line in the graph, CoBEVFlow exhibits a decrease of only  $4.94\%$  and  $14.0\%$  in AP@0.50 and AP@0.70, respectively, on the IRV2V dataset under different asynchrony. These results far exceed the performance of single-object detection, even under extreme asynchrony.
Trade-off between detection results and communication cost. CoBEVFlow allows agents to share only sparse perceptual features and the ROI set, which is communication bandwidth-friendly. Figure 5 compares the proposed CoBEVFlow with the previous methods in terms of the trade-off between detection performance (AP@0.50/0.70) and communication bandwidth under asynchrony. We adopt the same asynchrony settings mentioned before and choose  $300\mathrm{ms}$  as the expectation of the time interval. We see: i) CoBEVFlow consistently outperforms the state-of-the-art communication efficient solution, where2comm, as well as the other baselines in the setting of asynchrony; ii) as the communication volume increases, the performance of CoBEVFlow continues to improve
Figure 5: Trade-off between detection performance (AP@0.50/0.70) and communication bandwidth under asynchrony (expected 300ms latency) on IRV2V dataset. CoBEVFlow outperforms even with a much smaller communication volume.
steadily, while the performance of where2comm and where2comm+SyncNet fluctuates due to improper information transformation caused by asynchrony.
Robustness to pose error. We conduct experiments to validate the performance under the impact of both asynchrony and pose error. To simulate the pose error, we add Gaussian noise  $\mathcal{N}(0,\sigma_t)$  on  $x,y$  and  $\mathcal{N}(0,\sigma_r)$  on  $\theta$  during the inference phase, where  $x,y,\theta$  are 2D centers and yaw angle of accurate global poses. Our pose noise setting follows the Gaussian distribution with a mean of  $0\mathrm{m}$ , a standard deviation of  $0\mathrm{m} - 0.5\mathrm{m}$ , a mean of  $0^{\circ}$  and a standard deviation of  $0^{\circ} - 0.5^{\circ}$ . This experiment is conducted under the expectation of time interval is  $300\mathrm{ms}$  to simulate the time asynchrony. We compare our CoBEVFlow and other baseline methods including V2X-ViT[5], Where2comm[1]
Figure 6: Visualization of detection results for V2X-ViT, SyncNet, and CoBEVFlow with the expectation of time intervals are 100, 300, and 500ms on IRV2V dataset. CoBEVFlow qualitatively outperforms the others under different asynchrony. Red and green boxes denote detection results and ground-truth respectively.
Figure 7: Visualization of detection results for Where2comm, V2X-ViT, SyncNet, and our CoBEVFlow with the expectation of time intervals are 100, 300, and 500ms on the DAIR-V2X dataset. Red and green boxes denote detection results and ground-truth respectively.
and SyncNet[20]. Table 1 shows the results on IRV2V and DAIR-V2X[14] dataset. We see that CoBEVFlow still performs well even when both pose errors and time asynchrony appear. CoBEVFlow consistently outperforms other methods across all noise settings on the IRV2V dataset. In the case of noise levels of 0.4/0.4, our approach achieves 0.133 and 0.043 improvement over SyncNet.
5.3 Qualitative evaluation
Visualization of detection results. We illustrate the detection results of V2X-ViT, SyncNet, and CoBEVFlow at three asynchrony levels on the IRV2V dataset in figure 6 and the DAIR-V2X dataset in figure 7. The expectations of time intervals are 100, 300, and 500ms. The red box represents the detection result and the green box represents the ground truth. V2X-ViT shows significant deviations in collaborative perception under asynchrony, while SyncNet shows poor compensation due to introducing noise in feature regeneration and irregularity-incompatible design. The third row shows the results of CoBEVFlow, which achieves precise compensation and outstanding detections.
Visualization of BEV flow map. Figure 8 visualizes the feature map before/after compensation of CoBEVFlow in Plot(a)(b), the corresponding flow map in Plot(c), and matching, detection results
(a) Asynchronous feature map
(c) BEV flow map
(b) Feature map compensated by BEV flow
(d) Matching & compensation results
Figure 8: Visualization of compensation with CoBEVFlow on IRV2V dataset. In subfigure(d), green boxes are the objects' ground truth locations, blue boxes are the detection results based on the historical asynchronous features and red boxes are the detection results after compensation. CoBEVFlow achieves precise matching and compensation with the BEV flow map and mitigates the negative impact of asynchrony to a great extent.
after compensation in Plot(d). The green boxes in Plot (d) are the ground truth, the blue boxes are the historical detections with the matched asynchronous ROIs and the red boxes are the compensated detections. We see that the BEV flow map can be precisely estimated and is beneficial for perceptual feature alignment. The compensated detection results are more accurate than the uncompensated ones.
We conduct ablation studies on the IRV2V dataset. Table 2 assesses the effectiveness of the proposed operations, including time encoding, the object feature/detected box) to warp, and our ROI mixer. We see that: i) time encoding encodes the irregular continuous timestamp and makes the estimation more accurate; ii) Warping the features outperforms warping the boxes directly a lot, which means the operation for features shows superiority over the operation for the detection results; and iii) our ROI mixer shows more proper matching results than traditional Hungarian[41] matching.
Table 2: Ablation Study on IRV2V dataset. Time encoding(TE), BEV flow on features, and the proposed matcher all improve the performance.
Modules	AP@0.50 / AP@0.70 ↑
TE	Warped by Flow	Matcher	300ms	500ms
Box	Hungarian	0.724 / 0.473	0.644 / 0.388
✓	Box	Hungarian	0.747 / 0.595	0.668 / 0.438
✓	Box	Ours	0.764 / 0.611	0.571 / 0.399
✓	Feature	Hungarian	0.779 / 0.690	0.739 / 0.614
✓	Feature	Ours	0.831 / 0.757	0.815 / 0.687
6 Conclusion and limitation
We formulate the asynchrony collaborative perception task, which considers various unideal factors that may cause communication latency or information misalignments during collaborative communication. We further propose CoBEVFlow, a novel asynchrony-robust collaborative perception framework. The core idea of CoBEVFlow is BEV flow, which is a collection of the motion vector corresponding to each spatial location. Based on BEV flow, asynchronous perceptual features can be reassigned to appropriate positions, mitigating the impact of asynchrony. Comprehensive experiments show that CoBEVFlow achieves outstanding performance under all settings and far superior robustness with asynchrony.
Limitation and future work. The current work focuses on addressing the asynchrony problem in collaborative perception. It is evident that effective prediction can compensate for the negative impact of temporal asynchrony in collaborative perception. Moreover, the generated flow can also be utilized not only for compensation but also for prediction. In the future, we expect more works on exploring the ROI-based flow generation design for collaborative perception and prediction tasks.
Acknowledgment. This research is supported by NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 21511100900, 22511106101, and 22DZ2229005.
[32] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention-based recurrent neural network for time series prediction. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 2627-2633. ijcai.org, 2017.
[33] Satya Narayan Shukla and Benjamin M. Marlin. Multi-time attention networks for irregularly sampled time series. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
[34] Satya Narayan Shukla and Benjamin M. Marlin. Interpolation-prediction networks for irregularly sampled time series. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[35] Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan B. Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 651-659. AAAI Press, 2021.
[36] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning, pages 4353-4363. PMLR, 2020.
[37] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In *The Tenth International Conference on Learning Representations*, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008, 2017.
[39] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference on robot learning, pages 1-16. PMLR, 2017.
[40] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12697-12705. Computer Vision Foundation / IEEE, 2019.
[41] Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2(1-2):83-97, March 1955.
A IRegular V2V (IRV2V)
To facilitate the research on asynchrony for collaborative perception, we use CARLA [39] (under MIT license) to simulate IRregular V2V (IRV2V) dataset, which is the first collaborative perception dataset with multiple asynchronies.
Asynchronous data collection. The number of collaborative vehicles in a scene ranges from 2 to 5. Each collaborative vehicle is equipped with 4 cameras for  $360^{\circ}$  view, a 32-channel LiDAR, and GPS/IMU sensors. The ideal sample interval of the sensor is  $100\mathrm{ms}$ . Due to different asynchronous factors, collaborative messages have asynchronous timestamps. There is a time offset  $\delta_s \sim \mathcal{U}(-50, 50)\mathrm{ms}$  at the sampling starting point of non-ego vehicles. And all non-ego vehicles' collaborative messages are sampled with time turbulence  $\delta_d \sim \mathcal{U}(-10, 10)\mathrm{ms}$ . Sensing information at each timestamp of each agent contains 4 camera images with resolution  $600 \times 800$ , and 32-channel LiDAR points.
Data size. Assuming the model requires the use of information from the past 10 frames, our dataset consists of a total of 8,449 collaborative samples, which include 8,449 point cloud inputs and 33,796 RGB images. We have split the dataset into training, validation, and testing sets, which contain 5,445, 994, and 2,010 samples, respectively.
Data analysis. Figure 9 presents some statistical analysis results regarding the IRV2V dataset. The IRV2V dataset contains a total of 1,564,033 vehicles, with an average of 48.302 vehicles per scene. It should be noted that the figure only displays the distribution of vehicles with speeds greater than  $1\mathrm{km/h}$ . Considering real-world scenarios, there are around 1,203,793 moving vehicles in the dataset. Plot (a) illustrates the distribution of moving vehicles with different speeds across all samples, ranging from 1 to  $105\mathrm{km/h}$ , with an average speed of  $25.586\mathrm{km/h}$ , which achieves around  $15\mathrm{km/h}$  faster compared to the majority of vehicles in the V2X-Sim[15]. Plot (b) shows the distribution of the total number of vehicles per sample in the dataset, with the maximum number of vehicles being 113.
(a) Vehicle speed distribution.
(b) Number of vehicles in the scene distribution.
Figure 9: Data distribution of IRV2V dataset. (a) shows the speed distribution of moving vehicles; (b) shows the vehicle numbers distribution.
B Detailed information about experimental settings
Implement details. We conduct experiments on LiDAR-based part of IRV2V and DAIR-V2X[14] dataset. Our feature encoder is PointPillars[40] based. And our backbone follows the setting in CoAlign[17]. The difference is that we change the fusion method from self-attention to max-fusion. We conduct training for a total of 60 epochs, starting with an initial learning rate of 2e-3. Subsequently, at the 10th and 20th epochs, the learning rate decreases to  $10\%$  of its previous value. For IRV2V dataset, we set the lidar range as  $x \in [-140.8, +140.8]\mathrm{m}$ ,  $y \in [-40, +40]\mathrm{m}$ . The voxel size is  $h = w = 0.4\mathrm{m}$ . The feature map's size is  $H = 200$ ,  $W = 704$ . For DAIR-V2X dataset, we set the lidar range as  $x \in [-100.8, +100.8]\mathrm{m}$ ,  $y \in [-40, +40]\mathrm{m}$ . The voxel size is  $h = w = 0.4\mathrm{m}$ . The feature map's size is  $H = 200$ ,  $W = 504$ .
Communication volume. Our communication volume is the same as Where2comm[1]. For CoBEVFlow, we control the communication volume by adjusting the maximum number of generated ROIs. Specifically, the average number of voxels contained in each ROI region is 40, and we limit the maximum number of generated ROIs to  $K_{\| \mathcal{R}\|}$ . Correspondingly, we modify the information exchange in Where2comm to include the top  $40 \times K_{\| \mathcal{R}\|}$  blocks based on their scores on the spatial confidence map. In practical scenarios, the actual communication volume is influenced by factors such as the feature dimension and floating-point precision. To simplify the expression, we uniformly represent the communication volume using the logarithm to the base 2 of the voxel count. The communication volume is
$$
\log_ {2} \left(4 0 \times K _ {\| \mathcal {R} \|}\right), \tag {6}
$$
where  $K_{\| \mathcal{R}\|}$  is the maximum number of generated ROIs, and 40 is the average number of voxels in each ROI.
C Benchmarks
We conduct extensive experiments on current collaborative perception methodologies. Table 3 and Table 4 present the detection performance under the expectation of time interval from 0 to  $500\mathrm{ms}$  on IRV2V and DAIR-V2X[14] respectively, which correspond to the numerical results shown in Figure 4 in the main text. We see that CoBEVFlow consistently achieves significant improvements over previous methods on both datasets and the leading gap is bigger when the expectation of the time interval is higher.