Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow
Authors: Sizhe Wei
Yuxi Wei
Yue Hu
Yifan Lu
Yiqi Zhong
Siheng Chen
Ya Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Collaborative Perception; BEV Flow; Time Asynchronization

Abstract: Collaborative perception can substantially boost each agent's perception ability by facilitating communication among multiple agents. However, temporal asynchrony among agents is inevitable in the real world due to communication delays, interruptions, and clock misalignments. This issue causes information mismatch during multi-agent fusion, seriously shaking the foundation of collaboration. To address this issue, we propose CoBEVFlow, an asynchrony-robust collaborative perception system based on bird's eye view (BEV) flow. The key intuition of CoBEVFlow is to compensate motions to align asynchronous collaboration messages sent by multiple agents. To model the motion in a scene, we propose BEV flow, which is a collection of the motion vector corresponding to each spatial location. Based on BEV flow, asynchronous perceptual features can be reassigned to appropriate positions, mitigating the impact of asynchrony. CoBEVFlow has two advantages: (i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous time stamps without discretization; and (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, avoiding additional noises. To validate CoBEVFlow's efficacy, we create IRregular V2V(IRV2V), the first synthetic collaborative perception dataset with various temporal asynchronies that simulate different real-world scenarios. Extensive experiments conducted on both IRV2V and the real-world dataset DAIR-V2X show that CoBEVFlow consistently outperforms other baselines and is robust in extremely asynchronous settings. The code is available at https://github.com/MediaBrain-SJTU/CoBEVFlow.

Asynchrony-Robust Collaborative Perception via Bird's Eye View Flow
Sizhe Weilt
Yuxi Wei Yue Hu
Yifan Lu Ya Zhang!
Yiqi Zhong? Siheng Chen
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
University of Southern Californa 3 Shanghai Al Laboratory
(sizhewei, xyx3590236732, 18671129361, vifan_lu]@sjtu weu.uuuuuuuuuuuu...uu.. edu. cn {sihenge, ya_ ra_zhang) @s eet.u.edu.ce edu. cn yiqizhon@usc. edu
Abstract
Collaborative perception can substantially boost each agent's perception ability by facilitating communication among multiple agents. However, temporal asyn- chrony among agents is inevitable in the real world due to communication delays, interruptions, and clock misalignments. This issue causes information mismatch during multi-agent fusion, seriously shaking the foundation of collaboration. To address this issue, we propose CoBEVFlow, an asynchrony- -robust collaborative perception system based on bird's 's eye view (BEV) flow. The key intuition of CoBEVFlow to compensate motions to align asynchronous collaboration mes- sages sent by multiple agents. To model the motion in a scene, we propose BEV flow, which is a collection of the motion vector corresponding location. Based on BEV flow, asynchronous perceptual features can be reassigned appropriate positions, mitigating the impact of asynchrony. CoBEVFlow has two advantages: (i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous time stamps without discretization; and (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, avoiding additional noises. To validate CoBEVFlow 's efficacy, create IRregular V2V(RVVVV,, the first synthetic collaborative percep- tion dataset with various temporal asynchronies that simulate different real-world scenarios. Extensive experiments conducted on both IRV2V and the real-world dataset DAIR- V2X show that CoBEVFlow consistently outperforms other base- lines and is robust in extremely asynchronous settings. The code is available https:/ggthhh..coommmmacammmnntt
each spatial
Introduction
Multi-agen collaborative perception allows agents to exchange complementary perceptual informa- tion through communication. This can overcome inherent limitations single-agent perception, such as occlusion and long-range issues. Recent studies have shown that collaborative perception can substantially boost the performance of perception systems ( and has great potential for wide range of real-world applications, such multi-robot automation system 6 77, aehicle-tt-everythinn communication-aico autonomous driving[8] 9 and multi-UAV UAVs (unmanned aerial vehicles) [2] As an emerging area, the study of collaborative perception has many challenges to be tackled, such as high-quality datasets [3H15,, model-agnostic and aagnstic formulation [16] and the robustness to pose enrors [17] and adversarial attacks [18] 19].
Equal contribution.
"Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

However, vast majority of existing works do not seriously account for the harsh realities of real-world communication among agents, such as congestion, heavy computation, interruptions, and the lack of calibration. These factors in- troduce delays or misalignments that severely impact the reliability and quality of informa- tion exchange among agents. Some prior works have touched upon the issue of communication latency. For instance, V2VNet [4] and V2X- in-
Asynchranous communication
egn agen time
Perception results under
Perception results with CoBEVFLow
asynchronous communication
0 0g
G O
nag
B
Dg
VIT [S] incorporated the delay time put for feature compensation. However, they only account for a single frame without leverag- ing historical frames, making them inadequate for high-speed scenarios (above 20m/s) or high- latency scenarios (above 0.3s) scenarios. Mean- while, SyncNet 200 uses historical features to predict the complete feature map at the current timestamp 21 Nevertheless, this
og
0 Do
Cgll B
0g
ag
0 P
Figure 1: Illustration of asynchronous collaborative per- ception and the perception result w.o./w. CoBEVFlow. Red boxes are detection results and green boxes are the ground-truth.
method assumes equal time intervals for its input, causing failures when delays are irregular. Overall, previous works have not addressed the issues raised by common irregular time delays, rendering existing collaborative perception systems can never reach their full potential in real-world scenarios.
To fill the research gap, we specifically formulate setting of asynchronous collaborative perception; see Fig. for a visual demonstration. Here asynchrony indicates that the time stamps of the collabo- ration messages from other agents are not aligned and the time interval of two consecutive messages from the same agent is irregular. Due to the universality and inevitability of temporal asynchrony in practical applications, handling this setting is critical to the further development of collaborative collaborative perception
perception. To address this, we propose CoBEVFlow, an asynchrony system based on bird's eye view (BEV) flow. The key idea is to align perceptual information from other agents by compensating relative motions. Specifically, CoBEVFlow uses historical frames to estimate BEV flow map. which encodes the motion information in each grid cell. Using the BEV flow map, CoBEVFlow can reassign asynchronous perceptual features to appropriate spatial locations, which aligns perceptual features on the time dimension, mitigating the impact caused by asynchrony. The proposed CoBEV has two major advantages: i) CoBEVFlow can handle asynchronous collaboration messages sent at irregular, continuous timestamps without discretization; ii) CoBEVFlow better adheres to the essence of compensation, which is to move features to the desig- nated spatiotemporal positions. Not like SyncNet 20) that regenerates features, this motion-guides position adjustment fundamentally prevents introducing extra noises to the features.
When validating the effectiveness of CoBEVFlow, we noticed that there no appropriate collabora- tive perception dataset that contains asynchronous samples. To facilitate research on asynchronous collaborative perception, we create IRregular V2V(IVVVV), the first synthetic asynchronous collabo- rative perception dataset with irregular time delays, simulating various real-world scenarios. Sec. [ show the experiment results and analysis on both IRV2V and real-world dataset DARI-V2X [14. Results show that CoBEVFlow consistently achieves the best compensation performance across various latencies. When the expected latency on the IRV2V dataset is set to 500ms, CoBEVFlow outperforms other methods by more than 18.9%. In the case of a 300ms latency with an additional 200ms disturbance, the decrease in AP@0.50 is only 0.25%.
2 Related work
2.1 Collaborative Perception
Factors such as limited sensor fields of view and physical environmental occlusions can negatively impact perception tasks for individual agents 23 To address the aforementioned challenges, collaborative perception based on aueti-agent systems has emerged as a promising solution 2 24 It can enhance the performance of perception tasks by leveraging the exchange of information among different agents within the same scenario. V2VNet uses multi-round message passing via graph neural networks to achieve better perception and prediction performance 4: DiscoNet adopts knowledge distillation to take advantage of both early and intermediate collaboration 9 V2X- VIT proposes heterogencous multi-agent attention module to aggregate information from heterogeneous
2

---

agents; Where2comm [ introduces spatial confidence map which achieves pragmatic compression and improves perception performance with limited communication bandwidth.
As unideal communication is an inevitable issue that negatively impacts the performance and applica- tion of collaborative perception, some methods have been studied for robust collaborative perception. V2VNet[4] utilizes a convolutional neural network to learn how to compensate for communication delay by taking time information and relative pose as input; V2X-ViT designs a Delay-aware Positional Encoding module to learn the influence caused by latency, but these methods do not con- sider the historical temporal information for compensation. SyneNet [20] uses historical multi-frame information and compensates for the current time by Conv-LSTM[21, but its compensation for framework can not handle temporal irregular inputs. This work formulates asynchronous collaborative perception, which considers real-world communication asynchrony.
the whole feature map leads noises to the feature channels, and
2.2 Time-Series Forecasting
Time series analysis aims to extract temporal information of variables, giving rise to numerous downstream tasks. Time series forecasting leverages historical sequences to predict observations of variables in future time periods. Classical methods for time series forecasting include the Fourier transform (7), autoregressive models [8), and Kalman filters (29]. In the era of deep learning, plethora of RNN- based and attention based methods have emerged model sequence input and address time series prediction problems
The issue of sampling irregularity, which may arise due to physical interference or device issues, calls for irregularity-robut time series analysis systems. Approaches such m33 IP-Neu[34] and DGM2 (35] employ imputation- basedd techniques to tackle irregular sampling, which estimate the observation or hidden embedding of series at regular timestamps and then apply the methods designed for regular time series analysis, while ScFT and Raindrop 37 utilize operations that are insensitive to sampling intervals for processing irregularly sampled data. In this work, we consider each collaboration message one irregular sample and use irregular time series forecasting to estimate the BEV flow map for asynchronous feature alignment.
3 Problem Formulation
Consider N agents in a scene, where each agent can send and receive collaboration messages from other agents, and store & historical frames of messages. For the nihh agent, let x and y be the raw obscrvation and the perception ground-truth at time current t,, respectively, where e, is the i-th
timestamp of agent n, and p .n be the collaboration message sent from agent m at time th The key of the asynchronous setting is that the timestamp of each collaboration message t, E RR is continuous value, those messages from other agents are not aligned, e * t. and the time interval between consecutive timestamps + t, is irregular. Therefore, each agent has encounter collaboration messages from other agents sent at arbitrary times. Then, the task of asynchronous collaborative perception is formulated as:
max O.P
g
ni
Y co( (Pt P mm
k+1 Pm i }m=l),
subject
where g(:,)) is the perception evaluation metric, 4% is the perception result of agent time i c+(-) is the collaborative perception network with trainable parameters 0, and ti tj-k+1 < t-=++2 < t/ I f.. Note that: i) when the collaboration messages from other agents are all aligned and the time interval between two consecutive timestamps is regular; that is, t = er for all agent's pairs m, n, and f r is constant for all agents n, the task degenerates to standard well-synchronized collaborative perception; and ii) when the collaboration messages from other the agents are not aligned, yet the time interval between two consecutive timestamps is regular; that is, t + t", and
1 1 is constant, the task degenerates to the setting in SyncNet (20)
Given such irregular asynchrony, the performances of collaborative perception systems would be significantly degraded since features from asynchronized timestamps differ from the actual current features, and using asynchronized features may contain erroneous information during the perception process. In the next section, we will introduce CoBEVFlow to address this critical issue.
3

---

Fliw
indicates historical BEV Flow Map Min
Festure F
Sparse Feazure
Massage
Warped Pro
Moxi
ROI R
Fool Fasture
A
Encoder
Message
Packing
Message
Fusion
Decoder
Feature Fa
Point Cloud
Message Exchange
Collsborative Detections
Figure 2: System overview. Message packing process prepares ROI and sparse features as message efficient communication and BEV flow map generation. Message fusion process generates and applics BEV flow map for compensation, and fuses the features at the current timestamp from all agents.
CoBEVFlow: Asynchrony-Robur Collaborative Perception System
This section proposes an asynchrony-robust collaborative perception system, CoBEVFlow. Figure overviews the overall scheme of CoBEVFlow. We introduce the overall framework of the CoBEVFlow system in Sec. 4.1 The details of three key modules of CoBEVFlow can be found in Sec. 4.2 4.4 demonstrates the training details and loss functions of the whole system.
Sec. 4.3
4.1
Overall architecture
The problem of asynchrony results in the misplacements of moving objects in the collaboration messages. That is, the collaboration messages from multiple agents would record various positions for the same moving object. The proposed CoBEVFlow addresses this issue with two key ideas: i) we use BEV flow map to capture the motion in scene, enabling motion-guided reassigning asynchronous perceptual features to appropriate positions; and ii) we generate the region of interest(ROI) to make sure that the reassignment only happens to the areas that potentially contain objects. By following these two ideas, we eliminate direct modification of the features and keep the background feature unaltered, effectively avoiding unnecessary noise in the learned features.
Mathematically. let the n-th agent be the ego agent and X be its raw observation at the i-th timestamp of agent denoted as i. The proposed asynchrony-robus, collaborative perception system CoBEVFlow is formulated as follows:
F fenc (X%:
(2a)
F R. froi gon (F
(2b)
-1, fllww genw (t (Ri 19
(2c)
Fthh swrp Frlm M -tt).
(2d)
i fags F (F meN.))
(2e)
4 feee H
(2
where F. E HxWx is the BEV perceptual feature map of agent n at timestamp é with H, W the size of BEV map and D the number of channels; R the of region of interest (ROI); F HxWxD is the sparse version of F which only contains features inside Rt and zero- padding outside; M +e, E =R is the m-th agent's the BEV flow map that reflects each grid cell's movement from timestamp 1/, to timestamp 1 (Rih A
ROI sets sent by agent m; F E R is the realigned feature map from the mn-hh agent's at timestamp 1 after motion compensation; H E HR the aggregated features from all of the agents; Nr, the collaboration neighbors of the nith agent; and T the final output of the system.
Step extracts BEV perceptual feature from observation data. Step generates the ROIs for each feature map, enabling BEV flow generation in Step 2c Now all the agents exchange their messages,

---

including Fr and the R Step 2c generate the BEV flow map M -
by leveraging historical
ROIs from the same agent. Step 2dlgets the estimated feature map by applying the BEV flow map to reassign the asynchronized features. Step aggaggres the feature maps of all agents. Finally, Step
21 outputs the final perceptual results.
Note that i) Steps 2a 2b are done before communication. Steps 2c 2f are performed after receiving the message from others. During the communication process, both sparse perceptual features and the ROI set are sent to other agents, which is communication bandwidth friendly; and ii) CoBEVFlow adopts the feature representations in bird's eye view (BEV), where the feature maps of all agents are projected to the same global coordinate system, avoiding complex coordinate transformations and supporting easier cross-agent collaboration.
The proposed asynchronous collaborative perception system has three advantages: i) CoBEVFlow can deal with asynchronous collaboration messages sent at irregular, continuous timestamps without discretization; (ii) with BEV flow, CoBEVFlow only transports the original perceptual features, instead of generating new perceptual features, causing additional noises; and (iii) CoBEVFlow promotes robustness to asynchrony by introducing minor communication cost (ROI set).
We now elaborate on the details of Steps 26 2e in the following subsections.
4.2 ROI generation
Given the perceptual feature map of an agent, Step 2b aims to generate a set of spatial regions of interest (ROIs) for the areas that possibly contain objects. Each ROI indicates one potential object's region in the scene. The intuition is that foreground objects are the only ones that move, while the background remains static. Therefore, using ROI can enable the subsequent BEV flow map to concentrate on critical regions and simplify the computation of the BEV flow map.
To implement, we use the structure of the object detection decoder to produce the ROIs. Given agent
m's perceptual feature map at timestamp 1// F'i- the corresponding detection result obtained as:
Droi (F m E #x#*x
m
where Proi_gen/ (.) is the ROI generation network with detection decoder structure, and each element (0% )r = (c, x. y, h, w, cos a, sin a)) represents one detected ROI with its class confidence, posi- tion, size, and orientation. We threshold the class confidence, apply non-max suppression, and obtain a set of detected boxes, whose occupied spaces form the set of ROIs, R.
Based on this ROI set, we can also get binary mask H # whose values inside ROIs are 1 and the others are O. We then get the sparse feature map F
<e! F ) H, which only contains the
features within ROIs Then, agent m packs its sparse feature map F and the ROI set R as the message and sends out for collaboration.
4.3
BEV flow map generation
After receiving collaboration messages from other agents at various timestamps, Step 2c aims to generate the BEV flow map to correct feature misalignment due to asynchrony. The proposed BEV flow map encodes the motion vector of each spatial location. The main idea of obtaining this BEV flow is to associate correlated ROIs based on a sequence of messages sent by the same collaborator. In this step, each ROI is regarded as an instance that has its own attributes generated by Step 2b After the ROI association, we are able to compute the motion vector and further estimate the positions where the corresponding object would appear at certain timestamp. The generation of the BEV flow map includes two key steps: adjacent timestamp' ROI matching and BEV flow estimation.
Adjacent frames' ROI matching. The purpose of adjacent frames' ROI matching is to match the ROIs in two consecutive messages sent by the same agent. The matched ROIs are essentially the same instance perceived at different timestamps. This module contains three processes: cost matrix construction, greedy matching, and post-processing. We first construct a cost matrix C E R X where o₁ and 02 are the numbers of ROIs in the two frames to be matched. Each element Cp.g is the matching cost between ROI p in the earlier frame and ROI q in the later frame. To determine the value of C Cp.q. we define the vicinity of the front and rear directions as a feasible angle range for matching.
We set Cp. = dp.g when q is within the feasible angle range of p. otherwise Cp.4 = +00, where dp.g is the Euclidean distance between the center of ROI p and g. We then use the greedy matching
5

---

strategy to search the paired ROIs. For each row we search the q with minimum C Cp.. and match p. q as pair. To avoid invalid matching, we further post-process matched pairs by removing those with excessively large values of Cp.g. Through these processes, we can get the matched ROI pairs in adjacent frames. For a sequence of frames, we can track each ROI's multiple locations across frames.
BEV flow estimation. We now retrieve each ROI's historical locations series of irregu- lar timestamps. In this module, we use those irregular tracklets to predict the location and orientation of ROIs at the ego agent's 's current timestamp t and generate the corresponding
Location and orientation estimation
Iregubrly sampled sequence
Cm
Value
Time uncoding
MLP
BEV flow map M -ti To formulate the r- th ROI's irregular tracklet perceived by m--hh agent, we extract the motion-related attributes (i.e. location and orientation) from the r-th ROI's attributes set each timestamp. Let 12 (i-ktl
Quary
MHA
Estimated location oniennation
V m.r th #5-1
Vr" .... .Vr" } be histor- ical sequence of the r-th ROI's attributes sent by
BEV flow map generation
map
the m-th agent, where v = (IT" yyr" )
with (a, th ,,/"" ) is the 2D BEV center position e and ar is the orientation. Note that V Vm,r an irregularly sampled sequence due to time asyn- chrony. Fig 33 this process.
Figure 3: The process of the BEV flow estimation.
Based on Mm,,, now predict v which is the location and orientation of the r-th ROI at the ego agent's 's current timestamp. Unlike the common motion estimation, we need to handle an irregularly sampled sequence. To enable the errgularity-omppatiie of the timestamp should be taken into account, Here we propose to use traditional trigonometric functions (38) for timestamp encoding, by which map the continuous-valued timestamp t into its corresponding time code utt) through:
motion estimation method, the information
(u(t)) sin( 1000020/a) (u(t))2e+1 cos( 100002e/4
(4)
where e is the index of temporal encoding. The timestamp information now can be input into the estimation process along with the irregularly sampled sequence and make the estimation process capable of multi-head attention(MHAA), The query MHA the time code of the target timestamp t and the key and value both are the sum of the features of the irregularly sampled sequence and its corresponding time code set Ug:
compatiilee motion estimation. We implement the estimation process using
MHA (L)PV m.r) + ULPML m.r) + UR),
(5)
where of is the estimation of ROI's location and orientation vf MLP(.) is the encoding function of the irregular historical series V and M-A(-) is the multi-head attention for temporal estimation.
With the estimated location and orientation of ROIs in the ego agent's current timestamp along with the ROIs' sizes predicted by Step 2b we calculate the motion vector each grid cell by an affine transformation of the associated ROI's motion, constituting the whole BEV flow map M +t" nxxx Note that the grid cells outside ROIs are zero-padded.
Compared to Syncnet 20 that uses RNNs to handle regular communication latency, the generated BEV flow map has two benefits: it handles irregular asynchrony via the attention-based estimation with appropriate time encoding; and ii) facilitates the motion-guided feature warping, which avoids regenerating the entire feature channels.
4.4
Feature warp and aggregation
The BEV flow map M - is applied on the sparse feature map F which implements Step 2d
The features each grid cell are moved to the estimated position based on M tm -r The warping

---

Single
V2VNet V2XVVT
DiscoNet
Where2comm + SyncNet
Late
(ours)
0.90
0.7)
APgo 50
0.70
6:4
390 1x ax Se Expectation time
Expectation time Expectation
468 303
232 103 463
TEE TE xo Expectation
tmm))
IRV2V
(b) DAIR-V2X
Figure 4: Comparison of the performance of CoBEVFlow and other baseline methods under the expectation of time interval from 0 to 500ms. CoBEVFlow oulperforms all the baseline methods and shows great robustness under any level of asynchrony on both two datasets.
process is: Fil + +t
w. O], M +t [b, w. 1] Flr [h, w). After adjusting each
non-ego agent's feature map, these estimated feature maps and ego feature map are aggregated together by an aggregation function Saga (.), which implements Step The fusing function can be any common fusion operation. All our experiments adopt Multi-scale Max-fusion.
4.5 Training details and loss function
To train the overall system, we supervise three tasks: ROI generation, flow estimation, and the final fusion detector. As mentioned before, the functionality of the ROI generator and final fusion detector share the same architecture but do not share the parameters. During the training process, the ROI generator and flow estimation module are trained separately, and later the final fusion detector trained with the pre-trained two modules. Common loss functions in detection tasks: cross entropy loss and weighted smooth LI loss are used for the classification and regression of ROI generation and final fusion detector, and MSE loss is used for flow estimation.
5 Experimental Results
We propose the first asynchronous collaborative perception dataset and conduct extensive experiments on both simulated and real-world scenarios. The task of the experiments on the two datasets point- cloud-based object detection. The detection performance at
evaluated using Average Precision (AP) (IoU) thresholds of 0,50 and 0.70.
5.1 Datasets
IRregular V2V(IRVVV. To facilitate research asynchrony for collaborative perception, simulate the first collaborative perception dataset with different temporal asynchronies based on CARLA (39], named IRregular V2V(IRV2V). We 100ms ideal sampling time interval and simulate various asynchronies in real-world scenarios from two main aspects: i) considering that agents are unsynchronized with the unified global clock, we uniformly sample time shift 8, u(-50, ms for each agent in the same scene, and ii) considering the trigger noise of the sensors, we uniformly sample a time turbulence 6, ud-10, 10)ms for each sampling timestamp. The final asynchronous time interval between adjacent timestamps is the summation of the time shift and time turbulence. experiments, we also sample the frame intervals achieve large-scuu scale and diverse asynchrony. Each scene includes multiple collaborative agents ranging from 2 to 5. Each agent is equipped with 4 cameras with resolution 600 x 800 and a 32-channel LiDAR. The detection range is 281.6m x 80m. It results in 34K images and 8.5K LiDAR sweeps. See more details in the Appendix.
DAIR-V V2X. DAIR-V2X [14] is real-world collaborative perception dataset. There is one ego agent and one roadside unit each frame. All frames are captured from real scenarios 10 Hz with 3D annotations. Lu et al. [17] complemented the missing annotations outside the camera view on the vehicle side cover 360-degree view detection. We adopt the complemented annotations and set the perceptual range to x E [-100.8m, + +100.8m) 8m], y E [-40m, -40m].
5.2 Quantitative evaluation
Benchmark comparison. The baseline methods include late fusion, DiscoNet V2VNet V2X- 5) and Where2comm[[]. The red dashed line represents single-agent detection without
7

---

Table 1: Detection performance on IRV2V and DAIR- -2X Gaussian distribution in the testing phase.
dataset with pose noises following
Dataset
IRV2V
DAIR-V2X
Noise Level o (m/o/
0.0/0.0
0.1/0.1
0.2/0.2
0.3/03
0.4M0.4
0.0/0.0
0.1/0.1
0.2/0.2
0.3/0.3
0,4/0.4
Model/ / Metric
AP@0.50 1
V2X-V1 Where2comm
0.641
0.626
0.627
0.625
0.619
0.693
0.692
0.545
0.685
0.681
0,510
0,411
0.411
0,411
0.411
0.702
0.693
0.679
0.658
0.643
Where2comn-SyncNe
0.654
0.653
0.652
0.651
0.648
0.711
0.692
0.583
0.579
0.671
CoBEVFlow (ours)
0.831
0.820
0.815
0.802
0.781
0.738
0.743
0.732
0.723
0.703
Model/ / Metric
AP@0,70 1
V2X-VIT
0.511
0.504
0.502
0.504
0.501
0.545
0.545
0.545
0.685
0.543
Where2comm
0,388
0.323
0,312
0,302
0.293
0,577
0.577
0.561
0.658
0.543
Where2comm- SyncNet
0.549
0.550
0.545
0.538
0.527
0.587
0,583
0.579
0,570
0,567
CoBEVFlow (ours)
0.757
0.730
0.686
0.628
0.570
0.599
0.593
0.579
0.571
0.560
collaboration. We also consider the integration of SyncNet with Where2comm[[] [I, which presents the SOTA method Where2comm [I] with resistance to time delay. All methods use the same feature encoder based on PointPillars (4). To simulate temporal asynchrony, we sample the frame intervals of received messages with binomial distribution to get random irregular time intervals. Fig. 4 detection performances LAP@Io==0.50/0. 70) of the proposed CoBEVFlow and the baseline methods under varying levels of temporal asynchrony the expectation of the time interval of delay of the latest received information and interval between adjacent frames and y-axis the detection performance. Note that, when the r-axis is at O, it represents standard collaborative perception without any asynchrony. We see that i) the proposed CoBEV Vllow achieves the best performance in both simulation and real-world datasets all asynchronous settings. On the IRV2V dataset, CoBEVFlow outperforms the best methods by 23.3% and 35.3% 3% in terms of AP@0.50 and AP@0.70, respectively, under a 300ms interval expectation. Similarly, under 500ms interval expectation, we achieve 30.3% and 28.2% improvements, respectively. On the DAIR-V2X dataset, CoBEVFlow still performs best. ii) CoBEVFlow demonstrates remarkable robustness to asynchrony. As shown by the red line in the graph, CoBEVFlow exhibits decrease of only 4.94% and 14.0% in A.50.50 and AP 0.70. respectively, on the IRV2V dataset under different asynchrony. These results far exceed the performance of single-object detection, even under extreme asynchrony.
shows the
both IRV2V and DAIR-V22 where the z-axis is
Trade-off between detection results and com- munication cost. CoBEVFlow allows agents to share only sparse perceptual features and the ROI set, which is communication bandwidth- friendly. Figure 5 compares the proposed CoBEVFlow with the previous methods in terms of the trade-off between detection performance (AP@0.500) 50/0.000000. 70) and communication bandwidth under asynchrony. We adopt the same asyn- chrony settings mentioned before and choose 300ms as the expectation of the time interval. We see: i) CoBEVFlow consistently outper- forms the state-of-the-re -hrt----tttttttt communication effi- cient solution, where2comm, as well as the other baselines in the setting of asynchrony; ii) as the communication volume increases, the per- formance of CoBEVFlow continues to improve steadily, while the performance of where2comm and where2comm-SyncNNe fluctuates due to im- proper information transformation caused by asynchrony.
0.
CoBEVFlow
2.5
CoBEVFlow
Wherezcomm Synne
Where2comm-Synettee
ARg0 50
02 RE0OBU
DiscoNer
DiscNNet
Single
V2VNet V2X-VIT
Single
V2X-VIT
WhereZcomm
WhereZcomm
3.5 3.75
7:50
11:33
33.00
5.75
7:0
11.23
12:00
Communication volume flog2)
Communication
(leg2)
Figure 5: Trade-off between detection performance (P@00.5000.70) and communication bandwidth under asynchrony (expected 300ms latency) on IRV2V dataset. CoBEVFlow outperforms even with much smaller communication volume.
Robustness to pose error. We conduct experiments to validate the performance under the impact of both asynchrony and pose error. To simulate the pose error, we add Gaussian noise AO o,) on x, and (O, σr,) on 0 during the inference phase, where x, y. A are 2D centers and yaw angle of accurate global poses. Our pose noise setting follows the Gaussian distribution with a mean of Om, a standard deviation of 0m-0.5m, mean of (1 and standard deviation of 0 0.5%. This experiment conducted under the expectation of time interval is 300ms to simulate the time asynchrony. We compare our CoBEVFlow and other baseline methods including V2X- ViT Where2comm[]
8

---

V2X-VIT
SyncNet
CoBEVFlOw
100ms
300ms
500ms
Figure 6: Visualization of detection results for V2X- SyneNet, and CoBEVFlow with the expectation of time intervals are 100, 300, and 500ms IRV2V dataset. CoBEVFlow qualitatively outperforms the others under different asynchrony. Red and green boxes denote detection results and ground-truth respectively. CoBEVFlow
Where2comm
V2X-VIT
SyncNet
100ms
300ms
500ms
Figure 7: Visualization of detection results for Where2comm, V2X-VIT, SyncNet, and our CoBEVFlow with the expectation of time intervals 100, 300, and 500ms the DAIR-V2X dataset. Red and green boxes denote detection results and ground-truth respectively.
and SyncNet 20) Table shows the results on IRV2V and DAIR-V2X 14 dataset. We see that CoBEVFlow still performs well even when both pose errors and time asynchrony appear. CoBEVFlow consistently outperforms other methods across all noise settings on the IRV2V dataset. In the case of noise levels of 0.4/0.4, our approach achieves 0.133 and 0.043 improvement over SyncNet.
5.3 Qualitative evaluation
Visualization of detection results. We illustrate the detection results of V2X-ViT, SyncNet, and CoBEVFlow at three asynchrony levels on the IRV2V dataset in figure and the DAIR- V2X dataset in figure The expectations of time intervals are 100, 300, and 500ms. The red box represents the detection result and the green box represents the ground truth. V2X-VIT shows significant deviations in collaborative perception under asynehrony, while SyncNet shows poor compensation due to introducing noise in feature regeneration and iregulaarty.inccomatbbl design. The third row shows the results of CoBEVFlow, which achieves precise compensation and outstanding detections.
Visualization of BEV flow map. Figure visualizes the feature map before/after compensation of CoBEVFlow in Plot(a)(b), the corresponding flow map in Plot(c), and matching, detection results

---

# #
Asynchronous feature
(c) BEV flow map
(b) Feature map compensated by BEV flow
(d) Matching compensation results
Figure 8: Visualization of compensation with CoBEVFlow IRV2V dataset. In subfigure(d), green boxes the objects' ground truth locations, blue boxes are the detection results based on the historical asynchronous features and red boxes are the detection results after compensation. CoBEVFlow achieves precise matching and compensation with the BEV flow map and mitigates the negative impact of asynchrony to a great extent.
after compensation Plot(d). The green boxes in Plot (d) are the ground truth, the blue boxes are the historical detections with the matched asynchronous ROIs and the red boxes are the compensated detections. We see that the BEV flow map can be precisely estimated and is beneficial for perceptual feature alignment. The compensated detection results are more accurate than the uncompensated ones.
5.4 Ablation Study
We conduct ablation studies on the IRV2V dataset. Table Soiieeaasssssss the effectiveness of the proposed operations, including time encoding, the object(feature/deeeeeed box) to warp, and our ROI matcher. We see that: i) time encoding encodes the irregular continuous timestamp and makes the estimation more accurate; ii) Warp- ing the features outperforms warping the boxes directly a lot, which means the operation for fea-
Table 2: Ablation Study on IRV2V dataset. Time en- coding(TE), BEV flow features, and the proposed matcher all improve the performance.
Modules
AP AP@0.70
TE
Warped Flow
Matcher
300ms
500ms
Box
Hungarian
0.724/0.473
0.644 / 00,38
Box
Hungarian
0.747 / 0.595
0.668/00 /0.438
Box
Ours
0,764 0.764/0.611 0.611
0,571 0,339
tures shows superiority over the operation for the detection results; and iii) our ROI matcher shows more proper matching results than tradi- tional Hungarian [1 matching.
Feature
Hungarian
0.739/ 0.614
Feature
Ours
0.831/0.75 0.7757
0.815 0.687
6 Conclusion and limitation
We formulate the asynchrony collaborative perception task, which considers various unideal factors that may cause communication latency or information misalignments during collaborative com- munication. We further propose CoBEVFlow, a novel asynchrony-robut framework. The core idea of CoBEVFlow BEV flow, which is collection of the motion vector corresponding to each spatial location. Based on BEV flow, asynchronous perceptual features can be reassigned to appropriate positions, mitigating the impact of asynchrony. Comprehensive experi- ments show that CoBEVFlow achieves outstanding performance under all settings and far superior robustness with asynchrony.
collaborative perception
Limitation and future work. The current work focuses on addressing the asynchrony problem in collaborative perception. is evident that effective prediction can compensate for the negative impact of temporal asynchrony in collaborative perception. Moreover, the generated flow can also be utilized only for compensation but also for prediction. In the future, expect more works on exploring the ROI-based flow generation design for collaborative perception and prediction tasks.
Acknowledgment. This research is supported by NSFC under Grant 62171276 and the Science and Technology Commission of Shanghai Municipal under Grant 2151 100900, 2 1006000,, and 2217/2229005.
10

---

References
[1] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication--fficccn collaborative perception via spatial confidence maps. In Advances in Neural Information Processing Systems, volume 35, pages 4874 4886, 2022.
[2] Yen-Cheng Liu, Junjiao Tian, Nathaniel Glaser, and Zsolt Kira. When2com: Multi-agent perception via communication graph grouping. In Proceedings of the IEEE/CVF Conference computer vision and pattern recognition, pages 4106 4115, 2020.
[3] Yen- Ven-Cheng Liu, Junjiao Tian, Chih- Yao Ma, Nathan Glaser, Chia- Wen Kuo, and Zsolt Kira. Who2com: Collaborative perception via learnable handshake communication. In 2020 IEEE International Conference on Roboties and Automation (ICRA), pages 6876-6883. IEEE, 2020.
[4] #sun-Hsuan Wang, Sivabalan Manivasagam, Ming Liang, Bin Yang, Wenyuan Zeng, and Raquel communication for joint perception and prediction. In Computer Vision-ECCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 1! 16, pages 605-621. Springer, 2020.
Urtasun. V2vnet:
[5] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuar Yang. and Jiaqi Ma. V2x-vit: Shhicle-to-everythine cooperative perception with vision transformer. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIX, pages 107-124. Springer, 2022.
[6] Zhi Li, Ali Vatankhah Barenji. Jiazhi Jiang, Ray Y Zhong, and Gangyan Xu. A mechanism for scheduling multi robot intelligent warehouse system face with dynamic demand, Journal of Intelligent Manufacturing, 31(2):469-48)) 2020.
[7] Michela Zaccaria, Mikhail Giorgini, Riccardo Monica, and Jacopo Aleotti. Multi-robot multiple camera people detection and tracking in automated warchouses. In 2021 IEEE 19th International Conference on Industrial Informatics (INDIN) pages 1-6. IEEE, 2021.
[8] Siheng Chen, Baoan Liu, Chen Feng, Carlos Vallespi-Gonzalez, and Carl K. Wellington. 3d point cloud processing and learning for autonomous driving: Impacting map creation, localization, and perception. IEEE Signal Processing Magazine, 38:68-86, 2021.
[9] Yiming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng. and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Processing Systems, 34:29541-2955 2021.
[10] Ebtehal Turki Alotaibi, Shahad Saleh Alqefari, and Anis Koubaa. Lsar: Multi-uav collaboration for search and rescue missions. IEEE Access, 7:55817-55832, 2019.
[11] Yue Hu, Shaoheng Fang, Weidi Xie, and Siheng Chen. Aerial monocular 3d object detection. IEEE Robotics Autom. Lett., 1(4):1959-1966 1966, 2023.
[12] Jürgen Scherer, Saced Yahyanejad, Samira Hayat, Evsen Yanmaz, Torsten Andre, Asif Khan, Vladimir Vukadinovic, Christian Bettstetter, Hermann Hellwagner, and Bernhard Rinner. An autonomous multi-uav system for search and rescue. Proceedings of the First Workshop on Micro Aerial Vehicle Networks, Systems, and Applications for Civilian Use, pages 33-38, 2015.
[13] Runsheng Xu, Hao Xiang. Xin Xia, Xu Han, Jinlong Li, and Jiaqi Ma. OPV2V: an open benchmark dataset and fusion pipeline for perception with rehicle-to-vehicl communication. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, pages 2583-2589. IEEE, 2022.
[14] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. DAIR- V2X: large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In IEEE/CVE Conference on Computer Vision and Pottern Recognition, CVPR 2022, New Orleans, IA, USA, June 18-24, 2022, pages 21329-21338 IEEE, 2022.

---

[15] Yiming Li, Dekun Ma. Ziyan An, Zixun Wang. Yiqi Zhong, Siheng Chen, and Chen Feng. V2x-sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. IEEE Robotics Autom. Lett., 104):10914-1092,, 2022.
[16] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning, pages 2062- 2072. PMLR, 2023.
[17] Yifan Lu, Quanhao Li, Baoan Liu, Mehrdad Dianati, Chen Feng, Siheng Chen, and Yanfeng Wang. Robust collaborative 3d object detection in presence of pose errors. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 June 2, 2023, pages 4812 4818. IEEE, 2023.
[18] Hao Xiang, Runsheng Xu, Xin Xia, Zhaoliang Zheng, Bolei Zhou, and Jiaqi Ma. V2XP-ASG: generating adversarial scenes for vehicle-to-everythingg perception. In IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 June 2, 2023, pages 3584-3591. IEEE, 2023.
[19] Yiming Li, Qi Fang, Jiamu Bai, Siheng Chen, Felix Juefei-Xu, and Chen Feng. Among us: Adversarially robust collaborative perception by consensus. arXiv preprint 2031v22003.09495 2023.
[20] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception. In Computer ViCion-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXII, pages 316-332. Springer, 2022.
[21] Xingjian Shi, Zhourong Chen, Hao Wang, Din Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. Advances in neural information processing systems, 28, 2015.
[22] Jacques Ferber and Gerhard Weiss. Multi-agent systems: an introduction to distributed artificial intelligence, volume 1. Addison-wesley Reading, 1999.
[23] Anurag Mittal and Larry S Davis. general method for sensor planning multi-sensor systems: Extension to random occlusion. International journal of computer vision, 76(1):31, 2008.
[24] Hao Xiang, Runsheng Xu, and Jiaqi Ma. Hm-vit: Hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[25] Tianhang Wang, Guang Chen, Kai Chen, Zhengfa Liu, Bo Zhang, Alois Knoll, and Changjun Jiang. Ume: unified handwidth-efficieet and multi-resolution based collaborative perception framework. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[26] Kun Yang, Dingkang Yang, Jingyu Zhang, Mingcheng Li, Yang Liu, Jing Liu, Hanqi Wang, Peng Sun, and Liang Song. Spatio-temporal domain awareness for multi-agent collaborative perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[27] E Oran Brigham. The fast Fourier transform and its applications. Prentice-Hall, Inc., 1988.
[28] Timo Teräsvirta. Specification, estimation, and evaluation of smooth transition autoregressive models. Journal of the american Statistical association, 89(425):088-218 1994.
[29] Gary Bishop, Greg Welch, et al. An introduction to the kalman filter. Proc of SIGGRAPH, Course, 8(27599-23175)41, 2001.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1185:1735-1780, 1997.
[31] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv: 1412.3555 2014.
12

---

[32] Yao Qin, Dongjin Song, Haifeng Chen, Wei Cheng, Guofei Jiang, and Garrison W. Cottrell. A dual-stage attention- based recurrent neural network for time series prediction. In Carles Sierra, editor, Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017, pages 2627-2633. ijcai.org, 2017.
[33] Satya Narayan Shukla and Benjamin M. Marlin. Multi-time attention networks for irregularly sampled time series. In 9th International Conference on Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021, OpenReview.net, 2021.
[34] Satya Narayan Shukla and Benjamin M. Marlin. Interpolation-prediieee networks for irregu- larly sampled time series. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[35] Yinjun Wu, Jingchao Ni, Wei Cheng, Bo Zong, Dongjin Song, Zhengzhang Chen, Yanchi Liu, Xuchao Zhang, Haifeng Chen, and Susan B. Davidson. Dynamic gaussian mixture based deep generative model for robust forecasting on sparse multivariate time series. In Thirty- AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thind Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances Artificial Intelligence, FAAI 2021, Virtual Event, February 2-9, 2021, pages 651-659, AAAI Press, 2021.
[36] Max Horn, Michael Moor, Christian Bock, Bastian Rieck, and Karsten Borgwardt. Set functions for time series. In International Conference on Machine Learning, pages 2020.
PMLR,
[37] Xiang Zhang, Marko Zeman, Theodoros Tsiligkaridis, and Marinka Zitnik. Graph-guided network for irregularly sampled multivariate time series. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[38] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6999999 6008, 2017.
[39] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. Carla: An open urban driving simulator. In Conference robot learning, pages 1-16. PMLR, 2017.
[40] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 12697-12705. Computer Vision Foundation IEEE, 2019.
[41] Harold W. Kuhn. The Hungarian Method for the Assignment Problem. Naval Research Logistics Quarterly, 2(1-2):83 97, March 1955.
13

---

A IRegular V2V (IRV2V)
To facilitate the research on asynchrony for collaborative perception, we use CARLA 39 (under MIT license) to simulate IRregular V2V (IRV2V) dataset, which is the first collaborative perception dataset with multiple asynchronies.
Asynchronous data collection. The number of collaborative vehicles in scene ranges from to 5. Fach collaborative vehicle is equipped with 4 cameras for 360 view, a 32-channel LiDAR, and GPS/IMU sensors. The ideal sample interval of the sensor is 100ms. Due different asyn- chronous factors, collaborative messages have asynchronous timestamps. There is time offset 6, ((-50, 50)ms at the sampling starting point of non-ego vehieles. And all non-ego vehicles' collaborative messages are sampled with time turbulence dg ~ U(-10, 10)ms. Sensing information each timestamp of each agent contains camera images with resolution 600 x 800, and 32-channel
LiDAR points.
Data size. Assuming the model requires the use of information from the past 10 frames, dataset consists of total of 8,449 collaborative samples, which include 8,449 point cloud inputs and 33,796 RGB images. We have split the dataset into training, validation, and testing sets, which contain 5,445, 994, and 2,010 samples, respectively.
Data analysis. Figure presents some statistical analysis results regarding the IRV2V dataset. The IRV2V dataset contains a total of 1,564,033 vehicles, with an average of 48.302 vehicles per scene. should be noted that the figure only displays the distribution of vehicles with speeds greater than 1 km/h. Considering veal-world scenarios, there are around 1,203,793 7,3 moving vehicles in the dataset. Plot (a) illustrates the distribution of moving vehicles with different speeds across all samples, ranging from to 105 km/h, with an average speed of 25.586 km/h, which achieves around 15km/h faster compared to the majority of vehicles in the V2X-Sim(15) Plot (b) shows the distribution of the total number of vehicles per sample in the dataset, with the maximum number vehicles being 113.
60000
3000
Number of vehicles
50000
Number of scenes
2500
40000
2000
30000
1500
20000
1000
10000
500
40
80
100
20
60
100
Speed the vehicle (km/h)
Number of vehicles the scene
(a) Vehicle speed distribution.
(b) Number vehicles scene distribution.
Figure 9: Data distribution of IRV2V dataset. (a) shows the speed distribution of moving vehicles; (b) shows the vehicle numbers distribution.
B Detailed information about experimental settings
Implement details. We conduct experiments on LiDAR-based part of IRV2V and DAIR- [4] dataset. Our feature encoder is PointPillars 40 based. And our backbone follows the setting in [1]] The difference is that we change the fusion method from self-attention to max max-fusion. We conduct training for total of 60 epochs, starting with an initial learning rate of 2e-3. Subsequently. at the 10th and 20th epochs, the learning rate decreases to 10% of its previous value. For IRV2V dataset, we set the lidar range as x E 1440.8, + +140.8 m, y/ E [040, +40 m. The voxel size is h = = 0.4m. The feature map's size is H 200, W 704. For DAIR-V2X dataset, we set the lidar range as x E |- 100.8, + 100.8 8) m, y E [-40, +-40 m. The voxel size is h w 0.4m. The feature map's 's size is H 200, W 504.
14

---

Communication volume. Our communication volume is the same Where2comm]] [ For CoBEVFlow, we control the communication volume by adjusting the maximum number of generated ROIs. Specifically, the average number of voxels contained in each ROI region is 40, and we limit the maximum number of generated ROIs to |R] Correspondingly, we modify the information exchange in Where2comm to include the top 40 X K [| blocks based on their scores on the spatial confidence map. In practical scenarios, the actual communication volume is influenced by factors such the feature dimension and floating-poin precision. To simplify the expression, we uniformly represent the communication volume using the logarithm to the base of the voxel count. The communication volume is
loga (40 x K K(Rl)
(6)
where K the maximum number of generated ROIs, and 40 is the average number of voxels in each ROI.
C
Benchmarks
We conduct extensive experiments on current collaborative perception methodologies. Table 3 and Table present the detection performance under the expectation of time interval from 0 to 500ms IRV2V and DAIR- [2) respectively, which correspond to the numerical results shown in Figure 4 in the main text. We see that CoBEVFlow consistently achieves significant improvements previous methods on both datasets and the leading gap is bigger when the expectation the time interval is higher.
Table 3: Performance of CoBEVFlow and other baseline methods under the expectation of time interval from 0 to 500ms IRV2V dataset. CoBEVFlow outperforms all the baseline methods and shows great robustness under any level of asynchrony.
Expectation of interval (ms)
100
200
300
400
500
Model / Metric
AP@ t
Single
0.647
Late Fusion
0,828
0.638
0.478
0.371
0.324
0.308
V2VNet
0.811
0.747
0.710
0.663
0.626
0.591
VIX-VIT
0.781
0.737
0.692
0.641
0.598
0.575
DiscoNet
0.742
0.728
0.704
0.673
0.647
0.625
Where2comm
0.864
0.758
0.609
0.510
0.455
0.431
Where2comm--SnnNeet
0.864
0.721
0.672
0.654
0.649
0.625
CoBEVFlow (ours)
0.864
0.841
0.834
0.831
0.812
0.815
Model / Metric
AP@ 0.70 +
Single
0.535
Late Fusion
0.751
0.385
0.285
0.235
0.219
0.223
V2VNet
0.744
0.607
0.540
0.480
0.439
0.408
V2X-VIT
0.630
0.577
0.545
0.511
0.489
0.479
DiscoNet
0.624
0.612
0.586
0.559
0.537
0.519
Where2comm
0.827
0.613
0.458
0.388
0.362
0.359
Where2comm--yneNNe
0.827
0.602
0.555
0.549
0.545
0.536
CoBEVFlow (ours)
0.827
0.781
0.761
0.757
0.714
0.687
We extended our experiments to include the V2XSet dataset. V2XSet is simulated dataset where each scenario involves at most one roadside unit and 2 to vehicles as collaborative objects. The outcomes of these experiments are summarized Table Under time delays of 300ms and 500ms, the AP@0.70 scores achieved by CoBEVFlow are 0.776 and 0.713 respectively. These values surpass the best baseline methods by 17.0% and 10.6% respectively, and are notably higher than the results of single-object detection(0.5555 (0.556). This once again underscores CoBEVFlow's 's ability maintain high levels of collaborative perception performance in scenarios involving temporal asynchrony.
15

---

Table 4: Performance of CoBEVFlow and other baseline methods under the expectation of time interval from 0 to 500ms on DAIR-V2X dataset. CoBEVFlow outperforms all the baseline methods and shows great robustness under any level of asynchrony.
Expectation of interval (ms) 0
100
200
300
400
500
Model / Metric
AP@ 0.50 f
Single
0.674
Late Fusion
0.709
0.664
0.616
0.600
0.595
0.593
V2VNet
0.626
0.623
0.622
0.617
0.612
0.608
V2X-ViT ViT
0.725
0.714
0.704
0.693
0.681
0.681
DiscoNet
0.645
0.630
0.620
0.606
0.597
0.590
Where2comm
0.807
0.739
0.603
0.698
0.686
0.676
Where2comm--SneNNe
0.807
0.719
0.706
0.695
0.687
0.679
CoBEVFlow (ours)
0.807
0.765
0.749
0.738
0.728
0.726
Model / Metric
AP@ 0.70 t
Single
0.587
Late Fusion
0.538
0.479
0.467
0.464
0.466
0.465
V2VNet
0.556
0.555
0.554
0.552
0.548
0.548
V2X-VIT
0.556
0.553
0.550
0.545
0.541
0.541
DiscoNet
0.553
0.537
0.530
0.523
0.519
0.518
Where2comm
0.662
0.603
0.587
0.577
0.569
0.566
Where2comm-SyneNa
0.662
0.602
0.588
0.587
0.584
0.580
CoBEVFlow (ours)
0.662
0.621
0.601
0.599
0.592
0.588
Table 5: We conduct experiments on the V2XSet dataset, comparing the performance of CoBEVFlow and other baseline methods under different expected time intervals. CoBEVFlow outperforms all the baseline methods and shows great robustness under any level of asynchrony.
Expectation of interval (ms)
0
300
500
Model Metric
AP@0.50/AP@0.701
Single
0
Late Fusion
0.881/0.751
0.460/0.307
0.442/0.333
V2VNet
0.906/0.781 781
0.581/0.334
0.524/0.341
V2X-VIT
0.912/0.751
0.712/0.529
0.642/0.499
Where2comm
0.901/0.847
0.677/0.542
0.612/0.511
WoenSScommt
0.901/0.847
0,801/0.663
0.781/0.66 /4.5
CoBEVFlow (ours)
0.901/0.847
0.871/0.776
0.841/0.713
16


Reviews:
[{"review_id": "arIV80PRnR", "reviewer": "Reviewer_ZUTg", "summary": "The paper introduces CoBEVFlow, a new asynchrony-robust collaborative 3D perception system designed to enhance multi-agent perception abilities. The method compensates motions to align asynchronous collaboration messages from different agents, aiding in dealing with real-world issues such as communication delays. The authors developed BEVflow to reassign asynchronous perceptual features to appropriate positions and thus mitigating the impact of asynchrony. \n\nTo evaluate its effectiveness, a new dataset IRregular Vehicle-to-Vehicle (IRV2V) was created that simulates various real-world scenarios. Extensive experiments on both this new dataset and an existing dataset DAIR-V2X showed that CoBEVFlow consistently outperforms other baseline methods (evaluated on Vehicle detection).", "questions": "1. Will the dataset be released for future research?", "limitations": "There is a discussion about limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The challenge of temporal asynchrony in multi-agent collaborative perception is quite new and the proposed CoBEVFlow firstly addresses it by reassigning asynchronous perceptual features (at RoI level) to ensure better alignment.\n2. The creation of the IRV2V dataset to simulate real-world scenarios provides an important resource for future research.\n3. Detailed evaluations of two datasets provide good results (Fig. 4) for the superiority of CoBEVFlow over existing methods.\n4. With smaller bandwidth, Fig.5 shows CoBEVFlow actually shows the potential to greatly improve the performance over single-object detection. \n\n", "weaknesses": "1. The main concern is the limited evaluation. There might be limitations in applying CoBEVFlow to other types of multi-agent systems as it was mainly tested on vehicle-to-vehicle communication scenarios. Is there any reported performance of Pedestrian or Cyclist?\n2. A background paragraph (Sec.3) about the comparison between collaborative and original settings (perhaps a visual comparison rather than a tedious explanation in text) could be more elaborate for better understanding for new readers. Even if I was working on the 3D detection, I cannot understand the key difference quickly.\n3. While CoBEVFlow shows potential in simulated scenarios, its performance in broader, real-world situations remains untested. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Limited Evaluation and Applicability\n1. Besides vehicle-to-vehicle collaboration communication, we also report the result of experiments on the vehicle-to-infrastructure dataset DAIR-V2X in the main text. This dataset includes information from both the vehicle side and the roadside units, enabling communication between vehicles and infrastructure. Figure.4(b) in the main text and Table 4 in the appendix show that our approach -- CoBEVFlow outperforms the best methods by 5.73% and 2.04% in terms of AP@0.50 and AP@0.70, respectively, under a 300ms interval expectation, which exhibits commendable performance on the vehicle-infrastructure collaborative dataset in real-world scenarios. The visualization of detection results can be referenced in Figure 8 in the Appendix. All these results suggest that our method can effectively handle arbitrary multi-agent systems and exhibits superior efficacy in the real-world setting.  \nIn addition to the two datasets mentioned in the main text, based on suggestions from other reviewers, we also conducted supplementary experiments on the **V2XSet** dataset. V2XSet is a simulated vehicle-to-infrastructure dataset, which does not consider asynchrony issues. The experimental setup was consistent with the other datasets. The specific results can be seen in Table 1 of the global response PDF.CoBEVFlow outperforms the best SOTA methods by 8.74%, and 17.04% for AP@0.50 and AP@0.70 respectively under the expectation of time intervals is 300ms. The findings obtained from experiments on the V2XSet dataset align with the conclusions drawn from experiments on the other two datasets in this paper.\n2. Considering more object categories in the evaluation of multi-agent perception systems is a valuable suggestion. The two datasets reported in the main text – the IRV2V dataset and the collaborative portion of the DAIR-V2X dataset, as well as the additional V2XSet dataset provided in the rebuttal, all of them do not include labels for Pedestrians and Cyclists. Consequently, we cannot present corresponding detection results. Moreover, as far as we know, none of the existing collaborative detection datasets currently have annotations for these two entities.\n\n---\n\n## W2: Comparing Collaborative and Original Settings\nWe appreciate the reviewer's suggestion! \n1. In the future version, we will provide a more detailed description of collaborative sensing. \n2. The difference between original and collaborative settings is: Under the original setting, the ego agent can only use the information obtained from its own sensors for detection. However, in a collaborative setting, **different agents can transmit and share information, meaning the ego agent can use information from other agents for detection in addition to its own information**, thereby expanding and enhancing its field of perception. By leveraging complementary perceptual information through communication, agents overcome the inherent limitations of single-agent perception, such as occlusion and long-range issues. In Figure 4 of the main text, the gray dashed lines represent individual perception, while the solid lines represent collaborative perception methods. Under the condition of zero expectation of time interval, the performance of collaborative perception methods is better than that of single perception. However, as the degree of temporal asynchrony increases, the performance of methods like late fusion, V2VNet, V2X-ViT, Disconet, and Where2comm are affected by temporal asynchrony and even fall below the level of single perception. This is precisely the problem we aim to address. As shown by the red line in Figure 4, in the case of temporal asynchrony, our objective is to maintain the performance of collaborative perception at a higher level as much as possible.\n\n---\n\n## W3: Real-World Performance of CoBEVFlow\nIn the main text, the proposed method is validated on two datasets, our new dataset IRV2V and DAIR-V2X. We believe the experiment results on the DAIR-V2X should reveal the potential of CoBEVFlow in real-world scenarios. DAIR-V2X is the first real-world collaborative perception dataset, which is collected from real road scenes. In this dataset, there is one vehicle and one roadside unit. The communication process takes place between the vehicle and the roadside unit. The dataset is collected from real-world scenarios, encompassing a wide range of conditions including clear, rainy, and foggy days, as well as daytime, nighttime, urban roads, and highways. It is extensively employed in collaborative perception research. Figure.4(b) in the main text and Table 4 in the appendix show that our approach -- CoBEVFlow outperforms the best methods by 5.73% and 2.04% in terms of AP@0.50 and AP@0.70, respectively, under a 300ms interval expectation. In Figure 4(b), the red line represents our method, CoBEVFlow, and it can be observed that our method consistently outperforms others. In Appendix Figure 8, we present the visualization of detection results on the DAIR-V2X dataset with an expected interval of 300ms. The red boxes indicate the predicted results, while the green boxes represent the ground truth. In the presence of temporal asynchrony, our method's predictions closely match the ground truth, resulting in fewer false positives. These results demonstrate that CoBEVFlow continues to achieve superior performance in real-world scenarios.\n\n---\n\n## Q1: Dataset Availability for Future Research\n**Yes**, the dataset and the code both will be fully accessible to support research in the field.\n"}, {"review_id": "kxsYwZnFe8", "reviewer": "Reviewer_iS3b", "summary": "This work points out that there is a time delay among agents and the delay period is not fixed. Thus, when these agents share their environment perception information in BEV, there will be an uneven spatial mismatch. To address the aforementioned problem, this work constructs a benchmark and proposes a strategy to align the BEV feature of various agents by predicting BEV flow map.", "questions": "not.", "limitations": "Please see the weakness section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThis work finds a new problem and develops a corresponding benchmark. This studied problem is interesting and meaningful.\n2.\tThe proposed strategy for tackling the asynchronous perception problem is easy to understand and makes sense.\n", "weaknesses": "1.\tThe proposed algorithm predicts the BEV flow estimation rather than movement speed, so the communication delay among agents should be known. How to know that in practical applications. \n2.\tIn the developed benchmark, there is a random time turbulence. How to tackle this turbulence?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Communication Delay in Practical Applications\nIn practical applications, it is straightforward to know the communication delay because different vehicles can easily acquire a unified world timestamp. \n1. A unified world timestamp is easily obtainable for agents. Several existing technologies can achieve this capability, such as GPS systems, network time servers, and online time services.\n2. When vehicles send information, they also transmit the corresponding world timestamp. Thus, the information received by the ego agent comes with its specific timestamp, which allows the computation of the exact time interval and delay. Given the globally synchronized time stamps, the time intervals or delays can be easily calculated.\n---\n## W2: Tackle Random Time Turbulence\nHere we assume that \"Tackle this turbulence\" mentioned by the review means the process of mitigating the effect of temporal asynchrony. Here are the responses:\n1. Regarding temporal asynchrony in real-world scenarios: The 'turbulence' is a crucial factor of the temporal asynchrony in the data. Correspondingly, in real scenarios, this turbulence emulates a variety of potential unfavorable factors, including but not limited to situations where the sensors or associated hardware are unable to handle their tasks, resulting in data collection not proceeding entirely at the predetermined frequency; disruptions like illumination, electromagnetic waves, and unstable hardware and software drivers that could interfere with the precise moments when the sensor collects information. These adverse factors all contribute to turbulence at the time of data collection. The method we propose in our paper can address asynchrony caused by any factor.\n2. About the datasets: IRV2V is a synthetic dataset created by adjusting the sampling times to simulate temporal asynchrony. The sampling times are continuous and more complex, which better reflects real-world applications. The other dataset we used,  DAIR-V2X, is a real dataset sampled at 10Hz. We simulate temporal asynchrony in situations like communication delays by selecting non-aligned perception information.\n3. About the baseline methods: State-of-the-art collaborative perception methods like late fusion, DiscoNet, Where2comm, etc., do not consider the impact of temporal asynchrony. In our testing, we directly use asynchronous collaboration information as inputs to these models. SyncNet addresses temporal delays but assumes fixed time intervals between input information. In our experiments, we directly use irregularly spaced perception information as input, which proves to be challenging for these methods to handle effectively.\n\nIn summary, CoBEVFlow addresses the challenges of temporal asynchrony in real-world scenarios, and our experiments (Figure 4) show that it outperforms other methods that do not explicitly consider such 'turbulence'. Please let me know if you have any further questions.\n\n"}, {"review_id": "IRyX6Qv6uk", "reviewer": "Reviewer_NCLy", "summary": "The paper proposes CoBEVFlow, a new system for collaborative 3D perception that can handle temporal asynchrony among multiple agents. CoBEVFlow compensates for motion to align asynchronous collaboration messages and has two advantages: it can handle irregular time stamps without discretization and only transports original perceptual features. The authors validate CoBEVFlow's efficacy with a newly proposed synthetic dataset IRV2V and an existing real-world dataset (DAIR-V2X), showing that CoBEVFlow outperforms other methods and is robust in extremely asynchronous settings. Overall, the paper presents a novel approach to collaborative perception that improves collaboration among multiple agents.", "questions": "- Time complexity analysis of the proposed ROI matching method and the Hungarian matching method.\n\n- Based on the information provided, it is unclear why the performance of Where2comm+SyncNet increases with a decrease in communication volume in Figure 5. Further investigation is needed to understand the reasons behind this phenomenon.", "limitations": "Yes, they have addressed them.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper contributes to the formulation of the asynchrony collaborative perception task and proposes a novel asynchrony-robust collaborative perception framework. Compared with the existing methods to solve the delay, the proposed benchmark is more challenging and closer to real-world scenarios.\n\n- CoBEVFlow has two advantages over other methods. Firstly, it can handle asynchronous collaboration messages sent at irregular, continuous time stamps without discretization. Secondly, it only transports the original perceptual features, avoiding additional noise and generating new perceptual features.\n\n- Comprehensive experiments conducted on both IRV2V and a real-world dataset, DAIR-V2X, show that CoBEVFlow consistently outperforms other baselines and is robust in extremely asynchronous settings.\n\n- The writing in this paper is good, and the figures and charts are clear.", "weaknesses": "- The training process requires three stages, which are complex and time-consuming, which limits practical application scenarios.\n\n- The collaboration performance is constrained by the performance of the ROI generator.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Training Process Complexity and Practical Application\nThe training process does require three stages, but it is not time-consuming and does not limit practical application scenarios for two reasons. \n1. First, each of the three modules is lightweight. The training of the individual detection module, and the finetuning of the collaborative components, are common to all collaborative perception tasks. The stage of training the prediction module is both lightweight and fast. The entire prediction training stage takes only a matter of minutes and does not entail significant additional time consumption during the inference time.\n2. Additionally, the inference process is also not time-consuming. The time required for inference remains unaffected by the intricacies of the training stages. It is noteworthy that the inference time doesn't exhibit a mere increase as a result of its nature of multi-stage training. To support our point, We examine the inference time increment compared to the Where2Comm approach, which does not account for temporal asynchrony, our method introduces only a marginal 3% increment in time consumption during the inference time with proper implementation optimization. Furthermore, we observed that when compared to SyncNet, another method that equally considers delay compensation, our approach remarkably demonstrates a reduction of 33% in time consumption. \n---\n## W2: Performance Dependency on ROI Generator\nThat's a good question. We agree with the reviewer's insightful comment. But we want to emphasize two points:\n1. ROI generator generally provides reliable ROIs. The performance of the ROI generator is strongly correlated with the performance of the single-object detection. There are many mature and powerful single-object detection methods available now. In most cases, we can easily find an excellent single-object detection method and adopt it as the ROI generator with minor modifications. According to our experiments, the ROI generator can easily achieve commendable results even using the most common 3D detectors, such as PointPillar, which we use to generate the experiments results table in the manuscript. We will add this information to the revised version. \n2. When the ROI generator fails, our collaboration performance is still better than baselines. In extreme cases where the ROI generator performs poorly, it indicates that it is difficult to find a sufficiently good single-object detection method. In this case, all collaborative perception methods would struggle to perform well. However, even if only a portion of the ROIs are accurate and effective, our method can still compensate for them, thereby achieving stronger robustness to time asynchrony compared to other methods.\n---\n\n## Q1: Time Complexity Analysis of ROI Matching Methods\n\nThe theoretical time complexity of our matching scheme is **$O(n)$**, while the general theoretical time complexity of the Hungarian algorithm is **$O(n^3)$**, so our scheme has a significant advantage in terms of theoretical time complexity. Given that about 50 ROIs are typically encountered during the detection process, our method only consumed 12ms to complete the matching task, while the Hungarian algorithm took about 16ms. In practice, the time consumption for matching is a very small part of the overall inference process and can basically be ignored.\n\n---\n## Q2: Investigating SyncNet's Performance Increase with Decreased Communication Volume\nThe reason that the performance of Where2comm+SyncNet increases with a decrease in communication volume might be that SyncNet applies a Conv-LSTM-based compensation strategy which struggles to accurately conduct motion compensation when features have lower quality. With the increase in communication volume, features with lower confidence are more likely to be transmitted compared to situations when the communication volume is lower. It indicates that the overall qualities of the features become lower when the communication volume increase. Therefore, the motion compensation capability of SyncNet tends to be worse in such situations and results in worse performance. To support our point, Figure 1 in the global response PDF visualizes the detection results of CoBEVFlow and Syncnet under different bandwidths. We can observe that as the bandwidth increases, the number of false positives in Syncnet's detection results noticeably grows, showing some misalignment with the ground truth.\n"}, {"review_id": "IKNS5ISyIa", "reviewer": "Reviewer_vzpF", "summary": "This article proposes a robust detection algorithm to address the issue of detection errors caused by asynchronous information transmission in multi-agent collaborative perception tasks. For late fusion, a robust prediction algorithm is designed using a BEV flow map generation algorithm, which provides a method to align perception information from different time frames. Additionally, the authors claim to have introduced the first asynchronous collaborative perception dataset and tested the algorithm on this dataset as well as the DAIR-V2X dataset.", "questions": "1. The experiments in this paper are not comprehensive enough, as they only validate the proposed methods on the dataset introduced in this paper, but lack testing on datasets such as V2XSet[1].\n2. The research motivation in this paper focuses on the time intervals between timestamps, but it seems insufficient to be a major concern. Even if the time intervals for information transmission between agents are inconsistent, the prediction algorithm can still rely on predicting algorithms combined with several consecutive data transmissions. In other words, does the algorithm's performance significantly differ when the time intervals are the same or different?\n3. In the section \"Adjacent frames' ROI matching,\" the author uses the Euclidean distance between the same ROI in two consecutive data transmissions from the same agent as the association criterion. However, when there is high vehicle density in the environment, this can significantly impact the association algorithm. How is this issue addressed?\n4. Also in the section \"Adjacent frames' ROI matching,\" how is the \"feasible angle range\" defined in line 213?\n\n[1]. Xu, et al. V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer. ECCV 2022.\n", "limitations": "The proposed algorithm in this paper seems to be affected by complex vehicle scenarios.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper addresses an important research problem in multi-agent collaborative perception tasks and proposes a structurally concise and effective 3D detection framework. By utilizing BEV flow, the framework significantly reduces the bandwidth consumption in information transmission among different agents. To overcome the challenge of low matching accuracy in information fusion during asynchronous collaborative perception, the paper introduces a BEV flow map that can align neighboring frames of vehicles. By predicting the BEV flow, the framework aligns detection results with time intervals. Additionally, the paper presents a benchmark dataset specifically designed for asynchronous collaborative perception tasks, which contributes to the advancement of research in this field", "weaknesses": "The writing in this paper can be further improved to enhance readers' understanding of the formulas and methods described. In the main experiments, such as the benchmark comparison in the quantitative evaluation, it would be helpful to include a table that presents the comparative results for easier reference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## W1: Enhancing Clarity of Formulas, Methods, and Experiments\n\nThank you for your feedback. We will revise the corresponding expressions in the final version for better understanding. We have provided data tables in the appendix corresponding to the main text, which can be referred to more accurately.\n\n---\n\n## Q1: Dataset Validation and Comparison\n\nIn the main text, the proposed method is validated on two datasets, our new dataset **IRV2V** and **DAIR-V2X**. Note that DAIR-V2X is the first real-world collaborative perception dataset, which is collected from real road scenes. In this dataset, there is one vehicle and one roadside unit. The communication process takes place between the vehicle and the roadside unit. The dataset is collected from real-world scenarios, encompassing a wide range of conditions including clear, rainy, and foggy days, as well as daytime, nighttime, urban roads, and highways. It is extensively employed in collaborative perception research. The experiments conducted on DAIR-V2X are mainly shown in Figure 4(b), Table 4(Appendix), and Figure 8(Appendix).  Figure.4(b) in the main text and Table 4 in the appendix show that CoBEVFlow outperforms the best methods by 5.73% and 2.04% in terms of AP@0.50 and AP@0.70, respectively, under a 300ms interval expectation. The red line represents our method, CoBEVFlow, and it can be observed that our method consistently outperforms others. In Appendix Figure 8, we present the visualization of detection results on the DAIR-V2X dataset with an expected interval of 300ms. The red boxes indicate the predicted results, while the green boxes represent the ground truth. In the presence of temporal asynchrony, our method's predictions closely match the ground truth, resulting in fewer false positives. These results demonstrate that CoBEVFlow continues to achieve superior performance in real-world scenarios.\n    \nThe V2XSet dataset mentioned by the reviewer is a simulation dataset that does not explicitly consider latency issues. We show the **additional experimental results** on V2XSet in **Table 1** of the global response pdf. CoBEVFlow outperforms the best SOTA methods by 8.74%, and 17.04% for AP@0.50 and AP@0.70 respectively under the expectation of time intervals is 300ms. The findings obtained from experiments on the V2XSet dataset align with the conclusions drawn from experiments on the other two datasets in this paper. CoBEVFlow demonstrates its ability to alleviate the impact of temporal asynchrony and maintain robustness in the presence of temporal asynchrony.\n\n---\n\n## Q2: Impact of Time Intervals\n\nDifferent time intervals lead to significantly different performance for methods that rely on temporal information and assume known fixed time intervals. Under the assumption of regular sampling, the temporal variation information between continuous frames is fixed, and the model does not need to consider or extract the specific time interval between consecutive timestamps. When facing inputs with random different time intervals, the model may mistakenly perceive them as having a fixed interval, leading to incorrect analysis of temporal change speed and incorrect compensation for specific delays. Syncnet, discussed in the paper, is such a method. To validate this point, we trained Syncnet (which is an LSTM-based method) with a fixed time interval of 100ms, while using different time intervals during the inference process. We compensate based on two frames of historical information, fixing the delay of the first frame at 500ms. And we vary the time interval between the second and the first frame at 500, 600, 700, 800, and 900ms. In the table, $\\Delta t$ represents the difference between two time intervals. The experimental comparison is shown in Table 2 in the global response PDF. Syncnet cannot cope well with varying time intervals, its performance is significantly affected. However, our method remains highly robust in the face of time irregularities.\n\n---\n\n## Q3: Addressing Association Algorithm Impact in Dense Environments\n\nFor such situations, we can leverage object tracking methods. During matching, not only the distance but also the similarity of the features covered within the ROI range are considered to solve this problem. However, in most cases, such situations are very rare in all our datasets. Our current matching method still has very accurate matching results even when vehicles are relatively dense, so there is not an urgent need for this kind of requirement.\n\n---\n\n## Q4: \"Feasible Angle Range\" in ROI Matching\n\nThe \"feasible angle range\" is actually the range of angles to which a vehicle may move during regular driving. It is generally a certain angular range with the vehicle's front or rear as the centerline. In our experiments, we set it uniformly to plus or minus 45 degrees."}]

Author rebuttal:
Thank you for your valuable feedback! 

In this work, we propose CoBEVFlow, an asynchrony-robust collaborative 3D perception system based on bird’s eye view flow, to address the issues caused by temporal asynchrony among agents. 

In the main text, we conducted experiments on two datasets: IRV2V and DAIR-V2X[1]. IRV2V is the first synthetic collaborative perception dataset with various temporal asynchronies that simulate different real-world scenarios that we created, containing 2 to 5 vehicles as collaborative agents in each frame. **DAIR-V2X is a real-world dataset** consisting of a roadside unit and a vehicle as collaborative agents in each frame. Our experimental results conducted on both datasets are depicted in Figure 4. The corresponding numerical outcomes are detailed in Appendix Table 4 and Table 5. In the case of the IRV2V dataset, CoBEVFlow demonstrates substantial performance, achieving 83.1% and 75.7% in AP@0.50 and AP@0.70 respectively, within the 300ms time interval. This performance exceeds that of the leading baseline method by 23.3% and 35.3%. Similarly, on the DAIR-V2X dataset, CoBEVFlow achieves remarkable results of 73.8% and 59.9% in AP@0.50 and AP@0.70 respectively, within the 300ms time interval, outperforming the best baseline method by 5.73% and 2.04%.

Furthermore, during this rebuttal process, we extended our experiments to include the V2XSet[2] dataset. V2XSet is a simulated dataset where each scenario involves at most one roadside unit and 2 to 4 vehicles as collaborative objects. The outcomes of these experiments are summarized in Table 1 of the global response PDF. Under time delays of 300ms and 500ms, the AP@0.70 scores achieved by CoBEVFlow are 0.776 and 0.713 respectively. These values surpass the best baseline methods by **17.0%** and **10.6%** respectively, and are notably higher than the results of single-object detection(0.556). This once again underscores CoBEVFlow's ability to maintain high levels of collaborative perception performance in scenarios involving temporal asynchrony.

The attached PDF contains various tables and figures. Please review it. 

[1] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang, Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 21329–21338. IEEE, 2022.

[2] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang, and Jiaqi Ma. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIX, pages 107–124. Springer, 2022.

Meta-review: In order to address the issues caused by temporal asynchrony among agents, this paper introduces a novel asynchrony-robust collaborative 3D perception framework, based on bird’s eye view flow. All the four reviewers recommend accept, and they pointed out that the task is important and new, and the proposed method is interesting and effective. The authors also did comprehensive evaluations. Though there are some weaknesses such as limited efficiency, considering the novelty of the task and method, the AC recommends accept.
Decision: Accept (poster)