Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection
Authors: Haibao Yu
Yingjuan Tang
Enze Xie
Jilei Mao
Ping Luo
Zaiqing Nie
Venue: NeurIPS 2023 poster (2023)
Keywords: vehicle-infrastructure cooperative autonomous driving
3D object detection
feature flow
self-supervised learning

Abstract: Cooperatively utilizing both ego-vehicle and infrastructure sensor data can significantly enhance autonomous driving perception abilities. However, the uncertain temporal asynchrony and limited communication conditions that are present in traffic environments can lead to fusion misalignment and constrain the exploitation of infrastructure data. To address these issues in vehicle-infrastructure cooperative 3D (VIC3D) object detection, we propose the Feature Flow Net (FFNet), a novel cooperative detection framework. FFNet is a flow-based feature fusion framework that uses a feature flow prediction module to predict future features and compensate for asynchrony. Instead of transmitting feature maps extracted from still-images, FFNet transmits feature flow, leveraging the temporal coherence of sequential infrastructure frames. Furthermore, we introduce a self-supervised training approach that enables FFNet to generate feature flow with feature prediction ability from raw infrastructure sequences. Experimental results demonstrate that our proposed method outperforms existing cooperative detection methods while only requiring about 1/100 of the transmission cost of raw data and covers all latency in one model on the DAIR-V2X dataset. The code  is available https://github.com/haibao-yu/FFNet-VIC3D.

Flow-Based Feature Fusion for Vehicle-Infrastructure Cooperative 3D Object Detection
Haíbao Yu 1.2, Yingjuan Tang2.3 Enze Xie Jillii Mao', Ping Luo!.A, Zaiqing Nie?
The University of Hong Kong
Institute for Al Industry Research (AIR), Tsinghua University Beijing Institute of Technology "Shanghai Al Laboratory
Abstract
Cooperatively utilizing both ego gehicle and infrastructure sensor data can signifi- cantly enhance autonomous driving perception abilities. However, the uncertain temporal asynchrony and limited communication conditions can lead fusion misalignment and constrain the exploitation of infrastructure data. To address these issues in rehicle-infrastructure cooperative 3D (VIC3D) object detection, we propose the Feature Flow Net (FFNet), novel cooperative detection framework. FFNet flow-based feature fusion framework that uses feature flow predic- tion module to predict future features and compensate for asynchrony. Instead of transmitting feature maps extracted from still-images, FFNet transmits feature flow, leveraging the temporal coherence of sequential infrastructure frames. Furthermore, we introduce self-supervised training approach that enables FFNet to generate feature flow with feature prediction ability from raw infrastructure sequences. Ex- perimental results demonstrate that our proposed method outperforms existing cooperative detection methods while only requiring about 1/100 of the transmission cost of raw data and covers all latency in one model on the DAIR-V2 V2X dataset. The code is available at http:/ghwww...oooaaaaaeeeee/eeee
1 Introduction
Accurate 3D object detection is a critical task in autonomous driving as it provides crucial informa- tion about the location and classification of surround- ing obstacles. Traditional 3D object detection meth- ods rely on onboard sensor data from the ego ve- hicle, which has limited perception field and of- ten fails blind or long-range zones, resulting in safety concerns. To address these challenges, vehicle- infrastructure cooperative autonomous driving has gained much attention, particularly using infrastruc- ture sensors like cameras and LiDARs, which are usually installed higher than ego vehicles, providing a broader field of view (40) 42 25 24).. By utilizing possible
12
FFNet (Ours)
FFNet-C1
Early Fusion
mpp
22
Late Fusion
DiscoNet
17
V2VNet
Transmission (Byte)
additional infrastructure sensor data, to obtain more meaningful information and improve autonomous driving perception ability. In this paper, we focus on solving the vehicle-ing intdee--titrttuture cooperative 3D (VIC3D) object detection problem to enhance the safety and performance of autonomous driving systems in challenging traffic scenarios.
Figure I: Performance us. Transmission Cost on DAIR-V2X Dataset. All results reported with 200ma lntency. FFNet achieves new saat-oof-theare 62.87% mAP@BEV while only requiring about
early
"Correspondngg author. Work done while AIR.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Row Seact
bnrerefiate Dare
Dinction Cupues
Transmiszion Schemes
(o Aliggne Features Point Clonds
Lavited Car Commenaniaannn Cordiion
After diy
Afler d
Affec d
Alter la
(4) Infreimatinn and Vehiele Servor Dat
irattiiii Casperdive Sydcm
(d) Non- Featinas / Pointt ClaesClln Cloads
Cooperative 3D Object Detection. (a) Infrastructure Vehicle Sensor Data. Infrastructure sensor data can provide abundant information for autonomous driving with a broader perception field compared to vehicle sensor data. (b) Vehicle-Infin ihbice-Infrastructure Cooperative System. There are three potential data forms for transmission: raw data for early fusion, sntermediate-levee data for middle fusion, and detection outputs for late fusion. Due limited communication conditions, the infrastructure information may be received by any vehicle with an uncertain latency, resulting in uncertain temporal asynchrony. (c-d) Aligned and Non- aligned Point Clouds and Features. Non-aligned point clouds and features can cause fusion misalignment and affect the exploitation of infrastructure data, potentially impacting the performance of the cooperative detection.
Figure 2:
The VIC3D problem can be formulated as a multi-sensor detection problem under constrained communication bandwidth, presenting two main challenges. First, infrastructure data can be received by any vehicle, and the data captured by ego-vehicle sensors and received from infrastructure devices have asynchronous timestamps with uncertain differences. Second, the communication bandwidth between the two-side devices is limited. Recent studies (40) [4] 33) have attempted to address this problem and proposed three major fusion frameworks for cooperative detection: early fusion, late fusion, and middle fusion. Early fusion involves transmitting raw data like raw point clouds, while late fusion uses detection outputs for object-level fusion. Middle fusion utilizes intermediate-leve features for feature fusion, striking a balance between preserving valuable information and reducing redundant transmission. However, existing middle-fusion solutions [15] 14 33) overlook the challenge of temporal asynchrony explicitly, leading to fusion misalignment that affects detection results, as depicted in Figure 2 This paper aims address these challenges in a simple and unified manner. Specifically, we propose the Feature Flow Net (FFNet), novel cooperative detection framework that simultaneously overcomes the issues of uncertain temporal asynchrony and communication bandwidth limitations in VIC3D object detection.
As depicted in Figure FFNet comprises several steps, including generating feature flow from sequential infrastructure frames, transmitting the compressed feature flow, and fusing it with ego- vehicle features to obtain detection output. The feature flow critical component of FFNet, serving as a feature prediction function that enables alignment with ego-vehicle features and eliminates fusion errors arising from temporal asynchrony. To reduce transmission costs while preserving valuable information and temporal prediction ability, we employ attention masks and quantization methods to further compress the feature flow before transmission. Furthermore, we introduce a self-supervised approach to train the feature flow generator. This approach involves constructing ground truth features using raw infrastructure sequences, eliminating the need for manual labeling. The feature flow captures rich temporal correlations extracted from the raw infrastructure sequence and exhibits the ability to predict infrastructure features at any future time, making it well-suited for addressing the challenge of uncertain temporal asynchrony in VIC3D object detection. To the best of our knowledge, this is the first time feature flow has been utilized in multi-sensor object detection to address the issue of temporal misalignment in intermediate levels.
We implemented the proposed FFNet framework on the DAIR-VAS V2X dataset 40)) which consists of real-world driving scenarios in challenging traffic intersections. To demonstrate the effectiveness of FFNet, we conducted performance comparisons with several existing cooperative detection methods, including V2VNet (30) and DiscoNet (19). The experimental results reveal that FFNet surpasses all other cooperative methods while utilizing only about 1/100 of the transmission cost required for transmitting raw data. Furthermore, our method effectively addresses the challenge of temporal asynchrony and overcoming latency variations ranging from 100ms to 500ms in one model. Experiments encompassing additional V2V (vchicle-to-vehicl) scenarios will soon be public.

---

The main contributions of this work are as follows:
We propose Feature Flow Net (FFNet), flow-based feature fusion framework for VIC3D object detection, FFNNet transmits feature flow to generate aligned features for data fusion, providing a simple and unified manner to transmit valuable information for fusion while addressing the challenges of uncertain temporal asynchrony and transmission cost.
We introduce a self- self-supervised approach to train the feature flow generator, enabling FFNet with feature prediction ability to mitigate temporal fusion errors across various latencies. This training is independent of cooperative view and labeling, allowing full utilization of infrastructure sequences.
We evaluate the proposed FFNet the DAIR-VA V2X dataset, demonstrating superior performance compared to all cooperative methods while requiring only about 1/100 of the transmission cost of raw data. Furthermore, FFNet is robust across various latencies, requiring only one model.
2 Related Work
Egocentric 3D Object Detection. Perceiving objects, especially 3D obstacles in the road environ- ment, is a fundamental task in egocentric autonomous driving. Egocentric 3D object detection can be classified into three categories based on sensor types: Camera- based methods. LiDAR-baa b-seddd methods, and mulli-senssr-basee
methods. Camera-based methods, such as FCOS3D (29). directly detect 3D bounding boxes from a single image. BEVformer (2I) and B (31, project 2D images onto bird's s-eye view (BEV) to conduct multi-camera joint 3D detection. LiDAR-basee methods, such as VoxelNet (43), SECOND (37), and PointPillars (16), divide the LiDAR point cloud into voxels or pillars and extract features from them. Multi Multi-sensor- sensor--ssrseeee based methods (28 23) utilize both Camera and LiDAR data. In contrast to these methods for single vehicle view object detection, our proposed method focuses on cooperative detection with point clouds as inputs. It utilizes both infrastructure and vehicle sensor data to overcome the perception limitations of single-vehicle view detection.
VIC3D Object Detection. With the development of V2X communication 12), utilizing information from the road environment has attracted much attention. Several works, such as V2VNet (30). DiscoNet (19), StarNet (20) and SyneNet (17), utilize information from other vehicles to expand
the perception field. V2X-Sim (18), OPV2V (34) and V2V4Real (32) are datasets for multi-vehicle cooperative perception research. Controlling (27) and Coopernaut (6) integrate infrastructure
data for end-to-end autonomous driving. Some works like Rope3D (38), BEVHeight (36), and A9-Dataset (5) that focus on utilizing roadside sensor data for 3D object detection. DAIR- V2X (40) cooperative 3D object detection, which introduces the VIC3D object detection task and provides carly and late fusion basclines. Then V2X-Seq (42) extends the tasks into cooperative tracking and motion forecasting. Existing approaches such 24 15 g
a pioneering work in
focus on transmitting feature maps or queries for cooperative detection, without considering the challenges of temporal asynchrony. In this paper, we propose a flow-based feature fusion framework to address the issue of temporal asynchrony and reduce transmission costs in simple and unified manner. is important note that our method is fundamentally different from SyneNet (1),, which transmits common features and integrates per-frame features to compensate for latency.
Feature Flow. Flow is concept originating from mathematics, which formalizes the idea of the motion of points over time ( It has been successfully applied to many computer vision tasks, such as optical flow 33, scene flow (26), and video recognition (45). As a concept extended from optical flow (13), feature flow describes the changing of feature maps over time, and it has been widely used in various video understanding tasks. Zhu et al. (44) propose flow-guided feature aggregation to improve video detection accuracy. In this paper, we introduce the feature flow for feature prediction to overcome the challenge of temporal asynchrony VIC3D object detection.
3 Method
In this section, we present the proposed FFNet (Feature Flow Net) to solve sabicle-infrastructtte cooperative 3D (VIC3D) object detection. We begin by introducing the VIC3D problem in Section 3.1 then explaining the inference process in Section 3.2 and explaining the training methodology of FFNet, including the incorporation of self-supervised learning, Section 3.3 the Appendix, we provide comprchensive comparison of various potential solutions for reference.
3

---

Infrastructure Side at
Po doud Point doul fi-1
Pseuco Image 4 (BPV)
Feature P.RG)))
Feature Flow
Fist-order De F((G)
(B2V)
Compresser
Vehicle Side at
Feature Fusion
Festure (F,(G)
Temporal and Spatial Fosture Aligmment
Pazudoo magic
3D Deteztion kod
3D Ourput
Figure 3: FFNet Overview. In the infrastructure system, we represent the feature flow using linear forms by extracting both the feature and the first-order derivative, shown in Equation 2 To further reduce the transmission cost, employ attention masks and quantization techniques addition to compress feature flow. the vehicle system, we utilize feature flow to generate temporally and spatially aligned features. These aligned features are then fused with the vehicle feature to obtain 3D outputs.
common compressor
3.1 VIC3D Object Detection
Problem Definition. The VIC3D object detection aims to improve the performance of localizing and recognizing the surrounding objects by utilizing both the infrastructure and vehicle sensor data under limited wireless communication conditions. This paper focuses on point clouds captured from LiDAR as inputs. The input of VIC3D consists of two parts:
Point cloud P.(4.) captured by the ego-vehicle sensor with timestamp 4, as well as its relative pose (t,),, where P-(-) denotes the capturing function of ego-vehicle LiDAR.
Point cloud (t,) captured by the infrastructure sensor with timestamp t, as well its relative pose (t,),, where P:(-) denotes the capturing function of infrastructure LiDAR. Previous frames captured by the infrastructure sensor can also be utilized in cooperative detection.
Note that the timestamp t; should be earlier than timestamp t, since receiving the data through long- range communication from infrastructure devices vehicle devices requires a significant amount of transmission time. Moreover, the latency (t, - t)) should be uncertain before receiving the data, as the transmitted data could be obtained by various autonomous driving vehicles in different locations after data broadcasting. The illustration of the uncertain latency is also provided in Figure
Challenges. Compared 3D object detection single-vehicle autonomous driving scenarios, VIC3D object detection encounters additional challenges related to temporal asynchrony and trans- mission cost. Directly fusing infrastructure data can lead to significant fusion errors and negatively impact detection performance due to scene changes and the movement of dynamic objects. This asynchronous behavior is show Figure 2 and evident in the experimental results presented in Section 4.3 Moreover, reducing the amount of transmitted data can effectively decrease the overall latency, as transmission time is directly influenced by the volume of data being transmitted
Evaluation Metrics. We evaluate the 3D object detection performance using mean Average Preci- sion (mAP) with cooperative annotations as the ground truth, as outlined (10). focus on egocentric surroundings, objects outside the designated evaluation area are excluded. For measuring the transmission cost, we adopt the Average Byte (AB) metrie, as suggested (40). The detailed explanations of these two metries and the computation of AB are provided in the Appendix.

---

3.2 Feature Flow Net
As depicted in Figure 3 Feature Flow Net (FFNet) consists of three main modules: (1) generating the feature flow, (2) compressing, transmitting, and decompressing the feature flow, and (3) fusing the feature flow with vehicle feature to generate the detection results.
Feature Flow Generation. We adopt the feature flow as prediction function to describe the infrastructure feature changes over time in the future. Given the current point cloud frame (t(4:) and the infrastructure feature extractor F:(-), the feature flow over the future time f after i; is defined as:
F((t)= = ((),( t 2 t..
(I)
Compared with the previous approaches of transmitting per-frame feature (t.) produced from per frames (30), which lacks temporal and predictive information, feature flow enables the direct prediction of the aligned feature at the timestamp by the vehicle sensor data.
Two issues need to be addressed in order to apply the feature flow to transmission and cooperative detection: expressing and transmitting the continuous feature flow changes over time, and enabling the feature flow with prediction ability. Considering that the time interval f, + t is generally short, we address the expressing issue by using the simplest first-order expansion to represent the continuous feature flow over time, which takes the form of Equation (2)
(t, + At) = (((((tt)) + Ff
(2)
where F (1:) denotes the first-order derivative of the feature flow and At denotes short time period in the future. Thus, we only need to obtain the feature (P,, and the first-order derivative of the feature flow F. (1:) to approximate the feature flow. When an autonomous driving vehicle receives (t.))) and F; (f:) after an uncertain latency, we can generate the infrastructure feature aligned with the vehicle sensor data with minor computation because it only needs linear calculation. To en- able the feature flow with prediction ability, we use network to extract the first-order derivative of the feature flow F. (t+) from the historical infrastructure frames [1 N + 1),"." 1, 1), T, (t,). Generally, the larger N will generate more accurate estimations. In this paper, take N as two and use two consecutive infrastructure frames P. (f: - 1) and (t.).
Specifically, we first use the Pillar Feature Net (16) to convert the two consecutive point clouds into two aseudo-images with bird-eye view (BEV) and with the size of [384, 288, 288). Then, we concatenate the two BEV pssudd-imagen into the size of [768, 288, 288], and input the concatenated pseudo-images into a 13-layer Backbone and a 3-layer FPN (Feature Pyramid Network), as in SECOND (35), to generate the estimated first-order derivative F. (t+) with the size of [364, 288, 288). The detailed network configuration is provided in the Appendix.
Compression, Transmission and Decompression. In order to eliminate redundant information and reduce the transmission cost, we apply two compressors to the feature ((4) and the derivative F. (t+), compressing them from size [384, 288, 288] [12, 36, 36) using three Conv-Bn-ReLU blocks in each compressor. We broadcast the compressed feature flow along with the corresponding timestamp and calibration file on the infrastructure side. Upon receiving the compressed feature flow, the vehicle uses two decompressors, each composed of three Decony-Bn- Baey--Bn-RLLL blocks, to decompress the compressed feature and compressed first-order derivatives to the original size [384, 288, 288).
We incorporate optional attention masks and quantization techniques to further compress the feature flow. Firstly, we use attention mask to identify regions of interest and transmit the complete feature along with only the first-order derivative of the feature flow within these regions. Since the infrastructure sensors have fixed positions, the correspondence between elements in the infrastructure feature and real physical space remains constant. The most significant changes in the feature flow over time occur in regions where dynamic instances moving. To capture these regions, we employ a binary attention mask M that predicts potential dynamic instance locations in the near future. We transmit the feature flow multiplied element-wisg by the attention mask, denoted as M O (t,). where 3 represents the sement-wwis product. Secondly, we apply quantization to both the feature and the first-order derivative, reducing them to b-bit representations using linear quantization approach. The quantization is performed according to the following equation:
Q(x; a) = clamp(æ, a) s(a),
(3)
s(x)
5

---

where clamp(', a:) truncates values to the range [-a, al,, H denotes rounding, and a is the clipping value. We set as the maximum value of the imput tensor, as larger values tend to contain more valuable information (39, 41 [T). We determine s(a) أ By transmitting b-bit numbers instead of the original 32-bit floating-poing point values and transmitting data only within the regions of interest, we achieve more compression in the data transmission process.
Cchicle-Infrastructur Feature Fusion. We use the feature flow to predict the infrastructure feature at timestamp L,, aligned with the vehicle feature, as follows:
(t,t.)) = (,(((tt.)) (t,, - t4) (t,))
This linear prediction operation effectively compensates for uncertain latency and requires minimal computation. The predicted feature (.,...) is then transformed into the vehicle coordinate system using the corresponding calibration files. The bird's-eye view of the infrastructure and vehicle features are obtained, both at the vehicle coordinate system, while preserving spatial alignment. The feature located outside the vehicle's 's interest area is discarded for the infrastructure feature, and empty locations are padded with zero elements.
Subsequently, we concatenate the infrastructure and vehicle feature and employ a Conv-Bn-Redl block to fuse the concatenated features. Finally, input the fused feature into 3D detection head, utilizing the Single Shot Detector (SSD) (22) setup as the 3D object detection head, to generate 3D outputs for more accurate localization and recognition. The experimental results indicate that the infrastructure feature flow significantly enhances the detection ability.
3.3 Training Feature Flow Net
The FFNet training consists of two stages: training a basic fusion framework in an end-to-end way and then using self-supervised learning to train the feature flow generator.
In the first stage, we train a basic fusion framework in an end-to-end manner without considering latency. This stage aims at enabling FFNet fusing the infrastructure feature with the vehicle feature to enhance detection performance. Specifically, we train FFNet using cooperative data and annota- tions obtained from both the vehicle and the infrastructure. The localization regression and object classification loss functions used in SECOND (35) are applied in this stage.
In the second stage, we use self superviie learning to train the feature flow generator by exploiting the temporal correlations in infrastructure sequences, as shown in Figure F The idea construct the ground truth features by using nearby infrastructure frames that do not require any manual annotations. Specifically, we generate training frame pairs D = [dh_,. = (P, - 1), (.(),,, (1 + k))). where P. 1) and (44) are two consecutive infrastructure point cloud frames, and P. (4: + k)) is the (Is + 1)-th frame after (t.).
We construct the loss function to optimize the feature flow generator. The objective to gener- ate the feature flow to predict F, (4, + k) as close possible to (t, k)). We use the cosine similarity to measure the similarity between the predicted feature and the ground truth feature as
-((1-11)
Feature Flow
P2(t,)
Time ti +k
Predicted Feature
Fl + k) @ P((Pa(le + k)) similarity [1, + 12 * (P,, + k)) (5)
Similarity Loss
p1 +k)
Ground Truth
Feature
where @ denotes the inner product, denotes the scalar multiplication, and ||| ||l denotes the L2
Figure Illustration training feature flow generator using supervised learning and similarity lnss. The upper red circle represents the first-order derivarive generator, while the lower purple cireles with solid and dushed lines share the same infrastructure feature extracter.
norm. We use this similarity as the loss function to train the feature flow generator
F.(4, + h) の (.(P.(4 + k))
L(D,0)
Σ
(1-
[[ + k) * [[F,(P.(t, + +))ll2
where / is the parameter of the feature flow generator, and we only update the parameters in first-order derivative generator F. (-) and frozen other parameters.

---

4 Experiments
In this section, we implement FFNet on the DAIR-V2X dataset (40), comparing it with existing cooperative detection methods on different latencies. Our proposed FFNet outperforms all other methods, including early fusion, V2VNet (30), and DiscoNet (19), at 200ms latency, while only requiring about 1/100 of the transmission cost of raw point clouds. We demonstrate how FFNet overcomes the challenge of temporal asynchrony with feature flow prediction. Our results show that temporal asynchrony significantly reduces the performance of the cooperative detection model, but feature flow can effectively compensate for this drop. We evaluate FFNet different latencies and show that it can robustly solve uncertain latency challenges with just one model. Furthermore, we show that self- supervieed training can utilize extra infrastructure sequences. In the Appendix, compare the performance of feature flow extraction on the infrastructure and ego vehicle sides.
4.1 Experiment Settings
Dataset. We used public and real-world DAIR-V2, dataset (40), which comprises over 100 scenes and 18,000 data pairs captured from infrastructure and vehicle sensors (Cameras and LiDARs) at 28 challenging traffic intersections. The dataset includes cooperative 3D annotations with vehicle- infrastructure cooperative view for 9,311 pairs, where each object is labeled with its corresponding category (Car, Bus, Truck, or Van). The dataset is divided into train/val/test sets in 5:2:3 ratio, with all models evaluated on the val set. Additionally, raw sensor data is only released for the test set.
Note that the timestamps of the data from infrastructure and vehicle sensors in each pair are not precisely synchronized. The time difference of each pair in 9,311 pairs, is within the range of [-30, 30]ms. As the actual data cannot be altered after collection, we simulate latency of * 100ms by replacing the first & frames of the infrastructure frame with current infrastructure frame for each pair.
Implementation details. We utilized MMDetection3D ( as our codebase and trained the feature fusion base model on the DAIR-V V2X training set for 40 epochs, with a learning rate of 0.001 and weight decay of 0.01. To form D for training the feature flow generator, we select each pair from training part and randomly set k from the range [1, 2]. More information on l: and DD can be found in S.c The pretraining of FFNet was done using the trained feature fusion base model. We trained the feature flow generator on D, for 10 epochs with learning rate of 0,001 and weight decay of 0.01. All training and evaluation were performed on an NVIDIA GeForce RTX 3090 GPU. The detection performance was measured using KITTI (0) evaluation detection metrics, which include bird-eye view (BEV) mAP and 3D mAP with 0.5 IoU and 0.7 loU, respectively. Only the Car class was taken into account for objects located in the rectangular area [0. -39.12, 100, 39.12) More implementation details regarding FFNet and the fusion methods are provided in the Appendix.
4.2 Comparison to Different Fusion and State of the Art Methods
We compare FFNet with four categories of fusion methods: non-fusion (e.g., PointPillars (6) and AutoAlign 44), early fusion, late fusion, middle fusion (e.g., DiscoNet (19). and V2VNet (18).
Result Analysis. Table presents summary of our experimental results. The table is divided into three parts: the top section displays the evaluation results for non-fusion methods, the middle section shows the results for 200ms latency, and the bottom section presents the results for 300ms latency. Our proposed FFNet achieves new SOTA on DAIR-V21 V2X. Notably, FFNet-C1 surpasses early fusion while it only requires about 1/100 of the transmission cost. Firstly, our proposed FFNet outperforms the non-fusion method PointPillars by 9.32% mAP@ BEV (IoU=0.5) and 7.32% mAP@ BEV (loU=0.5) in 200ms and 300ms latency, respectively. This result indicates that utilizing infrastructure data can improve 3D detection performance. Secondly, although late fusion requires little transmission cost, the mAP@ BEV (loU=0.5) of late fusion is much lower than that of FFNet, up to 5.10% in 200ms latency. Thirdly, compared with early fusion methods, FFNet achieves similar detection performance in 200ms latency and outperforms 2.92% mAP in 300ms latency, while only requires no more than 1/10 of the transmission cost. Moreover, FFNer-C1 outperforms early fusion more than 2% mAP in 300ms latency while only requiring 1/100 of the transmission cost. Fourthly, our FFNet achieves the best detection performance with the exact transmission cost the
7

---

Table 1: Comparison Different Fusion Methods. FFNett significantly outperforms all other fusion methods.
mAP@3D)
mAP@ BEVT
Model
FusionType
Latency (ms)
AB (Byte) 4
1oU=0.5
ToU=0.7
ToU=0.5
100=0.7
PointPillars (16
non-fusion
48.06
52.24
AutoAlignV2 4
non-fusion
50.32
53.88
Early Fusion
early
$4.63
38.23
61.08
50.06
1.4x 10
Late Fusion
late
52.43
36.54
49.25
5.1x10"
DiscoNet (99
middle
200
50.76
28.57
58.20
48.90
1.2x 10
V2VNet 330
middle
200
49.67
26.96
56.02
46.32
1.2x10
FFNet (Ours)
middle
200
55.37
31,66
63,20 (+9.32)
54.69
102x10
FFNet-C1 (Ours)
middle
200
55.17
31,20
62.87 (+8.99)
54.28
1.7x10
Early Fusion
early
300
51.37
37.25
58.28
49.81
14x10"
Late Fusion
late
300
51.35
36,24
56.89
48.79
5,1x102
DiscoNet (19
middle
300
49.03
27.39
$5.81
47.28
1.2x 10
V2VNet [30
middle
300
48.51
27.00
55.81
46.32
1.2x 10
FFNet (Ours)
middle
300
53.46
30.42
61.20 (+7.32)
52.44
1.2x 10
FFNet-C1 (Ours)
middle
300
54.10
29.887
60.76 (+6.88)
53.28
1.7x10
Table 2: Comparison between with and without Feature Prediction. Compared with no prediction models,
FFNet with feature prediction has significantly lower performance drop when there communication latency.
mAP@
mAP@ BEV
Model
Latency (ms)
AB (Byte) +
0.5
loU=0.7
loU=0.5
loU=0.7
FFNet
55.81
30.23
63.54
54.16
1.2x 10
FFNet (without prediction)
55.81
30.23
63.54
54.16
6,2x10*
FFNet- (without prediction)
55.78
30.22
64.23
55.00
1.2x10
FFNet
55.37
31.66
63.20 (-0.34)
54.69
1,2x10'
FFNet (without prediction)
200
50.27
27.57
57.93 (-5.61)
48.16
6,2x10*
FFNet-V2 V2 (without prediction)
200
49.90
27.33
58.00 (-6.23)
48.22
1.2x 10
middle fusion methods. For example, FFNet surpasses DiscoNet by 5.0% mAP@BEY (IoU=0.5) and 5.39% mAP@BEV (IoU=0.5) in 200ms and 300ms latency, respectively.
4.3 Ablation Study
We conducted a series of experiments to demonstrate the effectiveness of the feature flow module in overcoming the temporal asynchrony challenge and to show that FFNet performs robustly under vari- ous latencies. Additionally, we studied how self-supervised learning can fully exploit infrastructure sequences that are independent of cooperative view and labeling.
Feature prediction can well solve temporal asynchrony. We conducted series of experiments evaluate the effectiveness of the feature flow module in overcoming the temporal asynchrony challenge. We evaluated FFNet under two different latency conditions: Oms and 200ms, where Oms indicates temporal asynchrony between infrastructure data and vehicle data within [-30, 30]ms. To investigate the impact of temporal asynchrony on FFNet's performance, we also removed the prediction module from FFNet and directly fused the infrastructure feature. We refer this version as FFNet (without prediction), abbreviated as FFNet-O, and evaluated it under both Omns and 200ms latency. Since FFNet-O does not require the transmission of the first-order derivative the feature flow, it only requires half the transmission cost of FFNet. To ensure fair comparison, we trained another version of FFNet called FFNet-V2, which compressed the feature flow from (384, 288, 288) to (384/16, 288/8, 288/8). FFNet-V2-O has the same transmission cost as FFNet, and we evaluated under both Omn and 200ms latency as well.
The evaluation results, presented in Table 2 demonstrate that FFNet-O and FFNet- V2-O exhibit significant performance drop under 200ms latency. For example, FFNet-O experiences 5.61% mAP@ BEV (IoU=0.5) drop in 200ms latency compared to Oms latency. Although "FNet-V2-0 performs slightly better than FFNet and FFNet-O in Orns latency, FFNet significantly outperforms FFNet-V2-O in 200ms latency. These results show that temporal asynchrony can significantly impact performance when we directly fuse the infrastructure feature, and that our feature prediction module can effectively compensate for the performance drop caused by temporal asynchrony.

---

FFNet-O
64
FFNet-O
FFNet
BEV mAP (1oU==0.5)
FFNet-V2-O
FFNet
mAP (1oU-0.5)
62
FFNet-V3
FFNet-V4
60
60
58
58
56
54
54
200
400
100
200
300
400
500
Average Latency Time (ms)
Average Latency Time (ms)
Figure 5: Ablation study of FFNet robustness.
Figure 6: Ablation study of FFNet training.
FFNet is robust to uncertain latency. We conducted additional experiments assess the perfor- mance of FFNet, FFNet-O, and PFNet-V2-O under varying latency cases, ranging from 100ms to 500ms. The experiment results are presented Figure As depicted in the figure, both FFNet-O and FFNet-V2-O exhibit continuous performance degradation the latency increases from 100ms to 500ms. Specifically, 500ms latency, FFNet-O and FFNet- V2-O show a significant 9.38% mAP@BEV (loU=0.5) drop and 9.76% mAP BEV (IoU=0.5) drop, respectively. Conversely, FFNet demonstrates minimal performance degradation within 200ms latency and only experiences a 4.39% mAP@ BEV (loU=0.5) drop. These results suggest that FFNet is resilient varying latencies and can effectively handle uncertain latency in VIC3D problem. The ability of our feature flow to make predictions at arbitrary future time before transmission is crucial since it could be received by different vehicles with different latencies.
FFNet training can fully utilize infrastructure sequences. We additionally trained the feature flow generator using the extra test portion of the DAIR-V2X dataset. For each frame in the test part, randomly set k from the range [1, 2] to form Dtest- We first pretrained FFNet with the trained feature fusion base model and trained the feature flow generator solely with Dtest without D. We refer this trained FFNet as FFNet- V3. Subsequently, we pretrained FFNet with the trained feature fusion base model and trained the feature flow generator using Drest U D. We denote this trained FFNet as V4. We evaluated V3 and FFNet- under latency from 100ms to 500ms, In Figure 5) we present the mAP@ BEV (IoU=0.5) results. FFNet-VS V3 demonstrates significantly better performance than FFNet-O, indicating that the training of the feature flow generator can be independent of cooperative-view data. FFNet- V4 performs slightly better than FFNet, suggesting that incorporating more infrastructure sequences enhances the feature flow prediction ability.
Conclusion
This paper introduces FFNet, an innovative intermediate-level cooperative framework designed for VIC3D object detection. FFNet effectively addresses challenges related to temporal asynchrony and transmission cost by utilizing compressed feature flow for cooperative detection. Through exten- sive experiments conducted on the DAIR-V2X dataset, FFNet demonstrates superior performance compared to existing state-of-the-re art methods. Furthermore, FFNet be extended various modalities, including image and mudti-modality data, making it a versatile solution. Moreover, FFNet holds promise in the domain of multi-vehicle cooperative perception and leverages the utilization of additional frames to enhance feature prediction capabilities. The proposed FFNet framework, incorporating feature prediction and self-supervised learning, presents a promising avenue for VIC3D object detection and holds potential for addressing diverse cooperative perception tasks in the future.
Acknowledgements
This paper is partially supported by the National Key R&D Program of China No. N0.2002ZD0161000 and the General Research Fund of Hong Kong No.17200622 This work was also supported by Baidu Inc. through the Apollo-AIR Joint Research Center.

---

References
[1] MMDetection33: OpenMMLab next-generation platform for general 3D object detection, 2020.
[2] Eduardo Amold, Mchrdad Dianati, Robert de Temple, and Saber Fallah. Cooperative perception for 3d object detection in driving scenarios using infrastructuree sensors. IEEE Transactions on Intelligent Transportation Systems, 2020.
[3] Steven S. Beauchemin and John L. Barron. The computation of optical flow. ACM computing surveys (CSUR), 27(3))433--46, 1995.
[4] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. European conference on computer vision, 2022.
[5] Christian BreB, Walter Zimmer, Leah Strand, Maximilian Fortkord, Siyi Dai, Venkatnarayanan Lakshmi- narasimhan, and Alois Knoll. A9-dataset: Multi- sensor infrastructure based dataset for mobility research.
In 2022 IEFE Intelligent Vehicles Symposium (IV), pages 965-970. IEFF, 2022.
[6] Jiaxun Cui, Hang Qiu, Dian Chen, Peter Stone, and Yuke Zhu. Coopernaut: End-to-end driving with cooperative penception for networked vehicles. In Proceedings the IEEE/CVF Conference on Computer Vision and Pottern Recognition. pages 17252-17262, 2022.
[7] Michel Deville and Thomas B Gatski. Mathematical modeling for complex fluids and flows. Springer Science & Business Media, 2012.
[8] Siqi Fan, Haibao Yu. Wenxian Yang. Jirui Yuan, and Zaiqing Nie. Quest: Query stream for vehicle- infrastructure cooperative perception. arXiv preprint arXiv:2308.00 230.000444 2023.
[9] Robert M Fano. Transmission information: A statistical theory of communications. American Journal of Physics, 29(1 1)7933-94, 1961.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 335--361.. IEFE 2012.
[11] Song Han. Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint wrXivv150000114: 2015.
[12] Laurens Hobert, Andreas Festag, Ignacio Llatser, Luciano Altomare, Filippo Visintainer, and Andras v2x communication support of cooperative autonomous driving. IEEE
Kovacs. Enhancements communications magazine, 53(12):64-70, 2015.
[13] Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3):185 203, 1981.
[14] Yue Hu, Shaoheng Fang, Zixing Lei, Yiqi Zhong, and Siheng Chen. Where2comm: Communication- efficient collaborative perception via spatial confidence maps. arXiv preprint (xXiv2209.12836 2022.
[15] Yue Hu, Yifan Lu, Runsheng Xu, Weidi Xie, Siheng Chen, and Yanfeng Wang. Collaboration helps camera overtake lidar in 3d detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[16] Alex H Lang. Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars:
Fast encoders for object detection from point clouds, 2019.
[17] Zixing Lei, Shunli Ren, Yue Hu, Wenjun Zhang, and Siheng Chen. Latency-aware collaborative perception.
In European Conference on Computer Vision. Springer, 2022.
[18] Yiming Li, Dekun Ma, Ziyan An, Zixun Wang. Yiqi Zhong, Siheng Chen, and Chen Feng. V2x-sim: agent collaborative perception dataset and benehmark for autonomous driving. IEEE Robotics and Automation Letters, 7(4):10914 10,21,, 2022.
[19] Viming Li, Shunli Ren, Pengxiang Wu, Siheng Chen, Chen Feng, and Wenjun Zhang. Learning distilled collaboration graph for multi-agent perception. Advances in Neural Information Procesingg Systems, 34,
2021.
[20] Yiming Li, Juexiao Zhang, Dekun Ma, Yue Wang, and Chen Feng. Multi-robot seene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning, pages 2062 2072. PMLR, 2023.
[21] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird's- s-eye-view representation from multi-camera images via spatiotemporal trans- formers. arXiv preprint orXiv:2203.177 1727700..77...... 2022.
[22] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yan Fu, and Alexander Berg. Ssd: Single shot multibox detector. Computer Vision-ECCV 2016: 14th European Conference. Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 21-37.
Springer, 2016
[23] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task: multi-sensor fusion with unified bird's-eye view representation. arXiv preprint arXiv:2205. 13552, 2022.
[24] Changxi Ma. Wei Hao, Aobo Wang, and Hongxing Zhao. Developing coordinated signal control system for urban ring road under the vehicle-infrastructuuee connected environment. leee Access, 6:52471-52477
2018.
[25] Yongchang Ma, Mashrur Chowdhury, Adel Sadck, and Mansourch Jeihani. Real-time highway traffic condition assessment framework using veticll-inffaattructuee integration (vii) with artificial intelligence
(ai). IEEE Trunsactions on Intelligent Transportation Systems, 10(4):615-627, 2009.
[26] Moritz Menze and Andreas Geiger. Object scene flow for autonomous vehicles. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3061-3070, 2015.
10

---

[27] Rodolfo Valiente, Mahdi Zaman, Sedat Ozer, and Yaser P Fallah. Controlling steering angle for cooperative self-driving vehicles utilizing cun and lstm-based deep networks. In 2019 IEEE intelligent vehicles symposium (IV), pages 2423-2428. IEEE, 2019.
[28] Sourabh Vora, Alex Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 4604-4612, 2020.
[29] Tai Wang, Xinge Zhu, Jiangmiao Pang. and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
913-922, 2021.
[30] Tsun-Hsuan Wang. Sivabalan Manivasagam, Ming Liang, Bin Yang. Wenyuan Zeng, and Raquel Urtasun. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. In European Conference on Computer Vision, pages 605 621. Springer. 2020.
[31] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. M 2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view
representation. arXiv preprint 22Xi::200000008 2022.
[32] Runsheng Xu, Xin Xia, Jinlong Li, Hanzhao Li, Shuo Zhang. Zhengzhong Tu, Zonglin Meng. Hao Xiang, Xiaoyu Dong. Rui Song, et al. V2v4real: A real-world large-scale dataset for vehicle-to-vehice cooperative perception. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13712-13722, 2023.
[33] Runsheng Xu, Hao Xiang, Zhengzhong Tu, Xin Xia, Ming-Hsuan Yang. and Jiaqi Ma. V2x-vit: Vehicle- to-everything cooperative perception with vision transformer. arXiv preprint MrXiv:2203.10633 2022. [34] Runsheng Xu, Hao Xiang, Xin Xia, Xu Han, Jinlong Liu, and Jiaqi Ma. Opv2v: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vchicle communication. ICRA, 2022.
[35] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors,
18(10)3337,, 2018.
[36] Lei Yang. Kaicheng Yu, Tao Tang, Jun Li, Kun Yuan, Wang. Xinyu Zhang, and Peng Chen. Bevheight: rohust framework for vision-based roadside 3d object detection. Proceedings of the JEEF/CVF Conference Computer Vision and Pattern Recognition, 2023.
[37] Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object dctector for point cloud. In Proceedings of the IEEE/CVF international conference computer vision, pages
1951-1960, 2019.
[38] Xiaoqing Ye, Mao Shu, Hanyu Li, Yifeng Shi, Yingying Li, Guangjie Wang. Xiao Tan, and Errui Ding. Rope3d: The roadside perception dataset for autonomous driving and monocular 3d object detection task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21341-21350, 2022.
[39] Haibao Yu, Qi Han, Jianbo Li, Jianping Shi, Guangliang Cheng, and Bin Fan. Scarch what you want: Barrier panelty nas for mixed precision guantization. In Computer Vision-ECCV 2020: 16th European Conference. Glasgow, UK. August 23-28, 2020, Proceedings. Part IX 16, pages 1-16. Springer, 2020.
[40] Haibao Yu, Yizhen Luo, Mao Shu, Yiyi Huo, Zebang Yang. Yifeng Shi, Zhenglong Guo, Hanyu Li, Xing Hu, Jirui Yuan, and Zaiqing Nie. Dair-v2x: large-scale dataset for vehicle-infrastructure cooperative 3d object detection. In JEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), June 2022. [41] Haibao Yu, Tuopu Wen, Guangliang Cheng. Jiankai Sun. Qi Han, and Jianping Shi. Low-bit quantization needs good distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 680-681, 2020.
[42] Haibao Yu, Wenxian Yang, Hongzhi Ruan, Zhenwei Yang, Yingjuan Tang, Xu Gao, Xin Hao, Yifeng Shi, Yifeng Pan, Ning Sun, Juan Song, Jirui Yuan, Ping Luo, and Zaiqing Nie. V2x-seq: A large-scale sequential dataset for rehicle-infrastructuee cooperative perception and forecasting. In Proceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.
[43] Yin Zhou and Oncel Tuzel. Voxelnet: End-ta-end learning for point cloud based 3d object detection. In Proceedings of the IEFF conference on computer vision and pattern recognition, pages 4490-4499, 2018.
[44] Xizhou Zhu, Yujie Wang, Jifeng Dai, Lu Yuan, and Yichen Wei. Flow-guided feature aggregation for video object detection. In Proceedings of the IEEE international conference on computer vision, pages 407-417, 2017.
[45] Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition.
In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2349-2358.
2017.
11


Reviews:
[{"review_id": "V4JwpCBiIu", "reviewer": "Reviewer_CL6E", "summary": "In this paper, a cooperative detection framework, named Feature Flow Net (FFNet), is presented to address temporal asynchrony and limited communication condition challenges in vehicle-infrastructure cooperative.  \nSpecifically, FFNet transmits feature flow to generate aligned features for data fusion, providing a unified manner to transmit valuable information for fusion while addressing the challenges of uncertain temporal asynchrony and transmission cost.\nIncorporating with self-supervised learning, the proposed FFNet framework presents a performacne improvement for VIC3D object detection with less communication cost.", "questions": "1. Experiments and visualizations are expected to analyze and verify why FFNet is deficient in high-quality detection boxes.\n2. Can a finer supervision method, not limited to the global similarity, improve the performance of FFNet on map@3D?", "limitations": "See Weakness.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. FFNet transmits feature flow to generate aligned features for data fusion. This idea is simple yet can transmit temporal asynchrony information for fusion with less transmission cost.\n2. The proposed self-supervised training approach enables FFNet with feature prediction ability to mitigate temporal fusion errors across various latencies without cooperative view and labeling.\n3. FFNet demonstrates superior performance compared to other cooperative methods and demonstrates robustness in terms of latency.", "weaknesses": "1.  For 3D object detection, map@3D proves the detector accuracy better than map@BEV. However, it can be seen from Table 1 that FFNet's performance advantage in map@3D is much smaller than that of map@BEV, and the advantage on IoU@0.7 is less than IoU@0,5. This may indicate  \n\n     a) The proposed method does not learn the changes in the height of the environment, resulting in poor accuracy in high dimensions of FFNet.  \n\n     b) The flow-based prediction strategy is not accurate enough, which limits the upper quality of FFNet detection results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Resolving Anomalies in Evaluation Results using the mAP@3D Metric**\n\nDear Reviewer CL6E:  \nThank you for your insightful question and for bringing to our attention the anomalous results in the mAP@3D metric.\n\n**Phenomenon.** We acknowledge the existence of abnormal performance in mAP@3D. As evident in Table 1, \"FFNet's performance advantage in mAP@3D is much smaller than that of mAP@BEV.\" Moreover, both FFNet and other middle fusion methods like V2VNet[1] and DiscoNet[2] achieve significantly lower mAP@3D (IoU=0.7) compared to early fusion and late fusion methods.\n\n**Explanation.** The aforementioned issues can be attributed to the assumption of a strictly parallel ground in the implementation of feature fusion methods. Specifically, in the conversion of infrastructure Bird's Eye View (BEV)  feature/feature flow into consistent local coordinate systems, we assumed that the x-y planes would remain parallel to the ground. Unfortunately, as the DAIR-V2X dataset[4] originates from  real-world capture, the driving area does not adhere to strict parallelism. Consequently, an unintended rotation component surfaces in high dimensions when transitioning from the infrastructure's local coordinate system to that of the vehicle. Regrettably, we overlooked this high component, resulting in reduced accuracy in the high dimension. On the contrary, we used real transformation matrices in the implementation of early fusion and late fusion, mitigating the impact on the high dimension and achieving much better mAP@3D (IoU=0.7) results than middle fusion methods.\n\n**Solution**. To rectify this issue, we conducted additional experiments by unifying the bottom of all detection boxes and ground truth boxes to the same height. This effectively eliminates the influence of the high component, and we re-evaluated the detection results of FFNet. The table below shows a significant improvement in mAP@3D performance for FFNet.\n\n| High Standardization | Latency (ms) | mAP@3D |   |    mAP@BEV | |\n| --- | --- | --- | --- | --- |--- |\n| |   | IoU=0.5 | IoU=0.7 | IoU=0.5 | IoU=0.7 |\n| No | 200 | 55.37 | 31.66 | 63.20 | 54.69 |\n| Yes | 200 | **62.48 (+7.11)** | **47.92 (+16.26)** | 63.20 | 54.69 |\n| No | 300 | 53.46 | 30.42 | 61.20 | 52.44 |\n| Yes | 300 | **60.39 (+6.93)** | **45.82 (+15.40)** | 61.20 | 52.44 |\n|||||||\n\nTable 2. Evaluation results with high standardization on DAIR-V2X dataset[4].\n\n**More Discussion:** mAP@BEV is widely recognized as a more relevant metric in the autonomous driving context. Since the driving scene typically involves no objects above traffic participants, focusing on the BEV (Bird's Eye View) dimension becomes critical for evaluation. Notably, related approaches like V2VNet [1], DiscoNet [2], and V2X-ViT [3] exclusively report evaluation results that prioritize the BEV dimension. This alignment in evaluation underscores the industry's consensus on the significance of BEV-based metrics for assessing autonomous driving systems.\n\n[1] Wang et al. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. ECCV2020   \n[2] Li et al. Learning distilled collaboration graph for multi-agent perception. NuerIPs2021  \n[3] Xu et al. V2x-vit: Vehicle-to-everything cooperative perception with vision transformer. ECCV2022  \n[4] Yu et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. CVPR2022  \n\n\n"}, {"review_id": "e1r9hBTbsN", "reviewer": "Reviewer_NKqy", "summary": "This paper introduces FFNet, a flow-based feature fusion framework that incorporates a feature flow prediction module to predict future features and addresses asynchrony issues. Instead of transmitting feature maps extracted from static images, FFNet transmits feature flow by leveraging the temporal coherence of sequential infrastructure frames. To evaluate the effectiveness of FFNet, experiments are conducted on the DAIR-V2X dataset.", "questions": "Considering the major concern about the limited scope and application, the authors may consider submitting this work to a more appropriate venue.", "limitations": "N.A.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and the idea is simple yet effective.\n\n2. The overall results look good.", "weaknesses": "1. The major limitation of this work pertains to its narrow scope, focusing exclusively on vehicle-infrastructure cooperative 3D object detection and mainly addressing the latency issue. While the proposed idea demonstrates effectiveness within this specific application, its applicability to broader contexts (such as multiple vehicle scenarios) remains uncertain. \n\n2. Another weakness lies in the experimental section. The comparison made with V2VNet and DiscoNet seems unfair since these models were not originally designed to address communication latency. To ensure a more comprehensive evaluation, it would be beneficial for the authors to incorporate additional modules into these methods or compare their approach to more advanced models, such as SyncNet (ECCV 2022). This would provide a more accurate assessment of the proposed method's performance in comparison to state-of-the-art techniques.\n\n3. A minor weakness is that the literature review is not sufficient. The authors should include more closely-related works such as [1-5]\n\n\n[1] Xu, R., Xia, X., Li, J., Li, H., Zhang, S., Tu, Z., Meng, Z., Xiang, H., Dong, X., Song, R. and Yu, H., 2023. V2v4real: A real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 13712-13722).\n\n[2] Li, Y., Zhang, J., Ma, D., Wang, Y. and Feng, C., 2023, March. Multi-robot scene completion: Towards task-agnostic collaborative perception. In Conference on Robot Learning (pp. 2062-2072). PMLR.\n\n[3] Xu, R., Tu, Z., Xiang, H., Shao, W., Zhou, B. and Ma, J., 2023, March. CoBEVT: Cooperative Bird’s Eye View Semantic Segmentation with Sparse Transformers. In Conference on Robot Learning (pp. 989-1000). PMLR.\n\n[4] Li, J., Xu, R., Liu, X., Ma, J., Chi, Z., Ma, J. and Yu, H., 2023. Learning for vehicle-to-vehicle cooperative perception under lossy communication. IEEE Transactions on Intelligent Vehicles.\n\n[5] Su, S., Li, Y., He, S., Han, S., Feng, C., Ding, C. and Miao, F., 2023. Uncertainty quantification of collaborative detection for self-driving. ICRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer NKqy,\n\nThank you for providing valuable feedback on our work. We will address each of the limitations you have pointed out in your comments.\n\n**W1.** Regarding the potential limitations of our work concerning application scenarios, **we have extended our  experiments to encompass the OPV2V dataset[1], which exclusively  focuses on cooperative 3D detection in multiple-vehicle scenarios.** Our forthcoming version will also include experiments conducted on the V2X-Sim[2] dataset.  Notably, the comment at the top offers a concise summary of our experiment results. These results, spanning the DAIR-V2X[3] and OPV2V[1]  datasets, show the impressive efficacy of our proposed approach across diverse V2X scenarios. \n\nIn addition, we not only pay attention to the latency challenge, but also the transmission cost challenge. Specifically, we propose the Feature Flow Net (FFNet), a unified framework to overcome the hurdles posed by  uncertain temporal asynchrony and communication bandwidth limitations in cooperative 3D object detection.\n\nOverall, we kindly request that you re-evaluate the scope of our work in light of the applications we have focused on and the issues we have successfully addressed.\n\n**W2.** Addressing your concern in Weakness 2, we recognize the significance of  comparing FFNet with existing latency-aware techniques like SyncNet[4]. We wish to clarify that **we have indeed analyzed the distinctions between FFNet and SyncNet[4] in our paper (see lines 109-112).  Moreover, we have conducted comparative experiments and provide comprehensive analysis in the supplementary material** (specifically, sections F \"Comparison of Feature Flow Extraction on  Different Sides\" and G \"Relationship to Other Existing Possible Solutions\" in \"S1-Appendix.pdf\").\n\nIt is worth noting that FFNet and SyncNet[4] approach the cooperative problem from fundamentally different angles. While SyncNet[4] focuses on leveraging historical features for latency compensation, FFNet takes a  more comprehensive approach by integrating transmission and reception considerations. **In our experiments (as outlined in section F of the supplementary material), we incorporate SyncNet's compensation module for VIC3D detection**, that is utilizing historical features for feature prediction on vehicle side. The results, as showcased in Table 2 of the appendix, illustrate FFNet's superiority in overcoming latency through feature flow extraction on infrastructure side. We also list part results in Table 3 below. This outcome aligns with the inherent challenges of extracting temporal  information from compressed features, which lack the richness found in raw sequential point clouds.\n\n| Model | Latency (ms) | mAP@3D |   |    mAP@BEV | |\n| --- | --- | --- | --- | --- |--- |\n| |   | IoU=0.5 | IoU=0.7 | IoU=0.5 | IoU=0.7 |\n| SyncNet[4]  | 100   |    50.5    |  28.25   | 58.02                   | 50.03 |\n| FFNet   | 100        |       53.46   |  30.42   | **61.20 (+3.18)** | 52.44 |\n|||||\n\nTable 3. Comparison with SyncNet[4]. Refer to appendix for more detailed experimental results.\n\nFurthermore, FFNet optimizes vehicle computing resources, as feature  flow generation transpires on infrastructure devices rather than within vehicles. This presents FFNet as a more computing-efficient solution  for resource-constrained vehicle devices. In contrast, SyncNet[4] demands  increased computational resources to leverage historical per-frame features. Additionally, SyncNet[4] necessitates heightened storage due to its dependence on past received frames. SyncNet's vulnerability to frame drops impacts its execution and performance, a concern mitigated by FFNet's storage-friendliness and robustness.\n\n**W3.** Thanks to your suggestion, we will consider adding more related papers in the next version.\n\nBest regards,  \n7569 Authors  \n\n[1] Xu et al. OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. ICRA2022  \n[2] Li et al. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. RA-L 2022  \n[3] Yu et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. CVPR2022  \n[4] Lei et al. Latency-aware collaborative perception. ECCV2022  "}, {"review_id": "RyLTMc6bXF", "reviewer": "Reviewer_enaU", "summary": "This work proposes a flow-based feature fusion framework called Feature Flow Net (FFNet) for vehicle-infrastructure cooperative 3D object detection (VIC3D).\nFFNet can generate aligned features for data fusion to transmit information for fusion and solve uncertain temporal asynchrony and transmission costs.\nThe proposed self-supervised training of the feature flow generator lead FFNet to mitigate temporal fusion errors across various latencies.\nThe results on the DAIR-V2X dataset show superior performance compared to other cooperative methods while consuming only about 1/100 of the transmission cost of raw data and using a single model.", "questions": "- Although the latency and the bandwidth problem is indeed a real-world challenge, the Vehicle-Infrastructure Cooperative looks a little unrealistic. In that case, the autonomous driving vehicle will only be able to use such kind of extra information when there is an infrastructure sensor. Wouldn't it?", "limitations": "No.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The writing is clear and easy to follow.\n- Using predicted features to solve the latency issue is neat and effective.\n- The whole pipeline is simple but well-designed. It covers the transmission speed and bandwidth while keeping the model performance in the meanwhile.\n- The results are solid and expressive. For different latency scenarios, FFNet achieves the best performance with less transmission data.", "weaknesses": "- The abbreviation of models is not easy to distinguish. For instance, FFNet versus FFNet-V2 versus FFNet-O, etc.\n- The ablation of compression is missed. Without proposed compression and decompression, will it affect the tolerance of the latency of transmission?\n- The last paragraph of the ablation study is not clear. The definition of the test part of the Dataset is unclear. If the models are exposed to the dataset test set, then it will be better than the one that doesn't (FFNet-O) for sure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer enaU,  \nThank you for your valuable feedback on our work. We have carefully considered your suggestions and would like to respond to each of your main comments regarding our weaknesses and questions.\n\n\n**W1.** We sincerely acknowledge your suggestion, and in our upcoming version, we will enhance the clarity of  abbreviations by incorporating a dedicated lookup table in the appendix.\n\n**W2.** Your concern about the impact of compression on latency tolerance is valid. The real-time dynamics of the road scene underscore the importance of minimizing latency for accurate predictions. As exemplified in Table 5 in our paper,  increasing latency results in a discernible accuracy loss, even with our latency compensation. Moreover, uncompressed  transmission can escalate both bandwidth consumption and latency challenges. **To illustrate, transmitting and downloading original uncompressed feature flow at 10HZ would demand a substantial 200Mb/s communication bandwidth.** This could severely impede communication responsiveness and stability, particularly in intricate traffic and communication environments. Conversely, our compression module substantially mitigates bandwidth consumption while preserving overall accuracy. The compression and decompression modules within FFNet are indeed pivotal components of our approach.\n\n| Model                         | Latency (ms) | mAP@3D |  | mAP@BEV| | Transmission cost (Bytes) |\n|-------------------------------|--------------|--------|--|-------------|-------|----|\n|                                    |                       |IoU=0.5 |IoU=0.7|      IoU=0.5 |IoU=0.7|                      |\n| FFNet                         | 200          | 55.37  | 31.66 | 63.20  | 54.69                |      1.2× 10^5    |\n| FFNet-C1                      | 200           |  55.17 |31.20 | 62.87  | 54.28                    |        **1.7×10^4 (~1/10^4)**     |\n| FFNet (without any compression) | 200           |  55.44 |31.89 | 63.97  | 55.90                    |    2.5×10^8      |\n||||||  |   |\n\nTable 4. Evaluation results of FFNet with different compression on DAIR-V2X[1] dataset.\n\n**W3.** We wish to clarify that there are no data breach concerns. FFNet's training, including the feature flow prediction modules, is exclusively based on the training part of the DAIR-V2X[1] dataset.\n\n**Q1. Vehicle-infrastructure cooperative autonomous driving is a highly promising and dynamic field of research.** On one hand, the strategic deployment of sensors to forge intelligent transportation systems has gained attention from a diverse array of nations and research institutions, including Germany[4], China[5], and the United States[6]. Additionaly, the infrastructure sensors, including cameras, is also a common sight on city roads, highways, and various traffic networks. On the other hand, despite of achieving great progress recently, autonomous driving still faces great safety challenges due to a lack of global perspective and the limitation of long-range perception capability.  A promising solution to address these challenges is to leverage infrastructure information via Vehicle-to-Everything (V2X) communication, which has been shown to significantly expand perception range and enhance autonomous driving safety[1, 2]. Overcoming the issues in vehicle-infrastructure cooperation, thereby enabling the seamless integration of roadside information to autonomous driving vehicles, constitutes a captivating realm of research.\n\n**Furthermore, our work can be extended to more diverse V2X scenarios.** Our experiment results on the DAIR-V2X[1] and OPV2V[3] datasets confirm FFNet's efficacy across different datasets and its remarkable effectiveness in a wide range of V2X applications, including vehicle-infrastructure interactions and complex multiple vehicle scenarios. In essence, our approach facilitates autonomous driving vehicles using extra information from infrastructure or other vehicle sensors through V2X communication. \n\n\nBest regards,  \n7569 Authors  \n\n\n[1] Yu et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. CVPR2022  \n[2] Eduardo et al. Cooperative perception for 3d object detection in driving scenarios using infrastructure sensors. TITS2020  \n[3] Xu et al. OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. ICRA2022  \n[4] https://innovation-mobility.com/en/project-providentia/  \n[5] https://thudair.baai.ac.cn/index  \n[6] https://www.transportation.gov/tags/v2i  \n\n"}, {"review_id": "v7Ger3xLUm", "reviewer": "Reviewer_Wmh8", "summary": "The authors explore and analyze the existing vehicle-infrastructure cooperative 3D object detection framework, and propose to adopt flow-based rather than still images to extract temporal features from multiple LiDAR input frames. The experimental results on DAIR-V2X dataset is better compared to the existing methods in terms of car objects.", "questions": "Please refer to the questions that I describe in the Weakness part. I would also consider the rebuttal and other reviews.", "limitations": "NA", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The task of vehicle-infrastructure cooperative 3D object detection is very important in the 3D community and has real-world practical applications. Interestingly, the authors propose to extract temporal features for different input frames. Also, they propose a self-supervised method to extract the flow-based features.", "weaknesses": "1. The evaluation of one single class on a single dataset is not strong enough to prove the effectiveness of the proposed method. It would be convincing if the authors could conduct the experiments on other labeled classes, i.e., bus, truck, or van. Also, experiments on another existing dataset would be better as well. \n\n2. Memory footprint comparison. The authors didn't make a comparison of their work in terms of memory with the existing vehicle-infrastructure cooperative 3D detection methods. As far as I am concerned, the memory consumption/additional efforts are needed since the compression and transmission are included.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer Wmh8,  \n\nThank you for providing valuable feedback on our work. We will address each of the limitations you have pointed out in your comments.\n\n\n**W1. We have conducted experiments on more datasets like OPV2V[1],** which also focuses on cooperative 3D detection. Our forthcoming version will also include experiments conducted on the V2X-Sim[2] dataset.  Notably, the comment at the top offers a concise summary of our experiment results. These results, spanning the DAIR-V2X[3] and OPV2V[1] datasets, show the effectiveness of our proposed method.\n\n**W2. We have augmented our analysis to include memory comparisons with existing cooperative 3D detection methods.** To ensure a focused evaluation, we specifically compare our FFNet with  DiscoNet[4], as the inference network structure of DiscoNet aligns closely with that of V2VNet[5]. Detailed implementation specifics for both methods are meticulously outlined in the supplementary materials (refer to Section E  \"Implementation Details of V2VNet and DiscoNet for VIC3D\" in  \"S1-Appendix.pdf\"). This comparison is summarized in Table 5 below. Notably, memory calculations were executed using an NVIDIA A100 GPU. In contrast to DiscoNet[5], **FFNet exhibits a slightly elevated memory footprint due to the processing of multiple frames on the infrastructure side, while the vehicle side memory remains consistent.** However, this increased memory usage remains well within acceptable limits for infrastructure computing servers.\n\n|               | Infrastructure Side |            |            |  | Vehicle Side |            |             |\n|---------------|---------------------|------------|-----------| --|--------------|------------|-------------|\n|               | Feature Flow     Generation      | Compression | Total       |  |     |\n| DiscoNet[5]      | /                   | 122.1M     | 243.9M     |  | 144.3M       |            |             |\n| FFNet         | 486.0M              | 122.1M     | 610.4M     |  | 144.3M       |            |             |\n| FFNet-C1      | 486.0M              | 286.1M     | 610.4M    |   | 144.3M       |            |             |\n| | | | | | | | | \n\nTable 5. Comparison of Memory Footprints. FFNet-C1 refers to FFNet with additional proposed compression modules.\n\nFurthermore, autonomous vehicles, with limited resources, are more sensitive to the memory footprint and computing consumption than infrastructure servers. **Our FFNet is memory and computing-friendly for the autonomous driving vehicles.** It extracts feature flow on the infrastructure side to predict future features, eliminating reliance on past data and conserving memory.  Latency compensation involves light linear computations, saving  vehicle-side computing resources.\n\nBest regards,  \n7569 Authors  \n\n[1] Xu et al. OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. ICRA2022  \n[2] Li et al. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. RA-L 2022  \n[3] Yu et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. CVPR2022  \n[4] Wang et al. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. ECCV2020  \n[5] Li et al. Learning distilled collaboration graph for multi-agent perception. NuerIPs2021  "}]

Author rebuttal:
**Conducting FFNet on More Datasets and Driving Contexts**

In response to some reviewers' concerns regarding the sufficiency of our experiments (specifically, Reviewer Wmh8's concern about limited dataset usage and Reviewer NKqy's concern about limited application scenarios), we have taken their feedback into consideration and conducted further experiments to address these issues.

**Datasets.** For our extended experiments, we chose to utilize two widely-used simulated datasets: OPV2X[1] and V2X-Sim[2]. These datasets are renowned for their focus on cooperative 3D detection tasks in multiple vehicle scenarios, making them ideal candidates to validate the capabilities of FFNet. Due to the constraints of the rebuttal time, we were able to complete only a portion of the experiments. Consequently, in this submission, we only report experiment results for the OPV2V dataset[1]. We will provide additional experimental results on the V2X-Sim dataset[2] in the next version.


**Experiment Setting and Results.**  Each scene in OPV2V[1] involves 2-7 autonomous vehicles. To align with  FFNet's existing framework, we adopted a two-vehicle configuration. We train FFNet, as well as the feature flow modules, using the training part of OPV2V[1]. Subsequently, we evaluate FFNet with and without feature flow prediction on the test part respect, under different latencies (100ms and 200ms). Experiment results are reported in the following table. From Table 1 below, it can be seen that:
- FFNet can leverage data from other vehicles to enhance detection performance effectively, compared to PointPillars[5].
- FFNet can mitigate performance drops caused by latency, outperforming methods like V2VNet[4] across both 100ms and 200ms latency scenarios.

| Model | FusionType       | Latency (ms) | mAP@BEV  || mAP@3D ||
|----------------------------------------|--------------|--------------|----------------|-------|----|-----|
|              |      |     |        IoU=0.5        |     IoU=0.7           |IoU=0.5| IoU=0.7  |
| PointPillars[5] | non-fusion | / |   71.7 | 56.7 | 70.0 | 44.6 |
| V2VNet[4]      | middle | 100           |       79.6        |    59.9   | 76.6                      | 49.8 |
| FFNet              | middle | 100           |       **82.1 (+10.4)**       |    63.3   | 79.6 | 54.6 |
| V2VNet[4]      | middle |  200          |       71.3        |    50.2   |             65.8          | 39.7 |
| FFNet              | middle |  200          |        **80.0 (+8.3)**       |    60.4   |  77.5 | 51.8 |
|||||||

Table 1. Experiments results on OPV2V[1] dataset. 

**Conclusion.** Our experiment results on the DAIR-V2X[3] and OPV2V[1] datasets confirm FFNet's efficacy across different datasets and its remarkable effectiveness in diverse V2X scenarios, including vehicle-infrastructure interactions and complex multiple vehicle scenarios. This showcases FFNet's adaptability and potential for excelling in a wide range of V2X applications, making it a powerful and reliable solution for cooperative driving tasks, reinforcing its value and significance in V2X research and application.

[1] Xu et al. OPV2V: An open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. ICRA2022  
[2] Li et al. V2X-Sim: Multi-agent collaborative perception dataset and benchmark for autonomous driving. RA-L 2022  
[3] Yu et al. DAIR-V2X: A large-scale dataset for vehicle-infrastructure cooperative 3d object detection. CVPR2022  
[4] Wang et al. V2vnet: Vehicle-to-vehicle communication for joint perception and prediction. ECCV2020  
[5] Lang et al. Pointpillars: Fast encoders for object detection from point clouds. CVPR2019  


Meta-review: This paper proposes the Feature Flow Net, which is presented to address temporal asynchrony and limited communication condition challenges in vehicle-infrastructure cooperative detection, and it demonstrates superior performance compared to other cooperative methods and demonstrates robustness in terms of latency. All reviewers have positive comments on the technical part, and the ACs agree with the reviewers that this paper is simple, intuitive, and effective. After reading the comments and the draft, ACs recommend accepting this paper and encourage the authors to take the comments into consideration.
Decision: Accept (poster)