SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 4: Illustration of training a feature flow generator using self-supervised learning and similarity loss. The upper red circle represents the first-order derivative generator, while the lower purple circles with solid and dashed lines share the same infrastructure feature extractor.
$$
\mathcal {L} (\mathcal {D}, \theta) = \sum_ {d _ {t _ {i}, k} \in \mathcal {D}} (1 - \frac {\widetilde {F} _ {i} \left(t _ {i} + k\right) \odot F _ {i} \left(P _ {i} \left(t _ {i} + k\right)\right)}{\left| \left| \widetilde {F} _ {i} \left(t _ {i} + k\right) \right| \right| _ {2} * \left| \left| F _ {i} \left(P _ {i} \left(t _ {i} + k\right)\right) \right| \right| _ {2}}), \tag {6}
$$
where  $\theta$  is the parameter of the feature flow generator, and we only update the parameters in first-order derivative generator  $\widetilde{F}_i'(\cdot)$  and frozen other parameters.
In this section, we implement FFNet on the DAIR-V2X dataset (40), comparing it with existing cooperative detection methods on different latencies. Our proposed FFNet outperforms all other methods, including early fusion, V2VNet (30), and DiscoNet (19), at 200ms latency, while only requiring about 1/100 of the transmission cost of raw point clouds. We demonstrate how FFNet overcomes the challenge of temporal asynchrony with feature flow prediction. Our results show that temporal asynchrony significantly reduces the performance of the cooperative detection model, but feature flow can effectively compensate for this drop. We evaluate FFNet on different latencies and show that it can robustly solve uncertain latency challenges with just one model. Furthermore, we show that self-supervised training can utilize extra infrastructure sequences. In the Appendix, we compare the performance of feature flow extraction on the infrastructure and ego vehicle sides.
Dataset. We used public and real-world DAIR-V2X dataset (40), which comprises over 100 scenes and 18,000 data pairs captured from infrastructure and vehicle sensors (Cameras and LiDARs) at 28 challenging traffic intersections. The dataset includes cooperative 3D annotations with vehicle-infrastructure cooperative view for 9,311 pairs, where each object is labeled with its corresponding category (Car, Bus, Truck, or Van). The dataset is divided into train/val/test sets in a 5:2:3 ratio, with all models evaluated on the val set. Additionally, raw sensor data is only released for the test set.
Note that the timestamps of the data from infrastructure and vehicle sensors in each pair are not precisely synchronized. The time difference of each pair in 9,311 pairs, is within the range of [-30, 30]ms. As the actual data cannot be altered after collection, we simulate a latency of  $k * 100ms$  by replacing the first  $k$  frames of the infrastructure frame with current infrastructure frame for each pair.
Implementation details. We utilized MMDetection3D (1) as our codebase and trained the feature fusion base model on the DAIR-V2X training set for 40 epochs, with a learning rate of 0.001 and weight decay of 0.01. To form  $\mathcal{D}$  for training the feature flow generator, we select each pair from training part and randomly set  $k$  from the range [1, 2]. More information on  $k$  and  $\mathcal{D}$  can be found in Sec.3.3. The pretraining of FFNet was done using the trained feature fusion base model. We trained the feature flow generator on  $D_{u}$  for 10 epochs with a learning rate of 0.001 and weight decay of 0.01. All training and evaluation were performed on an NVIDIA GeForce RTX 3090 GPU. The detection performance was measured using KITTI(10) evaluation detection metrics, which include bird-eye view (BEV) mAP and 3D mAP with 0.5 IoU and 0.7 IoU, respectively. Only the Car class was taken into account for objects located in the rectangular area [0, -39.12, 100, 39.12]. More implementation details regarding FFNet and the fusion methods are provided in the Appendix.
4.2 Comparison to Different Fusion and State of the Art Methods
We compare FFNet with four categories of fusion methods: non-fusion (e.g., PointPillars (16) and AutoAlignV2 (4)), early fusion, late fusion, middle fusion (e.g., DiscoNet (19)), and V2VNet (18).
Result Analysis. Table 1 presents a summary of our experimental results. The table is divided into three parts: the top section displays the evaluation results for non-fusion methods, the middle section shows the results for  $200ms$  latency, and the bottom section presents the results for  $300ms$  latency. Our proposed FFNet achieves new SOTA on DAIR-V2X. Notably, FFNet-C1 surpasses early fusion while it only requires about 1/100 of the transmission cost. Firstly, our proposed FFNet outperforms the non-fusion method PointPillars by  $9.32\%$  mAP@BEV (IoU=0.5) and  $7.32\%$  mAP@BEV (IoU=0.5) in  $200ms$  and  $300ms$  latency, respectively. This result indicates that utilizing infrastructure data can improve 3D detection performance. Secondly, although late fusion requires little transmission cost, the mAP@BEV (IoU=0.5) of late fusion is much lower than that of FFNet, up to  $5.10\%$  in  $200ms$  latency. Thirdly, compared with early fusion methods, FFNet achieves similar detection performance in  $200ms$  latency and outperforms  $2.92\%$  mAP in  $300ms$  latency, while it only requires no more than 1/10 of the transmission cost. Moreover, FFNet-C1 outperforms early fusion more than  $2\%$  mAP in  $300ms$  latency while only requiring 1/100 of the transmission cost. Fourthly, our FFNet achieves the best detection performance with the exact transmission cost as the
Table 1: Comparison to Different Fusion Methods. FFNet significantly outperforms all other fusion methods.
Model	FusionType	Latency (ms)	mAP@3D ↑	mAP@BEV ↑	\(\mathcal{A}\mathcal{B}\) (Byte) ↓
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
PointPillars (16)	non-fusion	/	48.06	-	52.24	-	0
AutoAlignV2 (4)	non-fusion	/	50.32	-	53.88	-	0
Early Fusion	early	200	54.63	38.23	61.08	50.06	\(1.4\times 10^6\)
Late Fusion	late	200	52.43	36.54	58.10	49.25	\(5.1\times 10^2\)
DiscoNet (19)	middle	200	50.76	28.57	58.20	48.90	\(1.2\times 10^5\)
V2VNet (30)	middle	200	49.67	26.96	56.02	46.32	\(1.2\times 10^5\)
FFNet (Ours)	middle	200	55.37	31.66	63.20 (+9.32)	54.69	\(1.2\times 10^5\)
FFNet-C1 (Ours)	middle	200	55.17	31.20	62.87 (+8.99)	54.28	\(1.7\times 10^4\)
Early Fusion	early	300	51.37	37.25	58.28	49.81	\(1.4\times 10^6\)
Late Fusion	late	300	51.35	36.24	56.89	48.79	\(5.1\times 10^2\)
DiscoNet (19)	middle	300	49.03	27.39	55.81	47.28	\(1.2\times 10^5\)
V2VNet (30)	middle	300	48.51	27.00	55.81	46.32	\(1.2\times 10^5\)
FFNet (Ours)	middle	300	53.46	30.42	61.20 (+7.32)	52.44	\(1.2\times 10^5\)
FFNet-C1 (Ours)	middle	300	54.10	29..87	60.76 (+6.88)	53.28	\(1.7\times 10^4\)
Table 2: Comparison between with and without Feature Prediction. Compared with no prediction models, FFNet with feature prediction has a significantly lower performance drop when there is communication latency.
Model	Latency (ms)	mAP@3D ↑	mAP@BEV ↑	AB (Byte) ↓
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
FFNet	0	55.81	30.23	63.54	54.16	1.2×10^5
FFNet (without prediction)	0	55.81	30.23	63.54	54.16	6.2×10^4
FFNet-V2 (without prediction)	0	55.78	30.22	64.23	55.00	1.2×10^5
FFNet	200	55.37	31.66	63.20 (-0.34)	54.69	1.2×10^5
FFNet (without prediction)	200	50.27	27.57	57.93 (-5.61)	48.16	6.2×10^4
FFNet-V2 (without prediction)	200	49.90	27.33	58.00 (-6.23)	48.22	1.2×10^5
middle fusion methods. For example, FFNet surpasses DiscoNet by  $5.0\%$  mAP@BEV (IoU=0.5) and  $5.39\%$  mAP@BEV (IoU=0.5) in 200ms and 300ms latency, respectively.
We conducted a series of experiments to demonstrate the effectiveness of the feature flow module in overcoming the temporal asynchrony challenge and to show that FFNet performs robustly under various latencies. Additionally, we studied how self-supervised learning can fully exploit infrastructure sequences that are independent of cooperative view and labeling.
Feature prediction can well solve temporal asynchrony. We conducted a series of experiments to evaluate the effectiveness of the feature flow module in overcoming the temporal asynchrony challenge. We evaluated FFNet under two different latency conditions: 0ms and  $200ms$ , where 0ms indicates temporal asynchrony between infrastructure data and vehicle data within  $[-30, 30]ms$ . To investigate the impact of temporal asynchrony on FFNet's performance, we also removed the prediction module from FFNet and directly fused the infrastructure feature. We refer to this version as FFNet (without prediction), abbreviated as FFNet-O, and evaluated it under both 0ms and  $200ms$  latency. Since FFNet-O does not require the transmission of the first-order derivative of the feature flow, it only requires half the transmission cost of FFNet. To ensure a fair comparison, we trained another version of FFNet called FFNet-V2, which compressed the feature flow from (384, 288, 288) to (384/16, 288/8, 288/8). FFNet-V2-O has the same transmission cost as FFNet, and we evaluated it under both 0ms and  $200ms$  latency as well.
The evaluation results, presented in Table 2, demonstrate that FFNet-O and FFNet-V2-O exhibit a significant performance drop under  $200ms$  latency. For example, FFNet-O experiences a  $5.61\%$  mAP@BEV (IoU=0.5) drop in  $200ms$  latency compared to  $0ms$  latency. Although FFNet-V2-O performs slightly better than FFNet and FFNet-O in  $0ms$  latency, FFNet significantly outperforms FFNet-V2-O in  $200ms$  latency. These results show that temporal asynchrony can significantly impact performance when we directly fuse the infrastructure feature, and that our feature prediction module can effectively compensate for the performance drop caused by temporal asynchrony.
Figure 5: Ablation study of FFNet robustness.
Figure 6: Ablation study of FFNet training.
FFNet is robust to uncertain latency. We conducted additional experiments to assess the performance of FFNet, FFNet-O, and FFNet-V2-O under varying latency cases, ranging from 100ms to 500ms. The experiment results are presented in Figure 5. As depicted in the figure, both FFNet-O and FFNet-V2-O exhibit continuous performance degradation as the latency increases from 100ms to 500ms. Specifically, in 500ms latency, FFNet-O and FFNet-V2-O show a significant  $9.38\%$  mAP@BEV (IoU=0.5) drop and  $9.76\%$  mAP@BEV (IoU=0.5) drop, respectively. Conversely, FFNet demonstrates minimal performance degradation within 200ms latency and only experiences a  $4.39\%$  mAP@BEV (IoU=0.5) drop. These results suggest that FFNet is resilient to varying latencies and can effectively handle uncertain latency in VIC3D problem. The ability of our feature flow to make predictions at an arbitrary future time before transmission is crucial since it could be received by different vehicles with different latencies.
FFNet training can fully utilize infrastructure sequences. We additionally trained the feature flow generator using the extra test portion of the DAIR-V2X dataset. For each frame in the test part, we randomly set  $k$  from the range [1, 2] to form  $\mathcal{D}_{test}$ . We first pretrained FFNet with the trained feature fusion base model and trained the feature flow generator solely with  $\mathcal{D}_{test}$  without  $\mathcal{D}$ . We refer to this trained FFNet as FFNet-V3. Subsequently, we pretrained FFNet with the trained feature fusion base model and trained the feature flow generator using  $\mathcal{D}_{test} \cup \mathcal{D}$ . We denote this trained FFNet as FFNet-V4. We evaluated FFNet-V3 and FFNet-V4 under latency from 100ms to 500ms. In Figure 5, we present the mAP@BEV (IoU=0.5) results. FFNet-V3 demonstrates significantly better performance than FFNet-O, indicating that the training of the feature flow generator can be independent of cooperative-view data. FFNet-V4 performs slightly better than FFNet, suggesting that incorporating more infrastructure sequences enhances the feature flow prediction ability.
This paper introduces FFNet, an innovative intermediate-level cooperative framework designed for VIC3D object detection. FFNet effectively addresses challenges related to temporal asynchrony and transmission cost by utilizing compressed feature flow for cooperative detection. Through extensive experiments conducted on the DAIR-V2X dataset, FFNet demonstrates superior performance compared to existing state-of-the-art methods. Furthermore, FFNet can be extended to various modalities, including image and multi-modality data, making it a versatile solution. Moreover, FFNet holds promise in the domain of multi-vehicle cooperative perception and leverages the utilization of additional frames to enhance feature prediction capabilities. The proposed FFNet framework, incorporating feature prediction and self-supervised learning, presents a promising avenue for VIC3D object detection and holds potential for addressing diverse cooperative perception tasks in the future.
This paper is partially supported by the National Key R&D Program of China No.2022ZD0161000 and the General Research Fund of Hong Kong No.17200622. This work was also supported by Baidu Inc. through the Apollo-AIR Joint Research Center.