SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we outline the experimental setup and assess our proposed VCD-E model, as well as the newly introduced trajectory-based distillation and occupancy reconstruction modules, using the nuScenes dataset [3]. This includes a presentation of the evaluation metrics, baseline models, ablation studies, and a comparative analysis of our approach with the current state-of-the-art methods.
Table 2: Comparison among the camera-only methods on the nuScenes test set. Methods marked with * denote long-term baseline implemented by us, based on BEVDet4D-Depth [22]. † depicts test time augmentation adopted during the inference phase. VCD-A achieves SOTA under critical metrics and surpasses its baseline by 2 points in NDS.
Methods	Backbone	Image Size	mAP↑	NDS↑	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
FCOS3D† [52]	R101-DCN	900 × 1600	0.358	0.428	0.690	0.249	0.452	1.434	0.124
DETR3D† [53]	V2-99	900 × 1600	0.412	0.479	0.641	0.255	0.394	0.845	0.133
UVTR [33]	V2-99	900 × 1600	0.472	0.551	0.577	0.253	0.391	0.508	0.123
BEVDet4D† [22]	Swin-B [41]	900 × 1600	0.451	0.569	0.511	0.241	0.386	0.301	0.121
BEVFormer [36]	V2-99	900 × 1600	0.481	0.569	0.582	0.256	0.375	0.378	0.126
PolarFormer [28]	V2-99	900 × 1600	0.493	0.572	0.556	0.256	0.364	0.439	0.127
BEVDistill [11]	ConvNeXt-B	900 × 1600	0.496	0.594	0.475	0.249	0.378	0.313	0.125
PETRv2 [40]	RevCol [4]	640 × 1600	0.512	0.592	0.547	0.242	0.360	0.367	0.126
BEVDepth [35]	ConvNeXt-B	640 × 1600	0.520	0.609	0.445	0.243	0.352	0.347	0.127
AeDet† [15]	ConvNeXt-B	640 × 1600	0.531	0.620	0.439	0.247	0.344	0.292	0.130
SOLOFusion [43]	ConvNeXt-B	640 × 1600	0.540	0.619	0.453	0.257	0.376	0.276	0.148
StreamPETR [51]	ConvNeXt-B	640 × 1600	0.550	0.631	0.493	0.241	0.343	0.243	0.123
Baseline*	ConvNeXt-B	640 × 1600	0.522	0.610	0.457	0.253	0.391	0.271	0.142
VCD-A	ConvNeXt-B	640 × 1600	0.548	0.631	0.436	0.244	0.343	0.290	0.120
Table 3: Comparison among the multi-modality methods on the nuScenes val set. Our proposed VCD-E only adopts image backbone while achieving comparable performance with state-of-the-art multi-modal methods who predominately rely on the LiDAR backbone.
Methods	Venue	Backbone	mAP↑	NDS↑	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
BEVFusion [38]	NeurIPS 2022	LiDAR & Image	0.642	0.680	-	-	-	-	-
FUTR3D [10]	Arxiv 2022	LiDAR & Image	0.645	0.683	-	-	-	-	-
UVTR [33]	NeurIPS 2022	LiDAR & Image	0.654	0.702	0.332	0.258	0.268	0.212	0.177
CMT [57]	Arxiv 2023	LiDAR & Image	0.679	0.708	-	-	-	-	-
VCD-E	-	Image	0.677	0.711	0.308	0.254	0.317	0.189	0.201
Camera-only 3D detection. In order to evaluate the effectiveness of our proposed Revenge model and the trajectory-based distillation modules, we have conducted rigorous experiments on the nuScenes validation and test sets. As presented in Tab. 1, the performance of VCD-A surpasses other cutting-edge methods, achieving a record of  $44.6\%$  and  $56.6\%$  on the nuScenes benchmark. This provides robust evidence of the effectiveness of our approach. Utilizing an image resolution of  $256 \times 704$  and a ResNet-50 backbone, VCD outperforms the state-of-the-art method from [43] by improving mAP by  $1.9\%$  and NDS by  $3.2\%$ . Additionally, VCD-A exhibits an improvement over our baseline by  $2.8\%$  mAP and  $2.4\%$  NDS, thereby indicating that our methods can considerably advance state-of-the-art results. To further ascertain the generalizability of our methods, we conducted experiments on the nuScenes test set, as shown in Tab. 2. With the adoption of the ConvNext-B backbone [42], VCD achieved  $54.8\%$  mAP and  $63.1\%$  NDS, outperforming the state-of-the-art detector, SOLOFusion [43], by  $0.8\%$  mAP and  $1.2\%$  NDS. Furthermore, our proposed approach led to a  $2.6\%$  increase in mAP and a  $2.1\%$  improvement in NDS compared to our baseline, thereby demonstrating the efficacy of our method for large-scale vision-based models.
Multi-modality Fusion. We first compare our proposed vision-centric expert with other state-of-the-art multi-modality fusion methods. VCD-E does not require any complex fusion strategies or three-stage training schedule, yet it achieves  $67.7\%$  mAP and a  $71.1\%$  NDS as shown in Tab. 3. Despite employing only a single image backbone, the model's performance is comparable to state-of-the-art methods such as BEVFusion [38], and it even surpasses UVTR [33] by  $2.3\%$  mAP and  $0.9\%$  NDS, which utilizes two backbones, by a considerable margin. Owing to its homogeneous nature and compatibility with camera-only models, the VCD-E can serve as a powerful expert for knowledge transfer, facilitating performance improvements for small to large vision detectors.
To verify the effectiveness and necessity of each component, we conduct various ablation experiments on the nuScenes validation set.
Table 4: Ablation study of the proposed distillation framework on different temporal lengths. The design works with different length of time windows and the gain grows with the temporal length.
Temporal Length	Distill	mAP (%)↑	NDS (%) ↑	mATE↓	mAOE↓	mAVE↓
1	X	26.6	37.9	0.815	0.645	0.556
✓	30.1 (+3.5)	41.5(+3.6)	0.732	0.629	0.476
2	X	26.9	38.4	0.804	0.706	0.461
✓	31.3 (+4.4)	43.2 (+4.8)	0.717	0.615	0.403
4	X	28.4	39.8	0.748	0.739	0.432
✓	33.0 (+4.6)	44.1 (+4.3)	0.707	0.632	0.389
8	X	29.7	40.9	0.762	0.714	0.415
✓	35.4 (+5.7)	45.9 (+5.0)	0.690	0.625	0.370
Table 5: The performance gains of the apprentice which benefit from different experts. CM denotes cross-modal and UM represents Uni-modal. It indicates the success of uni-modal distillation remains.
Expert	Paradigm	mAP	NDS
-	-	0.297	0.409
CenterPoint [60]	CM	0.281	0.420
Transfusion [2]	CM	0.292	0.435
BEVDepth [35]	UM	0.341	0.442
VCD-E	UM	0.354	0.459
Table 6: Effect of different distillation methods. All models are trained with VCD-E as expert. Our proposal significantly surpasses previous SOTA methods.
Methods	mAP	NDS
Baseline [35]	0.297	0.409
FitNet [44]	0.318	0.421
CWD [46]	0.311	0.412
BEVDistill [11]	0.316	0.439
VCD-A	0.354	0.459
Effectiveness of the General Framework. We first validate the effectiveness of the general framework for various temporal lengths. In Tab. 4, we examine different timestamps for temporal modeling and find that our method significantly outperforms the baseline. By employing the general framework, we achieve  $5.0\%$  NDS and  $5.7\%$  mAP performance gains with 8 temporal modeling instances. For other temporal modeling scenarios, our method continues to exhibit substantial performance improvements, ranging from  $3.6\%$  to  $5.0\%$  NDS. As the duration of temporal fusion extends, the advantages of the model become increasingly pronounced, suggesting that this framework exhibits higher compatibility with extended sequences. This indicates the effectiveness of our proposed trajectory-based distillation module, which alleviates the issue of motion misalignment.
Effectiveness of VCD-E. In this study, it is crucial to verify the effectiveness of our proposed vision-centric experts. We select various expert models for a fair comparison, including LiDAR-based expert Centerpoint [60], vision-based expert BEVDepth [35], and fusion-based expert Transfusion, based on our proposed distillation strategy. In Tab. 5, we observe that homogeneous expert models significantly influence the success of knowledge transfer. The camera-based expert BEVDepth demonstrates superior performance gains compared to the other two heterogeneous expert models. Our advanced model, VCD-E, operates within the same modality as vision-based models, and it is equipped with precise geometric information. Consequently, the distillation effect outperforms the Transfusion [2] expert by a significant margin, achieving a  $6.2\%$  increase in mAP. Moreover, it surpasses the BEVDepth expert by an additional  $1.7\%$  in NDS. This demonstrates the superior performance and effectiveness of our model.
Comparison to SOTA. To further demonstrate the effectiveness of our proposed distillation strategy, we compare our method with other state-of-the-art methods. To ensure a fair comparison, we conduct all experiments based on our vision-centric expert VCD-E. We find that our method consistently outperforms other distillation methods by a significant margin. In 2D detection, FitNet [44] and CWD [46] are two classic distillation methods, we adapt them for 3D object detection to facilitate a fair comparison. BEVDistill [11] represents a state-of-the-art distillation method for multi-view 3D object detection, and we compare our results with this approach as well. As shown in Tab. 6, our method achieves the best results when compared to these state-of-the-art distillation strategies, demonstrating the effectiveness of our approach. Our method surpasses BEVDistill by  $2\%$  NDS and  $3.8\%$  mAP.
Table 7: Gains of different image backbone on multi-modal models [38]. The stronger backbone still demonstrates better performance in this case.
Methods	Backbone	mAP	NDS
BEVFusion	ResNet-50	0.598	0.662
BEVFusion	ConvNext-B	0.597	0.665
VCD-E	ResNet-50	0.611	0.656
VCD-E	ConvNext-B	0.664	0.693
Table 8: Gains of different depth fusion strategy. The proposed fusion depth is optimal among different methods.
Methods	mAP	NDS
Predicted depth	0.495	0.585
LiDAR depth	0.638	0.687
Weighted depth	0.644	0.690
Fusion depth	0.646	0.690
Table 9: The performance gains of different trajectory length for trajectory-based distillation. As the trajectory length increases, the benefits derived from the distillation process become more pronounced.
Trajectory Length	Distill	mAP (%)	NDS (%)
-	X	29.7	40.9
0	✓	31.8	42.1
1	✓	33.1	44.5
3	✓	34.6	45.6
5	✓	35.4	45.9
9	✓	33.9	44.7
Gains of the Image Backbone. To verify that VCD-E benefits from contemporary image backbones, we conducted experiments outlined in Tab. 7. We selected ResNet-50 and ConvNext-B as two distinct modern image backbones for BEVFusion and VCD-E. Our findings indicate that VCD-E achieves substantial improvements of  $5.3\%$  mAP and  $3.7\%$  NDS when using ConvNext-B compared to ResNet-50, demonstrating that VCD-E can indeed benefit from modern image backbones. However, existing fusion-based methods, such as BEVFusion, which primarily rely on the capabilities of LiDAR backbones, show limited gains from employing modern image backbones.
Fusion Strategy of Expert. To determine the optimal fusion strategy for LiDAR points and images, we propose four distinct depth fusion approaches, including (1) Predicted depth that directly uses predicted depth based on image features. (2) LiDAR depth that utilizes the projected depth from LiDAR points. (3) Fusion depth that employs the projected depth when corresponding LiDAR points exist for each pixel, otherwise using the predicted scores based on image features, and (4) Weighted depth. Applying the weighted average depth of the predicted depth and LiDAR depth. As demonstrated in Tab. 8, fusion depth yields the best results which indicates that using corresponding predicted scores where LiDAR depth is unavailable is advantageous.
The Effectiveness of Trajectory-based Distillation The results presented in Table 9 indicate that as the trajectory length increases, the benefits derived from the distillation process become more pronounced. The temporal fusion length for this experiment is set at eight. The first row denotes the baseline model which does not use the distillation method. The second row depicts that the VCD-A model directly conducts distillation under the full BEV feature without the Trajectory-based distillation module. When the trajectory length is set to 1, we only distill ground truth locations in the current frame. However, when the trajectory length exceeds five, there is a noticeable decrease in accuracy. We hypothesize that this decrease may be attributed to the model's distracted attention towards distant motions. The density of traffic can lead to distant motion locations being occupied by other objects, which may not necessarily require additional trajectory supervision. This suggests that the application of excessive trajectory supervision in such scenarios could be unnecessary and inefficient.
Ablation Study of Each Component. In Tab. 10, we conduct an ablation study on the components employed in VCD to verify their contributions to the final result. These components include (1) Long-term temporal fusion, which allows our model to benefit from historical information, thereby enhancing its performance. (2) Trajectory-based distillation, which transfers knowledge from VCD-E to VCD-A in areas with motion misalignment to mitigate this issue. (3) Occupancy reconstruction,
Table 10: Ablation study of important components in VCD-A. It shows that the motion module contributes most to the performance improvement, indicating the potential of future research here.
Methods	Long	Occ	Motion	mAP↑	NDS↑	mATE↓	mAVE↓
Baseline	0.271	0.319	0.767	1.065
Longer Temporal Fusion	✓	0.297	0.409	0.762	0.415
Trajectory-based Distillation	✓	0.283	0.356	0.772	0.870
All but Longer Temporal	✓	✓	0.290	0.367	0.724	0.834
Occupancy Reconstruction	✓	✓	0.308	0.416	0.734	0.379
VCD	✓	✓	✓	0.341	0.449	0.715	0.389
which serves as dense depth supervision in the perspective view, improving the performance of VCD-A. Long-term temporal modeling enables our model to reference prior information from history, assisting in object velocity estimation and resulting in a significantly lower mATE. We also evaluate the impact of the trajectory-based distillation module when all other components are incorporated. As shown in Tab. 10, trajectory-based distillation increases NDS by  $3.3\%$  and mAP by  $3.3\%$ , contributing to the majority of the improvement.
In this paper, we presented a novel vision-centric model as the expert, which leverages the strengths of both modalities, our proposed model achieves exceptional performance, rivaling that of state-of-the-art multimodal fusion models, while also mitigating domain gap issues, making it suitable for distilling vision-based models. Furthermore, we introduced two innovative modules, the trajectory-based distillation and occupancy reconstruction modules. These modules enhance the geometric perception capabilities of multi-camera 3D object detection models and improve the detection of both static and dynamic objects in the scene. Our experiments demonstrate the effectiveness of our proposed methods on the widely-used nuScenes 3D object detection benchmark.
Limitation. In this work, we have not delved into more details of the vision-centric expert, which may hold significant potential for improvement. Additionally, we have not explored further applications, such as automatic dimension estimation based on VCD-E.
This work was supported by National Key R&D Program of China (2022ZD0160104) and NSFC (62206172). We would like to thank anonymous reviewers for active discussions.
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. 2
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. TransFusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 1, 3, 8
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 6, 14
[4] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang. Reversible column networks. In ICLR, 2023. 7
[5] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022. 3
[6] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306.16927, 2023. 3
A.1 Dataset and Evaluation Metrics
We conduct our experiments on the nuScenes dataset [3], a widely used benchmark for autonomous driving tasks. The dataset encompasses diverse driving scenarios captured using cameras and LiDAR sensors, offering rich information for both visual and LiDAR-based 3D object detection. The dataset comprises 700 training scenes, 150 validation scenes, and 150 testing scenes. Each scene spans approximately 20 seconds, with key frames annotated at a  $2\mathrm{Hz}$  frequency.
The two dominant metrics for the nuScenes detection task are the nuScenes Detection Score (NDS) and mean Average Precision (mAP). The mAP for nuScenes is computed based on the center distance between predictions and ground truth annotations on the ground plane. Moreover, the nuScenes dataset defines five true positive metrics (mATE, mASE, mAOE, mAVE, mAAE) for measuring translation, scale, orientation, velocity, and attribute, respectively. The NDS for nuScenes is a weighted sum of mAP and the five true positive metrics, defined as  $NDS = \frac{1}{10} [5mAP + \sum_{mTP}(1 - \min(1,mTP))]$ .
We conduct experiments on BEVDepth [35]. The codebase is developed upon MMDetection3D [13]. Main experiments are trained on 8 NVIDIA A100 GPUs, while ablation experiments are conducted on 8 NVIDIA V100 GPU. For BEVDepth, the model is trained for 20 epochs with an initial learning rate of 2e-4. In the distillation process, the per-GPU batch size is set to 4, whereas during the training of the baseline model, it is set to 8. Normal data augmentations are introduced in the training process such as flip and rotate. In our apprentice models, future frames are not incorporated into the long-term temporal fusion throughout the training phase to ensure a fair comparison. As for ablation study, we conduct experiments with an online training strategy, and we have employed the ResNet-50 configuration in the absence of CBGS.
In our research, we implement distinct temporal modeling strategies for both apprentice and expert models. For the apprentice models, we incorporate a sequence of eight frames into the temporal modeling process. In contrast, the expert models integrate four future frames into the temporal modeling as demonstrated in our primary results. However, in our ablation study, we deviate from this approach and instead employ eight historical frames for temporal modeling. Besides, We use the last sweep LiDAR frame with the current LiDAR frame to create the depth map.
Table 11: Experiment settings. * denotes that the training schedule for VCD-E is approximately one-fourth of the original schedule. This reduction was implemented to expedite the training process during the ablation study. The first group is engaged in training on the main results, whereas the second group is utilized in the ablation study.
Method	Backbone	Image Size	Frames	mAP (%)	NDS (%)
VCD-E	ConvNext-B [42]	512 × 1408	8+1	67.7	71.1
VCD-A	Res-50 [20]	256 × 704	8+1	41.8	54.2
VCD-E*	ConvNext-B [42]	256 × 704	8+1	54.2	58.8
VCD-A	Res-50 [20]	256 × 704	8+1	29.7	40.9
A.3 Experiments Settings
The setting of adopted expert-apprentice pairs is depicted in Tab. 11. We categorize the distillation setting into two distinct groups. The primary group is engaged in training on the main results, whereas the second group is utilized for the ablation study. Both our baseline and VCD-A adopt 8 history frames for temporal fusion.
Figure 3: Effects of VCD on static objects. Our distillation framework VCD can still consistently improves static objects, demonstrating  $3.2\%$  and  $7.6\%$  improvements in precision-recall (mAP) and localization quality (mATE), respectively.
B The Analysis of Temporal Fusion
B.1 The Misalignment of Motion Objects
As highlighted in preceding studies [43], long-term temporal fusion may face misalignment issues in motion estimation, which can be discerned through a reduction in performance on metrics like mATE. Let's consider a moving object and analyze the impact of inaccurate motion estimation on its position in the fused frame. We will assume that the environment is static, except for the moving object. Let the position of the moving object in the world coordinate system be represented by  $\boldsymbol{P}_i^w = (x_i^w, y_i^w, z_i^w, 1)^T$  in each of the  $N$  frames captured at times  $t_1, t_2, \ldots, t_N$ . The actual motion of the moving object between frames is represented by  $M_i^{obj}$ , and the estimated motion is represented by  $\hat{M}_i^{obj}$ . The difference between the estimated and actual motion of the object can be denoted as:
$$
\Delta M _ {i} ^ {\text {o b j}} = M _ {i} ^ {\text {o b j}} - \hat {M} _ {i} ^ {\text {o b j}}. \tag {7}
$$
As we have already computed the transformation matrix  $\pmb{T}_i$  based on the estimated ego motion, we can calculate the transformed object position in the current frame, considering its actual motion, as:
$$
\boldsymbol {P} _ {i} ^ {w ^ {\prime}} = \boldsymbol {T} _ {i} \times \boldsymbol {M} _ {i} ^ {o b j} \times \boldsymbol {P} _ {i} ^ {w}. \tag {8}
$$
The error in the transformed object position can be computed as:
$$
e _ {i} ^ {o b j} = P _ {i} ^ {w ^ {\prime}} - \hat {P} _ {i} ^ {w ^ {\prime}}. \tag {9}
$$
In the long-term fusion process, we integrate the information from all  $N$  frames. Assuming we use a fusion function  $F$ , the fused position in the current frame can be represented as:
$$
\boldsymbol {P} _ {f u s i o n} ^ {o b j} = F \left(\boldsymbol {P} _ {1} ^ {w ^ {\prime}}, \boldsymbol {P} _ {2} ^ {w ^ {\prime}} \dots , \boldsymbol {P} _ {N} ^ {w ^ {\prime}}\right). \tag {10}
$$
The inaccuracies in the motion estimation of the moving object for each frame can propagate through the fusion function and result in a misaligned object in the fused frame. The overall error in the fused position can be represented as a function of the errors in each frame:
$$
\boldsymbol {e} _ {\text {f u s i o n}} ^ {\text {o b j}} = G \left(\boldsymbol {e} _ {1} ^ {\text {o b j}}, \boldsymbol {e} _ {2} ^ {\text {o b j}} \dots ., \boldsymbol {e} _ {N} ^ {\text {o b j}}\right), \tag {11}
$$
where  $G$  represents a function that combines the errors from each frame. The fused position of the moving object will be less accurate due to these motion estimation errors, leading to a decline in object detection performance in the long-term setting. To address the issue mentioned earlier, we introduce the trajectory-based distillation module, which compensates for the misalignment of moving objects. We will provide further details in the subsequent discussion.
B.2 The Improvements of Static and Dynamic Objects
In this section, we present visualizations to demonstrate the improvements achieved in dynamic objects. Particularly noteworthy is the significant enhancement in the representation of dynamic objects through trajectory-based distillation, thereby highlighting the effectiveness of the trajectory-based module. As depicted in Fig. 3 and Fig. 4, our distillation framework consistently enhances static and dynamic objects across various metrics.
mAOE↓
mAVE↓
Figure 4: Effects of VCD on movable objects. Our distillation framework VCD consistently improves dynamic objects across a range of metrics.
CAM_FRONT_LEFT
CAM_FRONT
CAM_BACK
CAM_FRONT_RIGHT
CAM_BACK_RIGHT
CAM_BACK_LEFT
Figure 5: Visualization of the predictions for 3D object detection generated by the VCD-A.
We have performed several visualizations in Fig. 5 to showcase the advancements achieved by our distillation framework. Our findings indicate that our models excel in accurately predicting 3D bounding boxes for the target objects.
Our research introduces a novel perspective for multi-modal methodologies and a fresh distillation paradigm for camera-only techniques. We believe that it can establish a robust baseline for the broader scientific community. However, while our methods contribute to the enhancement of autonomous driving, they are not yet capable of addressing more complex corner cases. Consequently, these limitations could potentially introduce risks in real-world autonomous systems.