SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Ablation on our distillation losses.
CD	TD	RD	NDS(%)	mAP(%)
-	-	-	51.44	40.51
-	-	✓	52.88	42.07
-	✓	-	53.21	42.66
✓	-	-	53.91	43.26
-	✓	✓	53.70	42.86
✓	-	✓	53.89	43.42
✓	✓	-	54.00	43.41
✓	✓	✓	54.31	44.03
Table 2: Comparisons of feature-level distillation methods. Bottom results are obtained by applying different representation learning strategies to CD loss.
Method	NDS(%)	mAP(%)
w/o KD	51.44	40.51
MSE	51.67	41.01
MSE w/ GT	51.84	41.03
CD w/ VICReg [2]	52.04	40.89
CD w/ CLIP [45]	52.47	41.69
CD w/ Barlow Twins [67]	53.91	43.26
Comparison with Feature-Level Distillation Methods. To solely examine the effectiveness of our correlation regularizing distillation (CD), we compared it to the following feature-level distillation methods without applying other distillation approaches. MSE is a baseline method [19, 29] that simply minimizes the element-wise  $\mathcal{L}_2$ -distance between the features of a teacher and a student. MSE w/ GT is similar to MSE, but it only distills features near the ground truth point [11, 9]. Our CD loss in Eq.2 draws inspiration from the Barlow Twins [67] approach for representation learning. To validate our choice, we also conducted a comparison with other alternative methods. Specifically, we applied VICReg [2] and CLIP [45] to CD loss and evaluated their impact.
Tab.2 highlights that even in the absence of other distillation losses, the CD loss alone delivers the most promising performance, resulting in a significant increase of  $2.47\%$  in NDS. At the bottom of Tab.2, we observe that other representation learning methods also lead to performance improvements. Notably, the Barlow Twins method achieves the best performance among the tested methods. From a distillation perspective, VICReg solely relies on MSE loss to transfer the teacher's knowledge, which is referred to as the invariance term in [2]. As a result, the structural knowledge from the teacher may not be fully exploited, leading to suboptimal performance. CLIP is a contrastive learning method that typically requires a large and diverse set of positive and negative pairs of features for effective learning. In the case of 3DOD, however, cross-modal feature batches extracted from a scene typically contain only a limited number of positive pairs, while the number of negative pairs is much larger. Consequently, the effectiveness of contrastive learning may be compromised in the context of 3DOD. In contrast, Barlow Twins leverages the structural knowledge by exploiting cross-correlation between the cross-modal features (see Eq.1), and is robust to the size of feature set [67]. Thus, our correlation regularizing distillation (CD) is the most effective when implemented with Barlow Twins.
Correlation Regularization Effect of CD Loss. In Sec.3.2, we discussed that distillation with correlation regularization aims to maximize the information contained in the learned features, allowing the individual components of the feature to become more informative. Here we validate our argument using two qualitative measures: (1) effective dimension and (2) dimensional redundancy.
(1) Effective dimension [46] quantitatively measures the dimensionality of features. A higher effective dimension indicates lower redundancy among the features. In Tab.3, we compare effective dimensions  $d_{\mathrm{eff}}$  of features learned through various feature-level distillation methods. Since  $d_{\mathrm{eff}}$  is bounded by  $\sqrt{D} = 16$ , we also provide the squared values for a intuitive interpretation, bounded by  $D = 256$ . For example, the features learned through CD can be effectively represented in a dimensional space of at least 20 dimensions, approximately. Notably, our CD loss achieves the highest effective dimension compared to other feature distillation methods.
Table 3: Effective dimension  $d_{\mathrm{eff}}$  for different feature-level distillation methods and teacher.
Method	deff	d2eff
MSE	3.810	14.519
MSE w/ GT	4.059	16.474
CD (Ours)	4.389	19.267
Teacher	5.757	33.137
(2) Dimensional redundancy: Redundant feature components exhibit higher correlation among themselves. From this intuition, we quantify the redundancy in features by utilizing Eq.1 and 2. For camera features  $\mathbf{G}$ , we define a dimensional self-correlation matrix  $\mathbf{C}_{\mathrm{self}} = \hat{\mathbf{G}}^T\hat{\mathbf{G}}$ . We then calculate the second term of Eq.2 for  $\mathbf{C}_{\mathrm{self}}$  and refer to the resulting value as the dimensional redundancy of  $\mathbf{G}$ . Fig.3 depicts the dimensional redundancy of features learned from the MSE and CD loss throughout the training progress. Results demonstrate that the CD loss is more effective in reducing dimensional redundancy compared to the MSE loss. Furthermore, even when the MSE loss is replaced with the CD loss during training, a decrease in dimensional redundancy is still observed (red dashed lines).
Figure 3: Trend of dimensional redundancy of features over training progress of the student model.
These results indicate that distilling features through a simple element-wise comparison can lead to dimensional collapse. In contrast, our information maximization distillation method leverages the structural knowledge embedded in the feature space by considering the cross-correlation of feature components, and ultimately contributes to significantly improved prediction performance (see Tab.1).
Table 4: Performance comparisons on the nuScenes validation set. For fair comparisons, we use ResNet-101-DCN as backbone for all the experiments.  $\dagger$  indicates our implementation of BEVFormer [34] with modified BEV grid size of  $128\times 128$ . * indicates our implementation of Object-DGCNN [34] with a modified transformer-decoder module. In modality column, 'L', 'C', and  $\mathrm{L}\rightarrow \mathrm{C}$  represent LiDAR, Camera, and knowledge distillation from LiDAR to Camera, respectively.
Method	Modality	NDS(%)	mAP(%)	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
Obj-DGCNN* [55]	L	66.7	60.9	0.332	0.260	0.307	0.264	0.202
UVTR-L [29]	L	66.4	59.3	0.345	0.259	0.313	0.218	0.185
DETR3D [54]	C	42.5	34.6	0.773	0.268	0.383	0.842	0.216
PETR [35]	C	44.2	37.0	0.711	0.267	0.383	0.865	0.201
PolarFormer [24]	C	52.8	43.2	0.648	0.270	0.348	0.409	0.201
BEVDepth [33]	C	53.8	41.8	-	-	-	-	-
BEVFormer† [34]	C	51.4	40.5	0.690	0.275	0.360	0.362	0.194
+BEVDistill [9]	L → C	52.4	41.7	-	-	-	-	-
+STXD (Ours)	L → C	54.3	44.0	0.635	0.264	0.365	0.309	0.198
UVTR-C [29]	C	44.1	36.2	0.758	0.272	0.410	0.758	0.203
+L2C [29]	L → C	45.0	37.2	0.735	0.269	0.397	0.761	0.193
+STXD (Ours)	L → C	46.1	39.0	0.698	0.265	0.413	0.755	0.203
UVTR-CS [29]	C	48.3	37.9	0.731	0.267	0.350	0.510	0.200
+L2CS [29]	L → C	48.8	39.2	0.720	0.268	0.354	0.534	0.206
+STXD (Ours)	L → C	50.8	41.4	0.662	0.265	0.381	0.484	0.199
Effect of  $K$  on TD Loss. The temporal consistency distillation (TD) in Eq.4 utilizes the BEV features extracted from the past  $K$  frames. Tab.5 demonstrates increasing the value of  $K$  results in improved 3DOD performance, particularly in terms of mAVE (mean Average Velocity Error). By leveraging the information from consecutive frames through the TD loss, the student can effectively learn the dynamics of target objects, leading to accurate predictions of their dynamic attributes. This highlights the impor
tance of referencing past frames, as it allows the teacher to transfer a greater amount of information to the student, resulting in improved 3DOD performance. Similar results have also been echoed in concurrent works [32, 41], which also leverage temporal information for the single-modal 3DOD.
Table 5: Performance of the proposed method depending on  $K$  .
K	NDS(%)	mAP(%)	mAVE↓
1	54.06	43.48	0.3246
2	54.13	43.45	0.3148
3	54.31	44.03	0.3093
Visualization of Temporal Similarity Map.
The similarity map defined in Eq.3 calculates the element-wise similarity between the BEV features of the current and past frames. In Fig.4, we select the feature element corresponding to the location of the car object in the current BEV space and calculate its similarity to the entire features of the past frames. Notably, the student model trained with TD exhibits improved similarity patterns with the teacher (see Fig.4-(a) and (b)). This indicates that the student model has learned to capture similar temporal relations and representations
from the teacher by referring to the past frames. The competence of our temporal distillation approach ultimately contributes to the improved 3DOD performance, as demonstrated in our ablation studies (see Tab.1 and 5). We provide more temporal similarity visualization results in the Appendix.
(a) Teacher
(b) w/ TD
(c) w/o TD
Figure 4: Examples of temporal similarity maps for (a) teacher model, (b) student model trained with TD, and (c) student model trained without TD.
4.3 Comparisons with Other Methods
To demonstrate the generalizability of our approach, we apply STXD to both UVTR [29] and BEVFormer [34] baselines. For a fair comparison, we use ResNet-101-DCN [18, 13] as the backbone of our student models, unless otherwise specified. Tab.4 shows the evaluation results on the nuScenes
Table 6: Performance comparisons on the nuScenes testing set.  $\dagger$  indicates implementation of BEV-Former [34] with modified BEV grid size of  $128\times 128$ .  $\ddagger$  indicates the use of V2-99 [28] as backbone, otherwise ResNet-101-DCN is used for a fair comparison. In modality column, 'L', 'C', and  $\mathrm{L}\rightarrow \mathrm{C}$  represent LiDAR, Camera, and knowledge distillation from LiDAR to Camera, respectively.
Method	Modality	NDS(%)	mAP(%)	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
DETR3D [54]	C	47.9	41.2	0.641	0.255	0.394	0.845	0.133
PETR [33]	C	45.5	39.1	0.647	0.251	0.433	0.933	0.143
PolarFormer [24]	C	54.3	45.7	0.612	0.257	0.392	0.467	0.129
BEVFormer† [34]	C	52.6	42.4	0.620	0.263	0.457	0.387	0.132
+STXD (Ours)	L → C	55.5	46.5	0.579	0.259	0.462	0.363	0.113
BEVFormer‡ [34]	C	55.5	45.7	0.605	0.263	0.371	0.375	0.116
+STXD (Ours)	L → C	58.3	49.7	0.546	0.256	0.383	0.356	0.117
UVTR-C [29]	C	43.0	36.4	0.724	0.266	0.486	0.898	0.143
+L2C [29]	L → C	44.0	38.2	0.677	0.262	0.493	0.925	0.150
+STXD (Ours)	L → C	45.8	40.2	0.634	0.271	0.465	0.918	0.143
UVTR-CS [29]	C	48.6	39.0	0.702	0.263	0.435	0.548	0.144
+L2CS [29]	L → C	48.7	39.8	0.679	0.262	0.432	0.612	0.133
+STXD (Ours)	L → C	51.8	43.5	0.614	0.260	0.407	0.589	0.132
validation set, where STXD significantly improves the performance of the based student models, surpassing the performance of existing distillation methods [29, 9], up to  $2.9\%$  in NDS and  $3.5\%$  in mAP. In most cases, the addition of STXD contributes to the improvement of mATE (translation), mASE (scale), and mAVE (velocity) metrics. Both mATE and mASE are closely related to the geometric features of 3D objects, and the teacher effectively transfers such important information to the student models. The improvement in mAVE can be attributed to our temporal distillation method.
Similar results are echoed in our experiments on the nuScenes testing dataset, as shown in Tab.6. We did not use CBGS [70] or test time augmentations to evaluate the sole effect of our method. STXD consistently improves NDS and mAP by up to  $3.2\%$  and  $4.5\%$ , respectively, compared to the various baseline methods (UVTR-C/CS, L2C/L2CS, BEVFormer†). STXD also improves performances for the student model with heavy backbone (BEVFormer‡) by  $2.8\%$  in NDS and  $4.0\%$  in mAP. These results are significant as models with heavy backbones already achieve high performance, making it challenging to further improve through knowledge distillation. In the Appendix, we show the competence of STXD on lightweight backbones as well. These results demonstrate the effectiveness and versatility of STXD in enabling cross-modal distillation between the teacher and the student models. Overall, STXD enables the student to learn the maximum amount of information from the teacher, including geometric and temporal knowledge, leading to significantly enhanced performance.
In this paper, we presented STXD, a cross-modal knowledge distillation framework for the multi-view 3D object detection task. STXD is capable of effectively distilling structural and temporal knowledge across different modalities through Correlation Regularizing Distillation (CD) and Temporal Consistency Distillation (TD). To further enhance the distillation quality, STXD also adopts Response-Level Distillation (RD), which transfers task-specific knowledge at the output level. Our extensive experiments and ablation studies demonstrated the effectiveness of STXD on the nuScenes dataset, where the NDS and mAP of the based student detectors are improved by up to  $3.2\%$  and  $4.5\%$  on the testing set. Inspired by recent works on multi-modal fusion [36, 63, 1] and other types of 3D perception tasks (e.g., BEV Segmentation [68, 20], Occupancy Detection [23, 30]), we plan to further explore the potential of STXD in such various tasks leveraging cross-modal distillation. It would also be valuable to apply and validate our approach with other models that utilize long-term information such as [17, 41]. We hope our work provides a solid baseline and new perspectives on structural and temporal cross-modal distillation.
Societal Impact. Though the proposed method significantly improves the performance of multiview camera-based 3D object detection, it does not guarantee perfect predictions for all cases. Therefore, when applied to actual autonomous driving scenes, a contingency plan must be included.
In this appendix, we provide additional details of the implementation and training of our method (Appendix A). We also present additional experimental results, including ablations on lightweight backbones, additional baselines, and qualitative examples (Appendix B). These results further demonstrate the effectiveness of our STXD framework in cross-modal distillation for 3D object detection.
A.1 Details on Cross-Modal Distillation Baselines
In Sec.4.1 of the manuscript, we introduced two cross-modal distillation baselines for 3D object detection: (1) BEVFormer [34] as the student model and Object-DGCNN [55] as the teacher model; and (2) UVTR-C/CS as the student models and UVTR-L as the teacher model [29]. To be more specific, in the first baseline (Object-DGCNN  $\rightarrow$  BEVFormer), we set the spatial dimensions of the bird's-eye view (BEV) feature to be  $X = Y = 128$ , and  $Z = 1$  for both the teacher and student models, indicating a flat BEV feature space. In accordance with the concurrent work [9], we replaced the DGCNN attention module of the transformer-decoder in the teacher model with vanilla multi-head attention. Apart from these modifications, we followed the original implementations of both the student and teacher models. In the second baseline (UVTR-L  $\rightarrow$  UVTR-C/CS), we utilized the original implementation of UVTR [29] without any modifications. The dimensions of the bird's-eye view (BEV) features were set to  $X = Y = 128$  and  $Z = 11$  for UVTR-C/CS; and  $Z = 5$  for UVTR-L. Here,  $Z$  represents the height of the Bird's Eye View (BEV) feature space. The BEV features were further sampled using the  $(x,y,z)$  positions of object queries  $Q$ , where  $|Q| = 900$ .
Note that we adopted two baseline student models that utilize multi-frame inputs. BEVFormer generates the BEV feature of the current frame by referring to the BEV feature of the past frames via temporal self-attention mechanism. In UVTR-CS, multi-frame images are concatenated and forwarded to the image encoder.
For a fair comparison, we used ResNet-101-DCN pretrained by FCOS3D [53] as the backbone network for the student models in our extensive experiments (see Sec.4.2 and 4.3 in the manuscript) following prior works [34, 29]. We also showcased the competence of our STXD framework with other types of backbones, including VoVNet-99 [28] initialized from DD3D [40] as well as lightweight ResNet-18 and ResNet-50 networks (see B.1). We set the balancing parameter  $\gamma$  to 0.8 and 0.6 for classification and localization in Eq.5 of the manuscript, following [10, 59]. The weights for each loss term in Eq.7 of the manuscript are set to (0.1, 100.0, 1.0, 1.0), respectively, in the order mentioned. The student models were trained for 24 epochs using a learning rate of  $2 \times 10^{-4}$ , and a batch size of 1 per GPU. All models were trained on 8 of NVIDIA A100 GPU while following the original codebase for each model. We employed AdamW [37] as the optimizer with a weight decay of  $1 \times 10^{-2}$ . To isolate the effect of our STXD framework in cross-modal distillation, we deliberately did not use the CBGS strategy [70] or test-time augmentation in our experiments.
A.3 Instance Matching Function in Response-Level Distillation
In Sec.3.4, we adopted the commonly used Hungarian algorithm [26] to define set-to-set matching between the ground-truth and candidates from the teacher and the student, separately. Then, we constructed a mapping function  $\pi(j)$  between candidates from the teacher and the student based on matched ground-truth indices.
B.1 Ablations on Lightweight Backbones
To further validate the effectiveness of our STXD framework in practical scenarios where lighter backbone networks are preferred for deployment on edge devices, we conducted experiments using smaller backbone networks for the student model. This allows us to assess the performance of our approach under resource-constrained settings while improving the level of accuracy in 3D object detection. We trained BEVFormer with ResNet-18 and ResNet-50 backbone networks on nuScenes training set and evaluated on the validation set. The results in Tab.7 demonstrate that STXD significantly improves the performance of BEVFormer with smaller backbone networks where it achieves an improvement of  $3.24\%$  and  $4.00\%$  in NDS and  $3.33\%$  and  $4.49\%$  in mAP, respectively. These improvements are consistent with the results presented in Tab.4 and 6 of the manuscript, further confirming the versatility and compatibility of our approach across different backbone architectures.
Table 7: Performance of STXD on lightweight backbone networks for the student model.
Method	Backbone	NDS(%)	mAP(%)
BEVFormer [34]	ResNet-18	40.42	28.70
+STXD (Ours)	ResNet-18	43.66	32.03
BEVFormer [34]	ResNet-50	44.06	32.86
+STXD (Ours)	ResNet-50	48.06	37.35
B.2 Experiments on Additional Teacher-Student Baseline
To further validate the effectiveness of our approach across various teacher-student baselines, we extend our experiments to additional student baseline, BEVDepth [33], which relies on LSS-based feature generation [43], depth supervision, and multi-frame inputs. Here we followed the official implementation of TiG-BEV [22] for cross-modal distillation with CenterPoint [64] as the teacher model, but added our feature-level distillation losses for fair comparisons. Results are reported in Tab.8, where our approach outperforms prior distillation methods, such as BEVDistill [9] and TiG-BEV [22], by up to  $+3.2\%$  of NDS. These results are consistent to the results from BEVFomer and UVTR-C/CS baselines in Tab.4 and 6.
Table 8: Performance comparison of distillation methods on the baseline with CenterPoint as the teacher and BEVDepth as the student. TiG-BEV (FD): applying only the feature-level distillation (FD) losses ( $\mathcal{L}_{\text{inter-channel}}$ ,  $\mathcal{L}_{\text{inter-keypoint}}$ ). TiG-BEV: leveraging full set of losses including inner-depth supervision proposed in TiG-BEV. STXD (CD): applying our correlation regularizing distillation loss. STXD (TD): applying our temporal consistency distillation loss.
Method	Backbone	Modality	NDS(%)	mAP(%)
CenterPoint [64]	VoxelNet	L	64.6	56.4
BEVDepth [33]	ResNet-50	C	43.1	32.9
+BEVDistill [9]	ResNet-50	L → C	45.4	33.2
+TiG-BEV (FD) [22]	ResNet-50	L → C	45.2	35.8
+TiG-BEV [22]	ResNet-50	L → C	46.1	36.6
+STXD (CD)	ResNet-50	L → C	48.4	37.1
+STXD (TD)	ResNet-50	L → C	48.3	36.1
B.3 Comparison to Correlation-based Distillation Method
Our Correlation Regularizing Distillation (CD) and TiG-BEV [22] both consider dimensional correlation of features. However, these methods are distinct in their underlying objectives and implementations. TiG-BEV proposes the inter-channel BEV distillation loss (IC loss) that encourages the student's features to replicate the dimensional self-correlation of the teacher's features (i.e., minimize  $||\mathbf{F}^T\mathbf{F} - \mathbf{G}^T\mathbf{G}||_2$ ). In contrast, the core contribution of CD is introducing cross-modal
cross-correlation regularization to prevent student features from being redundant. Specifically, CD exploits the decorrelation mechanism (the second term of Eq.2) to prevent information collapse in the student model. However, IC loss does not utilize such an information maximization mechanism, and only relies on the global one-to-one MSE distance (L2-norm) between modality-specific self-correlation matrices. Hence, CD pursues a different learning principle compared to TiG-BEV.
These differences lead to the competence of CD over TiG-BEV. Tab.9 shows that CD effectively reduces duplicate dimensional feature components compared to the correlation-based distillation of TiG-BEV. Furthermore, in Tab. 8, CD ultimately achieves performance improvements over TiG-BEV and BEVDistill by up to  $+3.2\%$  of NDS and  $+3.9\%$  of mAP. These results consistently demonstrate the significance of mitigating feature redundancy in cross-modal knowledge distillation and enhancing feature quality through our feature-level distillation approaches.
Table 9: Validation for correlation regularization effect of CD loss. For both TiG-BEV and our STXD, we measured the effective dimension  $(d_{\mathrm{eff}})$  and dimensional redundancy values defined in Sec.4.2. TiG-BEV (FD) represents the results of applying only the feature distillation loss from TiG-BEV.
Method	deff↑	d2eff↑	Dim.Red.↓
TiG-BEV (FD) [22]	1.620	2.634	0.174
STXD (CD)	1.694	2.871	0.062
Teacher [33]	2.194	4.813	0.038
B.4 Qualitative Results
B.4.1 Prediction Results on BEV Space
Fig.5 and 6 provide examples of 3DOD results in the BEV space from the LiDAR-based model (top-left), the multi-view camera-based model trained with our STXD (top-middle), and the camera-based model trained without knowledge distillation (camera-only model, top-right). Additionally, at the bottom of each example, we have included camera views that correspond to the highlighted regions, which can help in better understanding the results. Overall, STXD enables more accurate prediction of bounding box locations, scales, and orientations compared to the camera-only model (highlighted by red dashed boxes). This observation is consistent with the quantitative results in Tab.4 and 6 of the manuscript. STXD also leads to a reduction of false positives. In certain examples, we observe a decrease in false positives even when comparing the results to the LiDAR-based model (highlighted by pink dashed boxes). Notably, in Fig.6, STXD effectively reduces false positive detections in 3DOD compared to the LiDAR-based model and the camera-only model. These results indicate that our STXD framework effectively transfers valuable information from the LiDAR-based teacher to the camera student, leading to improved detection performance.
B.4.2 Qualitative Comparison on BEV Features
As an extension of Fig.1 in the manuscript, we provide additional examples of BEV features learned using different feature-level distillation methods in Fig.7. In various samples, the model trained with the correlation regularizing distillation (CD) consistently generates more similar feature patterns (Fig.7-(b)) to the teacher model (Fig.7-(a)) compared to other methods (Fig.7-(c) and (d)). These results demonstrate the effectiveness of the proposed method in transferring the rich information contained in the teacher model to the student model.
B.4.3 Temporal Similarity Maps