STXD: Structural and Temporal Cross-Modal Distillation for Multi-View 3D Object Detection
Authors: Sujin Jang
Dae Ung Jo
Sung Ju Hwang
Dongwook Lee
Daehyun Ji
Venue: NeurIPS 2023 poster (2023)
Keywords: knowledge distillation
cross-modal learning
3d object detection
TL;DR: We improve multi-view-based 3D object detection through a cross-modal knowledge distillation framework, which effectively transfers structural and temporal knowledge from the LiDAR-modality teacher to the image-modality student.

Abstract: 3D object detection (3DOD) from multi-view images is an economically appealing alternative to expensive LiDAR-based detectors, but also an extremely challenging task due to the absence of precise spatial cues. Recent studies have leveraged the teacher-student paradigm for cross-modal distillation, where a strong LiDAR-modality teacher transfers useful knowledge to a multi-view-based image-modality student. However, prior approaches have only focused on minimizing global distances between cross-modal features, which may lead to suboptimal knowledge distillation results. Based on these insights, we propose a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3DOD. First, STXD reduces redundancy of the feature components of the student by regularizing the cross-correlation of cross-modal features, while maximizing their similarities. Second, to effectively transfer temporal knowledge, STXD encodes temporal relations of features across a sequence of frames via similarity maps. Lastly, STXD also adopts a response distillation method to further enhance the quality of knowledge distillation at the output-level. Our extensive experiments demonstrate that STXD significantly improves the NDS and mAP of the based student detectors by 2.8%~4.5% on the nuScenes testing dataset.

STXD; Structural and Temporal Cross-Modal Distillation for Multi-View 3D Object Detection
Sujin Jang"!
Dae Ung Jo Sung Ju Hwang? Dongwook Lee'
Daehyun Ji
Samsung Advanced Institute of Technology (SAIT) 1
Korea Advanced Institute of Science and Technology (KAIST)*
st.steve. daeung jo, dw12. lee, derek.
d (1)a@aamunng.com com
sjhwang820kaist..ce
Abstract
3D object detection (3DOD) from multi-view images is economically appealing alternative to expensive LiDAR- baseed detectors, but also an extremely challenging task due the absence of precise spatial cues. Recent studies have leveraged the teacher-student paradigm for cross moos- distillation, where a strong LiDAR- image-modality
modality teacher transfers useful knowledge to multi-view-hand student. However, prior approaches have only focused on minimizing global dis- tances between cross-modal features, which may lead to suboptimal knowledge distillation results. Based these insights, we propose novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3DOD. First, STXD reduces redundancy of the feature components of the student by regularizing the cross-correlation of cross-modal features, while maximizing their similarities. Second, to effectively transfer temporal knowledge, STXD en- codes temporal relations of features across a sequence of frames via similarity maps. Lastly. STXD also adopts response distillation method to further enhance the quality knowledge distillation at the output-level. Our extensive experiments demonstrate that STXD significantly improves the NDS and mAP of the based student detectors by 2.8% 4.5% on the nuScenes testing dataset.
Introduction
3D object detection (3DOD) is the task of locating and classifying objects in 3D space using input data from specific modalities, which typically include LiDAR point clouds [61, 14, 7] and camera images [54, 34, 24]. 3DOD is central to understanding the surrounding environment and has been widely applied to various complex vision systems, such as autonomous driving [44]. robotic manipulation [71, 65], and augmented reality [42]. Recently, camera-basse based multi-view 3DOD has emerged as an attractive alternative to expensive LiDAR-based methods, thanks its ubiquity, low cost, and the semantically rich information from colorized pixels. In general, current works on multi-view 3DOD aim to extract unified bird's-eye s-eye view (BEV) feature map from multiple camera images. However, 3DOD from only camera-specific BEV features is extremely challenging due to the lack of precise spatial cues [39], which leads to inferior performance of camera-basee based models compared to LiDAR- based models. To bridge the gap, some prior works [16, 11, 29, 9] introduced a teacher-student paradigm for cross-modal knowledge distillation to transfer geometrically-riet 3D information from LiDAR models (teacher) to camera-based 3DOD models (student). The current state-of-the the-art art 3DOD models typically follow a similar detection paradigm, where moodality-speciffe encoders are used to extract BEV features, followed by 3D object detection head [29, 34, 55, 7].
'Theee authors equally contributed this work
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a) LiDAR
(b) Ours
(c) MSE w/ GT
(d) MSE
(c) Camera
Figure 1: Examples of BEV features generated by (a) the LiDAR model, (b)-(d) the model trained with different reature-lever distillation methods, and (e) the camera-only model. Here we visualize the C2-morm value of the BEV features. More examples are provided in the Appendix.
Hence, recent works have taken the BEV features from different modalities as the focal point of knowledge distillation [16, 29], and then applied additional response-lewe distillation [11, 19, 9].
One of the main challenges in cross moosl knowledge distillation is transferring modality-specifft knowledge from the teacher to the student in a different modality. This is primarily because the features learned by different modalities are typically non-homogeneous, and there exist distributional divergences between the feature spaces as shown in Fig.1-(a) and (e). Most existing distillation methods aim to minimize global distances between cross-modal features (i.e., Cz-distance). While this approach has achieved some success, it fails to capture the structural knowledge inherent in the modality-speciffe features, which can lead to suboptimal knowledge distillation, as shown in 1-g. and (d). To address these challenges, we first propose distillation method that regularizes cross-correlation of BEV features from the teacher and the student models. In particular, we intro- duce a decorrelation mechanism [67, 2] to reduce redundancy among feature components, thereby maximizing the information contained in the features. Our structural knowledge distillation approach ultimately prevents information collapse in the student model and enhances the component-wist feature similarities across different modalities, as illustrated in Fig. and (b).
We also explore the integration of temporal knowledge embedded in features of previous frames. Although temporal information has been actively exploited recent 3DOD approaches [34, 29, 32, 41J, it has been relatively understudied for cross-modal distillation. The main challenge is the spatio- temporal misalignment between past and current frames. For instance, in driving scenes, foreground objects may have different locations or disappear over time. Thus, directly matching the misaligned cross-modal features could result in incorrect distillation. Instead, we leverage the temporal relations of the BEV features by generating temporal similarity maps between the teacher's s current and past frames. Specifically, we train the student model to generate the BEV feature of the current frame, which mimics the temporal relations encoded in the similarity maps from the teacher. In this way, we can avoid false matching issues when transferring temporal information over sequence of frames. To further improve the quality of transferred knowledge, we also adopt the response distillation approach [10, 59, 19, 9], which assigns quality scores to the predictions of the teacher model, thereby enabling the distillation of only meaningful information at the prediction-level.
Our structural and temporal modal distillation (STXD) framework integrates the proposed feature- and response-level distillation approaches (see Fig.2), and demonstrated a sigmificant im- provement on the nuScenes 3DOD benchmark [3], leading to up to 3.2% NDS and 4.5% mAP improvement over the based student models. We also conducted detailed ablation studies to validate the effectiveness of each component of STXD. Notably, our framework does not require additional computational cost at testing time, while improving the prediction performance of the student models.
Related Work
based- 3DOD. Recent LiDAR-based 3DOD detectors can be categorized based on the representation types of 3D space: voxel-based and point-based approaches. The voxel-based ap- proaches [69, 58, 27, 48, 14, 7] convert point clouds into a regular voxel-grid representation. Point- based approaches [62, 49, 64, 61] extract point-wise features directly from raw point clouds. Both approaches involve transforming low-level features into a BEV representation, which is then followed by an object detection head that generates response features.

---

LIDAR Hoint Clouds
LIDAR BEV Features
UDAR Predictions
Point Clouds Backbone
Detection Head
Feature-l evel Distillation
Response- Distillation
30 EOx
Lyo
LIAAR Feetur |Previous Framnes)
Matching
LAD
Camrero Festures
30 Box
Matrik
Cross Pross-Correaaion
Simlarity Map
Correlation Regularizing Distillation
Temporal Temporal Consistency Distillation
Image Backbone
Detection Head
Multi-View Camera Images
Camera BEV Features
Camera Predictions
Figure An overview STXD framework for multi-view 3D object detection. Structural and temporal feature-level distillation is applied to the unified bird's-eye s-eee view (BEV) features extracted and transformed from both LiDAR- and camera-bases backbones (Sec.3.2 and 3.3). Response-lewe distillation is applied to the 3D box candidates predicted by both LiDAR- and camera-base detectors 3.).. During testing, only the camera model is utilized without additional computational cost.
Camera-based 3DOD. Camera-based approaches typically perform 3DOD on either single- or multi-view images. Prior works on single-view 3DOD aim detect 3D boxes directly from image features [5, 38, 53], or using intermediate geometric representations (e.g., pseudo-lidar [66, 12, 50], probabilistic depth-cues [47, 52]). More recently multi-view 3DOD approaches [21, 33, 29] use the sift-splat-shoot method [43, 47] to transform multi-image features into unified bird s-eye view (BEV) representation followed by an object detection head. Other works on multi-view images [54, 34, 35, 24] aim to implicitly learn query-based BEV features sampled from corresponding 2D image features based on attention mechanisms [4, 72]. While some of these works [50, 47, 33] utilize LiDAR point clouds as input for explicit supervision of depth cues, they do not fully exploit the potential of LiDAR- based 3DOD features, which have appeared to be effective for improved prediction performance of camera-based models. In light of this work, we further provide new perspectives on the structural and temporal knowledge embedded in the LiDAR-based BEV features.
Multi-Modal 3DOD. Recently, integrating complementary information from different modality sensors, especially LiDAR and cameras, has demonstrated improved 3DOD performance. One line of work fuses intermediate feature representations in the BEV space [8, 31, 36, 63], while other approaches try fuse object proposals from each modality in the detection head [25, 6, 1]. Another line work augments point clouds with semantic information extracted from corresponding images [51, 51, 57]. However, these methods require both sensors to be available during test time. In contrast, our cross-modal distillation approach leverages both modalities only during training to transfer useful knowledge from LiDAR-based camera-based 3DOD models. During testing, use only multi-camera images without any additional computational costs.
Knowledge Distillation for 3DOD. Most prior works on knowledge distillation for object detection have focused on transferring knowledge among single-modal detectors, particularly for 2D object detection (e.g., [15, 59]) and LiDAR-base models (e.g., [60, 56]). Relatively fewer works have studied cross-modal distillation for object detection. MonoDistill [11] projects LiDAR point clouds into perspective image space to distill spatially aligned LiDAR features to camera- based student model. LIGA-Stereo [16] aims to transfer intermediate features from LiDAR to a stereo camera-based student model. CMKD [19] and BEVDistill [9] employ knowledge distillation from LiDAR-base bbsse models at both the feature and response levels to minimize the gap between modalities. Similarly, UVTR [29] tries directly match the BEV representations of the camera model to the LiDAR-basc

---

teacher. Although these methods have achieved some success, they still do not carefully consider the distributional divergences between non-homogeneous modalities. To mitigate the modality gup between LiDAR and camera features, TIG-BEV [22] learns the inner geometry of foreground objects by encouraging the student's features to replicate the self-correlation of the teacher's features. However, this method still does not consider cross-modal associations among BEV features from the teacher and student models. Differs from previous methods, our distillation approach considers structural knowledge in the features by introducing cross-modal cross-correlation regularization inspired by the decorrelation mechanism [67, 2]. Furthermore, we introduce a method for effectively distilling temporal information contained in previous frames of the teacher. We also utilize response- level distillation to further improve the quality of knowledge distillation at the predictions level.
3 Methodology
3.1 Preliminary
Recent 3D object detectors share similar detection paradigm, where low-level features are extracted from modality-speciien encoders and subsequently transformed into BEV features (see 2). For example, modern LiDAR-based detectors [69, 27, 55] use 3D encoders to extract low-level features from the raw point clouds, then the features are further transformed to the BEV features F E xxxxx2xx with spatial extent of X x Y Z and the number of channels D. For recent multi-view image detectors, there are two mainstream approaches for transforming the low-level image features to the BEV features: (1) estimating depth cues [21, 33, 29] via lift-splat-shoot [43] and (2) implicit learning of depth via cross- cosss-atttntionn mechanism [35, Regardless of the type of transform methods, the BEV features can be represented by G R xxxxx Z similar the LiDAR- detectors. In this work, as illustrated in Fig.2, we aim to train high-ppeforming camera-based detector (student) by distilling useful knowledge from a LiDAR- detector (teacher) while considering the structural and temporal information embedded in the modality-specin BEV featuers, F and G.
3.2 Correlation Regularizing Distillation
Prior works on cross-modal distillation directly force the student to mimic the BEV feature from
the teacher by calculating element-wise distance loss (i.e., || Gll2- 2 [19, 29I), or weighted via foreground mask (i.e., || fg- (F G)lla [11, 9I) However, they ignore the structural knowledge inherent in poddalttysspecifee features and may not effectively distill rich information contained in the features. Recently, several self-supervised learning methods have incorporated cross-correlation regularization to maximize the information contained in features [67, 2]. These methods can also be applied to the problem of LiDAR-to --too-aamera knowledge distillation, since LiDAR and camera features can be regarded as different representations obtained from the same scene, and have a spatial one-to-one correspondence. From this intuition, we propose the Correlation Regularizing Distillation (CD) loss that maximizes the similarity between aligned features while regularizing cross-correlation along feature dimensions. Let batches of N aligned D-dimensional LiDAR and camera features be given as F = [f, f2.,...f.. ]r] e TRN' xD and G [g g2...., | E TR xD respectively. N is the number of serialized BEV features, given by X - ZZ. Following [67], each feature is transformed to be mean-centeres along the batch dimension, denoted by F and G. Then, CD loss is defined as:
C - DDD
CCD :- : E(- de MX C(i,))
where C denotes a dimensional cross-correlatee correlatoon matrix and A 2 is balancing hyper- aeper--rraeeter. The first term of Eq.2 encourages the aligned LiDAR and camera feature components to be similar, while the second term reduces the correlation among feature components that are not aligned. Hence, CD loss can reduce duplicated information in the learned features, allowing for the student to learn diverse information from the teacher. We analyzed the effectiveness of the CD loss in Sec.4.2.
3.3 Temporal Consistency Distillation
In driving scenes, sensor data is received sequentially. Therefore, the past few frames may contain valuable information for detecting objects in the current frame, and incorporating temporal infor- mation has brought significant improvements to 3DOD [34, 29, 32, 41]. However, directly learning

---

from the LiDAR features of the past frames (i.e., via L--distance) may not be effective due to the spatially and temporally misaligned con-homogeneous features. To address this issue, we propose the Temporal Consistency Distillation (TD) loss, which indirectly distills the teacher's information from past frames by introducing temporal similarity map, Let batches of aligned LiDAR and camera features of the current frame be given as F(O) and G respectively. We are also given the LiDAR features of the k-th previous frame, denoted by F(-*), k E [1, KJ. where K is the maximum number of past frames referred to. Then, the cross-modal temporal similarity map S and the intra-modal temporal similarity map T between the current and the k-th previous frame are calculated as:
( = G F ( E ER RRNN T = =F F (m)" E RNN, h E [1, K].
(3)
Then TD loss is defined as:
CTD : DKL
(4)
where D is the Kullback-Leible divergence (KLD). Since S and T are matrices, apply a softmax function to each row of them, and then calculate the KLD for each row. Based on the temporal similarity map, TD loss distills the teacher's information of past frames indirectly, avoiding spatial false matching issues across the time frames. We also demonstrated and analyzed the effectiveness of TD in Sec.4.2.
3.4 Response-Level Distillation
In general, detector models infer a large number of candidates, but not all candidates contain meaningful information. Since only a few candidates detect target objects, it is necessary for a student selectively learn from the predictions of the teacher. Inspired by recent works on prediction-guie -guieed distillation [10, 59, 19, 9], we define the Response-Leve Level Distillation (RD) loss that assigns quality scores to the predictions of the teacher model, enabling the distillation of only meaningful information. Let us define {(b; ). as prediction results of teacher model, where b; and c; are regression and classification results for the i-th of M box candidates, respectively. In a similar manner, student
prediction can be defined as {(by. ر
-=1- Following [10, quality score qi for i-th candidate of the teacher predictions can be calculated follow:
q: -= <<(x) 1-7 (IoU(b;.b.))
where cτ is the predicted probability for the class of the ground truth box that is matched to i-th can- didate of the teacher predictions, by is regression parameter of the ground truth box that is matched to i-th candidate of the teacher predictions, and is a hyper-parameter balancing classification and localization. Then RD loss is defined as:
LRD Σ (T())) (Ibb+)) -byll + DKZ(Cro)l(e))).
(6)
where Tji) is function that returns the index of the corresponding teacher's prediction matched with the j-th student's prediction. By combining RD with our structural and temporal distillation (CD,TD), achieved additional improvement the detection performance reported in Sec.4.2.
3.5 Optimization Objective
Finally, the overall optimization objective for the student models can be obtained by integrating the aforementioned three distillation losses and the task loss Ltask:
Liotal LCD + LTD + LRD + Crask:
where Crask is adopted from a general set-to-set target assignment for the student with ground truth, including regression and classification losses [54], based on the Hungarian algorithm [26].
Experiments
We now present extensive experimental results to validate our structural and temporal cross-modal knowledge distillation framework (STXD) for 3D object detection (3DOD) task. We first introduce the details of experimental setting (Sec.4. 1) followed by various ablation studies to validate the effectiveness of each component in STXD (Sec.4.2). Then, we compare STXD against several leading methods the nuScenes [3] dataset (Sec.4.3). Additional experiments are provided in the Appendix.
5

---

4.1 Experimental Settings
Dataset. We evaluate our STXD framework on the nuScenes [3] benchmark dataset, which is large-scale and widely used by recent research works on 3D object detection. The nuScenes dataset comprises 1,000 driving scenes with a duration of approximately 20s, which are divided into 700, 150, and 150 scenes for training, validation, and testing, respectively. Fach scene consists of RGB images collected by six cameras, which capture a 360 field of view at 12Hz and 32-beam LiDAR point clouds at 20Hz. Annotations for 23 categories of objects are provided for each sample at a rate of 2Hz, and the metries for the 3DOD task are evaluated using 10 classes of objects, as defined by the official evaluation metrics. These metrics include mean Average Precision (mAP) and nuScenes Detection Score (NDS). detail, mAP measures the recall and precision predicted bounding boxes, while NDS is weighted sum of mAP and five true positive nuScenes metrics: mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mA0E), mean Average Velocity Error (mAVE), and mean Average Attribute Error (mAAE).
Baseline Models. To validate the effectiveness of our approach, we adopt two state-of of-the-art cross- modal distillation approaches baselines. Similar concurrent work [9], as the image-modalit student model and Object-DGCNN [55] as the LiDAR-modality teacher model, where the DGCNN attention module of the dransdermerddcooder is replaced with vanilla multi-head attention. For the second baseline, we adopt the multi-view camera (single frame of UVTR-C, multi-frame of UVTR-CS) and LiDAR (UVTR-L) models from UVTR [29] the student and teacher models, respectively. Our base student models represent two mainstream methods for transforming 2D perspective image features into 3D BEV features, where BEVFormer uses cross- attention mechanism [4, 72] and UVTR-C/CS is based on estimating depth distributions through lift-splat-shoot approach [43, 47].
BEVFormer [34]
Implementation Details. The input image size for the student models is set to 1600 x 900. For both teacher and student models, we use voxel size of (0.1m, 0.1m, 0.2m) and the input point clouds are filtered within range of [-51.2m 51. 51.2m] for the X and Y axes and [-5.0m, 3.0m] for the Z axis. The dimensions of the BEV feature grids are 128 128, and the number of channel D for BEV features is set to 256. We train all student models for 24 epochs without using CBGS [70]. A Eq.2 set to 0.01 according to [67]. More implementation details are provided the Appendix.
4.2 Ablation Studies
For our ablation studies, we implemented BEVFormer [34] with ResNet-101-DCN as backbone network for the student model, and evaluated the trained model on the nuScenes validation dataset.
Ablation on Distillation Methods. To validate the effects of each proposed distillation approach in Sec.3, we conducted ablation study on the loss function. As shown in Tab. 1, each loss term contributes to the improvement in performance. Specifically, the proposed feature-level distillation (CD and TD) plays crucial role in enhancing the performance. When CD and TD losses applied solely, NDS is increased by 2.47% and 1.77%, respectively. When all loss terms are combined, the performance is further improved without any conflicts, resulting in the best performance with a significant improvement of NDS by 2.87% and mAP by 3.52%. These results demonstrate the effectiveness of structural and temporal distillation as well as the response distillation method.
Table 1: Ablation on our distillation losses.
Table 2: Comparisons of feature-level distillation methods. Bottom results are obtained by applying dif- ferent representation learning strategies to CD loss.
CD
TD
RD
NDS(%) %)
mAP(%)
51.44
40.51
Method
NDS( %)
mAP(%)
v
52.88
42.07
53.21
42,66
w/o KD
51.44
40.51
53.91
43.26
MSE
51.67
41.01
MSE w/ GT
51.84
41.03
53.70
42.86
53.89
43,42
CD w/ VICReg [2]
52.04
40.89
54.00
43,41
CD w/ CLIP [45]
52.47
41.69
54.31
44.03
CD w/ Barlow Twins [67]
53.91
43.26

---

Comparison with Feature-Level Distillation Methods. To solely examine the effectiveness of our correlation regularizing distillation (CD), we compared it to the following feature-level distillation methods without applying other distillation approaches. MSE is baseline method [19, 29] that simply minimizes the element-wise L2-distance between the features of teacher and a student. MSE w/ GT is similar to MSE, but it only distills features near the ground truth point [11, 9]. Our CD loss in Eq.2 draws inspiration from the Barlow Twins [67] approach for representation learning. To validate our choice, we also conducted a comparison with other alternative methods. Specifically, we applied VICReg [2] and CLIP [45] to CD loss and evaluated their impact.
Tab.2 highlights that even in the absence of other distillation losses, the CD loss alone delivers the most promising performance, resulting in significant increase of 2.47% in NDS. At the bottom of Tab.2. we observe that other representation learning methods also lead to performance improvements. Notably, the Barlow Twins method achieves the best performance among the tested methods. From a distillation perspective, VICReg solely relies on MSE loss to transfer the teacher's 's knowledge, which referred as the invariance term in [2]. As result, the structural knowledge from the teacher may not be fully exploited, leading to suboptimal performance. CLIP is a contrastive learning method that typically requires a large and diverse set of positive and negative pairs of features for effective learning. In the case of 3DOD, however, cross-modal feature batches extracted from a scene typically contain only a limited number of positive pairs, while the number of negative pairs is much larger. Consequently, the effectiveness of contrastive learning may be compromised in the context of 3DOD. In contrast, Barlow Twins leverages the structural knowledge by exploiting cross- correlation between the cross- mooss- features (see Iq. and is robust to the size of feature set [67]. Thus, our correlation regularizing distillation (CD) is the most effective when implemented with Barlow Twins.
Correlation Regularization Effect of CD Loss. In Sec.3.2, we discussed that distillation with correlation regularization aims to maximize the information contained in the learned features, allowing the individual components of the feature become more informative. Here validate our argument using two qualitative measures: (1) effective dimension and (2) dimensional redundancy.
(1) Effective dimension [46] quantitatively measures the di- mensionality of features. higher effective dimension indi- cates lower redundancy among the features. In Tab.3, we com- pare effective dimensions deff of features learned through var- ious feature-level distillation methods. Since deff is bounded by vD = 16, we also provide the squared values for a intu- itive interpretation, bounded by D 256. For example, the features learned through CD can be effectively represented in dimensional space of at least 20 dimensions, approximately. Notably, our CD loss achieves the highest effective dimension compared to other feature distillation methods.
Table 3:
Effective dimension dett for different feature-level distillation methods and teacher.
Method MSE
delt
deff
3,810
14.519
MSE w/ GT
4.059
16.474
CD (Ours)
4.389
19.267
Teacher
5.757
33.137
(2) Dimensional redundancy: Redundant feature com- ponents exhibit higher correlation among themselves. From this intuition, we quantify the redundancy in fea- tures by utilizing Eq.1 and 2. For camera features G. we define a dimensional self-correlation matrix Czelf = GTG. We then calculate the second term of Eq.2 for Cuel and refer to the resulting value the dimensional redundancy of G. Fig.3 depicts the dimensional redun- dancy of features learned from the MSE and CD loss throughout the training progress. Results demonstrate that the CD loss is more effective in reducing dimen- sional redundancy compared to the MSE loss. Further- more, even when the MSE loss is replaced with the CD loss during training, a decrease in dimensional redun- dancy is still observed (red dashed lines).
Dimensional Redundancy
0.08
CD (Ours)
Late Appliance
0.07
0.06
0.05
8
12
16
24
Training Epoch
Figure 3: Trend of dimensional redun- dancy of features over training progress of the student model.
These results indicate that distilling features through a simple element-wise comparison can lead to dimensional collapse. In contrast, our information maximization distillation method leverages the structural knowledge embedded in the feature space by considering the cross-correlation of feature components, and ultimately contributes to significantly improved prediction performance (see Tab.
7

---

Table 4: Performance comparisons the nuScenes validation For fair comparisons, BesNet-101-DCN. as backbone for all the experiments. f indicates our implementation of BEV- Former [34] with modified BEV grid size of x28x 128. indicates our implementation of Object-
DGCNN [34] with modified transformer-deccoder module. In modality column, 'L', 'C", and 'L +C
represent LiDAR, Camera, and knowledge distillation from LiDAR Camera, respectively.
Method
Modality
NDS(%)
mAP(%%
mATE,
mASEL
mAOEJ mAVEJ mAAE|
Obj-DGCNN" [55]
66.7
60.9
0.332
0.260
0.307
0.264
0.202
UVTR-L. [29]
L
66.4
59.3
0.345
0.259
0.313
0.218
0.185
DETR3D [54]
C
42.5
34.6
0.773
0.268
0.383
0.842
0.216
PETR [35]
44.2
37.0
0.711
0.267
0.383
0.865
0.201
PolarFormer [24]
C
52.8
43.2
0.648
0,270
0.348
0.409
0.201
BEVDepth [33]
53.8
41.8
BEVFormer' [34]
C
51.4
40.5
0.690
0.275
0.360
0.362
0.194
+BEVDistill [9]
L L->
52.4
41.7
+STXD (Ours)
L
54.3
44.0
0.635
0.264
0.365
0.309
0.198
UVTR-C [29]
C L
44.1
36.2
0.758
0.272
0.410
0.758
0.203
+L.2C [29]
45.0
37.2
0.735
0.269
0.397
0.761
0.193
+STXD (Ours)
L-C
46.1
39.0
0.698
0.265
0.413
0.755
0.203
UVTR-CS [29]
C
48.3
37.9
0.731
0.267
0.350
0.510
0.200
+L2CS [29]
L+C
48.8
39.2
0.720
0.268
0.354
0.534
0.206
+STXD (Ours)
L-C
50.8
41.4
0.662
0.265
0.381
0.484
0.199
Effect of on TD Loss. The temporal consistency distillation (TD) in Eq.4 utilizes the BEV features extracted from the past K frames. Tab.5 demonstrates increasing the value of K results in improved 3DOD performance, particularly in terms of mAVE (mean Average Velocity Error). By leveraging the information from consecutive frames through the TD loss, the student can ef- fectively learn the dynamics of target objects, leading to accurate 3
Table 5: Performance of the pro-
posed method depending on K.
K NDS(%)
mAP(%)
mAVEJ
54.06
43.48
0.3246
2
54.13
43.45
0.3148
54.31
44.03
0.3093
predictions of their dynamic attributes. This highlights the impor- tance referencing past frames, as it allows the teacher to transfer a greater amount of information to the student, resulting in improved 3DOD performance. Similar results have also been echoed in concurrent works [32, 41], which also leverage temporal information for the single-modal 3DOD.
Visualization of Temporal Similarity Map. The similarity map defined in Eq.3 calculates the element-wise similarity between the BEV features of the current and past frames. In Fig.4, we select the feature element corre- sponding the location of the car object in the current BEV space and calculate its sim- ilarity to the entire features of the past frames. Notably, the student model trained with TD exhibits improved similarity patterns with the teacher (see 4-a) and (b)). This indicates that the student model has learned to capture similar temporal relations and representations
(a) Teacher
(b) w/ TD
(c) w/o TD
Figure 4: Examples of temporal similarity maps for (a) teacher model, (b) student model trained with TD, and (c) student model trained without TD.
from the teacher by referring to the past frames. The competence of our temporal distillation approach ultimately contributes to the improved 3DOD performance, demonstrated in ablation studies (see Tab. and 5). We provide more temporal similarity visualization results in the Appendix.
4.3 Comparisons with Other Methods
To demonstrate the generalizability of our approach, we apply STXD to both UVTR [29] and BEVFormer [34] baselines. For a fair comparison, we use MONNet--11--CCS [18, 13] as the backbone of our student models, unless otherwise specified. Tab.4 shows the evaluation results the nuScenes

---

Table 6: Performance comparisons the nuScenes testing indicates implementation of BEV-
Former [34] with modified BEV grid size of 128x 128. ş indicates the use V2-99 [28] as backbone,
otherwise ResNet-101-DCN is used for fair comparison. In modality column, 'L', 'C', and 'L
represent LiDAR, Camera, and knowledge distillation from LiDAR to Camera, respectively.
Method
Modality
NDS(%)
mAP(%)
mATEJ mASEL
mAOE
mAVE, mAAEl
DETR3D [54]
C
47.9
41.2
0,641
0.255
0.394
0.845
0.133
PETR [33]
45.5
39.1
0.647
0.251
0.433
0.933
0.143
PolarFormer [24]
C
54.3
45.7
0.612
0.257
0.392
0.467
0.129
BEVFormer [34]
52.6
42.4
0.620
0.263
0.457
0.387
0.132
+STXD (Ours)
L- C
55.5
46.5
0.579
0.259
0.462
0.363
0.113
BEVFormer [34]
C
55.5
45.7
0.605
0.263
0.371
0.375
0.116
+STXD (Ours)
- C
58.3
49.7
0.546
0.256
0.383
0.356
0.117
UVTR-C [29]
43.0
36.4
0.724
0.266
0.486
0.898
0.143
+L2C [29]
L- C
44.0
38.2
0.677
0.262
0.493
0,925
0.150
+STXD (Ours)
L-C
45.8
40.2
0.634
0.271
0.465
0.918
0.143
UVTR-CS [29]
C
48.6
39.0
0.702
0.263
0.435
0.548
0.144
+L2CS [29]
L L-C
48.7
39.8
0.679
0.262
0.432
0.612
0.133
+STXD (Ours)
L-C
51.8
43.5
0.614
0.260
0.407
0.589
0.132
validation set, where STXD significantly improves the performance of the based student models, surpassing the performance of existing distillation methods [29, 9]. up to 2.9% NDS and 3.5% in mAP. In most cases, the addition of STXD contributes to the improvement of mATE (translation), mASE (scale), and mAVE (velocity) metrics. Both mATE and mASE are closely related to the geometric features of 3D objects, and the teacher effectively transfers such important information the student models. The improvement in mAVE can be attributed to our temporal distillation method.
Similar results echoed in experiments the nuScenes testing dataset, shown in Tab.6. We did not use CBGS [70] or test time augmentations to evaluate the sole effect of our method. STXD consistently improves NDS and mAP by up to 3.2% and 4.5%, respectively, compared the various baseline methods (UVTR-CICS, L2C/LLCS, BEVFormer") ). STXD also improves performances for the student model with heavy backbone (BEVFormer") by 2.8% in NDS and 4.0% in mAP. These results are significant as models with heavy backbones already achieve high performance, making it challenging to further improve through knowledge distillation. In the Appendix, competence of STXD on lightweight backbones as well. These results demonstrate the effectiveness and versatility of STXD in enabling cross-modal distillation between the teacher and the student models. Overall, STXD enables the student to learn the maximum amount of information from the teacher, including geometric and temporal knowledge, leading to significantly enhanced performance.
show the
5 Conclusion
In this paper, we presented STXD, a cross-modal knowledge distillation framework for the multi-view 3D object detection task. STXD is capable of effectively distilling structural and temporal knowledge across different modalities through Correlation Regularizing Distillation (CD) and Temporal Consis- tency Distillation (TD). To further enhance the distillation quality, STXD also adopts Response-Level Distillation (RD), which transfers task-specific knowledge at the output level. Our extensive experi- ments and ablation studies demonstrated the effectiveness of STXD on the nuScenes dataset, where the NDS and mAP of the based student detectors are improved by up to 3.2% and 4.5% on the testing set. Inspired by recent works on multi-mode fusion [36, 63, and other types of 3D perception tasks (e.g., BEV Segmentation [68, 20], Occupancy Detection [23, 30]). we plan to further explore the potential of STXD in such various tasks leveraging cross-modal distillation. It would also be valuable to apply and validate our approach with other models that utilize long-term information such as [17, 41]. We hope our work provides a solid baseline and new perspectives on structural and temporal cross-modal distillation.

---

Though the proposed method significantly improves the performance of multi- view camera-based based 3D object detection, it does not guarantee perfect predictions for all cases. Therefore, when applied to actual autonomous driving scenes, contingency plan must be included.
Societal Impact.
References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[2] Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: aaaaee---naarinnc--ovaaaaaaa regularization for self-supervised learning. In International Conference on Learning Representations, 2022.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijhom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IRFE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to End object detection with transformers. In Proceedings of the European Conferenee on Computer Vision, 2020.
[5] Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular object detection for autonomous driving. In Proceedings the JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[6] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, 2017.
[7] Yukang Chen, Yanwei Li. Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recogmition, 2022.
[8] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[9] Zchui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang. Qinhong Jiang, and Feng Zhao. BEVDistill: Cross-modal BEV distillation for multi-view 3d object detection. In International Conference on Learning Representations, 2023.
[10] Zehui Chen. Chenhongyi Yang. Qiaofei Li, Feng Zhao, Zheng Jun Zha, and Feng Wu, Disentangle your dense object detector. In Proceedings of the 29th ACM International Conference Multimedia, 2021.
[11] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. Monodis- till: Learning spatial features for monocular 3d object detection. In International Conference on Learning Representations, 2022.
[12] Xiaomeng Chu, Jiajun Deng, Yao Li, Zhenxun Yuan, Yanyong Zhang, Jianmin Ji, and Yu Zhang. Neighbor- vote: Improving monocular 3d object detection through neighbor distance voting. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
[13] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li. Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
[14] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel cnn: Towards high performance voxel-based 3d object detection. In Proceeding of Association for the Advancement of Artificial Intelligence, 2021.
[15] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recogaition, 2021.
[16] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang. and Hongsheng Li. Liga-stereo: Learning lidar geom- etry aware representations for vereo-based 3d detector. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
10

---

[17] Chunrui Han, Jianjian Sun, Zheng Ge Jinrong Yang. Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303.0520 2023.
[18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun, Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
[19] Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In Proceedings of the European Conference on Computer Vision, 2022.
[20] Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: future instance prediction in sird' view from surround monocular cameras, In Proceedings the IEEE/CVF International Conference on Computer Vision, 2021.
[21] Junjic Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du Bevdet: righ-perffrmance multi-camera 3d object detection in bird-eye-view. arXiv preprint 1xXiv:2112.11722 2021.
[22] Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu. Tig- bew: Multi-view bev 3d object detection via target inner-geometry learning. arXiv preprint arXiv:2212. 13999, 2022.
[23] Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspeetive view for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2302.07817, 2023.
[24] Yangin Jiang. Li Zhang. Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu- fu-Gang Jiang. Polar- former: Multi-camera 3d object detection with polar transformers. In Proceeding of Association for the Advancement of Artificial Intelligence, 2023.
[25] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal gen- eration and object detection from view aggregation. In IEFE/RSJ International Conference on Intelligent Robots and Systems, 2018.
[26] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. 2(1-2):83 97, 1955.
[27] Alex H Lang. Sourabh Vora, Holger Caesar. Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[28) Youngwan Lee, Joong-wor won Hwang, Sangrok Lee, Yuseok Bac, and Jongyoul Park. An energy and gpu- computation efficient backbone network for real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.
[29] Yanwei Li, Yilun Chen, Xiaojuan Qi. Zeming Li. Jian Sun, and Jiaya Jia. Unifying voxel-based represen- tation with transformer for 3d object detection. In Advances in Neural Information Processing Systems. 2022.
[30] Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-basse 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[31] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng. Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V Le. al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recogmition, 2022.
[32] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. In Proceeding of Association for the Advancement of Artificial Intelligence, 2023.
[33] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Procceding of Association for the Advancement of Artificial Intelligence, 2023.
[34] Zhiqi Li, Wenhai Wang. Hongyang Li. Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bev- former: Learning bird' s-eye-view representation from multi- images via spatiotemporal transformers. In Proceedings of the European Conference Computer Vision, 2022.
11

---

[35] Yingfei Liu. Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In Proceedings of the B Conference on Computer Vision, 2022.
[36] Zhijian Liu, Haotian Tang. Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L Rus, and Song Han. Bev- fusion: Multi-task multi-sensor fusion with unified seee view representation. IEEE International Conference on Rohotics and Automation, 2023.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[38] Shujie Luo, Hang Dai, Ling Shao, and Yong Ding. M3dssd: Monocular 3d single stage object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[39] Xinzhu Ma. Yinmin Zhang. Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[40] Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
[41] Jinhyung Park, Chenfeng Xu, Shijia Yang. Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka. and Wei Zhan. Time will tell: New outlooks and baseline for temporal multi-view 3d object detection. In International Conference on Learning Representations, 2023.
[42] Youngmin Park, Vincent Lepetit, and Woontack Woo. Multiple 3d object tracking for augmented reality. In IEEE/ACM International Symposium on Mixed and Augmented Reality, 2008.
[43] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.
[44] Rui Qian, Xin Lai, and Xirong Li. 3d object detection for autonomous driving: a survey. Pattern Recognition, 130: 108796, 2022.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine learming, 2021.
[46]
Kavya Ravichandran, Ajay Jain, and Alexander Rakhlin. Using effective dimension to analyze feature transformations in deep neural networks. In ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.
[47] Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021.
[48] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang. Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renm: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[49] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattemn Recognition, 2019.
[50] Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Peter Kontschieder. and Elisa Ricci, Are we missing confidence in pseudo-lidar methods for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[51] Chunwei Wang, Chao Ma, Ming 7hu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IERE/VF Conference on Computer Vision and Pattem Recognition, 2021.
[52] Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Proceedings of the Conference on Robot Learning, 2022.
[53] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
12

---

[54] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queriees. In Proceedings of the Conference on Robot Learning, 2022.
[55] Yue Wang and Justin M. Solomon. Object dgenn: 3d object detection using dynamic graphs. In Advances in Neural Information Processing Systems, 2021.
[56] Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: bridging the beam- induced domain gap for 3d object detection. In Proceedings of the European Conference on Computer Vision, 2022.
[57] Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin. Zhou Bin, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In IEEE International Intelligent Transportation Systems Conference, IEEE, 2021.
[58] Yan Yan. Yuxing Mao, and Bo Li. 18(10):3337, 2018
Second: Sparsely embedded convolutional detection. Sensors,
[59] Chenhongyi Yang. Mateusz Ochal, Amos Storkey, and Elliot J Crowley. Prediction-guideer distillation for dense object detection. In Proceedings of the European Conference on Computer Vision, 2022.
[60] Jihan Yang, Shaoshuai Shi, Runyu Ding. Zhe Wang. and Xiaojuan Qi. Towards efficient 3d object detection with knowledge distillation. In Advances in Neural Information Processing Systems. 2022.
[61] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia, 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[62] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Spanse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[63] Zeyu Yang, Jiaqi Chen. Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. In Advances in Neural Information Processing Systems, 2022.
[64] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[65] Zhiyun Yin and Yujie Li. Overview of robotic grasp detection from 2d to 3d. Cognitive Robotics, 2022.
[66] Yurong You, Yan Wang. Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Camphell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference on Learning Representations, 2020.
[67] Jure Zbontar. Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference Machine Learning, 2021.
[68] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic seg- mentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. 2022.
[69] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, 2018
[70] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arv:: MwXiv:1908.000 1908.0949,, 2019.
[71] Menglong 7hu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mahel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis, Single image 3d object detection and pose estimation for grasping. In IEEE International Conference on Robotics and Automation, 2014.
[72] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable (detr): Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.
13

---

Appendix
In this appendix, we provide additional details of the implementation and training of our method (Appendix A). We also present additional experimental results, including ablations on lightweight backbones, additional baselines, and qualitative examples (Appendix B). These results further demon- strate the effectiveness of our STXD framework in cross-modal distillation for 3D object detection.
A
Additional Implementation Details
A.1 Details on Cross-Modal Distillation Baselines
In Sec.4.1 of the manuscript, we introduced two cross-modal distillation baselines for 3D object detection: (1) BEVFormer [34] the student model and Object-DGCNN [55] as the teacher model; and (2) UVTR CC/CC as the student models and UVTR-L as the teacher model [29]. To be more specific, in the first baseline (Object-DGCNN + BEVFormer), bird's-cye view (BEV) feature to be models, indicating flat BEV feature space. In accordance with the concurrent work [9]. we replaced the DGCNN attention module of the transformer-decode attention. Apart from these modifications, student and teacher models. In the second baseline (UVTR-L + UVTR-CICS), we utilized the original implementation of UVTR [29] without any modifications. The dimensions of the bird's view (BEV) features were set to X = Y = 128 and Z = 11 for UVTR-C/CS; and = 5 for UVTR-L. Here, Z represents the height of the Bird's Eye View (BEV) feature space. The BEV features were further sampled using the (4r, y, z) positions of object queries Q, where [Q] 900.
set the spatial dimensions of the Y = 128, and Z = for both the teacher and student
in the teacher model with vanilla multi-heal followed the original implementations of both the
Note that we adopted two bascline student models that utilize multi-frame inputs. BEVFormer generates the BEV feature of the current frame by referring to the BEV feature of the past frames via temporal self-attention mechanism. In UVTR-CS, multi-frame images are concatenated and forwarded to the image encoder.
A.2 Training Details
For a fair comparison, we used ResNet-101-DCN pretrained by FCOS3D [53] as the backbone network for the student models in extensive experiments (see Sec.4.2 and 4.3 in the manuscript) following prior works [34, 29]. We also showcased the competence of our STXD framework with other types of backbones, including Vo [28] initialized from DD3D [40] well as lightweight ResNet-18 88 and ResNet-50 networks (see B. I)).. We set the balancing parameter to 0.8 and 0.6 for classification and localization in Eq.5 of the manuscript, following [10, 59]. The weights for each loss term in Eq.7 of the manuscript are set to (0.1, 100.0, 1.0, 1.0), respectively. in the order mentioned. The student models were trained for 24 epochs using learning rate of 2 x 10-4, and a batch size of GPU. All models were trained on 8 of NVIDIA A100 GPU while following the original codebase for each model. We employed Adam W [37] as the optimizer with a weight decay of 1 x 10 2 To isolate the effect of our STXD framework in cross-modal distillation, we deliberately did not use the CBGS strategy [70] or test-time augmentation in our experiments.
A.3 Instance Matching Function in Response-Leve Distillation
In Sec.3.4, we adopted the commonly used Hungarian algorithm [26] to define set-to-set matching between the ground-truth and candidates from the teacher and the student, separately. Then, we constructed a mapping function π (f)) between candidates from the teacher and the student based on matched ground-truth indices.
14

---

B Additional Experimental Results
B.1
Ablations on Lightweight Backbones
To further validate the effectiveness of our STXD framework in practical scenarios where lighter backbone networks are preferred for deployment on edge devices, we conducted experiments using smaller backbone networks for the student model. This allows us to assess the performance of our approach under resource-constranee settings while improving the level of accuracy in 3D object detection. We trained BEVFormer with ResNet-18 and ResNet-50 backbone networks on nuScenes training set and evaluated on the validation set. The results in Tab.7 demonstrate that STXD significantly improves the performance of BEVFormer with smaller backbone networks where it achieves an improvement 3. 24% and 4.00% in NDS and 3.33% and 4.49% in mAP, respectively. These improvements are consistent with the results presented in Tab.4 and 6 of the manuscript, further confirming the versatility and compatibility of our approach across different backbone architectures.
Table 7: Performance of STXD on lightweight backbone networks for the student model.
Method
Backhone
NDS(%)
mAP(%)
BEVFormer [34]
ResNet-18
40.42
28.70
+STXD (Ours)
ResNet-18
43.66
32.03
BEVFormer [34]
ResNet-50
44.06
32.86
+STXD (Ours)
ResNet-50
48.06
37.35
B.2 Experiments on Additional Teacher-Student Baseline
To further validate the effectiveness of approach across various teacher-student basclines, extend our experiments to additional student baseline, BEVDepth [33]. which relies on LSS-based feature generation [43], depth supervision, and multi-frame inputs. Here we followed the official implementation of TiG-BEV [22] for cross- moddl distillation with CenterPoint [64] as the teacher model, but added our feature-level distillation losses for fair comparisons. Results reported in Tab.8, where our approach outperforms prior distillation methods, such as BEVDistill [9] and TIG-BEV [22], by up to +3.2% of NDS. These results are consistent the results from BEVFormer and UVTR-CICS baselines in Tab.4 and 6.
Table 8: Performance comparison of distillation methods on the baseline with CenterPoint as teacher and BEVDepth as the student. TiG-BEV (FD): applying only the feature-level distillation (FD) losses (Ciner-hhannel Linter-koypoint................) TiG-BEV: leveraging full set of losses including inner-depth supervision proposed in TiG-BEV. STXD (CD): applying correlation regularizing distillation loss. STXD (TD): applying our temporal consistency distillation loss.
Method
Backbone
Modality
(D)(%)
mAP(%%
CenterPoint [64]
VoxeINet
L
64.6
56.4
BEVDepth [33]
ResNet-50
43.1
32.9
+BEVDistill [9]
ResNet-50
L-C
45.4
33.2
+TiG-BEV (FD) [22]
ResNet-50
L-CC
45.2
35.8
+TiG-BEV [22]
ResNet-50
L-C
46.1
36.6
+TXD (CD)
ResNet-50
L-CC
48.4
37.1
+STXD (TD)
ResNet-50
L-C
48.3
36.1
B.3 Comparison to Correlation Distillation Method
Our Correlation Regularizing Distillation (CD) and TiG-BEV [22] both consider dimensional cor- relation of features. However, these methods are distinct their underlying objectives and imple- mentations. TiG-BEV proposes the inter-channel BEV distillation loss (IC loss) that encourages the student's features to replicate the dimensional self-correlation of the teacher's features (i.e, minimize | T F GT Gll2 ). In contrast, the core contribution of CD is introducing cross-modal
15

---

cross-correlatio regularization to prevent student features from being redundant. Specifically, CD exploits the decorrelation mechanism (the second term of Eq.2) to prevent information collapse in the student model. However, IC loss does not utilize such an information maximization mecha- nism, and only relies on the global one-to-one MSE distance (1.2-norm) between modality-specific self-correlation matrices. Hence, CD pursues a different learning principle compared to TiG-BEV.
These differences lead the competence of CD over TiG-BEV. Tab.9 shows that CD effectively reduces duplicate dimensional feature components compared to the correlation-base distillation of TiG-BEV. Furthermore, Tab. 8, CD ultimately achieves performance improvements over TiG-BEV and BEVDistill by up to +3.2% of NDS and +3.9% of mAP. These results consistently demonstrate the significance of mitigating feature redundancy cross- cross-modal knowledge distillation and enhancing feature quality through our feature-lever distillation approaches.
Table 9: Validation for correlation regularization effect of CD loss. For both TiG-BEV and our STXD, we measured the effective dimension (det) and dimensional redundancy values defined in Sec.4.2. TiG-BEV (FD) represents the results of applying only the feature distillation loss from TiG-BEV.
Method
den't
dart 1
Dem.Red.4
TiG-BEV (FD) [22]
1.620
2.634
0.174
STXD (CD)
1.694
2.871
0.062
Teacher [33]
2.194
4.813
0.038
B.4
Qualitative Results
B.4.1 Prediction Results on BEV Space
Fig.5 and 6 provide examples of 3DOD results in the BEV space from the LiDAR-based model (top- left), the multi-view camera-based model trained with our STXD) (top-middle), and the camera-based model trained without knowledge distillation (camera-only model, top-right). Additionally, at the bottom of each example, we have included camera views that correspond to the highlighted regions, which can help in better understanding the results. Overall, STXD enables more accurate prediction of bounding box locations, scales, and orientations compared to the camera-only model (highlighted by red dashed boxes). This observation consistent with the quantitative results Tab.4 and 6 of the manuscript. STXD also leads to reduction of false positives. In certain examples, observe decrease in false positives even when comparing the results to the LiDAR- bbased model (highlighted by pink dashed boxes). Notably, in Fig.6, STXD effectively reduces false positive detections in 3DOD compared to the LiDAR -ased model and the camera-only model. These results indicate that our STXD framework effectively transfers valuable information from the LiDAR- teacher the camera student, leading to improved detection performance.
B.4.2 Qualitative Comparison BEV Features
As an extension of Fig.1 in the manuscript, we provide additional examples of BEV features learned using different feature-level distillation methods in Fig.7. In various samples, the model trained with the correlation regularizing distillation (CD) consistently generates more similar feature patterns (Fig.7-(b)) to the teacher model (a)) compared to other methods (Fig.7-(c) and (d)). These results demonstrate the effectiveness of the proposed method in transferring the rich information contained in the teacher model to the student model.
B.4.3 Temporal Similarity Maps
As extension of 4 in the manuscript, Fig.8 illustrates the learning progress of the temporal
similarity map during the training phase for different values of k (k = 1,2, 3). As the training progresses, the cross-modal temporal similarity map S(-*) becomes similar to the intra-modal temporal similarity T-k)), This indicates that the student model effectively learns the temporal knowledge encoded in the similarity maps from the teacher, and ultimately leads to significant improvements in 3DOD, as demonstrated in our extensive experiments reported in the manuscript.
16

---

LiDAR
Ours
Caeau eioo
Camera
GT
LIDAR
ups
ups
Ours
Camera
ups
Ups
LiDAR
Ours
Camera
LIDAR
Ours
Camera
Figure 5: Examples of 3D object detection results from LiDAR-based model (top-left), multi-view camera-based model with STXD (top-middle), and camera-only model (top-right). Green rectangles indicate ground truth bounding boxes, while blue rectangles represent the predicted bounding boxes. Pink dashed boxes highlight the comparisons of STXD with the LiDAR-bass model, while red dashed boxes highlight the comparisons with the camera-only model. We also provide camera views corresponding to the highlighted regions the bottom.
17

---

LiDAR
Ours
Camera
GT
LIDAR
Ours
Camera
LiDAR
Ours
Camera
GT
LIDAR
Ours
Camera
Figure 6: Examples of 3D object detection results from LiDAR-based model (top-left), multi-view camera-based model with STXD (top-middle), and camera-only model (top-right). Green rectangles indicate ground truth bounding boxes, while blue rectangles represent the predicted bounding boxes. Pink dashed boxes highlight the comparisons of STXD with the LiDAR-basee model, while red dashed boxes highlight the comparisons with the camera-only model. We also provide camera views corresponding to the highlighted regions at the bottom.
18

---

(a) LiDAR
(b) Ours
(c) MSE w/ GT
(d) MSE
(e Camera
Figure 7: Examples of BEV features generated by (a) the LiDAR-based model, (b)-(d) the multi-view baser model trained with different feature-level distillation methods, and (e) the camera-only model. We visualize the norm value of BEV features.
19

---

0 epoch
epoch
24 epoch
S(-3)
2
s(-1)
(-3)
(-2
T(1)
Figure 8: Examples of the learning progress for the temporal similarities during the training phase.
20


Reviews:
[{"review_id": "PqUKKwHRVv", "reviewer": "Reviewer_4Udp", "summary": "The authors propose a new method for LiDAR-to-Camera 3D object detection model distillation. They propose distillation along three avenues. First, referencing Barlow Twins, they consider redundancy reduction through cross-correlation between LiDAR and Camera features. Second, they enforce consistency between temporal similarity maps of current lidar-past lidar and current camera-past lidar. Finally, they additionally have distillation at the output level.", "questions": "- L138: Why is N = X*Y*Z? It instead seemed like N should be X*Y, the flattened BEV feature map. Further, is this done over all the samples in a batch or separately for each BEV teacher-student pair?", "limitations": "Discussed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is clear and easy to read, with informative diagrams.\n- The motivation for the distillation approaches is sound. I especially appreciate the temporal similarity distillation, as well as its improvement on mAVE.\n- Step-by-step ablations demonstrate the improvement from each proposed component. \n- Analyses in pg 7 demonstrate the redundancy reduction of the core distillation method.\n", "weaknesses": "- Additional ablations on Temporal Consistency Distillation would be helpful. For instance, perhaps a more natural way to do this could be to generate temporal similarity maps independently for LiDAR and Camera (current lidar x past lidar, current camera x past camera), and enforce consistency between those instead. It is not entirely clear why current camera x past lidar is done instead.\n- Discussion and head-to-head comparison with TiG-BEV with settings shown in the TiG-BEV paper is strongly recommended.\n- As the proposed method considers inter-channel relationships, discussion of the proposed method’s relationship to works that use inter-channel distillation (such as in TiG-BEV) would be appreciated. \n- How are teacher and student outputs matched for the response distillation? Is it greedy, hungarian, or something else?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We express our gratitude to the reviewer \"4Udp\" for recognizing our fundamental contributions to feature-level distillation approaches, particularly highlighting the significance of Temporal Consistency Distillation, and acknowledging our thorough validation results on a component-wise basis. Below, we provide answers to your questions and hope they appear to be reasonable as well.\n\n**Design Rationale behind Temporal Consistency Distillation (TD):**\\\n\"4Udp\" suggested an ablation of different strategies for temporal similarity map generation, specifically focusing on modality-independent associations like \"current LiDAR ($F_0$) x past LiDAR ($F_k$), current camera ($G_0$) x past camera ($G_k$)\". In our early design experiments, we also tested the same similarity map generation strategy (i.e., $L_{TD}=D_{KL}(S || T)$ where $S=G_0 G_k^{T}, T=F_0 F_k^{T}$). However, this formulation did not result in a significant improvement in detection performance as we had anticipated. Please note that, in our cross-modal knowledge distillation setting, the LiDAR teacher model can provide informative and high-quality features across temporal frames, leveraging its pre-trained weights. However, the image features from past frames, generated via the student model, offer relatively unreliable and uninformative guidance compared to the teacher model. Consequently, relying on these untrustworthy past image features may even hinder the progressive learning of informative features from the teacher model. Thus, we devised the current formulation of $L_{TD}$, where the LiDAR teacher is primarily utilized to generate features from past frames and guide the student model, as presented in Eq.3 and 4. Consequently, TD led to significantly improved prediction performance in our component-wise analyses, as shown in Tab.1.\n\n**Comparison with TiG-BEV [R3]:**\\\nAs pointed out by \"4Udp\", our cross-correlation regularizing distillation (CD) and TiG-BEV both take into consideration dimensional (inter-channel) correlation. However, we want to emphasize that these losses are distinct in terms of their underlying objectives and specific implementations.\n\nAccording to our understanding of TiG-BEV, the inter-channel BEV distillation loss (\"IC loss\") encourages the student's features to replicate the dimensional **self-correlation** ($D \\times D$ matrix, $D$=the dimension of BEV features) of the teacher's features. However, IC loss does not consider cross-modal associations among BEV features from the teacher and student models.\n\nIn contrast, the core contribution and novelty of CD is introducing **cross-modal cross-correlation regularization** to prevent student features from being redundant. Inspired by recent info-max oriented self-supervised learning approaches (e.g., Barlow Twins [R1] and VICReg [R2]), CD exploits the **decorrelation mechanism** (the second term of Eq.2) which prevents information collapse in the student model. CD also enhances the dimensional component-wise feature similarities (the first term of Eq.2). Please refer to L43-L48 and L142-L145 for detailed explanations. As \"4Udp\" also remarked, our extensive analyses presented in Tab.3 and Fig.3 prove the effectiveness of CD in terms of eliminating redundant information in the features of the student, and ultimately contributes to significantly improved prediction performance. However, TiG-BEV does not have such information maximization mechanism, and only relies on the global one-to-one MSE distance (L2-norm) between modality-specific self-correlation matrices. Hence, CD pursues a different learning principle compared to IC loss of TiG-BEV. \n\nFor a clear comparison, we also have illustrated the differences in loss calculation in Fig.R2 of the rebuttal PDF. Again, one notable distinction is that TiG-BEV computes self-correlation for each modality separately (Fig.R2-(b)), while our CD loss calculates cross-correlation between LiDAR and camera features (Fig.R2-(a)).\n\nLast but not least, we provide an additional quantitative comparison with TiG-BEV using BEVDepth [R11] as a baseline student model. Tab.R1 of the rebuttal PDF shows that both CD and TD outperforms TiG-BEV (correlation-based distillation losses, FD), by achieving notable improvements of +3.2% and +3.1% in terms of NDS, respectively. It is noteworthy that CD achieves superior performance, even when compared with TiG-BEV leveraging additional inner-depth supervision (+2.3% of NDS). In addition, we also measure the effective dimension $d_{\\text{eff}}$ and dimensional redundancy to compare the redundancy in student features learned from the teacher in Tab.R2. These results consistently demonstrate the significance of mitigating feature redundancy in cross-modal knowledge distillation and enhancing feature quality through our feature-level distillation approaches. We will include these additional results in the final version of the Appendix.\n\n**Further Clarifications**\n- We adopted the commonly used Hungarian algorithm [R14] to define set-to-set matching between the ground-truth and candidates from the teacher and the student, separately. Then, we constructed a mapping function (i.e., $\\pi(j)$ in Eq.6) between candidates from the teacher and the student based on matched ground-truth indices. We will better explain this in Sec.3.4 in the final version.\n- We define a shape of BEV feature as $N = X \\cdot Y \\cdot Z$, using a generalized notation where $Z=1$ for BEVFormer [R4] and $Z=11$ for UVTR-C/CS [R5], following the official implementation of each method (see Sec.A.1 of the Appendix). Also we did not combine the BEV teacher-student pairs across batches; instead, we handled them separately."}, {"review_id": "KHLysVpmxH", "reviewer": "Reviewer_1M3u", "summary": "This paper introduces a cross-modal distillation framework named STXD, which addresses existing distillation problems from both structural and temporal perspectives. Structure-based distillation imposes constraints on the similarity matrix of the student and teacher, ensuring that corresponding positions have a higher similarity. Temporal distillation is accomplished by distilling the feature similarity matrix over time. Ultimately, response-level distillation is employed to further improve the performance. Experiments on the nuScenes dataset demonstrate that the proposed approach outperforms previous distillation techniques.", "questions": "1. The notion of 'inherent structural knowledge' has not been succinctly explicated. For example, whether it refers to the 'instance'. It appears no relation with the term 'structure'. \n2. As described in the weakness, the authors should give a reasonable explanation for the motivation of structural distillation and temporal distillation. \n3. More experiments are needed to prove the effectiveness of the method.", "limitations": "The authors have already provided the limitations in the paper.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "+ The proposed cross-modal distillation from LiDAR to image works well (a marked improvement ranging from 2.8% to 4.5%). \n+ Experiments illustrate that the proposed approach outperforms previous approaches on the nuScenes dataset. \n+The paper is well written with a clear description of the method and comprehensive component analysis.", "weaknesses": "- The concept of 'structural distillation' seems no distinction from existent Mean Squared Error (MSE) distillation. It applies a weight of 0.01 to different positioning areas, hence, the leading role is still played by the loss at corresponding positions, which essentially substitutes MSE with a measure of similarity. \n- The structural distillation appears not reasonable. In fact, spatially adjacent tokens should have some correlation. In Fig.2, enforcing the correlation of corresponding positions to 1, and other positions to 0 is unjustified. Additionally, correlation is unable to respond to the redundancy of features, so Fig.3 is void of significance.\n- The paper employs an abundance of similarity matrices, resulting in substantial consumption of memory during the training process. I have noticed that BEVFormer uses only a 128x128 Bird's Eye View (BEV) resolution in the experiments. Whether the computational constraints caused this deviation from the default settings of BEVFormer.\n- The motivation for temporal distillation is insufficient. Theoretically, both the teacher and student features of the current frame contain temporal information. Thus, it brings into question the need to formulate the distillation approach in this manifestation.\n- It is better to conduct distillation based on stronger baselines such as BEVDepth, following BEVDistill. Table 2 ought to incorporate some comparison with prevalent distillation methods in 2D detection, such as GID. Techniques such as CLIP are seldom used. For the generality of distillation, the distillation on the Waymo dataset is indispensable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer \"1M3u\" for taking time and effort to review our work. We also appreciate the reviewer's recognition of the competence of our knowledge distillation approach, clear presentation, and comprehensive component-wise analyses. Below, we provide responses to your concerns, and we hope they are considered reasonable and acceptable.\n\n**Structural Distillation through Correlation Regularization (CD):**\\\nFirst and foremost, we clarify the motivation and concept underlying our Structural Distillation method. As shown in Fig.1-(a) and (e), different modalities exhibit distinctive feature distributions. However, MSE-like simple cross-modal distillation losses (e.g., L2-Norm) primarily focus on position-wise spatial correspondence and do not take into account dimension-wise feature structures (e.g., the distribution of dimensional feature space). Consequently, such MSE-like losses fail to adequately transfer the structure of the feature space (i.e., the “structural knowledge” inherent in modality-specific feature space), as shown in Fig.1-(c) and (d). To address these challenges, we propose a new structural distillation method that transfers the knowledge of dimension-wise feature structures (L36-L48).\n\nTo our best understanding, however, “1M3u” construed our structural distillation approach as one of the position-wise one-to-one distillation methods. This is a grave misunderstanding of our approach and its contributions. In addition, **please note that the “Dimensional Cross-Correlation” matrix shown on the left of Fig.2 has a shape of $D \\times D$, where $D$ represents the dimension of BEV features, and each element does not denote the spatial position in the BEV space.**\n\nThe core of CD is the **decorrelation mechanism** [R1,R2], which helps to reduce redundancy among dimensional feature components. Consequently, our structural distillation method prevents information collapse in the student model (Tab.3 and Fig.3) and enhances the dimensional feature similarities, as shown in Fig.1-(a) and (b).  It's noteworthy that other reviewers, “4UdP” and “XsGg”, also acknowledged the effectiveness of the CD loss in reducing duplicated information within the student features.\n\n**Temporal Consistency Distillation (TD):**\\\nOur baseline student models (BEVFormer [R4], UVTR-CS [R5]) leverage intra-modal temporal features and have shown improved prediction performance compared to single-frame learning. Expanding on the use of temporal information, we propose to further integrate temporal information across different modalities through our Temporal Consistency Distillation (TD). Note that TD exhibits significantly improved prediction performance in our comprehensive component-wise analyses (Tab.1 and 5, Fig.4) as well as comparisons with prior approaches (Tab.4 and 6). To the best of our knowledge, we are the first to offer such new perspectives on temporal cross-modal distillation for 3D object detection, a recognition also shared by other reviewers, “U3eQ”, “X2eU”, and “4Udp”. Hence, we believe that our temporal distillation approach brings a notable contribution to the relevant research fields.\n\n**BEV Feature Shape:**\\\nRegarding the modified BEV feature shape parameters (i.e., (200 x 200) $\\rightarrow$ (128 x 128)) for BEVFormer [R4], our intention was not to manage GPU memory consumption. Instead, we aimed to maintain consistency with BEVDistill's implementation [R6] for **fair comparisons and to maintain alignment between the feature shape of LiDAR and camera models**.\n\n**Additional Validation on BEVDepth [R11]:**\\\nWe conducted thorough experiments and comprehensive ablation studies on representative student baselines, including BEVFormer (attention-based model, [R4]) and UVTR (LSS-based model, [R5]), using the nuScenes dataset (see Sec.4). We also compared our method against state-of-the-art distillation methods for 3D object detection. As noted by the majority of other reviewers, these results have extensively and sufficiently validated the effectiveness of our STXD framework. In response to the concerns raised by \"1M3u\", we extend our experiments to additional baseline student model, BEVDepth [R11], which relies on LSS-based feature generation, depth supervision, and multi-frame inputs. We report results in Tab.R1 of the rebuttal PDF, where our approach outperforms prior distillation methods, such as BEVDistill [R6] and TiG-BEV [R3], by up to +3.2% of NDS. These results are consistent to the results from BEVFormer and UVTR-C/CS baselines (see Tab.4 and 6).\n\n**Comparison with Prior Distillation Approaches:**\\\nUntil recently, MSE and its variant with focusing on regions with ground-truth (MSE w/ GT) have been commonly adopted in state-of-the-art cross-modal distillation for object detection [R6-R9]. To our best understanding, General Instance Distillation (GID, [R12]) is also conceptually similar to \"MSE w/ GT\" concerning its sampling strategy around the foreground objects. Based on this observation, we compared our approach with such prevalent methods (MSE, MSE w/ GT) in Tab.2.\nAs mentioned above, our correlation regularizing distillation (CD) drew inspiration from self-supervised learning methods [R1,R2]. The lower section of Tab.2. presents comparisons of various learning strategies, including CLIP [R13], VICReg [R2], Barlow-Twins [R1], to validate our design of CD loss (L236-L239).\n\n**Additional Validation on Waymo Datatset:**\\\nGenerally, the method that performs well on the nuScenes dataset often achieves favorable outcomes in the Waymo dataset, as observed in [R4,R15]. Therefore, we believe our extensive validation and testing results on the nuScenes sufficiently demonstrate the effectiveness of our distillation approach. To accommodate your suggestion, we are conducting experiments on the Waymo dataset, but we cannot report the results from such a large-scale dataset (230K frames over 1TB) within the short rebuttal period. We will report results in the final version of our paper."}, {"review_id": "qgRMXNbURv", "reviewer": "Reviewer_XsGg", "summary": "This paper proposes a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3D Object Detection. STXD leverages the Correlation Regularizing Distillation loss to address the distributional divergences between non-homogeneous modalities. Besides, it effectively distills temporal information contained in previous frames of the teacher and utilizes response-level distillation to further improve the quality of knowledge distillation at the prediction-level. Extensive experiments on nuScenes show that STXD significantly improves the NDS and mAP of the base student detectors.", "questions": "1. The paper does not clearly explain how to utilize sweep images in the student models. Additionally, it's worth noting that most of the camera-based baseline models seem not to use multi-frames, as evidenced by the relatively high mATE (>0.6) and mAVE (>0.5) values.\n2. Could the authors clarify why CBGS is not employed, given its potential to facilitate full model training?\n3. Could the authors offer the results on the nuScenes test sets? (Leaderboard web: https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Camera)", "limitations": "There are still many bad cases when using camera based 3D object detection. Achieving high performance with most of the existing methods often requires significant computational resources, such as training on 8 NVIDIA A100 GPUs.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. The paper provides valuable insights into addressing distributional divergences between non-homogeneous modalities through the use of the Correlation Regularizing Distillation loss.\n2. The effectiveness of the Correlation Regularizing Distillation loss is well demonstrated through ablation studies.\n3. The writing is clear and the paper is easy to follow.", "weaknesses": "1. The novelty of Response-Level Distillation (Sec 3.4) is limited, as a very similar method is proposed in BEVDistill [1].\n2. The effectiveness of Temporal Consistency Distillation is somewhat uncertain. Many previous works demonstrate the effectiveness of using multi-frames [2-4], and their methods are relatively simple (such as concatenating the temporal BEV features) yet effective in achieving state-of-the-art performance. However, this paper does not include experiments to demonstrate the effectiveness of Temporal Consistency Distillation in models that utilize multi-frames. If the authors can provide these experiments, I may raise my rating.\n3. While the insight of using the Correlation Regularizing Distillation loss to address distributional divergences is valuable, it should be noted that the authors do not propose the Correlation Regularizing Distillation loss itself.\n\n[1] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. BEVDistill: Cross- modal BEV distillation for multi-view 3d object detection. ICLR 2023.\n[2] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, Xiangyu Zhang. Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perception. \n[3] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, Wei Zhan. Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection.\n[4] Junjie Huang, Guan Huang. BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the acknowledgment from reviewer \"XsGg\" for recognizing \"effectiveness\" and \"valuable insights\" of our non-homogeneous cross-modal knowledge distillation framework, particularly the use of Correlation Regularizing Distillation (CD) as well as our comprehensive ablation studies. Below, we provide answers to your questions to further address your concerns.\n\n**Response-Level Distillation (RD):**\\\nAs pointed out by \"XsGg,\" we are also aware of that similar approaches to our response-level distillation have been introduced in prior works [R6-R9]. **However, please note that we acknowledge that and do not claim the introduction of RD as our main contribution**. Rather, we explicitly indicated our \"adoption\" and \"inspiration\" from such output distillation strategies to further enhance our knowledge distillation performance (L59-L61, L167-L168). Please also note that **the improvement gained from our core feature-level distillation methods (CD,TD) is larger than RD** (i.e., gain of NDS(%) and mAP(%) over based student model BEVFormer [R4]: RD=1.44% and 1.56%, TD=1.77% and 2.15%, and CD=2.47% and 2.75%). Moreover, **STXD without RD already outperforms BEVDistill [R6]** (STXD w/o RD=54.0% of NDS and 43.4% of mAP in Tab.1 vs. BEVDistill=52.4% of NDS and 41.7% of mAP in Tab.4). Hence, we can consider RD as an add-on method for improved cross-modal knowledge distillation, given its widespread use in most state-of-the-art distillation methods for object detection [R6-R9].\n\nTo draw a clear comparison with BEVDistill, our output distillation formulation is primarily inspired by the quality-score approaches used in output-guided knowledge distillation for 2D object detectors [R7,R9] (L171-L177). In contrast, BEVDistill employs a critic network to approximate distributional similarities in output-level features between teacher and student models. Hence, technically speaking, RD is different from BEVDistill.\n\n**Temporal Consistency Distillation on Multi-Frame Models:**\\\nIn our experiments, we adopted two baseline student models that utilize multi-frame inputs. In UVTR-CS [R5], multi-frame images are concatenated and forwarded to the image encoder. BEVFormer [R4] generates the BEV feature of the current frame by referring to the BEV feature of the past frames via temporal self-attention mechanism. We followed the implementation of each student models and incorporated the past three frames in cross-modal distillation learning. As shown in Tab.1, 4, 5, and 6, TD demonstrated its effectiveness when it is incorporated with student models that utilize multi-frame  (i.e., BEVFormer and UVTR-CS) as well as single-frame (i.e., UVTR-C). In the final version, we will provide further clarification regarding the implementation details of the student models in Sec.4.1.\n\n**Correlation Regularizing Distillation (CD):**\\\nWe indeed did not create the decorrelation mechanism itself [R1,R2] in current submission. However, it also should be noted that we are the first to bring such self-supervised representation learning approach into the field of cross-modal distillation and 3D object detection based on a strong motivation (L36-L48). As “XsGg” also agreed with, our component-wise validation provides informative and valuable insights in this direction. We hope the follow-up research considers to use our CD loss instead of MSE-like loss functions based on the new perspectives and proof of effectiveness of CD presented in our extensive experiments.\n\n**Further Clarifications:**\n- Above, we have explained how our baseline student models (BEVFormer [R4], UVTR-CS [R5]) utilize multi-frame inputs. Additionally, for mATE and mAVE metrics, applying STXD (especially for TD in Tab.1 and Tab.5) results in an overall enhancement compared to the baselines and other KD methods (Tab.4 and 6). It is noteworthy that most reviewers, \"4Udp\", \"X2eU\", and \"U3eQ\", also noted the effectiveness and novelty of our temporal cross-modal distillation approach.\n- Regarding CBGS [R10], which mainly focuses on a sampling strategy for training, it has limited relevance to knowledge distillation. To isolate and specifically validate the effectiveness of KD methods without any potential compounding effects, we did not utilize CBGS during training (L326). Furthermore, applying CBGS to our baseline model, BEVFormer, led to an extension of 1 epoch to 4.5 epochs, resulting in the entire training time exceeding two weeks even on NVIDIA A100 GPUs. Therefore, we find it impractical and challenging to apply CBGS to our baseline student models, making it difficult to ensure reproducibility.\n- Tab.6 presents the evaluation results on the nuScenes test set, which can also be verified on the official eval.ai webpage (https://eval.ai/web/challenges/challenge-page/356/overview). We included captured results from the webpage in Fig.R1 of the rebuttal PDF for reference. Reviewers can also find our results by searching the keyword \"STXD\" in the official leaderboard (https://eval.ai/web/challenges/challenge-page/356/leaderboard/1012). Upon acceptance, we will request the authorities to display our testing results on the nuScenes dataset webpage, as you have suggested."}, {"review_id": "AaH3vNqGmm", "reviewer": "Reviewer_X2eU", "summary": "This work presents a cross-modal knowledge distillation framework for the multi-view, called STXD. It tries to distill structural and temporal knowledge across different modalities (from LiDAR to Camera) from three aspects, namely Correlation Regularizing Distillation (CD), Temporal Consistency Distillation (TD), and Response-Level Distillation (RD). Experiments prove the effectiveness of the proposed approach.", "questions": "Please refer to the weakness section.", "limitations": "The authors discuss the social impact in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work focuses on distilling structural features from LiDAR to the camera-based setting, which is promising and could have potential usage for application.\n2. The method to utilize temporal information via feature comparison is interesting.\n3. Experiments are sufficient and show significant improvement over the baseline on several benchmarks.\n4, Overall, the writing is clear and easy to follow.", "weaknesses": "1. It's interesting to apply correlation regularization to cross-modality distillation. However, the experimental comparison with classic feature-level distillation and analysis are missing. I'm wondering why the correlation regularization achieves much better results than the classic one.\n2. Because temporal info seems to bring better results. How about the performance with a longer temporal sequence and stronger benchmark, like SOLOFusion [37].\n3. Some typos should be fixed: L58 This way-> In this way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer \"X2eU\" for acknowledging our cross-modal distillation framework is \"promising\" and has \"potential usage for application\" as well as recognizing our \"sufficient experiments\" showing \"significant improvements\" over the baselines. We hope you find our responses provided below helpful to address your questions.\n\n**Comparison with \"classic feature-level distillation\" and Analyses:**\\\nTo our best understanding, we interpret \"classic feature-level distillation\" referred to by \"X2eU\" is closely related to conventional MSE-like global one-to-one distance minimization methods [R6-R9]. If so, please note that we provided the comparisons between our Correlation Regularizing Distillation (CD) and conventional feature-level distillation approaches in Tab.2. In particular, we showed that CD outperforms the conventional \"MSE\" and \"MSE w/ GT\" distillation methods by up to +2.24% of NDS and +2.25% of mAP.\\\nWe discussed that the observed performance gain over conventional feature-level distillation methods can be attributed to maximizing the information in the student features learned from the teacher (L142-L145). To validate our argument, we presented two quantitative measures: (1) Effective Dimension and (2) Dimension Redundancy (L254-L257). The results from Tab.3 and Fig.3 depicted the reduction of redundant feature components as we anticipated. In addition, Fig.1 and Fig.A3 of Appendix demonstrated the improved quality of feature similarities between the teacher and the student compared to classical distillation approaches (e.g., MSE and MSE w/ GT sampling).\n\n**Further validation with a longer temporal sequences:**\\\nThanks for suggesting such an interesting and valuable extension of our temporal distillation approach. We plan to apply our Temporal Consistency Distillation (TD) over longer temporal sequences and this could lead to another interesting contribution to the field of temporal cross-modal distillation and 3D object detection.\n\nLastly, we will correct typos in the final version."}, {"review_id": "zx2UB6Q5XQ", "reviewer": "Reviewer_U3eQ", "summary": "This paper presents a novel cross-modal knowledge distillation framework for multi-view 3D object detection, namely STXD, with three effective strategies. \n\n(1) correlation-based cross-modal feature distillation, which gets rid of the domain gap on the feature level during distillation. \n\n(2) feature similarity guided temporal feature distillation, which considers the dynamic scenes (especially moving objects) across multiple frames through similarity maps.\n\n(3) response-level distillation via bipartite matching between teacher and student predictions.\n\nOverall, STXD significantly improves current 3D detectors with its effective KD framework on nuScenes dataset, and provide a clear architecture.", "questions": "1. The paper presents remarkable improvements on BEVFormer and UVTR, however, there are other types of detectors, for instance, BEVDepth4D, BEVDet4D-Depth. What about these LSS-based detectors? Can STXD improve them even in a stronger setting? (e.g. with a ConvNeXt-based backbone and achieves 60+NDS on nuscenes test set?) \n\n2. What about the performance on Waymo dataset? The authors are suggested to conduct experiments on other datasets except nuScenes to make up for the limited novelty in this paper.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow.\n\n2. The paper presents new perspectives on knowledge distillation for multi-view 3D detectors compared to recent works, i.e., temporal-based feature distillation.\n\n3. The improvements on nuScenes dataset validate the effectiveness of STXD and the experiments on nuScenes dataset are extensive.\n\n4. The paper is a practical work with solid enhancement to current 3D detectors.", "weaknesses": "1. The main concern of the paper is the novelty of this paper. Actually, the correlation-based feature distillation to decouple the domain knowledge for LiDAR and image is not novel, even in the cross-modal distillation (for example TiG-BEV[1], on arxiv 5 months ago). The response-level distillation is also similar to BEVDistill [2]. \n\n[1] TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning\n[2] BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the recognition from reviewer \"U3eQ\" for acknowledging our \"new perspectives on knowledge distillation for 3D detectors,\" particularly our \"temporal-based feature distillation\" approach. \"U3eQ\" also noted the effectiveness of STXD, validated through extensive experiments, and its \"practicability\" and \"solid/remarkable (performance) enhancement\". Below, we address your concerns regarding the contributions of STXD, including Correlation Regularizing Distillation (CD) and Response-Level Distillation (RD).\n\n**Correlation Regularizing Distillation (CD) vs. TiG-BEV [R3]:**\\\nOur Correlation Regularizing Distillation (CD) and TiG-BEV [R3] both consider dimensional correlation of features, which might appear to be similar at first glance. However, we want to emphasize that these methods are distinct in terms of their underlying objectives and implementations. Consequently, there exist clear differences between CD and TiG-BEV in several aspects. TiG-BEV proposes two correlation-based losses:\n\n$L_{\\text{inter-channel}}=\\|\\|C_F - C_G\\|\\|_2$, where $C_F=F F^T, C_G=G G^T \\in \\mathbb{R}^{D\\times D}$, \n\n$L_{\\text{inter-keypoint}}=||\\tilde{C}_F - \\tilde{C}_G||_2$, where $\\tilde{C}_F=F^T F, \\tilde{C}_G=G^T G \\in \\mathbb{R}^{N\\times N}$. \n\nBoth losses try to minimize distance between **intra-modal self-correlation** of BEV features from the teacher ($F$) and student models ($G$) based on the MSE distance function.\n\nIn contrast, the core contribution and novelty of CD lies in its introduction of **cross-modal cross-correlation regularization** to maximize the information transferred from the teacher to the student. Specifically, our inspiration comes from recent self-supervised learning methods [R1, R2], which utilize a **decorrelation mechanism** to reduce duplicated information in the learned features. We found such info-max approach interesting and valuable, especially for knowledge distillation of the BEV features containing noisy and redundant background information. Our CD loss is defined as follows (quoted from Eq.2):\n\n$L_{\\text{CD}}= \\sum_{i} (1-C(i,i))^2 + \\lambda_{c} \\sum_{i} \\sum_{j\\neq i} C(i,j)^2, \\text{where } C = \\hat{F}^T \\hat{G} \\in \\mathbb{R}^{D \\times D}.$\n\nCD loss utilizes the decorrelation mechanism (the second term of Eq.2) to prevent information collapse in the student model, while enhancing the dimensional component-wise feature similarities (the first term of Eq.2) (refer to L43-L48 and L141-L145). However, TiG-BEV does not utilize such an information maximization mechanism. For a clear comparison, we have also depicted relevant diagrams to clarify these differences in Fig.R2 of the rebuttal PDF. These differences lead to the competence of CD over TiG-BEV, as presented in Tab.R1 and Tab.R2. From these results, we confirmed that (1) CD effectively reduces duplicate dimensional feature components compared to the correlation-based distillation of TiG-BEV (Tab.R2), and (2) ultimately outperforms prior cross-modal distillation methods (i.e., improvements over TiG-BEV [R3] and BEVDistill [R6] by up to +3.2% of NDS and +3.9% of mAP (Tab.R1)). Such results are consistent with our extensive component-wise ablation and main results presented throughout the manuscript and the Appendix.\n\n**Additional Contribution of Response-Level Distillation (RD):**\\\nWe are also aware of that prior works [R6-R9] have introduced similar approaches to RD. Please note that we acknowledge that and do not claim the introduction of RD as our main contribution. Rather we explicitly indicated our \"adoption\" and \"inspiration\" from such output distillation strategies to further enhance our knowledge distillation performance (L59-L61, L167-L168). Please note the improvement gained from our core feature-level distillation methods (CD,TD) is larger than RD (i.e., gain of NDS(%) and mAP(%) over based student model BEVFormer [R4]: RD=1.44% and 1.56%, TD=1.77% and 2.15%, and CD=2.47% and 2.75%) as shown in Tab.1. More importantly, STXD without RD already outperforms BEVDistill [R6] (i.e., STXD w/o RD=54.0% of NDS and 43.4% of mAP in Tab.1 vs. BEVDistill=52.4% of NDS and 41.7% of mAP in Tab.4). Hence, we can consider RD as an add-on method for improved cross-modal knowledge distillation, given its widespread use in most state-of-the-art distillation methods for object detection [R6-R9].\n\nTo draw a clear comparison with BEVDistill, our output distillation formulation is primarily inspired by the quality-score approaches used in output-guided knowledge distillation for 2D object detectors [R7,R9] (refer to L171-L177). In contrast, BEVDistill employs a critic network to approximate distributional similarities in output-level features between teacher and student models. Hence, technically speaking, RD is different from BEVDistill.\n\n**Additional Baseline for Student Model:**\\\nPlease note that we extensively validated our STXD framework with two of baseline student models, BEVFormer [R4] and UVTR-C/CS [R5], representing attention-based and LSS-based BEV feature generation approaches, respectively. To further validate the effectiveness of our approach, we also present additional validation experiments with BEVDepth [R11] in Tab.R1 of the rebuttal PDF, where CD and TD enhances BEVDepth, by achieving notable improvements of +5.3% and +5.2% in terms of NDS, respectively.\n\n**Additional Validation on Waymo Dataset:**\\\nGenerally, the method that performs well on the nuScenes dataset often achieves favorable outcomes in the Waymo dataset, as observed in [R4,R15]. Therefore, we believe our extensive validation and testing results on the nuScenes sufficiently demonstrate the effectiveness of our cross-modal distillation approach. To accommodate your suggestion, we are conducting experiments on the Waymo dataset, but we are unable to report the results from such a large-scale dataset (230K frames over 1TB) within the short rebuttal period. We will report results in the final version of our paper."}]

Author rebuttal:
Dear AC and Reviewers,

First of all, we sincerely appreciate your time and effort to review our work. At the end of this comment, we have attached our **Rebuttal PDF**, which includes additional experimental results and diagrams illustrating the comparisons with prior methods. Tables and figures in the rebuttal PDF are referenced in our rebuttal comments as Tab.**R**x and Fig.**R**x. We will include these additional experimental results (Tab.R1 and Tab.R2) in the final version. Also, we have listed the references we used in our rebuttal below. Please refer to this list in each rebuttal comment and in the rebuttal PDF.

**References:**

**[R1]** Zbontar, J., Jing, L., Misra, I., LeCun, Y., & Deny, S. (2021, July). Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (pp. 12310-12320). PMLR.

**[R2]** Bardes, A., Ponce, J., & Lecun, Y. (2022, April). VICReg: Variance-Invariance-Covariance Regularization For Self-Supervised Learning. In ICLR 2022-International Conference on Learning Representations.

**[R3]** Huang, P., Liu, L., Zhang, R., Zhang, S., Xu, X., Wang, B., & Liu, G. (2022). TiG-BEV: Multi-view bev 3d object detection via target inner-geometry learning. arXiv preprint arXiv:2212.13979.

**[R4]** Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., ... & Dai, J. (2022, October). BEVFormer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision (pp. 1-18). Cham: Springer Nature Switzerland.

**[R5]** Li, Y., Chen, Y., Qi, X., Li, Z., Sun, J., & Jia, J. (2022). Unifying voxel-based representation with transformer for 3d object detection. Advances in Neural Information Processing Systems, 35, 18442-18455.

**[R6]** Chen, Z., Li, Z., Zhang, S., Fang, L., Jiang, Q., & Zhao, F. (2022). BEVDistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:2211.09386.

**[R7]** Chen, Z., Yang, C., Li, Q., Zhao, F., Zha, Z. J., & Wu, F. (2021, October). Disentangle your dense object detector. In Proceedings of the 29th ACM international conference on multimedia (pp. 4939-4948).

**[R8]** Hong, Y., Dai, H., & Ding, Y. (2022, October). Cross-modality knowledge distillation network for monocular 3d object detection. In European Conference on Computer Vision (pp. 87-104). Cham: Springer Nature Switzerland.

**[R9]** Yang, C., Ochal, M., Storkey, A., & Crowley, E. J. (2022, October). Prediction-guided distillation for dense object detection. In European Conference on Computer Vision (pp. 123-138). Cham: Springer Nature Switzerland.

**[R10]** Zhu, B., Jiang, Z., Zhou, X., Li, Z., & Yu, G. (2019). Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492.

**[R11]** Li, Y., Ge, Z., Yu, G., Yang, J., Wang, Z., Shi, Y., ... & Li, Z. (2023, June). BEVDepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 2, pp. 1477-1485).

**[R12]** Dai, X., Jiang, Z., Wu, Z., Bao, Y., Wang, Z., Liu, S., & Zhou, E. (2021). General instance distillation for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 7842-7851).

**[R13]** Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., ... & Sutskever, I. (2021, July). Learning transferable visual models from natural language supervision. In International conference on machine learning (pp. 8748-8763). PMLR.

**[R14]** Kuhn, H. W. (2005). The Hungarian method for the assignment problem. Naval Research Logistics (NRL), 52(1), 7-21.

**[R15]** Yin, T., Zhou, X., & Krahenbuhl, P. (2021). Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 11784-11793).

Meta-review: This paper presented a novel approach for LiDAR-to-Camera distillation. Three key schemes are introduced, and experimental results demonstrate the proposed approach. The reviewers provided many constructive comments. Most comments were well handled by the author rebuttal. There remain two key issues: add experiments with longer past frames and on an additional dataset. Please do include the two experiments in the final version.
Decision: Accept (poster)