Title: STXD: Structural and Temporal Cross-Modal Distillation for Multi-View 3D Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: knowledge distillation

cross-modal learning

3d object detection
Abstract: 3D object detection (3DOD) from multi-view images is an economically appealing alternative to expensive LiDAR-based detectors, but also an extremely challenging task due to the absence of precise spatial cues. Recent studies have leveraged the teacher-student paradigm for cross-modal distillation, where a strong LiDAR-modality teacher transfers useful knowledge to a multi-view-based image-modality student. However, prior approaches have only focused on minimizing global distances between cross-modal features, which may lead to suboptimal knowledge distillation results. Based on these insights, we propose a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3DOD. First, STXD reduces redundancy of the feature components of the student by regularizing the cross-correlation of cross-modal features, while maximizing their similarities. Second, to effectively transfer temporal knowledge, STXD encodes temporal relations of features across a sequence of frames via similarity maps. Lastly, STXD also adopts a response distillation method to further enhance the quality of knowledge distillation at the output-level. Our extensive experiments demonstrate that STXD significantly improves the NDS and mAP of the based student detectors by 2.8%~4.5% on the nuScenes testing dataset.

STXD: Structural and Temporal Cross-Modal Distillation for Multi-View 3D Object Detection

Abstract

3D object detection (3DOD) from multi-view images is an economically appealing alternative to expensive LiDAR-based detectors, but also an extremely challenging task due to the absence of precise spatial cues. Recent studies have leveraged the teacher-student paradigm for cross-modal distillation, where a strong LiDARmodality teacher transfers useful knowledge to a multi-view-based image-modality student. However, prior approaches have only focused on minimizing global distances between cross-modal features, which may lead to suboptimal knowledge distillation results. Based on these insights, we propose a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3DOD. First, STXD reduces redundancy of the feature components of the student by regularizing the cross-correlation of cross-modal features, while maximizing their similarities. Second, to effectively transfer temporal knowledge, STXD encodes temporal relations of features across a sequence of frames via similarity maps. Lastly, STXD also adopts a response distillation method to further enhance the quality of knowledge distillation at the output-level. Our extensive experiments demonstrate that STXD significantly improves the NDS and mAP of the based student detectors by 2 . 8% ∼ 4 . 5% on the nuScenes testing dataset.

1 Introduction

3D object detection ( 3DOD ) is the task of locating and classifying objects in 3D space using input data from specific modalities, which typically include LiDAR point clouds [61, 14, 7] and camera images [54, 34, 24]. 3DOD is central to understanding the surrounding environment and has been widely applied to various complex vision systems, such as autonomous driving [44], robotic manipulation [71, 65], and augmented reality [42]. Recently, camera-based multi-view 3DOD has emerged as an attractive alternative to expensive LiDAR-based methods, thanks to its ubiquity, low cost, and the semantically rich information from colorized pixels. In general, current works on multi-view 3DOD aim to extract a unified bird's-eye view ( BEV ) feature map from multiple camera images. However, 3DOD from only camera-specific BEV features is extremely challenging due to the lack of precise spatial cues [39], which leads to inferior performance of camera-based models compared to LiDAR-based models. To bridge the gap, some prior works [16, 11, 19, 29, 9] introduced a teacher-student paradigm for cross-modal knowledge distillation to transfer geometrically-rich 3D information from LiDAR models ( teacher ) to camera-based 3DOD models ( student ). The current state-of-the-art 3DOD models typically follow a similar detection paradigm, where modality-specific encoders are used to extract BEV features, followed by a 3D object detection head [29, 34, 55, 7].

(a) LiDAR

(b) Ours

(c) MSE w/ GT

(d) MSE

(e) Camera

Hence, recent works have taken the BEV features from different modalities as the focal point of knowledge distillation [16, 29], and then applied additional response-level distillation [11, 19, 9].

One of the main challenges in cross-modal knowledge distillation is transferring modality-specific knowledge from the teacher to the student in a different modality. This is primarily because the features learned by different modalities are typically non-homogeneous, and there exist distributional divergences between the feature spaces as shown in Fig.1-(a) and (e). Most existing distillation methods aim to minimize global distances between cross-modal features ( i.e., L 2 -distance). While this approach has achieved some success, it fails to capture the structural knowledge inherent in the modality-specific features, which can lead to suboptimal knowledge distillation, as shown in Fig.1-(c) and (d). To address these challenges, we first propose a distillation method that regularizes cross-correlation of BEV features from the teacher and the student models. In particular, we introduce a decorrelation mechanism [67, 2] to reduce redundancy among feature components, thereby maximizing the information contained in the features. Our structural knowledge distillation approach ultimately prevents information collapse in the student model and enhances the component-wise feature similarities across different modalities, as illustrated in Fig.1-(a) and (b).

We also explore the integration of temporal knowledge embedded in features of previous frames. Although temporal information has been actively exploited in recent 3DOD approaches [34, 29, 32, 41], it has been relatively understudied for cross-modal distillation. The main challenge is the spatiotemporal misalignment between past and current frames. For instance, in driving scenes, foreground objects may have different locations or disappear over time. Thus, directly matching the misaligned cross-modal features could result in incorrect distillation. Instead, we leverage the temporal relations of the BEV features by generating temporal similarity maps between the teacher's current and past frames. Specifically, we train the student model to generate the BEV feature of the current frame, which mimics the temporal relations encoded in the similarity maps from the teacher. In this way, we can avoid false matching issues when transferring temporal information over a sequence of frames. To further improve the quality of transferred knowledge, we also adopt the response distillation approach [10, 59, 19, 9], which assigns quality scores to the predictions of the teacher model, thereby enabling the distillation of only meaningful information at the prediction-level.

Our structural and temporal cross-modal distillation ( STXD ) framework integrates the proposed feature- and response-level distillation approaches (see Fig.2), and demonstrated a significant improvement on the nuScenes 3DOD benchmark [3], leading to up to 3 . 2% NDS and 4 . 5% mAP improvement over the based student models. We also conducted detailed ablation studies to validate the effectiveness of each component of STXD. Notably, our framework does not require additional computational cost at testing time, while improving the prediction performance of the student models.

2 Related Work

LiDAR-based 3DOD. Recent LiDAR-based 3DOD detectors can be categorized based on the representation types of 3D space: voxel-based and point-based approaches. The voxel-based approaches [69, 58, 27, 48, 14, 7] convert point clouds into a regular voxel-grid representation. Pointbased approaches [62, 49, 64, 61] extract point-wise features directly from raw point clouds. Both approaches involve transforming low-level features into a BEV representation, which is then followed by an object detection head that generates response features.

LiDAR

LiDAR

PointClouds

BEVFeatures

LiDARPredictions

Detection

Backbone

Head

Feature-Level Distillation

Response-Level Distillation

3DBoX

.

Attributes

LiDAR Features

LTD

LiDARFeatures

(Previous Frames)

LCD

BoxMatching

LRD

Identity

Matrix

Cross-Correlation

Dimensional

Camera Features

Similarity Map

Temporal

Attributes

3DBoX

CorrelationRegularizingDistillation

Temporal ConsistencyDistillation

Image

Detection

Backbone

Head

Multi-View

Camera

CameraPredictions

Camera Images

BEVFeatures

Camera-based 3DOD. Camera-based approaches typically perform 3DOD on either single- or multi-view images. Prior works on single-view 3DOD aim to detect 3D boxes directly from image features [5, 38, 53], or using intermediate geometric representations ( e.g., pseudo-lidar [66, 12, 50], probabilistic depth-cues [47, 52]). More recently multi-view 3DOD approaches [21, 33, 29] use the lift-splat-shoot method [43, 47] to transform multi-image features into a unified bird's-eye view (BEV) representation followed by an object detection head. Other works on multi-view images [54, 34, 35, 24] aim to implicitly learn query-based BEV features sampled from corresponding 2D image features based on attention mechanisms [4, 72]. While some of these works [50, 47, 33] utilize LiDAR point clouds as input for explicit supervision of depth cues, they do not fully exploit the potential of LiDARbased 3DOD features, which have appeared to be effective for improved prediction performance of camera-based models. In light of this work, we further provide new perspectives on the structural and temporal knowledge embedded in the LiDAR-based BEV features.

Multi-Modal 3DOD. Recently, integrating complementary information from different modality sensors, especially LiDAR and cameras, has demonstrated improved 3DOD performance. One line of work fuses intermediate feature representations in the BEV space [8, 31, 36, 63], while other approaches try to fuse object proposals from each modality in the detection head [25, 6, 1]. Another line of work augments point clouds with semantic information extracted from corresponding images [51, 51, 57]. However, these methods require both sensors to be available during test time. In contrast, our cross-modal distillation approach leverages both modalities only during training to transfer useful knowledge from LiDAR-based to camera-based 3DOD models. During testing, we use only multi-camera images without any additional computational costs.

Knowledge Distillation for 3DOD. Most prior works on knowledge distillation for object detection have focused on transferring knowledge among single-modal detectors, particularly for 2D object detection ( e.g., [15, 59]) and LiDAR-based models ( e.g., [60, 56]). Relatively fewer works have studied cross-modal distillation for object detection. MonoDistill [11] projects LiDAR point clouds into perspective image space to distill spatially aligned LiDAR features to a camera-based student model. LIGA-Stereo [16] aims to transfer intermediate features from LiDAR to a stereo camera-based student model. CMKD [19] and BEVDistill [9] employ knowledge distillation from LiDAR-based models at both the feature and response levels to minimize the gap between modalities. Similarly, UVTR [29] tries to directly match the BEV representations of the camera model to the LiDAR-based

teacher. Although these methods have achieved some success, they still do not carefully consider the distributional divergences between non-homogeneous modalities. To mitigate the modality gap between LiDAR and camera features, TIG-BEV [22] learns the inner geometry of foreground objects by encouraging the student's features to replicate the self-correlation of the teacher's features. However, this method still does not consider cross-modal associations among BEV features from the teacher and student models. Differs from previous methods, our distillation approach considers structural knowledge in the features by introducing cross-modal cross-correlation regularization inspired by the decorrelation mechanism [67, 2]. Furthermore, we introduce a method for effectively distilling temporal information contained in previous frames of the teacher. We also utilize responselevel distillation to further improve the quality of knowledge distillation at the predictions level.

3 Methodology

3.1 Preliminary

Recent 3D object detectors share a similar detection paradigm, where low-level features are extracted from modality-specific encoders and subsequently transformed into BEV features (see Fig.2). For example, modern LiDAR-based detectors [69, 27, 55] use 3D encoders to extract low-level features from the raw point clouds, then the features are further transformed to the BEV features F ∈ R X × Y × Z × D with spatial extent of X × Y × Z and the number of channels D . For recent multi-view image detectors, there are two mainstream approaches for transforming the low-level image features to the BEV features: (1) estimating depth cues [21, 33, 29] via lift-splat-shoot [43] and (2) implicit learning of depth via cross-attention mechanism [35, 34]. Regardless of the type of transform methods, the BEV features can be represented by G ∈ R X × Y × Z × D , similar to the LiDAR-based detectors. In this work, as illustrated in Fig.2, we aim to train a high-performing camera-based detector (student) by distilling useful knowledge from a LiDAR-based detector (teacher) while considering the structural and temporal information embedded in the modality-specific BEV featuers, F and G .

3.2 Correlation Regularizing Distillation

Prior works on cross-modal distillation directly force the student to mimic the BEV feature from the teacher by calculating element-wise distance loss ( i.e., || F -G || 2 , [19, 29]), or weighted via foreground mask ( i.e., || W fg · ( F -G ) || 2 , [11, 9]). However, they ignore the structural knowledge inherent in modality-specific features and may not effectively distill rich information contained in the features. Recently, several self-supervised learning methods have incorporated cross-correlation regularization to maximize the information contained in features [67, 2]. These methods can also be applied to the problem of LiDAR-to-camera knowledge distillation, since LiDAR and camera features can be regarded as different representations obtained from the same scene, and have a spatial one-to-one correspondence. From this intuition, we propose the C orrelation Regularizing D istillation ( CD ) loss that maximizes the similarity between aligned features while regularizing cross-correlation along feature dimensions. Let batches of N aligned D -dimensional LiDAR and camera features be given as F = [ f 1 , f 2 , ..., f N ] T ∈ R N × D and G = [ g 1 , g 2 , ..., g N ] T ∈ R N × D , respectively. N is the number of serialized BEV features, given by X · Y · Z . Following [67], each feature is transformed to be mean-centered along the batch dimension, denoted by ˆ F and ˆ G . Then, CD loss is defined as:

glyph[negationslash]

where C denotes a dimensional cross-correlation matrix and λ c ≥ 0 is a balancing hyper-parameter. The first term of Eq.2 encourages the aligned LiDAR and camera feature components to be similar, while the second term reduces the correlation among feature components that are not aligned. Hence, CD loss can reduce duplicated information in the learned features, allowing for the student to learn diverse information from the teacher. We analyzed the effectiveness of the CD loss in Sec.4.2.

3.3 Temporal Consistency Distillation

In driving scenes, sensor data is received sequentially. Therefore, the past few frames may contain valuable information for detecting objects in the current frame, and incorporating temporal information has brought significant improvements to 3DOD [34, 29, 32, 41]. However, directly learning

from the LiDAR features of the past frames ( i.e., via L 2 -distance) may not be effective due to the spatially and temporally misaligned non-homogeneous features. To address this issue, we propose the T emporal Consistency D istillation ( TD ) loss, which indirectly distills the teacher's information from past frames by introducing a temporal similarity map. Let batches of aligned LiDAR and camera features of the current frame be given as F (0) and G (0) , respectively. We are also given the LiDAR features of the k -th previous frame, denoted by F ( -k ) , k ∈ [1 , K ] , where K is the maximum number of past frames referred to. Then, the cross-modal temporal similarity map S ( -k ) and the intra-modal temporal similarity map T ( -k ) between the current and the k -th previous frame are calculated as:

Then TD loss is defined as:

where D KL is the Kullback-Leibler divergence ( KLD ). Since S ( -k ) and T ( -k ) are matrices, we apply a softmax function to each row of them, and then calculate the KLD for each row. Based on the temporal similarity map, TD loss distills the teacher's information of past frames indirectly, avoiding spatial false matching issues across the time frames. We also demonstrated and analyzed the effectiveness of TD in Sec.4.2.

3.4 Response-Level Distillation

In general, detector models infer a large number of candidates, but not all candidates contain meaningful information. Since only a few candidates detect target objects, it is necessary for a student to selectively learn from the predictions of the teacher. Inspired by recent works on prediction-guided distillation [10, 59, 19, 9], we define the R esponse-Level D istillation ( RD ) loss that assigns quality scores to the predictions of the teacher model, enabling the distillation of only meaningful information. Let us define { ( b i , c i ) } i M =1 as prediction results of teacher model, where b i and c i are regression and classification results for the i -th of M box candidates, respectively. In a similar manner, student prediction can be defined as { ( ˜ b j , ˜ c j ) } ˜ j M =1 . Following [10, 59], quality score q i for i -th candidate of the teacher predictions can be calculated as follow:

where c ∗ i is the predicted probability for the class of the ground truth box that is matched to i -th candidate of the teacher predictions, b ∗ i is a regression parameter of the ground truth box that is matched to i -th candidate of the teacher predictions, and γ is a hyper-parameter balancing classification and localization. Then RD loss is defined as:

where π ( j ) is a function that returns the index of the corresponding teacher's prediction matched with the j -th student's prediction. By combining RD with our structural and temporal distillation (CD,TD), we achieved additional improvement on the detection performance as reported in Sec.4.2.

3.5 Optimization Objective

Finally, the overall optimization objective for the student models can be obtained by integrating the aforementioned three distillation losses and the task loss L task :

where L task is adopted from a general set-to-set target assignment for the student with ground truth, including regression and classification losses [54], based on the Hungarian algorithm [26].

4 Experiments

We now present extensive experimental results to validate our structural and temporal cross-modal knowledge distillation framework (STXD) for 3D object detection (3DOD) task. We first introduce the details of experimental setting (Sec.4.1) followed by various ablation studies to validate the effectiveness of each component in STXD (Sec.4.2). Then, we compare STXD against several leading methods on the nuScenes [3] dataset (Sec.4.3). Additional experiments are provided in the Appendix.

4.1 Experimental Settings

Dataset. We evaluate our STXD framework on the nuScenes [3] benchmark dataset, which is large-scale and widely used by recent research works on 3D object detection. The nuScenes dataset comprises 1,000 driving scenes with a duration of approximately 20s, which are divided into 700, 150, and 150 scenes for training, validation, and testing, respectively. Each scene consists of RGB images collected by six cameras, which capture a 360 ◦ field of view at 12Hz and 32-beam LiDAR point clouds at 20Hz. Annotations for 23 categories of objects are provided for each sample at a rate of 2Hz, and the metrics for the 3DOD task are evaluated using 10 classes of objects, as defined by the official evaluation metrics. These metrics include mean Average Precision (mAP) and nuScenes Detection Score (NDS). In detail, mAP measures the recall and precision of predicted bounding boxes, while NDS is a weighted sum of mAP and five true positive nuScenes metrics: mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), and mean Average Attribute Error (mAAE).

Baseline Models. To validate the effectiveness of our approach, we adopt two state-of-the-art crossmodal distillation approaches as baselines. Similar to concurrent work [9], we use BEVFormer [34] as the image-modality student model and Object-DGCNN [55] as the LiDAR-modality teacher model, where the DGCNN attention module of the transformer-decoder is replaced with vanilla multi-head attention. For the second baseline, we adopt the multi-view camera (single frame of UVTR-C, multi-frame of UVTR-CS) and LiDAR (UVTR-L) models from UVTR [29] as the student and teacher models, respectively. Our base student models represent two mainstream methods for transforming 2D perspective image features into 3D BEV features, where BEVFormer uses crossattention mechanism [4, 72] and UVTR-C/CS is based on estimating depth distributions through lift-splat-shoot approach [43, 47].

Implementation Details. The input image size for the student models is set to 1600 × 900 . For both teacher and student models, we use a voxel size of (0 . 1 m, 0 . 1 m, 0 . 2 m ) , and the input point clouds are filtered within a range of [ -51 . 2 m, 51 . 2 m ] for the X and Y axes and [ -5 . 0 m, 3 . 0 m ] for the Z axis. The dimensions of the BEV feature grids are 128 × 128 , and the number of channel D for BEV features is set to 256 . We train all student models for 24 epochs without using CBGS [70]. λ c in Eq.2 is set to 0.01 according to [67]. More implementation details are provided in the Appendix.

4.2 Ablation Studies

For our ablation studies, we implemented BEVFormer [34] with ResNet-101-DCN as backbone network for the student model, and evaluated the trained model on the nuScenes validation dataset.

Ablation on Distillation Methods. To validate the effects of each proposed distillation approach in Sec.3, we conducted an ablation study on the loss function. As shown in Tab.1, each loss term contributes to the improvement in performance. Specifically, the proposed feature-level distillation (CD and TD) plays a crucial role in enhancing the performance. When CD and TD losses are applied solely, NDS is increased by 2 . 47% and 1 . 77% , respectively. When all loss terms are combined, the performance is further improved without any conflicts, resulting in the best performance with a significant improvement of NDS by 2 . 87% and mAP by 3 . 52% . These results demonstrate the effectiveness of our structural and temporal distillation as well as the response distillation method.

Comparison with Feature-Level Distillation Methods. To solely examine the effectiveness of our correlation regularizing distillation (CD), we compared it to the following feature-level distillation methods without applying other distillation approaches. MSE is a baseline method [19, 29] that simply minimizes the element-wise L 2 -distance between the features of a teacher and a student. MSE w/ GT is similar to MSE, but it only distills features near the ground truth point [11, 9]. Our CD loss in Eq.2 draws inspiration from the Barlow Twins [67] approach for representation learning. To validate our choice, we also conducted a comparison with other alternative methods. Specifically, we applied VICReg [2] and CLIP [45] to CD loss and evaluated their impact.

Tab.2 highlights that even in the absence of other distillation losses, the CD loss alone delivers the most promising performance, resulting in a significant increase of 2 . 47% in NDS. At the bottom of Tab.2, we observe that other representation learning methods also lead to performance improvements. Notably, the Barlow Twins method achieves the best performance among the tested methods. From a distillation perspective, VICReg solely relies on MSE loss to transfer the teacher's knowledge, which is referred to as the invariance term in [2]. As a result, the structural knowledge from the teacher may not be fully exploited, leading to suboptimal performance. CLIP is a contrastive learning method that typically requires a large and diverse set of positive and negative pairs of features for effective learning. In the case of 3DOD, however, cross-modal feature batches extracted from a scene typically contain only a limited number of positive pairs, while the number of negative pairs is much larger. Consequently, the effectiveness of contrastive learning may be compromised in the context of 3DOD. In contrast, Barlow Twins leverages the structural knowledge by exploiting cross-correlation between the cross-modal features (see Eq.1), and is robust to the size of feature set [67]. Thus, our correlation regularizing distillation (CD) is the most effective when implemented with Barlow Twins.

Correlation Regularization Effect of CD Loss. In Sec.3.2, we discussed that distillation with correlation regularization aims to maximize the information contained in the learned features, allowing the individual components of the feature to become more informative. Here we validate our argument using two qualitative measures: (1) effective dimension and (2) dimensional redundancy.

(1) Effective dimension [46] quantitatively measures the dimensionality of features. A higher effective dimension indicates lower redundancy among the features. In Tab.3, we compare effective dimensions d eff of features learned through various feature-level distillation methods. Since d eff is bounded by √ D = 16 , we also provide the squared values for a intuitive interpretation, bounded by D = 256 . For example, the features learned through CD can be effectively represented in a dimensional space of at least 20 dimensions, approximately. Notably, our CD loss achieves the highest effective dimension compared to other feature distillation methods.

(2) Dimensional redundancy: Redundant feature components exhibit higher correlation among themselves. From this intuition, we quantify the redundancy in features by utilizing Eq.1 and 2. For camera features G , we define a dimensional self-correlation matrix C self = ˆ G T ˆ G . We then calculate the second term of Eq.2 for C self and refer to the resulting value as the dimensional redundancy of G . Fig.3 depicts the dimensional redundancy of features learned from the MSE and CD loss throughout the training progress. Results demonstrate that the CD loss is more effective in reducing dimensional redundancy compared to the MSE loss. Furthermore, even when the MSE loss is replaced with the CD loss during training, a decrease in dimensional redundancy is still observed (red dashed lines).

These results indicate that distilling features through a simple element-wise comparison can lead to dimensional collapse. In contrast, our information maximization distillation method leverages the structural knowledge embedded in the feature space by considering the cross-correlation of feature components, and ultimately contributes to significantly improved prediction performance (see Tab.1).

Effect of K on TD Loss. The temporal consistency distillation (TD) in Eq.4 utilizes the BEV features extracted from the past K frames. Tab.5 demonstrates increasing the value of K results in improved 3DOD performance, particularly in terms of mAVE (mean Average Velocity Error). By leveraging the information from consecutive frames through the TD loss, the student can effectively learn the dynamics of target objects, leading to accurate predictions of their dynamic attributes. This highlights the impor-

tance of referencing past frames, as it allows the teacher to transfer a greater amount of information to the student, resulting in improved 3DOD performance. Similar results have also been echoed in concurrent works [32, 41], which also leverage temporal information for the single-modal 3DOD.

Visualization of Temporal Similarity Map.

The similarity map defined in Eq.3 calculates the element-wise similarity between the BEV features of the current and past frames. In Fig.4, we select the feature element corresponding to the location of the car object in the current BEV space and calculate its similarity to the entire features of the past frames. Notably, the student model trained with TD exhibits improved similarity patterns with the teacher (see Fig.4-(a) and (b)). This indicates that the student model has learned to capture similar temporal relations and representations

(a) Teacher

(b) w/ TD

(c) w/o TD

from the teacher by referring to the past frames. The competence of our temporal distillation approach ultimately contributes to the improved 3DOD performance, as demonstrated in our ablation studies (see Tab.1 and 5). We provide more temporal similarity visualization results in the Appendix.

4.3 Comparisons with Other Methods

To demonstrate the generalizability of our approach, we apply STXD to both UVTR [29] and BEVFormer [34] baselines. For a fair comparison, we use ResNet-101-DCN [18, 13] as the backbone of our student models, unless otherwise specified. Tab.4 shows the evaluation results on the nuScenes

validation set, where STXD significantly improves the performance of the based student models, surpassing the performance of existing distillation methods [29, 9], up to 2 . 9% in NDS and 3 . 5% in mAP. In most cases, the addition of STXD contributes to the improvement of mATE (translation), mASE (scale), and mAVE (velocity) metrics. Both mATE and mASE are closely related to the geometric features of 3D objects, and the teacher effectively transfers such important information to the student models. The improvement in mAVE can be attributed to our temporal distillation method.

Similar results are echoed in our experiments on the nuScenes testing dataset, as shown in Tab.6. We did not use CBGS [70] or test time augmentations to evaluate the sole effect of our method. STXD consistently improves NDS and mAP by up to 3 . 2% and 4 . 5% , respectively, compared to the various baseline methods (UVTR-C/CS, L2C/L2CS, BEVFormer † ). STXD also improves performances for the student model with heavy backbone (BEVFormer ‡ ) by 2 . 8% in NDS and 4 . 0% in mAP. These results are significant as models with heavy backbones already achieve high performance, making it challenging to further improve through knowledge distillation. In the Appendix, we show the competence of STXD on lightweight backbones as well. These results demonstrate the effectiveness and versatility of STXD in enabling cross-modal distillation between the teacher and the student models. Overall, STXD enables the student to learn the maximum amount of information from the teacher, including geometric and temporal knowledge, leading to significantly enhanced performance.

5 Conclusion

In this paper, we presented STXD, a cross-modal knowledge distillation framework for the multi-view 3D object detection task. STXD is capable of effectively distilling structural and temporal knowledge across different modalities through Correlation Regularizing Distillation (CD) and Temporal Consistency Distillation (TD). To further enhance the distillation quality, STXD also adopts Response-Level Distillation (RD), which transfers task-specific knowledge at the output level. Our extensive experiments and ablation studies demonstrated the effectiveness of STXD on the nuScenes dataset, where the NDS and mAP of the based student detectors are improved by up to 3 . 2% and 4 . 5% on the testing set. Inspired by recent works on multi-modal fusion [36, 63, 1] and other types of 3D perception tasks ( e.g., BEV Segmentation [68, 20], Occupancy Detection [23, 30]), we plan to further explore the potential of STXD in such various tasks leveraging cross-modal distillation. It would also be valuable to apply and validate our approach with other models that utilize long-term information such as [17, 41]. We hope our work provides a solid baseline and new perspectives on structural and temporal cross-modal distillation.

Societal Impact. Though the proposed method significantly improves the performance of multiview camera-based 3D object detection, it does not guarantee perfect predictions for all cases. Therefore, when applied to actual autonomous driving scenes, a contingency plan must be included.

References

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

Adrien Bardes, Jean Ponce, and Yann LeCun. VICReg: Variance-invariance-covariance regularization for self-supervised learning. In International Conference on Learning Representations, 2022.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Proceedings of the European Conference on Computer Vision, 2020.

Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monocular 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.

Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. BEVDistill: Cross-modal BEV distillation for multi-view 3d object detection. In International Conference on Learning Representations, 2023.

Zehui Chen, Chenhongyi Yang, Qiaofei Li, Feng Zhao, Zheng-Jun Zha, and Feng Wu. Disentangle your dense object detector. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.

Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. Monodistill: Learning spatial features for monocular 3d object detection. In International Conference on Learning Representations, 2022.

Xiaomeng Chu, Jiajun Deng, Yao Li, Zhenxun Yuan, Yanyong Zhang, Jianmin Ji, and Yu Zhang. Neighborvote: Improving monocular 3d object detection through neighbor distance voting. In Proceedings of the 29th ACM International Conference on Multimedia, 2021.

Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel rcnn: Towards high performance voxel-based 3d object detection. In Proceeding of Association for the Advancement of Artificial Intelligence, 2021.

Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303.05970, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.

Yu Hong, Hang Dai, and Yong Ding. Cross-modality knowledge distillation network for monocular 3d object detection. In Proceedings of the European Conference on Computer Vision, 2022.

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: future instance prediction in bird's-eye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11790, 2021.

Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu. Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning. arXiv preprint arXiv:2212.13979, 2022.

Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2302.07817, 2023.

Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. In Proceeding of Association for the Advancement of Artificial Intelligence, 2023.

Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2018.

Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.

Youngwan Lee, Joong-won Hwang, Sangrok Lee, Yuseok Bae, and Jongyoul Park. An energy and gpucomputation efficient backbone network for real-time object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2019.

Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. In Advances in Neural Information Processing Systems, 2022.

Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.

Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoc V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. In Proceeding of Association for the Advancement of Artificial Intelligence, 2023.

Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceeding of Association for the Advancement of Artificial Intelligence, 2023.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In Proceedings of the European Conference on Computer Vision, 2022.

Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In Proceedings of the European Conference on Computer Vision, 2022.

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In IEEE International Conference on Robotics and Automation, 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.

Shujie Luo, Hang Dai, Ling Shao, and Yong Ding. M3dssd: Monocular 3d single stage object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Dennis Park, Rares Ambrus, Vitor Guizilini, Jie Li, and Adrien Gaidon. Is pseudo-lidar needed for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. In International Conference on Learning Representations, 2023.

Youngmin Park, Vincent Lepetit, and Woontack Woo. Multiple 3d object tracking for augmented reality. In IEEE/ACM International Symposium on Mixed and Augmented Reality, 2008.

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision, 2020.

Rui Qian, Xin Lai, and Xirong Li. 3d object detection for autonomous driving: a survey. Pattern Recognition, 130:108796, 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.

Kavya Ravichandran, Ajay Jain, and Alexander Rakhlin. Using effective dimension to analyze feature transformations in deep neural networks. In ICML 2019 Workshop on Identifying and Understanding Deep Learning Phenomena, 2019.

Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.

Andrea Simonelli, Samuel Rota Bulo, Lorenzo Porzi, Peter Kontschieder, and Elisa Ricci. Are we missing confidence in pseudo-lidar methods for monocular 3d object detection? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Proceedings of the Conference on Robot Learning, 2022.

Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.

Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Proceedings of the Conference on Robot Learning, 2022.

Yue Wang and Justin M. Solomon. Object dgcnn: 3d object detection using dynamic graphs. In Advances in Neural Information Processing Systems, 2021.

Yi Wei, Zibu Wei, Yongming Rao, Jiaxin Li, Jie Zhou, and Jiwen Lu. Lidar distillation: bridging the beaminduced domain gap for 3d object detection. In Proceedings of the European Conference on Computer Vision, 2022.

Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In IEEE International Intelligent Transportation Systems Conference. IEEE, 2021.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.

Chenhongyi Yang, Mateusz Ochal, Amos Storkey, and Elliot J Crowley. Prediction-guided distillation for dense object detection. In Proceedings of the European Conference on Computer Vision, 2022.

Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi. Towards efficient 3d object detection with knowledge distillation. In Advances in Neural Information Processing Systems, 2022.

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.

Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. In Advances in Neural Information Processing Systems, 2022.

Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.

Zhiyun Yin and Yujie Li. Overview of robotic grasp detection from 2d to 3d. Cognitive Robotics, 2022.

Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference on Learning Representations, 2020.

Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, 2021.

Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.

Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.

Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint arXiv:1908.09492, 2019.

Menglong Zhu, Konstantinos G Derpanis, Yinfei Yang, Samarth Brahmbhatt, Mabel Zhang, Cody Phillips, Matthieu Lecce, and Kostas Daniilidis. Single image 3d object detection and pose estimation for grasping. In IEEE International Conference on Robotics and Automation, 2014.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable {detr}: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations, 2021.

Appendix

In this appendix, we provide additional details of the implementation and training of our method (Appendix A). We also present additional experimental results, including ablations on lightweight backbones, additional baselines, and qualitative examples (Appendix B). These results further demonstrate the effectiveness of our STXD framework in cross-modal distillation for 3D object detection.

A Additional Implementation Details

A.1 Details on Cross-Modal Distillation Baselines

In Sec.4.1 of the manuscript, we introduced two cross-modal distillation baselines for 3D object detection: (1) BEVFormer [34] as the student model and Object-DGCNN [55] as the teacher model; and (2) UVTR-C/CS as the student models and UVTR-L as the teacher model [29]. To be more specific, in the first baseline (Object-DGCNN → BEVFormer), we set the spatial dimensions of the bird's-eye view (BEV) feature to be X = Y = 128 , and Z = 1 for both the teacher and student models, indicating a flat BEV feature space. In accordance with the concurrent work [9], we replaced the DGCNN attention module of the transformer-decoder in the teacher model with vanilla multi-head attention. Apart from these modifications, we followed the original implementations of both the student and teacher models. In the second baseline (UVTR-L → UVTR-C/CS), we utilized the original implementation of UVTR [29] without any modifications. The dimensions of the bird's-eye view (BEV) features were set to X = Y = 128 and Z = 11 for UVTR-C/CS; and Z = 5 for UVTR-L. Here, Z represents the height of the Bird's Eye View (BEV) feature space. The BEV features were further sampled using the ( x, y, z ) positions of object queries Q , where | Q | = 900 .

Note that we adopted two baseline student models that utilize multi-frame inputs. BEVFormer generates the BEV feature of the current frame by referring to the BEV feature of the past frames via temporal self-attention mechanism. In UVTR-CS, multi-frame images are concatenated and forwarded to the image encoder.

A.2 Training Details

For a fair comparison, we used ResNet-101-DCN pretrained by FCOS3D [53] as the backbone network for the student models in our extensive experiments (see Sec.4.2 and 4.3 in the manuscript) following prior works [34, 29]. We also showcased the competence of our STXD framework with other types of backbones, including VoVNet-99 [28] initialized from DD3D [40] as well as lightweight ResNet-18 and ResNet-50 networks (see B.1). We set the balancing parameter γ to 0.8 and 0.6 for classification and localization in Eq.5 of the manuscript, following [10, 59]. The weights for each loss term in Eq.7 of the manuscript are set to (0 . 1 , 100 . 0 , 1 . 0 , 1 . 0) , respectively, in the order mentioned. The student models were trained for 24 epochs using a learning rate of 2 × 10 -4 , and a batch size of 1 per GPU. All models were trained on 8 of NVIDIA A100 GPU while following the original codebase for each model. We employed AdamW [37] as the optimizer with a weight decay of 1 × 10 -2 . To isolate the effect of our STXD framework in cross-modal distillation, we deliberately did not use the CBGS strategy [70] or test-time augmentation in our experiments.

A.3 Instance Matching Function in Response-Level Distillation

In Sec.3.4, we adopted the commonly used Hungarian algorithm [26] to define set-to-set matching between the ground-truth and candidates from the teacher and the student, separately. Then, we constructed a mapping function π ( j ) between candidates from the teacher and the student based on matched ground-truth indices.

B Additional Experimental Results

B.1 Ablations on Lightweight Backbones

To further validate the effectiveness of our STXD framework in practical scenarios where lighter backbone networks are preferred for deployment on edge devices, we conducted experiments using smaller backbone networks for the student model. This allows us to assess the performance of our approach under resource-constrained settings while improving the level of accuracy in 3D object detection. We trained BEVFormer with ResNet-18 and ResNet-50 backbone networks on nuScenes training set and evaluated on the validation set. The results in Tab.7 demonstrate that STXD significantly improves the performance of BEVFormer with smaller backbone networks where it achieves an improvement of 3 . 24% and 4 . 00% in NDS and 3 . 33% and 4 . 49% in mAP, respectively. These improvements are consistent with the results presented in Tab.4 and 6 of the manuscript, further confirming the versatility and compatibility of our approach across different backbone architectures.

B.2 Experiments on Additional Teacher-Student Baseline

To further validate the effectiveness of our approach across various teacher-student baselines, we extend our experiments to additional student baseline, BEVDepth [33], which relies on LSS-based feature generation [43], depth supervision, and multi-frame inputs. Here we followed the official implementation of TiG-BEV [22] for cross-modal distillation with CenterPoint [64] as the teacher model, but added our feature-level distillation losses for fair comparisons. Results are reported in Tab.8, where our approach outperforms prior distillation methods, such as BEVDistill [9] and TiG-BEV [22], by up to +3.2% of NDS. These results are consistent to the results from BEVFormer and UVTR-C/CS baselines in Tab.4 and 6.

B.3 Comparison to Correlation-based Distillation Method

Our Correlation Regularizing Distillation (CD) and TiG-BEV [22] both consider dimensional correlation of features. However, these methods are distinct in their underlying objectives and implementations. TiG-BEV proposes the inter-channel BEV distillation loss (IC loss) that encourages the student's features to replicate the dimensional self-correlation of the teacher's features ( i.e., minimize || F T F -G T G || 2 ). In contrast, the core contribution of CD is introducing cross-modal

cross-correlation regularization to prevent student features from being redundant. Specifically, CD exploits the decorrelation mechanism (the second term of Eq.2) to prevent information collapse in the student model. However, IC loss does not utilize such an information maximization mechanism, and only relies on the global one-to-one MSE distance (L2-norm) between modality-specific self-correlation matrices. Hence, CD pursues a different learning principle compared to TiG-BEV.

These differences lead to the competence of CD over TiG-BEV. Tab.9 shows that CD effectively reduces duplicate dimensional feature components compared to the correlation-based distillation of TiG-BEV. Furthermore, in Tab. 8, CD ultimately achieves performance improvements over TiG-BEV and BEVDistill by up to +3.2% of NDS and +3.9% of mAP. These results consistently demonstrate the significance of mitigating feature redundancy in cross-modal knowledge distillation and enhancing feature quality through our feature-level distillation approaches.

B.4 Qualitative Results

B.4.1 Prediction Results on BEV Space

Fig.5 and 6 provide examples of 3DOD results in the BEV space from the LiDAR-based model (topleft), the multi-view camera-based model trained with our STXD (top-middle), and the camera-based model trained without knowledge distillation (camera-only model, top-right). Additionally, at the bottom of each example, we have included camera views that correspond to the highlighted regions, which can help in better understanding the results. Overall, STXD enables more accurate prediction of bounding box locations, scales, and orientations compared to the camera-only model (highlighted by red dashed boxes). This observation is consistent with the quantitative results in Tab.4 and 6 of the manuscript. STXD also leads to a reduction of false positives. In certain examples, we observe a decrease in false positives even when comparing the results to the LiDAR-based model (highlighted by pink dashed boxes). Notably, in Fig.6, STXD effectively reduces false positive detections in 3DOD compared to the LiDAR-based model and the camera-only model. These results indicate that our STXD framework effectively transfers valuable information from the LiDAR-based teacher to the camera student, leading to improved detection performance.

B.4.2 Qualitative Comparison on BEV Features

As an extension of Fig.1 in the manuscript, we provide additional examples of BEV features learned using different feature-level distillation methods in Fig.7. In various samples, the model trained with the correlation regularizing distillation (CD) consistently generates more similar feature patterns (Fig.7-(b)) to the teacher model (Fig.7-(a)) compared to other methods (Fig.7-(c) and (d)). These results demonstrate the effectiveness of the proposed method in transferring the rich information contained in the teacher model to the student model.

B.4.3 Temporal Similarity Maps

As an extension of Fig.4 in the manuscript, Fig.8 illustrates the learning progress of the temporal similarity map during the training phase for different values of k ( k = 1 , 2 , 3 ). As the training progresses, the cross-modal temporal similarity map S ( -k ) becomes similar to the intra-modal temporal similarity T ( -k ) . This indicates that the student model effectively learns the temporal knowledge encoded in the similarity maps from the teacher, and ultimately leads to significant improvements in 3DOD, as demonstrated in our extensive experiments reported in the manuscript.

LiDAR

Ours

Camera

LiDAR

Ours

Camera

Jps

LiDAR

Ours

Camera

LiDAR

日

amera,

LiDAR

Ours

Camera

可

LiDAR

Ours

Camera

LiDAR

Ours

Camera

GT

LiDAR

Ours

Camera

福

.....

(a) LiDAR

(b) Ours

(c) MSE w/ GT

(d) MSE

(e Camera

0 epoch

4 epoch

24 epoch

s(-3)

(z-)S

s(-1)

T(-3)

T(-2)

T(-1)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel cross-modal knowledge distillation framework for multi-view 3D object detection, namely STXD, with three effective strategies. \n\n(1) correlation-based cross-modal feature distillation, which gets rid of the domain gap on the feature level during distillation. \n\n(2) feature similarity guided temporal feature distillation, which considers the dynamic scenes (especially moving objects) across multiple frames through similarity maps.\n\n(3) response-level distillation via bipartite matching between teacher and student predictions.\n\nOverall, STXD significantly improves current 3D detectors with its effective KD framework on nuScenes dataset, and provide a clear architecture.\n\nSTRENGTHS:\n1. The paper is well-written and easy to follow.\n\n2. The paper presents new perspectives on knowledge distillation for multi-view 3D detectors compared to recent works, i.e., temporal-based feature distillation.\n\n3. The improvements on nuScenes dataset validate the effectiveness of STXD and the experiments on nuScenes dataset are extensive.\n\n4. The paper is a practical work with solid enhancement to current 3D detectors.\n\nWEAKNESSES:\n1. The main concern of the paper is the novelty of this paper. Actually, the correlation-based feature distillation to decouple the domain knowledge for LiDAR and image is not novel, even in the cross-modal distillation (for example TiG-BEV[1], on arxiv 5 months ago). The response-level distillation is also similar to BEVDistill [2]. \n\n[1] TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning\n[2] BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection\n\nQUESTIONS:\n1. The paper presents remarkable improvements on BEVFormer and UVTR, however, there are other types of detectors, for instance, BEVDepth4D, BEVDet4D-Depth. What about these LSS-based detectors? Can STXD improve them even in a stronger setting? (e.g. with a ConvNeXt-based backbone and achieves 60+NDS on nuscenes test set?) \n\n2. What about the performance on Waymo dataset? The authors are suggested to conduct experiments on other datasets except nuScenes to make up for the limited novelty in this paper.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This work presents a cross-modal knowledge distillation framework for the multi-view, called STXD. It tries to distill structural and temporal knowledge across different modalities (from LiDAR to Camera) from three aspects, namely Correlation Regularizing Distillation (CD), Temporal Consistency Distillation (TD), and Response-Level Distillation (RD). Experiments prove the effectiveness of the proposed approach.

STRENGTHS:
1. This work focuses on distilling structural features from LiDAR to the camera-based setting, which is promising and could have potential usage for application.
2. The method to utilize temporal information via feature comparison is interesting.
3. Experiments are sufficient and show significant improvement over the baseline on several benchmarks.
4, Overall, the writing is clear and easy to follow.

WEAKNESSES:
1. It's interesting to apply correlation regularization to cross-modality distillation. However, the experimental comparison with classic feature-level distillation and analysis are missing. I'm wondering why the correlation regularization achieves much better results than the classic one.
2. Because temporal info seems to bring better results. How about the performance with a longer temporal sequence and stronger benchmark, like SOLOFusion [37].
3. Some typos should be fixed: L58 This way-> In this way.

QUESTIONS:
Please refer to the weakness section.

LIMITATIONS:
The authors discuss the social impact in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3D Object Detection. STXD leverages the Correlation Regularizing Distillation loss to address the distributional divergences between non-homogeneous modalities. Besides, it effectively distills temporal information contained in previous frames of the teacher and utilizes response-level distillation to further improve the quality of knowledge distillation at the prediction-level. Extensive experiments on nuScenes show that STXD significantly improves the NDS and mAP of the base student detectors.\n\nSTRENGTHS:\n1. The paper provides valuable insights into addressing distributional divergences between non-homogeneous modalities through the use of the Correlation Regularizing Distillation loss.\n2. The effectiveness of the Correlation Regularizing Distillation loss is well demonstrated through ablation studies.\n3. The writing is clear and the paper is easy to follow.\n\nWEAKNESSES:\n1. The novelty of Response-Level Distillation (Sec 3.4) is limited, as a very similar method is proposed in BEVDistill [1].\n2. The effectiveness of Temporal Consistency Distillation is somewhat uncertain. Many previous works demonstrate the effectiveness of using multi-frames [2-4], and their methods are relatively simple (such as concatenating the temporal BEV features) yet effective in achieving state-of-the-art performance. However, this paper does not include experiments to demonstrate the effectiveness of Temporal Consistency Distillation in models that utilize multi-frames. If the authors can provide these experiments, I may raise my rating.\n3. While the insight of using the Correlation Regularizing Distillation loss to address distributional divergences is valuable, it should be noted that the authors do not propose the Correlation Regularizing Distillation loss itself.\n\n[1] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. BEVDistill: Cross- modal BEV distillation for multi-view 3d object detection. ICLR 2023.\n[2] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, Xiangyu Zhang. Exploring Recurrent Long-term Temporal Fusion for Multi-view 3D Perception. \n[3] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, Wei Zhan. Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection.\n[4] Junjie Huang, Guan Huang. BEVDet4D: Exploit Temporal Cues in Multi-camera 3D Object Detection.\n\nQUESTIONS:\n1. The paper does not clearly explain how to utilize sweep images in the student models. Additionally, it's worth noting that most of the camera-based baseline models seem not to use multi-frames, as evidenced by the relatively high mATE (>0.6) and mAVE (>0.5) values.\n2. Could the authors clarify why CBGS is not employed, given its potential to facilitate full model training?\n3. Could the authors offer the results on the nuScenes test sets? (Leaderboard web: https://www.nuscenes.org/object-detection?externalData=all&mapData=all&modalities=Camera)\n\nLIMITATIONS:\nThere are still many bad cases when using camera based 3D object detection. Achieving high performance with most of the existing methods often requires significant computational resources, such as training on 8 NVIDIA A100 GPUs.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a cross-modal distillation framework named STXD, which addresses existing distillation problems from both structural and temporal perspectives. Structure-based distillation imposes constraints on the similarity matrix of the student and teacher, ensuring that corresponding positions have a higher similarity. Temporal distillation is accomplished by distilling the feature similarity matrix over time. Ultimately, response-level distillation is employed to further improve the performance. Experiments on the nuScenes dataset demonstrate that the proposed approach outperforms previous distillation techniques.\n\nSTRENGTHS:\n+ The proposed cross-modal distillation from LiDAR to image works well (a marked improvement ranging from 2.8% to 4.5%). \n+ Experiments illustrate that the proposed approach outperforms previous approaches on the nuScenes dataset. \n+The paper is well written with a clear description of the method and comprehensive component analysis.\n\nWEAKNESSES:\n- The concept of 'structural distillation' seems no distinction from existent Mean Squared Error (MSE) distillation. It applies a weight of 0.01 to different positioning areas, hence, the leading role is still played by the loss at corresponding positions, which essentially substitutes MSE with a measure of similarity. \n- The structural distillation appears not reasonable. In fact, spatially adjacent tokens should have some correlation. In Fig.2, enforcing the correlation of corresponding positions to 1, and other positions to 0 is unjustified. Additionally, correlation is unable to respond to the redundancy of features, so Fig.3 is void of significance.\n- The paper employs an abundance of similarity matrices, resulting in substantial consumption of memory during the training process. I have noticed that BEVFormer uses only a 128x128 Bird's Eye View (BEV) resolution in the experiments. Whether the computational constraints caused this deviation from the default settings of BEVFormer.\n- The motivation for temporal distillation is insufficient. Theoretically, both the teacher and student features of the current frame contain temporal information. Thus, it brings into question the need to formulate the distillation approach in this manifestation.\n- It is better to conduct distillation based on stronger baselines such as BEVDepth, following BEVDistill. Table 2 ought to incorporate some comparison with prevalent distillation methods in 2D detection, such as GID. Techniques such as CLIP are seldom used. For the generality of distillation, the distillation on the Waymo dataset is indispensable.\n\nQUESTIONS:\n1. The notion of 'inherent structural knowledge' has not been succinctly explicated. For example, whether it refers to the 'instance'. It appears no relation with the term 'structure'. \n2. As described in the weakness, the authors should give a reasonable explanation for the motivation of structural distillation and temporal distillation. \n3. More experiments are needed to prove the effectiveness of the method.\n\nLIMITATIONS:\nThe authors have already provided the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new method for LiDAR-to-Camera 3D object detection model distillation. They propose distillation along three avenues. First, referencing Barlow Twins, they consider redundancy reduction through cross-correlation between LiDAR and Camera features. Second, they enforce consistency between temporal similarity maps of current lidar-past lidar and current camera-past lidar. Finally, they additionally have distillation at the output level.\n\nSTRENGTHS:\n- The paper is clear and easy to read, with informative diagrams.\n- The motivation for the distillation approaches is sound. I especially appreciate the temporal similarity distillation, as well as its improvement on mAVE.\n- Step-by-step ablations demonstrate the improvement from each proposed component. \n- Analyses in pg 7 demonstrate the redundancy reduction of the core distillation method.\n\nWEAKNESSES:\n- Additional ablations on Temporal Consistency Distillation would be helpful. For instance, perhaps a more natural way to do this could be to generate temporal similarity maps independently for LiDAR and Camera (current lidar x past lidar, current camera x past camera), and enforce consistency between those instead. It is not entirely clear why current camera x past lidar is done instead.\n- Discussion and head-to-head comparison with TiG-BEV with settings shown in the TiG-BEV paper is strongly recommended.\n- As the proposed method considers inter-channel relationships, discussion of the proposed method’s relationship to works that use inter-channel distillation (such as in TiG-BEV) would be appreciated. \n- How are teacher and student outputs matched for the response distillation? Is it greedy, hungarian, or something else?\n\nQUESTIONS:\n- L138: Why is N = X*Y*Z? It instead seemed like N should be X*Y, the flattened BEV feature map. Further, is this done over all the samples in a batch or separately for each BEV teacher-student pair?\n\nLIMITATIONS:\nDiscussed", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}