SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Sujin Jang\*1 Dae Ung Jo\*1 Sung Ju Hwang2 Dongwook Lee1 Daehyun Ji
Samsung Advanced Institute of Technology (SAIT)  
Korea Advanced Institute of Science and Technology (KAIST)  
{s.steve.jang, daeung.jo, dw12.lee, derek.ji}@samsung.com  
sjhwang82@kaist.ac.kr
3D object detection (3DOD) from multi-view images is an economically appealing alternative to expensive LiDAR-based detectors, but also an extremely challenging task due to the absence of precise spatial cues. Recent studies have leveraged the teacher-student paradigm for cross-modal distillation, where a strong LiDAR-modality teacher transfers useful knowledge to a multi-view-based image-modality student. However, prior approaches have only focused on minimizing global distances between cross-modal features, which may lead to suboptimal knowledge distillation results. Based on these insights, we propose a novel structural and temporal cross-modal knowledge distillation (STXD) framework for multi-view 3DOD. First, STXD reduces redundancy of the feature components of the student by regularizing the cross-correlation of cross-modal features, while maximizing their similarities. Second, to effectively transfer temporal knowledge, STXD encodes temporal relations of features across a sequence of frames via similarity maps. Lastly, STXD also adopts a response distillation method to further enhance the quality of knowledge distillation at the output-level. Our extensive experiments demonstrate that STXD significantly improves the NDS and mAP of the based student detectors by  $2.8\% \sim 4.5\%$  on the nuScenes testing dataset.
3D object detection (3DOD) is the task of locating and classifying objects in 3D space using input data from specific modalities, which typically include LiDAR point clouds [61, 14, 7] and camera images [54, 34, 24]. 3DOD is central to understanding the surrounding environment and has been widely applied to various complex vision systems, such as autonomous driving [44], robotic manipulation [71, 65], and augmented reality [42]. Recently, camera-based multi-view 3DOD has emerged as an attractive alternative to expensive LiDAR-based methods, thanks to its ubiquity, low cost, and the semantically rich information from colorized pixels. In general, current works on multi-view 3DOD aim to extract a unified bird's-eye view (BEV) feature map from multiple camera images. However, 3DOD from only camera-specific BEV features is extremely challenging due to the lack of precise spatial cues [39], which leads to inferior performance of camera-based models compared to LiDAR-based models. To bridge the gap, some prior works [16, 11, 19, 29, 9] introduced a teacher-student paradigm for cross-modal knowledge distillation to transfer geometrically-rich 3D information from LiDAR models (teacher) to camera-based 3DOD models (student). The current state-of-the-art 3DOD models typically follow a similar detection paradigm, where modality-specific encoders are used to extract BEV features, followed by a 3D object detection head [29, 34, 55, 7].
*These authors equally contributed to this work
(a) LiDAR
(c) MSE w/ GT
Figure 1: Examples of BEV features generated by (a) the LiDAR model, (b)-(d) the model trained with different feature-level distillation methods, and (e) the camera-only model. Here we visualize the  $\mathcal{L}_2$ -norm value of the BEV features. More examples are provided in the Appendix.
(d) MSE
(e) Camera
Hence, recent works have taken the BEV features from different modalities as the focal point of knowledge distillation [16, 29], and then applied additional response-level distillation [11, 19, 9].
One of the main challenges in cross-modal knowledge distillation is transferring modality-specific knowledge from the teacher to the student in a different modality. This is primarily because the features learned by different modalities are typically non-homogeneous, and there exist distributional divergences between the feature spaces as shown in Fig.1-(a) and (e). Most existing distillation methods aim to minimize global distances between cross-modal features (i.e.,  $\mathcal{L}_2$ -distance). While this approach has achieved some success, it fails to capture the structural knowledge inherent in the modality-specific features, which can lead to suboptimal knowledge distillation, as shown in Fig.1-(c) and (d). To address these challenges, we first propose a distillation method that regularizes cross-correlation of BEV features from the teacher and the student models. In particular, we introduce a decorrelation mechanism [67, 2] to reduce redundancy among feature components, thereby maximizing the information contained in the features. Our structural knowledge distillation approach ultimately prevents information collapse in the student model and enhances the component-wise feature similarities across different modalities, as illustrated in Fig.1-(a) and (b).
We also explore the integration of temporal knowledge embedded in features of previous frames. Although temporal information has been actively exploited in recent 3DOD approaches [34, 29, 32, 41], it has been relatively understudied for cross-modal distillation. The main challenge is the spatiotemporal misalignment between past and current frames. For instance, in driving scenes, foreground objects may have different locations or disappear over time. Thus, directly matching the misaligned cross-modal features could result in incorrect distillation. Instead, we leverage the temporal relations of the BEV features by generating temporal similarity maps between the teacher's current and past frames. Specifically, we train the student model to generate the BEV feature of the current frame, which mimics the temporal relations encoded in the similarity maps from the teacher. In this way, we can avoid false matching issues when transferring temporal information over a sequence of frames. To further improve the quality of transferred knowledge, we also adopt the response distillation approach [10, 59, 19, 9], which assigns quality scores to the predictions of the teacher model, thereby enabling the distillation of only meaningful information at the prediction-level.
Our structural and temporal cross-modal distillation (STXD) framework integrates the proposed feature- and response-level distillation approaches (see Fig.2), and demonstrated a significant improvement on the nuScenes 3DOD benchmark [3], leading to up to  $3.2\%$  NDS and  $4.5\%$  mAP improvement over the based student models. We also conducted detailed ablation studies to validate the effectiveness of each component of STXD. Notably, our framework does not require additional computational cost at testing time, while improving the prediction performance of the student models.
LiDAR-based 3DOD. Recent LiDAR-based 3DOD detectors can be categorized based on the representation types of 3D space: voxel-based and point-based approaches. The voxel-based approaches [69, 58, 27, 48, 14, 7] convert point clouds into a regular voxel-grid representation. Point-based approaches [62, 49, 64, 61] extract point-wise features directly from raw point clouds. Both approaches involve transforming low-level features into a BEV representation, which is then followed by an object detection head that generates response features.
Figure 2: An overview of STXD framework for multi-view 3D object detection. Structural and temporal feature-level distillation is applied to the unified bird's-eye view (BEV) features extracted and transformed from both LiDAR- and camera-based backbones (Sec.3.2 and 3.3). Response-level distillation is applied to the 3D box candidates predicted by both LiDAR- and camera-based detectors (Sec.3.4). During testing, only the camera model is utilized without additional computational cost.
Camera-based 3DOD. Camera-based approaches typically perform 3DOD on either single- or multi-view images. Prior works on single-view 3DOD aim to detect 3D boxes directly from image features [5, 38, 53], or using intermediate geometric representations (e.g., pseudo-lidar [66, 12, 50], probabilistic depth-cues [47, 52]). More recently multi-view 3DOD approaches [21, 33, 29] use the lift-spat-shoot method [43, 47] to transform multi-image features into a unified bird's-eye view (BEV) representation followed by an object detection head. Other works on multi-view images [54, 34, 35, 24] aim to implicitly learn query-based BEV features sampled from corresponding 2D image features based on attention mechanisms [4, 72]. While some of these works [50, 47, 33] utilize LiDAR point clouds as input for explicit supervision of depth cues, they do not fully exploit the potential of LiDAR-based 3DOD features, which have appeared to be effective for improved prediction performance of camera-based models. In light of this work, we further provide new perspectives on the structural and temporal knowledge embedded in the LiDAR-based BEV features.
Multi-Modal 3DOD. Recently, integrating complementary information from different modality sensors, especially LiDAR and cameras, has demonstrated improved 3DOD performance. One line of work fuses intermediate feature representations in the BEV space [8, 31, 36, 63], while other approaches try to fuse object proposals from each modality in the detection head [25, 6, 1]. Another line of work augments point clouds with semantic information extracted from corresponding images [51, 51, 57]. However, these methods require both sensors to be available during test time. In contrast, our cross-modal distillation approach leverages both modalities only during training to transfer useful knowledge from LiDAR-based to camera-based 3DOD models. During testing, we use only multi-camera images without any additional computational costs.
Knowledge Distillation for 3DOD. Most prior works on knowledge distillation for object detection have focused on transferring knowledge among single-modal detectors, particularly for 2D object detection (e.g., [15, 59]) and LiDAR-based models (e.g., [60, 56]). Relatively fewer works have studied cross-modal distillation for object detection. MonoDistill [11] projects LiDAR point clouds into perspective image space to distill spatially aligned LiDAR features to a camera-based student model. LIGA-Stereo [16] aims to transfer intermediate features from LiDAR to a stereo camera-based student model. CMKD [19] and BEVDistill [9] employ knowledge distillation from LiDAR-based models at both the feature and response levels to minimize the gap between modalities. Similarly, UVTR [29] tries to directly match the BEV representations of the camera model to the LiDAR-based
teacher. Although these methods have achieved some success, they still do not carefully consider the distributional divergences between non-homogeneous modalities. To mitigate the modality gap between LiDAR and camera features, TIG-BEV [22] learns the inner geometry of foreground objects by encouraging the student's features to replicate the self-correlation of the teacher's features. However, this method still does not consider cross-modal associations among BEV features from the teacher and student models. Differs from previous methods, our distillation approach considers structural knowledge in the features by introducing cross-modal cross-correlation regularization inspired by the decorrelation mechanism [67, 2]. Furthermore, we introduce a method for effectively distilling temporal information contained in previous frames of the teacher. We also utilize response-level distillation to further improve the quality of knowledge distillation at the predictions level.
Recent 3D object detectors share a similar detection paradigm, where low-level features are extracted from modality-specific encoders and subsequently transformed into BEV features (see Fig.2). For example, modern LiDAR-based detectors [69, 27, 55] use 3D encoders to extract low-level features from the raw point clouds, then the features are further transformed to the BEV features  $\mathbf{F} \in \mathbb{R}^{X \times Y \times Z \times D}$  with spatial extent of  $X \times Y \times Z$  and the number of channels  $D$ . For recent multi-view image detectors, there are two mainstream approaches for transforming the low-level image features to the BEV features: (1) estimating depth cues [21, 33, 29] via lift-splat-shoot [43] and (2) implicit learning of depth via cross-attention mechanism [35, 34]. Regardless of the type of transform methods, the BEV features can be represented by  $\mathbf{G} \in \mathbb{R}^{X \times Y \times Z \times D}$ , similar to the LiDAR-based detectors. In this work, as illustrated in Fig.2, we aim to train a high-performing camera-based detector (student) by distilling useful knowledge from a LiDAR-based detector (teacher) while considering the structural and temporal information embedded in the modality-specific BEV featuers,  $\mathbf{F}$  and  $\mathbf{G}$ .
3.2 Correlation Regularizing Distillation
Prior works on cross-modal distillation directly force the student to mimic the BEV feature from the teacher by calculating element-wise distance loss (i.e.,  $||\mathbf{F} - \mathbf{G}||_2$ , [19, 29]), or weighted via foreground mask (i.e.,  $||\mathbf{W}_{fg}\cdot (\mathbf{F} - \mathbf{G})||_2$ , [11, 9]). However, they ignore the structural knowledge inherent in modality-specific features and may not effectively distill rich information contained in the features. Recently, several self-supervised learning methods have incorporated cross-correlation regularization to maximize the information contained in features [67, 2]. These methods can also be applied to the problem of LiDAR-to-camera knowledge distillation, since LiDAR and camera features can be regarded as different representations obtained from the same scene, and have a spatial one-to-one correspondence. From this intuition, we propose the Correlation Regularizing Distillation (CD) loss that maximizes the similarity between aligned features while regularizing cross-correlation along feature dimensions. Let batches of  $N$  aligned  $D$ -dimensional LiDAR and camera features be given as  $\mathbf{F} = [\mathbf{f}_1,\mathbf{f}_2,\dots,\mathbf{f}_N]^T\in \mathbb{R}^{N\times D}$  and  $\mathbf{G} = [\mathbf{g}_1,\mathbf{g}_2,\dots,\mathbf{g}_N]^T\in \mathbb{R}^{N\times D}$ , respectively.  $N$  is the number of serialized BEV features, given by  $X\cdot Y\cdot Z$ . Following [67], each feature is transformed to be mean-centered along the batch dimension, denoted by  $\hat{\mathbf{F}}$  and  $\hat{\mathbf{G}}$ . Then, CD loss is defined as:
$$
\mathbf {C} = \hat {\mathbf {F}} ^ {T} \hat {\mathbf {G}} \in \mathbb {R} ^ {D \times D}, \tag {1}
$$
$$
\mathcal {L} _ {C D} := \sum_ {i} (1 - \mathbf {C} (i, i)) ^ {2} + \lambda_ {c} \sum_ {i} \sum_ {j \neq i} \mathbf {C} (i, j) ^ {2}, \tag {2}
$$
where  $\mathbf{C}$  denotes a dimensional cross-correlation matrix and  $\lambda_{c} \geq 0$  is a balancing hyper-parameter. The first term of Eq.2 encourages the aligned LiDAR and camera feature components to be similar, while the second term reduces the correlation among feature components that are not aligned. Hence, CD loss can reduce duplicated information in the learned features, allowing for the student to learn diverse information from the teacher. We analyzed the effectiveness of the CD loss in Sec.4.2.
3.3 Temporal Consistency Distillation
In driving scenes, sensor data is received sequentially. Therefore, the past few frames may contain valuable information for detecting objects in the current frame, and incorporating temporal information has brought significant improvements to 3DOD [34, 29, 32, 41]. However, directly learning
from the LiDAR features of the past frames (i.e., via  $\mathcal{L}_2$ -distance) may not be effective due to the spatially and temporally misaligned non-homogeneous features. To address this issue, we propose the Temporal Consistency Distillation (TD) loss, which indirectly distills the teacher's information from past frames by introducing a temporal similarity map. Let batches of aligned LiDAR and camera features of the current frame be given as  $\mathbf{F}^{(0)}$  and  $\mathbf{G}^{(0)}$ , respectively. We are also given the LiDAR features of the  $k$ -th previous frame, denoted by  $\mathbf{F}^{(-k)}$ ,  $k \in [1, K]$ , where  $K$  is the maximum number of past frames referred to. Then, the cross-modal temporal similarity map  $\mathbf{S}^{(-k)}$  and the intra-modal temporal similarity map  $\mathbf{T}^{(-k)}$  between the current and the  $k$ -th previous frame are calculated as:
$$
\mathbf {S} ^ {(- k)} = \mathbf {G} ^ {(0)} \mathbf {F} ^ {(- k) ^ {T}} \in \mathbb {R} ^ {N \times N}, \quad \mathbf {T} ^ {(- k)} = \mathbf {F} ^ {(0)} \mathbf {F} ^ {(- k) ^ {T}} \in \mathbb {R} ^ {N \times N}, \quad k \in [ 1, K ]. \tag {3}
$$
Then TD loss is defined as:
$$
\mathcal {L} _ {T D} := \sum_ {k} D _ {K L} \left(\mathbf {S} ^ {(- k)} \| \mathbf {T} ^ {(- k)}\right), \tag {4}
$$
where  $D_{KL}$  is the Kullback-Leibler divergence (KLD). Since  $\mathbf{S}^{(-k)}$  and  $\mathbf{T}^{(-k)}$  are matrices, we apply a softmax function to each row of them, and then calculate the KLD for each row. Based on the temporal similarity map, TD loss distills the teacher's information of past frames indirectly, avoiding spatial false matching issues across the time frames. We also demonstrated and analyzed the effectiveness of TD in Sec.4.2.
3.4 Response-Level Distillation
In general, detector models infer a large number of candidates, but not all candidates contain meaningful information. Since only a few candidates detect target objects, it is necessary for a student to selectively learn from the predictions of the teacher. Inspired by recent works on prediction-guided distillation [10, 59, 19, 9], we define the Response-Level Distillation (RD) loss that assigns quality scores to the predictions of the teacher model, enabling the distillation of only meaningful information. Let us define  $\{(\mathbf{b}_i,\mathbf{c}_i)\}_{i = 1}^M$  as prediction results of teacher model, where  $\mathbf{b}_i$  and  $\mathbf{c}_i$  are regression and classification results for the  $i$ -th of  $M$  box candidates, respectively. In a similar manner, student prediction can be defined as  $\{(\tilde{\mathbf{b}}_j,\tilde{\mathbf{c}}_j)\}_{j = 1}^M$ . Following [10, 59], quality score  $q_{i}$  for  $i$ -th candidate of the teacher predictions can be calculated as follow:
$$
q _ {i} = \left(c _ {i} ^ {*}\right) ^ {1 - \gamma} \cdot \left(\operatorname {I o U} \left(\mathbf {b} _ {i} ^ {*}, \mathbf {b} _ {i}\right)\right) ^ {\gamma}, \tag {5}
$$
where  $c_{i}^{*}$  is the predicted probability for the class of the ground truth box that is matched to  $i$ -th candidate of the teacher predictions,  $\mathbf{b}_{i}^{*}$  is a regression parameter of the ground truth box that is matched to  $i$ -th candidate of the teacher predictions, and  $\gamma$  is a hyper-parameter balancing classification and localization. Then RD loss is defined as:
$$
\mathcal {L} _ {R D} := \sum_ {j} q _ {\pi (j)} \cdot \left(\| \mathbf {b} _ {\pi (j)} - \tilde {\mathbf {b}} _ {j} \| _ {1} + D _ {K L} \left(\mathbf {c} _ {\pi (j)} \| \tilde {\mathbf {c}} _ {j}\right)\right), \tag {6}
$$
where  $\pi (j)$  is a function that returns the index of the corresponding teacher's prediction matched with the  $j$ -th student's prediction. By combining RD with our structural and temporal distillation (CD,TD), we achieved additional improvement on the detection performance as reported in Sec.4.2.
3.5 Optimization Objective
Finally, the overall optimization objective for the student models can be obtained by integrating the aforementioned three distillation losses and the task loss  $\mathcal{L}_{task}$ :
$$
\mathcal {L} _ {\text {t o t a l}} = \mathcal {L} _ {C D} + \mathcal {L} _ {T D} + \mathcal {L} _ {R D} + \mathcal {L} _ {\text {t a s k}}, \tag {7}
$$
where  $\mathcal{L}_{task}$  is adopted from a general set-to-set target assignment for the student with ground truth, including regression and classification losses [54], based on the Hungarian algorithm [26].
We now present extensive experimental results to validate our structural and temporal cross-modal knowledge distillation framework (STXD) for 3D object detection (3DOD) task. We first introduce the details of experimental setting (Sec.4.1) followed by various ablation studies to validate the effectiveness of each component in STXD (Sec.4.2). Then, we compare STXD against several leading methods on the nuScenes [3] dataset (Sec.4.3). Additional experiments are provided in the Appendix.
Dataset. We evaluate our STXD framework on the nuScenes [3] benchmark dataset, which is large-scale and widely used by recent research works on 3D object detection. The nuScenes dataset comprises 1,000 driving scenes with a duration of approximately 20s, which are divided into 700, 150, and 150 scenes for training, validation, and testing, respectively. Each scene consists of RGB images collected by six cameras, which capture a  $360^{\circ}$  field of view at  $12\mathrm{Hz}$  and 32-beam LiDAR point clouds at  $20\mathrm{Hz}$ . Annotations for 23 categories of objects are provided for each sample at a rate of  $2\mathrm{Hz}$ , and the metrics for the 3DOD task are evaluated using 10 classes of objects, as defined by the official evaluation metrics. These metrics include mean Average Precision (mAP) and nuScenes Detection Score (NDS). In detail, mAP measures the recall and precision of predicted bounding boxes, while NDS is a weighted sum of mAP and five true positive nuScenes metrics: mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), and mean Average Attribute Error (mAAE).
Baseline Models. To validate the effectiveness of our approach, we adopt two state-of-the-art cross-modal distillation approaches as baselines. Similar to concurrent work [9], we use BEVFormer [34] as the image-modality student model and Object-DGCNN [55] as the LiDAR-modality teacher model, where the DGCNN attention module of the transformer-decoder is replaced with vanilla multi-head attention. For the second baseline, we adopt the multi-view camera (single frame of UVTR-C, multi-frame of UVTR-CS) and LiDAR (UVTR-L) models from UVTR [29] as the student and teacher models, respectively. Our base student models represent two mainstream methods for transforming 2D perspective image features into 3D BEV features, where BEVFormer uses cross-attention mechanism [4, 72] and UVTR-C/CS is based on estimating depth distributions through lift-splat-shoot approach [43, 47].
Implementation Details. The input image size for the student models is set to  $1600 \times 900$ . For both teacher and student models, we use a voxel size of  $(0.1m, 0.1m, 0.2m)$ , and the input point clouds are filtered within a range of  $[-51.2m, 51.2m]$  for the X and Y axes and  $[-5.0m, 3.0m]$  for the Z axis. The dimensions of the BEV feature grids are  $128 \times 128$ , and the number of channel  $D$  for BEV features is set to 256. We train all student models for 24 epochs without using CBGS [70].  $\lambda_{c}$  in Eq.2 is set to 0.01 according to [67]. More implementation details are provided in the Appendix.
For our ablation studies, we implemented BEVFormer [34] with ResNet-101-DCN as backbone network for the student model, and evaluated the trained model on the nuScenes validation dataset.
Ablation on Distillation Methods. To validate the effects of each proposed distillation approach in Sec.3, we conducted an ablation study on the loss function. As shown in Tab.1, each loss term contributes to the improvement in performance. Specifically, the proposed feature-level distillation (CD and TD) plays a crucial role in enhancing the performance. When CD and TD losses are applied solely, NDS is increased by  $2.47\%$  and  $1.77\%$ , respectively. When all loss terms are combined, the performance is further improved without any conflicts, resulting in the best performance with a significant improvement of NDS by  $2.87\%$  and mAP by  $3.52\%$ . These results demonstrate the effectiveness of our structural and temporal distillation as well as the response distillation method.