SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Linyan Huang<sup>1</sup> Zhiqi Li<sup>2</sup> Chonghao Sima<sup>1</sup> Wenhai Wang<sup>3</sup> Jingdong Wang<sup>4</sup> Yu Qiao<sup>1</sup> Hongyang Li<sup>1</sup>
$^{1}$ Shanghai AI Lab  $^{2}$ Nanjing University  $^{3}$ CUHK  $^{4}$ Baidu
Current research is primarily dedicated to advancing the accuracy of camera-only 3D object detectors (apprentice) through the knowledge transferred from LiDAR- or multi-modal-based counterparts (expert). However, the presence of the domain gap between LiDAR and camera features, coupled with the inherent incompatibility in temporal fusion, significantly hinders the effectiveness of distillation-based enhancements for apprentices. Motivated by the success of uni-modal distillation, an apprentice-friendly expert model would predominantly rely on camera features, while still achieving comparable performance to multi-modal models. To this end, we introduce VCD, a framework to improve the camera-only apprentice model, including an apprentice-friendly multi-modal expert and temporal-fusion-friendly distillation supervision. The multi-modal expert VCD-E adopts an identical structure as that of the camera-only apprentice in order to alleviate the feature disparity, and leverages LiDAR input as a depth prior to reconstruct the 3D scene, achieving the performance on par with other heterogeneous multi-modal experts. Additionally, a fine-grained trajectory-based distillation module is introduced with the purpose of individually rectifying the motion misalignment for each object in the scene. With those improvements, our camera-only apprentice VCD-A sets new state-of-the-art on nuScenes with a score of  $63.1\%$  NDS. The code will be released at https://github.com/OpenDriveLab/Birds-eye-view-Perception.
The camera-only 3D perception has garnered increasing attention in autonomous driving perception tasks [31, 21, 47, 50]. Although camera-only models possess the advantages of low deployment cost and ease of widespread application, they still fall behind state-of-the-art models that leverage LiDAR sensors regarding perception accuracy. Researchers have recently employed distillation methods to transfer knowledge from a powerful expert model into a camera-only apprentice model, with the expectation of leveraging the expertise of these stronger expert models to enhance the capability of the camera-only models. Existing 3D perception distillation methods often adopt expert models with the best performance, such as LiDAR-based models [60] or multi-modal fusion models [2, 59, 33]. However, the presence of the domain gap between LiDAR and camera features hampers knowledge transfer during distillation, resulting in limited improvements in practical applications. An alternative expert model is the large-scale camera-only model [32, 61]. Despite eliminating the domain gap between the camera-only expert model and the apprentice model, the expert model falls short in terms of effectiveness due to the inherent lack of precise geometry information. Likewise, it fails to yield a satisfactory improvement to the apprentice model. Hence, a desirable expert should meet two essential requirements: attaining state-of-the-art performance and minimizing the domain gap.
Furthermore, current distillation methods fall short in compatibility with long-term temporal fusion, which is an essential component in cutting-edge camera-only 3D detectors [18, 43]. Long-term
(a) Comparison of various multimodal fusion methods.
(b) The misalignment of moving objects.
Figure 1: (a) The existing pipelines require camera and LiDAR backbones, while our pipeline eliminates the need for LiDAR. Using point cloud depth, we directly transform image features into BEV space to create a vision-centric expert. (b) The warping of an object in the historical frame into the current timestamp results in a false position in the current frame due to assuming the object is stationary. The green rectangle represents true positives, while the pink rectangle indicates false positives.  $x_{i}$  denotes the various positions of the object in the historical timestamp.
temporal modeling has shown considerable potential in enhancing the accuracy of depth estimation and detection performance, but it introduces the issue of motion misalignment. Previous methods for BEV distillation have followed two distinct approaches, either distilling the entire BEV space without sufficient attention to the foreground objects [44] or exclusively distilling the foreground object regions [11, 25], thereby overlooking the motion misalignment issue resulting from the long-term temporal fusion. As shown in Fig. 1 (b), this misalignment occurs when past scenes are transformed into the current scene coordinates based solely on ego-motion, assuming all objects are stationary. While in reality, dynamic objects will cause the misalignment, thus interfering with the temporal fusion features. This is more challenging in the case of long-term temporal fusion. Existing methods, such as StreamPETR [51], introduce LayerNorm [1] for dynamic object modeling, but the effects of incorporating velocity and time variables in the model are relatively minor.
To address the aforementioned challenges, we first propose a vision-centric expert, termed as VCD-E, which incorporates LiDAR information to enhance the accuracy of depth input. In this context, the term "vision-centric" refers to the utilization of prominent features derived from camera input, distinguishing it from approaches that heavily rely on LiDAR-based features. This model is distinct from conventional multi-modality fusion techniques by eliminating LiDAR backbone. By solely integrating LiDAR depth and long-term temporal fusion under bird's-eye-view (BEV), our model achieves comparable performance to state-of-the-art multi-modal fusion methods [38] by only encoding image modality. As illustrated in Fig. 1 (a), different from previous fusion methods that adopt two modality-specific backbones, we only leverage a single branch to generate semantic features based on image input, and point clouds only provide depth information. Compared to previous fusion methods, our approach eliminates the need for intricate training strategies or specialized fusion module designs while reaching a comparable performance to current fusion methods [38, 33]. More importantly, the vision-centric multi-modal model has the exact same architecture as the camera-only apprentice model, and the generated BEV spatial representation is solely from image features. The domain gap is significantly alleviated through the distillation of knowledge from the proposed multimodal expert model to the camera-only apprentice model. Due to the advantageous characteristic of domain consistency, our apprentice model acquires substantial benefits from the expert, surpassing previous distillation methods [11, 46].
To mitigate the incompatibility arising from the motion misalignment of dynamic objects, we further propose a trajectory-based distillation module. In this paper, our primary focus revolves around foreground objects, while simultaneously incorporating a meticulous consideration of their historical trajectories. Specifically, by warping dynamic targets from history to the current frame, we derive the motion trajectory associated with each individual object. Then we use the trajectory of each object to query BEV features of the apprentice and expert models respectively. By leveraging the trajectory features of the expert model to optimize the corresponding features of the apprentice model, the latter can acquire the ability to mitigate the interference arising from motion misalignment. In addition, to
enhance the depth perception ability, we diffuse the depth of the foreground part into the 3D space, modeling occupancy to obtain grid-based supervision to assist in depth prediction for objects.
In summary, we propose the multi-modality expert and camera-only apprentice models, termed as VCD-E and VCD-A, respectively. Our contributions are summarized as follows:
- We construct a vision-centric multi-modal expert that solely encodes the image modality, eliminating the need for a LiDAR backbone. For the first time, we demonstrate that the expert can deliver performance on par with other state-of-the-art multi-modal methods while being significantly simpler.
- Due to its homogeneous characteristics and superior performance, the vision-centric expert has been proven effective in distilling knowledge to vision-based models. The effects are significant across a range of model sizes, from compact to more extensive architectures.
- We propose trajectory-based distillation and occupancy reconstruction modules, which supervise both static and dynamic objects to alleviate misalignment during long-term temporal fusion. Combined with the constructed expert model, we enhance the performance of the vision-based models and achieve state-of-the-art on the nuScenes val and test leaderboard.
In this section, we review previous studies in the areas of 3D object detection, multi-modality fusion, knowledge distillation, focusing on the techniques and methods most relevant to our research.
3D Object Detection. 3D object detection has recently gained significant popularity in the context of autonomous driving and robotics. Detection methods generally fall into two categories: vision-based 3D object detection [64, 65, 5, 37, 52, 36, 53, 58, 22, 56] and LiDAR-based 3D detection [30, 60, 45]. Vision-based approaches [34, 54, 15, 35] exploit image information and frequently involve deep learning techniques to estimate depth. In contrast, LiDAR-based methods capitalize on precise geometric information from LiDAR sensors to achieve superior object detection accuracy. Our proposed vision-centric expert shares the same modality as vision-based detectors, while exhibiting superior performance compared to LiDAR-based detectors.
Long-term modeling has been employed to improve the performance of 3D object detection models [18, 43]. SOLOFusion [43] utilizes long-term temporal modeling to achieve excellent performance. VideoBEV [18] maintains comparable performance with SOLOFusion while being more efficient, using long-term recurrent temporal modeling. However, previous research [34, 51] has highlighted that long-term temporal fusion can lead to inadequate detection of dynamic objects. Although StreamPETR [51] proposes the propagation transformer [48, 7, 9, 8] to conduct object-centric temporal modeling, the improvement of dynamic object modeling remains relatively modest. Our proposed trajectory-based distillation module alleviates this limitation, enabling accurate detection of both static and dynamic objects by camera-based 3D object detection models that utilize long-term modeling.
Multi-modality Fusion. Multi-modality fusion techniques [49, 2, 33, 16] have been extensively investigated to enhance 3D object detection performance by integrating complementary information from different sensor modalities, such as cameras and LiDAR sensors. These methods [38, 59] typically require complex training strategies and the development of specialized fusion modules to effectively merge the distinct sources of information. In contrast, we propose a streamlined architecture that utilizes an image backbone for feature extraction, obviating the need for a LiDAR backbone. This efficient approach augments vision-based models by incorporating LiDAR information, maintaining homogeneity with vision-based models while preserving exceptional performance.
Knowledge Distillation. Knowledge distillation [44, 46] is a technique facilitating the transfer of knowledge from a larger, more complex model (expert) to a smaller, more efficient model (apprentice). This approach has been successfully applied in various domains, including image classification [14], natural language processing [48], and policy learning [55, 27, 26, 6]. Therefore, recent distillation methods [17, 25, 32, 24, 29] build upon 3D object detection aims to transfer the accurate geometry knowledge from LiDAR to camera. MonoDistill [12] projects the LiDAR points into the image plane to serve as the expert to transfer knowledge. BEVSimDet [62] simulates fusion-based methods to alleviate the domain gap between the two different modalities. BEVDistill [11] projects the LiDAR points and images into the BEV space to align the LiDAR feature and image feature. Due to the
non-homogenous nature between the LiDAR and camera, transferring knowledge from LiDAR to images is challenging. Instead, our work constructs a vision-centric expert, which possesses a homogeneous modality with vision-based models. The vision-centric expert can leverage knowledge distillation to transfer the geometric perception capabilities to various vision-based models, hence enhancing performance accordingly.
In this section, we present our approach in detail. The overall architecture is presented in Sec. 3.1. Our method involves two main components: (1) the vision-centric expert in Sec. 3.2, and (2) the trajectory-based distillation and occupancy reconstruction modules as elaborated in Sec. 3.3. The pipeline of our method is depicted in Fig. 2.
3.1 Overall Architecture
In this paper, we construct a pair of harmonious expert and apprentice models. The expert and apprentice models adopt the consistent model architecture. The only difference is that the expert additionally leverages the accurate depth map generated from the point cloud, while the apprentice model predicts the depth map from the image. Although our expert model only uses an image backbone to encode high-level scene information, it is on par with state-of-the-art multi-model fusion methods that use several modality-specific backbones and complex interaction strategies. More importantly, we eliminate the domain gap between the multi-model expert and the camera-only apprentice model, which is deemed as one of the most challenging topics in the cross-modality distillation literature.
As illustrated in Fig. 2, we construct a distillation framework between the expert network and the apprentice network. The vision-centric expert fuses feature  $\boldsymbol{I}^{E}$  extracted from the image backbone and the temporal depth map  $D$  projected from LiDAR points to create a unified BEV representation  $\boldsymbol{F}^{E}$  used for 3D object detection. Therefore, although we adopt a cross-modality approach for 3D object detection, the resulting representation remains homogeneous with image modality features.
After obtaining the pretrained vision-centric expert and corresponding apprentice network, we freeze the expert network and leverage its intermediate features as auxiliary supervision for the apprentice network. Since current advanced vision-based detectors employ long-term temporal modeling to attain state-of-the-art performance, we utilize a standard long-term temporal vision-based detector based on BEVDepth as the apprentice model in our context. The expert model also utilizes long-term temporal modeling to ensure consistency and achieve higher performance.
3.2 The Generation of Expert Model
We construct a vision-centric expert by integrating LiDAR information as accurate depth input into a vision-based model. Nevertheless, given the sparsity of LiDAR depth data, we rely on the predicted depth values obtained from images for pixels lacking LiDAR depth information. We also utilize future frames to further improve the performance of the expert model in the offline 3D detection setting. The vision-based detector serves as the primary model, and LiDAR information complements it by providing precise depth information. This approach eliminates the need for complex training strategies or custom-designed fusion modules, streamlining the fusion process.
For the expert model, we project the last sweep LiDAR frame with the current LiDAR frame onto images to obtain corresponding depth maps  $D$ . Since the depth maps  $D$  generated from point clouds can not cover every pixel of the images, we also predict depth distribution for each pixel based on the image features. Then we will project the image features onto the BEV space to obtain BEV features  $F^{E}$  based on their depth. Furthermore, we transform the BEV features  $F_{T-N}^{E}$  from previous timestamps to the current BEV features  $F_{T}^{E}$ , modeling the long-term relationship.  $N$  denotes the time interval between the current frame and the history frame. The unified BEV features  $F^{E}$  are then combined to produce 3D object detection predictions  $P$ . The model is trained using a multi-task loss function that considers both 3D detection loss  $\mathcal{L}_{Det}$  and depth estimation loss  $\mathcal{L}_{Depth}$ .
Figure 2: Algorithm Overview. Expert utilizes LiDAR data to enhance depth estimation accuracy before view transformation in BEV pipeline. Apprentice represents a standard long-term vision-based detection model. The occupancy is built using the depth information from the expert model, which serves as the supervision for the apprentice model. Motion trajectory is constructed by warping the time series GT query into the current timestamp. Projecting the motion trajectory of each object into BEV space can rectify the misalignment of object motion. With the knowledge transferred from the expert, the apprentice can deliver higher performance than before.
3.3 The Procedure of Distillation
In this section, we elucidate the methodology adopted to overcome the constraints inherent in long-term modeling for multi-camera 3D object detection. This is achieved through the incorporation of two innovative modules within the distillation process: the trajectory-based distillation module and the occupancy reconstruction module.
Trajectory-based Distillation. The trajectory-based distillation module aims to improve the detection of dynamic objects by focusing on the inconsistent portion of the objects' motion. For  $i$ -th historical frame at timestamp  $t_i$  that contains  $K$  objects, extract the  $j$ -th ground truth object position  $\pmb{P}_i^j = (x_i^j, y_i^j, z_i^j, 1)^T$  in the ego coordinate system. Determine the actual ego motion matrix  $M_i$  between the current frame  $t_0$  and each historical frame  $t_i$ . Apply the ego-motion transformation matrix  $M_i$  to the ground truth object positions  $\pmb{P}_i^j$  to obtain the transformed positions  $\pmb{P}_i^{j'}$  in the current frame's coordinate system:
$$
\boldsymbol {P} _ {i} ^ {j ^ {\prime}} = \boldsymbol {M} _ {i} \times \boldsymbol {P} _ {i} ^ {j}. \tag {1}
$$
We amalgamate the transformed ground truth object positions,  $P_{i}^{j^{\prime}}$ , from all historical frames to construct the motion trajectories. The trajectory of  $j$ -th object can thus be represented as a sequence of object positions within the current frame as  $(P_{1}^{j^{\prime}}, P_{2}^{j^{\prime}}, \ldots, P_{N}^{j^{\prime}})$ , where  $N$  is the number of points on each motion trajectory. Let  $F_{ij}^{E}$ ,  $F_{ij}^{A}$  represent the sampled features on identical point  $P_{i}^{j^{\prime}}$  from the expert BEV feature  $F^{E}$  and apprentice BEV feature  $F^{A}$ , respectively. They are sampled via bilinear interpretation and then are normalized as:
$$
\boldsymbol {F} _ {i j} ^ {E} = \operatorname {n o r m} \left(\boldsymbol {F} ^ {E} \left(P _ {i} ^ {j ^ {\prime}}\right)\right), \quad \boldsymbol {F} _ {i j} ^ {A} = \operatorname {n o r m} \left(\boldsymbol {F} ^ {A} \left(P _ {i} ^ {j ^ {\prime}}\right)\right). \tag {2}
$$
Finally, the trajectory-based distillation loss  $\mathcal{L}_{TD}$  is computed between the normalized key sampled features:
$$
\mathcal {L} _ {T D} = \frac {1}{K \times N} \sum_ {j = 1} ^ {K} \sum_ {i = 1} ^ {N} L _ {2} \left(\boldsymbol {F} _ {i j} ^ {A}, \boldsymbol {F} _ {i j} ^ {E}\right). \tag {3}
$$
Ultimately, by using the motion trajectory as queries, we conduct trajectory-based distillation on these representative positions. This approach enables the expert to rectify the motion misalignment in the apprentice.
Table 1: Comparison among the camera-only methods on the nuScenes val set. * denotes the long-term baseline implemented by us based on BEVDet4D-Depth [22]. † depicts that the size of BEV feature is  $256 \times 256$ . VCD-A surpasses previous SOTA by 2 points in NDS and achieves SOTA under the same setting.
Methods	Backbone	Image Size	Frames	mAP↑	NDS↑	mATE↓	mASE↓	mAOE↓	mAVE↓	mAAE↓
BEVDet [23]	ResNet-50	256 × 704	1	0.298	0.379	0.725	0.279	0.589	0.860	0.245
PETR [39]	ResNet-50	384 × 1056	1	0.313	0.381	0.768	0.278	0.564	0.923	0.225
BEVDet4D [22]	ResNet-50	256 × 704	2	0.322	0.457	0.703	0.278	0.495	0.354	0.206
BEVDepth [35]	ResNet-50	256 × 704	2	0.351	0.475	0.639	0.267	0.479	0.428	0.198
BEVStereo [34]	ResNet-50	256 × 704	2	0.372	0.500	0.598	0.270	0.438	0.367	0.190
STS [54]	ResNet-50	256 × 704	2	0.377	0.489	0.601	0.275	0.450	0.446	0.212
VideoBEV [19]	ResNet-50	256 × 704	8	0.422	0.535	0.564	0.276	0.440	0.286	0.198
SOLOFusion [43]	ResNet-50	256 × 704	16+1	0.427	0.534	0.567	0.274	0.411	0.252	0.188
StreamPETR [51]	ResNet-50	256 × 704	8	0.432	0.540	0.581	0.272	0.413	0.295	0.195
Baseline*	ResNet-50	256 × 704	8+1	0.401	0.515	0.595	0.279	0.489	0.291	0.198
VCD-A	ResNet-50	256 × 704	8+1	0.426	0.540	0.547	0.271	0.433	0.268	0.207
Baseline*†	ResNet-50	256 × 704	8+1	0.418	0.542	0.522	0.267	0.428	0.262	0.188
VCD-A†	ResNet-50	256 × 704	8+1	0.446	0.566	0.497	0.260	0.350	0.257	0.203
Occupancy Reconstruction. The expert model demonstrates outstanding performance in 3D object detection, resulting in a more precise 3D geometric representation of the objects. We utilize depth estimation modules to predict the depth, denoted as  $D(u,v)$ , for each image pixel  $(u,v)$ . Subsequently, the depth map  $D$  is back-projected into a 3D point cloud, each image pixel  $(u,v)$  is transformed into a 3D coordinate  $(x,y,z)$ . Then the scores from different pixels in the multi-camera within the same occupancy region, are accumulated to make decisions about the presence of an object in that region. In this way, we can generate the occupancy  $O_e$  and  $O_a$  for expert and apprentice model, respectively.
The occupancy reconstruction module improves the model's capability to discern the 3D geometric properties of objects. The grid-based supervisory signals effectively direct the model to enhance its prediction accuracy for object depths. Different voxels within the occupancy structure aggregate the fused depth distribution from various perspective views, thereby enhancing robustness against depth errors. Drawing inspiration from CenterPoint [63], we simply extend the Gaussian distribution applied to each target into the 3D space for more focused 3D object modeling:
$$
\boldsymbol {G} _ {x y z} = \exp \left[ - \frac {\left(x - \tilde {p} _ {x}\right) ^ {2} + \left(y - \tilde {p} _ {y}\right) ^ {2} + \left(z - \tilde {p} _ {z}\right) ^ {2}}{2 \sigma_ {p} ^ {2}} \right], \tag {4}
$$
where  $(\tilde{p}_x,\tilde{p}_y,\tilde{p}_z)$  represents the center of the 3D object, while  $\sigma_{p}$  denotes the standard deviation of each object's size. The model utilizes the expert model's occupancy  $O_{e}$  as supplementary supervision by adopting a straightforward  $\mathcal{L}_1$  regularization loss for the occupancy status  $O_{a}$  of the apprentice model, which optimizes depth prediction capabilities for both static and dynamic objects. The occupancy reconstruction loss can be formulated as
$$
\mathcal {L} _ {O R} = \mathcal {L} _ {1} \left(\boldsymbol {G} _ {x y z} \cdot \boldsymbol {O} _ {e}, \boldsymbol {G} _ {x y z} \cdot \boldsymbol {O} _ {a}\right). \tag {5}
$$
Training Loss. During the distillation phase, the joint training loss  $\mathcal{L}_{Total}$  is formulated as
$$
\mathcal {L} _ {\text {T o t a l}} = \mathcal {L} _ {A} + \lambda_ {1} \cdot \mathcal {L} _ {T D} + \lambda_ {2} \cdot \mathcal {L} _ {O R}, \tag {6}
$$
where  $\mathcal{L}_A$  is the perceptual loss of the apprentice model. Besides,  $\lambda_{1}$  and  $\lambda_{2}$  represent hyperparameters employed to effectively balance the scales of the respective loss functions. The utilization of trajectory-based distillation loss  $\mathcal{L}_{TD}$  and occupancy reconstruction loss  $\mathcal{L}_{OR}$  collectively facilitates the transfer of semantic and geometry knowledge from the expert model to the apprentice model.