Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection
Authors: Linyan Huang
Zhiqi Li
Chonghao Sima
Wenhai Wang
Jingdong Wang
Yu Qiao
Hongyang Li
Venue: NeurIPS 2023 poster (2023)
Keywords: camera-only detection
multi-modal distillation
multi-view object detection
TL;DR: This paper introduces a novel framework that enhances camera-only 3D object detection by implementing an apprentice-friendly multi-modal expert and a temporal-fusion-friendly distillation strategy.

Abstract: Current research is primarily dedicated to advancing the accuracy of camera-only 3D object detectors (apprentice) through the knowledge transferred from LiDAR- or multi-modal-based counterparts (expert). However, the presence of the domain gap between LiDAR and camera features, coupled with the inherent incompatibility in temporal fusion, significantly hinders the effectiveness of distillation-based enhancements for apprentices. Motivated by the success of uni-modal distillation, an apprentice-friendly expert model would predominantly rely on camera features, while still achieving comparable performance to multi-modal models. To this end, we introduce VCD, a framework to improve the camera-only apprentice model, including an apprentice-friendly multi-modal expert and temporal-fusion-friendly distillation supervision. The multi-modal expert VCD-E adopts an identical structure as that of the camera-only apprentice in order to alleviate the feature disparity, and leverages LiDAR input as a depth prior to reconstruct the 3D scene, achieving the performance on par with other heterogeneous multi-modal experts. Additionally, a fine-grained trajectory-based distillation module is introduced with the purpose of individually rectifying the motion misalignment for each object in the scene. With those improvements, our camera-only apprentice VCD-A sets new state-of-the-art on nuScenes with a score of 63.1% NDS. The code will be released at https://github.com/OpenDriveLab/Birds-eye-view-Perception.

Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection
Linyan Huang
Zhiqi Li
Chonghao Sima' Wenhai Wang
Jingdong Wang
Yu Qiao
Hongyang Li
'Shanghai Al Lab "Nanjing University
3CUHK
4Baidu
Abstract
Current research is primarily dedicated advancing the accuracy of camera-only 3D object detectors (apprentice) through the knowledge transferred from LiDAR- multi-modal- based counterparts (expert). However, the presence of the domain gap between LiDAR and camera features, coupled with the inherent incompatibil- ity temporal fusion, significantly hinders the effectiveness of distillation bisslllation-baeed based enbancements for apprentices. Motivated by the success of uni-modal distillation, an apprentice-friendll expert model would predominantly rely on camera features, while still achieving comparable performance to multi-modal models. To this end, we introduce VCD, framework improve the camera-only apprentice model, including an snrrentice-freendl multi-modal expert and cemporal-fusion-frienee distillation supervision. The multi-modal expert VCD-E adopts an identical struc- ture as that of the camera-only apprentice in order to alleviate the feature disparity, and leverages LiDAR input as a depth prior to reconstruct the 3D scene, achieving experts. Addi-
the performance tionally, a fine-grad fine-grained trajectory- based distillation module is introduced with the purpose of individually rectifying the motion misalignment for each object in the scene. With those improvements, our camera-only apprentice VCD-A sets new state-of-the-art nuScenes with score of 63.1 1% NDS. The code will be released at https: / github. cww hwtp///eeeeeebbbbbbbeeeeeee
par with other heterogeneous
eye- -ewwwwee Perception.
Introduction
The camera-only 3D perception has garnered increasing attention in autonomous driving perception tasks [31, 21, 47, 50]. Although camera-only models possess the advantages of low deployment cost and case widespread application, they still fall behind state-of-the-at aht models that leverage LiDAR sensors regarding perception accuracy. Researchers have recently employed distillation methods to transfer knowledge from a powerful expert model into a camera-only apprentice model, with the expectation of leveraging the expertise of these stronger expert models to enhance the capability of the camera-only models. Existing 3D perception distillation methods often adopt expert models with the best performance, such LiDAR-based models [60] multi-modal fusion models [2, 59, 33]. However, the presence of the domain gap between LiDAR and camera features hampers knowledge transfer during distillation, resulting in limited improvements in practical applications. An alternative expert model is the large-scale camera-only model [32, 61]. Despite eliminating the domain gap between the camera-only expert model and the apprentice model, the expert model falls short in terms of effectiveness due to the inherent lack of precise geometry information. Likewise, it fails to yield satisfactory improvement to the apprentice model. Hence, a desirable expert should meet two essential requirements: attaining ofate-of-the-- art performance and minimizing the domain gap.
Furthermore, current distillation methods fall short in compatibility with long-term temporal fusion, which is an essential component in cutting-edge camera-only 3D detectors [18, 43). Long-term
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Image Backbone
LIDAR Backbone
1-T-2
t-T-1
t-T
Image Backbone
BEV
Motion object
Moving
Moving
x;
X2
X3
Expert
BEV
LÍDAR
Prjezt
Depth
Domain Gap X-
(a.1) Previous Pipeline
(a.2) Our Pipedire
Warp
Warp
Actual Motion V
Apprettcee
Image Backhone
BEV
Detection
X;
X2
Xg
Misalignment
Vision-based Models
(a) Comparison of various multimodal fusion methods.
(b) The misalignment of moving objects.
Figure 1:
(a) The existing pipelines require camera and LiDAR backbones, while our pipeline eliminates the need for LiDAR. Using point cloud depth, we directly transform image features into BEV space to create a cistocccentric expert. (b) The warping of an object in the historical frame into the current timestamp results false position in the current frame due to assuming the object is stationary. The green rectangle represents true positives, while the pink rectangle indicates false positives. z; denotes the various positions of the object the historical timestamp.
temporal modeling has shown considerable potential enhancing the accuracy of depth estimation and detection performance, but introduces the issue of motion misalignment. Previous methods for BEV distillation have followed two distinct approaches, either distilling the entire BEV space without sufficient attention the foreground objects [44] exclusively distilling the foreground object regions [11, 25], thereby overlooking the motion misalignment issue resulting from the long-term temporal fusion. As shown in Fig. (b), this misalignment occurs when past scenes are transformed into the current scene coordinates based solely on ego-motion, assuming all objects are stationary. While in reality, dynamic objects will cause the misalignment, thus interfering with the temporal fusion features. This is more challenging in the case of long- gergtterm temporal fusion. Existing methods, such StreamPETR [51], introduce LayerNorm [1] for dynamic object modeling, but the effects of incorporating velocity and time variables in the model are relatively minor.
To address the aforementioned challenges, first propose vision-centric expert, termed as VCD-E, which incorporates LiDAR information to enhance the accuracy of depth input. In this context, the term "vision-centric" refers the utilization of prominent features derived from camera input, distinguishing it from approaches that heavily rely on LiDAR-based features. This model is distinct from conventional multi-modality fusion techniques by eliminating LiDAR backbone. By solely integrating LiDAR depth and long-term temporal fusion under bird's seey''-ee-view (BEV), our model achieves comparable performance state-of-the-an multi-modal fusion methods [38] by only encoding image modality. As illustrated in Fig. (a), different from previous fusion methods that adopt two modality-speeiif backbones, we only leverage single branch generate semantic features based on image input, and point clouds only provide depth information. Compared to previous fusion methods, our approach eliminates the need for intricate training strategies or specialized fusion module designs while reaching comparable performance to current fusion methods [38, 33]. More importantly, the vision-centric multi-modal model has the exact same architecture as the camera-only apprentice model, and the generated BEV spatial representation is solely from image features. The domain gap significantly alleviated through the distillation of knowledge from the proposed multi- modal expert model to the camera-only apprentice model. Due to the advantageous characteristic of domain consistency, our apprentice model acquires substantial benefits from the expert, surpassing previous distillation methods [11, 46].
To mitigate the incompatibility arising from the motion misalignment of dynamic objects, we further propose trajectory-basee based distillation module. In this paper, our primary focus revolves around foreground objects, while simultaneously incorporating meticulous consideration of their historical trajectories. Specifically, by warping dynamic targets from history to the current frame, we derive the motion trajectory associated with each individual object. Then we use the trajectory of each object to query BEV features of the apprentice and expert models respectively. By leveraging the trajectory features of the expert model to optimize the corresponding features of the apprentice model, the latter can acquire the ability to mitigate the interference arising from motion misalignment. In addition, to

---

enhance the depth perception ability, we diffuse the depth of the foreground part into the 3D space, modeling occupancy to obtain grid-based supervision assist in depth prediction for objects.
In summary, we propose the multi-modality expert and camera-only apprentice models, termed as VCD-E and VCD-A, respectively. Our contributions summarized as follows:
We construct vision-centric multi-modal expert that solely encodes the image modality, eliminating the need for LiDAR backbone. For the first time, we demonstrate that the expert can deliver performance on par with other state-of-the-art multi-modal methods while being significantly simpler.
Due to its homogeneous characteristics and superior performance, the ciston-centric expert has been proven effective in distilling knowledge vision-based models. The effects are significant across a range of model sizes, from compact to more extensive architectures.
We propose trajectory-based distillation and occupancy reconstruction modules, which supervise both static and dynamic objects to alleviate misalignment during long-term temporal fusion. Com- bined with the constructed expert model, we enhance the performance of the vision-based models and achieve state-of-the-art on the nuScenes val and test leaderboard.
2 Related Work
In this section, we review previous studies in the areas of 3D object detection, multi-modalle fusion, knowledge distillation, focusing on the techniques and methods most relevant to our research.
3D Object Detection. 3D object detection has recently gained significant popularity in the context of autonomous driving and robotics. Detection methods generally fall into two categories: vision-based 3D object detection [64, 65, 5, 37, 52, 36, 53, 58, 22, 56] and LiDAR-based 3D) detection [30, 60, 45). basen approaches [34, 54, 15, 35] exploit image information and frequently involve deep learning techniques to estimate depth. In contrast, LiDAR-basee methods capitalize on precise geometric information from LiDAR sensors to achieve superior object detection accuracy. Our proposed vision-centric expert shares the same modality as vision-based detectors, while exhibiting superior performance compared to LiDAR-base detectors.
Long-tew tengttermmn modeling has been employed to improve the performance of 3D object detection mod- els [18, 433. SOLOFusion [43] utilizes long-term temporal modeling to achieve excellent performance. while being more efficient, using long-term recurrent temporal modeling. However, previous research [34, 51] has highlighted that long-term temporal fusion can lead to inadequate detection of dynamic objects. Although Stream- PETR [51] proposes the propagation transformer [48, 7, 9, 8] to conduct object-centric temporal modeling, the improvement of dynamic object modeling remains relatively modest. Our proposed trajectory-bases distillation module alleviates this limitation, enabling accurate detection of both static and dynamic objects by camera-based based 3D object detection models that utilize long-te modeling.
VideoBEV [18] maintains comparable performance with
Multi-modality Fusion. Multi-modality fusion techniques [49, 23, 16) have been extensively investigated to enhance 3D object detection performance by integrating complementary information from different sensor modalities, such as cameras and LiDAR sensors. These methods [38, 59] typically require complex training strategies and the development of specialized fusion modules ef- fectively merge the distinct sources of information. In contrast, we propose streamlined architecture that utilizes an image backbone for feature extraction, obviating the need for LiDAR backbone. This efficient approach augments vision- models by incorporating LiDAR information, maintaining homogeneity with vision-basse models while preserving exceptional performance.
Knowledge Distillation. Knowledge distillation [44, 46] is a technique facilitating the transfer of knowledge from larger, more complex model (expert) smaller, more efficient model (apprentice). This approach has been successfully applied in various domains, including image classification [14], natural language processing [48], and policy learning [55, 27, 26, 6]. Therefore, recent distillation methods [17, 25, 32, 24, 29] build upon 3D object detection aims to transfer the accurate geometry knowledge from LiDAR to camera. MonoDistill [12] projects the LiDAR points into the image plane to serve as the expert to transfer knowledge. BEVSimDet [62] simulates fusion-based methods to alleviate the domain gap between the two different modalities. BEVDistill [11] projects the LiDAR points and images into the BEV space to align the LiDAR feature and image feature. Due to the
3

---

oon-homogenous nature between the LiDAR and camera, transferring knowledge from LiDAR to images is challenging. Instead, our work constructs a iation-centrc expert, which possesses homogeneous modality with vision-based models. The vision-centric expert can leverage knowledge distillation transfer the geometric perception capabilities to various vision-based models, hence enhancing performance accordingly.
3
Method
In this section, we present our approach in detail. The overall architecture is presented Sec. 3.1. Our method involves main components: (1) the vision-centric expert in Sec. 3.2, and (2) the trajectory-baase distillation and occupancy reconstruction modules as elaborated in Sec. 3.3. The pipeline of our method is depicted in Fig.
3.1 Overall Architecture
In this paper, construct a pair of harmonious expert and apprentice models. The expert and apprentice models adopt the consistent model architecture. The only difference is that the expert additionally leverages the accurate depth map generated from the point cloud, while the apprentice model predicts the depth map from the image. Although our expert model only uses an image backhone encode high-level scene information, it on par with state-of-the-ar multi-model fusion methods that use several modality-specifie backbones and complex interaction strategies. More importantly, we eliminate the domain gap between the multi-model expert and the camera-only apprentice model, which is deemed as one of the most challenging topics in the cross-modality distillation literature.
As illustrated in Fig. 2, we construct a distillation framework between the expert network and the apprentice network. The cxiiocccettic expert fuses feature I extracted from the image backbone and the temporal depth map D projected from LiDAR points create unified BEV representation F used for 3D object detection. Therefore, although adopt cross-modality approach for 3D object detection, the resulting representation remains homogeneous with image modality features.
After obtaining the pretrained vision-centric expert and corresponding apprentice network, we freeze the expert network and leverage its intermediate features as auxiliary supervision for the apprentice network. Since current advanced vision-based detectors employ long-term temporal modeling to attain state-of-the-art performance, we utilize a standard long-term temporal vision-based detector based on BEVDepth the apprentice model in context. The expert model also utilizes long-term temporal modeling to ensure consistency and achieve higher performance.
3.2 The Generation of Expert Model
We construct vision-centric expert by integrating LiDAR information as accurate depth input into a vision-based model. Nevertheless, given the sparsity of LiDAR depth data, rely the predicted depth values obtained from images for pixels lacking LiDAR depth information. We also utilize future frames further improve the performance of the expert model in the offline 3D detection setting. The vision-bart based detector serves as the primary model, and LiDAR information complements by providing precise depth information. This approach eliminates the need for complex training strategies or custom-designed fusion modules, streamlining the fusion process.
For the expert model, project the last sweep LiDAR frame with the current LiDAR frame onto images to obtain corresponding depth maps D. Since the depth maps D generated from point clouds can not cover every pixel of the images, we also predict depth distribution for each pixel based on the image features. Then we will project the image features onto the BEV space to obtain BEV features _ based on their depth. Furthermore, we transform the BEV features F from previous timestamps to the current BEV features F FF, modeling the long-term relationship. N denotes the time interval between the current frame and the history frame. The unified BEV features F are then combined to produce 3D object detection predictions P. The model trained using multi-task loss function that considers both 3D detection loss LDt and depth estimation loss Depth.

---

Training Inference Training Only
Image Features
Features
Time series GI query
Expert
View Transform
Depth Prediction
ccupancc
Warp
Warp
LiDAR
Fine-grained
Proj.
Depth
Trajectery-based Distillation
Proj
Proj.
Proj.
Reconstruction
Supervise
Apremtice
View Transform
Det.
Head
Multi-view Camera
BEV Features
Motion Trajactory
Algorithm Overview. Expert utilizes LiDAR data to enhance depth estimation accuracy before view transformation in BEV pipeline. Apprentice represents standard long-term vision-based detection model. The occupancy is built using the depth information from the expert model, which serves the supervision for the apprentice model. Motion trajectory constructed by warping the time series GT query into the current timestamp. Projecting the motion trajectory of each object into BEV space can rectify the misalignment object motion. With the knowledge transferred from the expert, the apprentice can deliver higher performance than before.
Figure 2:
3.3 The Procedure of Distillation
In this section, elucidate the methodology adopted overcome the constraints inherent in long- term modeling for multi-camera 3D object detection. This is achieved through the incorporation of two innovative modules within the distillation process: the trajectory- -based distillation module and the occupancy reconstruction module.
Trajectory-based Distillation. The trajectory-basee distillation module aims to improve the de- tection of dynamic objects by focusing on the inconsistent portion of the objects' motion. For i-th historical frame timestamp that contains K objects, extract the j-th ground truth object position P! (1,, vl of 1)" in the ego coordinate system. Determine the actual ego motion matrix M; between the current frame to and each historical frame f4. Apply the ego-motion transformation matrix M, to the ground truth object positions P obtain the transformed positions P! in the current frame's 's coordinate system:
M x P. P
(1)
We amalgamate the transformed ground truth object positions, P from all historical frames to construct the motion trajectories. The trajectory of j-th object thus be represented sequence of object positions within the current frame as (Pi P ..... PK), where N the number of points each motion trajectory. Let F F. represent the sampled features identical point P from the expert BEV feature F and apprentice BEV feature F respectively. They are sampled via bilinear interpretation and then are normalized as
F = norm( B (PP')) )), F norm F (P( )).
Finally, the trajectory- based distillation loss LTD computed between the normalized key sampled features:
CTD KXN
ΣΣ La (F F.). j=
Ultimately, by using the motion trajectory queries, we conduct wrajectorybbaee distillation on these representative positions. This approach enables the expert to rectify the motion misalignment in the apprentice.
5

---

Table 1: Comparison among the camera-only methods on the nuScenes val denotes the
long-term baseline implemented by based on BEVDet4D- Depthh [22]. depicts that the size of
BEV feature 256x VCD-A surpasses previous SOTA by 2 points NDS and achieves SOTA under the same setting.
Methods
Backbone
Image Size
Frames
mAPt
NDS1
mATEL
mASEL
mAOEL
mAVE!
mAAE,
BEVDet [23]
ResNet-50
256 x 704
0.298
0.379
0.725
0.279
0.589
0.860
0.245
PETR [39]
ResNet-50
384 1056
1
0.313
0.381
0.766
0.278
0.564
0.923
0.225
BEVDet4D [22]
ResNet-50
256 x 704
2
0.322
0.457
0.703
0.278
0.495
0.354
0.206
BEVDepth
ResNet-50
256 704
0.351
0.475
0.639
0.267
0.428
BEVSterco [34]
ResNet-50
256 x 704
2
0.372
0,500
0.598
0,270
0.438
0.367
0.190
STS
VideoBEV [19]
ResNet-50
256 x 704
8
0.422
0.535
0,564
0.276
0.440
0.286
0.198
SOLOFusion [43]
ResNet-50
256 x 704
16+1
0.427
0.534
0.567
0,274
0.411
0.252
0.188
StreamPETR [51]
ResNet-50
256 x 704
8
0.432
0,540
0.581
0.272
0.413
0.295
0.195
Baseline
ResNet-50
256 x 704
8+1
0.401
0.515
0.395
0,279
0.489
0.291
0.198
VCD-A
0.271
0.268
Baseline
ResNet.50
256 704
8+1
0.418
0.542
0.522
0.267
0.428
0.262
0.188
VCD-A!
ResNer-50
256 704
8+1
0,446
0.566
0.497
0.260
0.350
0.257
0.203
Occupancy Reconstruction. The expert model demonstrates outstanding performance in 3D object detection, resulting in a more precise 3D geometric representation of the objects. We utilize depth estimation modules to predict the depth, denoted as (uu, v), for each image pixel (u, v). Subsequently, the depth map D is pack-projected into 3D point cloud, each image pixel (u, v) is transformed into 3D coordinate (x, y, z). Then the scores from different pixels in the multi-camera within the same occupancy region, are accumulated to make decisions about the presence of an object in that region. In this way, can generate the occupancy O and O₂ for expert and apprentice model, respectively.
The occupancy reconstruction module improves the model's capability to discern the 3D geometric properties objects. The grid-based supervisory signals effectively direct the model to enhance its prediction accuracy for object depths. Different voxels within the occupancy structure aggregate the fused depth distribution from various perspective views, thereby enhancing robustness against depth errors. Drawing inspiration from CenterPoint [63], simply extend the Gaussian distribution applicd to each target into the 3D space for more focused 3D object modeling:
(x - pz )2 + (y p + (z P=)²
Gay: = exp
(4)
20
where (fz, py P2)) represents the center of the 3D object, while op denotes the standard deviation of each object's s size. The model utilizes the expert model's 's occupancy O2 supplementary supervision by adopting straightforward L1 regularization loss for the occupancy status O₂ of the apprentice model, which optimizes depth prediction capabilities for both static and dynamic objects. The occupancy reconstruction loss can be formulated as
Lor = Li (G Oc, Guyz On).
(5)
Training Loss.
During the distillation phase, the joint training loss Crotal is formulated as
Drotal = LA + A1 LTD + X2 LoR,
(6)
where LA is the perceptual loss of the apprentice model. Besides, A and X2 represent hyperparam- eters employed effectively balance the scales of the respective loss functions. The utilization of trajectory-basse distillation loss LTD and occupancy reconstruction loss LoR collectively facilitates the transfer of semantic and geometry knowledge from the expert model the apprentice model.
Experiments
In this section, we outline the experimental setup and assess our proposed VCD-E model, as well as the newly introduced enjeetorrybased. distillation and occupancy reconstruction modules, using the nuScenes dataset [3]. This includes presentation the evaluation metries, baseline models, ablation studies, and comparative analysis of our approach with the current of-e-of-theear methods.

---

Table 2: Comparison among the camera-only methods on the nuScenes test set. Methods marked with denote long-term baseline implemented by us, based on DEVDer4D-Depis [22]. f depicts test time augmentation adopted during the inference phase. VCD-A achieves SOTA under critical metrics and surpasses its baseline by 2 points in NDS.
Methods
Backbone
Image Size
mAPt
NDS1
MATEL
mASE,
mAQE
mAVE,
mAAE|
FCOS3D1 [52]
R10I-DCN
900 1600
0.358
0.428
0.690
0.249
0.452
1.434
0.124
DETR3Di [53]
V2-99
900 1600
0.412
0.479
0.641
0.255
0.394
0.845
0.133
UVTR [33]
V2-99
900 1600
0.472
0.551
0.577
0.253
0.391
0.508
0.123
BEVDet4Di [22]
Swin-B [41]
900 1600
0.451
0.569
0.511
0.241
0.386
0,301
0.121
BkVFormer [36]
V2-99
900 1600
0.481
0.582
0.256
0.375
0.378
0.126
PolarFormer [28]
V2-99
900 1600
0.493
0.572
0.556
0.256
0.364
0.439
0.127
BEVDistll [1]]
ConvNeXi-B
900 1600
0.496
0.594
0.475
0.249
0.378
0.313
0.125
PETRv2 [40]
RevCol
640 1600
0.512
0.592
0.547
0.242
0.360
0.367
0.126
BEVDepth [35]
ConvNeXt-B
640 1600
0.520
0.609
0,445
0.243
0.352
0.347
0.127
AeDett [15]
ConvNeXi-B
640 1600
0.531
0.620
0.439
0.247
0.344
0.292
0.130
SOLOFusion [43]
ConvNeXt-B
640
1600
0.540
0.619
0,453
0.257
0.376
0.276
0.148
StreamPETR [51]
ConvNeXi-B B
640
1600
0.550
0.631
0.493
0.241
0,343
0,243
0.123
Baseline"
ConvNeXi- B
640
1600
0.522
0.610
0.457
0.253
0.391
0.271
0.142
VCD-A
ConvNeXi-B
640 1600
0.548
0.631
0,436
0.244
0.343
0.290
0.120
Table 3: Comparison among the multi-modality methods the nuScenes val set. Our proposed VCD-E only adopts image backbone while achieving comparable performance with sfate-of-the-art multi-modal methods who predominately rely on the LiDAR backbone.
Methods
Venue
Backbone
MAPt
NDS!
MATE, MASEJ
MAOEJ
mAVEL
mAAEI
BEVFusion [38]
NeuriPS 2022
LiDAR & Image
0.642
0.680
FUTR3D [10]
Arxiv 2022
LiDAR & Image
0.645
0.683
UVTR [33]
NeurIPS 2022
LiDAR & Image
0.654
0.702
0.332
0.258
0.268
0.212
0.177
CMT [57]
Arxiv 2023
LIDAR & Image
0.679
0.708
VCD-E
Image
0.677
0.711
0.308
0.254
0.317
0.189
0.201
4.1 Main Results
Camera-only 3D detection. In order to evaluate the effectiveness of our proposed Revenge model and the trajectory- -baee distillation modules, we have conducted rigorous experiments on the nuScenes validation and test sets. As presented in Tab. 1, the performance of VCD-A surpasses other cutting-edge methods, achieving record of 44.6% and 56.6% on the nuScenes benehmark. This provides robust evidence of the effectiveness of our approach. Utilizing an image resolution of 256x704 and a ResNet-50 backhone, VCD outperforms the state-of-the-ar method from [43] by improving mAp by 1.9% and NDS by 3.2%. Additionally, VCD-A exhibits an improvement over our baseline by 2.8% mAP and 2.4% NDS, thereby indicating that our methods can considerably advance state-of-the-an results. To further ascertain the generalizability of our methods, we conducted experiments on the nuScenes test set, as shown in Tab. With the adoption of the ConvNext-B backbone [42], VCD achieved 54.8% MAP and 63.1 %% NDS, outperforming the state-of-the-ard detector, SOLOFusion [43], by 0.8% mAp and 1.2% NDS. Furthermore, our proposed approach led to a 2.6% increase in mAP and a 2.11% 1% improvement in NDS compared to our baseline, thereby demonstrating the efficacy of our method for large-scale vision-based models.
Multi-modality Fusion. We first compare our proposed vision-centric expert with other state-of- the-art art multi-modallg fusion methods. VCD-E does not require any complex fusion strategies or stage training schedule, yet achieves 67.7% mAP and 71.1% NDS as shown in Tab. 3. Despite employing only single image backbone, the model's performance is comparable to state-of- the-art methods such as BEVFusion [38], and even surpasses UVTR [33] by 2.3% mAP and 0.9% NDS, which utilizes two backbones, by considerable margin. Owing to its homogeneous nature and compatibility with camera-only models, the VCD-E can serve as powerful expert for knowledge transfer, facilitating performance improvements for small to large vision detectors.
4.2 Ablation Study
To verify the effectiveness and necessity of each component, we conduct various ablation experiments on the nuScenes validation set.
7

---

Table Ablation study of the proposed distillation framework on different temporal lengths.
The design works with different length of time windows and the gain grows with the temporal length.
Temporal Length
Distill
mAP (%)
NDS (%)
mATE,
mAOE)
mAVEt
26.6
37.9
0.815
0.645
0.556
30.1 (43.5)
41.5(+3.6)
0.732
0.629
0.476
x
26.9
38.4
0.804
0.706
0.461
2
31.3 (+4.4)
43.2 (+4.8)
0.717
0.615
0.403
x
28.4
39.8
0.748
0.739
0,432
<
33.0 (+4.6)
44.1 (+4.3)
0.707
0.632
0.389
29.7
40.9
0.762
0.714
0.415
35.4 (+5.7)
45.9 (+5.0)
0.690
0.625
0.370
Table 5: The performance gains of the appren- tice which benefit from different experts. CM de- notes cross-modal and UM represents Uni-modal. It indicates the success of uni-modal distillation remains.
Table Effect of different distillation meth- ods. All models are trained with VCD-E as expert. Our proposal significantly surpasses previous SOTA methods.
Methods
mAP
NDS
Expert
Paradigm
mAP
NDS
Baseline [35]
0.297
0,409
0.297
0.409
FitNet [44]
0.318
0.421
CenterPoint [60]
CM
0.281
0.420
CWD [46]
0.311
0,412
Transfusion [2] CM
0.292
0.435
BEVDistill [11]
0.316
0.439
BEVDepth [35]
UM
0.341
0.442
VCD-A
0.354
0.459
VCD-E
UM
0.354
0.459
Effectiveness of the General Framework. We first validate the effectiveness of the general framework for various temporal lengths. In Tab. 4, we examine different timestamps for temporal modeling and find that our method significantly outperforms the baseline. By employing the general framework, we achieve 5.0% NDS and 5.7% mAP performance gains with 8 temporal modeling instances. For other temporal modeling scenarios, our method continues to exhibit substantial performance improvements, ranging from 3.6% to 5.0% NDS. As the duration of temporal fusion extends, the advantages of the model become increasingly pronounced, suggesting that this framework exhibits higher compatibility with extended sequences. This indicates the effectiveness of our proposed trajectory-baan based distillation module, which alleviates the issue of motion misalignment.
In this study, it is crucial verify the effectiveness of our proposed vision-centric experts. We select various expert models for fair comparison, including LiDAR-bases expert Centerpoint [60], vision-bast based expert BEVDepth [35], and fusion-bast based expert Transfusion, based on our proposed distillation strategy. In Tab. 5, we observe that homogeneous expert models significantly influence the success of knowledge transfer. The camera-based expert BEVDepth demonstrates superior performance gains compared to the other two heterogeneous expert models. Our advanced model, VCD-E, operates within the same modality as vision-based models, and is equipped with precise geometric information. Consequently, the distillation effect outperforms the Transfusion [2] expert by a significant margin, achieving 6.2% increase in mAP. Moreover, it surpasses the BEVDepth expert by additional 1.7% in NDS. This demonstrates the superior performance and effectiveness of our model.
Effectiveness of VCD-E.
Comparison to SOTA. To further demonstrate the effectiveness of our proposed distillation strategy, we compare our method with other state-of-the-an methods. To ensure a fair comparison, we conduct all experiments based on our vision-centric expert VCD-E. We find that our method consistently outperforms other distillation methods by significant margin. In 2D detection, FitNet [44] and CWD [46] are two classic distillation methods, we adapt them for 3D object detection to facilitate a fair comparison. BEVDistill [11] represents a state-of-the-as distillation method for multi-view 3D object detection, and we compare our results with this approach as well. As shown Tab. our method achieves the best results when compared to these state-of-the-arr distillation strategies, demonstrating the effectiveness of our approach. Our method surpasses BEVDistill by 2% NDS and
3.8% mAP.
8

---

Table Gains of different image backbone on multi-moda models [38]. The stronger backbone still demonstrates better performance this case.
Table Gains of different depth fusion strategy. The proposed fusion depth is opti- mal among different methods.
Methods
Backbone
mAP
NDS
Methods
mAP
NDS
BEVFusion
ResNet-50
0.598
0.662
Predicted depth
0.495
0.585
BEVFusion
ConvNext-B
0.597
0,665
LiDAR depth
0.638
0.687
VCD-E
ResNet-50
0.611
0.656
Weighted depth
0.644
0.690
VCD-E
ConvNext-B
0.664
0.693
Fusion depth
0.646
0.690
Table 9: The performance gains of different trajectory length for trajectory- based distillation. As the trajectory length increases, the benefits derived from the distillation process become more pronounced.
Trajectory Length
Distill
mAP (%)
NDS (%)
x
29.7
40.9
0
/
31.8
42.1
1
/
33.1
44.5
3
/
34.6
45.6
5
/
35.4
45.9
9
/
33.9
44.7
Gains of the Image Backbone. To verify that VCD-E benefits from contemporary image backbones, we conducted experiments outlined in Tab. 7. We selected ResNet-50 and ConvNext-B as two distinct modern image backhones for BEVFusion and VCD-E. Our findings indicate that VCD-- achieves substantial improvements of 5.3% mAP and 3.7% NDS when using ConvNext-I compared to ResNet-50, demonstrating that VCD-E can indeed benefit from modern image backbones. However, existing fusion-based methods, such as BEVFusion, which primarily rely on the capabilities of LiDAR backbones, show limited gains from employing modern image backbones.
Fusion Strategy of Expert. To determine the optimal fusion strategy for LiDAR points and images, propose four distinct depth fusion approaches, including (1) Predicted depth that directly uses predicted depth based on image features. (2) LiDAR depth that utilizes the projected depth from LiDAR points. (3) Fusion depth that employs the projected depth when corresponding LiDAR points exist for each pixel, otherwise using the predicted scores based on image features, and (4) Weighted depth. Applying the weighted average depth of the predicted depth and LiDAR depth. As demonstrated in Tab. 8, fusion depth yields the best results which indicates that using corresponding predicted scores where LiDAR depth is unavailable advantageous.
The Effectiveness of Trajectory-based Distillation The results presented in Table 9 indicate that as the trajectory length increases, the benefits derived from the distillation process become more pronounced. The temporal fusion length for this experiment is set at eight. The first row denotes the baseline model which does not use the distillation method. The second row depicts that the VCD-A model directly conducts distillation under the full BEV feature without the Trajectory-basee distillation module. When the trajectory length set we only distill ground truth locations in the current frame. However, when the trajectory length exceeds five, there is noticeable decrease in accuracy. We hypothesize that this decrease may be attributed to the model's distracted attention towards distant motions. The density of traffic can lead to distant motion locations being occupied by other objects, which may not necessarily require additional trajectory supervision. This suggests that the application of excessive trajectory supervision in such scenarios could be unnecessary and inefficient.
Ablation Study of Each Component. In Tab. 10, we conduct an ablation study on the components employed VCD to verify their contributions to the final result. These components include (1) Long-term temporal fusion, which allows our model to benefit from historical information, thereby enhancing its performance.
(2) Trajectory-based distillation, which transfers knowledge from VCD-E to VCD-A in areas with motion misalignment to mitigate this issue. (3) Occupancy reconstruction.

---

Table 10: Ablation study of important components in VCD-A. shows that the motion module
contributes most to the performance improvement, indicating the potential of future research here.
Methods
Long
Occ
Motion
MAPt
NDS1
mATE|
mAVEJ
Baseline
0.271
0.319
0.767
1.065
Longer Temporal Fusion
0.297
0.409
0.762
0.415
Trajectory-basse -based Distillation
/
0,283
0.356
0.772
0.870
/
All but Longer Temporal
0.290
0.367
0.724
0.834
Occupancy Reconstruction
0.308
0,416
0.734
0.379
VCD
0.341
0.449
0.715
0.389
which as dense depth supervision in the perspective view, improving the performance of VCD-A. Long-ter cong-term temporal modeling enables our model to reference prior information from history. assisting object velocity estimation and resulting in a significantly lower mATE. We also evaluate the impact of the vajectory-basee distillation module when all other components are incorporated. As shown Tab. 10, trajectory-based distillation increases NDS by 3.3% and mAP by 3.3%, contributing to the majority of the improvement.
Conclusion
In this paper, we presented novel vision-centrie model as the expert, which leverages the strengths of both modalities, our proposed model achieves exceptional performance, rivaling that of state-of- the-art multimodal fusion models, while also mitigating domain gap issues, making it suitable for distilling vision-based models. Furthermore, we introduced two innovative modules, the trajectory- based distillation and occupancy reconstruction modules. These modules enhance the geometric perception capabilities of multi-camera 3D object detection models and improve the detection of both static and dynamic objects the scene. Our experiments demonstrate the effectiveness of our proposed methods the widely-used nuScenes 3D object detection benchmark.
Limitation. this work, we have not delved into more details the vision-centre expert, which may hold significant potential for improvement. Additionally, we have not explored further applica- tions, such as automatic dimension estimation based on VCD-E.
Acknowledgements
This work supported by National Key R&D Program of China (2022ZD0160104) and NSFC (62206172). We would like to thank anonymous reviewers for active discussions.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607. (000::607.6450 06450, 2016.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. TransFusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 3.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan. Giancarlo Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 6, 14
[4] Yuxuan Cai, Yizhuang Zhou, Qi Han, Jianjian Sun, Xiangwen Kong, Jun Li, and Xiangyu Zhang. Reversible column networks. ICLR, 2023.
[5] Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, et al. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. In ECCV, 2022.
[6] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. arXiv preprint arXiv:2306. 16927,
2023.
10

---

[7] Mengzhao Chen, Mingbao Lin, Ke Li, Yunhang Shen, Yongjian Wu, Fei Chao, and Rongrong
Ji. Cf-vit: A general coarse-to-fine fine method for vision transformer. In AAAI, 2023.
[8] Mengzhao Chen, Mingbao Lin, Zhihang Lin, Yuxin Zhang, Fei Chao, and Rongrong Ji. Smmix: Self-motivated image mixing for vision transformers. In ICCV, 2023. 3
[9] Mengzhao Chen, Wenqi Shao, Peng Xu, Mingbao Lin, Kaipeng Zhang, Fei Chao, Rongrong Ji, Yu Qiao, and Ping Luo. Diffrate: Differentiable compression rate for efficient vision transformers. arXiv preprint arXiv:2305. 17997, 2023. 3
[10] Xuanyao Chen, Tianyuan Zhang. Yue Wang. Yilun Wang, and Hang Zhao. Futr3d: unified sensor fusion framework for 3d detection. arXiv preprint arXiv:2203.10642, 2022. 7
[11] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Cross-modal bev distillation for multi-view 3d object detection. arXiv preprint arXiv:22/ 10t::22/1109380. 1.09386, 2022. 2, 7,8
[12] Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang, and Wanli Ouyang. MonoDistill: Learning spatial features for monocular 3d object detection. arXiv preprint arXiv:2201.00830, 2022. 3
[13] MMDetection33 Contributors. MMDDrtection3D:: OpenMML.ab next-gemeration platform for
general 3D object detection. https://gitne com/ oetooppenmmmmm
2020.
14
[14] Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
[15] Chengjian Feng, Zequn Jie, Yujie Zhong, Xiangxiang Chu, and Lin Ma. Aedet: Azimuth- invariant multi-view 3d object detection. arXiv preprint arXiv:2211.12500) 2022. 3,
[16] Yulu Gao. Chonghao Sima, Shaoshuai Shi, Shangzhe Di, Si Liu, and Hongyang Li. Sparse dense fusion for 3d object detection. In IROS, 2023. 3
[17] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. LIGA-Stereo: Learning lidar geometry aware representations for stereo-based 3d detector. In ICCV, 2021. 3
[18] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303.0551
2023. 1,3
[19] Chunrui Han, Jianjian Sun, Zheng Ge, Jinrong Yang, Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. arXiv preprint arXiv:2303. 05970, 2023, 6
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image
recognition. In CVPR, 2016. 14
[21] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In CVPR, 2023.
[22] Junjie Huang and Guan Huang. BEVDet4: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint ar03r: 17054 2022. 3, 6.
[23] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. BEVDet: figh-performance multi- camera 3d object detection in bird-eye-views arXiv preprint arXiv:21 1790 2021.
[24] Linyan Huang, Huijie Wang, Jia Zeng, Shengchuan Zhang, Liujuan Cao, Rongrong Ji, Junchi Yan, and Hongyang Li. Geometric-aware pretraining for vision-centric 3d object detection. arXiv preprint arXivv2304.03305. 2023. 3
[25] Peixiang Huang, Li Liu, Renrui Zhang, Song Zhang, Xinli Xu, Baichao Wang, and Guoyi Liu. Tig-bev: Multi-view bev 3d object detection via target inner-geometry learning. arXiv preprint
arXiv:2212.13977 13979, 2022. 3,3
[26] Xiaosong Jia, Yulu Gao, Li Chen, Junchi Yan, Patrick Langechuan Liu, and Hongyang Li. Driveadapter: Breaking the coupling barrier of perception and planning in end-to-end au- tonomous driving. In ICCV, 2023.
[27] Xiaosong Jia, Penghao Wu, Li Chen, Jiangwei Xie, Conghui He, Junchi Yan, and Hongyang Li. Think twice before driving: Towards scalable decoders for end-to-end autonomous driving. In CVPR, 2023. 3
11

---

[28] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. arXiv preprint arXiv:2206. /5398, 2022. 7
[29] Marvin Klingner, Shubhankar Borse, Varun Ravi Kumar, Behnaz Rezaei, Venkatraman Narayanan, Senthil Yogamani, and Fatih Porikli. kd: Knowledge distillation across modali- ties, tasks and stages for multi-camera 3d object detection. arXiv preprint arXiv:2303.02203: 2023.
[30] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. PointPillars: Fast encoders for object detection from point clouds. In CVPR, 2019. 3
[31] Hongyang Li, Chonghao Sima, Jifeng Dai, Wenhai Wang, Lewei Lu, Huijie Wang, Enze Xie, Zhiqi Li, Hanming Deng, Hao Tian, et al. Delving into the devils of sird's-eye-view perception: review, evaluation and recipe. arXiv preprint arXiv:2209.0533
2022.
[32] Jianing Li, Ming Lu, Jiaming Liu, Yandong Guo, Li Du, and Shanghang Zhang. Bev-Igkd: A unified lidar- lidar-guided knowledge distillation framework for bev 3d object detection. arXiv preprint arXiv:22/2.0062: 2022. 1,3
[33] Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. arXiv preprint arXiv:2206.0066 006600 2022.
1, 3,7
[34] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.
arXiv preprint arXiv:2209. 10248, 2022. 3,6 6
[35] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. BEVDepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv: 1009,, 2022. 3, 6, 7, 8, 14
[36] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. BEVFormrr: Learning bird's s-eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203. /7270, 2022. 3,7
[37] Zhuoling Li, Chuanrui Zhang, Wei-Chiu Ma, Yipin Zhou, Linyan Huang, Haoqian Wang, SerNam Lim, and Hengshuang Zhao. Voxelformer: Bird's-eye- s-eee-view feature generation based on dual-view attention for multi-view 3d object detection. arXiv preprint arXiv:2304.01055 2023.
[38] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. BEVFusion: A simple and robust lidar-camera fusion framework. arXiv preprini arXiv:2205. 1 33900, 2022. 2, 3, 7,9
[39] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETR: Position embedding transfor- mation for multi-view 3d object detection. arXiv preprint arXiv:2203.0 05625, 2022.
[40] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETRv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.. 2565, 2022. 7
[41] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
7
[42] Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In CVPR, 2022. 7, 14
[43] Jinhyung Park, Chenfeng Xu, Shijia Yang, Kurt Keutzer, Kris Kitani, Masayoshi Tomizuka, and Wei Zhan. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.024443, 2022. 1, 7, 15
[44] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. 2, 3, 8
[45] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN++: Point-voxel feature set abstraction with local vector repre- sentation for 3d object detection. arXiv preprint arXiv: 102. 00463, 2021. 3
arXiv:2
12

---

[46] Changyong Shu, Yifan Liu, Jianfei Gao, Zheng Yan, and Chunhua Shen. Channel-wise
knowledge distillation for dense prediction. In ICCV, 2021. 2. 3, 8
[47]
Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng. Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, and Hongyang Li. Scene as occupancy. In /CCV, 2023. 1
[48]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,
Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPSS 2017. 3
[49] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. PointPainting: Sequential fusion for 3d object detection. CVPR, 2020.
[50] Huijie Wang, Tianyu Li, Yang Li, Li Chen, Chonghao Sima, Zhenbo Liu, Yuting Wang, Shengyin Jiang, Peijin Jia, Bangjun Wang, Feng Wen, Hang Xu, Ping Luo, Junchi Yan, Wei Zhang, and Hongyang Li. Openlane-v2: A topology reasoning benchmark for scene understanding in
autonomous driving, 2023. 1
[51] Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection, arXiv preprint 2rX1v:2303.11926. 2023. 2, 3, 6,7
[52] Tai Wang. Xinge Zhu, Jiangmiao Pang. and Dahua Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. In ICCV, 2021. 3,7
[53] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, 2022. 7,7
[54] Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang. and Di Huang. STS: Surround-view temporal stereo for multi-view 3d detection. arXiv preprint arXiv:2208. 10145,
2022. 3,6 6
[55] Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory- guided control prediction for end-to-end autonomous driving: A simple yet strong baseline. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[56] Kaixin Xiong, Shi Gong, Xiaoqing Ye, Xiao Tan, Ji Wan, Errui Ding, Jingdong Wang, and Xiang Bai. Cape: Camera view position embedding for multi-view 3d object detection. In CVPR, pages 21570-21579, June 2023. 3
[57] Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal transformer via coordinates encoding for 3d object dectection. arXiv preprint arXiv:2301.01228. 2023. 7
[58] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern im- age backbones to bird's vvew recognition via perspective supervision. arXiv preprint arXiv:2211. 10439, 2022.
[59] Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. DeepInteraction: 3d object detection via modality interaction. arXiv preprint arXiv:2208. /// 12, 2022. 1,3
[60] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021. 1,
[61] Jia Zeng, Li Chen, Hamming Deng, Lewei Lu, Junchi Yan, Yu Qiao, and Hongyang Li. Distilling
focal knowledge from imperfect expert for 3d object detection. In CVPR, 2023. 1
[62] Haimei Zhao, Qiming Zhang, Shanshan Zhao, Jing Zhang, and Dacheng Tao. Bevsimdet: Simulated multi-modal distillation in bird's-eye view for multi-view 3d object detection. arXiv
preprint 27Xiv:2303.168888 2023. 3
[63] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv preprint
1rXiv:1904.07600 2019. 6
[64] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang. Monoc- ular 3d object detection: An extrinsic parameter free approach. In CVPR, 2021. 3
[65] Yunsong Zhou, Yuan He, Hongzi Zhu, Cheng Wang, Hongyang Li, and Qinhong Jiang. Monoef: Extrinsic parameter free monocular 3d object detection. IEEE TPAMI, 2022. 3
13

---

A Experiment Details
A.1 Dataset and Evaluation Metries
We conduct our experiments on the nuScenes dataset [3], widely used benchmark for autonomous driving tasks. The dataset encompasses diverse driving scenarios captured using cameras and LiDAR sensors, offering rich information for both visual and LiDAR-based 3D object detection. The dataset comprises 700 training scenes, 150 validation scenes, and 150 testing scenes. Each scene spans Hz frequency.
approximately 20 seconds, with key frames annotated
The two dominant metrics for the nuScenes detection task are the nuScenes Detection Score (NDS) and mean Average Precision (mAP). The mAP for nuScenes is computed based the center distance between predictions and ground truth annotations on the ground plane. Moreover, the nuScenes dataset defines five true positive metries (mATE, mASE, mAOE, mAVE, mAAE) for measuring translation, scale, orientation, velocity, and attribute, respectively. The NDS for nuScenes is weighted sum of mAP and the five true positive metrics, defined as N DS TO [5m.P++ 1 Dmmp(< (miimmmin,,,
A.2 Implementation Details
We conduct experiments on BEVDepth [35]. The codebase developed upon MMDetection3D [13]. Main experiments
trained 8 NVIDIA A100 GPUs, while ablation experiments conducted on 8 NVIDIA V100 GPUS, For BEVDepth, the model is trained for 20 epochs with an initial learning rate of 2c-4. In the distillation process, the per-GPU batch size is set to 4, whereas during the training of the baseline model, it is set to Normal data augmentations are introduced the training process such flip and rotate. In our apprentice models, future frames temporal fusion throughout the training phase to ensure a fair comparison. As for ablation study, conduct experiments with an online training strategy, and have employed the ResNet-50 configuration in the absence of CBGS.
incorporated into the long-term
In our research, implement distinct temporal modeling strategies for both apprentice and expert models. For the apprentice models, we incorporate a sequence of eight frames into the temporal modeling process. In contrast, the expert models integrate four future frames into the temporal modeling as demonstrated in our primary results. However, in our ablation study, we deviate from this approach and instead employ eight historical frames for temporal modeling. Besides, We the last sweep LiDAR frame with the current LiDAR frame to create the depth map.
Table 11: Experiment settings. denotes that the training schedule for VCD-E is approximately one-fourth of the original schedule. This reduction was implemented to expedite the training process during the ablation study. The first group is engaged in training the main results, whereas the second group utilized in the ablation study.
Method
Backbone
Image Size
Frames
mAP (%)
NDS (%)
VCD-E
ConvNext-B [42]
512 x 1408
8+1
67.7
71.1
VCD-A
Res-50 [20]
256 704
8+1
41.8
54.2
VCD-E*
ConvNext-B [42]
256 704
8+1
54.2
58.8
VCD-A
Res-50 [20]
256 x 704
8+1
29.7
40.9
A.3 Experiments Settings
The setting of adopted expert-apprentice pairs is depicted in Tab. 11. We categorize the distillation setting into two distinct groups. The primary group is engaged in training on the main results, whereas second group is utilized for the ablation study. Both our baseline and VCD-A adopt history frames for temporal fusion.
14

---

mAP1
MATE|
1.05
0.45
+3.2%
0.35
0.85
Baseline
0.25
0.65
VCD (ours)
0.15
-7.6%
0.05
0.45
Con Veh
Traffic
Barrier
Average
Con Veh
Traffic
Barrier
Average
Cone
Cone
Figure 3: Effeets of VCD static objects. Our distillation framework VCD can still consistently improves static objects, demonstrating 3.2% and 7.6% improvements in precision-recall (mAP) and localization quality (mATE), respectively.
B
The Analysis of Temporal Fusion
B.1
The Misalignment of Motion Objects
As highlighted in preceding studies [43], long-term temporal fusion may face misalignment issues in motion estimation, which can discerned through a reduction in performance on metrics like mATE. Let's consider moving object and analyze the impact of inaccurate motion estimation its position in the fused frame. We will assume that the environment is static, except for the moving object. Let the position of the moving object in the world coordinate system be represented by P = (x:", ( ,"", , ,)" each the N frames captured at times f1, .... ,t. The actual motion of the moving object between frames is represented by M and the estimated motion is represented obj
by M; The difference between the estimated and actual motion of the object can be denoted as:
AM; obj
M; M, obj
As we have already computed the transformation matrix T, based on estimated ego motion, we can calculate the transformed object position in the current frame, considering its actual motion,
P.P'
T. M P."
(8)
The error in the transformed object position be computed as:
obj
(9)
In the long-term fusion process, integrate the information from all N frames. Assuming we use fusion function F, the fused position in the current frame can be represented as:
pobj Jusson F(P"' P. PPK))
(10)
The inaccuracies in the motion estimation of the moving object for each frame propagate through the fusion function and result a misaligned object in the fused frame. The overall error in the fused position can be represented as function of the errors in each frame:
obj e fusion = G(e
obj obj ee
(11)
where represents a function that combines the errors from each frame The fused position of the moving object will be less accurate due to these motion estimation errors, leading decline in object detection performance in the long-term setting. To address the issue mentioned earlier, introduce the trajectory-baset distillation module, which compensates for the misalignment of moving objects. We will provide further details the subsequent discussion.
B.2
The Improvements Static and Dynamic Objects
In this section, we present visualizations to demonstrate the improvements achieved in dynamic objects. Particularly noteworthy is the significant enhancement the representation of dynamic objects through trajectory based distillation, thereby highlighting the effectiveness of the trajectory- based module. As depicted Fig. and Fig. 4, our distillation framework consistently enhances static and dynamic objects across various metrics.
15

---

mAP 1
MATE
0.6
0.5
+5.7%
0.75
0.4
0.65
-6.7%
0.3
0.55
0.2
0,45
Car
Truck
Bus
Pedlestrian
Matorcyde
Average
Car
Truck
Bus
Pedestrian
Matorcydle
Aver age
Baseline
(ours)
mADE
mAVE
0.8
1.05
0.7
0,6
0.55
0.5
-13.6%
0.4
0.05
0.3
6.5%
Truck
Bus
Pedestrian
Motorcydle
Average
Car
Truck
Pedestrian
Motorgydle
Average
Figure 4: Effects of VCD movable objects. Our distillation framework VCD consistently improves dynamic objects across a range of metrics.
CAM FRONT LEFT
SAM_FRONT
CAM_FROMM RIGHT
CAM BACK_BACBBEEEE LEFT
CAM_BACC BACK
CAM BACK_RIGHT RIGHT
GAITI
Figure 5: Visualization of the predictions for 3D object detection generated by the VCD-A.
C
Visualization
We have performed several visualizations in Fig. 5 to showcase the advancements achieved by our distillation framework. Our findings indicate that our models excel in accurately predicting 3D bounding boxes for the target objects.
D Broader Impact
Our research introduces a novel perspective for multi-modal methodologies and a fresh distillation paradigm for camera-only techniques. We believe that it can establish robust baseline for the broader scientific community. However, while our methods contribute to the enhancement of autonomous driving, they yet capable of addressing limitations could potentially introduce risks real-world autonomous systems.
complex corner cases. Consequently, these
16


Reviews:
[{"review_id": "eBdMzoUEfW", "reviewer": "Reviewer_LE1a", "summary": "This paper seeks to improve camera-only 3D object detection by distilling a multi-modal expert model. The authors start by developing a multi-modal expert with essentially the same architecture as the camera-only model - intended to reduce the domain gap. Then, they propose trajectory-based distillation as well as occupancy reconstruction to supervise the student. The resulting camera-only model achieves state-of-the-art in nuScenes, and the multi-modal expert is a simple, strong method in its own right.", "questions": "- Additional details on the expert model are necessary. How many past frames were used to accumulate LiDAR to create the sparse depth map? Does using past LiDAR not cause significant depth artefacts for moving objects?", "limitations": "The authors have included a limitations section.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "- The intuition that the expert should have similar architecture to the student is intuitive.\n- The proposed expert model’s significant increased performance when using a stronger image backbone in Table 7 is interesting. \n- Extensive experiments demonstrate that having an aligned architecture between the teacher and student is helpful (BEVDepth teacher outperforms TransFusion as well, which is a nontrivial observation).\n- The final model achieves state-of-the-art in nuScenes.", "weaknesses": "- The Occupancy Reconstruction section is difficult to understand. It appears that the predicted expert depths are outprojected to XYZ. However, it is unclear how the 3D voxel grid for occupancy reconstruction is generated. Further, it’s actually not clear how the outprojected XYZ is used, as G_xyz is defined in (5) and used in (6) without further mention of the outprojected XYZ. In L207, Should G_xy be G_xyz? It is also unclear how this optimizes depth prediction capabilities for static and dynamic objects. I would appreciate a clear explanation of this part, as it is one of the main contributions of this work.\n- What model/settings was used for the Table 4 ablation? The numbers seem too drastically worse compared to Table 1. In Table 1 in the supplementary, the baseline VCD-A model in the first and second sections appear to be the same, but the results differ.\n- While Trajectory-based Distillation improves performance in Table 9, additional ablations are missing. More specifically, the “simplest” form of Trajectory-based distillation would be to simply just distill GT instance locations in the current frame, without considering past trajectory locations. This ablation appears to be missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer LE1a,\n\nYour acknowledgment of our approach is sincerely appreciated. We will address your comments below.\n\n**1. Occupancy Reconstruction**\n\n1.1. Details of Occupancy Reconstruction\n\nThe **annotations of occupancy** are generated by gathering depth scores from multi-camera pixels within its boundaries. Then, the accumulated depth scores are averaged to get a single value. This single value represents how likely the voxel is occupied. Our process includes two main steps. First, we project depth information into 3D space. Then, we gather scores from different pixels in the same occupancy region. These scores help us decide if there's an object in that region. We `will` update our paper to make it more clear in the revised version. \n\n1.2. In L207, Should G_xy be G_xyz\n\nYes, thanks for pointing it out. It should be G_xyz. We will correct it in the revised version.\n\n1.3. How occupancy optimizes depth prediction capabilities\n\nAfter the expert model generates occupancy annotations, these annotations are used to guide the apprentice model's occupancy reconstruction process using its depth scores. In this way, this module enables the enhancement of depth prediction capabilities within the apprentice model. Besides, it does not discriminate between dynamic and static objects; rather, this module provides direct supervision for scenarios involving both types of objects.\n\n\n**2. The settings for different tables**\n\n2.1. The different settings between Tab.4 ablation and Tab.1\n\nThe different numbers between Tab.1 and Tab.4 come from the different number of training iterations and different resolutions of BEV. In Tab.1, we reported the results with 90 epochs and 256x256 resolution of BEV feature. Tab. 4 reports ablation studies that use 20 epochs and setting 128x128 resolution of BEV to save computation resources and time. We will add it to the implementation details in Supplementary Sec.A.2 to make it more clear in our updated paper.\n\n2.2. The difference between the first row and the second row in Supplementary Tab.1\n\nExcuse for the confusion. In Supplementary Tab.1, the first row denotes the baseline model which does not use the distillation method. The second row depicts that the VCD-A model directly conducts distillation under the full BEV feature without the Trajectory-based distillation module. We will add the explanation for Supplementary Tab.1 to make it more clear in our updated paper.\n\n**3. More experiments of distilling GT locations in the current frame.**\n\nThanks for your kind suggestions. It is stated in Supplementary Tab.1. As shown in the Table below, The Exp.A is the `simplest` form of Trajectory-based distillation. With the increasing of the trajectory length, our performance of VCD is consistently improved. We will add the explanations for this table in the revised version.\n\n| Exp. | Trajectory length | mAP(%) | NDS(%) |\n| :---: | :---: | :---: | :---: |\n| A   | 1   | 33.1 | 44.5 |\n| B   | 3   | 34.6 | 45.6 |\n| C   | 5   | 35.4 | 45.9 |\n\n**4. Additional details and the number of LiDAR frames to create a depth map, will cause depth artifacts?**\n\nThanks for your suggestions! We use the last sweep LiDAR frame with the current LiDAR frame to create the depth map. The selection of a last sweep LiDAR for depth generation results in a relatively limited time interval, mitigating the occurrence of pronounced depth artifacts. In the nuScenes dataset, the temporal gap between the last sweep frame and the current frame amounts to 0.05 seconds. We will update the details of the expert model in the revised manuscript and `will` release our code to the public. "}, {"review_id": "CO6VGB4rzj", "reviewer": "Reviewer_KRDk", "summary": "The paper presents an innovative approach for improving camera-only 3D object detection. It introduces a vision-centric multi-modal expert and a trajectory-based distillation module to address key challenges in the field. The framework includes an apprentice-friendly multi-modal expert and a fine-grained trajectory-based distillation module to rectify motion misalignment for each object in the scene. The proposed VCD-A model achieves state-of-the-art performance on nuScenes dataset.", "questions": "1. Could you provide provide the dynamic object detection performance gain brought by the trajectory-based distillation module?\n2. You can add a column about the number of frames used in the table. As far as I know, the number of frames used by different methods is inconsistent. How many frames does the VCD use?", "limitations": "How to improve the detection performance of dynamic objects is an urgent problem to be solved at the current stage of 3D perception. The starting point of this article is promising. I hope that the authors can conduct more in-depth research on the real improvement of dynamic object detection performance.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents an innovative approach of incorporating a vision-centric multi-modal expert that exclusively relies on camera features, eliminating the need for a LiDAR backbone. This approach simplifies the model architecture while delivering comparable performance to multi-modal methods.\n2. The trajectory-based distillation module alleviates the problem of motion misalignment in long-term temporal fusion, improving the accuracy of object detection.\n3. This paper provides clear figures and tables, making it easy for readers to understand and follow.", "weaknesses": "1. Typesetting problem, Reference[23] and Reference[24] are repeated. \n2. The Fine-grained Trajectory-based Distillation Module dose not really solve the problem of dynamic target detection, but only enhances the feature representation through multi-frame alignment, thereby improving performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer KRDk,\n\nThank you for the thoughtful and valuable comments. We will address your concerns below.\n\n**1. The repeated Reference[23] and Reference[24]**\n\nIt indeed confuses easily, while these are two different papers.\n\n[23] Li, Yinhao, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. \"Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo.\" arXiv preprint arXiv:2209.10248 (2022).\n\n[24] Li, Yinhao, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. \"Bevdepth: Acquisition of reliable depth for multi-view 3d object detection.\" arXiv preprint arXiv:2206.10092 (2022).\n\n**2. Does Trajectory-based Distillation Module solve the dynamic target detection**\n\nThe Fine-grained Trajectory-based Distillation Module indeed enhances the performance of dynamic objects by a large margin, which can be observed in the Table of Q3 and Supplementary Fig.1. \nThere exists `motion misalignment` during long-term temporal fusion, which will damage the performance of dynamic object detection. In Supplementary Sec.B.1, we have analyzed the motion misalignment. By employing the Trajectory-based Distillation Module during the multi-frame alignment process, we enhance our capability to detect dynamic objects by mitigating motion misalignment. Besides, our expert model possesses a stronger ability to perceive dynamic objects. As a result, we leverage the Trajectory-based Distillation Module to facilitate knowledge transfer from the expert to the apprentice model, thereby augmenting the dynamic object perception capacity of the latter.\n\n**3. The performance gain of dynamic objects brought by the trajectory-based distillation module**\n\nThe trajectory-based distillation module can enhance the performance of dynamic objects, demonstrating 4.0% and 3.2% improvements in `precision-recall` (mAP) and `localization quality` (mATE), respectively. \nTo assess the performance of dynamic objects, we specifically exclude static entities from our analysis. As illustrated in the table presented below, trajectory-based distillation module (`TRM`) yields a substantial enhancement in the performance of dynamic object detection. The dynamic object detection improvement brought by VCD can also be found in Supplementary Fig. 1.\n\n| Dynamic objects  | mAP $\\uparrow$  | mATE $\\downarrow$  | mAOE $\\downarrow$   | mAVE $\\downarrow$  |\n|:--------:|:-----:|:-----:|:-----:|:-----:|\n| baseline | 0.306 | 0.740 | 0.689 | 0.468 |\n| TRM      | 0.346 | 0.708 | 0.602 | 0.404 |\n\n\n**4. Add a column about the number of frames used in the table**\n\nWe appreciate your suggestions and will add them to the revised manuscript. For baseline and VCD experiments, we use 8 frames for temporal fusion. For more detail, please refer to Tab.1 in the attached PDF. \n\n**5. More in-depth research on the dynamic object detection**\n\nThanks for your suggestions! The application of VCD enhances the capability to detect dynamic objects, which highlights the significant role played by knowledge transfer and motion misalignment resolution. Moreover, we will delve deeper into dynamic object detection in the future."}, {"review_id": "dki3rhlSwr", "reviewer": "Reviewer_xwmh", "summary": "This paper aims to design a multi-modal expert teacher with little domain gap to distillate the LSS-based 3D object detector. Different from existential work, it just leverages the LiDAR depth information to design a teacher model instead of using a cumbersome LiDAR feature extractor. The proposed method shows effectiveness to transfer the knowledge from teacher to student detector, which shows significantly results.", "questions": "When using a large backbone like ConvNeXt, you can not provide any evidence to verify the proposed method can boost performance.", "limitations": "Please see the weakness section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Without the aid of a LiDAR feature extractor, it designs an apprentice-friendly multi-modal expert to conduct distillation.\n2. VCD-E achieving comparable performance with state-of-the-art multi-modal methods who predominately rely on the LiDAR backbone.", "weaknesses": "1. The comparison with other multi-modal methods should report the latency to highlight the superiority.\n2. The performance seems to be saturated since the performance of VCD-A with ConvNeXt-B and 8 frames is not good. The Baseline is the   BEVDepth with 2 frames. If the frame is set to 8 frames, what is your improvement compared with the baseline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer xwmh,\n\nThank you for the constructive and thoughtful feedback. We will address your concerns below.\n\n**1. The latency of VCD-E.**\n\nThanks for your suggestions. As shown in the Table below, we add the FPS in the comparison with other multi-modal methods. The FPS is measured by a single A100.\n\n| Methods | Backbone | mAP | NDS | FPS |\n| :---: | :---: | :---: | :---: | :---: |\n| FUTR3D | LiDAR & Image | 0.642 | 0.680 | 2.5 |\n| TransFusion | LiDAR & Image | 0.675 | 0.713 | 3.2 |\n| Ours | Image | 0.677 | 0.711 | 3.5 |\n\n**2. The improvement compared with the baseline using ConvNeXt-B and 8 frames.**\n\nVCD demonstrates a noteworthy enhancement of 2.6% in mAP when compared to our baseline model. This improvement is observed when utilizing the ConvNeXt-B architecture with an input of 8 frames.\n\nOur baseline is the BEVDepth with 8 frames. We want to clarify that both the VCD-A model and our baseline model were implemented using 8 frames. So the main contribution to the improvement of scores lies in the effectiveness of the VCD distillation. Our choice of implementation involves the BEVDet-Depth code, distinct from BEVDepth. It is stated in the caption of Tab.2. The tabulated data below illustrates that the replication of BEVDepth using a 2-frame configuration yields 49.1% mAP, given the unavailability of the BEVDepth `leaderboard` code. We will add the details of the experiment settings in the revised version.\n\n| Model      | Backbone   | # frames | mAP(%) | NDS(%) |\n|:----------:|:----------:|:--------:|:------:|:------:|\n| BEVDepth   | ConvNext-B | 2        | 49.1   | 58.9   |\n| BEVDepth   | ConvNext-B | 8        | 52.2   | 61.0   |\n| VCD (ours) | ConvNext-B | 8        | 54.8   | 63.1   |\n\n\n**3. The consistent improvement of performance based on a large backbone.**\n\nKindly consider referring to our response provided in Q2. We would like to clarify that our VCD distillation method does indeed boost performance by 2.6% mAP with ConvNeXt, which is a significant improvement. Moreover, we `will` release our code to the public to show the improvement achieved through VCD."}, {"review_id": "B5wq05DxNI", "reviewer": "Reviewer_bEge", "summary": "This work presents vision-centric distillation, which utilizes multi-modalities for the expert and images for the student. It includes two distillation parts, namely Trajectory-based Distillation and Occupancy Reconstruction. Experiments prove the effectiveness of the proposed method.", "questions": "Please refer to the Weakness section.", "limitations": "Broader Impact is given in the supplementary material.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method tries to improve the performance via distillation from depth (Occupancy) and motion (Trajectory), which is reasonable for image-based 3D detection.\n2. Experiments show significant improvement over various benchmarks.\n3. The paper presentation is overall clear.", "weaknesses": "1. It is interesting to see the work improves the framework from depth and motion. However, the definition of Occupancy in Figure 2 and Section 3 could be confusing. Because there is no annotation for occupancy optimization in the framework, which cannot guarantee occupancy generation. It's more like depth supervision from expert to apprentice from my point of view.\n2. Because the motion trajectory is generated from the predicted velocity. The difference between Trajectory-based Distillation and previous prediction-based distillation should be made clear. The classic prediction-based distillation also includes velocity for teacher-student optimization.\n3. What's the advantage of a.2 over a.1 in Figure 1(a). Because this work focuses on the camera-based setting. And the backbone for experts (not included in inference) utilizing the image-only or modality-specific approach in Figure 1(a) seems not to make an inherent difference. The essential advantage of the point cloud is accurate depth, which is already utilized in this framework.\n4. It's better to add the results StreamPETR[37] in Table 2, which achieves better results.\n5. It's unclear how the LiDAR-camera depth fusion in Figure 2 is conducted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer bEge,\n\nThank you for the valuable and detailed comments. We will address your concerns below.\n\n**1. The confusion about the definition of Occupancy and how to generate the occupancy annotation.**\n\nExcuse for the confusion. The **definition** of occupancy used in the paper is to determine the presence of objects in a given 3D space based on depth scores generated from multi-camera pixel space. \n\nThe **generation of occupancy annotation** is obtained by projecting the depth information into the 3D space. Then the scores from different pixels in the multi-camera within the same occupancy region, are accumulated to make decisions about the presence of an object in that region. \n\nIMHO, our occupancy task `differs` from depth supervision. Occupancy is defined within a three-dimensional voxelized space, whereas depth is defined in a two-dimensional image space.\n\nWe `will` revise accordingly to make it more clear in the manuscript.\n\n**2. The motion trajectory is generated from the predicted velocity. The difference between Trajectory-based Distillation and previous prediction-based distillation methods.**\n\nNote that, the motion trajectory is indeed generated by the past GT instances rather than the predicted velocity (As stated in L177).\nThis is a notable difference with previous prediction-based distillation. Specifically, we transform the history's GT instances into the current coordinate system to construct the trajectory. In this way, we can capture the moving objects to mitigate their misalignment during the long-term temporal fusion process which is ignored by previous distillation methods. As shown in Tab.6, our method significantly outperforms previous methods by alleviating motion misalignment.\n\n**3. What's the advantage of a.2 over a.1 in Figure 1(a)**\n\nThe most advantage of the VCD a.2 over a.1 is the elimination of the domain gap between expert and apprentice models. Our objective is to create an expert model that eliminates the domain gap typically present between pure vision models and multi-modal models. Our expert model a.2 only encodes visual information while utilizing LiDAR solely for geometric data, thus minimizing the domain gap. The experiments in Tab.5 demonstrate that our expert model a.2 has the prominent advantage over other multi-modality models a.1.\n\n**4. Add the results StreamPETR[37] in Table 2**\n\nThanks for pointing it out, we will revise accordingly.\n\n**5. The process of LiDAR-camera depth fusion in Fig. 2**\n\nAs stated in L284, VCD-E employs the point cloud data for pixels with available Ground Truth (GT) depth information. For pixels lacking GT depth information, we rely on the predicted depth values obtained from images. \n\nWe will add it to Sec.3 accordingly. "}, {"review_id": "bX5bGbKbUD", "reviewer": "Reviewer_oCet", "summary": "This paper introduces a novel cross-modal BEV distillation approach, namely VCD, but adopts identical feature extractors for the teacher and the student model, which eases the distillation difficulties during the knowledge-transferring phase. \n\n(1) It proposes a new multi-modal teacher VCD-E, with an image-based backbone only to serve as the teacher model, and achieves similar performance with multi-modal 3D detectors. \n\n(2) It introduces a novel trajectory-based feature distillation approach, to enhance the feature distillation quality on moving objects.\n\n(3) The final model achieves 63.1 NDS on nuScenes test leaderboard, setting SOTA performance for camera-based detectors.", "questions": "1. The official BEVDet4D-depth is 51.9NDS, while the author report 54 NDS. What's the modification to the model?\n2. The reviewer is interested in the static objects' performance similar to Figure 1(supplementary). It is not straightforward to see moving objects' improvements only, since VCD has already improved over its baseline. \n2. The author is suggested to clarify that VCD-E utilizes future frames during training.\n3. The author is suggested to use \\times rather than x in Table 1, which is not formal.", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper points out that a LiDAR-based detector may not be essential in camera-based 3D detectors' distillation: a dedicated image-based detector with future frames temporal fusion and LiDAR-guided depth input is enough. Based on this, the author overcomes the domain gap between LiDAR and images easily.\n2. The paper presents a novel trajectory-based distillation approach to deal with moving objects, which is simple yet effective according to ablations. ", "weaknesses": "1. The paper lacks some theoretical analysis, which is practical work.\n2. Though the trajectory-based distillation is effective, the reviewer does not think VCD truly captures the moving objects. A more detailed experiment may help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer oCet,\n\nWe sincerely appreciate your valuable feedback. We will address your comments below.\n\n**1. lacks some theoretical analysis**\n\nThanks for your suggestions. We have done some analysis on the motion misalignment during the process of temporal fusion in Supplementary Sec.B.1. We will delve deeper into the relevant research areas to conduct further analysis in the future.\n\n\n**2. An experiment to show VCD captures the moving objects**\n\nIn response to the reviewer's concerns, we conducted the following experiments. We categorized object velocities into three tiers: 0-1, 1-5, and 5-. The tabulated results underscore a substantial enhancement achieved by our VCD method for objects with higher velocities, demonstrating 32.6% and 20.0% improvements in `precision-recall` (mAP) and `localization quality` (mATE), respectively. \\* denotes the percentage improvement relative to the baseline. \n\n| mAP$\\uparrow$ |  velocity (0 - 1) | velocity (1 - 5) | velocity (5 - )  |\n| :---: | :---: | :---: | :---: |\n| Baseline | 0.289 | 0.115 | 0.146 |\n| VCD | 0.338 (+16.8%)\\* | 0.155 (+34.9%) | 0.194 (+32.6%) |\n\n| mATE$\\downarrow$ | velocity (0 - 1) | velocity (1 - 5) | velocity (5 - ) |\n| :---: | :---: | :---: | :---: |\n| Baseline | 0.729 | 0.731 | 0.885 |\n| VCD | 0.685 (-6.1%) | 0.680 (-7.0%) | 0.708 (-20.0%) |\n\n**3. The modification based on the official BEVDet4D-depth**\n\nWe modify the resolution of the BEV feature based on the official BEVDet4D-depth from 128 to 256, which causes the gap. Notably, in the main experiments, including those involving the baseline and VCD, the resolution was consistently adopted for the 256.\n\n**4. The static objects' performance**\n\nThanks for your advice. As depicted in the table below, the performance for static objects exhibits a notable improvement of over 3.3% in mAP. We also provide Fig.2 similar to Supplementary Fig.1 in the attached PDF.  We will add it in the corresponding section in the revised version.\n| mAP$\\uparrow$ | Con Veh | Traffic Cone | Barrier |\n| :---: | :---: | :---: | :---: |\n| Baseline | 0.060 | 0.463 | 0.495 |\n| VCD | 0.099 | 0.505 | 0.512 |\n\n**5. Clarify that VCD-E utilizes future frames during training.**\n\nThanks for your kind advice. It is stated in line 29 of the supplement. We will clarify it more clearly in the revised version.\n\n**6. Suggested to use \\times rather than x in Table 1**\n\nWe appreciate your suggestion and will fix them in the revised version."}]

Author rebuttal:
Dear Reviewers and AC(s):

We extend our gratitude. Below, we address all review comments and incorporate accordingly into the revised manuscript of our work.

The attached PDF below includes a table and a figure. The table illustrates the number of frames used by different approaches, addressing the concern from `Reviewer KRDk`. The figure presents the performance evaluation of static objects using VCD, addressing the remarks from `Reviewer oCet`.





Meta-review: This paper proposes a camera-only 3D object detection method using distillation techniques from expert models with richer sensor information, such as LiDAR. The proposed model achieved state-of-the-art performance on the nuScenes dataset. All reviewers appreciated the novel and innovative ideas and experimental results of this paper, and there were very few negative comments in the weaknesses section, mostly questions about unclear issues and suggestions for additional experiments. Almost all reviewers were satisfied with the authors' rebuttals. One reviewer lowered his/her rating once after the rebuttal, but after further discussion with the authors, the final rating (=WA) was restored. In conclusion, we believe that this paper is solid and valuable and should be accepted by this conference.
Decision: Accept (poster)