BEVDistill: Cross-Modal BEV Distillation for Multi-View 3D Object Detection
Authors: Zehui Chen
Zhenyu Li
Shiquan Zhang
Liangji Fang
Qinhong Jiang
Feng Zhao
Venue: ICLR 2023 poster (2023)
Keywords: object detection
3d detection
BEV perception

Abstract: 3D object detection from multiple image views is a fundamental and challenging task for visual scene understanding. Owing to its low cost and high efficiency, multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely difficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deployment. Different from the images, LiDAR points are superior in providing spatial cues, resulting in highly precise localization. In this paper, we explore the incorporation of LiDAR-based detectors for multi-view 3D object detection. Instead of directly training a depth prediction network, we unify the image and LiDAR features in the Bird-Eye-View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm. To this end, we propose BEVDistill, a cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection. 
Extensive experiments demonstrate that the proposed method outperforms current KD approaches on a highly-competitive baseline, BEVFormer, without introducing any extra cost in the inference phase. Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-arts in comparison with various image-based detectors. Code will be available at https://github.com/zehuichen123/BEVDistill.

Published as a conference paper at ICLR 2023
BEVDISTILL: CROSS-MODAL BEV DISTILLATION FOR MULTI-VIES 3D OBJECT DETECTION
Zehui Chen', Zhenyu Shiquan Zhang², Liangji Fang?, Qinhong Jiang', Feng Zhao
1 University of Science and Technology of China
2 Harbin Institute of Technology
3 TemseTime Research
lovesnow@mail. www.ustt.c... edu. cn, fzhao356lustc.edu zhenyali17@hit. edu.cn
(zhangshiquan, fangl l
ABSTRACT
3D object detection from multiple image views is a fundamental and challeng- ing task for visual scene understanding. Owing its low cost and high efficiency. multi-view 3D object detection has demonstrated promising application prospects. However, accurately detecting objects through perspective views is extremely dif- ficult due to the lack of depth information. Current approaches tend to adopt heavy backbones for image encoders, making them inapplicable for real-world deploy- ment. Different from the images, LiDAR points superior in providing spatial cues, resulting in highly precise localization. In this paper. we explore the incor- poration of LiDAR- based detectors for multi-view 3D object detection. Instead of directly training a depth prediction network, we unify the image and LiDAR fea- tures in the Bird-Bye-Viiee View (BEV) space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm. To this end, we propose BEvtistil,, cross- cross-modal BEV knowledge distillation (KD) framework for multi-view 3D object detection. Extensive experiments demonstrate that the proposed method outperforms current KD approaches on highly-competttive baseline, BEVFormer, without introducing any extra cost in the inference phase. Notably, our best model achieves 59.4 NDS on the nuScenes test leaderboard, achieving new state-of-the-arts in comparison with various image-based detectors. Code will be available at https///ittubb.com/cccucchhennn 23/BEVDistill.
INTRODUCTION
3D object detection, aiming at localizing objects in the 3D space, is crucial ingredient for 3D scene understanding. has been widely adopted various applications, such as autonomous driving (Chen et al., 2022a; Shi et al., 2020; Wang et al., 2021b), robotic navigation (Antonello et al., 2017), and virtual reality (Schuemic et al., 2001). Recently, multi-view 3D object detection has drawn great attention thanks to its low cost and high efficiency. As images offer a discriminative appearance and rich texture with dense pixels, detectors can easily discover and categorize the objects, even a far distance. Despite the promising deployment advantage, accurately localizing instances from camera view only is extremely difficult, mainly due to the ill-posed nature of monocular imagery. Therefore, recent approaches adopt heavy backbones g., GesNet-101-DCC (He et al., 2016), VoVNetV2 (Lee & Park, 2020)) for image feature extraction, making it inapplicable for real-world applications.
LiDAR points, which capture precise 3D spatial information, provide natural guidance for camera- based object detection. In light of this, recent works (Guo et al.. 2021b; Peng et al., 2022) start to explore the incorporation of point clouds in 3D object detection for performance improvement. One line of work (Wang et al., 2019b) projects each points to the image to form depth map labels, and subsequently trains a depth estimator explicitly extract the spatial information. Such paradigm generates intermediate products, i.e., depth prediction maps, therefore introducing extra computa- tional cost. Another line of work (Chong et al., 2021) I is to leverage the teacher-student paradigm for
"Corresponding author

---

Published as a conference paper at ICLR 2023
knowledge transfer. (Chong et al., 2021) projects the LiDAR points to the image plane, constructing a 2D imput for the teacher model. Since the student and teacher models are exactly structurally iden- tical, feature mimicking can be naturally conducted under the framework. Although it solves the alignment issue across different modalities, it misses the opportunity of pursuing a strong LiDAR- based teacher, which is indeed important in the knowledge distillation (KD) paradigm. Recently, UVTR (Li et al., 2022a) propose to distill the cross-modal knowledge in the voxel space, while maintaining the structure of respective detectors. However, it directly forces the 2D branch to imi- tate the 3D features, ignoring the divergence between different modalities.
In this work, by carefully examining the non -omooooggeouu features represented in the 2D and 3D spaces, we explore the incorporation of knowledge distillation for multi-view 3D object detection. Yet, there are two technical challenges. First, the views of images and LiDAR points are different, Le., camera features are in the perspective view, while LiDAR features are in the 3D//irdd Such view discrepancy indicates that natural one-to-one imitation (Romero et al., 2014) may not be suitable. Second, RGB images and point clouds hold respective representations in their own modalities. Therefore, it can be suboptimal to mimic features directly, which is commonly adopted in the 2D detection paradigm (Yang al., 2022a; Zhang & Ma, 2020).
view.
We address the above challenges by designing a cross-modal BEV knowledge distillation frame- work, namely BEVDistill. Instead of constructing separate depth estimation network or explicitly projecting one view into the other one, we convert all features to the BEV space, maintaining both geometric structure and semantic information. Through the shared BEV representation, features from different modalities are naturally aligned without much information loss, After that, we adap- tively transfer the spatial knowledge through both dense and sparse supervision: (i) we introduce soft foreground-guides distillation for non-homogenous dense feature imitation, and (ii) sparse style instance- wise distillation paradigm is proposed to selectively supervise the student by maxi- mizing the mutual information.
Experimental results on the competitive nuScenes dataset demonstrate the superiority and gener- alization of our BEVDistill. For instance, we achieve 3.4 NDS and 2.7 NDS improvements on a competitive multi-view 3D detector, BEVFormer (Li et al., 2022c), under the single-frame and multi-frame settings, respectively. Besides, we also present extensive experiments on lightweight backbones, as well as detailed ablation studies to validate the effectiveness of our method. Notably, our best model reaches 59.4 NDS on nuScenes test leaderboard, achieving new state-of-ie of-the-art re- sults among all the published multi view 3D detectors.
RELATED WORK
2.1 VISION-BASED 3D OBJECT DETECTION
-bsion- 3D object detection aims to detect object locations, scales, and rotations, which is of great importance in autonomous driving (Xie et al., 2022; Jiang al., 2022; Zhang al., 2022) and augmented reality (Azuma, 1997). One line of work is to detect 3D boxes directly from single images. Mono3D (Chen et al., 2016) utilizes traditional methods to lift 2D objects into 3D space with semantic and geometrical information. In the consideration that objects located at different distances appear on various scales, D4LCN (Ding et al., 2020) proposes to leverage depth prediction for convolutional kernel learning. Recently, FCOS3D (Wang et al., 2021a) extends the classical 2D paradigm FCOS (Tian et al., 2019b) into monocular 3D object detection. It transforms the regression targets into the image domain by predicting its 2D- -D-ddsed attributes. Further, PGD (Wang et al., 2022b) introduces relational graphs to improve the depth estimation for object localization. MonoFlex (Zhang et al., 2021) argues that objects located at different positions should not be treated equally and presents auto-adjusted supervision.
Another line of work predicts objects from multi-view images. DETR3D (Wang et al., 2022c) first incorporates DETR (Carion et al., 2020) into 3D detection by introducing a novel concept: 3D ref- erence points. After that, Graph-DETR3D (Chen et al., 2022b) extends it by enriching the feature representations with dynamic graph feature aggregation. Different from the above methods, BEVDet (Huang et al., 2021) leverages the Lift-splat-shoot (Philion & Fidler, 2020) to explicitly project im- ages into the BEV space, followed by a traditional 3D detection head. Inspired by the recently de- veloped attention mechanism, FoVrrrerr (Li et al., 2022c) automates the cam2bev process with
2

---

Published as a conference paper at ICLR 2023
learnable attention manner and achieves superior performance. PolarFormer (Jiang et al., 2022) introduces polar coordinates into the model construction of BEV space and greatly improves the performance. Moreover, BEVDepth (Li al., 2022b) improves BEVDet by explicitly supervising depth prediction with projected LiDAR points and achieves state-of-the-art performance.
2.2 KNOWLEDGE DISTILLATION IN OBJECT DETECTION
Most KD approaches for object detection focus on transferring knowledge among two homogenous detectors by forcing students' predictions to match those of the teacher (Chen et al., 2017: Dai et al., 2021; Zheng al., 2022). More recent works (Guo al., 2021a; a; Wang et al., 2019a) find that imitation of feature representations is more effective for detection. A vital challenge is to determine which feature regions should distilled from the teacher model. FGFI (Wang et al., 2019a) chooses features that are covered by anchor boxes whose IoU with GT is larger than certain threshold. PGD (Yang et al., 2022a) only focuses on a few key predictive regions using a combination of classification and regression scores measure of quality.
Despite a large number works discussing the KD in object detection, only a few of them consider the multi-modal setting. MonoDistill (Chong al., 2021) projects the points into the image plane and applies a modified image-based 3D detector as the teacher to distill knowledge. Such a paradigm solves the alignment issue naturally, however, it misses the chance pursue a much stronger point- based teacher model. LIGA-stereo (Guo et al.. 2021b) leverages the information of LiDAR by supervising the BEV representation of the vision-based model with the intermediate output from the SECOND (Yan et al., 2018). Recently, UVTR (Li et al., 2022a) presents a simple approach by directly regularizing the voxel representations between the student and teacher models. Both of them choose to mimic the feature representations across the models, while ignoring the divergence between different modalities.
METHODS
In this section, we introduce our proposed BEVDistill in detail. We first give an overview of the whole framework in Figure and clarify the model designs of the teacher and student model in Section 1. After that, we present the cross-modal knowledge distillation approach in Section 3.2, which consists of two modules: dense feature distillation and sparse instance distillation.
3.1 BASELINE MODEL
Student Model. We adopt current state-of-the-art camera-based detector, BEVFormer (Li et al., 2022c), as our student model. It consists of an image backbone for feature extraction, a spatial cross-altention module for cam2bev view transformation, and transformer head for 3D object detection. Besides, provides a temporal cross-attention module to perceive subsequential multi- frame information for better predictions.
Teacher Model. In order keep consistent with the student model, we select Object-DGCNN
(Wang & Solomon, 2021) as our teacher model. For simplicity and generality, we replace the DGCNN attention with a vanilla multi-scale attention module. It first casts 3D points into the BEV plane and then conducts one-to-one supervision with transformer-based label assignment. We train the model by initializing it from a pre-trained CenterPoint model.
3.2 BEVDISTILL
BEVDistill follows the common knowledge distillation paradigm, with 3D point cloud detector as teacher and an image detector as student. Different from previous knowledge distillation meth- ods, which keep the same architecture for both student and teacher models (except for backbone), BEVDistill explores a more challenging setting with non-hooogenous representations.
3.2.1 DENSE FEATURE DISTILLATION
In order to conduct dense supervision for feature distillation, we first need to determine the BEV features generated from both models. For the student, directly adopt the BEV features maps
3

---

Published as a conference paper at ICLR 2023
Voxellzation
|00
DETR Encoder
BEV Feature
DETR Decoder
Supervise
Pred
30 Points
Backbone
Dense Feature
Sparse Instance
20 Surrounding Images
Distillation
Distillation
2D Backbone
DETR Encoder
DETR Decoder
BEV Feature
GT
Figure 1: The overall architecture of the proposed BEVDistill consists of three main components: a transformer- based LiDAR detector, transformer-basee image detector, and the proposed cross- modal distillation modules. It follows the teacher-student paradigm with both RGB images and LiDAR points as the inputs and keeps the original structure of respective model structures.
D2D produced by the BEVTrans frrrrannsforerrncodeee In order align the feature representations between the teacher and student, we choose the same BEV feature outputted by the transformer encoder for the teacher model. Different from LIGA-Stereo (Guo al., 2021b) which directly mimics the BEV features extracted from the 3D backbone, we postpone such supervision after the transformer encoder (illustrated in Figure 1), which offers more opportunities for the network to align information across different modalities.
Previous works (Romero al., 2014; et al., 2022a) directly force the student to mimic the feature map from the teacher and achieve significant performance improvement:
feat
H W ΣΣ
F
F2D 1l2..
where H, W denotes the width and height of the distilled feature map, and || ||2 is the L2 However, such strategy may not work well under the cross-modal feature setting. Although the view discrepancy eliminated by projecting both features into the BEV plane, there is still domain discrepancy between different modalities: despite the same scenes being captured with the point clouds and images with careful alignment, the representation itself can be diverse under different modalities. For instance, images holds pixels for both foreground and backgrounds regions, however LiDAR points only appears when there exists objects to reflect the ray.
Consider that the regional 3D features can hold meaningful information only if the point exists, we regularize the distillation within the foreground regions. Besides, we notice that boundary of foreground could also provides useful information. Therefore, instead of performing hard super- vision on each foreground region, we introduce a soft-supervision manner similar to that in (Zhou et al., 2019). Concretely, draw Gaussian distribution for each ground truth center (I,, yi) in the BEV space with,
(x) 22)2 + (y/ 2,)
W1,z3 exp(
20
where o; is constant (set 2 by default), indicating the object size standard deviation. Since the feature maps are class-agnostic, we merge all Wi..,y into one single mask W. For overlapping regions among different Wi,z,y at the same position, simply take the maximum value of them.
After that, we enforce the student to mimic the feature with the bnngrroud-guideee mask W for dense feature distillation:

---

Published as a conference paper at ICLR 2023
W
Leat H W x Σ max (Ws))
Σ
Σ
maz (W;) (W)))ll
F3
(3)
This foreground-guie -uguddddddddddddddd-dd re- we-ghttht strategy allows the model to focus the foreground regions from the teacher, and at the same time avoid imitation of useless empty 3D features in the background regions. Such manner is counter to the finding in previous homogeneous knowledge distillation works (Yang et al., 2022c).
3.2.2 SPARSE INSTANCE DISTILLATION
Instance-level distillation can be easily conducted under the dense prediction setting, where only pixel-to-pixel mapping is needed for supervision. However, both the student and teacher models in BEVDistill act in a sparse- preddccpeeintion style. Therefore, set-to-set mapping is required to ensure instance-level distillation. To achieve the goal, we simply follow the practice in (Wang & Solomon, 2021) to construct the correspondence between the student and teacher predictions. Specifically, suppose the categorical and localization predictions output by the in query from the teacher model are of and 6f" and the student can be represented as cf and be permutation of & can be found between the output set the teacher and the student with the lowest cost:
ō arg min Σ Lmatch (yis 9t),
(4)
where Lmatch (y/s Ui) is pair-wise matching cost, which is defined as:
Liaatcc (yi, Us) log ( (cF) I bonl
(5)
However, we empirically find such a vanilla distillation approach works poorly in a cross-domain supervision manner. There are two main issues that impede the model from further performance improvement. On one hand, not all predictions from the teacher should be treated equally as valuable cues, since most of the predictions are false positives with low classification score. On the other hand, though the classification logits can represent rich knowledge (Hinton et al., 2015), it may not hold when the input data are different. Directly distilling these prediction will introduce great noise to the model and deteriorate the performance. To tackle the problem, we exploit reliable quality score derived from the teacher model measure the importance of the instance-level pseudo labels and use the score soft re- re-weighted factors to eliminate the potential noise predictions made by the teacher. Specifically, we consider the classification score c; (categorical information) and the IoU between the prediction grred and the ground truth &GT (localization information) together to formulate the quality-score g::
Q: = (c)) LOU 1pred)1-v.
(6)
The quality score serves as an indicator to guide the student on which teacher's predictions should be paid more weight. Therefore, the final instance-level distillation be written as:
N
Linst
Σ qi (aLeii e4') + BLoo (bt 88 ()))))
where Lets is the KL. divergence loss, Lbor the L2 loss in (Wang & Solomon, 2021), and a, B are reweighted factors to balance the supervision (set to 1.0 and 0.25 by default). Given that the prediction probability mass function over classes contains richer information than one-hot labels, student models are proven to be benefited from such extra supervision (Hinton al., 2015). How- ever, this low dimensional prediction distribution (number of classes) means that only a few amount of knowledge is encoded, thus limiting the knowledge that be transferred, especially under the cross cross-modal modal setting. Besides, the representational knowledge is often structured, with implicit com- plex interdependencies across different dimensions, while the KI. objective treats all dimensions
5

---

Published as a conference paper at ICLR 2023
independently. To overcome the problem, instead of minimizing the KL divergence two distri- butions, we choose to directly maximize the mutual information (Mf) between the representations h h² the penultimate layer (before logits) from the teacher and student networks:
(8) where phh",h" is the joint distribution and the marginal distributions p(h"), v(h"). We then define a distribution & conditioned on 7) that captures whether the pair is congruent (q(1) 1)) or incongruent (1(1) O)):
N("";~²> (L(p(h", (lp(h")) (u(h)) )())())))))))))
qhh h n 1) p(h (ilh,hh) ),)), 1 0) =
(9)
With Bayes' rule, we can obtain the posterior for 1j
p(h² 1:
<n = 11-= p(hT, h15) u(h (o(s))
(10)
By taking the logarithm of both sides, we have
p(h²", J.5) p(h", h2) ) + u(h") ((hT)u(h)) = I(h?) ,15).
log q() = 1|h", .0i,,,5)) & log
(11)
Therefore, the objective be transformed maximize the lower bound of MI. Since there is no closed form for q(7) = 1|h" h)S), we utilize a neural network g. called critic (Tian et al., 2019a) to approximate with NCE loss (see Appendix A for detailed implementation)::
Let ,1 E (0007,00==1)1000 g(h [T)]] + Eygae 10 [lo (1 - s(h ,,))))
(12)
To this end, the final sparse instance distillation loss can be represented as:
Linst Σ as (o (T) + BLoo (b% by
(13)
4 EXPERIMENTS
4.1 DATASET AND EVALUATION METRICS
We conduct the experiments on the NuScenes dataset (Caesar et al., 2020), which one of the most popular datasets for 3D object detection. It consists of 700 scenes for training, 150 scenes for validation, and 150 scenes for testing. For each scene, it includes 6 camera images (front, front left, front right, back left, back right, back) to cover the whole viewpoint, and 360 LiDAR point cloud. Camera matrixes including both intrinsic and extrinsic are provided, which establish a one-to-one correspondence between each 3D point and the 2D image plane. There are 23 categories in total. and only 10 classes are utilized to compute the final metrics according to the official evaluation code.
We adopt the official evaluation toolbox provided by nuScenes. We report nuScenes Detection Score (NDS) and mean Average Precision (mAP), along with mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Ve- locity Error (mAVE), mean Average Attribute Error (mAAE) in our experiments. The NDS score weighted sum of mean Average Precision (mAP) and the aforementioned True Positive (TP) metrics, which is defined as NDS [5 mAP + EmmPETT (1-min1,,
min(1, mTP)).
4.2 IMPLEMENTATION DETAILS
Our codebase is built on MMDetection3D (Contributors, 2020) toolkit. All models are trained on 8 NVIDIA A100 GPUs. We first train the 3D detector with voxel size of (0.1m, 0.1m, 0.2m) followed in (Yin et al., 2021). During the distillation phase, the batch size is set to per GPU with an initial learning rate of 2e-4. Unless otherwise specified, we train the models for 2 x schedule (24 epochs) with a cyclic policy. The input image size is set to 1600 x 900 and the grid size of the BEV plane in BEVFormer is set to 128 128. FP16 training is enabled for GPU memory management.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of recent distillation works the nuScenes validation set with ResNet-50 image backbone. * indicates our own re-implementation with modifications.
Detector
Setting
NDS1
MAPt
Teacher
67.4
61.5
Student
42.3
35.2
FitNet (Romero et al., 2014)
41.1
34.4
BEVFormer
Set2Set (Wang Solomon, 2021)
41.0
33.1
MonoDistills (Chong et al., 2021)
42.9
36.4
UVTR (Li al., 2022a)
43.1
36.2
BEVDistill (Ours)
45.7(+3.4)
38.6(+3.4)
Teacher
67.4
61.5
Student
48.8
38.3
FitNet (Romero et al., 2014)
48.0
37.3
BEVFormer-T
Set2Set (Wang & Solomon, 2021)
47.9
37.5
MonoDistills (Chong et al., 2021)
49.5
39.0
UVTR (Li et al., 2022a)
50.1
39.4
BEVDistill (Ours)
51.5(+2.7)
40.7(+2.4)
4.3
MAIN RESULTS
We first implement BEVFormer and BEVFormer- on the nuScenes validation subset with ResNet- 50 backbone and compare BEVDistill with other knowledge distillation methods. The final results are shown in Table 1. Our BEVDistill greatly boosts its vanilla student model, BEVFormer, by more than 3.4/2.3 mAP and 3.4/2.7 NDS, respectively, without introducing any extra inference cost. Clas- sical and effective distillation methods like FitNet (Romero et al., 2014). fail to bring any enhance- ment and even deteriorate the final performance, i.e., -1. 1.2% NDS. The Set2Set (Wang & Solomon, 2021) also gets struck in the detection accuracy (-1.3% NDS) due to the miss consideration of the divergence between different modality features. UVTR (Li al., 2022a), which is the first work to leverage 3D points for multi-view 3D object detection, manages to get 0.8 NDS improvement. However, only focuses on feature-based distillation while missing the potential benefits other aspects. We also reimplement MonoDistill (Chong et al., 2021) by extending its monocular version to multi-view one, yielding an improvement 0.7 NDS.
4.4 COMPARISON WITH STATE-OF-THE-TA
In addition to offline results, we also report the detection performance nuScenes test leader- board compared to various multi-view 3D detection approaches to further explore the potential of BEVDistill. The results shown in Table 2. Our final model is based current sate-offtthear model BEVDepth with an input image size of 640 x 1600. We adopt ConvNeXI-base (Liu et al., 2022d) as image encoder with BEV space of 256 256. The model is trained 20 epochs with CBGS and no test-time augmentation applied. It surpasses all other detectors, including the recently developed PETRv2, BEVFormer, and BEVDet4D by more than 1.0 NDS, achieving new state-of-the-art on this competitive benchmark.
4.5
ABLATION STUDIES
In this section, provide detailed ablation studies different components of our model. For efficiency, we adopt BEVFormer R50 as base model with schedule on 1/2 training subset.
4.5.1 MAIN ABLATIONS
To understand how each module contributed to the final detection performance in BEVDistill, test each component independently and report its performance in Table 3. The overall baseline starts from 42.2 NDS. When dense feature distillation is applied, the NDSS is raised by 2.5 points, which

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of recent works the nuScenes testing "I." and "C" indicate LiDAR and Camera, respectively. & denotes our reimplementatton borrowed from BEVDet.
Method
M.
NDS
mAP
mATE
mASE
mAOB
mAVE mAAE
Reference
PointPillar
L
45.3
30.5
CVPR2019
CenterPoint
I.
65.5
58.0
CVPR2021
BEVFusion
L+C
72.9
70.2
0.239
0.329
0.260
0.134
0.153
Arxiv2022c
FCOS3D
C
42.8
35.8
0.690
0.249
0.452
1.434
0.124
ICCVW20211
Ego3RT
47.3
42.5
0.549
0.264
0.433
1.014
0.145
ECCV2022
DETR3D
C
47.9
41.2
0.641
0.255
0.394
0.845
0.133
CoRL2022c
BEVDet
48.8
42.4
0.524
0.242
0.373
0.950
0.148
Arxiv2021
G-DETR3D
C
49.5
42.5
0.621
0.251
0.386
0.790
0.128
MM2022b
PETR
50.4
44.1
0.593
0.249
0.384
0.808
0.132
ECCV2022a
UVTR
C
55.1
47.2
0.577
0.253
0.391
0.508
0.123
NeurIPS2022a
BEVDet4D
56.9
45.1
0.511
0.241
0.386
0.301
0.121
Arxiv2022
BEVFormer
C
56.9
48.1
0.582
0.256
0.375
0.378
0.126
ECCV2022e
PETRv2
58.2
49.0
0.561
0.243
0.361
0.343
0.120
Arxiv2022b
BEVDepth
C
58.9
49.1
0.484
0.245
0.377
0.320
0.132
Arxiv2022b
BEVDistill
59.4
49.8
0.472
0.247
0.378
0.326
0.125
Table 3: Effectiveness of each component in BEVDistill. Results reported BEVFormer-RSS with standard x schedule setting.
Dense Feature Distill
Sparse Instance Distill
NDSt
mAPt
mAOB +
42.3
35.2
0.428
44.7
38.0
0.416
43.9
37.1
0.410
45.7
38.6
0.399
validates the finding that disentangling the foreground regions with soft supervision can greatly promote the student's detection ability. Then, we add the sparse instance distillation, which brings us a 1.7% NDS enhancement. Finally, the NDS score achieves 45.7 when all components are applied, yielding 3.4 % absolute improvement, validating the effectiveness BEVDistill.
4.5.2 ABLATIONS ON DENSE FEATURE DISTILLATION
Comparison with other feature-based distillation mechanisms.
Feature-basee distillation methods have been extensively explored in both image classification (Hin- al., 2015) and object detection tasks (Yang al., 2022b:c). Therefore, we compare our dense feature distillation module with other competitive feature- based distillation approaches. The results are shown in Table FitNet distills all pixels across 2D and 3D features with equal supervision, yielding little improvement. FGD (Yang et al., 2022b) is current state-of-the tion approach in 2D object detection. However, neglects the divergence between different modal- ities across features, therefore gets the worse performance. UVTR (Li et al., 2022a) also mimics the features directly but it only selects positive regions (query points) for feature imitation. Compared with the above methods, BEVDistill considers both modality divergence and region importance for feature imitation, providing the highest accuracy.
knowledge distilla-
Strategies foreground mask generation.
As mentioned in Section 3.2.1, not all positions in BEV feature representation should be treated equally. How to select the most informative regions for feature imitation is also non-trivial. Hence, we provide detailed experiments on how to generate the foreground imitation mask for distillation. Intuitively, directly select the center points of the ground truth objects (GT-Center) or the
8

---

Published as a conference paper at ICLR 2023
Table 4:
Comparison feature-based
Table 5: Comparison foreground mask genera- tion approaches.
knowledge distillation approaches.
Method
NDS
mAP
mAOE
Method
NDS
mAP
mAOE
37.5
33.3
0.515
37.5
33.3
0.515
FitNet
37.8
34.4
0.549
GT-Center
37.7
34.2
0.546
FGD
36.5
32.9
0.534
Query-Center
38.1
34.4
0.513
UVTR
38.1
34.5
0.513
Pred- Pred-Heatmap
38.6
34.5
0.533
BEVDistill
38.9
34.9
0.501
GTT-meamap
38.9
34.9
0.501
query reference points (Query-Center) as imitation instances. However, such a strategy directly abandons most areas, which may contain useful information. Therefore, we attempt to convert such a 'hard' decision into the 'soft' manner with a heatmap mask, which can be the model classification prediction from the teacher (Pred-Heatmap) or the ground truth heatmap (GT-Heatmap). The final results are shown in Table The soft foreground mask generation is much better compared to the hard one. We choose the GT GT-Heatmap strategy in experiments due to its highest score.
4.3.3 ABLATIONS ON SPARSE INSTANCE DISTILLATION
Table Ablations on loss selection for clas- sification and regression branches.
Table Ablations on critic g selection and pair generation.
cls
reg
NDS
mAP
mAOE
critic g
Pair
NDS
mAP
CE
35.8
31.7
0.544
37.5
33.3
L1
38.1
33.9
0.521
NCS Loss
Pos
38.0
33.7
CE
L1
37.0
32.7
0.531
CE Loss
Pos
35.8
31.7
InfoNCE
L1
39.1
34.6
0.474
InfoNCE Loss
Pos
Neg
38.6
34.5
Ablations the instance distillation loss.
Instance-level distillation a necessary module in detection- knowledge distillation. However, such a component has been seldom discussed in transformer-based sparse prediction setting. In this ablation, we conduct detailed experiments on the loss selection in Table 6. We first conduct the commonly adopted cross-entropy and LI loss for the classification and regression branches. The sult is surprisingly lower than its vanilla baseline. When adding them respectively, we figure out that the cross-entropy the key issue leading to performance degradation. To avoid the problem, we re- place it with the InfoNCE loss along with the contrastive paradigm, and an immediate improvement observed, which validates our assumption in Section 3.2.2.
Strategies on selection of critic g.
Since most self self- supervised learning approaches are based on homogenous representations, exploring the suitable contrastive paradigm for cross-modality is important. Note that features located in the same positions the and 3D BEV feature maps are considered positive pairs while the rest are negative pairs We compare three prototypes of critic g and report the results in Table 7. We first adopt the classic InfoNCE loss, which provides the most improvement. We also conduct experiments on positive pairs only with NCS (negative cosine similarity) and CE (cross-entropyy loss, while gets little enhancement, validating the proof in Section 3.2.2.
5 CONCLUSION
this paper, we introduce cross-modal knowledge distillation framework for multi-view 3D object detection, namely BEVDistill. It unifies different modalities in the BEV space to case the modal alignment and feature imitation. With the dense feature distillation module, it can effectively transfer rich teacher knowledge the student across 2D and 3D data. BEVDistill also explores sparse instance distillation approach under the transformer bassedd detection paradigm. Without bells and whistles, achieves state-of-the-art performance the competitive nuScenes test leaderboard.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported by the JKW Research Funds under Grant 20-166-14412-001-0044 and the Anhui Provincial Natural Science Foundation under Grant 2108085UD12. We acknowledge the support of GPU cluster built by MCC Lab of Information Science and Technology Institution, USTC.
REFERENCES
Morris Antonello, Marco Carraro, Marco Pierobon, and Emanuele Menegatti. Fast and robust de- tection of fallen people from a mobile robot. In IROS, pp. 4159-4166, 2017.
Ronald Azuma. A survey of augmented reality. Presence: teleoperators & virtual environments, 6(4):355-385, 1997.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancario Baldan, and Oscar Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR. pp. 11621-11631, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, pp. 213-229, 2020.
Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeurlPS, volume 30, 2017.
Xiaozii Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoc- ular 3D object detection for autonomous driving. In CVPR, pp. 2147-2156, 2016.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao. AutoAlign: Pixel-instance feature aggregation for multi-modal 3D object detection. In IJCAI, 2022a.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Graph- DETR3D: Rethinking overlapping regions for multi-view 3D object detection. In ACM MM, pp. 5999-6008, 2022b.
Zhiyu Chong, Xinzhu Ma, Hong Zhang, Yuxin Yue, Haojie Li, Zhihui Wang. and Wanli Ouyang. MonoDistill: Learning spatial features for monocular 3D object detection. ICLR, 2021.
MMDetection3D Contributors. MMDetection3D: OpenMMLab next-generation platform for gen- eral 3D object detection. https: //tp://///tttt/// cot///ccooo///ooooooo//oo/ open wwwpommmnaamppameme wttpemmmmmmmmmeeeeeeiieeo htt////neeeeeiinnneeeee 2020.
Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou. General instance distillation for object detection. In CVPR, pp. 7842-7851, 2021.
Mingyu Ding, Yuqi Huo, Hongwei Yi, Zhe Wang, Jianping Shi, Zhiwu Lu, and Ping Luo. Learning depth-guided convolutions for monocular 3D object detection. In CVPR, pp. 1000-1009, 2020.
Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Dis- tilling object detectors via decoupled features. CVPR. pp. 2154-2164, 2021a.
Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. LIGA-Stereo: Learning lidar
geometry aware representations for stereo-based 3D detector. In ICCV, pp. 3153-3163, 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9738-9738, 2020.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. In NeurIPS Workshop, volume 2015.
10

---

Published as a conference paper at ICLR 2023
Junjie Huang and Guan Huang. BEVDet4D: Exploit temporal cues in multi-camert 3D object detection. arXiv preprint arXiv:2203. 17054, 2022.
Junjie Huang. Guan Huang, Zheng Zhu, and Dalong Du. BEVDet: High-performance nulti-camera 3D object detection in bird-eye-view. arXiv preprint urKiv:21 2021.
Wei-Chih Hung, Henrik Kretzschmar, Vincent Casser, Jyh-Jing Hwang, and Dragomir Anguelov. LET-3D-AP: Longitudinal error tolerant 3D average precision for camera-only 3D detection. arXiv preprint arXiv:2206.07705. 2022.
Yanqin Jiang, Li Zhang. Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3D object detection with polar transformers. arXiv preprint 1xXiv:2206.15399 2022.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- Pillars: Fast encoders for object detection from point clouds. In CVPR, pp. 12697-12705, 2019.
Youngwan Lee and Jongyoul Park. CenterMask: Real-time anchor-free instance segmentation. In CVPR. pp. 13905, 2020.
Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3D object detection. In NeurIPS, 2022a.
Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. BEVDepth: Acquisition of reliable depth for multi-view 3D object detection. arXiv preprint arXiv:2206. 10092 2022b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. BEVFormer: Learning -eyee--eee representation from multi-camera images via spa- tiotemporal transformers. In ECCV, pp. 1-18, 2022c.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETR: Position embedding transforma- tion for multi-view 3D object detection. In ECCV, pp. 531-548, 2022a.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. PETRv2: unified framework for 3D perception from multi-camera images. arXiv preprint 2rXiv:2206.01256, 2022b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. BEVFusion: Multi-task multi-sensor fusion with unified bird's- seeye view representation. arXiv preprint arXiv:2205. 13442, 2022c.
Zhuang Liu, Hanzi Mao, Chao-Yuse Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A ConvNet for the 2020s. In CVPR, pp. 11976-11986, 11986, 2022d.
Jiachen Lu, Zheyuan Zhou, Xiatian Zhu, Hang Xu, and Li Zhang. Learning ego 3D representation as ray tracing. In ECCV, pp. 129-144, 2022.
Liang Peng, Fei Liu, Zhengxu Yu, Senbo Yan, Dan Deng, Zheng Yang, Haifeng Liu, and Deng Cai. LiDAR point cloud guided monocular 3D object detection. In ECCV, pp. 123-139, 2022.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D. In ECCV. pp. 194-210, 2020.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. FitNets: Hints for thin deep nets. In ICLR, 2014.
Martijn Schuemie, Peter Van Der Straaten, Merel Krijn, and Charles APG Van Der Mast. Research on presence in virtual reality: A survey. CyberPsychology & Behavior, 4(2):183 (22):183-21, 2001.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: Point-voxel feature set abstraction for 3D object detection. In CVPR, pp. 10529- 10538, 2020.
11

---

Published as a conference paper at ICLR 2023
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In CVPR. pp. 2446-2454, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2019a.
Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. FCOS: Fully convolutional one-stage object detection. In ICCV, pp. 9627-9636, 2019b.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. FCOS3D: Fully convolutional one-stage monocular 3D object detection. In ICCV Workshop, pp. 913-922, 2021a.
Tai Wang, Qing Lian, Chenming Zhu, Xinge Zhu, and Wenwei Zhang. MV-FCOS3D++: Multi-
view camera-only 4D object detection with pretrained monocular backbones. arXiv preprint arXiv:2207.12716. 2022a.
Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detect- ing objects in perspective. CoRL, pp. 1475- 1485. PMLR, 2022b.
Tao Wang. Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine-grained feature imitation. In CVPR, pp. 4933-4942, 2019a.
Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Wein- berger. Pseudo-lidar from visual depth estimation: Bridging the gap 3D object detection for autonomous driving. In CVPR, pp. 8445-8453, 2019b.
Yingjie Wang, Qiuyu Mao, Hanqi Zhu, Yu Zhang, Jianmin Ji, and Yanyong Zhang. Multi-modal 3D object detection in autonomous driving: a survey. arXiv preprint arXiv:2106. 120.533,,,1 2021b.
Yue Wang and Justin M Solomon. Object dgenn: 3D object detection using dynamic graphs. In NeurlPS, volume 34, pp. 20745-20758, 2021.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. DETR3D: 3D object detection from multi-view images via 3D-to-2D queries. In CoRL, pp. 180-191, 2022c.
Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. M2BEV: Multi-camera joint 3D detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204.0500 2022.
Yan Yan, Yuxing Mao, and Bo Li. SECOND: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Chenhongyi Yang, Mateusz Ochal, Amos Storkey, and Elliot J Crowley. Prediction-guider guided distilla- tion for dense object detection. In ECCV, pp. 123-133 138, 2022a.
Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan. Focal and global knowledge distillation for detectors. In CVPR, pp. 4643 4652, 2022b.
Zhendong Yang, Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked gener- ative distillation. In ECCV, pp. 53-69, 2022c.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, pp. 11784-11793 2021.
Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distilla- tion: Towards accurate and efficient detectors. ICLR, 2020.
Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3D object detection. In CVPR, pp. 3298, 2021.
Yumpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang. Guan Huang, Jie Zhou, and Jiwen Lu. BEVerse: Unified perception and prediction birds-eye-view for ceston-centric autonomous driving. arXiv preprint 2rXiv:2205.09743, 2022.
12

---

Published as a conference paper at ICLR 2023
Zhaohui Zheng, Rongguang Ye, Ping Wang, Dongwei Ren, Wangmeng Zuo, Qibin Hou, and Ming- Ming Cheng, Localization distillation for dense object detection. In CVPR, pp. 9407-9416, 2022.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. 19X0::100,,07850, 2019.
Objects as points.
arXiv preprint
13

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION OF CRITIC 9
Firstly, we adopt bilinear Fbilinear to obtain the paired 2D/3D features f.D f on student and teacher BEV features maps F2D F&D according to the reference location ri predicted by each student query q::
PDD Fbilinear (F P.D Fbilinear (F3D rz).
(14)
Note that we adopt query points from stages which actually form 900 x feature pairs. Each 2D and 3D feature located the same position viewed positive pair. We use separate projection heads for 2D and 3D features since they originated from different modalities. Each projection head 3-layer MLP (y20, R) and the final loss be written
222 g2 = (155)
explx? x22/T))
(15)
= -log Lz=oj exp (19D/T)
where 7 is the temperature. Different from the original implementation in (He et al., 2020), we do not collate samples from other cards during distributed training since there enough negative samples (5400-1=5399) for each card.
B
MORE EXPERIMENTAL RESULTS
B.1 EXPERIMENTS ON BEVFORMER WITH LARGER BACKBONES
Instead of the results conducted on ResNet-50 backhone, we also provide detailed experiments with stronger backbones, ResNet-101. We keep all experimental settings the same Section 4.5 except for the image backbone and the results shown in Table 8. From the table, we can conclude that BEVDistill can still consistently improve the competitive baseline BEVFormer, even with much stronger ResNet-101 backbone by more than 2.3 and 1.8 NDS.
Table 8: Experimental results of BEVDistill with BEVFormer ResNet-101 DCN backbone, under both single-frame and multi-frame settings.
Detector
Setting
NDSt
mAPt
Teacher
67.4
61.5
BEVFormer
Student
44.5
37.4
+BEVDistill
46.8(42.3)
38.9(+1.5)
Teacher
67.4
61.5
BEVFormer-T
Student
50.6
40.5
+BEVDistill
52.4(+1.8)
41.7(+1.2)
B.2 DISTILLING TO LIGHTWEIGHT BACKBONE
Knowledge distillation is actually one of the most common solutions to model compression. It is usually adopted to transfer useful information from large stronger model to a lightweight weaker model, which is suitable for deployment on the edge. Considering that vision-based detectors are fed with batch of images for surrounding view inputs, is extremely important to reduce the size of the image backbone. With this in mind, we conduct experiments with smaller backbones, including ResNet- 18 and MobileNetV2, which a more demanded setting in real applications. The final results are shown in Table 9. BEVDistill improves the ResNet- 18 and MobileNetV2 backbones by more than 2 NDS, pointing its great potential for resource- limited applications including autonomous driving and edge computing.
14

---

Published as a conference paper at ICLR 2023
Table 9: Experimental results of BEVDistill smaller image backbones: R8 and Mo-
bileNetV2 on nuScenes validation subset.
Detector
Backbone
Setting
NDS1
MAPt
Teacher
67.4
61.5
BEVFormer
ResNet-18
Student
37.3
30.0
+ BEVDistill
40.2(+2.9)
32.0(+2.0)
Teacher
67.4
61.5
BEVFormer-T
ResNet-18
Student
43.7
32.1
+ BEVDistill
45.8(+2.1)
(3.9(+1.8)
Teacher
67.4
61.5
BEVFormer
MobileNet-V2
Student
37.0
29.3
+ BEVDistill
38.8(+1.8)
30.8(+1.5)
Teacher
67.4
61.5
BEVFormer-T
MobileNet-V2
Student
43.1
32.0
+ BEVDistill
44.8(+1.7)
33.5(+1.5)
Table 10: Experimental results of BEVDistill with BEVDepth ResNet-50 backbone.
Detector
Setting
NDS
MAPt
Teacher
66.4
60.3
BEVDepth
Student
44.0
31.7
+BEVDistill
45.2(+1.2)
33.0(+1.3)
B.3 EXPERIMENTS ON BEVDEPTH
To fully exploit the capability of this simple framework, extend it adapt the current state- of-the-ar art multi-view 3D detector, BEVDepth. Since BEVDepth and BEVFormer both share BEV space as the intermediate feature representation, we can directly apply dense feature distillation. As for the instance distillation, BEVDepth conducts dense prediction similar to CenterPoint. Therefore, we directly choose CenterPoint as our teacher model and only conduct instance distillation on the foreground regions (ie.,, the center of ground truth objects). We conduct experiments on ResNet-50 backbone on nuScenes 1/2 training subset and report the results on validation subset in Table Surprisingly, BEVDistill can still improve its vanilla baseline by 1.2 NDS, even though BEVDepth has already leveraged the information from LiDAR data with explicit depth supervision.
Table 11: Experimental results BEVDistill with MV-FCOS3D+ and MV-FCOS4D++ on Waymo Open Dataset. Following the practice in (Wang et al., 2022a), we sample every 5 frame to form the training set.
Detector
Backbone
BEVDistill
mAPL1
MAP
MAPH!
ResNet-50
32.6
45.1
42.7
MV-FCOS3D++
ResNet-50
34.3
46.8
44.4
ResNet-50
34.0
46.5
43.8
MV-FCOS4D++
ResNet-50
35.1
47.5
44.9
B.4 EXPERIMENTS ON WAYMO OPEN DATASET
In this section, report BEVDistill another competitive public dataset, Waymo Open Dataset (Sun et al., 2020) to further demonstrate the generalization of our approach. Due to the characteris- ties of the camera-bace based detections, instead of adopting original mAP metrics, choose the official
15

---

Published as a conference paper at ICLR 2023
released LET-mAP (Hung al., 2022) to evaluate the model performance. Since most multi-view 3D detectors do not report their performances on Waymo dataset, we select another competitive multi-view 3D detector, MV-FCOS3D++ (Wang et al., 2022a), which achieves 2nd place at the Track, as our baseline. The follow the same exper-
Waymo Open Dataset imental settings in the official MV-FCOS3D++ repository. We select our teacher model as SEC- used by MV-FCOS3D++. The final results are reported
OND to adapt with the Anchor in Table 11. It can be concluded that BEVDistill can consistently improves MV-FCOS3D++ and MV-FCOS4D++ (multi-frames input version of MV-FCOS3D++) by 1.7/1.1 mAPL, respectively, indicating the robustness and general of the proposed method.
C DISCUSSIONS
C.I WHY KL DIVERGENCE DISTILLATION ON SPARSE QUERY LOGITS FAILS TO IMPROVE?
KL logit distillation has been proven to be effective in plenty of works (Tian et al., 2019a; Hinton et al., 2015). However, it fails to yield improvements under the cross- moosl distillation. By visu- alizing the training data and the predicted logits, attribute one reason to this problem: for most previous distillations, both teacher and student are fed with the data in the same modality. There- fore, both of them should behave in similar manner in every region. Such hypothesis may not hold under different modalities: due to the characteristics of the modal sensor, the information col- lected by each sensor can be diverse. For instance, even if one person is occluded by another one in the image view, he may still be perceived by the LiDAR sensors. Besides, some instances can be easily detected by one modal sensor, while it may be hard on the other one, leading to different logits distribution predicted by the model. Directly forcing the student to mimic such distribution can deteriorate the inner knowledge learned by the student itself. Different from logit imitation, the contrastive paradigm only enforces the contexts encoded in each modality are the same, providing the freedom for the model to learn the representation encoded in the respective modality.
C.2 WHAT DOES THE MODEL LEARN FROM ITS PROSS-MODAL TEACHER?
Different from the previous knowledge distillation framework, BEVDistill leverages LiDAR models to transfer knowledge. A natural question is: what does a multi-view vision-based student learn from its LiDAR-based 3D teacher? Actually, it hard to figure student, however, we can get some hints to some extent by comparing the detection performance evaluated from various aspects. The detailed results are shown in Table 12.
the explicit knowledge learned by the
Table 12:
Detailed metrics comparison between BEVFormer and BEVDistill on nuScenes validation dataset with ResNet-50 image backbone.
Method
NDS
mAP
mATE
mASE
mAOE
mAVE
mAAE
BEVFormer
42.3
35.2
0.755
0.271
0.428
0.870
0.208
+ BEVDistill
45.7
38.6
0.693
0.264
0.399
0.802
0.199
BTVerrerr-T
48.8
38.3
0.707
0.281
0.435
0.414
0.194
+ BEVDistill
51.5
40.7
0.663
0.268
0.393
0.374
0.184
From the table localization, orientation and velocity estimation. LiDAR point clouds provide accurate localization information, which is of great benefit to localizing the object, including the absolute 3D space and its orientation. Both of them are more challenging to acquire from the camera view only. Note that the velocity estimation gets 10% enhancement since the LiDAR model introduces more spatial information (i.e., the LiDAR model takes 10 consecutive sweeps of point clouds as inputs instead of a single frame). As for the attribute prediction, which mostly bound up with the classification tasks, BEVDistill fails to gain satisfying performance. We attribute the reason to the deficiency of point clouds in semantic representation compared to the imagery data, which may also account for the failure in KL distillation on the classification branch discussed in Section 3.2.2.
find that the most significant improvements mainly related to object
16

---

Published as a conference paper at ICLR 2023
D QUALITATIVE RESULTS
We provide some qualitative results in Figure 2. More visualizations can refer to the supplementary materials, which presented in video format.
9010
0
0
Figure 2: Visualization of the detection results predicted by BEVDistill. Best view in color: the prediction and ground truth are in blue and orange, respectively.
17


Reviews:
[{"review_id": "bwEGlU4Mm", "reviewer": "Reviewer_2Rwe", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a cross-modal knowledge distillation framework for mutli-view 3D object detection, with a 3D point cloud detector as teacher and an image detector as student.  The proposed method unifies different modalities in the BEV space and conducts knowledge distillation through the dense feature distillation and the sparse instance distillation. Extensive experiments demonstrate that the proposed method outperforms current KD approaches, and the method improves baseline model with a significant boost and achieves new state-of-the-art performance on nuScenes test leaderboard.", "summary_of_the_review": "The paper presents a cross-modal knowledge distillation framework for mutli-view 3D object detection which unifies the image and LiDAR features in the BEV space. The paper has good writing and extensive experiments. However, the paper is lack of more visualizations to verify the proposed method. \n\nMy biggest concern is regarding the motivation of using the teacher-student distillation paradigm, since it is also possible to supervise the image detector with LiDAR point cloud (more specifically, depth). If authors can provide comparisons with BEVDepth and show the necessarity of the adopted distillation setting, I may raise my rating.", "strength_and_weaknesses": "Strength\n+ The overall design is relatively novel,  it unifies the image and LiDAR features in the BEV space and adaptively transfer knowledge across non-homogenous representations in a teacher-student paradigm.\n+ The method is well ablated and  achieves state-of-the-art performance on nuScenes.\n+ The writing is good and easy to follow.\n\nWeakness\n- It would be better if some visualizations are provided to verify the effectiveness of proposed method.\n- More experiments on comparison with BEVDepth will be interesting, which directly uses point clouds as depth ground truth and supervises the image detector. This comparison is necessary to clarify the motivation of the teacher-student paradigm, since it is also possible to supervise the image detector with LiDAR point cloud (more specifically, depth). ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GeMCgdaQJ2", "reviewer": "Reviewer_BsUP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method named BEVDistill to distill knowledge from LiDAR-based 3D object detectors (teacher) to enhance multi-view 3D detectors (student). \nFirst, since both detectors project their inputs to the BEV space, the proposed method performs feature distillation between the BEV feature maps under the guidance of a ground-truth gaussian mask. \nSecond, the teacher and student both adopt DETR-style prediction heads, the prediction-level distillation requires a matching between the two sets of instance predictions generated by the teacher and student models. Instance-level distillation is then performed between each pair of matched instances through a bounding box regression loss and a mutual information loss between the teacher and student’s hidden representations before logits. To suppress low-quality instance predictions, the authors also design an IoU-weighted quality score as the weight for each instance pair in the distillation loss.\n\nThe experiments on the nuScenes dataset proves that the proposed method works well for the chosen student model (BEVFormer) with various backbones. The ablation studies demonstrated the effectiveness of feature and instance distillation, as well as several specific designs within each component.\n", "summary_of_the_review": "Although this paper proposed an effective method for distilling knowledge from LiDAR-based detectors to multi-view image-based detectors, which is a novel attempt towards enhancing image-based detectors, the technical novelty of each proposed component is limited, since IoU reweighting and mutual information-based KD are previously studied. The experiments demonstrate reasonable improvements but some necessary implementation details are missing. Overall, this paper is marginally below the acceptance threshold.", "strength_and_weaknesses": "Pros) \nDistilling knowledge from LiDAR-based detectors to multi-view image-based detectors is a novel attempt and can be a practical technique to enhance image-based detectors.\nThe effectiveness of the method and its components is shown by the experiments.\n\nCons) \nThe designed IoU-based quality-score itself is not technically novel (explored in LiDAR-based methods [1,2]), and its effectiveness is not analyzed in the experiments.\nAlthough effective, mutual information-based knowledge distillation is also studied in previous works [3]. This paper directly used an existing formulation.\nSome necessary implementation details are missing, e.g., the overall loss and the hyper-parameters (\\alpha and \\beta in Eq.(13), \\sigma_i in Eq(2), and \\gamma in Eq(6)) are not given. These could affect the reproducibility of the proposed method.\n\n[1] Zheng W, Tang W, Chen S, Jiang L, Fu CW. Cia-ssd: Confident iou-aware single-stage object detector from point cloud. AAAI 2021.\n[2] Hu Y, Ding Z, Ge R, Shao W, Huang L, Li K, Liu Q. Afdetv2: Rethinking the necessity of the second stage for object detection from point clouds. AAAI 2022.\n[3] Tian Y, Krishnan D, Isola P. Contrastive representation distillation. ICLR 2020.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "9oxeedzsCp", "reviewer": "Reviewer_diMS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a cross-modality knowledge distillation framework for multi-view 3D object detection, called BEVDistill. It introduces two types of feature distillation, i.e., dense feature distillation and sparse instance distillation. Experiments prove the effectiveness of the proposed modules on the nuScenes dataset.", "summary_of_the_review": "My main concern is the contribution of this work. It could be limited compared with previous cross-modality distillation methods. Please refer to the Weakness section for more details.", "strength_and_weaknesses": "Strength:\n---------------\n1. This work focus on cross-modality distillation. It's reasonable to transfer geometry-aware knowledge to images during training.\n2. The experiments are sufficient and proven to be effective on the nuScenes dataset.\n3. The overall writing is clear and easy to follow.\n\nWeakness:\n---------------\n1. Compared with previous work that uses feature distillation like LIGA-stereo and UVTR, the main contribution of this work is the sparse instance distillation. However, target-level distillation is not a novel thing in the distillation domain. And it only contributes 0.6% mAP gain and 1.0% NDS gain on top of the dense feature distill baseline in Table 3. That means most of the improvements are still from the dense feature distillation. This could make the contribution of this work limited. \n2. The author claims \"we postpone such supervision after the transformer encoder, which offers more opportunities for the network to align information across different modalities.\" in Section 3.2.1. But according to Figure 1, the dense feature distillation is still operated in the BEV representation. The reviewer is confusing the difference between it and UVTR, and why it offers more opportunities for the network to align information. Maybe the author can provide more details on the dense feature distillation.\n3. Why the BEV feature in Figure 1 is 3D BEV? Is the height axis preserved here? It's better to add some illustrations in Section 3.2.1.\n4. Writing: Section 4.2 can be moved to supplementary material. And it's better to report Table 10 in the main paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9_8aXJ0-G4", "reviewer": "Reviewer_zDXj", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper explores LiDAR-to-Monocular distillation methods for BEV object detection. Their method, BEVDistill, builds upon the work of BEVFusion and uses the intermediate BEV representation to ease the multi-modal alignment and feature imitation. This allows the use of an asymmetrical LiDAR detector, Object-DGCNN, which this paper chose to use. The method also uses a “dense feature distillation” and “sparse instance distillation”, which supervises both the feature level (with a soft-focus on the foreground region) and the instance level (focusing on high quality teacher predictions). This method is evaluated extensively on the nuScenes dataset, with a strong leaderboard performance (5th place on NDS score at the time of review). The authors also conduct extensive ablation studies to validate the components of the model.", "summary_of_the_review": "In summary, this work builds upon the BEVFusion work and utilizes the intermediate BEV representation to ease the distillation process between detectors using LiDAR input to detectors using monocular camera input. By using their  “dense feature distillation” and “sparse instance distillation”, they are able to achieve very strong performance on the nuScenes dataset. The strong analysis section and ablations support their claim. However, their results are slightly limited by the fact that their results are only on a single dataset. Overall, the work is clear; I would be willing to increase the score if additional results on another dataset support their claims.", "strength_and_weaknesses": "**Strengths**\nThe paper is very well written, and analytically thorough. The strongest point of this paper is the clear and complete experimental section, that exhaustively ablates the key components of the model. Reading over the tables, I observe that it is competitive as compared to a strong set of baselines (Table 1), is robust across backbones (Table 2), and is a well motivated distillation method with each component contributing to its performance (Table 3). Baseline wise, the authors compare against a strong set of methods, that – to the best of my knowledge – is comprehensive. Further ablations on the method’s distillation components and losses are interesting and informative, and convincingly beat a set of baseline comparisons.\n\nOne interesting point is that this method cleverly deals with different representations between the Student and Teacher models, which doesn’t limit the architecture of both (up until the distillation representation). This distillation process (foreground focusing) is intuitive and is validated experimentally (Table 4).\n\n\n**Weakness**\nOne weakness is that the method appears to only be evaluated on a single dataset, nuScenes. It is strange to me, however, since the model BEVfusion from which they are building off of is evaluated both on nuScenes dataset and Waymo dataset. **This paper would be much stronger if there is a second dataset that they compare their method on**. This would ensure that their results – while excellent – are also general.\n\n**Misc**\nI am also curious if this distillation process (across BEV features) works on different BEV type detectors? It is a minor point, but does this distillation improve other Pseudo-LiDAR (Wang 2019) type methods, Lift-Splat (Philion, 2020) type methods, or Inverse Perspective Mapping (Reiher, 2020)? In theory, the method should improve upon all of the detectors that utilize intermediate BEV representations, and I would be curious to see if this is indeed the case.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster