VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception
Authors: JI Yuzhe
Yijie CHEN
Liuqing Yang
Rui Ding
Meng Yang
Xinhu Zheng
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Perception
Multi-modal Fusion
Cross-modal Knowledge Distillation
Primary area: machine_vision
TL;DR: A simple yet versatile framework that jointly considers multi-modal fusion and cross-modal knowledge distillation.

Abstract: Recent advancements in 3D perception have led to a proliferation of network architectures, particularly those involving multi-modal fusion algorithms. While these fusion algorithms improve accuracy, their complexity often impedes real-time performance. This paper introduces VeXKD, an effective and Versatile framework that integrates Cross-Modal Fusion with Knowledge Distillation. VeXKD applies knowledge distillation exclusively to the Bird's Eye View (BEV) feature maps, enabling the transfer of cross-modal insights to single-modal students without additional inference time overhead. It avoids volatile components that can vary across various 3D perception tasks and student modalities, thus improving versatility. The framework adopts a modality-general cross-modal fusion module to bridge the modality gap between the multi-modal teachers and single-modal students. Furthermore, leveraging byproducts generated during fusion, our BEV query guided mask generation network identifies crucial spatial locations across different BEV feature maps in a data-driven manner, significantly enhancing the effectiveness of knowledge distillation. Extensive experiments on the nuScenes dataset demonstrate notable improvements, with up to 6.9\%/4.2\% increase in mAP and NDS for 3D detection tasks and up to 4.3\% rise in mIoU for BEV map segmentation tasks, narrowing the performance gap with multi-modal models.

VeXKD: The Versatile Integration of Cross-Modal Fusion and Knowledge Distillation for 3D Perception
Yuzhe JI Yijie Chen', Liuqing Yang!. 2,2, Rui Ding², Meng Yang", Xinhu Zhengü 1,2
Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China 2 Hong Kong University of Science and Technology. Hong Kong SAR, China
3 Xi'an Jiaotong University, Xi'an, I'an, China
{yji755, ychen324]@connect srconnect.kkust--ee hhusttgzz edu. en iqyangShkust-gee edu. cn dingrui@stu aejtssttu.utu....... xjtu. edu, cn mengyang@xjtu tu. edu, en
xinhuzhengehkust-eeee edu, en
Abstract
Recent advancements in 3D perception have led proliferation of network archi- tectures, particularly those involving multi-mode fusion algorithms. While these fusion algorithms improve accuracy, their complexity often impedes real-time per- formance. This paper introduces VeXKD, an effective and Versatile framework that integrates Cross-Modal Fusion with Knowledge Distillation. VeXKD applies knowledge distillation exclusively to the Bird's Eye View (BEV) feature maps, enabling the transfer of cross-modal insights single-modal students without additional inference time overhead. It avoids volatile components that can vary across various 3D perception tasks and student modalities, thus improving ver- satility. The framework adopts modality-general cross-modal fusion module to bridge the modality gap between the multi-modal teachers and single-modal students. Furthermore, leveraging byproducts generated during fusion, our BEV query guided mask generation network identifies crucial spatial locations across different BEV feature maps from different tasks and semantic levels in a data- driven manner, significantly enhancing the effectiveness tion. Extensive experiments provements, with up to 6.9%/4.2% increase in mAP and NDS for 3D detection tasks and up 4.3% rise in mloU for BEV map segmentation tasks, narrowing the performance gap with multi-modal models.
knowledge distilla- the nuScenes dataset demonstrate notable im-
Introduction
3D perception, encompassing 3D object detection [50 II] and BEV map segmentation
3 22
72
38 Achiev- ing high accuracy and real-time performance simultaneously presents the desirable yet challenging pursuit in this field [I2]. Many studies [2], 20. 28] focus on multi-modal fusion, particularly the fusion of LiDAR and multi-view cameras, to enhance perception accuracy. These methods necessi- tate handling additional input data, inevitably employ more complex networks, and extend inference time. Conversely, some studies [6%, 52, 66, 63, 60] continue to focus single-modal algorithms to maintain system simplicity and enhance accuracy with better training strategies and data pipelines.
crucial for understanding 3D scenes and controlling autonomous vehicles [48.
Cross modal Knowledge Distillation (KD) [14, 3d 59 has emerged as promising strategy for transferring insights across modalities. using mulli-moda models as teachers and single-modal mod- els as students. Cross-modal KD can thus improve the accuracy of student models without incurring
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

additional inference time. However, cross- cross-modal KD faces significant challenges, including capacity discrepancies between the student and teacher models, and information gaps stemming from the dif- ferent input modalities. Xue ef al. [5D]] propose and tentatively validate that teacher models focusing on modality-speecffe information can exacerbate the modality gap between student, reducing the ef- fectiveness of cross-modal KD. In contrast, teacher models that base decisions on modality-gente -enneral information can minimize disparities with single-modal students, facilitating a more effective cross-
modal KD process.
However, the current research paradigm, which separates cross-modal fusion from KD, lim- its potential synergies. On one hand, multi- modal fusion research [24], 622 28] has become performance-orientte Despite achieving im- pressive results on benchmarks like nuScenes [a]. BEVFusion [28] stands out as the state- of-the-art fusion method. However, the visu- alization of fusion and LiDAR feature maps in BEVFusion [28], and the significant per- formance drops under conditions like LiDAR failure depicted in Fig. M reveal over- reliance on LDDR-sppecffi information rather than the general information of multi- features. This makes these high performance fusion methods less suitable teachers in cross-modal KD. On the other hand, research cross- modal KD in 3D perception [68, Z|] seldom considers the selection of teachers, often directly using a LiDAR or fusion model with higher accuracy, such as BEVFusion [28], as the teacher [68], which potentially limits the effectiveness of cross-modal KD.
Defore Fuslon
After Fusion
Deatatic DESTAN Poree
Caniaa nalling
LEWE WaAn
Figure 1: (Left) Visualization of BEVFusion feature maps before and after fusion, indicating minimal infor- mation gain. (Right) Significant performance degrada- tion in the LiDAR missing scenario, highlighting the over-reliance on LiDAR features.
The versatility of KD algorithms is another facet often overlooked in previous research, yet it sig- nificantly impacts the vitality and breadth of KD applicability. especially in a rapidly evolving field like 3D perception. If KD algorithm cannot be directly applied to new student algorithms, its re- implementation may require numerous difficult-to tiffiuultto-generalize empirical engineering decisions and result in an unpredictable performance drop. The versatility of KD algorithm primarily hinges on two aspects. Firstly, it depends on whether the KD algorithm is tied to the processing steps or feature space of a specific modality. For instance, MonoDistill spective view and uses the same model architecture as the camera, serving as depth supervision for camera branches but is not applicable to LiDAR students. In contrast, using the BEV space, which is representation-friee friinddyy to different modalities, alleviates this limitation. Secondly, and most impor- tantly, the versatility of KD algorithm depends on its compatibility with specific network architec- tures, especially task-specific heads, which exhibit the greatest design variations across different 3D perception algorithms and downstream tasks. These can range from dense heads [[55]
[Н] projects LiDAR points onto per-
B
E7. 34 [Q]
and transformer- based heads [2, A5, 58, 27]] to segmentation heads [722, Ad. RР, 67 This diversity limits the versatility of KD methods that rely on response distillation across different perception tasks and various detection algorithms. Consequently, previous 3D perception KD methods often require identical heads and similar model architectures for both student and teacher models, and are generally limited to single downstream task, primarily 3D object detection.
The current reliance on response distillation primarily stems from the underutilization of the rich information available in the teacher's feature maps. Previous efforts have focused on the ground truth locations, using Gaussian masks [68, / or sampling a few points within these areas [68]. These methods tend to overlook the valuable background information, which has proven useful in 2D perception [I3 61]]. Moreover, by confining feature distillation solely ground truth locations, these methods not only fail to account for variations in spatial perceptual fields across different levels of BEV feature maps but also struggle to generalize to tasks requiring dense supervision, such as map segmentation. This reduce the versatility of their KD methods.
To address the aforementioned challenges, we propose an effective and versatile framework that integrates Cross-Modality Fusion and Knowledge Distillation within the BEV feature space. This framework effectively narrows the modality gap between the teacher and different single-modal stu- dent models by training modality- fusion teacher. Moreover, we leverage the learned BEV query, the byproduct of the fusion process, guide mask generation network in creating unique

---

spatial masks for different feature levels and tasks in data-driven manner. These learned masks significantly enhance the effectiveness of feature distillation by enabling the selective filtration and transfer of valuable information from the teacher's s feature maps. Furthermore, our KD framework is independent of specific processing steps or network architectures, making it versatile to be applied across any downstream 3D perception task and adaptable to various student modalities, as well as future advancements in this field.
In summary, our contributions are summarized as follows:
We present early effort to integrate cross-modal fusion and KD in 3D perception, enhancing the efficacy of moosl KD with the modallty-geneeet fusion teacher and the fusion byproduct.
We pioneer data-driven spatial masks learning for feature distillation on BEV feature maps, se- lectively transferring only beneficial feature information by applying distinct masks tailored to different BEV feature maps.
Our KD approach is designed to be task- and BEV-based 3D perception task and adaptable to various student modalities.
making it highly versatile for any
2 Related Work
2.1
Fusion
LiDAR and camera are the two most common sensors used 3D perception. Recent advancements in both camera-based [32] [6. 23, 25, 5 and LiDAR-based [69, bll I9, 551 methods have achieved notable results, providing a solid foundation for LiDAR-cames fusion. Since the BEV space offers feature space that is friendly to both modalities and can easily be applied to various downstream tasks, fusion research based on BEV space has also become a trend [26 24, F2, 28]. In BEV-based perception methods, both single-mon modal and fusion models [32 P from LiDAR or cameras into the BEV space using mppelity- encoders. Then only the fusion models being integrated through fusion module, but finally BEV encoder and task-specific heads are applied to all models for perception tasks. Although the abundance of research related to BEV poses challenges in selecting appropriate student and teacher architectures for KD, the common paradigm in BEV-based work offers the potential for a unified KD framework.
[8] initially transform inputs IT
2.2 Knowledge Distillation
Knowledge distillation (KD) facilitates the efficient transfer of implicit knowledge from the teacher to the student without increasing the student's inference time [G] [14]. Besides aligning the teacher's soft outputs with the student's, some works indicate that mimicking the feature map can also boost performance [ 2T, 93, 56]. Direct distillation over the entire feature maps can potentially degrade performance due to the noisy teacher feature maps. Based on this observation, some approaches have resorted to attentive distillation, concentrating KD on the less noisy foreground features and using ground truth as a mask guide the process [G55, 571]. On the other hand, recent studies [[3. 61] emphasize the value of background information and suggest decoupled KD processes for both foreground and background features. DistillBEV [PZ]] advances this concept by decomposing the regions of feature maps and enhance attention to false positive regions. Recently, inspired by pretext tasks in large language models, generative distillation has been proposed [[44]. However, random masks utilized in generative distillation can destabilize the performance, especially in 3D object detection with pronounced foreground- foreground-backkroundd imbalance. Our approach still aligns more closely with attentive distillation to selectively transfer knowledge from teacher's 's feature map.
Another key research focus is the efficacy of KD, as explored by Cho et al. [8]. Studies indicate that a high-accuracy teacher model does not necessarily improve KD results. Similarly, in cross-modal KD scenarios, Xue et al. [60] investigate factors influencing KD's effectiveness. They reveal that if a teacher makes decisions based on modality
features, KD performance can be improved
even when the teacher's accuracy not superior. Recently, Huang el al. [LZ]] develop a "vision- centric" multi-modal teacher, reducing reliance on LiDAR align more closely with camera-based students. Our work aims to develop a modality- fusion model without modifying the pipeline of the teacher network.
3

---

Due to spatial inaccuracies in RGB images, numerous studies have used KD to improve the ac- curacy of camera models. BEV-LGKD [20] uses LiDAR enhance the camera-based students' depth estimation but it is limited to ISSS-baed [[2] projection methods, excluding certain camera students like BEVFormer [[5]] BEVDistill [ introduces a sparse instance distillation method coupled with transformer-based detection heads, which is unsuitable for common dense detection heads [ Compared to camera student models, cross-modal KD applied to LiDAR students is relatively less explored. Zheng et al. [b4]]] employ the PointPainting [[9]] model as a teacher, pro- viding supervision only for voxel-based LiDAR students during the voxelization process. Unidistill [6X] supports various student-teacher modality combinations, but its response distillation is limited to dense detection heads, and feature distillation depends on 3D detection ground truths, restricting its use in tasks like BEV map segmentation. To date, universal KD paradigm covers diverse 3D perception tasks and student modalities.
3 Methodology
3.1 Overall architecture of Cross-Modal Fusion and Knowledge Distillation Framework
LibAR Teput
LIDAR
Hlgh Postro BEV feauure PANN
Faste
LibAR enowler
Dcverden
Imege BFV Seatuee Find
Lonoknd Marthed Dteiiladion
Higitbol Shated Doailaiio
Drsetee
Odges Detnciio Boadk
Inadder
Teput 1338
Sugasssaii
strsk
lecel eingle
Tecadee High-leus modil DEY banure
LIDAR Ingut fuse
Mult-vioe carrera Ingax (00
Kodalapocfic ennder
Fooks
plae.
Figure 2: Overall architecture of VeXKD: Building upon the common BEV fusion pipeline, tailor Modality-General Fusion Module and design a masked feature distillation method with learned masks assisted by the byproduct of the fusion module, applied across both low-level and high-level BEV features. Our feature distillation framework circumvents variations in different model architectures, making it modality- and task- agnostic.
Sigmoodd
Manked teacher
Teacher BEV Festure stene
Marked tracher Festure
parception on loss
Feed Buvazd
oecmtalaation
Toundee
afvenice
loss Fenokt
Deformaaia Crows Ameriin
Student
Teae ver BEV Festure Pietis
perception
arviuery
tack loss
Studer: SEV
Masked student Festure
Festure slew
(b) Deformable Attention
Query Q
Teachet Fecture Fanie
(c) Masked Fecture Distilaation
(o) BEY Query Guided Mask Generation
represents
operation.
Figure 3: Illustration of BEV query guided mask generation and masked feature distillation. (a) Overview of the transformer-based block for mask generation, which adopts the byproducts from the fusion module as the BEV query. (b) In the deformable maps identify crucial spatial locations. (c) the masked feature distillation stage, learned spatial masks are applied to both teacher and student feature maps before calculating the distillation loss.
attention operation, the BEV query interacts with the teacher feature
The overall architecture of the proposed framework is depicted in Fig. D. The modality-genent fusion module operates on the low-level BEV feature Flon capturing high spatial granularity and rich semantic information from different modalities. Enhanced by the byproducts of the fusion

---

module, masked feature distillation is applied to both low and high-level BEV features to selectively mimic the semantic information from the feature maps of the fusion teacher.
3.2 Modality-General Fusion Module (MGFM)
The architecture of the fusion module is shown in Fig. g Inspired by BEVFormer [25]] employ deformable attention [ZT] as the central mechanism of fusion module. As detailed in Eq. II q.P. represent the query, reference point, and input features, respectively. The num- ber of attention heads and the number of key points sampled per head are denoted by Nhead and Nkey. Wy and W; are learnable parameters. Aij and Apt E IR? represent the pre- dicted weight and the offset rel- ative the reference point p for these sampled key points, both are learned from q. Deformable attention enables learnable sam- pling offsets, which provide larger and more adaptable recep- tive field while maintaining com- putational efficiency. It is ideal for correcting varying degrees of spatial misalignments caused by ill-posed view projections and fickle sensor extrinsic matrices. Our fusion module adheres to the traditional transformer block structure, incorporating deformable cross-modal attention and query self-attention operations to it- eratively fuse multi-modal BEV features.
Fused BEV Feature Vinten
BEV Encoder & Task Specific Hleods
Normalization
Normaltation
Deformable Query Selff Attention
Updated Query
Query Q
Normalitation
(c) Deformable Query Set--ittttionn
Deformable Cross modal Amendinn
BEV Query
Camera Feature Fimp
Learnable BEV Query Q
LEBAR Feature Fire
Camera
EDDA Feature
Feature Fing
(a) Overview of Modality general Pasion Module Deformable modal Attention
Figure 4: The architecture of the Modality- General Fusion Module: (a) Overview of the transformer based block. (b) Deformable cross- modal attention operation: the BEV query symmetrically interacts with features sampled from both LiDAR and camera. (c) Deformable query
self-attention operation: the BEV query interacts with itself to integrate correlational relationships.
During the deformable cross-modal attention operation, a set of learnable dense BEV queries Q E R is initialized. These queries interact with low-level feature maps from each modal- ity Flane for modal {lidar, camera). The sampling offset generation process is modified by concatenating plow with the query Q to produce modality-specific sampling offsets Ap and atten- tion weights W. The resulting feature map is updated by summing the outputs, denoted in Eq.
This symmetrical and equitable fusion approach forces the preservation of the information from each modality. promotes the extraction of modality-generr information from the fusion features, rather than disproportionately relying tween the fusion teacher and individual modalities. In the deformable query self-attention operation, modality-specine feature inputs substituted with the query Q itself, indicated in Eq. GL This self-attention operation not only expands the receptive field but also capture the inter-correlations among BEV features, making the fusion features geometrically more accurate and semantically richer. After stacking transformer blocks similar to BEVFormer(25)), the MGFM module yields the fusion result p Flow Jusion
specific modality, thereby narrowing the information gap be-
Nhead
Nkev
DeformAttn(q.p.z)
W
Aij W. Apt)
(1)
Qmoddt IQ;
Flow
Q Σ
DeformAttn (Qmoddts P. Flow
(2)
modal
Q = DetormAttn (Q.p. Q)
(3)
5

---

3.3 BEV Query Guided Mask Generation
Due to the inherent differences among various downstream tasks and feature levels, the specific areas within the teacher feature map that require the student's focus can vary. For example, in detection tasks, information near ground truth positions is crucial for the high-level BEV features. However, since high-level features selectively aggregate broader range of low-level features, important low- level features may be more widely distributed, not just at ground truth positions. Based on these insights, we opt learn ad hoc masks in data-driven manner. Nevertheless, efficiently learning these masks rapidly necessitates set of initial parameters that are easy to generalize. Once trained within the fusion module, the learned BEV query Q can effectively extract valuable information from the BEV feature maps, facilitating quick adaptation to ad hoc spatial masks. To maximize the effectiveness of Q, we employ the deformable cross- attention operation in the mask generation network, similar to the setup in the fusion module as depicted in Fig. B().. Here, Q and the teacher BEV feature map Fifwaior where level E {low, high), serve the and z for DeformAttn, re- spectively, allowing for continuous updates to Qlevel, Finally, the channel dimension is reduced to I using a 1 X convolution, and the values interval for the learned spatial mask Mievel is adjusted to range (0,1) through sigmoid activation function.
Flevel is In designing the mask generation loss, inspired by [T]], the original teacher feature map fasion
substituted with the masked version F/ fusion obtained by applying Hadamard product operation @ between the teacher feature and the learned spatial mask, as shown in Eq. EL This masked feature serves as input to derive the final masked teacher loss for downstream tasks maskedd Minimizing this loss ensures the preservation of valuable features in the teacher feature map through the learned mask.
However, using only L as the mask learning loss can result in an all-ones mask. We also incorporate the feature distillation loss LmaskD D into the objective stabilize the mask generation process. This inclusion allows the student feature map to participate, helping avoid optimization pitfalls that arise from hard-to-mimit locations due to modality gaps. The final mask generation loss Timask gen is depicted in Eq. EL where is a factor to balance the scale of the two losses.
plevel Jusson
Mlevel @ plavet
fusion: level E (low, high}
geask_gen
Ctask mosked + J.
(5)
3.4 Masked Feature Distillation
After applying the learned mask to both the student and teacher feature maps as depicted in Eq. and Fig. (ic), various loss functions be applied to quantify the discrepancies between them. Finally, Attention Transfer [566] is adopted to compute the KD loss between the student and teacher. This method effectively mitigates the adverse impacts of channel-wise heterogeneity that arise from different architectures. By applying 1.2-norm
Attention Transfer as shown in Eq. M greater em- phasis is placed on spatial locations with higher activations more discriminative features, thereby enhancing the robustness and effectiveness of feature distillation.
We apply mask generation and attention transfer across both low- and high-level BEV features. The mask generation process, which also incorporates Lmask D. runs concurrently with the KD process. It can be halted after a certain number of epochs using controllable hook. The overall loss for the student model Latudant (see Eq. M combines the downstream task loss Estudent with KD loss L.nas D. using À to balance the magnitude of these losses.
Qlevel (F) vec (Fsum (Mievel + OF))
(6)
Qlevel
Qlevel( (Ijeaet fusion)
Cleved mask
student)
IQQeeell Pleveeteelle
[[Qlevel
fusion 12
DmosKD
Σ
Clevel
Estudent
estudent Lask
+ XLmaskD
(8)
levele (low,, high

---

4 Experiments
4.1 Experimental Setup
The versatility of the KD framework is verified by adopting both camera and LiDAR student modali- ties, and by evaluating on two 3D perception tasks: 3D object detection and BEV map segmentation. Notably, our framework can be easily extended to support additional student modalities such as radar and event-based cameras, and to support other downstream tasks, including 3D object tracking and motion prediction.
Dataset & Evaluation Metries The training and evaluation are conducted using the nuScenes [ large-scale dataset under the CCBCCA 4.0 license comprising 1,000 driving se- quences (700/150/15) for train/val/test). This dataset provides diverse annotations and sensor data, featuring six monocular camera images along with 32-beam LiDAR system, making it ideal for assessing our method across various tasks and modalities. We employ evaluation metrics that are widely adopted by state-of-the-an art methods. For 3D object detection, we adopt the mean Average Precision (mAP) along with the nuScenes detection score (NDS), the official evaluation metrics of the nuScenes dataset [[]] For BEV map segmentation, following BEVFusion [28] we use the class-averaged mean Intersection-oveer Uninn (mloU) across six background classes (drivable space, pedestrian crossing, walkway, stop line, car- car-parking area, and lane divider) as the evaluation metrics.
Models & Evaluation Configuration We use MMDetection3D [+X]]] and MMRazor [[[I]] under the Apache License 2.0 to implement the fusion module and the knowledge distillation framework. We replace BEVFusion's [28] fusion module with Modality-general Fusion Module to serve as the teacher, keeping all other settings identical to BEVFusion. To verify the versatility of our frame- work, We conduct KD on various representative student models, including the CenterPoint [6.]] Li- DAR student model, BEVDe([[6] with a ResNet-50 image backbone, and BEV [25]] (with- out temporal fusion) the camera student models. Additionally, periments on BEVDet4D-Depthiil
conduct cross-modal KD ex-
student model, which adopts the long-term temporal fusion operation. We maintain the same model architectures and data pipeline as original student models. For the segmentation task, we simply replace the detection head with the BEVFusion [28] segmen- tation head for binary segmentation across all classes. The teacher model utilizes a complex DETR detection head [2]] contrasting with the student models' simpler dense detection heads. This helps us assess the KD method's robustness against heterogeneity in model architecture, besides input modality. Detailed experiment settings can be referred to appendix. IB.
4.2 Comparison with the Staee-of-the-Arts
Table M1 presents the comparative results. We choose representative and influential single-modal algorithms our baseline student models, including CenterPoint [53]] for LiDAR, BEVDet-R50
[166), and BEVFormer-S [25]] for cameras. In 3D object detection, our framework achieves signifi- cant improvements over these baseline models. Our simple LiDAR student performs comparably to those of single-modal state-of-the-art methods, such as TransFusion-L [2] and BEVFusion-L. [28]] which employ more complex DETR detection head, thereby narrowing the performance gap with more sophisticated multi-modal fusion methods. Additionally, our simple BEVDet-R50 student out- performs FCOS3D [ and BEVFusion-C [28], which use heavier image backbones. Our method also exceeds the performance of state-of-the-art cross-modal KD methods, including UniDistill [b8] and BEVDistill [I)], both of which involve elaborate response KD. This underscores the effective- ness of our feature distillation approach, supported by a superior teacher model and learned spatial masks. For BEV map segmentation, our method also shows significant improvements compared to the baselines, validating the inseinndeeedeence of our integrated fusion and cross- cross-modal KD frame- work. Additionally, based on the performance comparison with VCD [[ZZ] in the 3D object detection task, we observe that while our NDS is slightly lower than that of VCD, this may be due to VCDs use of fine-grained trajectory-based distillation, which provides greater advantages in predicting ob- ject motion and velocity. Nevertheless, our method achieves comparable results to VCD in terms of mAP, demonstrating that the multi-sweep LiDAR information in the teacher model used by VeXKD significantly enhances the student's localization capabilities.
7

---

Table 1: Performance of VeXKD nuScenes for 3D object detection and BEV map segmentation tasks. "L" "C" denote LiDAR and the Camera modality, respectively. "L+C" denotes multi-modal fusion model. "L+C ++ C", "L C", and "L+C - L." represent the knowledge distillation from the teacher model to the respective single-modal modal student."+ "+" indicates the addition of cross cross-modal modal KD methods to the above student models. The FPS results evaluated GTX 4090 GPU with batch size of one. "e" denotes re ee-ipllementatinn results. rest-time augmentation applied during testing.
nuScenes test
nuScenes val
Method
Modality
GFLOPs
FPS
mAP
NDS
mAP
NDS
mloU
TransFusion[ BEVFusion
L+C L+C
972
1.8
68.9
71.6
67.5
71.3
507
2.3
70.2
72.9
68.5
71.4
62.9
TransFusion-L[0] BEVFusion-L[22]
L
340
5.8
65.5
70.2
65.1 1
70.1
L
322
5,4
64.7
69.3
48.6
CenterPoin([55]]
L
308
11.3
60.3
67.3
57.4
65.6
48.6
+S2M2-SSD [-]|
L+C+L
308
11.3
63.6
69.6
+Unidistill [6]|
L+C -L
308
11.3
63.9
70.1
59.7
67.5
+VeXKD(Ours)
L+C +
308
11.3
65.1
70.5
64.2
69.6
52.1
FCOS3d[EE]
C
2008
1.7
34.3
41.5
29.5
37.2
BEVDet-TinylID
C
370
6.3
33.3
41.0
56.8*
BEVDet-R50Jin
C
184
14.0
28.9
38.4
28.6
37.2
56.4"
+Unidistill[68]
L+C+C
184
14.0
29.6
39.3
+VeXKD(OOrs)
L+C
184
14.0
35.8
42.6
34.7
40.6
60.7
BEVFormer-S(INy
C
1152
2.6
40.9
46.2
37.5
44.8
61.8*
+Unidistill[BN/
-
1152
2.6
37.7
45.5*
+BEVDistilI
L--C
1152
2.6
38.6
45.7
+VeXKID(Ours)
-+C
1152
2.6
42.5
48.3
41.2
47.7
64.2
BEVDee4DD-Deph((II)
C
220
12.3
39.4
51.5
61.6*
+VCD
L+C
220
12.3
42.6
54.0
+VeXKD(Ours)
L+C
220
12.3
42.8
53.5
63.5
Furthermore, the comparison of giga floating-poing operations (GFLOPs) and inference time clearly illustrates the performance and real-time trade-off introduced by cross-moda KD methods, particu- larly with our VeXKD the student model.
4.3 Ablation Studies
In this section, we validate the effectiveness of each module. Given the diversity modalities and tasks applicable to method, some ablation studies conducted specific modality combinations due to computational constraints.
Effectiveness of Each Proposed Module this paper, we introduce three modules: the Modality- General Fusion Module (MGFM) and Masked Feature Distillation modules for both Low- and High- level BEV features (L-MFD and H-MFD), Configurations without MGFM use the original BEVFu- sion as teachers. We conduct experiments students to evaluate the impact of these modules on different student modalities and tasks. Results in Table indicate each module positively contributes the effectiveness of the KD, with MGFM showing the most substantial impact. This confirms the previously overlooked critical role of the teacher model in cross cros-mmodal KD settings.
CenterPoint detection and BEVDet-R50 segmentation
Effectiveness of BEV Query Guided Mask Generation Network To validate the effectiveness of the learned spatial mask, comparisons are made with state-of- of-the-rrt masking methods on the CenterPoint student model: complete feature mimic (all I's mask) (5), key point sampling [68], Gaussian masks centered on ground truth [7]. and masks derived from teacher's normalized acti- vation map statistics [57]]. map segmentation, key point sampling and Gaussian masks, which rely on ground truth, are inapplicable. Table G shows minimal improvements from complete fea-

---

Table 2: Ablation study of three proposed algorithm components nuScenes val. MGFM denotes the Modality-Geners Fusion Module, L-MFD denotes Low-level Masked Feature Distillation, H-MFD de- notes High-level Masked Feature Distillation. Performance metrics include mAP/NDS for detection and mloU for the segmentation task.
Component
Student Modality & Task
Setting
MGFM L-MFD
H-MFD
LiDAR Detection
Camera Segmentation
57. 4/65.6
56.4
2
59.2/66.8 (+1.8/1.2)
57.6 (+1.2)
58.0/66.1 (+0.6/0.5)
57.9 (+1.5)
4
59.8/66.9 (+2.4/1.3)
58.3 (+1.9)
62.8/68.9 (+5.4/3.4)
59.2 (+2.8)
6
V
61.9/68.5 (+4.5/2.9)
59.6 (+3.2)
64.2/69.6 (+6.8/4.0)
60.7 (+4.3)
ture mimic, underscoring the necessity for feature filtering through masking. Gaussian masks and key point sampling focus on limited foreground points for all levels of features, result in underuti- lized feature distillation. Using normalized activation statistics masks, assuming that regions with higher teacher activation contain critical information, shows correlation but not equivalence. Mean- while, the varying performance across tasks underscores the necessity of learning specialized masks for versatile KD framework. Additionally, comparisons between Mask Generation with Randomly Initialized Queries (MGTIQ) and Mask Generation with Learned Queries (MGLQ) demonstrate the benefits of using fusion byproducts in mask learning.
Table Ablation study different mask selection methods in feature distillation on nuScenes
Student Modality & Task
Method
LiDAR Detection
LiDAR Segementation
No feature distillation
$7.4/65.6
48,6
Feature distillation entire feature maps|E]
58.7/66.4
50.5
Feature distillation on foreground key points [6
60.3/67.0
Feature distillation masked by Gaussian[[]]
60.8/67.3
Feature distillation activation valuc|5.7]
61.4/67.7
50.9
Mask Generation with Randomly Initialized Query
63.7/69.1
51.6
Mask Generation with Learned Query from fusion
64.2/69.6
52.1
Effectiveness of Attention Transfer We use attention transfer (ATTN) [56] to compute the fea- ture distillation loss, as helps alleviate the channel dimension heterogeneity from differing archi- tectures. Table ! presents comparison of results using attention transfer versus traditional loss L1, L2, and Smooth LI on CenterPoint student model, each implemented with a simple convolu- tional adaptive layer. Although employing MGFM and learned masks renders the choice of specific distillation loss less critical, attention transfer consistently outperforms other losses across tasks.
Table 4: Ablation study of different loss functions in feature distillation nuScenes val. The choice of specific distillation loss is less critical while atten- tion transfer performs better.
Student Modality & Task
Method
LiDAR Det.
LiDAR Seg.
No KD
57, 6/756/6.6
48.6
Block Number
LI
62. 8/69/0
51.3
L2
63.7/69.2
51.6
Figure 5: Ablation study of the blacks number in mask generation network on nuScenes val.
Smooth-L1
63,5/69.3
51.7
ATTN|E6]
64.2/69.6
52.1

---

Influence of Mask Generation Network Architecture Our mask generation network consists of stacked transformer blocks. We examine how the number of blocks affects mask learning convergence speed and KD results on CenterPoint student model. Figure 5I shows that increasing the block count beyond two does not significantly enhance KD performance. However, employing three or more blocks accelerates convergence. To balance learning speed and performance, we opt for block count of three.
kow level
high level
(a) Object detection
4.4 Qualitative Results
Figure E shows spatial masks for different levels and tasks created by our BEV Query Guided Mask Generation module. In detection tasks, the mask emphasizes a wider area around foreground objects, especially in low-level BEV features; in segmentation tasks, it also highlights back- ground features. Figure / displays BEV features before and after our fu- sion module, demonstrating that fusion module effectively enhances crucial areas with camera contextual information. Figure & compares the BEV feature maps before and after KD, illustrating how method improves camera features with more deterministic depth projection ac- curacy and accentuates important LiDAR features by correlating point distribution with image textural information.
(b) BEV segmentation
Figure 6: Visualization of learned spatial mask. Our mask generation network can produce spatial masks for features at different lev- els, tailored for various 3D perception tasks.
(a) LiDAR
(b) MGFM
(a) Cam. before (b) Cam. KD (c) lidar before (d) lidar KD
Figure 7: Feature maps before and after MGFM, showing hancements in crucial areas with camera textural feature.
Figure 8: Comparison of feature maps without vs. with distillation. KD improves camera feature, offering more deterministic view projection accuracy and accentuating important LiDAR features.
5 Conclusion
We propose VeXKD, simple and versatile framework that integrates cross-mon modsll fusion and knowl- edge distillation, suitable for various student modalities and 3D perception tasks. Our approach constructs superior fusion model as the teacher and enhances the effectiveness of cross-moda knowledge distillation. Using BEV query guided mask generation network, we develop adapt- able feature distillation pipeline that produces spatial masks for features across different tasks and student models, demonstrating its potential for integration with future 3D perception methods. The effectiveness of our framework is confirmed through experiments on the nuScenes dataset, aiming to spur further research into versatile KD frameworks that move beyond model-specific and intricately engineered setups to more universally adaptable approaches.
Limitations While our framework is adaptable and has been tested across various modalities and tasks, many scenarios remain untested due to time and computational constraints, such as radar modality or the motion prediction task. The modality-gnnrral information utilized in our fusion model has only been shown through experimental outcomes and visualizations. Currently, no stud- ies can theoretically quantify this information. Future research could aim to extend knowledge distillation to more 3D perception tasks and explore the theoretical aspects that influence its effi- cacy. We utilize the implicit multi-sweep LiDAR information from the teacher model for temporal knowledge transfer to the student model. The comparison with VCD demonstrates the potential of integrating explicit temporal knowledge distillation operations. Developing versatile cross-modal KD frameworks based on explicit temporal knowledge transfer could be a promising future direc- tion.
Mask convergence is defined the condition where the difference in loss for the teacher's downstream task between the post-mask and original feature maps is less than 0.1.
10

---

Acknowledgements
This work was partially supported by the National Natural Science Foundation of China (Grants 62373315, U23A20339. 62373298), the Guangdong Provincial Key Lab of Integrated Commu- nication, Sensing and Computation for Ubiquitous Internet of Things (2023B 12BB1222011 201000 the Guangzhou Basic and Applied Basic Research Project (2023A0310683), the Guangzhou Munici- pal Science and Technology Project (2023A031001 1), and the Nansha Key Science and Technology Project (2023ZD006).
References
[1] Eduardo Arnold, Omar Al-Jarrah, Mehrdad Dianati, Saber Fallah, David Oxtoby, and Alex Mouzakitis. A survey on 3d object detection methods for autonomous driving applications. IEEE Trunsactions on Intelligent Transportation Systems, 20(10):782-3395 2019.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Trans- fusion: Rohust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on compuder vision and pattern recognition, pages 1090-1099. 2022.
[3] Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[4] Holger Cacsar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krish- nan, Yu Pan. Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[5] Weihan Cao, Yifan Zhang, Jianfei Gao, Anda Cheng, Ke Cheng, and Jian Cheng. Pkd: General distilla- tion framework for object detectors via pearson correlation coefficient. Advances in Neural Information
Processing Systems, 35:15394-155 15406. 2022.
[6] Xiaozhi Chen, Huimin Ma. Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pages 1905-1915, 2017.
Zehui Chen, Zhenyu Li, Shiquan Zhang, L Liangji Fang. Qinhong Jiang, and Feng Zhao. Bevdistill: Cross- modal bev distillation for multi-view 3d object detection. arXiv preprint veXiv:2211.09320 2022.
[8] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the IEFF/CVF international conference on computer vision, pages 4794-4802, 2019.
[9] Zhiyu Chong, Xinzhu Ma, Hong Zhang. Yuxin Yue, Haojie Li. Zhihui Wang, and Wanli Ouyang. Monodistill: Learning spatial features for monocular 3d object detection. arXiv preprint arXiv:2201. 2022,
[10] MMDetection3D Contributors. MMDetection3D: OpenMMLab next generation platform for general 3D object detection. fttps://githute com/ ptte//pen-mmmmmm wammllaammmettttinnnee 2020.
[1I] MMRazor Contributors. Openmmlab model compression toolbox and benchmark. https: //ithbb.
com/ hwtp://ennmmmaaaaaaaa mmrooooororrrrroom 2021.
[12] Jiajun Deng, Shaoshuai Shi, Poiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 1201-1209, 2021.
[13] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 2154-214 2164, 2021.
[14] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprini arXiv:1500 (0:::51033000 503.02330 2015.
[15] Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203.
2022.
[16] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint arXiv:21 1790, 2021.
[17] Linyan Huang, Zhiqi Li. Chonghao Sima, Wenhai Wang. Jingdong Wang. Yu Qiao, and Hongyang Li. Leveraging vision-centrie multi-modal expertise for 3d object detection. Advances in Neural Information Processing Systems, 36, 2024.
[18] Tao Huang, Yuan Zhang, Shan You, Fei Wang, Chen Qian, Jian Cao, and Chang Xu. Masked distillation with receptive tokens. arXiv preprint arXiv:2205. /4589, 2022.
[19] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Osear Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12700 2019.
11

---

[20] Jianing Li. Ming Lu, Jiaming Liu, Yandong Guo, Yuan Du, Li Du, and Shanghang Zhang. Bev-Igkd: A unified lidar- uida--giided knowledge distillation framework for multi-view bev 3d object detection. IEEE Transactions on Intelligent Vehicles, 2023.
[21] Quanquan Li, Shengying Jin, and Junjie Yan. Mimicking very efficient network for object detection. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 1556-6364, 2017. [22] Qi Li, Yue Wang. Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA), pages 4628 4634. IEEE, 2022.
[23] Yinhao I.,, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang. Yukang Shi, Jianjian Sun, and Zeming
Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 1477-1485, 2023.
[24] Yingwei Ii, Adams Wei Yu, Tianjian Meng. Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V Le, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition,
pages 17182-17191, 2022.
[25] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning birds-eye-view representation from multi-camera images via spatiotemporal trans- formers. In European conference on computer vision, pages 1-18. Springer, 2022.
[26] Ming Liang. Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European conference on computer vision (ECCV), pages 641-656, 2018.
[27] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position emhedding transformation for multi-view 3d object detection. In European Conference on Computer Vision, pages 531 331-548. Springer, 2022.
[28] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified s-rde view representation. In 2023 IEEE international conference robotics and automation (ICRA), pages 2774-2781. IEEE, 2023.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[30] Qianhui Luo, Huifang Ma, Li Tang. Yue Wang, and Rong Xiong. 3d-ssd: Learning hierarchical features from rgb-d images for amodal 3d ohject detection. Neurocomputting. 378:364-374, 2020.
[31] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection.
In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2906 2917, 2021.
[32] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by im- plicitly unprojecting 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow; UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 194-210. Springer, 2020.
[33] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: Fully convolutional anchor-free 3d object detection. In European Conference on Computer Vision, pages 477-493. Springer, 2022.
[34] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image to voxels projection for monocular and multi-view seneral-purpose 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2397-2406. 2022.
[35] Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian. Distilling object detectors with task adaptive regularization, arXiv preprint arXiv:2006.. 13108, 2020.
[36] Fida Mohammad Thoker and Juergen Gall. Cross-modal knowledge distillation for action recognition. In
2019 IEEE International Conference on Image Processing (ICIP), pages 6-10. IEEE, 2019.
[37] Zhi Tian, Chunhua Shen, Hao Chen, and Tong Hc. Fcos: A simple and strong archor-free object detector. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(4): 1922-193..919 2020. John Barry, Joseph Walsh, et al. Autonomous driving architectures, percep-
[38] Gustavo
tion and data fusion: A review. In 2020 IEEE 16th International Conference Intelligent Computer Communication and Processing (ICCP), pages 315-321. IEEE, 2020.
[39] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pages 4604-4612, 2020.
[40] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 11794-11803, 2021.
[41] Guo-Hu Wang. Yifan Ge. and Jianxin Wu. Distilling knowledge by mimicking features. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44(1))8818 8893,,19,,,18 2021.
[42] Jingdong Wang. Zhen Wei, Ting Zhang, and Wenjun Zeng. Deeply-fused nets. arXiv preprint arXiv: 1605.07716, 2016.
[43] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng. Distilling object detectors with fine- -rainer feature imitation. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Pattern Recognition. pages 4933 4942, 2019.
12

---

[44] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913-922, 2021.
[45] Yuc Wang. Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learn- ing. pages 180-191. PMLR, 2022.
[46] Yue Wang. Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Trunsactions on Graphics (tog), 38 2019.
[47] Zeyu Wang. Dingwen Li, Chenxu Luo, Cihang Xie, and Xiaodong Yang. Distillbev: Boosting multi- camera 3d object detection with cross-modal knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 8637-8646. October 2023.
[48]
Yutian Wu, Yueyu Wang, Shuwei Zhang, and Harutoshi Ogai. Deep 3d object detection networks using
lidar data: review. IEEE Sensors Journal, 21 1122:111221111111 2020.
[49] Mutian Xu, Runyu Ding, Hengshuang Zhao, and Xiaojuan Qi. Paconv: Position adaptive convolution with
dynamic kernel assembling on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3173-3182, 2021.
[50] Zihui Xue, Zhengqi Gao. Sucheng Ren, and Hang Zhao. The modality focusing hypothesis: Towards
understanding crossmodal knowledge distillation. arXiv preprint arXiv:2206. 06487, 2022.
[51] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
[52] Bin Yang. Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7660, 2018.
[53] Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones bird's-eye- view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 17830-17839 2023.
[54] Zhendong Yang. Zhe Li, Mingqi Shao, Dachuan Shi, Zehuan Yuan, and Chun Yuan. Masked generative distillation. In European Conference on Computer Vision, pages 53-69. Springer, 2022.
[55] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784-
11793, 2021.
[56] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint arXiv: 1olp:0392299 2016.
[S7] Linfeng Zhang and Kaisheng Ma. Improve object detection with feature-based knowledge distillation: Towards accurate and efficient detectors. In International Conference on Learning Representations, 2020.
[58] Renrui Zhang, Han Qiu, Tai Wang, Ziyu Guo, Ziteng Cui, Yu Qiao, Hongsheng Li, and Peng Gao. Mon- odetr: Depth-guided transformer for monocular 3d object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9155-9166, 2023.
[59] Zhang, Chuangao Tang, and Cuntai Guan. Visual-to-eeg tross-modal knowledge distillation for con- tinuous emotion recognition. Pattern Recognition, 130; 108833, 2022.
[60] Yunpeng Zhang, Wenzhao Zheng, Zheng Zhu, Guan Huang, Jiwen Lu, and Jie Zhou. simple baseline for multi-camera 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 37, pages 3507-3515, 2023.
[61] Borui Zhao. Quan Cui. Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 1 11953- 1962, 2022.
[62] Lin Zhao, Hui Zhou, Xinge Zhu, Xiao Song, Hongsheng Li, and Wenbing Tao. Lif-seg: Lidar and camera image fusion for 3d lidar semantic segmentation. IEEE Transactions on Multimedia, 2023.
[63] Xiaodong Zhao. Qichao Zhang, Dongbin Zhao, and Zhonghua Pang. Overview of image segmentation and its application on free space detection. In 20/8 IEEE 7th Data Driven Control and Learning Systems Conference (DDCIS), pages 166-11669 IEEF, 2018.
[64] Wu Zheng, Mingxuan Hong, Jiang, and Chi-Wing Fu. Boosting 3d object detection by simulating multimodality on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13638-13647, 2022.
[65] Wu Zheng, Weiliang Tang, Sijin Chen, Li Jiang, and Chi-Wing Fu. Cia-ssd: Confident aou-aware single- stage object detector from point cloud. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 3555-3562, 2021.
[66] Wu Zheng, Weiliang Tang. Li Jiang. and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14494 14503, 2021.
13

---

[67] Brady Zhou and Philipp Krähenbühl. Cross-view transformers for real-time map-view semantic segmen- tation. In Proceedings the IEEE/CVE conference computer vision and pattern recognition, pages 13760-13769, 2022.
[68] Shengchao Zhou, Weizhou Liu, Chen Hu, Shuchang Zhou, and Chao Ma. Unidistill: A universal cross- modality knowledge distillation framework for 3d object detection in bird's-eye view. In Proceedings
IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5116-5125 2023.
[69] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4490 4499, 2018.
[70] Benjin Zhu, Zhengkai Jiang. Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced Grouping and Sampling for Point Cloud 3D Object Detection. arXiv e-prints, page 1ii::1908.0900 Aug. 2019.
[71] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable
transformers for end-to-end object detection. arXiv preprint arXiv:2010.04155 2020.
[72] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical convolution networks for lidar segmentation. Proceedings of IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
14

---

Appendix
A Preliminary
In the realm of 3D perception conducted in BEV space, whether through single-modal or multi- modal fusion, the paradigm depicted in Figure A is commonly adopted. Within this paradigm, multi-view camera- comera-based algorithms utilize an image backbone to extract semantic information from multi-view image inputs sing
p piies This information is then projected onto BEV space from the image space through forward projection [32 [6] or backward projection
125 533, resulting in
low-level image BEV features Flow as in Eq. A.
img
Flow Projection (ImgBaakbonne (Tmult((vieee)) (mig
(A.1)
In the context of LiDAR-based algorithms, benefiting from the inherent geometric accuracy of Li- DAR point clouds, the LiDAR input Lider can be easily pooled into the BEV (Bird's Eye View) space, regardless of whether voxel-based or pillar-based encoders are used. This process yields low-level LiDAR BEV features, denoted Flow as Eq. B.
glow lidar
LiDAREncoder (Iudar)
(A.2)
Meanwhile, the multi-modality fusion pipeline incorporates both Filter and Ping as inputs. These subsequently processed through fusion module to produce fused low-level features Plaw fusion
smg lsdars followed by process- ing through several layers of simple convolution to obtain F as in Eq.
Specifically, BEVFusion [28] achieves this by concatenating Flog and Flow
F fusson
FusionModulle (mimg Flace
(A.3)
Regardless of whether it is LiDAR-only, camera-only or even fusion model, the low level BEV features are not suitable for direct 3D perception tasks due the spatial inaccuracy and low se- mantic level. Therefore, they are transformed into high-level BEV features Fhigh modal (lidar, img, fusion) through BEV encoders as in Eq. A.4
modal
BEVEncoder (Frove
modal {camera, lidar, fusion)
(A.4)
Subsequently, these high-level BEV features are processed through various task-specific heads to ob- tain the final results for 3D perception tasks, such as 3D map segmentation or object detection. Such common paradigm provide the foundation of building versatile knowledge distillation framework that can be applied to various student modalities and 3D perception tasks.
LIDAR BEY
Jeature Flee
HFLAAA hnggeetttedl algurtilu
LIBAR
Seree Thturden
High-lovsl KEV fattrr
Treatienas
Faliao
Onject
REY Eacodes
Mulliitor Japui
caronos sigge snglemmede sigrasitin
All kind ual. spadile
Mrp Segmesmation Rerui
Image frature yire
Imags Buckbote & Progection
Figure A.1: Common paradigm for BEV-based 3D perception tasks: Despite variations in model architectures, both lowelevel and high-level BEV features are common to all BEV-based algorithms.
15

---

B Detailed Training Settings
All experiments are conducted with a batch size of 32, training from scratch on 8 of NVIDIA A800 GPU. By default, each student is trained for 20 epochs using the AdamW optimizer [29] with weight decay of 10-2. Throughout the training process of KD, the teacher model remains frozen. The detection range was set to [-54.0, 54.0] for the X and Y axes, and the voxelization resulted in a resolution of (0.075m, 0.075m, 0.2m). Both low-level and high-level feature maps are sized at 180 x 180 in height and width. The hyper-parameters applied to the loss terms μ and À are simply set to 1.0 and 2.0 to balance the scale of losses in our experiment.
BEVDet-50 Knowledge Distillation Experiment Following the original BEVDet [TG]] experi- ments, a learning rate of 2 10 is used, with the ResNet-50 image backbone's ' learning rate reduced by a factor of 0.1 to stabilize training. The input image is cropped to 704 x 256, and identi- cal image augmentations are applied to both the student and the teacher prevent misalignment. For 3D object detection, the CenterPoint detection head is used, and for segmentation, the segmentation head from BEVFusion employed for comparative experiments before and after KD.
CenterPoint Knowledge Distillation Experiment For the 3D object detection task, rotate type of Non-Maximum Suppress (NMS) algorithm is used for processing results. For the segmentation task, the CenterPoint head is directly replaced with the segmentation head from BEVFusion for experimental comparison. Following the original settings, CBGS [ and a one-cycle learning rate policy are employed with an initial learning rate of x 10-4.
BEVFormer-S Knowledge Distillation Experiment To mitigate the impact of temporal self- attention in BEVFormer, BEVFormer-S is chosen as the student by adjusting the temporal self- attention into a vanilla self-attention without using historical BEV features. To maintain consistency with the teacher, the resolution of BEVs grid is adjusted from the original 0.512m to 0.6m, thereby reducing the BEV queries size to 180 x 180 instead of 200 200. Training for BEVFormer- is con- ducted over 24 epochs, using the same Cosine Annealing learning rate and momentum scheduling as the original. learning rate of x 10-4 used, with the ResNet-101 image backbone's 's learning rate reduced by factor of 0.1 to stabilize training.
iment configuration of Knowledge distillation Experiment This experiment follows the exper- The input image size is set to 2560E704, and the temporal fusion utilizes feature maps from the past 8 frames. ResNet-50 is employed as the image backbonn, and CenterHead is used for 3D object detection. Additionally, the depth estimation for the image is supervised using the depth loss proposed in BEVDepth.
BEVDet4D-Longten
C More Qualitative Results
Visualizaiton of BEV features before and after Knowledge Distillation As illustrated in Fig. Mll significant differences are evident in the feature maps of both camera and LiDAR students before and after knowledge distillation (KD). For camera students, KD can serve as a form of depth supervision during the projection process from the perspective view to the BEV (Bird's Eye View). Consequently, compared to the original feature map of BEVDet-R50, which is chaotically distributed across each depth bin, the KD-enhanced feature map contains more accurate and deter- ministic depth information, benefiting the precision of 3D perception localization. For the LiDAR student, KD helps establish connection between the point cloud distribution and the camera's 's tex- tural information. This enhances the visibility of key foreground features, partially compensating for the sparsity of point clouds at greater distances.
Visualization of detection result before and after Knowledge Distillation As previously men- tioned, cross- modal KD can act as a regularization mechanism in the projection process for camera- based systems. The detection visualization results, shown in Fig. C.ll demonstrate that KD en- hances confidence in depth estimates for camera students, helping to prevent the occurrence of false positives. For LiDAR students, KD facilitates the establishment of a correlation between point distribution and the camera's 's textural information. This enhancement aids in preventing the misclas- sification of objects into different categories and reduces false positives. Additionally, KD improves the recognition and accurate classification of objects with fewer points at greater distances.
16

---

(a) Visualization of camera BEV feature before knowledge distillation
(b) Visualization of camera BEV feature after knowledge distillation
(c) Visualization of LiDAR BEV feature hefore knowledge distillation
(d) Visualization of LiDAR BEV feature after knowledge distillation
Figure C.1: BEV feature map comparison between KD and no-KD ones
Visualization of segmentation result before and after Knowledge Distillation In the task of BEV map segmentation, a primary challenge for the camera modality is the unclear boundary de- lineation and boundary spreading caused by inaccuracies in the view projection process. As shown in Fig. OK
KD can significantly enhance the accuracy of depth estimation in this projection, facil- itating more precise reconstruction of the original BEV scene. Conversely, LiDAR models often underperform due to their inherent sparsity and reduced focus on background areas, which typi- cally have fewer reflective points. The framework introduced this study incorporates a learned spatial mask that increases focus these background locations, thereby preventing hallucinations near regions with sparse LiDAR points and improving the accuracy of background segmentation. Additionally, the established correlation between camera textural information and point cloud dis- tribution promotes more accurate pixel-wise classification, further enhancing model performance in densely populated nearby areas.
D Broader Impacts
Effective and real-time 3D perception is crucial for safety of autonomous vehicles. By integrat- ing cross-modal fusion and knowledge distillation, VeXKD transfers the multi-modal knowledge to improve the accuracy of single-modal student without additional inference time overhead, thus achieving better efficiency and accuracy trade-off. With its versatility, VeXKD can be applied to
17

---

NIE KHEAD REED
(a) Camera detection result before knowledge distillation
AAE H
(b) Camera detection result after knowledge distillation
(c) LiDAR detection result before knowledge distillation
(d) LiDAR detection result after knowledge distillation
Figure C.2: Detection result before and after knowledge distillation for camera and LiDAR students.
18

---

(a) Ground truth segmentation label for segmentation
(b) Camera segmentation result before knowledge distillation
(c) Camera segmentation result after knowledge distillation
(d) Ground truth segmentation label for LiDAR segmentation
(e) LiDAR segmentation result before knowledge distillation
(f) LiDAR segmentation result with knowledge distillation
Figure C.3: Segmentation result before or after knowledge distillation
19

---

various student models and downstream 3D perception tasks, It can help pave the way for safe and robust autonomous driving by integrating with the ever-evolving single-modal algorithms.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main contribution of our paper is the design and implementation of an effective and versatile framework that integrates cross-modal fusion and knowledge dis- tillation. In the methodology section, described in section.
we illustrate the overall architecture in subsection. B.I. the fusion module in subsection. 3.2 and the masked feature distillation in subsection. B.l and subsection. G detailing how they contribute to an effective and versatile framework. Finally, through experiments different student modalities and tasks, as discussed subsection. μ.2,, we have demonstrated the versatility and effectiveness of our proposed framework. We also conduct extensive ablation studies in subsection. ! to show the effectiveness of each proposed module.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the con- tributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings. It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have created separate "Limitations" section to point out the limitations of our work, which can be referred to section. B
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not diseussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specificatinn fication,,, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicaae,, the authors should discuss possible limitations of their approach to address problems of privacy and faimess..
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important
21

---

role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: As an experimental science, this paper proposes framework, implements it, and demonstrates both quantitative and qualitative results. Additionally, we mention in the Limitations section E that there has been no research on theoretically analyzing the mech- anisms of cross-modal knowledge distillation nor on the theoretical proof of the impact of different teachers on experimental outcomes.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have described the detailed architecture and the operation used for our methods in section. & elaborating our description with natural languages, figures and equa- tions. Additionally, we provide the source code along with the documentation in the sup- plemental materials to further improve the reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate the research performed.
be accomplished in various ways.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution primarily a new model architecture, the paper should describe the architecture clearly and fully.
22

---

(c) If the contribution is a new model (e.g., large language model), then there should either be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed- source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide source code and configuration files for training and evaluating methods, along with the instructions data access and preparation in supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (ittps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so No is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:)
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new pro- posed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed description of the training and testing details in subsec- tion. D and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
23

---

Answer: [No]
Justification: Acquiring statistical significance for our experiments requires running them multiple times. Since models trained from scratch, this process is both time- consuming and resource-intensee most of the existing research in this field. Therefore, follow the convention and omit the analysis of experiment statistical significance.
We have observed that this aspect is often omitted in
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main
claims of the paper.
The factors of variability that the bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1 I-sigma enror bars, but one should state it. The authors should prefer- ably report 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or fig- ures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provide the GPU resources we used to run the experiments ap- pendix. H
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 'ttt make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed the NeurlPS Code of Ethics. The experiments in our paper do not involve human subjects or participants. And the paper aims to improve the safety of autonomous vehicles as mentioned in appendix. D
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
24

---

The authors should make sure to preserve anonymity (e.g,, if there is special considera- tion due laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We aims to improve the safety of autonomous vehicles, And the societal impact illustrated in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is needed point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions access the model implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
25

---

Justification: We have cited all assets, including datasets and open-source platforms used for code implementation, and mentioned their licenses in subsection. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes]
Justification: We provide the code in supplementary materials and provide documentation with it.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limi- tations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation,
other labor should be paid least the minimum wage in the country of the data collec- tor.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
26

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
27


Reviews:
[{"review_id": "e9KtCWAIDD", "reviewer": "Reviewer_dGF9", "summary": "This paper proposes VeXKD, a method that performs Knowledge Distillation (KD) in the BEV feature space. By distilling cross-modal knowledge from a teacher model into a single-modal student model, VeXKD eliminates the need for additional inference time overhead. The distilled student model can be adapted to various tasks by attaching task-specific heads.", "questions": "1. Building on Weakness 1, is there an easy way to apply this pipeline to a model that does not use the BEV feature space?\n2. If possible, could the authors provide results for other modalities? If time or computational resources do not permit, could the authors clarify what additional efforts are required to apply this method to other modalities?", "limitations": "The authors have addressed the limitations of their work in the checklist, and there are no significant concerns regarding potential negative societal impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper employs a cross-modal KD approach to transfer insights from a multi-modal teacher model to a student model without adding extra overhead. Experimental results demonstrate that the student model’s performance significantly improves.\n2. The paper introduces a mask generation module, which ensures that only useful information is transferred during the KD process using these masks.\n3. The proposed KD method is highly flexible, capable of handling various modality inputs and downstream tasks.", "weaknesses": "1. The student and teacher structures must adhere to the design shown in Appendix Figure A.1, which is not compatible with models that do not use the BEV feature space.\n2. The masked teacher perception loss requires task-specific training, hindering the possibility of performing new tasks in a zero-shot manner.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: Adaptation on other feature representation**\n\nIn response to your valuable insights, we would like to offer some clarifications. When conducting KD, it is crucial for the feature maps of both the student and the teacher to reside within the same feature space to ensure spatial and semantic compatibility. In this regard, the BEV feature space has gained significant attention in recent years due to its favorable compatibility with multiple modalities and the similar perception pipeline across modalities in 3D perception. Indeed, the volume of research focused on the BEV feature space far surpasses that on other feature spaces in recent years.\n\nFurthermore, if student and teacher feature maps are in different feature spaces, projection can be used to align them. Once the spatial alignment is done, the specific feature space used for KD theoretically and empirically would not impact its effectiveness that much. For example, our masked distillation is inspired by previous work in the RGB image feature space[1] and is adapted to better address the semantic complexity of the BEV space through strategies like the initalizing and learning of dense BEV queries. Similarly, the methodologies we have developed for masked feature distillation and the construction of a modality-general fusion model can be adapted to other feature spaces, such as RGB or depth image features.\n\n### **Q2: Adaptation on other modalities**\n\nWe appreciate the opportunity to further clarify the adaptability of our methods with the inclusion of additional modalities. Our methodology is designed to handle each modality symmetrically, ensuring that integrating new modalities does not necessitate alterations to the existing codebase, but rather requires configuration updates to accommodate them. Additionally, these new modalities should be integrated into the training of the new fusion teacher to extract the modality-general information from new modalities.\n\nOnce the teacher model is trained, the masked feature distillation operation can be applied to facilitate specific feature mask learning and selective feature distillation for the newly integrated student modality model. For the student model, augmenting the existing task loss with the KD loss is sufficient to complete its training.\n\nIt is worth noting that our framework primarily conducts KD within the BEV feature space for LiDAR and camera students. However, BEV feature space is compatible with various modalities, including raw mmWave radar points[2, 3].\n\nIncorporating additional modalities necessitates training the new fusion teacher and new student models, adding the data processing operation for new modalities. Completing this retraining process within the rebuttal period is challenging. We hope this clarification is helpful and addresses your concerns effectively.\n\n**References**\n\n[1] Huang, Tao, et al. \"Masked distillation with receptive tokens.\" arXiv preprint arXiv:2205.14589 (2022).\n\n[2] Stäcker, Lukas, et al. \"RC-BEVFusion: A plug-in module for radar-camera bird’s eye view feature fusion.\" DAGM German Conference on Pattern Recognition. Cham: Springer Nature Switzerland, 2023\n\n[3] Harley, Adam W., et al. \"Simple-bev: What really matters for multi-sensor bev perception?.\" 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QOP8uZUTz9", "reviewer": "Reviewer_GBJi", "summary": "This paper presents VeXKD, an innovative framework that combines Cross-Modal Fusion and Knowledge Distillation (KD) to significantly enhance 3D perception capabilities. VeXKD employs knowledge distillation on BEV feature maps, facilitating the seamless transfer of multi-modal insights to single-modal student models without incurring additional computational overhead. The framework incorporates a versatile cross-modal fusion module designed to bridge the performance gap between multi-modal teacher models and their single-modal counterparts. Extensive experiments conducted on the nuScenes dataset have yielded substantial performance improvements, effectively reducing the disparity with state-of-the-art multi-modal models.", "questions": "See weakness.", "limitations": "The authors have discussed the limitations of their work in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The integration of Cross-Modal Fusion with Knowledge Distillation in the domain of 3D perception is a novel approach that offers a fresh perspective for enhancing single-modal models.\n\n2. The experimental outcomes on the nuScenes dataset, which show significant improvements in key metrics such as mAP, NDS, and mIoU, substantiate the effectiveness of the proposed methodology. Extensive ablation studies have also demonstrated the effectiveness of each module.\n\n3. The framework is designed to be modality- and task-agnostic, capable of being applied to various student modalities and downstream tasks without being constrained by specific network architectures or processing steps, ensuring the universality of the approach.", "weaknesses": "1. The framework's heavy reliance on BEV feature maps might limit its applicability to other types of feature representations.\n\n2. The paper's strategy for selecting teacher models is rather limited, lacking a comprehensive comparative analysis of the method's performance and effectiveness across various teacher model configurations.\n\n3. The authors have not evaluated whether the proposed distillation method remains effective when incorporating temporal information.\n\n4. Some of the tables lack sufficient detail, which affects readability. For instance, adding the types of teacher models in the caption of Table 1 would provide clearer context and enhance the table's usefulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **1. Q1: Adaptation on other feature representation**\n\nIn our manuscript, the experiments were conducted on the BEV feature map. The BEV feature space has become a focal point of research in recent years due to its favorable compatibility with multiple modalities and its similar processing pipeline.\n\nHowever, as long as the student and teacher models can achieve spatial alignment, the specific feature space in which KD is conducted would not impact its effectiveness that much empirically. Our approach of masked distillation is inspired by previous KD work conducted in the RGB image feature space [1] and has been adapted to address the BEV space's semantic complexity by initializing and learning dense BEV queries. Similarly, the methodology we have developed for masked feature distillation and the construction of a modality-general fusion model can be adapted to other feature spaces, such as RGB or depth image features.\n\n## **2. Q2 & Q4: Add teacher model type column to KD methods &  The Comparison of different teacher model**\n\n**Supplementary Table:**  Overview of Teacher Models Used in Knowledge Distillation Methods\n| Method              | Modality | Teacher Model |\n| :---------------- | :------: | :----: |\n| CenterPoint      |   L   | -- |\n| + S2M2-SSD           |  L+C -> L   | Multi-modality SSD |\n| + Unidistill    | L+C -> L   | BEVFusion |\n| + VeXKD(Ours) |  L+C -> L  | Modality-General Fusion Teacher (Ours) |\n| BEVDet-R50| C | -- |\n| +Unidistill | L+C -> C | BEVFusion |\n| +VexKD(Ours) | L+C -> C |  Modality-General Fusion Teacher (Ours) |\n| BEVFormer-S| C | -- |\n| +Unidistill | L+C -> C| BEVFusion |\n| +BEVDistill | L -> C| Object DGCNN[3] |\n| +VexKD(Ours) | L+C -> C |  Modality-General Fusion Teacher (Ours) |\n\nThank you for your constructive suggestions, which prompted me to add a table representing the teacher models used in various KD methods. As indicated in the table, most KD methodologies consider the architectural similarity between the teacher and student models when choosing the teacher model. For instance, S2M2-SSD employs a multi-modal teacher model similar to PointPainting[2], which fuses image segmentation results with LiDAR features before voxelization to supervise LiDAR student right from voxelization process. Unidistill adopts the BEVFusion model as a teacher, which aligns structurally with single-modal students. BEVDistill ensures structural similarity with the BEVFormer student by using Object DGCNN[3] as LiDAR teacher, which is based on the DETR encoder and decoder architecture.\n\nIn our VeXKD study, we adopted a BEVFusion pipeline similar to Unidistill but modified the fusion module to enhance the teacher's efficacy in KD. This modification was inspired by observing performance gaps between Unidistill and those KD methods exclusive to camera students like BEVDistill. Our ablation study also reveals that a significant portion of the performance gain is attributable to these modifications in the teacher models.\n\nAdditionally, when implementing our research, we explored building a multi-modal teacher model using the global attention method described in [4]. However, replicating the fusion teacher with this method resulted in significant GPU memory consumption, challenging the training process. This experience inspired the replacement of global attention with more efficient deformable attention for modality-general fusion.\n\nAs mentioned in the limitations of our paper, the lack of research quantifying the modality-general information contained in teacher models makes the process of experimenting with different teacher models time-consuming, effort-intensive, and fraught with uncertainty. Our paper aims to provide an example schema for extracting modality-general information from teacher models without adapting the teacher's model architecture. We hope that future research will include more theoretical analyses on the different teachers in cross-modal KD.\n\n## **3. Q3: Incorporation of temporal information**\n\nThank you for your suggestions. As clarified in the global response, both our teacher model and LiDAR students inherently incorporate multi-sweep LiDAR inputs, thereby implicitly integrating temporal information.\n\nInspired by your feedback, we conducted additional experiments with BEVFormer, a student model that combines temporal information, and observed performance gains, as detailed in the global response. As technologies for multi-modal explicit temporal fusion continue to mature and become open-sourced, we foresee the potential for integrating explicit temporal KD operations as a pluggable module into the VeXKD framework in future developments.\n\n\n**References**\n\n[1] Huang, Tao, et al. \"Masked distillation with receptive tokens.\" arXiv preprint arXiv:2205.14589 (2022).\n\n[2] Vora, Sourabh, et al. \"Pointpainting: Sequential fusion for 3d object detection.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[3] Wang, Y., & Solomon, J. M. (2021). Object dgcnn: 3d object detection using dynamic graphs. Advances in Neural Information Processing Systems, 34, 20745-20758\n\n[4] Man, Yunze, Liang-Yan Gui, and Yu-Xiong Wang. \"BEV-guided multi-modality fusion for driving perception.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ECES9GvhrC", "reviewer": "Reviewer_uk3B", "summary": "This paper introduces VeXKD, a Versatile framework that integrates Cross-Modal Fusion with Knowledge Distillation for 3D detection tasks. The framework adopts a modality-general cross-modal fusion module to bridge the modality gap between the multi-modal teachers and single-modal students. Extensive experiments on the nuScenes dataset demonstrate improvements for 3D detection and BEV map segmentation tasks.", "questions": "1. As shown in Table 1, CenterPoint+VeXKD (L+C->L) achieves lower detection accuracy than TransFusion-L (L). Then does it make sense to use a multi-modality teacher model to guide a lidar-based student model?\n2. The inference speed of the student model needs to be given and compared with existing state-of-the-art models and real-time inference models on detection and map segmentation tasks.\n3. What effect does temporal information have on the final results? (No experimental results are needed, just an exploratory question.)", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Cross-modal fusion and knowledge distillation for 3D perception is a significant problem, and this paper proposes an alternative strategy to current methods. The motivation for the paper is reasonable and inspiring. The proposed versatile knowledge distillation method for different downstream 3D perception tasks, such as 3d detection and map segmantation, is innovative. This paper is well-written and easy to follow.", "weaknesses": "1.As shown in Table 1, CenterPoint+VeXKD (L+C->L) achieves lower detection accuracy than TransFusion-L (L). Then does it make sense to use a multi-modality teacher model to guide a lidar-based student model?\n2.The inference speed of the student model needs to be given and compared with existing state-of-the-art models and real-time inference models on detection and map segmentation tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Q1: CenterPoint+VeXKD (L+C->L) Compared to TransFusion-L (L)**\n\nThank you for your question. To ensure fair comparisons with existing KD methods like S2M2-SSD and UniDistill, we chose CenterPoint as the LiDAR student model. CenterPoint inherently underperforms compared to TransFusion-L by 5.2 mAP and 7.1 NDS, largely because TransFusion-L utilizes a more advanced DETR head. However, as shown in the modified Table 1 attached to our global response, TransFusion-L operates nearly twice as slow as CenterPoint in terms of FPS, due to the time-consuming attention operations. By applying cross-modal KD to CenterPoint, while preserving its original fast inference speed, we brought its performance closer to that of TransFusion-L, thus achieving a more favorable balance between precision and real-time performance. Therefore, cross-modal KD does make sense for LiDAR-based student models.\n\n### **Q2: The inference speed comparison**\n\nThank you for your constructive feedback. As revised in the global response, we have added a comparison of the number of inference floating point operations(FLOPs) and required time for different models to Table 1. This adjustment allows for a clearer view of the performance-realtimeness tradeoff offered by knowledge distillation. Once again, we appreciate your suggestions. :)\n\n### **Q3: Incorporation of Temporal Information**\n\nThank you for your constructive questions. As clarified in the global response, our teacher model and LiDAR student models inherently incorporate multi-sweep LiDAR inputs, thus implicitly integrating temporal information. Inspired by your feedback, we conducted additional experiments with BEVFormer, a camera-based student model that explicitly incoporate temporal fusion operation, and observed performance gains, as detailed in the global response. As technologies for multi-modal explicit temporal fusion continue to mature and become open-sourced, we foresee the potential for integrating explicit temporal KD operations as a pluggable module into the VeXKD framework in future developments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N0Cfx8p8RW", "reviewer": "Reviewer_Vuwz", "summary": "The paper introduces a modality-general fusion teacher that narrows the gap between teacher and single-modal student models. This framework also includes a data-driven mask generation network that creates unique spatial masks for different feature levels and tasks. These masks enhance feature distillation by selectively transferring valuable information from the teacher’s feature maps.", "questions": "1. The related work section should be more comprehensive, and the experiments should compare with state-of-the-art methods.\n2. More experiments with different baselines mentioned above should be conducted to validate the effectiveness of the methods.", "limitations": "The paper has discussed its limitations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes the integration of cross-modal fusion and Knowledge Distillation in 3D perception, enhancing the efficacy of cross-modal KD through a modality-general fusion teacher.\n2. The paper develops a task- and modality-agnostic KD approach, making it highly versatile for any BEV-based 3D perception task and adaptable to various student modalities.", "weaknesses": "1. The baseline is too low. More baselines should be included to validate the effectiveness, such as BEVDepth [1] and long-term temporal fusion settings [2].\n2. The related work section is incomplete and does not include some state-of-the-art or similar prior works [3,4]. Additionally, the state-of-the-art work VCD [3] focusing on L+C->C is not compared in Table 1.\n3. The novelty is limited. The Masked Feature Distillation is actually proposed in FD3D [5]. \n\n\n[1] BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection\n\n[2] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection\n\n[3] Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection\n\n[4] BEVSimDet: Simulated Multi-modal Distillation in Bird's-Eye View for Multi-view 3D Object Detection\n\n[5] Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Q1: Experiment on More Baselines**\n\nThank you for your suggestion. We have conducted 3D object detection KD experiments on both BEVDepth and the temporally fused BEVFormer as student models. As shown in Table 1 attached to the global response, our KD framework was applicable to both models and improved their performance.\n\n### **2. Q2: Modified Related Work**\n\nThank you for your constructive suggestion and we have modified our related work. Due to space limitations and in order to avoid abusing the official comments, we abstract the modifications instead of posting the entire related work section.\n* **Adding different operations on utilization of feature distillation**\n\nDistillBEV[5] decomposes the region of feature maps and enhances attention to false positive regions. Recently, inspired by pretext tasks in large language models, masked generative distillation has been proposed [1]. Unlike attentive distillation, generative distillation masks part of the student’s feature map using a random mask. This masked map is then processed with a generator network to reconstruct feature maps that closely approximate the teacher’s, thus enhancing knowledge transfer. However, random masks can destabilize algorithms, especially in 3D object detection with pronounced foreground-background imbalance. Zeng et al. [2] address this by using a learned distillation head to predict coarse foreground boxes where random mask distillation is applied, focusing more on foreground regions to enhance stability. Our method, while inspired by masked generative distillation, aligns more closely with attentive distillation and eliminates the need for additional generator networks.\n\n* **The efficacy of KD**\n\nTo boost KD efficacy by minimizing the modality gap, Huang et al [3] developed a \"vision-centric\" multi-modal teacher, reducing reliance on LiDAR model operations to align more closely with camera-based students. Conversely, SimDistill[7] adds a branch to the student model to simulate multi-modal processing, narrowing the teacher-student gap but increasing the student model's size and inference time, deviating from traditional KD goals. Our work focuses on developing a modality-general fusion model without altering the existing pipelines of either teacher or student networks.\n\n* **More cross-modal research on camera-based students**\n\nRecent works have advanced short-term [8, 9] and long-term [4] temporal fusion in multi-camera 3D perception. Zheng et al.[6] use a long-term temporal fusion teacher to impart temporal cues to a short-term memory camera student, while work [3] warps time series ground truth into the current timestamp for long-term temporal supervision.\n\n### **3. W3: Novelty in Masked feature distillation**\n\nThank you for your thought-provoking comment. Indeed, our work shares some similarities with FD3D, as both are inspired by the work Masked Generative Distillation [1].\n\nBoth [1] and FD3D [2] are generative distillation in nature, where certain feature positions are masked and reconstructed from other feature locations to achieve KD. FD3D further generates bounding boxes through a learned distillation head, ensuring that only features within the bounding box are involved. In contrast, our method utilizes masks to selectively choose features to align pixel-wise akin to attentive distillation. This requiring only an auxiliary network for generating spatial masks, thereby eliminating the need for additional complexities of reconstruction.\n\nOur approach also differs in mask generation: FD3D is instance-based, using BEV queries to guide distillation within coarse bounding boxes. We have adapted our method to address the semantic complexities of the BEV space, initializing and learning dense BEV queries for fine-grained, pixel-wise mask generation that interacts directly with both student and teacher feature maps, independent of bounding boxes. This allows for greater generalization across various downstream tasks besides object detection.\n\nWhile both our method and FD3D utilize learned masks to identify useful feature locations and filter out noisy ones, FD3D focuses primarily on model compression in single-modality scenarios. Our study, however, considers cross-modal KD scenarios, including how to incorporate more modality-general information within the teacher and performing attentive distillation of this fine-grained modality-general information through a collaborative mask learning process involving student and teacher.\n\nIn summary, compared to the mentioned work, the novelties of our BEV guided masked learning and distillation include:\n\n1. Focusing on attentive over generative distillation.\n\n2. Our method generates masks based on interactions between the learned dense BEV queries and feature maps of both student and teacher, enhancing finegrained feature extraction and adaptability to various tasks.\n\n3. Our method explores cross-modal KD scenarios, aiming to enhance the teacher model with broader modality-general information and in return utilize the information contained in the learned BEV queries to perform mask guided selective attentive distillation.\n\n**References:**\n\n[1] Masked generative distillation\n\n[2] Distilling Focal Knowledge From Imperfect Expert for 3D Object Detection\n\n[3] Leveraging Vision-Centric Multi-Modal Expertise for 3D Object Detection\n\n[4] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection\n\n[5] Distillbev: Boosting multi-camera 3d object detection with cross-modal knowledge distillation\n\n[6] Distilling temporal knowledge with masked feature reconstruction for 3d object detection\n\n[7] Simdistill: Simulated multi-modal distillation for bev 3d object detection.\n\n[8] Bevdet4d: Exploit temporal cues in multi-camera 3d object detection\n\n[9] Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for your constructive comments and feedback, which have been invaluable in allowing us to improve our work.

We appreciate the recognition from all reviewers of the main contributions of this paper, including the **versatility** of the proposed knowledge distillation (KD) framework, which is both modality- and task-agnostic. Additionally, **the integration of cross-modal KD and fusion** to enhance the efficacy of the teacher in the KD process has been acknowledged(GBji, uk3B, Vuwz). Furthermore, the use of the **masked distillation module** to create unique masks for different feature maps, thereby facilitating the thorough mining and transfer of useful information contained in the teacher's feature map, has been positively noted (dGF9, uk3B, Vuwz).

In response to the constructive suggestions provided, we have adopted and made corresponding adjustments during the rebuttal period.

### **1. Addition of GFLOPs and Inference Speed Column to Table 1:**
Thanks to the constructive suggestion raised by reviewer uk3B regarding the inference computational resources and time required for different models, we utilized the open-source tool calflops to calculate the giga floating-point operations (GFLOPs) needed by all models mentioned in the comparative experiments during inference. We also conducted and compared inference time statistics on a commonly used GPU RTX 4090, and have included these results in the comparative experiments table attached. This addition more clearly illustrates the performance-realtimeness tradeoff brought about by the proposed cross-modal KD on the student model.

### **2. Revision of the Related Work Section on Knowledge Distillation:**
Thanks to the excellent papers mentioned by reviewer uk3B. We have thoroughly reviewed the recent literature and updated the related work section on knowledge distillation. The revised section now offers a more comprehensive overview of the latest developments in knowledge distillation, with a specific emphasis on cross-modal applications. This update ensures that our paper comprehensively reflects the current state of the field and its recent advancements, providing readers with a clearer understanding of the evolution and research gaps in cross-modal KD.

### **3. Clarification of Temporal KD and Additional Experimental Results on Temporal Camera Students:**
In response to the comments regarding temporal KD, we would like to clarify the integration of temporal information. The teacher model used in our cross-modal KD inherently incorporates temporal information. Here is a more detailed analysis:

* The LiDAR data pipeline naturally integrates multiple sweeps, embedding temporal information at the input level. This means that even without explicit temporal operations, our multi-modal teacher model—and LiDAR-only student models like CenterPoint—already leverage temporal cues.

* This is why influential work explicitly engaging in temporal fusion operations is predominantly focused on the camera branch, including BEVFormer[1], FB-BEV[2], PETRv2[3] and so on. To assess our framework's impact on multi-camera students employing explicit temporal fusion, we conducted supplementary experiments using BEVFormer as the student model on the nuScenes val set during the rebuttal period. The results in the attached Table 2 demonstrate that the implicit temporal information in the teacher model, along with modality-general information, enhances student model performance. These results are detailed in the attached PDF.

* The field of BEV perception is rapidly evolving, with recent projects like BEVFusion4D[4] and FusionFormer[5] starting to explore explicit multi-modal temporal fusion. However, these projects are not yet open-sourced, complicating their use as foundational teacher models for guiding various student modalities. The field of multi-camera perception is introducing diverse explicit temporal fusion operations at different feature levels, such as BEV-based, proposal-based, and query-based methods. Exploring the commonalities among those temporal fusion approaches could be challenging and require further study. As multi-modal temporal fusion research advances and becomes open-sourced, integrating explicit temporal KD operations as pluggable components into the VexKD framework could represent a promising direction for future research.

Please see the attached PDF with the modified tables and added experimental results and the reviewer-specific rebuttals for more information. Finally, I would like to extend my gratitude once again to all the reviewers for their valuable feedback and suggestions.

**Reference**

[1] Li, Zhiqi, et al. "Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers." European conference on computer vision. Cham: Springer Nature Switzerland, 2022.

[2] Li, Zhiqi, et al. "Fb-bev: Bev representation from forward-backward view transformations." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Park, Jinhyung, et al. "Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection." The Eleventh International Conference on Learning Representations. 2022.

[4] Cai, Hongxiang, et al. "BEVFusion4D: Learning LiDAR-Camera Fusion Under Bird's-Eye-View via Cross-Modality Guidance and Temporal Aggregation." arXiv preprint arXiv:2303.17099 (2023).

[5] Hu, Chunyong, et al. "FusionFormer: A Multi-sensory Fusion in Bird's-Eye-View and Temporal Consistent Transformer for 3D Objection." arXiv preprint arXiv:2309.05257 (2023).

Meta-review: All reviewers found the studied problem to be important, the proposed method to be effective, and the results promising. The rebuttal successfully addressed reviewer comments and all reviewers recommend acceptance. The authors are encouraged to improve the final paper version by following reviewer recommendations.
Decision: Accept (poster)