AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking
Authors: Shiqi Sun
Yantao Lu
Ning Liu
Bo Jiang
Jinchao Chen
Ying Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Network Pruning; Camera-LiDAR fusion models; Perception of Autonomous Driving;
Primary area: optimization_for_deep_networks

Abstract: Camera-LiDAR fusion models significantly enhance perception performance in autonomous driving. The fusion mechanism leverages the strengths of each modality while minimizing their weaknesses. Moreover, in practice, camera-LiDAR fusion models utilize pre-trained backbones for efficient training. However, we argue that directly loading single-modal pre-trained camera and LiDAR backbones into camera-LiDAR fusion models introduces similar feature redundancy across modalities due to the nature of the fusion mechanism. Unfortunately, existing pruning methods are developed explicitly for single-modal models, and thus, they struggle to effectively identify these specific redundant parameters in camera-LiDAR fusion models. In this paper, to address the issue above on camera-LiDAR fusion models, we propose a novelty pruning framework Alternative Modality Masking Pruning (AlterMOMA), which employs alternative masking on each modality and identifies the redundant parameters. Specifically, when one modality parameters are masked (deactivated), the absence of features from the masked backbone compels the model to reactivate previous redundant features of the other modality backbone. Therefore, these redundant features and relevant redundant parameters can be identified via the reactivation process. The redundant parameters can be pruned by our proposed importance score evaluation function, Alternative Evaluation (AlterEva), which is based on the observation of the loss changes when certain modality parameters are activated and deactivated. Extensive experiments on the nuScene and KITTI datasets encompassing diverse tasks, baseline models, and pruning algorithms showcase that AlterMOMA outperforms existing pruning methods, attaining state-of-the-art performance.

Alter.MOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking
Shiqi Sun Yantao Lu Ning Liu Bo Jinchao Chen', Ying Zhang' Jinchao
Department Computer Science, Northwestern Polytechnical University
Midea Group
3 Didi Chuxing
(shiqisun, yantaolu, cjc, ying_zhang) @nwpu. edu. cn
ningllu1220@gmaii.ccom boj wtw.hhooo@@gaaa.ccom
Abstract
Camera-LiDAR fusion models significantly enhance perception performance autonomous driving. The fusion mechanism leverages the strengths of each modal- ity while minimizing their weaknesses. Moreover, in practice, camera-LiDAR fusion models utilize pre-trained backhones for efficient training. However, we argue that directly loading single-modal pre-trained camera and LiDAR backbones into camera-LiDAR fusion models introduces similar feature redundancy across modalities due to the nature of the fusion mechanism. Unfortunately, existing pruning methods developed explicitly for mnnglemodal models, and thus, they struggle to effectively identify these specific redundant parameters in camera- LiDAR fusion models. In this paper, to address the issue above camera-LiDAR fusion models, we propose a novelty pruning framework Alternative Modality Masking Pruning (AlterMOMA), which employs alternative masking on each modality and identifies the redundant parameters. Specifically, when one modality parameters masked (deactivated), the absence of features from the masked backbone compels the model to reactivate previous redundant features of the other modality backbone. Therefore, these redundant features and relevant redundant parameters can be identified via the reactivation process. The redundant parameters can be pruned by our proposed importance score evaluation function, Alternative Evaluation (AlterEva), which is based on the observation of the loss changes when certain modality parameters are activated and deactivated. Extensive experiments on the nuScenes and KITTI datasets encompassing diverse tasks, baseline models, and pruning algorithms showcase that AlterMOMA outperforms existing pruning methods, attaining state-of-the-art performance.
Introduction
Camera-LiDAR fusion models are prevalent in autonomous driving, effectively leveraging the sensor properties, including the accurate geometric data from LiDAR point clouds and the rich semantic context from camera images [ 21, providing a more comprehensive understanding of the environment [3 4. However, the exponential increase in parameter counts due to fusion architectures introduces significant computational costs, especially when deploying these systems on resource-
"Joint first authorship. Fither author can he cited first. CCoresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Backward Propagation camera branch only
High finetune gradients due to the loss LDAR information
Task loss
Normalized Camera Gradients
LIDAR Input
backbone
UDAR feature
Low finetune gradients since the amendment LIDAR information
Backward Propegation with both camera and LIDAR branches
The corresponded pretrained parameters are redundant due the modality fusion
Camera Input
Camera feature
Figure 1: Motivating example of fusion-redundant features in the 3D object detection task. We employ backward propagation on camera-LiDAR fusion models with pre-trained backbones to observe the gradient difference (features utilization) between with camera backbone only and with both the camera and LiDAR backbone. Notably, certain pre-trained parameters in the camera backbone redundant due the amendment of LiDAR information. It reveals that similar feature extraction exists across modalities, which introduces additional redundancy when camera-LiDAR fusion models directly loads single-modal pre-trained backbones.
constrained edge devices, which a crucial challenge for autonomous driving [5] Network pruning is one of the most attractive methods for addressing the challenge above of identifying and eliminating redundancy in models. Existing pruning algorithms target single-modal models 66 7
g 9
10 [2]
or multi-moda models that merge distinct types of data [3 [4,, such as visual and language inputs. However, it's important to note that directly applying these algorithms to camera-LiDAe fusion models can lead to significant performance degradation. The degradation main factors that existing pruning methods overlooked: 1) the key fusion mechanism specific to vision sensor inputs within models, and 2) the training scheme where models typically load single-modal pre-trained parameters onto each backbone 22 [5.. Specifically, since single-modality models lack the modoliyy fusion mechanism, existing pruning algorithms traditionally do not consider inter- modality interactions. Furthermore, because the pre-traines backbones (image or LiDAR) are trained separately, they are not optimized jointly, exacerbating the redundancy in features extracted from each backbone. Though leveraging pre-trained backbone improves the training efficiency compared with models training from scratch, we argue that directly loading single-modal pre-trained camera and LiDAR backbones into camera-LiDAo fusion models introduces similar feature redundancy across modalities due the nature of the fusion mechanism.
be reasoned for two
In detail, since backbones are independently pre-trained on single-modal datasets, they extract features comprehensively. which leads to similar feature extraction across modalities. Meanwhile, the fusion mechanism selectively leverages reliable features while minimizing weaker ones across modalities to enhance model performance. This selective utilization upon similar feature extraction across modalities introduces the additional redundancy: Each backbone independently extracts similar features, which subsequent fusion modules will not potentially utilize. For instance, both camera and LiDAR backbones extract geometric features to predict depth during pre-training. However, geometric features extracted from the LiDAR backbone are considered more reliable during fusion because LiDAR input data contain more accurate geometric information than the cameras, e.g., object distance, due to the physical properties of sensors. Consequently, this leads to the redundancy of geometric features of the camera backbone. In summary, similar feature extraction across modalities, coupled with the following selective utilization fusion modules, leads to two counterparts of similar features across modalities: those utilized by fusion modules in one modality (i.e., fusion fusion-contributed) and those that are redundant in the other modality (i.c., fusion- redundant). We also illustrate the fusion-redundant features in Figure
To address the above challenge, we propose a novel pruning framework AlterMOMA, specifically designed for camera-LiDAR fusion models to identify and prune musion-redundan parameters. AlterMOMA employs alternative masking on each modality, followed by observing loss changes when certain modality parameters are activated and deactivated. These observations serve as important indications to identify fusiontrreunddnt parameters, which are integral to our importance scores evaluation function, AlterEva. Specifically, the camera and LiDAR backbones are alternatively masked. During this process, the absence of fusion-contribbuted features and relevant parameters in the masked (deactivated) backbone compels the fusion modules to reactivate their fusion-redundant

---

counterparts from the other backbone. Throughout this reactivation, changes in loss are observed as indicators for contributed and fusion-redundant parameters across modalities. These indicators are then combined in AlterEva to maximize the importance scores of contributed parameters while minimizing the scores of fusion-redundant parameters. Then, parameters with low importance scores will be pruned to reduce computational costs.
To validate the effectiveness of our proposed framework, extensive experiments are conducted on several popular 3D perception datasets with camera and LiDAR sensor data, including nuScenes [[6] and KITTI [7] These datasets encompass a range of 3D autonomous driving tasks, including 3D object detection, tracking, and segmentation.
The contributions of this paper effectively compress camera-LiDAA fusion models 2) we propose an importance score evaluation function AlterEva, which identifies fusion-redundant features and their relevant parameters across modalities 3) we validate the effectiveness of the proposed AlterMOMA on nuScenes and KITTI for 3D detection and segmentation tasks.
follows: 1) We propose pruning framework AlterMOMA to
2 Related Work
Camera-LiDAR Fusion. With the advancement of autonomous driving technology, the efficient
fusion of diverse sensors, particularly cameras and LiDARs, has become crucial [18 19]. Fusion
architectures can be categorized into three types based on the stage of fusion within the learning
framework: early fusion [20, 21], late fusion [22], and intermediate fusion 2 [5]. Current state-
of-the-art (SOTA) fusion models evolve primarily within intermediate fusion and combine low-level machine-learned features from each modality to yield unified detection results, thus significantly enhancing perception performance compared with early late fusion. Specifically, camera- LiDAR fusion models focus on aligning the camera and LiDAR features through dimension projection at various levels, including point 233 voxel (24, and proposal 3]. Notably, the SOTA fusion paradigm aligns all data to the bird's eye view (BEV)
15
25 26 has gained traction as an effective
approach maximize the utilization of heterogeneous data types.
Network Pruning. Network pruning effectively compresses deep models by reducing redundant parameters and decreasing computational demands, Pruning algorithms have been well-explored for 30 31 32], focusing on evaluating importance scores to identify and remove redundant parameters or channels. These scores are based on data attributes [7]] 33,, weight norms [34] 35), or feature map ranks (88). However, single-modal pruning algorithms are not suited for the complexities of camera-LiDAR fusion models. While some multi-modal pruning algorithms exist [13] [4]. they are mainly designed for models combining different data types like language and vision. Therefore, there is pressing need for pruning algorithms specifically devised for camera-LiDAL fusion models. From the perspective of granularity, pruning algorithms can be divided into two primary categories: 1) structured pruning, which entails removing entire channels rows from parameter matrices, and 2) unstructured pruning, which focuses on eliminating individual parameters. For practical applications, we have adapted our method support both types of pruning.
single-modal perception tasks [2 28 29
3 Methodology
3.1
Preliminaries
We firstly review some basic concepts including camera-LiDAs fusion models and pruning formula- tion. Gamera-LiDAs fusion models consist of 1) LiDAR feature extractor F, to extract features from point cloud inputs, 2) a camera feature extractor F, to extract features from image inputs, 3) the fusion module and following task heads F, to get the final task results. The parameters denote as 0= Oi, Oc. 0, for LiDAR backbone, camera backbone, and fusion and task heads, respectively. Take camera backbone for instance, A₂ {0), 82. NN.) denotes all weights in the camera backbone, where N₂ represents the total number of parameters in camera backbone. Therefore, for the LiDAR input X, and camera input X., the training process models could be denoted as
arg min C(Y,F,(0,: F (0g X.), Fo X.)): Be.ct
(I)
where Y denotes the ground truth, and L represents the task-specific loss functions.
3

---

Modality Masking Alternatieely mask one of backberws [LiDAR or Camera)
Redundancy Reactivation Reactvvte fusion-redundont parameters by training with batches
Importance Evaluation
Pruning AlterEvo Parameters with low importance scores will be pruned, then finetuned.
Altertia updete importance scores with two indictors: DeC RoA!
DeCl
Comeera LIDAR Fusion Models After Pruning
Altertua
importance scores
updete
moxk
mask
Comera LIDAR
Relnttiolization & Attemative Mosking
LiDAR backbone
Fuston Models
mask
Camera backbone
Refore Pruning
importance
scores
updote
task heeds
Output Results
Relti
Importance Evaluation
DeC1
Framework
AlterEva
Workflow
Figure 2: Overview of the AlterMOMA The framework begins with Modality Masking, where one of the backbones is imitially masked. This step is followed by Redundancy Reactivation and Importance Evaluation, where the parameter importance scores are initially calculated with AlterEva. Afterward, the models undergo Reinitialization and Alternative Masking of the other backbone, leading another round of Redundancy Reactivation and Importance Evaluation. When scores of all parameters in backbones are calculated fully with AlterEva (detailed Section (.3), models are pruned remove parameters with low importance scores and then finetuned. Notably, we use black lines to represent parameters of models and red lines to represent reactivated fusion-redundant parameters. The thickness of these lines indicates the contribution of parameters.
Importance-basee pruning typically involves using metries to evaluate the importance scores of parameters or channels. Subsequently, optimization methods are employed to prune the parameters with lower importance scores, that are nonessential within the model. For the camera-LiDAR fusion models, the optimization process can be formulated as follows:
N
N:
arg maN 95
Σ iE(
Σ
&. SO))) S.I. Σ
2
(2)
iE(l...))==
where de is an indicator which is if 0 will be kept or 0 if 07 is to be pruned. S is designed to measure the importance scores for parameters, and l represents the kept parameter number, where (1 p) -ieetee. N; with the pruning ratio p.
3.2 Overview of Alternative Modality Masking Pruning
Similar feature extraction across modalities, coupled with the selective utilization of features in the following fusion modules introduce redundancy in camera-LiDAR fusion models. Therefore, similar features and their relevant parameters can be categorized into two counterparts across modalities: those that contribute to fusion and subsequent task heads (fusion-contributte) redundant muision-redundant) In this section, we propose the pruning framework AlterMOMA, which alternatively employs masking on camera and Lidar backbones to identify and remove the fusion-redundan parameters. AlterMOMA is developed based on a novel insight: "The abxence of fusion-contributee features will compel fusion modules to 'reactivate' their fusion- fusion-reduunen redundant counterparts as supplementary, which, though less effective, are necessary to maintain functionality: " For instance, if the LiDAR backbone is masked, the previously cusion-contributed geometric features it provided are absent. To fulfill the need for accurate position predictions, the model still needs to process geometric features. Consequently, the fusion module is compelled utilize the geometric features from the unmasked camera backbone, which were previously fusion- redundant. We refer to this process Redundancy Reactivation. By observing changes during this Redundancy Reactivation, fusion-redt redundant parameters can be identified. The overview of AlterMOMA is shown in Figure and the detailed steps are in Algorithm of Appendix [ The key steps are introduced follows:
and those that are
Modality Masking. Three binary masks are denoted as μu, Pes and J E (0, 1), correspond to the parameters applied separately on the LiDAR backbone, the camera backbone, and the fusion and

---

tasks head. Our framework begins by masking either one of the camera backbones or the LiDAR backbone. Here we take masking the LiDAR backbone as the illustration. The masks are with μ Me = 1, 1i = 1. The camera backbone will be masked alternatively.
Redundancy Reactivation. To allow masked models to reactivate fusion-redundant parameters, train masked models with batches of data. Specifically, B batches of data D,, i E {1, ..., B3 are sampled from the multi-modal dataset D.
Importance Evaluation. After Redundancy Reactivation, the importance scores of parameters the camera backbones are calculated with proposed importance score evaluation function AlterEva detailed in Section 3.3 Since fusion modules need to consider the reactivation of both modalities, the importance scores of parameters in the fusion module and task heads will be updated once the importance scores of both the camera and Lidar backbones' parameters are calculated.
Alternative Masking. After Importance Evaluation of camera modality, models will reload the initialized parameters, and then the other backbone will alternatively be masked, with 14 = 1, Me 0, Ms 1. Then the step Redundancy Reactivation and Importance Evaluation will be processed again to update the importance scores of parameters in the LiDAR backbone and the fusion module.
Pruning with AlterEva. After evaluating the importance scores using AlterEva, parameters with low importance scores are pruned with global threshold determined by the pruning ratio. Once the pruning is finished, the model is fine-tuned with the task-specific loss, as indicated in Eqn.
3.3 Alternative Evaluation
In this section, we will detail the formulation of our proposed AlterEva, which consists of two distinct indicators evaluate the parameter importance scores. As outlined in section 3.2 the importance scores are alternatively calculated with AlterEva in Importance Evaluation. Then, parameters with low importance scores are removed in the pruning process. The goal of AlterEva is to maximize the scores of parameters that contribute to task performance while minimizing the scores of rediudand parameters. To achieve this, AlterEva incorporates two key indicators: 1) Deactivated Contribution Indicator (DeCI) evaluate the parameter contribution to the overall task performance of the fusion models, 2) Reactivated Redundancy Indicator (ReRI) identifies fusion-redundant parameters across both modalities. Since changes in loss can directly reflect the parameter contribution difference to task performance during alternative masking, both indicators are designed based on the observation of loss decrease or increase, when certain modality parameters are activated or deactivated. Specifically, take parameters in the camera backbone as an instance, DeCI observe the loss increases with masking camera backbone itself, while ReRI observe loss decrease with masking LiDAR backbone and reactivating camera backbone via Redundancy Reactivation. Formally, we formulate the loss for the fusion models with masks. With three binary masks and the dataset defined in Section 3.2 the loss is denoted as follows, by simplifying some of the extra notations used in Eqn. !
Lm (Her Mi, ue;; D) L(u c 8r, Me O Per 2y 3 by; D).
(3)
For brevity, we assume [4 = 1, 14 = 1, and j = 1, and we only specify in the formulation when mask is zero. For example, (14c 0; D) indicates that μ 0, Jyy and 14 1. Since the alternative masking is performed on both backbones, we illustrate our formulation by calculating two indicators for parameters in the camera backbone.
Deactivated Contribution Indicator. If a parameter is important and contributes to task performance, deactivating this parameter will lead to task performance degradation, which will be reflected in an increase in loss. Therefore, to derive the contribution of the i-th parameter 0: of the camera backbone, observe the changes in loss when this parameter is deactivating via masking, denoted follows:
0g (cm)(: (n. O;D)),
(4)
where μن represents the mask for 8% and Φøz denotes the indicator DeCI for 6:. However, the total number of parameters is enormous, deactivating and evaluating each parameter independently are computationally intractable. Therefore, we design an alternative efficient method to approximate the evaluation Eqn. 4 by leveraging the Taylor first-order expansion inspired by (36.. We first observe the loss changes 1: D) - L. (1te 0; D) by deactivating the entire camera backbones. Then, the first-order approximation of evaluation in Eqn. is calculated by expanding the loss change in each individual parameter 0% with Taylor expansion, considering 0_ = (0).... ....... 0N+.. This method
5

---

allows us to estimate the contribution for each parameter, denoted as follows:
Φ (m pà 0¹ dL (; D)
L. (Me 0; pi c 0: DLm (mc ( = 0; |)
(5)
:0:
00%
When 0g is deactivating with pe = 0, पº= 0 for i (I....,NN which means that the last term of Eqn. is zero. Meanwhile, when considering importance scores global scale, the Lm D) and Cm [r = 0; D) can be treated as constant for all 8:. Thus the first term and the third term can be disregarded. Therefore, the final indicator of each parameter's contribution, represented by our DeCI, can be expressed as follows:
0Cm(; ae:
60 0
(6)
This formulation enables tractable and efficient computation without Modality Masking of the camera backhone itself, achieved by performing a single backward propagation in the Importance Evaluation with initialized parameters.
Reactivated Redundancy Indicator. As discussed Section 3.2 the identification of fusion- redundant parameters relies on understanding of the fusion mechanism: when fusion-contributed features from the LiDAR backbone are absent due to masking, the previously fusion-redundant counterparts and their relevant parameters from the camera backbone will be reactivated during the Redundancy Reactivation. Therefore, to reactivate and identify fusion-redundan parameters in the camera backbone, the Modality Masking of the LiDAR backbone (n4 = and Redundancy Reactivation are processed first. Throughout this process, the loss evolves from L.. (1 = 0; Di) to fm (M 0: DB), and the parameters evolve from 8.,0 (i.e. 0.) to 0,, Similar to the formulation of DeCI, we observe the decrease in loss during Redundancy Reactivation and refer to this observation as our ReRI, denoted as follows:
L Cm (n = 0; D) (m = 0; D; + + C.. - (m = 0; PB)
Cm (14: D) L. (14 00 D2)).
Specifically, this process is designed to identify parameters that contribute to the task performance of models with the masked LiDAR backbone, highlighting those that are fusion-redundant, Since we want to observe reactivation rather than parameters updating of this masked model across training batches, apply the first-order Taylor expansion to the initial i-th parameters 0% denoted
8 = [6m = 0; D) + uhe 2 00,0 dc (pr 0: D)
00
(8)
0Lm (m = =0:
L. (mu 0; Da) nh @ 0%
08%
To derive the gradient on initial parameters 22,0 of the last term, we could use the chain rule and write out based on the gradient of the last step,
3Lm (I 0; DB).
00 DCm (MI DB)
B
a0
0e,0 (p 0, 0,De)
00,-
(9)
00 a0%
80..
00 00
00',8
According to the Proposition in the Appendix A this approximation is reached by dropping some small terms with sufficiently small learning rates (]. Since 0 is activating with μ and the
C. (pt = 0; D) and Lm (Re = 0: DB) can be treated as constant for all 0r, we could denote our final formulation by simplifying Eqn. [ and denoted as follow:
t 0 = - aLm (Hu = 0; D)
aCm(pu = 0; DR)
(10)
00g
00,
where 0% is the 02 and وyg represent the ReRI for 0g.
To the goal of parameters with significant contributions maintaining high importance scores while those identified as reduudate are assigned lower scores, AlterEva calculate the final importance scores by subtracting ReRI from DeCI. Since DeCI of parameters in the camera backbone could be calculated without masking the camera backbone itself, DeCI and ReRI of camera parameters could be calculated the same alternative masking stage (with LiDAR backbone masking), which simplifies the process of our framework AlterMOMA. Therefore, with the combination Eqn. aand Eqn. 10 the AlterEva of the camera backbones could be presented with the normalization:
g1
g5
S(A)) a
B
(11)
Σ
No
$ 20
Σ
= 2

---

where a and 8 are the hyper parameters and S(t)) represent the importance score evaluation function AlterEva for 0- Similarly, the AlterEva of parameters in the LiDAR backbones (i.e. f1) and in the fusion modules (i.e. 0g could be derived as:
D
S(O))
3
(12)
Σ $
Ni
Σ j
j ==
81
oi
4 0j
3
L (1 = 0)
3
oz (Me = 0)
s(0))
(13)
1
2
Σ N
of (+t = 0) 2
Σj= 20g (Her = 0)
==
==
where T0g and Φos (Jec 0) is calculated when camera backbone is masking, while B (M 0)
calculated with LiDAR backbone masking. AlterEva can efficiently calculate importance scores with backward propagation, enhancing the tractability of AlterMOMA. For brevity, we omit the derivations related to parameters in the LiDAR backbone and fusion modules, but additional details available in Appendix B and Appendix C
4
Experimental Results
4.1 Baseline Models and Datasets
To validate the efficacy of our proposed framework, empirical evaluations were conducted on several camera-LiDAR fusion models, including the two-stage detection models AVOD-FPI [3] as well as the end-to-end architecture based on BEV space, such as BEVfusion-ma mit [13] and BEVfusion-pee For AVOD-FPN, the point cloud input is processed using a voxel grid representation, while all input views are extracted using a modified VGG-16 [37]. Notably, the experiment on the AVOD-FPN demonstrates the efficiency of AlterMOMA on two-stage models, although this isn't the SOTA fusion architecture for recent 3D perception tasks. Current camera-LiDa' fusion models are moving towards a unified architecture that extracts camera and LiDAR features within BEV space. Thus, our primary results focus on BEV-based unified architectures, specifically BEVfusion-mit [15] and BEVfusion- pku 22. We conducted tests using various backbones. For camera backbones, we included Swin- Trin-Traasformer (Swin-T) (38) and ResNet [39]. For LiDAR backbones, we used SECOND [18]
[2].
VoxelNet 440 and PointPillars 41
We perform our experiments for both 3D object detection and BEV segmentation tasks on the KITTI 17 and nuScenes [16), which are challenging large-scale outdoor datasets devised for autonomous driving tasks. The KITTI dataset contains 14,999 samples in total, including 7,481 training samples and 7,518 testing samples, with comprehensive total of 80,256 annotated objects. To adhere to standard practice, we split the training samples into a training set and validation set in approximately a 1:1 ratio and followed the difficulty classifications proposed by KITTI, involving easy, medium, and hard. NuScenes is characterized by its comprehensive annotation scenes, encompassing tasks including 3D object detection, tracking, and BEV map segmentation. Within this dataset, each of the annotated 40,157 samples presents an assemblage of six monocular camera images, adept at capturing a panoramic 360-degree field of view. This dataset is further enriched with the inclusion of a 32-beam LiDAR scan, amplifying its utility and enabling multifaceted data-driven investigations.
4.2 Implementation Details
We conducted the 3D object detection and segmentation experiments with MMdetection3D 42 on NVIDIA RTX 3090 GPUs To ensure fair comparisons, consistent configurations of hyperparameters were employed across different experimental groups. To train the 3D object detection baselines, we utilize Adam as the optimizer with learning rate of le-4. We employ Cosine Annealing as the parameter scheduler and set the batch size to 2. For BEV segmentation tasks, we employ Adam as the optimizer with a learning rate of le-4. We utilize the one-cycle learning rate scheduler and set the batch size to 2. The hyperparameters a and / in Section 3.3 are both set with 1. The baseline pruning
methods include IMP 43), SynFlow 44 SNIP
and ProsPr (9, with the hyperparameters
specified in their papers respectively.
7

---

Table 1: 3D object detection performance comparison with the state-of-the-art pruning methods on the nuScenes validation dataset. We list the mAP and NDS of models pruned by different approaches within 80%, 85%, and 90% pruning ratios. The two baseline models trained with Swin'T and VoxeINet backbone.
Baseline Model
BFViusionmmit
ViVisioon-puu
Sparsity
80%
85%
90%
80%
85%
90%
Metric
mAP
NDS
MAP
NDS
mAP
NDS
mAP
NDS
mAP
NDS
mAP
NDS
[oo Pruning]
67.8
70.7
66.9
70.4
IMP
59.3
66.8
51.2
59.2
42.7
50.4
57.3
65.9
49.8
57.7
40.7
47.2
SynFlow
63.2
67.9
56.9
64.1
49.3
58.7
62.4
67.1
55.4
63.2
47.6
57.1
SNIP
62.2
67.5
56.4
63.6
50.2
58.8
61.8
67.5
54.7
62.9
47.8
57.3
ProsPr
64.3
69.6
61.9
66.1
58.6
62.5
63.6
68.4
59.9
66.1
56.7
62.7
AlterMOMA (Ours)
67.3
70.2
65.5
69.5
63.5
66.7
66.5
70.1
64.2
68.1
62.3
66.0
Table 2: 3D object detection performance comparison with the state-of-the-art pruning methods on the KITTI Validation dataset on car class. We list the models pruned by different approaches within 80%, and 90% pruning ratios. The baseline model is AVOD-FPN architecture.
Sparsity
Task
AP-3D
AP-BEV
AP-3D
AP-BEV
Difficulty
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
Car NNo Pruning]
82.4
72.2
66.5
89.4
83.9
18.7
IMP
65.8
57.7
51.3
69.2
64.66
59.7
52.1
45.7
43.2
59.6
54.2
$1.7
SynFlow
60.2
48.1
67.6
64.4
SNIP
73.5
64.9
59.8
79.1
75.8
69.6
62.7
52.3
45.8
72.4
66.9
63.7
ProsPr
78.9
62.1
75.7
63.4
59.1
71,9
AllerMOMA (Ours)
80.5
70.2
63.2
87,2
81.5
77.9
77.4
68.2
62.3
85.3
79.9
75.8
4.3 Experimental Results on Unstructured Pruning
To evaluate the efficiency of AlterMOMA with unstructured pruning, we conduct experiments across multiple fusion architectures and datasets for 3D object detection and semantic segmentation. Specifically, to evaluate the efficiency of AlterMOMA on two-stage fusion architectures, we applied AlterMOMA to AVOD-FPN (3], using the KITTI dataset with the task of 3D detection. Besides, BEVfusion-mit 15 and BEVfusion-pku (2). as two representative camera-LiDAR fusion models with unified based architectures, are applied with AlterMOMA using the nuScenes dataset to validate the efficiency on both 3D detection and semantic segmentation tasks. Additionally, to validate the robustness of AlterMOMA with various backbones, we conducted experiments with alternative images and point backbone, including ResNet [39 and PointPillars [11.
3D Object detection on nuScenes with BEV-based fusion Architectures. The experimental results are presented in Table [ Note that baseline models are BEVfusion-mit trained with SwinT and VoxeINet backbone. As reported in Table [ single-modal pruning methods, including IMP, SynFlow, SNIP, and ProsPr, experience significant declines in accuracy performance. Even the ProsPr, considered the best-performing method among these modal pruning techniques, demonstrates the mAP decrease of 3.5% in accuracy at the 80% pruning ratio and 9.2% at the 90% pruning ratio on BEVfusion-min Conversely, the incorporation of our AlterMOMA yielded promising results. For example, comparing with the baseline pruning method ProsPr, AlterMOMA boosts the mAP of BEVfusion-mii by 3.0% (64.3% + 67.3%), 3.6% (61.9 (61.9% + 65.5%), and 4.9% (58.6% - 63.5%) for the three different pruning ratios. Similarly, A AlterMOMA obtains much higher mAP and NDS than the other four pruning basclines with different pruning ratios on BEVFusion- and BEVFusion-ple -ku..
3D Object detection on KITTI with the two-stage fusion architecture. To validate the efficiency of AlterMOMA on the two-stage detection fusion architecture, we conduct experiments with various pruning ratios on KITTI with AVOD-FPN architecture as the baseline. The experimental results are presented in Table 2 Specifically, Table 2 presents the results for the car class on the KITTI, detailing AP-3D and AP-BEV across various difficulty levels including easy, moderate, and hard. Existing pruning methods experience significant declines performance on different metrics of different difficulties. Even the best-performing method among single-modal pruning methods, ProPr, shows a decrease in AP-3D of 3.5%, 2.6%, and 4.4% in the easy, moderate, and hard difficulty levels, respectively, at the 80% pruning ratio. Conversely, the AlterMOMA has yielded
8

---

Table 3: BEV segmentation performance parison on the nuScenes validation dataset.
Table 4: 3D object detection performance with various backbones on the nuScenes validation dataset.
Sparsity 80%
85%
90%
mloU
mIoU
mloU
80%
85%
90%
Sparsity
[No Pruning]
61.8
mAP
mAP
mAP
IMP
53.2
51.8
49.9
[No Pruning]
53.7
SynFlow
56.5
55.3
53.1
IMP
47.1
43.3
37.8
SNIP
55.9
54.9
53.2
SynFlow
49.5
45.8
40.3
ProsPr
57.7
56.2
54.1
SNIP
49.7
45.5
41.2
AlterMOMA
60.7
59.2
57.7
ProsPr
50.1
47.5
44.1
AlterMOMA
51.7
50.6
48.3
Table 5: 3D object detection performance structure pruning on the nuScenes validation dataset.
ResNet101 + SECOND
Sparsity
mAP
NDS
GFLOPs(L %)
[No Pruning]
64.6
69.4
610.66
IMP-30%
60.8
67.2
428.7 (29.8)
ProsPr-30%
64.2
69.1
413.4 (32.3)
AlterMOMA-30
65.3
69.9
420.13 (31.2)
IMP-50%
57.6
65.2
297.39 (51.3)
ProsPr-50%
62.5
68.4
285.79 (53.2)
AlterMOMA-50%
64.5
69.5
264.42 (56.7)
promising results. For instance, comparing with the pruning method ProsPr, AlterMOMA enhances both the AP-3D and AP-BEV on AVOP-FPN by 3.2% (74.2% - 77.4% and 3.9% (81.2% - 85.3%) for easy difficulties at the 90% pruning ratio. Furthermore. AlterMOMA consistently outperforms the other four pruning baselines across various difficulties and pruning ratios on AVOD- FPN. The comprehensive experimental results on 3D object detection validate the effectiveness of our AlterMOMA across different camera-LiDAE fusion architectures.
3D Semantic Segmentation on nuScenes To validate the robustness of work, extend performance evaluation of AlterMOMA to the semantic-centric BEV map segmentation task. Note that baseline models are BEVfusion-ni trained with Swin'T and VoxeINet backbone. We use the nuScenes dataset and utilize the BEV map segmentation validation set. The pivotal evaluation metric for this task is the mean Intersection over Union (mIoU). with the experimental configuration detailed in the work by [15], we perform our evaluation on the BEVfusion-mit as shown in Table 3 We observed that existing pruning methods still meet a significant accuracy drop by 8.6% (IMP), 5.3% (SynFlow), 5.9% (SNIP), and 4.1 .1% (ProsPr) for the 80% pruning ratio. Alternatively, our proposed approach yields significant advancements in performance. Specifically, compared with the ProsPr, AlterMOMA achieves substantial enhancement by 3.0% (57.7% - 60.7%), 3.0% (56.2% - 59.2%), , and 3.6% (54.1 > 57.7%) for the three different pruning ratios. These results empirically prove the efficacy of our pruning algorithms when applied to the BEV map segmentation task.
Results Various Backbone Architectures comprehensively assess the efficacy AlterMOMA. we conducted experiments with alternative images and point backbones which will influence fusion. Specifically, we replaced the original VoxeINet backbone with PointPillar and the SwinT backbone with 10 on the architecture of BEVFusion-mi mit. As depicted in Table the results obtained from these experiments consistently demonstrate state-of-the-ar performance with various pruning ratios. Particularly noteworthy is the achievement of substantial 1.6%, 3.1% 16, and 4.2% improvement compared to the ProsPr baseline under the pruning ratio of 80%, 85% and 90%. This consistent improvement is observed across different pruning ratios, affirming the effectiveness of AlterMOMA with different backbones employed. These outcomes robustly demonstrate the general applicability of AlterMOMA to various backbone architectures.

---

4.4 Structure Pruning Results on 3D object Detection
To assess the efficacy of our proposed pruning approach in structure pruning, we conducted experi- ments with BEVfusion-mit models with ResNer101 as the camera backbone and SECOND as the LiDAR backbones. Specifically, We measured the performance the pruned networks with a similar amount of FLOP reductions and reported the number of FLOPs(" FLOPs((GFLOO GFLOPs'). As depicted in Table 5 the results obtained from these experiments consistently demonstrate state-of-the-e art performance with various pruning sparsities. Our evaluations 30% and 50% pruning sparsities reveal that Alter- MOMA not only maintains a competitive mAP and NDS but also achieves a substantial reduction in computational overhead. Notably, with the 30% pruning sparsities, AlterMOMA achieves 0.7% on mAP and 0.5% on NDS compared with the unpruned baseline models, which reveals that removing similar feature redundancy improves the efficiency of models. Specifically, compared with the ProsPr baseline, AOterMOMA achieves a substantial enhancement by 1.1 % (64.6% + 63.3%), and 2.0% (62.5% + 64.5%). for the two different pruning sparsities.
Discussion and Conclusion
Although our approach identifies similar feature redundancy in camera-LiDAR fusion models, it is limited the perception field. Extending it to other multi-modal models, such vision- language models, requires further research. Fusion modules across various modalities exhibit different functionalities. In multi-sensor fusion models (camera, LiDAR, and Radar), the focus is on supplemening and spatially aligning data by leveraging the sensors' physical properties, fusing low-level features. However, in models with disparate data types like vision and language, fusion modules focus on matching high-level semantic contexts. Therefore, AlterMOMA primarily addresses redundancy from supplementary functionality multi-sensor fusion perception architectures.
In this paper, we explore the computation reduction of camera-LiDad fusion models. A pruning framework AlterMOMA is introduced to address redundancy in these models. AlterMOMA employs alternative masking on each modality and observes loss changes when certain modality parameters activated and deactivated. These observations integral to importance scores evaluation function AlterEva. Through extensive evaluation, our proposed framework AlterMOMA achieves better performance, surpassing the baselines established by single-modal pruning methods.
Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (62106202. 62102316 62403386), and in part by the Key Research and Development Projects of Shaanxi Province (2024GXYYBXM-I 18, D024G-YBBM-25555 and the Aeronautical Science Foundation China (2033M033053003).
References
[1] Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V Le, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 17182-17191, 2022.
[2] Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. Advances in Neural Information Processing Systems,
2022.
[3] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In 2018 JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1-8. IEEE, 2018.
[4] Yanlong Yang, Jianan Liu, Tao Huang, Qing-Long Han, Gang Ma, and Bing Zhu. Ralibev: Radar and lidar BEV fusion learning for anchor box free object detection system. CoRR, abs/221 1.06108, 2022.
10

---

Computer [5] Duy Thanh Nguyen, Tuan Nghia Nguyen, Hyun Kim, and Hyuk-Jae Lee. A high-throughput and power-efficient fpga implementation yolo enn for object detection. IEEE Transactions on Very Large Scale Iniegration (VLSI) Systems, 27(8):1611-187) 2019.
[6] Ning Liu, Geng Yuan, Zhengping Che, Xuan Shen, Xiaolong Ma, Qing Jin, Jian Ren, Jian Tang, Sijia Liu, and Yanzhi Wang. Lottery ticket preserves weight correlation: Is it desirable or not? In Marina Meila and Tong Zhang, editors. Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 701 7011-7020. PMLR, 18-24 Jul 2021.
[7] Yang Sui, Miao Yin, Yi Xie, Huy Phan, Saman Aliari Zonouz, and Bo Yuan. Chip: Channel independence- paaeedddeeee-aedd pruning for compact neural networks, Advances Neural Information Processing Systems, 14:24604-24611 2021.
[8] Chaoqi Wang, Guodong Zhang, and Roger B. Grosse. Picking winning tickets before training by preserving gradient flow. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net 2020.
[9] Milad Alizadeh, Shyam A. Tailor, Luisa M. Zintgraf, Joost van Amersfoort, Sebastian Farquhar, Nicholas Donald Lane, and Yarin Gal. Prospect pruning: Finding trainable weights at initializa- tion using meta-gradients. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022.
[10] Jianhui Liu, Yukang Chen, Xiaoqing Ye, Zhuotao Tian, Xiao Tan, and Xiaojuan Qi. Spatial pruned sparse convolution for efficient 3d object detection. Advances in Neural Information Processing Systems. 35:6735-6748, 2022.
[11] Jinyang Guo, Jiaheng Liu, and Dong Xu. Jointpruning: Pruning networks along multiple dimensions for efficient point cloud processing. IEEE Transactions on Circuits and Systems for Video Technology, 2021.
[12] Dong Chen, Ning Liu, Yichen Zhu, Zhengping Che, Rui Ma, Fachao Zhang, Xiaofeng Mou, Yi Chang, and Jian Tang. Epsd: Early pruning with self-distillation for efficient model com- pression. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 11258-11266 2024.
[13] Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Unified and progressive pruning for compressing vision-language transformers. In International Conference on Machine Learning, pages 31292-31311. PMLR, 2023.
[14] Jiangmeng Li, Wenyi Mo, Wenwen Qiang, Bing Su, and Changwen Zheng. Supporting vision-language model inference with causality-pruning knowledge prompt. arXiv preprint arXiv:2205. / 100, 2022.
[15] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye s-eee view representation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 2774-2781. IEEE, 2023.
[16] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan. Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages Vision Foundation IEEE, 2020.
[17] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):121-12333 2013.
[18] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
[19] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 7652-7660, 2018
11

---

[20] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4604-4614 2020.
[21] Chunwei Wang, Chao Ma. Ming Zhu, and Xiaokang Yang. Pointaugmenting: Ccoss-modal augmentation for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 11794-1 1803, 2021.
[22] Su Pang, Daniel Morris, and Hayder Radha. Clocs: Camera-lidar object candidates fusion for 3d object detection. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1 10386- 10393. IEEE, 2020.
[23] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. In proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 172-181, 2023.
[24] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangj Fang. Qinhong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao. Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection. In Luc De Raedt, editor, Proceedings of the Thirty-Firss International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022. pages 827-833, orgai.org, 2022.
[25] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in seddeey--viw.. arXiv preprint arXi::2112.11990 2021.
[26] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-viewy representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
[27] Zi Wang, Chengcheng Li, and Xiangyang Wang. Convolutional neural network pruning with structural redundancy reduction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14913-14922, 2021.
[28] Mingbao Lin, Rongrong Ji, Yan Wang. Yichen Zhang, Baochang Zhang, Yonghong Tian, and Ling Shao. Hrank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CV conference on computer vision and pattern recognition, pages 1529-1538, 2020.
[29] Jinyang Guo, Jiaheng Liu, and Dong Xu. 3d-pruning: A model compression framework for efficient 3d action recognition. IEEF Transactions Circuits and Systems for Video Technology. 32(12):77178722 2022.
[30] Fang Yu, Kun Huang, Meng Wang. Yuan Cheng, Wei Chu, and Li Cui. Width & depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 3143-3151, 2022.
[31] Zihao Xie, Li Zhu, Lin Zhao, Bo Tao, Liman Liu, and Wenbing Tao. Localization-aware channel pruning for object detection. Neurocomputing, 403:400-408, 2020.
[32] Ning Liu, Xiaolong Ma. Zhiyuan Xu, Yanzhi Wang, Jian Tang, and Jieping Ye. Autocompress: An automatic dnn structured pruning framework for ultra-high compression rates. Proceedings of the AAAI Conference Artificial Intelligence, 34(04):4876-4883, Apr. 2020.
[33] Yaomin Huang, Ning Liu, Zhengping Che, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Guixu Zhang, Xinmei Liu, Feifei Feng, and Jian Tang. Cp3: Channel pruning plug-in for point- based networks. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 5302-5312 2023.
[34] Yang He, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning filter pruning criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2009-2018, 2020.
12

---

[35] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based connection sensitivity. In 7th International Conference on Learning Represen- lations, ICIR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[36] Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
[37] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Truck Proceedings, 2015.
[38] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[39] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[40] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the pages 4490 4499, 2018.
the IEEE conference on computer vision and pattern recognition,
[41] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[42] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao. Yu Xiong, Xiaoxiao L.i, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: / 90199006.... 155, 2019.
[43] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse. trainable neural networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[44] Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. Advances in neural information processing systems, 33:6377-6333 6389, 2020.
13

---

A Derivation of Eqn in Section 3.3
Proposition. For a camera-LiDAl fusion model with parameters 0. for the camera backbone and parameters 0t for the LiDAR backbone, we can mask one of the backbones using masks lt = 0.ror the LiDAR backbone and pe 0 for the camera backbone. Take the models with masking LiDAR backbone instance. With sufficiently small learning rate the masked model is trained with butches of data Da, E {1, 2..., B} sampled from the dataset DD We assume the parameters update from Dc,0 to Be and the loss changes from fun (wu 0) to L.n. (1u = 0; DD). Then we could get the equation denoted as:
0Lm (14 DB)
dLm (14 0,
8 00
6%,0
(14)
00:
08 c,B
Proof. We can extend the left side of the equation using the chain rule, denoted as,
0L (pr DDD
3hm (M1 0; Dg)
00 c 0'
00,0
00 c,0
00',
00
0Lm (mu 0; D;D
00 c
00 0,2
00
(15)
00'
06 06
00. c,
00
c.0
2Cm (MI 0; D;D
00
0 0.0
00' c,B
00
Due to the updates with the learning rate can represent 02g as follows:
0% = 02-j11 - @Cm(H 01
(16)
00:j-1
Combining with Eqn. 15 and Eqn. 16 we could derive as following:
0L (pm 0; Dr)
0r 0Lm (w 0; DR)
00
II
c.j
g:
00%00
O6
09
cjj-1
0Lm (14 = 0: DD)
II
B
00
c.j-1
06m(Hi (M::D
36 36'
0
(17)
08:
00
j=1
0h (11
0: DD)
B
L (11 0; 0;
0
08: B,
06 a-j- )
where I represents the identity matrix, and 8 represents the second-order derivative. By dropping the terms with the sufficiently small learning rate inspired by 19, the approximation is as follows:
0L (W = =D:
2Lm @mm(4:00 DB)
8 Lm (pr 0; D;
00
0
00:
20'nn
0(03
1=1
dCm (1I = DB)
B I
I
0r
(18)
00
-=1
0Cm -,)
0 00
20',B
And we finally prove that
2L (11 = =0; OD:D))
dL (MI
0, 0,,
(19)
00
00
14

---

B Detailed AlterEva for parameters in the LiDAR backbone
Due to the page limits, in Section we only introduce the detailed formulation of the camera backbone. In this section, we will complete the details of the AlterEva of parameters from the LiDAR backbone, with two indicators DeCI and ReRI.
Deactivated Contribution Indicator. Similar to the parameters of camera backbones, the loss changes when parameters deactivating via masking is observed, denoted follows,
ΦØ. (: - L. (ni 0; D)).
(20)
Due to the enormous number of parameters, we generalize this deactivating approach to encompass the entire LiDAR backbone. We then denote the new evaluation function as follows:
L.. (m - L. (14 0; D)}.
(21)
Then, to accommodate the need for differentiation among various parameters, we apply the Taylor first-order expansion to be on each individual parameter 0f in the camera backbone, considering 0 = (0) ..... ON
(Lm(; (m 0: μ a of 8Lm (pu 0,
Oo; (: + Mi 0f
(22)
00;
00;
Since i when 0j is deactivating, and constant loss values can be disregarded when considering importance scores on a global scale, the final indicator of each parameter's contribution, represented by our DeCI, can be expressed as follows:
0L D)
6 = 1
(23)
08i
Reactivated Redundancy Indicator. Similarly, aiming to reactivate fusion-redundee redundant parameters in the camera backbone, the camera backbone will first be masked with #4e = 0 and then masked models are trained with sampled batches. Throughout this process, the loss evolves from (Hte = 0; D1) to (Mee 0; D and the parameters evolve from 01,0 (i.e. 02) to 01,B- The loss changes and parameter differences are observed via our ReRI, denoted as follows:
So.- 1. D; L. (Je 0; Di) + Lm DR-1) Cm(u Da)l
(24)
= [m = D) - L. = 0; Dz)l-
Then, apply the first-order Taylor expansion the initial i-th parameters Of denoted as:
dC (JLe D)
$ 0 = |mm = 0) + ui @ 0%
00i,
(25)
(Me D;
L. (μ 0; DB)
00%
we eliminate the identical parts and apply the known value of 1, and the final formulation could be denoted as,
According to Proposition A
10: 7L (Ne
0: D)
0f aCm (mc 0; DB)
(26)
00;
00,
Therefore, with the combination Eqn. 23 and Eqn. 26 the final importance scores evaluation function AlterMOMA of the LiDAR backbones could be presented with normalization:
S( = a
of
3
of
(27)
No -
N:
C Detailed Alter for parameters in the Fusion Modules and Following Task Heads
Similar with Appendix this section, we will complete the details of the AlterEva of parameters from the fusion backbone, with two indicators DeCI and ReRI.
15

---

Deactivated Contribution Indicator. Similar to the parameters of the camera and LiDAR backbones, the loss changes when parameters are deactivating masking is observed and then expand with Taylor first-order expansion, denoted as follows,
- (m (ms 0; -4 00 aLm = D)D))
20; - (m + μ o 03 00;
(28)
003
Then, after simplifying the constant loss and masked (deactivated) terms, formulation DeCI can be expressed as follows:
@Lm(:D) 00,
03 -0
(29)
Reactivated Redundancy Indicator. Different from the above camera backbone formulation in Section B.3 and LiDAR backbone formulation in Appendix B the fusion backbone experience the both alternative masking stages of AlterEva. Therefore, the ReRI of parameters in fusion modules and following task heads are calculated twice, denoted as,
to (Me 0) Cm (Me O;D) L. Eme 0;Dn)l,
(30)
bo, (ug = 0) = Cm (my 0; D) Lm 0; 0;DB)l,
(31)
Specifically, when loss evolves from L. (Me D2) to Ly (Mc - 0; DB) with masking camera backbone, the parameters evolve from = (i.e. f,)) to B.. Meanwhile, when loss evolves from (m4 0; D1) to L. (MI 0; DD) with masking backbone, the parameters evolve from 8 p10 (i.e. 0y) to f. Then, we apply the first-order Taylor expansion to the initial i-th parameters 0i denoted as:
0Lm(1t 0;
Toy (Jce =0) (mm(le D):) + ply 0f f.0.=000
00) 1,0,M.=0
0L = (Mo 0D)
- Lm (Mm = 0; 0;Dr)
1 3 0 f.0.
00; 000 1...p..0
(32)
0L (14 = O;D)
L0, (M 0) (m4 D) + sl f f
00' 00' f,0,1==0
0L = 0; 0;;
- -m (14 = 0; - ply 0 0
00,
1,0,=000
(33)
we eliminate the identical parts and apply the known value of and the final formulation could be denoted as,
According to Proposition A
1,
Φ (Mo = -) = |03 @cm (Mo = O;
2Lm (No = 0; DB)
0j.
(34)
80;
30
f.B
ac (M 0
0L (M DB;
t 0: (14 0) 10,
(35)
00;
00 f,B.u100
Specifically, since the ReRI are calculated twice for parameters in fusion models, we using the hyprpaarameters (8/2) to control the normalization scale of AlterMOMA of fusion modules. For- mally, with the combination Eqn. 29 and Equ. 34 the final importance scores evaluation function AlterMOMA of the LiDAR backbones could be presented with normalization:
$
0 (m 0)
TΦp (Ie = 0)
8
3
S(0))
(36)
$ 0
2
Σ
of (M =0)
Σn &op (Mr =
N,
D Pseudo Code of AlterMOMA
The overview of our framework AlterMOMA and the importance scores evaluation function AlterEva
respectively introduced in Section 3.2 and Section 3.3 Specifically, AlterMOMA employs
16

---

Algorithm Alternative Modality Masking Pruning
Input: Pruning ratio pi iteration steps Networks (Fi,Fos F Related parameters (0,, l.. 0y Dataset DD Initialise Masks
I: Initialise binary mask Peex MM 0. Me + Blnét O Mr ② mm
2: for m E c ) do
3: [Am +-
D Alternative Modality Masking b Redundancy Reactivation Importance Evaluation Reinitialization
4: Train Masked Model with sampled batches from Dj to DB
5: Update importance scores with AlterEva by Eqn 12 and 13
6: 0
8: Threshold 7 +- (1 p) percentile of S(0)
D Set pruning threshold Set pruning mask
9: - (T S S()))
0 @ ginit
b Apply mask on initial parameters > Train Pruned model
11: Finetune models with Dataset D
Table 6: 3D object detection performance comparison with the state-of-the-art pruning methods the KITTI Validation dataset on Car class. We list the models pruned by different approaches within 80%, and 90% pruning ratios. The baseline model is trained with AVOD-FPN architecture.
Sparsity
80%
90%
Tasks
AP-3D
AP-BEV
AP-3D
AP-BEV
Sparsity
Easy
Moderate
Hard 66.5
Moderate
Hard
Easy
Moderate
Hard
Easy
Moderate
Hard
Car (No Pruningl
82.4
72.2
89.4
83.9
78.7
IMP
45.7
43.2
54.2
51.7
SyaFlow
74.2
65.7
60.2
79.5
75.3
70.3
64.5
$4.7
48.1
73.5
67.6
64.4
SNIP
73.5
64.9
59.8
79.1
75.8
69.6
62.7
52.3
45.8
72.4
66.9
63.7
ProPr
78.9
69.6
62.1
85.2
79.1
75.7
74.2
63.4
59.1
81.2
75.1
71.9
AlterMOMA (Ours)
80.5
70.2
63.2
87.2
81.5
77.9
77.4
68.2
62.3
85,3
79.9
75.8
Pedestrian (No Pruningl
50.5
43.2
40.1
58.1
50.7
47.2
IMP
34.3
27.8
23.5
40.9
32.5
29.9
18.7
20.2
16.4
34.1
29.2
25.8
SynFlow
43.7
36.3
45.
41.5
33.9
25.4
20.6
43.5
38.9
SNIP
43.4
35.5
31.8
49.5
44.9
38.6
32.7
23.7
19.8
41.9
37.4
34.1
ProPt
36.9
33.5
AlterMOMA (Ours)
49.4
41.2
37.3
$7.0
49.7
46.5
47.8
39.6
36.1
55.9
46.2
43.2
Cyclist Pruningl
TMP
47.2
36.8
30.7
$1.2
41.4
35.7
39.1
31.5
27.3
45.5
36.9
31.2
SynFlow
56.2
42.2
47.5
36.2
31.1
54.8
44.3
38.2
SNIP
$5.5
43.1
36.2
58.4
47.5
41.9
47.2
35.4
29.2
53.3
42.3
37.5
ProPr
47.5
41.6
52.2
46.1
57.4
45.2
39.6
60.3
49.9
43.7
AlterMOMA (Ours)
62.1
49.8
44,9
65.2
$4.8
48.3
59.7
47.9
43.1
62.5
52.3
46.0
alternative masking on each modality, followed by the observation loss changes when certain modality parameters are activated and deactivated. These observations serve as important indications to identify fusion-redundant parameters, which are integral to our importance scores evaluation func- tion, Alterfiva. AlterMOMA begins with Modality Masking, where one of the backbones is initially masked. This step is followed by Redundancy Reactivation and Importance Evaluation, where the parameter importance scores initially calculated with AlterEva including the computation of DeCI and ReRL. Afterward, the models undergo Reinitialization and Alternative Masking of the other backbone, leading to another round of Redundancy Reactivation and Importance Evaluation. When scores all parameters in backbones are calculated fully with AlterEva, models are pruned to remove parameters with low importance scores and then finetuned.
We have also extended AlterMOMA to accommodate structured pruning, where instead of pruning individual weights, entire channels (or columns in linear layers) are removed. Although this approach imposes more restrictions, significantly enhances memory efficiency and reduces the computational costs associated with training and inference. Adapting AlterMOMA to structured pruning involves simply modifying the shape of the pruning mask μ and the parameter 0 in formulation to represent each channel (or column of the weight matrix). For experimental results, please refer to Section
E
Complete Experimental Results of KITTI dataset
As detailed Section 4.1
we conducted experiments using AVOD-FPN 3 on the KITTI dataset 17 to demonstrate the effcacy of our AlterMOMA on two-stage fusion architectures. Due to space constraints, we initially presented only a subset of our results on the KITTI dataset in Section 4.3 specifically excluding the results for the Pedestrian and Cyclist classes. In this section, aim to
17

---

Table 7: 3D object detection performance comparison with camera-radar fusion models on the nuScenes validation dataset. The baseline model is trained with ResNet and PointPillars backbone.
80%
90%
Sparsity
mAP
NDS
mAP
NDS
BEVfusion-R [No Pruning]
40.3
50.1
ProPr
35.8
43.6
32.4
40.1
AlterMOMA (Ours)
38.5
48.3
35.2
44.3
Table 8: 3D multi-object tracking (MOT) task performance comparison by performing tracking- by-detection on the nuScenes validation dataset. We list the AMOTA of models pruned within 80% and 90% pruning ratios. The baseline model is trained with SwinT and VoxelNet backbone.
80%
90%
Sparsity
AMOTA
AMOTA
BE BEVlusion-mit [No Pruning]
68.2
ProPr
65.2
61.4
AlterMOMA (Ours)
67.1
64.5
provide comprehensive view of our findings KITTI. The complete experimental results are displayed Table 6 Specifically, Table discloses results for classes (Car, Pedestrian, and Cyclist) the KITTI dataset, detailing AP-3D and AP-BEV accuracy across varying difficulty levels, including easy, moderate, and hard. As reported in Table single-modal pruning methods, including IMP, SynFlow, SNIP, and ProPr, experience significant declines in accuracy performance in all classes. Conversely, the incorporation our AlterMOMA yielded promising results.
F Extended Experimental Results on Camera-Radar Models
While our primary focus has been on camera and LiDAR modalities, we recognize that testing on additional modalities would make our proposed method more convincing. Therefore, conducted further evaluations on the camera-radar modality, as presented in Table This includes a comparison of pruming results on the object detection task using BEVFusion-RR with ResNet and PointPillars as backbones. We list the mAP and NDS of models at 80% and 90% pruning ratios. Compared to the bascline pruning method ProsPr, AlterMMMA boosts the mAP of BEVFusion-R R by 2.7% and 2.8% for the two different pruning ratios. The results demonstrate our method's superior performance and generality across multiple modalities, including camera-radas
camera-radar fusion model
G Extended Experimental Results on Tracking Tasks
To further validate the generalizability our method across different tasks, we conducted additional evaluations multi-object tracking (MOT), critical task in autonomous driving, shown in Table 8 We performed bockiig-by--etecttie evaluation on the nuScenes validation dataset, and list the AMOTA of models pruned 80% and 90% pruning ratios. The baseline model was trained with Swin'T and VoxeINet backbones. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the AMOTA of BEVFusion-mit by 1.9% and 3.1 % for the two pruning ratios. These results further demonstrate our method's superior performance and generality across multiple tasks, including tracking.
H Inference Speed of AlterMOMA
In this section, we report the running time in milliseconds in Table 9 We tested the inference time of models after pruning using the structure pruning settings of the Ta of the main paper. Here are the performance results for BEVFusion-mii models with SECOND and ResNet101 backbones on an single RTX 3090: the inference times are 124.04 ms for unpruned models, 106.39 ms for AlterMOMA-300 and 87.46 ms for AlterMOM-500% These results highlight the improvements in inference speed achieved through our pruning techniques.
18

---

Table 9: 3D object detection performance and inference speed comparison with the structure pruning methods on the nuScenes validation dataset. Note that baseline model is BEVfusion-mit with ResNet101 and SECOND backbone and inference is tested on the RTX3090.
ResNet101
SECOND
Sparsity
mAP
NDS
GFLOPs(L %)
Inference time(ms)
BEVlusion-mit [No Pruning]
64.6
69.4
610.66
124.04
AlterMOMA-300 (Ours)
65.3
69.9
420.13 (31.2)
106.39
AlterMOMA-500 (Ours)
64.5
69.5
264.42 (56.7)
87.46
Original Features
Reactivated Features
Fused Redundant Depth Features
Before Pruning
Features After Pruning
fin
LIDAR Features
Fused Features
Camera Features
Enlarged Redundant Depth Features
Figure 3: Visualization of the reactivated redundant features: The figure illustrates the features of different modalities at each stage of the entire pruning process of AlterMOMA, including LiDAR features, camera features and fused features in the states of original (before masking), reactivated (after reactivation), and pruned (after pruning with AlterMOMA).
I Analysis and Visualization of Features
The Figure 3 illustrates the features of different modalities at each stage of the entire pruning process of AlterMOMA. including LiDAR features, camera features and fused features in the states of original (before masking), reactivated (after reactivation). and pruned (after pruning with AlterMOMA) The fourth column provides enlarged views crucial redundant parts of the camera features. Notably, due to masking one side of backbones during reactivation, there are no fused features within reactivated states. Specifically, despite the absence of distant objects, camera features still provide some redundant depth features, as shown in the fourth column. These redundant parts of the original camera features are retained the subsequent original fused features after fusion. To address these redundant depth features. AlterMOMA reactivates these redundant parameters during the reactivation phase, as shown the middle row images of the fourth column. These redundant depth features are then pruned from both fused features and camera features, as observed by comparing the enlarged fused features in the middle row images of the second column and the pruned camera backbone features in the third and fourth columns of the third row.
19

---

30
mAP on Bevlusion-ni
66
654
on
69
B/a
B/a
Figure 4: Ablation study of hyperparameters a and 33 the nuScenes validation dataset. We list the relationship between mAP and B/a with our approaches within 80%, 85%, and 90% pruning ratios. The baseline models, BEVfusion-mit and BEVfusion-pku trained with Swin'T and VoxelNet backbone.
Ablation Study on hyper parameters a and
As described in Section 3.3 the hyperparameters and 3 determine the proportion of DeCI and ReRI within the importance score evaluation function AlterEva. To assess the impact of these indicators on overall pruning performance, we examine the relationship between mAP and the ratio B/a. Baseline models depicted the left subfigures of Figure f are from BEVfusion-mit trained with SwinT and VoxelNet backbones, while the right subfigures represent BEViusion-pki models trained with the same backbones. These experiments utilize the nuScenes dataset to evaluate the efficiency of 3D object detection tasks at pruming ratios of 80%, 85%, and 90%. The experimental results presented in Figure 4 Specifically, when 3/a = in the figure, indicating only relying on DeCI, there is significant drop in mAP compared to best-performing setup. This result underscore the critical role of addressing fusion-redundant parameters. As the ratio increases, indicating greater influence from ReRI, mAP increases, reflecting the beneficial impact of effectively identifying and pruning fusiontredundant parameters. However, as 3 surpasses a certain threshold, resulting in ReRI outweighing DeCI, mAP begins to decline again. It may be due that our redundancy reactivation also reactivates some contributed parameters, which may be accidentally pruned with the excessive usage of ReRI. These findings highlight the selection of hyper parameters and the ablation study for DeCI and ReRI indicators in AlterMOMA. The results are presented in Figure 4 which visually depicts these dynamics across different experimental settings.
K Further Discussion of AlterMOMA on General Multi-modal Fusion models
Although AlterMOMA explores similar feature extraction due to the fusion mechanism, it remains the perception fields, especially camera-LiDAR fusion models. However, extending it to wider range of multi-modal models, such as vision-language models, requires further refinement. As we hypothesize, fusion modules across various modalities and tasks exhibit different functionalitics. In perception-only tasks involving multiple sensors (camera, LiDAR and radar), the fusion mechanism primarily focuses on supplementing and spatial aligning across modalities by fully leveraging the physical properties of different sensors since all inputs consist of vision-based data. Formally, low- level machine features are fused in the multi-sensor fusion mechanism. However, in the fusion mechanism devised for different types of data, such vision and language, things differ. When input data is highly disparate, such as vision and language, fusion modules tend to focus more on matching, which means synchronizing the high-level semantic context between different input formats. Therefore, for the AlterMOMA framework, we concentrate on the redundancy stemming from supplementary functionality, which may only primarily exist in multi-sensor fusion architectures.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to the abstract and Section T
We introduce the problem our work solved and the experiment we conducted in both the abstract and final paragraph of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We discuss the scope of this work in Section Appendix K and the hyper parameters limitation on Appendix D Meanwhile, we define the scope our solved problem in the abstract and introduction.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Section 3.3 and Appendix A Appendix and Appendix C
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 4.1 and Section 4.2 for training details. Please refer to
Section 4.4 Section 4.4. Appendix Appendix E and Appendix (J for experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
22

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [No]
Justification: All the data used is from public available datasets (NuScenes and KITTI).
Please refer to Section[.]
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/nips. wttpcc/pcccccccccccooooooop httpc//gguuieeeeeeeuuuubiiiiiiii weneiisssonPoPiiiio
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 4.1 and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The training procedure for the proposed and baseline models spend consider- able amount of time. It is infeasible to perform enough repeated experiments calculate the statistical significance. Therefore, error bars are not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer to Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpss//neuripps http:///wurrpps.ccppllellleeeeee
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the development of our approach in further multi-modal models, such as vason-language models in Appendix K
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
25

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "UmpPnZpUOH", "reviewer": "Reviewer_hcok", "summary": "This paper introduces a novel and effective pruning framework (AlterMOMA) and outperforms existing pruning methods on multimodal 3D detection/segmentation tasks, based on a novel insight : \"The absence of fusion-contributed features will compel fusion modules to ’reactivate’ their fusion-redundant counterparts as supplementary to maintain functionality\".", "questions": "the paper presents a pruning strategy for multimodal strategy, which seems to work on any multimodal task. However, the authors only experimented on the Camera-LiDAR task. Did authors try more modal combinations (Radar, or even Language, etc.)? Is the approach applicable to other broader tasks?", "limitations": "the paper discussed the limitations in the Appendix.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- the paper presents a pioneering approach to pruning in camera-LiDAR fusion models, tackling redundancy through innovative masking strategies. The procedure is clearly explained and technically sound.\n- the proposed method demonstrates good performance on multiple tasks on two baseline model.\n- this paper proposes an interesting insight and tests it with experiments", "weaknesses": "- while AlterMOMA aims to reduce model complexity, the paper could clarify the computational overhead introduced by the pruning process itself, especially regarding the training time and resources required for reactivation and evaluation.\n- it is interesting to use some more direct methods (e.g., feature visualization) to prove that the \"reactivated\" parameters are \"redundant\".\n- some minor typing errors (e.g., line132, an incorrect quotation mark), which do not affect my rating", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the professional and insightful comments. We address each comment as follows:\n\n***Weakness 1:*** *The computataional overhead of pruning process.*\n- We agree that providing the computational overhead for the proposed pruning framework would significantly enhance the quality of this paper. Therefore, we present the computational expenses as follows:  \nThe pruning process involves **one round** of *\"Masking (LiDAR) - Reactivation (Camera) - AlterMasking (Camera) - Reactivation (LiDAR) - Importance evaluation\"* before the beginning of each fine-tuning epochs, and the **total fine-tuning epoch number is 3**. In each 'Reactivation' stage, we only **sample 100 out of 62k batches** (the entire nuScenes training set comprises 62k batches with a batch size of 2). We tested the time for **the reactivation stage, which is around 170 seconds**. In the Importance evaluation stage, the important scores for parameters were evaluated by using the gradients via backpropagation before and after reactivation for 5 batches, as mentioned in Eqn.11-13 of the main paper. We tested the time for **the importance evaluation stage, which is around 10 seconds**. We tested the time for **the entire round, which is an average of around 190 seconds** in total on a single RTX3090 device. Subsequently, a 3 epochs fine-tuning is performed, which incurs the same computational cost as normal training.  \nPlease note that unpruned baseline models, such as BEVFusion-mit, require a total of 30 epochs to train from scratch, including 20 epochs for LiDAR backbone pre-training and 10 epochs for the entire fusion architecture training. Since pruning is a one-time expense for generating a compact model, its overhead can be considered trivial, especially when considering the time-consuming nature of normal training.Due to the page limit, the aforementioned discussion will be included in the supplementary material for the camera-ready version.\n\n***Weakness 2:*** *Feature visualization of reactivaed redundant parameters.*\n- We appreciate the insightful comment and acknowledge that adding feature visualizations would make our paper more convincing. Therefore, we have included feature visualization figures **in Figure 1 of the uploaded global rebuttal PDF file**. These figures will also be added to the camera-ready paper.  \nThe figure illustrates the features of different modalities at each stage of the entire pruning process of AlterMOMA, including LiDAR features, camera features and fused features in the states of original (before masking), reactivated (after reactivation), and pruned (after pruning with AlterMOMA). For better clarity and understanding, we enlarge views of crucial redundant parts of camera features in the fourth column of Figure 1. Notably, due to masking one side of backbones during reactivation, there are no fused features within reactivated states.  \nSpecifically, despite the absence of distant objects, camera features still provide some redundant depth features, as shown in the fourth column. These redundant parts of the original camera features are retained in the subsequent original fused features after fusion. To identify these redundant depth features, AlterMOMA reactivates these redundant parameters during the reactivation phase, as shown in the middle row images of the fourth column. These redundant depth features are then pruned from both fused features and camera features, as observed by comparing the enlarged fused features in the middle row images of the second column and the pruned camera backbone features in the third and fourth columns of the third row. \n\n***Weakness 3:*** *Typing errors.*\n-  We appreciate the reminder about the typing errors. We have carefully proofread the entire paper and corrected the typographical errors, including the one on line 132.\n\n***Question:*** *Generalizability of AlterMOMA to additional modalities and tasks.*\n- We appreciate the insightful and inspiring comment. We currently focus on camera and LiDAR modalities, as well as 3D object detection and BEV map segmentation tasks, because these modalities and tasks are the most important factors for the perception of autonomous driving. We agree that testing on more modalities and tasks would make the proposed method more convincing. Therefore, we have performed additional evaluations on the **Camera-Radar modality** and and the **multi-object tracking (MOT) task**  seperately **in Table 1 and Table 2 of uploaded global rebuttal PDF file**.  \n**For the Camera-Radar modality, as presented in Table 1 of supplementary PDF file**, it includes a comparison of pruning results on the 3D object detection task using BEVFusion-R, a Camera-Radar fusion model with ResNet and PointPillars as backbones. We list the mAP and NDS of models at 80% and 90% pruning ratios. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the mAP of BEVFusion-R by 2.7% and 2.8% for the two different pruning ratios. The results demonstrate our method’s superior performance and generality across multiple modalities, including Camera-Radar.  \n**For the MOT task, as shown in Table 2 of supplementary PDF file**, we performed tracking-by-detection evaluation on the nuScenes validation dataset, and list the AMOTA of models pruned at 80% and 90% pruning ratios. The baseline model was trained with SwinT and VoxelNet backbones. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the AMOTA of BEVFusion-mit by 1.9% and 3.1% for the two pruning ratios. These results further demonstrate our method's superior performance and generality across multiple tasks, including tracking.  \nPlease refer to the global rebuttal PDF file for more details. Regarding the language modality, we are working on pruning the visual grounding task using natural language and camera modalities. We have added a related discussion in the supplementary material of the main papaer and will share our findings as soon as we obtain promising results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ahAZfVrJQi", "reviewer": "Reviewer_fjuq", "summary": "This paper addresses the problem of feature redundancy in camera-LiDAR fusion models. The authors propose a novel pruning framework, AlterMOMA, which employs alternative masking to identify and prune redundant parameters in these models. The paper demonstrates the effectiveness of AlterMOMA through extensive experiments on the nuScenes and KITTI datasets, showing superior performance compared to existing pruning methods.", "questions": "Can the proposed method be effective on other tasks like action recognition?", "limitations": "The paper mentions one limitation regarding its limited use in the perception field.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a novel pruning framework specifically designed for multi-modal fusion models, addressing the unique challenge of redundant features across different modalities\n2. Extensive results on nuScenes and KITTI showcasing the benefits and performance improvements of the proposed approach\n3. The paper is very well written and easy to follow", "weaknesses": "1. While the method is tested on two camera-LiDAR datasets, the generalizability of the approach to other fusion tasks and modalities beyond camera-LiDAR is not verified, such as video, optical flow and audio in action recognition [A].\n2. The running speed after pruning is not reported\n3. In Table 5, the performance after pruning is even better than the original version. Are there any reasons and analysis?\n\n[A] Munro et al., Multi-modal domain adaptation for fine-grained action recognition. In CVPR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your insightful comments and suggestions. We will address your concerns in the following answers:\n\n***Weakness 1:*** *Generalizability of AlterMOMA to additional modalities and tasks.*\n- We appreciate the inspiring comments. We currently focus on camera and LiDAR modalities, as well as 3D object detection and BEV map segmentation tasks, because these modalities and tasks are the most important factors for the perception of autonomous driving.  \nWe agree that testing on more modalities and tasks would make the proposed method more convincing. However, preparing new datasets, adapting our pruning frameworks to new fusion architectures, and training from scratch is time-consuming. We attempted to implement experiments on action recognition with [1], but it is challenging to accomplish this in the limited rebuttal period. Although we cannot test on action recognition tasks, we have expanded our models to more tasks and modalities to demonstrate our generalizability in autonomous driving. We have performed additional evaluations on the **Camera-Radar modality** and **the multi-object tracking (MOT) task** seperately in **Table 1 and Table 2 of uploaded global rebuttal PDF file**.  \n**For the Camera-Radar modality, as presented in Table 1 of supplementary PDF file,** it includes a comparison of pruning results on the 3D object detection task using BEVFusion-R, a Camera-Radar fusion model with ResNet and PointPillars as backbones. We list the mAP and NDS of models at 80% and 90% pruning ratios. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the mAP of BEVFusion-R by 2.7% and 2.8% for the two different pruning ratios. The results demonstrate our method’s superior performance and generality across multiple modalities, including Camera-Radar.  \n**For the MOT task, as shown in Table 2 of supplementary PDF file,** we performed evaluation on the nuScenes validation dataset, and list the AMOTA of models pruned at 80% and 90% pruning ratios. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the AMOTA of BEVFusion-mit by 1.9% and 3.1% for the two pruning ratios. These results further demonstrate our method's superior performance and generality across multiple tasks, including tracking.  \nIn future research, we plan to explore the applicability of AlterMOMA incorporating more tasks like action recognition. We appreciate your understanding of our current focus and look forward to exploring these avenues in future studies.\n\n***Weakness 2:*** *The running speed after pruning.*\n- We follow SOTA papers and report the computational efficiency in terms of GFLOPs. We agree that additionally reporting running speed would make the paper more practical and convincing. Therefore, we additionally report the running time in milliseconds.  We have tested the inference time of BEVFusion-mit models after pruning with structure pruning setting of Table 5 of the main paper on a single RTX3090 device. The inference times are **124.04 ms for unpruned models, 106.39 ms for AlterMOMA-30%, and 87.46 ms for AlterMOMA-50%**. These results highlight the improvements in inference speed achieved through our pruning techniques. The result table are also presented **in Table 3 of  uploaded gobal rebuttal PDF file** and will be included in the camera-ready version.\n\n***Weakness 3:*** *Analysis of outperformed results in Table 5.*\n- Thank you for reminding us of the need for further analysis on this positive result, which supports our observation on fusion redundancy and would make the paper in better form. Therefore, following analysis is performed and will be included in the camera-ready version:  \nAs discussed in Section 1, the use of pre-trained backbones results in similar feature extractions. With this similarity, low-quality similar features (e.g., depth features from the camera backbone) are also extracted and may be fused in the following fusion modules, thereby disturbing their high-quality counterparts during fusion (e.g., depth features from the LiDAR backbone). Therefore, when these low-quality redundant features and their related parameters are pruned using our proposed AlterMOMA (especially at low pruning ratios, which ensure the pruning of low-quality features is sufficient), overall performance improves during subsequent fine-tuning due to the elimination of redundant disturbances. We also **visualize the fusion redundancy in the uploaded global rebuttal PDF file**, which further explains why the elimination of low-quality features enhances performance.  \nNotablely, better performance after pruning compared to the original version has also been observed in several state-of-the-art studies, such as Table 4 and 5 in [2], Table 3 in [3], and Table 1 and 2 in [4].\n\n***Question:*** *The implement of AlterMOMA on action recognition.*\n- We sincerely agree that evaluating the proposed method on action recognition would greatly improve the generalizability of AlterMOMA. Currently this work focuses on perception tasks of autonomous driving, and applying the proposed method on the tasks other than perception tasks of autonomous driving (i.e. action recognition) needs further research. Since the limited rebuttal period, we could not adapt the proposed method and evaluate on action recognition. In order to response the question and enhance the generalizability of this paper on more tasks, we performed the proposed method on multi-object tracking tasks, which is the downstream task of 3D object detection and BEV map segmentation, which is detailed in **the answer of weakness 1 and also shown in the Table 1 of uploaded global rebuttal PDF file**.\n\n[1] Munro et al., Multi-modal domain adaptation for fine-grained action recognition. CVPR, 2020.  \n[2] Huang Y et al. CP3: Channel pruning plug-in for point-based networks. CVPR 2023.  \n[3] Lin M et al. Hrank: Filter pruning using high-rank feature map. CVPR, 2020.  \n[4] Fang G et al. Depgraph: Towards any structural pruning. CVPR, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MyLH1brv4c", "reviewer": "Reviewer_62BJ", "summary": "This paper introduces Alternative Modality Masking Pruning (AlterMOMA), a pruning framework utilizing alternative masking on each modality to pinpoint redundant parameters. It detects redundant features and relevant parameters through the reactivation process. These redundant parameters are then pruned using a proposed importance score evaluation function. Meanwhile, the observation of loss changes indicates which modality parameters are activated or deactivated.", "questions": "How many rounds does \"Modal masking-Redundant activation-importance score evaluation\" take?", "limitations": "Experiments indicate that the proposed Pruning method is effective. However, the effectiveness of other multi-modal fusion methods, such as VLM, still needs to be verified.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-written, and the concept of alternately applying masking to the camera and Lidar backbones to identify and eliminate fusion-redundant parameters is innovative.\n2. When designing the Deactivated Contribution Indicator (DeCI) and Reactivated Redundancy Indicator (ReRI) modules, thorough analysis and formula derivation were conducted for loss change processes using the masking method.\n3. Extensive experiments on the nuScenes and KITTI datasets have demonstrated the effectiveness of AlterMOMA.", "weaknesses": "The paper lacks an illustration of the costs associated with the proposed pruning method itself.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback and insightful comments. We will answer your questions in the following:\n\n***Weakness 1***:  *Lack of the computataional overhead of pruning process.*\n\n - Thank you for reminding us of the importance of detailing the cost associated with the proposed pruning method, including the execution process of algorithms and the rounds count of the entire pruning phases. We agree that providing this information will significantly enhance the quality of this paper and we will add them in the paper. Therefore, we present the detailed computation cost as follows:  \nThe pruning process involves **one round** of *\"Masking (LiDAR) - Reactivation (Camera) - AlterMasking (Camera) - Reactivation (LiDAR) - Importance evaluation\"* before the beginning of each fine-tuning epochs, and the total **fine-tuning epoch number is 3**. In each 'Reactivation' stage, we only sample **100 out of 62k batches** (the entire nuScenes training set comprises 62k batches with a batch size of 2). We tested the time for **the reactivation stage, which is around 170 seconds**. In the Importance evaluation stage, the important scores for parameters were evaluated by using the gradients via backpropagation before and after reactivation for 5 batches, as mentioned in Eqn.11-13 of the main paper. We tested the time for **the importance evaluation stage, which is around 10 seconds**. We tested the time for **the entire round, which is an average of around 190 seconds** in total on a single RTX3090 device. Subsequently, a 3 epochs fine-tuning is performed, which incurs the same computational cost as normal training.  \nPlease note that unpruned baseline models, such as BEVFusion-mit, require a total of 30 epochs to train from scratch, including 20 epochs for LiDAR backbone pre-training and 10 epochs for the entire fusion architecture training. Since pruning is a one-time expense for generating a compact model, its overhead can be considered trivial, especially when considering the time-consuming nature of normal training. Due to the page limit, the aforementioned discussion will be included in the supplementary material for the camera-ready version.\n\n***Question 1***:  *How many rounds does \"Modal masking-Redundant activation-Importance score evaluation\" take?*\n\n- We appreciate the professional question. Regarding the cost of the pruning methods in weakness 1, the process involves **one round** of *\"Masking (LiDAR) - Reactivation (Camera) - AlterMasking (Camera) - Reactivation (LiDAR) - Importance evaluation\"* before the beginning of each fine-tuning epochs, and the total fine-tuning epoch number is 3. Please let us know if any further adjustments or additional details are required.\n\n***Limitation:*** *However, the effectiveness of other multi-modal fusion methods, such as VLM, still needs to be verified.*\n\n- We appreciate the insightful and inspiring comment. We currently focus on camera and LiDAR modalities, because these modalities are the most important factors for the perception of autonomous driving. We agree that testing on more modalities would make the proposed method more convincing. Therefore, we have performed additional evaluations on the **Camera-Radar modality in Table 1 of the uploaded global rebuttal PDF file**. Regarding the vision-language fusion methods such as VLM, we are working on pruning the visual grounding task using natural language and camera modalities. We have added a related discussion in the supplementary material of the main paper and will share our findings as soon as we obtain promising results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to all the reviewers for your efforts in reviewing our work. Your insightful suggestions and comments are greatly appreciated and will significantly enhance the quality of our paper. We are grateful for the positive feedback on various aspects and are encouraged that the reviewers pointed out our work *"is clearly explained and technically sound"* and *"proposes an interesting insight"* (Reviewer hcok), *"introduces a novel pruning framework"* and *"very well written and easy to follow"* (Reviewer fjuq),  *"the concept of alternately applying masking is innovative"* and  *"extensive experiments"* (Reviewer 62BJ). We address the reviews below and will incorporate all changes in the revision.  

Although we were unable to conduct exhaustive extended experiments on AlterMOMA due to the time-consuming nature of the process (e.g., preparing code environments, adapting our pruning frameworks to new model architectures, and training models from scratch on large-scale autonomous driving datasets), we have included **several additional experiments and visualizations in the supplementary PDF of global rebuttals.** These additions address common concerns and demonstrate the efficiency and generality of our work, including:

***1. The generalizability of AlterMOMA on additional modalities.***
- While our primary focus has been on camera and LiDAR modalities, we acknowledge that testing on additional modalities would make our proposed method more convincing. Therefore, we conducted further evaluations on the **Camera-Radar modality, as presented in Table 1 of supplementary PDF file.** This includes a comparison of pruning results on the 3D object detection task using BEVFusion-R, a Camera-Radar fusion model with ResNet and PointPillars as backbones. We list the mAP and NDS of models at 80% and 90% pruning ratios. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the mAP of BEVFusion-R by 2.7% and 2.8% for the two different pruning ratios. The results demonstrate our method’s superior performance and generality across multiple modalities, including Camera-Radar.  
 Regarding the language modality, we are working on pruning the visual grounding task using natural language and camera modalities. We have added a related discussion in the supplementary material of the main paper and will share our findings as soon as we obtain promising results.

***2. The generalizability of AlterMOMA on additional tasks.***
- Similarly, to demonstrate our method's generalizability across different tasks, we have conducted additional evaluations on an important task in autonomous driving, **multi-object tracking (MOT), as shown in Table 2 of supplementary PDF file.** We performed tracking-by-detection evaluation on the nuScenes validation dataset, and list the AMOTA of models pruned at 80% and 90% pruning ratios. The baseline model was trained with SwinT and VoxelNet backbones. Compared to the baseline pruning method ProsPr, AlterMOMA boosts the AMOTA of BEVFusion-mit by 1.9% and 3.1% for the two pruning ratios. These results further demonstrate our method's superior performance and generality across multiple tasks, including tracking. 

***3. The running speed of models after pruning with AlterMOMA.***
- We agree that additionally reporting the running speed would make the paper more practical and convincing. Therefore, we report the running time in milliseconds. We tested the inference time of models after pruning using the structure pruning settings of the Table 5 of the main paper. Here are the performance results for BEVFusion-mit models with SECOND and ResNet101 backbones on an single RTX 3090: the inference times are **124.04 ms for unpruned models**, **106.39 ms for AlterMOMA-30%**, and **87.46 ms for AlterMOMA-50%**. These results highlight the improvements in inference speed achieved through our pruning techniques. The result table are also presented **in Table 3 of supplementary PDF file.**

***4. The computational overhead of proposed AlterMOMA.*** 
- The pruning process involves **one round** of *"Masking (LiDAR) - Reactivation (Camera) - AlterMasking (Camera) - Reactivation (LiDAR) - Importance evaluation"* before the beginning of each fine-tuning epochs, and the **total fine-tuning epoch number is 3**. In each 'Reactivation' stage, we only **sample 100 out of 62k batches** (the entire nuScenes training set comprises 62k batches with a batch size of 2). We tested the time for **the reactivation stage, which is around 170 seconds**. In the Importance evaluation stage, the important scores for parameters were evaluated by using the gradients via backpropagation before and after reactivation for 5 batches, as mentioned in Eqn.11-13 of the main paper. We tested the time for **the importance evaluation stage, which is around 10 seconds**. We tested the time for **the entire round, which is an average of around 190 seconds** in total on a single RTX3090 device. Subsequently, a 3 epochs fine-tuning is performed, which incurs the same computational cost as normal training.  
Please note that unpruned baseline models, such as BEVFusion-mit, require a total of 30 epochs to train from scratch, including 20 epochs for LiDAR backbone pre-training and 10 epochs for the entire fusion architecture training. Since pruning is a one-time expense for generating a compact model, its overhead can be considered trivial, especially when considering the time-consuming nature of normal training. Due to the page limit, the aforementioned discussion will be included in the supplementary material for the camera-ready version.

***5. The features visualization.***
- We appreciate the insightful comment and acknowledge that adding feature visualizations of reactivated redundant features would make our paper more convincing. Therefore, we have included feature visualizations **in Figure 1 of the supplementary PDF file.** These figures will also be added to the camera-ready paper.

Meta-review: he authors propose a novel pruning framework, AlterMOMA, which employs alternative masking to identify and prune redundant parameters in camera-LiDAR fusion models. The paper demonstrates the effectiveness of AlterMOMA through extensive experiments on the nuScenes and KITTI datasets, showing superior performance compared to existing pruning methods. The initial reviews consisted of two accepts and one borderline reject. The concerns raised were about the generalizability to additional modalities and tasks, as well as the running speed after pruning. In the rebuttal, the authors explained why camera-LiDAR is key for 3D object detection and demonstrated that the proposed method indeed speeds up the runtime after pruning. The final reviews resulted in two accepts and one weak accept. Therefore, the AC recommends acceptance.
Decision: Accept (poster)