SlimSAM: 0.1% Data Makes Segment Anything Slim
Authors: Zigeng Chen
Gongfan Fang
Xinyin Ma
Xinchao Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: segment anything model
model compression
Primary area: machine_vision
TL;DR: To address the challenging trade-off between performance and compression cost, we introduce SlimSAM, a novel data-efficient SAM compression method that achieves superior performance with extremely less training data.

Abstract: Current approaches for compressing the Segment Anything Model (SAM) yield commendable results, yet necessitate extensive data to train a new network from scratch. Employing conventional pruning techniques can remarkably reduce data requirements but would suffer from a degradation in performance. To address this challenging trade-off, we introduce SlimSAM, a novel data-efficient SAM compression method that achieves superior performance with extremely less training data. The essence of SlimSAM is encapsulated in the alternate slimming framework which effectively enhances knowledge inheritance under severely limited training data availability and exceptional pruning ratio. Diverging from prior techniques, our framework progressively compresses the model by alternately pruning and distilling distinct, decoupled sub-structures. Disturbed Taylor pruning is also proposed to address the misalignment between the pruning objective and training target, thereby boosting the post-distillation after pruning. SlimSAM yields significant performance improvements while demanding over 10 times less training data than any other existing compression methods. Even when compared to the original SAM, SlimSAM achieves approaching performance while reducing parameter counts to merely 1.4% (9.1M), MACs to 0.8% (23G), and requiring only 0.1% (10k) of the SAM training data.

SlimSAM: 0.1 % Data Makes Segment Anything Slim
Zigeng Chen, Gongfan Fang, Xinyin Ma, Xinchao Wang" National University of Singapore zigeng990u. nus. edu, xinchao@nus.edus edu.
Abstract
Current approaches for compressing the Segment Anything Model (SAM) yield commendable results, yet necessitate extensive data to train new network from scratch. Employing conventional pruning techniques can remarkably reduce data requirements but would suffer from degradation in performance. To address this challenging trade-off, we introduce SlimSAM, novel data-efficient SAM compression method that achieves superior performance with extremely less train- ing data. The essence of SlimSAM is encapsulated in the alternate slimming framework which effectively enhances knowledge inheritance under severely lim- ited training data availability and exceptional pruning ratio. Diverging from prior techniques, framework progressively compresses the model by alternately pruning and distilling distinct, decoupled sub-structures. Disturbed Taylor prun- ing is also proposed address the misalignment between the pruning objective and training target, thereby boosting the post-distillation after pruning. Slim- SAM yields significant performance improvements while demanding over 10 times less training data than any other existing compression methods. Even when compared the original SAM, SlimSAM achieves approaching performance while reducing parameter counts to merely 1.4% (9.1M). MACs to 0.8% (23G), and requiring only 0.1 (10k) of the SAM training data. Code available attps://githuu: com www.c/gggi2cseccoooo 25/55mmm/MM
Introduction
Segment Anything Model (SAM) 25 has attracted considerable attention from the community since 59 [55
53 35 have achieved substantial progress by incorporating SAM as fundamental component. Nevertheless, despite its remarkable performance, SAM's ' substantial model size and high computational demands render it inadequate for practical applications on sesource-constraine advancement and broader application of 4AM-based models.
its inception. A plethora of studies 48 19 34
43
31
18
devices. This limitation consequently hinders the
To mitigate these constraints, many efforts 29 62
have been made effectively compress SAM. Without exception, these endeavors opt to replace the originally heavyweight image network
60
50
encoder with lightweight and efficient architecture. This invariably entails training from scratch. With regard to scratch training, an unavoidable challenging trade-off arises between training costs and model performance. Existing methods all inevitably compromise performance when training with very limited data.
The crux of the above issue is their inability to fully exploit the capability of pre-trained SAM. To overcome the high training data demands by reusing the robust prior knowledge of pre-trained SAM,
straightforward strategy involves the application of pruning techniques [38 [4 3 52
g to directly
"ooresppddig Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Heavyweight Image Encoder
Embedding Pruning
Bottleneck Pruning
Our Lightweight Image Encoder
Loss
Loss Embedding Aligning
Bottleneck
Aligning
Figure 1: A simple overall diagram of the proposed alternate slimming process.
compress the sizable SAM by removing redundant parameters from the network and fine-tuning the streamlined model with a minimal dataset [16 36 10] 30]. Nevertheless, following this conventional procedure leads unexpected steep performance degradation, particularly when the pruning ratio is set aggressively high and the available data is extremely scarce.
In response to the challenges outlined above, we present SlimSAM, data-efficient method for SAM compression. Initiating with a standard the compression procedure by introducing our novel designs customized for severely limited data availability and the intricate coupled structure of SAM, culminating in exceptional efficacy while requiring minimal training data. Central to the method are pioneering contributions: the alternate slimming framework and the disturbed Taylor pruning.
workflow, we gradually "modernize"
The alternate slimming framework, presented in Figure [ boosts performance by minimizing di- vergence from the original model and enabling the intermediate feature alignment via consistent dimensionality. Diverging from prior methods, it alternates between pruning and distillation within decoupled model components. The process begins by targeting the embedding dimensions for pruning and aligning the consistent bottleneck dimensions for distillation. It then shifts focus to pruning the bottleneck dimensions in ViTs [6], aligning the unchanged embedding dimensions for distillation. Observing the misalignment between the pruning object and the distillation target impedes the efficacy of compression, we introduce novel label-free importance estimation criterion called disturbed Taylor importance to address this misalignment effectively, thereby enhancing the recovery process and obviating the need for labeled data.
Comprehensive assessments across performance metrics, efficiency, and training data requirements reveal that SlimSAM markedly enhances compression performance, concurrently achieving superior lightweight and efficiency with markedly reduced training data requirements. Notably, our entire compression can be completed using only 10k un-labeled images on a single Titan RTX GPU.
In summary, our contribution is data-efficient SAM compression method called SlimSAM, which effectively repurposes pre-trained SAMs without the necessity for extensive retraining. This is achieved through a novel modernized mouiingdisttillatinn procedure. By proposing the alternate slimming framework and introducing the concept of disturbed Taylor importance, we realize greatly enhanced knowledge retention in data-limited situations. When compared to the original SAM-H. SlimSAM achieves approaching performance while reducing the parameter counts 1.4% (9.1M). MACs to 0.8% % (23G), and requiring mere 0. % (10k) of the training data. Extensive experiments demonstrate that our method realizes significant superior performance while utilizing over 10 times less training data when compared to any other compression methods.
Related Works
Model Pruning. Due the inherent parameter redundancy in deep neural networks 13 model
pruning [16] 14
30
3 27,
33,
52,
36
10) has proved to be an effective approach for accelerating
and compressing models. Pruning techniques can generally classified into two main categories:
Structural pruning focused on eliminating parameter groups based on predefined criteria, while unstructured pruning involves the removal of individual weights, typically requiring hardware support.
structural pruning 30 27 56
56
7 9 and unstructured pruning 5 26 40
42
12

---

Efficient Learning. Efficient Learning refers to range of techniques [54 57 aimed reducing the training costs of deep models while maintaining performance. Knowledge Distillation (KD) 17] is a prominent method under this category, where knowledge is transferred from a larger, powerful teacher model to smaller, more efficient student model. This approach leverages soft targets and temperature parameter to enable the student model to learn more effectively. KD 37 39] has proven to be an effective strategy for model compression, making it highly applicable in scenarios requiring resource-cfficient deployment.
20 21 58
22 28
[46 [44 61 0 51 32
47
SAM Compression. The formidable model size and computational complexity SAM pose challenges for edge deployment, prompting an extensive array of research focused on devising compression techniques for SAM enhance its applicability. Notably, FastSAM 62 replaces
SAM's extensive ViT-based architecture with the efficient CNN-based YOLOv8-seg 22 model,
while MobileSAM 660] adopts the lightweight Tinyvit 49 to replace the image encoder and employs
knowledge distillation from the original encoder. EdgeSAM [63 introduces the prompt-in-the-
knowledge distillation to accurately capture the intricate dynamics between user input and mask generation. EfficientSAAM (50] innovatively adapts MAE [1] framework to obtain efficient image encoders for segment anything model but requires extensive training data even more than the SA- dataset. However, the above approaches all inevitably suffer from scratch training, resulting in unsatisfactory performance when training data is limited.
Remark. The application of common pruning and KD methods falls short in achieving superior performance due the unique challenges presented by limited training data and SAM's coupled struc- ture. To enhance performance, we propose an alternate slimming framework to minimize divergence from the original model and enable the intermediate feature alignment by consistent dimensionality. We also propose disturbed Taylor pruning to address the misalignment between pruning objectives and training targets. In contrast to other SAM compression methods, our SlimSAM achieves superior compression performance while significantly incurring lower training data requirements.
Methods
Our paramount objective to achieve substantial compression of the large image encoder while minimizing performance degradation in scenarios characterized by severe data limitations. To navigate the challenging trade-off between maintaining remarkable performance and the necessity for copious training data, we adopt a strategy of directly inheriting the core weights from the original SAM. This approach capitalizes on SAM's robust prior knowledge, derived from 11 million images. Adhering to this foundational principle, we begin with standard workflow: initial pruning of the model followed by refinement through post-distillatin.
3.1 Identifying SAM Redundancy
The initial phase dedicated the estimation of the importance of each parameter, determining the non-essential and redundant parameters of the image encoder to be pruned. To fulfill this objective, we endeavor to estimate the importance
parameter through the quantification of prediction errors engendered by its removal (38] Given a labeled dataset with image pairs (xi, Ur 3 and
model FF with M parameters W {wi)/< The output of the original model can be defined as I; = Fw (æ;). ). Our objective to identify the parameters that yield the minimum deviation in the loss. Specifically, the importance of a parameter wi; be defined
In AL 9:)l = [Cw, y:) Lue o y:)l,
(1)
where C(x,, y:) the loss between the model output and the label yi when input data is x;, We can approximate L in the vicinity of by its first-order Taylor expansion:
aL(xt,y)) V:) Dw;
Lo y1) Lwy (Iz, yi)
w; + RI(w; 0).
(2)
Substituting equation into equation
we can approximate the parameter importance as:
1w, s Lo,= v:) Lw,=0 v.) v1)+ + aC(x, U1)
aC(x,, 91)
w.
Wi
(3)
Ow;
Ow;
3

---

However, there exist two distinct limitations associated with the above Taylor importance estimation when pruning the image encoder of SAM. Firstly. the accuracy of Taylor importance relies heavily on the availability of sufficiently accurate hard labels yi. Unfortunately, due to the intricate nature of jointly optimizing the image encoder and combined decoder [60], the post-distillation process necessitates performing on the image embedding 1,, resulting in the utilization of soft labels exclu- sively. Secondly, a concern arises regarding the consistency of loss functions when employing Taylor importance estimation for SAM pruning. The importance estimation strategy's primary objective is to identify parameters w; that minimize the hard label discrepancy [ V.) In contrast, the goal of the dollllatio-bbssed recovery process is to minimize the soft label loss | t.)l. This misalignment in optimization objectives potentially impedes the efficacy of the distillation process. The experimental results in Section alsso strongly prove our conclusion.
Disturbed Taylor importance. address the unique limitations associated with Taylor importance estimation, we introduce an extremely simple yet effective solution known as disturbed Taylor importance. Given the absence of hard labels and the incongruity of loss functions, a logical approach is to identify parameters wy that minimize the soft label divergence [ f4)l. However, the gradients pc(#,,)) Sho resulting from applying the loss between encoder's outputs t, are consistently zero. Consequently, we calculate gradients based on the loss function between the original image embedding & and disturbed image embedding i; + + where N is Gaussian noise with mean μ = and standard deviation a 0.01. As the expectation E(t, + +W = t,, when the batch size is large enough, the importance of a parameter w, can be approximated as:
J. = AC(x, t:)l Re |AC 44 +N1
Lw, (Ii, to + +N
Lu, +N1
0L(x,, 4 + +N
Wi
ow;
As the generated gradients (C(x)
# O, the importance can be estimated.
Ow.
Remark. Leveraging our disturbed Taylor importance, the pruning objective is seamlessly aligned with the optimization target of subsequent distillation. Compared to previous pruning techniques, it results in a 0.85% MIoU enhancement when the pruning ratio reaches 77% and a 0.60% MloU improvement when the pruming ratio is set at 50%. Morcover, the adoption of disturbed Taylor importance transforms the entire compression workflow into a convenient label-free framework without incurring additional computational costs.
3.2 Alternate Slimming.
After estimating the weights' importance, our approach advances to implementing channel-wise structural pruning on the extensive image encoder, followed by distillation- based model finetuning. To attain an unprecedentedly high compression rate, the pruning ratio in this study is necessitated to be set significantly higher than in typical scenarios. With the pruning ratio exceeding 75%, we observe a marked performance degradation between the pruned model and its original counterpart, consequence of employing the conventional single-step pruning technique. Additionally, the extremely constrained data availability also poses unique challenges to distillation efficacy. Employing merely 0.1 % of the SA-1B dataset (10k images) for post- dost-distillatin underscores significant challenge in recuperating satisfactory performance for the pruned model.
To address identified challenges, we introduce an innovative alternate slimming framework, anchored by two principles: reducing the divergence between the original and pruned models, and enhancing post-distillation efficacy.
Our framework decomposes the model into two separate sub-structures: embedding (output di- mensions of each block) and bottleneck (intermediate features of each block). By sequentially pruning and restoring either sub-structure, we achieve a smoother compression loss, preventing the steep performance degradation typically associated with extreme pruning ratios. To improve post-distillation we exploit the hidden state information of the original model. Due to the structural resemblance between the pruned and original models, using intermediate hidden states for supervision facilitates superior knowledge transfer. Traditional pruning workflow struggles with dimensionality inconsistency, complicating hidden state supervision. Our method, by partitioning the model into

---

Mul: Hled Attenzion Block
MLP
Image
Ensheedinn
SAM-B
Pasa
Enbsalin
N Vits
7659200
LapoNvee
Lincr
760-6902
N x Vis
Can
Cundaye
3000000
Emhedding Proning
Can
Bottienecc Aigning
Can
Can
OKY Atstion
Drojetion
701-384
LaveNom
Lave 384:302
N Vits
Cas Laye
Cuntamer
Crmb
Bottieneck Proning
Camb
CEmb
CEmb
C
Embodling Allgunn
XX VIA
LeyeNonn
Pegertoe
36000N
Layethrm
Linel
36001556
N x Vits
Conn NU
Com MORDAAO
Slm8AMM
Figure 2: The provided figure depicts our alternate slimming process with a 50% pruning ratio on SAM-B. We utilize structural pruning the channel-wis group level compress SAM's image encoder, coupled with knowledge distillation from intermediate layers to restore the pruned encoder. The red numbers highlight the pruned dimensions each pruning step.
sub-structures, circumvents this issue. Whether pruning embedding or bottleneck dimensions, the intact remaining dimensions enable alignment through loss backpropagation. The effectiveness of this feature alignment, especially in data-scarce scenarios, highlights our framework's efficacy.
An overview of the alternate slimming framework is detailed in Figure 2 Given the Vit-based image encoder with k blocks, the output and intermediate features of each block within the encoder are denoted as E (MHABs), intermediate feature refers to the concatenated QKV features, while for the MI.Ps, it refers to the hidden features between two linear layers. The final output image embedding is represented as t. The original encoder is referred to as uo, while the pruned encoders after embedding pruning and bottleneck pruning denoted as and U2, respectively. The alternate slimming process can be described as the following progressive procedure: embedding pruning, bottleneck aligning, bottleneck pruning, and embedding aligning.
{ei),-1 and H {14) Specifically, for Multi-Head Attention Blocks
Embedding Pruning. The embedding dimension significantly impacts the encoder's performance as it determines the width of features extracted within the encoder. To begin with, we prune embedding dimensions D( E) while keeping the bottleneck dimensions D(H) constant. The presence of residual connections necessitates the preservation of uniformity in the pruned embedding dimensions D(fe)) across all blocks. Consequently, employed uniform local pruning.
Bottleneck Aligning. In the context incremental knowledge recovery, the pruned encoder learns from the original encoder's output b. and aligns with its dimensionality-cnniitt bottleneck features H, each block. The distillation loss function for bottleneck aligning combination bottleneck feature loss and final image embedding loss:
CBn a LMSE(H,,,))) v Ho,
+ +
a) LMSE ter )
(5)
where LMSE(..)) is mean-squared error, the dynamic weight a of nth epoch is defined as:
0.5 n n<N {
{
n n>=N
We N 10 for bottleneck aligning.
Bottleneck Pruning. Following the pruning the embedding dimension D(F) and its coupled struc- tures, we exclusively focus pruning the bottleneck dimension. As the dimension of intermediate
5

---

features ((fh))==)) in each block are entirely decoupled, we can systematically apply dimension pruning at various ratios for each block while maintaining the predetermined overall pruning ratio. This approach involves utilizing global ranking of importance scores to conduct global structural pruning.
Embedding Aligning. The pruned encoder 22 will learn from the embeddings E,, and final image embedding T, from the pruned encoder in to expedite knowledge recovery. Simultaneously, also computes loss functions based on the final image embedding Eno from the original encoder vo to enhance the precision of knowledge recovery. The total loss function for embedding aligning is defined as:
Bmb a. (CMSE Ev, Eus) + LMSE(te,, tva))
+(1 a)) toz)
where the dynamic weight a rath epoch is defined as:
N-n-1 n < N n カ>= N
(8)
The dynamic weight a will progressively diminish to zero as the distillation process unfolds. This transition in the learning objective of distillation gradually shifts from i1 to vo contributing smoother knowledge recovery. We also set N = 10 for embedding aligning.
Remark. The implementation of alternate slimming on decoupled sub- structures significantly reduces the disruption to the original model, particularly when the pruning ratio is quite high. This strategy also preserves consistent dimensionality, enabling effective intermediate feature distillation, which is especially beneficial data-scarce conditions. Consequently, in comparison to the previous pruning and distillation methods, alternate slimming achieves 3.40% and 0.92% increase in MIoU when the pruning ratios achieve 77% and 50%.
Experiments
4.1 Experimental Settings
Implementation Details. Our SlimSAM has been implemented in PyTorch 41 and trained single Nvidia Titan RTX GPU using only 0.1% (10,000 images) of the SA-IB 25 dataset. The base model of our framework is SAM-B (25). The model's parameters were optimized through the ADAM [24] algorithm with a batch size of 4. Training settings for both bottleneck aligning and embedding aligning are identical. The pruned models undergo distillation with an initial learning rate of Ie which will be reduced by half if validation performance does not improve for 4 consecutive epochs. The total training duration is 40 epochs for SlimSAM-50 (with a 50% pruning ratio) and 80 epochs for SlimSAM-7 (with a 77% pruning ratio). We exclusively compressed the image encoder while retaining SAM's original prompt encoder and mask decoder.
Evaluation Details. To ensure a fair quantitative evaluation of the compressed SAM models, we compute MIoU between the masks predicted by the model and the ground truth masks of the SA-1B dataset. We use the most challenging single-point prompts given in annotations for experiments. The results using box prompts also reported in our Appendix. For efficiency evaluation, we provide information on parameter counts and MAC's Additionally, we present details about training data, training iteration and training GPUs for evaluating the training cost. Qualitative comparison of results obtained using point prompts, box prompts, and segment-everythingg prompts are also shown in the following section.
4.2 Comparision and Analysis
Comparing with existing SAM compression methods. As depicted in Table I we conducted comprehensive comparison encompassing performance, efficiency, and training costs with other SOTA methods. Our SlimSAM-50 and SlimSAM-77 models achieve remarkable parameter reduction to only 4.0% (26M) and 1.4% (9. MM)of the original count, while also significantly lowering computational demands to just 3.5% (98G) and 0.8% (23G) MACs, all while maintaining performance levels comparable the original SAM-H. contrast to other compressed models, our approach yields substantial performance enhancements while simultancously achieving greater lightweight

---

Table I: Comparing with other existing SAM compression methods on SA-1B dataset. We report parameter counts, MACs, training costs, and Mean Intersection over Union (MloU) for a comprehen- sive and fair comparison.
Method
Params]
MACs|
TrainSet
BatchSize
GPUs
Iters
MloUt
SAM-H
641M
2736G
100%)
256
256
90%
78.30%
11MC100%)
180k
SAM-B
25
93M
372G
(LM(100%)
128
128
180k
73.37%
FastSAM
62
11M
37G
220k(298)
32
625K
30.72%
FastAAM-x
64
68M
3300
220K(258
32
625K
35.41%
MobileSAM
100k(158)
62.73%
FficientSAM-i
50
INM
28G
12.2M( 10%)
128
64
450k
69.42%
EfficienSAMMs
12.2M( 10%)
450k
71.19%
EdgeSAM
63
9.6M
23G
100K 1%8)
64
50k
65.96%
SlimSAMMSSKOOrr))
26M
98G
10%/0.11%
10DA
72.33%
SimmSAM-7((urs)
9.1M
23G
10k(0.1%
200k
67.40%
Table 2: Comparing with other structural pruning methods. 'Ratio' signifies the pruning ratio applied channel-wise groups. Training costs remain consistent for the same pruning ratio.
Ratio
Method
Labels
Params]
MACs}
MioU/
SAM-H
641M
2736G
78.30%
Ratio=0%
SAM-L
312M
13150
77.67%
Seratch Distillation
1.63%
Random Pruning
71.03%
Ratio=50%
Magnitude Pruning
H
26M
98G
69 96%
Pruning
Taylor Pruning
38
71.15%
SlimSAAM-50(Orrs
72.33 %
Scratch Distllation
1.34%
Random Pruning
62.5812
Magnitude Pruning
14
9.1M
23G
61.60%
Ratio=77%
Hessian Pruning
63.56%
Tay Pruning
38
64.26%
SlimSAM (Ours
67.40
and efficiency. SlimSAM consistently delivers more accurate and detailed segmentation results across various prompts, preserving SAM's robust segmentation capabilities to the greatest extent. This qualitative superiority over other models is visually evident in Figure and Our approach demonstrates outstanding levels of accuracy and correctness. Most notably, SlimSAM achieves these remarkable outcomes with exceptionally low training data requirements, utilizing merely 0.1% (10k) images of the SA-IB dataset. This represents a significant reduction in data dependency, requiring 10 times less data than both EdgeSAM and MobileSAM, and 1,100 times less data than EfficientSA..
Comparing with other structural pruning methods. Having demonstrated structural pruning's efficacy for SAM compression, we established benchmark for evaluating various pruning methods. SlimSAM is compared with four commonly used pruning methods: random pruning, magnitude pruning, Taylor pruning, and Hessian pruning, each employing different criteria for pruning. Addi- tionally, we conducted comparisons with crratch-iistiledd models, which are randomly initialized networks sharing the same architecture as the pruned models. To ensure a completely equitable comparison, models with the same pruning ratios were subjected to identical training settings. Table Showwases our method's consistent superiority over other structural pruning techniques, particularly higher pruning ratios. SlimSAM-S5 and SlimSAM-77 outperform existing methods, achieving minimum 1% and 3% MloU improvement while incurring the same training cost. It is noteworthy that the performance of scratch distillation is extremely low at such limited training cost. This further proves the effectiveness of our workflow in preserving knowledge from the original model.
Ablation Study and Analysis
We conducted series ablation experiments on the SlimSAM-77 model, which features an ambitious 77% pruning ratio. To ensure fair comparison in the ablation experiments, all evaluated
7

---

1001
Malt
Altemate Simming
Altemate Slimming
Stop
Tradung Epocks
Thining Spuchs
Disturbed Taylor Importance
Random Importance
Figure 3: Training results on SA-1B with the common one-step method and our alternate slimming framework. Left and right are results with disturbed Taylor importance and random importance.
Table Comparison between disturbed Tay- lor pruning and original Taylor pruning.
Table 4: Effect of distillation from intermediate
layers and final output image embeddings.
Method
MIoU/
Step
Distillation Objective
MIoU1
Taylor Pruning
62.04%
Final Image Embeeskings
65.10%
Disturhed Taylor Pruning
62.31
Step
Bottleneck Features
66.32%
SlimSAM-77 Taylor
63. 63.
Step 2
Final Image Embeddings Embedding Features
63.91%
SUmSAAM-77 Disturbed Taylor
64.48%
Stop
64.48%
Uccal huningg
Cloe HWanr!
cabel Fun Man
Gob hrvingg Guin
Gobal Prating Searrant
Gobal huning So Condarbttoo
Gsie moang
Staae Soane
Figure 4: The intermediate dimensions of QVK Attention (top row) and MLP (bottom row) within each VIT after pruning. We present the outcomes local pruning and global pruning under five distinct normalization methods.
models trained for 40 epochs the same 10k images from the SA-IB dataset. We also conduct additional experiments to evaluate the performance SlimSAM with even less training data.
Disturbed Taylor Pruning. First, conducted ablation study to assess the impact of our proposed disturbed Taylor pruning on distillation. This innovative approach aligns the pruning criteria with the optimization objectives of subsequent distillation, resulting in improved performance recovery. As depicted in Table 3 our disturbed Taylor pruning consistently achieves significantly superior performance at the same training cost. For both the common one-step pruning strategy and our alternate slimming strategy, our method demonstrates MIoU improvements of 0.3% and 0.85% % the original Taylor pruning, respectively.
Intermediate Aligning. We also evaluate the effect of incorporating aligning with intermediate layers into the distillation process. As depicted in Table distilling knowledge from intermediate layers leads to significant improvements in training results. Specifically, learning from bottleneck features and final image embeddings results in 1.22% MIoU improvement for step distillation, compared to learning solely from image embeddings. Similarly, for step 2 distillation, learning from embedding features and final image embeddings achieves a 0.57% MIoU improvement over the case where learning is based solely on image embeddings.
Alternate Slimming, In addition, we conducted experiments to investigate the impact of our alternate slimming framework. Unlike the common one-step pruning strategy, we partition the structural pruning process into two decoupled and progressive steps. In the first step, only the dimensions related to the embedding features are pruned, while in the second stage, only the dimensions related to the bottleneck features pruned. Following both embedding and bottleneck pruning,

---

Table Effect of global pruning evaluated un- der five different normalization approaches.
Table 6: Comparision of training results using varied amounts of training data.
Method
Normalization
MloUT
Pruning Ratio
Data
Iters
MloUt
Local Pruming
64.38%
10k
100k
72.33%
5atio=50%
5k
100k
71.89%
Mean
63.64%
69.79%
Max
64.35%
Global Pruning
Sum 63.55%
10%
200k
67.40%
Gaussian 64.48%
Ratio=77%
5k
200k
64,47%
Standardlizatin
64.14%
2k
200k
61.72%
knowledge distillation with intermediate layer aligning is employed the pruned model recover its performance. For a more exhaustive analysis, we present the results obtained using different pruning criteria assess whether the effectiveness of our method is influenced by importance estimation. As illustrated in Figure 3 our alternative slimming framework yields substantial improvements in MIoU, with gains of 3.9% and 3.5% observed under disturbed Taylor importance estimation and random importance estimation.
Global Pruning vs Local Pruning. Finally, we conducted experiments to evaluate the performance of local pruning and global pruning in bottleneck pruning. Given that the bottleneck dimensions in each block are entirely decoupled, systematically applied channel-wise group pruning at various ratios for each block while preserving the predefined overall pruning ratio in this step. To obtain consistent global ranking, normalized the group importance scores Ig of each layer in Ig:
Igi five ways: Sum: Ig. ETe Jo: (ii) Mean: Ig: E Jat/K'
Igs
(iii) Max: Igi
Mas (J..)'
(iv) Standarization: Iç; Maz! (T::) Minf (ee))+1e-8 (v) Gaussian: Ig: = Isc -E J../K
1g.- Moz (1g.)
of (I
As indicated in Table [ local pruning ensures consistent performance, whereas global pruning raises the model's upper-periee apper-performance limit. Global pruning's s efficacy is highly dependent on the chosen importance normalization method. For our model, we opted for global pruning with Gaussian normalization, which yielded the best training results. Following global pruning. Figure 4 illustrates the dimensions of bottleneck features (QKV embeddings and MLP hidden embeddings) within each ViT in the image encoder. When applying mean, sum, or Gaussian normalization, the ViTs in the middle exhibit more group redundancy compared to those at the beginning and end. However, the pruned dimensions do not display distinct patterns when utilizing max or standardization normalization. The impact of global pruning becomes more pronounced with increased number of training iterations. Specifically, when training extends to 80 epochs, the MIoU for global pruning exceeds that of local pruning by approximately 2%.
Even less data. As shown in Table 6 with pruning ratio of 50%, reduction in the volume of training data only marginally impacts the model's performance. Notably, even when trained with limited dataset just 2,000 images, our SlimSAM-50 model remarkably attains an MIoU of nearly 70%. However, the pruning ratio is elevated to 77%, decrease in training data more significantly affects performance. This leads to the inference that although our methodology, which integrates pruning and distillation techniques, mitigates the need for extensive training datasets, the availability of more training data can still enhance model performance, particularly at higher pruning rates.
6 Conclusion
In this paper, we present a novel data-efficient SAM compression method, SlimSAM, which achieves superior performance with minimal training data. The essence of our approach lies in the efficient reuse of pre-trained SAM, avoiding the need for extensive retraining. We introduce key designs to the compression method for enhancing knowledge retention from the original model in data- limited situations. Specifically, our alternate slimming framework carefully prunes and distills decoupled model structures in an alternating fashion, minimizing disruptions to the original model and enabling the intermediate feature alignment by consistent dimensionality. Furthermore, the proposed disturbed Taylor importance estimation rectifies the misalignment between pruning objectives and training targets, thus boosting post-distillation after pruning. SlimSAM convincingly demonstrates its superiority while imposing significantly lower training costs compared to any other existing methods.

---

Elfr EMTeerSAAM
Figure 5: Comparison of segmentation results using segment everything prompts
Eipe
SimSAM--0
Figure 6: Left columns: segmentation results obtained using point prompts; right 3 columns: segmentation results achieved with box prompts.
Acknowledgement
This project supported by the Ministry of Education, Singapore, under its Academic Research Fund Tier 2 (Award Number: MOE-T2EP20122-0006
10

---

References
[1] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Leamingg efficient object detection models with knowledge distillation. Advances neural information processing systems, 30, 2017.
[2] Keyan Chen, Chenyang Liu, Hao Chen, Haotian Zhang. Wenyuan Li, Zhengxia Zou, and Zhenwei Shi. Rsprompter: Learning to prompt for remote sensing instance segmentation based on visual foundation model. arXiv preprint arXiv:2306. 16269, 2023.
[3] Ting-Wu Chin, Ruizhou Ding, Cha Zhang, and Diana Marculescu. Towards efficient model compression learned global ranking. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 1518-1528, 2020.
[4] Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1943-4955 2019.
[5] Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in newral information processing systems, 30, 2017.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[7] Gongfan Fang, Xinyin Ma, Michael Mi, and Xinchao Wang. Isomorphic pruning for vision models. arXiv preprint 2rXiv:2407.046116 2024.
Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, and Xinehao Wang. Depgraph: Towards any structural pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pages 16091-16101, 2023.
[9] Gongfan Fang. Xinyin Ma, and Xinchao Wang. Structural pruning for diffuuson models. Advances in Neural Information Processing Systems, 2023.
[10] Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models, Advances in neural information processing systems, 36, 2024.
[11] Gongfan Fang, Kanya Mo, Xinchao Wang, Jie Song, Shitao Bei, Haofei Zhang, and Mingli Song. Up to 100x faster data-free knowledge distillation. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6597-6604, 2022.
[12] Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. Maskllm: Leamable cemi-structured sparsity for large language models. arXiv preprint arXiv:2409. 17481, 2024.
[13] Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv: 180X:v:1803.088 803.03635, 2018
[14] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections efficient neural network. Advances in neural information processing systems, 28, 2015.
[15] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogmition, pages 16000-16009, 2022.
[16] Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1389-1397, 2017.
[17] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.025: 2015.
[18] Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. arXiv preprint arXiv:2307.1295 2023.
[19] Yongcheng Jing, Xinchao Wang, and Dacheng Tao. Segment anything in non-euclidean domains: Chal- lenges and opportunities. arXiv preprint arXiv:2304.1 15595 2023.
[20] Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Amalgamating knowledge from heterogeneous graph neural networks. In CVPR, 2021.

---

[21] Yongcheng Jing. Yiding Yang, Xinchao Wang. Mingli Song, and Dacheng Tao. Meta-aggregator: Learning to aggregate for 1-bit graph neural networks. In ICCV, 2021.
[22] Yongcheng Jing, Chonghin Yuan, Li Ju, Yiding Yang, Xinchao Wang, and Dacheng Tao. Deep graph reprogramming. In CVPR, 2023.
[23] Glenn Jocher, Ayush Chaurasia, and Jing Qiu. Yolo by ultralytics, 2023.
[24] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 141206880 2014.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Lo, et al. Segment anything. arXiv preprint 00X4423304.02633, 2023.
[26] Namhoon Lec, Thalaiyasingam Ajanthan, Stephen Gould, and Philip HS Torr. A signal propagation perspective for pruning neural networks at initialization. arXiv preprint 19X:::90660060000 2019.
[27] Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint 100.:066000087/0 2016.
[28] Qi Li, Rumpeng Yu, and Xinchao Wang. Encapsulating knowledge in one prompt. arXiv preprint arXiv:2407.11902 2024.
[29] Weicong Liang. Yuhui Yuan, Henghui Ding. Xiao Luo. Weihong Lin, Ding Jia. Zheng Zhang, Chao Zhang, and Han Hu. Expediting large-scale vision transformer for dense prediction without fine-tuning. Advances in Neural Information Processing Systems, -3445762-33477, 2022.
[30] Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hat Xue, Xinjiang Wang. Yimin Chen, Wenming Yang. Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning, pages 7021-7032. PMIR, 2021.
[31] Songhua Liu, Jingwen Ye, and Xinchao Wang. Any-to-any style transfer: Making picasso and da vinci collaborate. arXiv e-prints, pages arXiv-2304, 2023.
[32] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang. Structured knowledge distillation for semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2604-2613, 2019.
[33] Zhuang Liu, Jianguo Li. Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang, Learning efficient convolutional networks through network slimming. In Proceedings of the IEFE international conference on computer vision, pages 2736-2744, 2017.
[34] Zhihe Lu, Zeyu Xiao, Jiawang Bai, Zhiwei Xiong, and Xinchao Wang. Can sam boost video super- resolution? arXiv preprint arXiv:2305.06524 2023.
[35] Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models, arXiv preprint arXiv:2305. /627, 2023.
[36] Xinyin Ma. Gongfan Fang, and Xinchao Wang. Lim- Lim-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36, 2024.
[37] Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques improve low- precision network accuracy. arXiv preprint 1..0v2171,.05852, 2017.
[38] Pavlo Molchanov, Arun Mallya, Stephen Tyree, Iuri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11264-1 1144-122 11272, 2019.
[39] Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj. Venkatesh Babu Radhakrishnan, and Anirban Chakraborty, Sero-shot knowledge distillation in deep networks. In International Conference on Machine Learning, pages 4743 4751. PMLR, 2019.
[40] Sejun Park, Jaeho Lee, Sangwoo Mo, and Jinwoo Shin. Lookahead: A far-sighted alternative of magnitude- based pruning. arXiv preprint 2002.000000048000 2020.
[41] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
12

---

[42] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine-tuning. Advances in Neural Information Processing Systems, 15:20338-203335 2020.
[43] Qiuhong Shen, Xingyi Yang. and Xinchao Wang. Anything 3d: Towards single view anything reconstruc- tion in the wild. arXiv preprint arXiv:2304. 2023.
[44] Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 09355, 2019.
[45] Xiaorui Sun, Jun Liu, Heng Tao Shen, Xiaofeng Zhu, and Ping Hu. On efficient variants segment anything model: A survey. arXiv preprint arXiv:2410.04960, 2024.
[46] Xu Tan, Yi Ren, Di He, Tao Qin, Zhou Zhao, and Yan Liu. Multilingual neural machine translation with knowledge distillation. arXiv preprint arXiv: 10461, 2019.
[47] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi. Kdgan: Knowledge distillation with generative adversarial networks. Advances in neural information processing systems. 31, 2018.
[48] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel. Medical sam adapter: Adapting segment anything model for medical image segmentation. arXiv preprint arXiv:2304. 12620, 2023.
[49] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. In European Conference on Computer Vision, pages 68-85. Springer, 2022.
[50] Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest landola, et al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprint 2xxi:23322000633 2023.
[51] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self-supervision. In European Conference Computer Vision, pages 588-604, Springer, 2020.
[52] Huanrui Yang, Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian aware saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18547-18557, 2023.
[53] Jinyu Yang, Mingqi Gao, Zhe Li, Shang Gao, Fangjing Wang, and Feng Zheng. Track anything: Segment anything meets videos. arXiv preprini arXiv:2304. 1968, 2023.
[54] Xingyi Yang, Daquan Zhou, Jiashi Feng, and Xinchao Wang. Diffusion probabilistic model made slim. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 2023.
[55] Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. Sam3d: Segment anything in 3d scenes. arXiv preprint arXiv:2306. 200.0000000308, 2023.
[56] Zhonghui You, Kun Yan, Jinmian Ye, Meng Ma, and Ping Wang. Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks. Advances in neural information processing systems, 32, 2019.
[57] Ruonan Yu, Songhua Liu, and Xinchao Wang. Dataset distillation: A comprehensive review. JEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[58] Ruonan Ya, Songhua Liu, Jingwen Ye, and Xinchao Wang. Teddy: Efficient large-scale dataset distillation via taylor-approximated matching. In European Conference on Computer Vision, pages 1-17. Springer. 2025.
[59] Tao Yu, Runseng Feng, Ruoyu Feng, Jinming Liu, Xin Jin, Wenjun Zeng. and Zhibo Chen. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arXiv:2304.06790. 2023.
[60] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ho Bac, Scungkyu Lee, and Choong Scon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023.
[61] Borui Zhao, Quan Cui. Renjie Song. Yiyu Qiu, and Jiajun Liang. Decoupled knowledge distillation. In Proceedings of the IEEE/CVF Conference on computer vision and patiern recognition, pages 11962. 2022.
13

---

[62] Xu Zhao, Wenchao Ding, Yongqi An. Yinglong Du, Tao Yu, Min Li. Ming Tang, and Jinqiao Wang. Fast segment anything. arXiv preprint wrXiv:2306.12166 2023.
[63] Chong Zhou, Xiangtai Li, Chen Change Loy, and Bo Dai. Edgesam: Prompt-in-the-loop distillation for on-device deployment of sam arXiv preprint arXiv:2312. 2023.
14

---

A Appendix
* In this document, we provide supplementary materials that extend beyond the scope of the main manuscript, constrained by space limitations. These additional materials include in-depth information about the ablation study of SlimSAM-50, further experiments assessing model efficiency, additional evaluations of training costs, analysis of dynamic loss, limitation discussion, and supplementary qualitative results.
A.1 Ablation Study on MlimSAM-50
We performed an extensive series of ablation studies on the SA-IB 25 dataset utilizing the SlimSAM- 50 model, characterized by its significant 50% pruning ratio. To guarantee a fair and consistent comparison across these ablation studies, each model under evaluation was uniformly trained over span of 20 epochs, employing training dataset comprising 10,000 images.
Disturbed Taylor Pruning. Initially, executed ablation study evaluate the effects of our innovative disturbed Taylor pruning technique on distillation processes. This approach strategically aligns pruning criteria with the optimization goals of the ensuing distillation, thereby facilitating enhanced performance recovery. As illustrated in Table 7 our disturbed Taylor pruning method consistently outperforms, achieving markedly better results at equivalent training expenditures. In comparison to the conventional one-step pruning strategy and our alternative slimming approach, our methodology registers MIoU enhancements of 0.31% % and 0.39% over the standard Taylor pruning method 38 respectively.
Intermediate Aligning. We further investigated the impact of integrating alignment with intermediate layers in the distillation process. As Table illustrates, leveraging knowledge from these intermediate layers substantially enhances the training outcomes. Specifically, when distillation in step incorpo- rates learning from both bottleneck features and final image embeddings, there is a notable 1. 1.19% improvement in MIoU compared in step 2 of the distillation process, a strategy that utilizes both embedding features and final image embeddings demonstrates a 0.29% MIoU improvement over approaches exclusively based on image embeddings.
methodology reliant solely on image embeddings. Similarly,
Table 7: Comparison between disturbed Tay- lor pruning and original Taylor pruning.
Table 8: Effect of distillation from intermediate layers and final output image embeddings.
Method
MloUt
Step
Distillation Objective
MloUt
Tayler Pruning
70.02%
Step
Final Image Embeddings
70.86%
Disturbed Taylor Pruning
70.33%
Step
Bottleneck Features
72.07%
SlimSAM-50 Taylor
70.42%
Step 2
Final Image Embeddings
70.52%
SlmmSAM-5 Disturbed Taylor
70.81 %
Step 2
Embedding Features
70.81 %
Alternate Slimming. Moreover, we conducted a series of experiments to examine the efficacy of our novel alternate slimming strategy. Diverging from the traditional one-step pruning approach, our method divides structural pruning into two distinct and progressive phases. In the initial phase, pruning is exclusively focused on the dimensions pertaining to embedding features. The subsequent stage then targets dimensions associated with bottleneck features. After completing both embedding and bottleneck pruning, we employ knowledge distillation with intermediate layer alignment on
Table 9: Effect of global pruning evaluated under five different normalization approaches.
Pruning Method
Normlization
MIoUU
Local Pruning
70.81%
Mcan
70.76%
Mux
70.77%
Global Pruning
Sum
70,80%
Gaussian
70.83%
Standardization
15

---

hoorg Mar
Gatal Filsay Huur
Ubde has (0art601
CON V.T.T.
Punney Xonsert
Oktall horiin (handerreirr)
Figure 7: The intermediate dimensions of QKV Attention (top row) and MLP (bottom row) within each VIT after pruning. We present the outcomes of local pruning and global pruning under five distinct normalization methods.
Alternate Simmiing
One Step
Toiring Epotks
Figure 8: Training results on SA-1B with common one-step strategy and our alternate slimming strategy.
the pruned model to facilitate performance restoration. The outcomes derived from our proposed disturbed Taylor importance estimation are displayed in Figure [ This figure demonstrates that our alternative slimming strategy significantly boosts MIoU, achieving an increase of 0.5%. When juxtaposed with the ablation study results of SlimSAM-77, becomes evident that our strategy exhibits more pronounced improvement, particularly when applied to models with higher pruning ratios.
Global Pruning vs Local Pruning. In our final set of experiments, we assessed the effectiveness of both local and global pruning approaches in the context of bottleneck pruning. Considering the complete decoupling of bottleneck dimensions in each block, we meticulously implemented channel-wise group pruning at varying ratios across different blocks. This was done while main- taining the predetermined overall pruning ratio for this phase of the study. To obtain consistent global ranking, we normalized the group importance scores Ig of each layer in five ways: (i) Sum: Tai
Ig: = ΣK Joi (ii) Mean: Içi EA l
(iii) Max: Igi
Io:
(iv) Standarization:
Tg: Maz. (F Min (1gs)+ (v) Gaussian: Ig. = I Tei/K Table reveals that o (1g
10. Mar (Ug.)
while local pruning maintains consistent performance, global pruning enhances the upper limit of the model's capabilities. In departure from the findings observed with SlimSAM-77, various normalization methods do not markedly influence post-pruning performance. This suggests that the necessity of selecting an optimal normalization technique increases with the pruning ratio. For our model, we chose global pruning combined with Gaussian normalization, which led the most favorable training outcomes. Figure 7 delineates the distribution of bottleneck feature dimensions (including QKV and MLP hidden embeddings) across each Vision Transformer (ViT) (6] in the image encoder. When mean, sum or Gaussian normalization is applied, the dimensions within the ViTs tend distribute more evenly. However, employing max or standardization normalization often results in significant variances in the intermediate dimensions of each VIT.
A.2 More Analysis on Efficiency
In the principal manuscript, the high efficiency of our SlimSAM model objectively substantiated through the disclosure of parameter counts and Multiply-Accuminn Operations (MACs). This section extends the evaluation by reporting on actual acceleration in inference, further affirming the model's efficiency. As delincated in Table I0 SlimSAM-50 outperforms the original SAM-H
16

---

Table 10: Inference acceleration was empirically tested on an NVIDIA TITAN RTX GPU, revealing that higher pruning rates significantly improve inference speed.
Pruning Ratio
Method
Speed Upt
SAM-H
Fasterxl.0 1.
Ratio=U%
SAM-I
Fasterxi.7
SAM-B
Fasterxe,3
Ratio=50%
SiamSAM ) S0XOns)
Fasterx6.9
Ratio=77%
SlimSAM-77(Oumm)
Fasterx8.6
Table 11: The training results were compared using the varied amounts of training data but maintaining the same training iterations.
Pruning Ratio
Truining Dula
Training Iters
MIoU1
10k
100k
72.33%
50t
5k
100k
71.89%
2k
100k
69.79%
10k
200k
67.40%
Ratio=77%
5k
200k
64.47%
2k
200k
61.72%
model by achieving a 6.9-fold increase in inference speed, while SlimSAM-77 attains an 8.3- fold acceleration. Our compression methodology markedly diminishes the actual inference time, concurrently effecting substantial reductions in both model size and MACs. The inference acceleration metrics were tested using NVIDIA TITAN RTX GPU.
A.3 More Analysis Training Costs
In our foundational manuscripts, we demonstrate that SlimSAM exhibits exceptional compression performance with minimal training cost. pertinent inquiry emerges: can SlimSAM maintain its competitive performance with reduced training costs? To address this, we have undertaken supplementary experiments focusing on the interplay between training cost and performance.
In Table II we present the results of additional experiments conducted with varying quantities of training data, while keeping the number of training iterations constant. We observe that with a pruning ratio of 50%, a reduction in the volume of training data only marginally impacts the model's performance. Notably, even when trained with limited dataset of just 2,000 images, our SlimSAM-50 model remarkably attains an MIoU of nearly 70%. However, as the pruning ratio is elevated 77%, a decrease in training data more significantly affects performance. This leads to the inference that although our methodology, which integrates pruning and distillation techniques, mitigates the need for extensive training datasets, the availability of more training data can still enhance model performance, particularly at higher pruning rates. It can be anticipated that with an increase in the volume of training data, our model may potentially achieve lossless compression of SAM.
Table 12 showcases the outcomes of experiments conducted by modifying the parameters for training iterations, while maintaining a constant training dataset size. The results clearly illustrate a direct relationship between the quantity of training iterations and the effectiveness of the model compression. is evident that more extensive training significantly improves the performance of our compressed models. Remarkably, SlimSAM maintains its superiority over other methods even when the training iterations are halved, demonstrating its robustness and efficiency in achieving nigh-performance compression.
A.4 More Analysis on Dynamic Loss
Further experiments were undertaken assess the efficacy of employing dynamic loss within our intermediate feature alignment procedure. The outcomes of these ablation studies are detailed Table
13
It was observed that a constant weight mechanism is more apt for scenarios involving robust
17

---

Table 12: Training outcomes were evaluated using the same amount of training data across different numbers of training iterations.
Pruning Ratio
Training Data
Training Iters
MloUt
10k
50k
70.83%
Ratio=50%
10k
100k
72.33%
I0k
100k
64.43%
Ratio=77%
10k
200k
67.40%
Table 13: Ablation study on dynamic loss weights for distillation
Step
Model
Constant (as = 0.5)
Dynamic (a
N-n-1
SlimSAM-50
Mloo::22.07%
MioU:71.60%%
SlimSAM-77
MIoU:66.322%
MIoU::6.79%5
SlimSAM-50
MloU:70.42%
MotU:70083%
SlimSAM-77
M1oU163363%%
MIOU:64.48%%
teacher model, whereas the implementation of a dynamic weight strategy enhances performance in instances where the teacher model exhibits lesser strength.
A.5 Limitations
In this analysis, we critically examine the constraints of our methodology.
First, our approach demonstrates robust compression performance with minimal training data. Nonetheless, an expanded training dataset could further enhance the model's capabilities. Our current pre-trained SlimSAMs, limited by hardware constraints, trained a dataset of only 10,000 images from the SA-IB dataset. Utilizing a more comprehensive training dataset could potentially enable method to achieve lossless compression.
Second, the essence of our method lies in employing structural pruning and knowledge distillation to preserve the knowledge of original pre-trained SAMs. This strategy inherently sets the performance ceiling of our model at the level of the original SAM. We found it challenging to surpass the performance of the original SAM, which acted as both the target for pruning and the target for optimization. key area for future research will exploring how to surpass performance the original SAM with limited parameter counts and reduced training costs.
A.6 More Qualitative Results
We present more visual comparisons with other existing compressed models and the original SAM-H. Figure provide detailed visual comparison using the regment-everythin prompt, while Figures 10 and showcase additional qualitative results obtained with box prompts and point prompts. respectively. Relative to established compression models such as MobileSAM (60], FastSAM (62). EdgeSAM 63 and EfficientSAM (50), our model distinetly outperforms achieving more precise segmentation, particularly noticeable at the object edges. Notably, even when benchmarked against SAM-H, our model demonstrates commensurate segmentation capabilities.
A.7 Societal impacts
In this paper, we introduce SlimSAM, novel data-efficient SAM compression method that delivers superior performance using minimal training data. SlimSAM achieves an outstanding compression ratio while preserving robust segmentation capabilities. This advancement enables the deployment of SAM on resource-constrained edge devices, underscoring its significant practical applications.
18

---

Images
FastSAM
MobileSAM
EfficienS
EdgeSAM
SlimSAM-77 77
SlimSAM-S 50
SAM-H
Figure 9: Comparison of segmentation results using segment everything prompts
19

---

Images
FastSAM
MobileSAM
EffciintSAM
EdaeS AM
SlimAM--77
SlimSAM-5n
SAM-H
Figure 10: Comparison of segmentation results using box prompts
20

---

Images
Fas:SAM
MobileSAM
ElficienSAA
SlimSAM-77
SlimSAM-50
SAM-H
Figure 11: Comparison of segmentation results using point prompts
21

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in our abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
22

---

Justification:
We discuss the limitation of our work in the Appendix.A.5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
The paper provide the full set of assumptions.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide mental results.
a detailed description of our method along with extensive experi-
Guidelines:
23

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We offer the full code along with relevant instructions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
wtw.eeuuubissiioooooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https c/iips.cc/public//uu
CodeeSumiiii wwabbbmmsssiinii ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
24

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all the details about the experiment in our paper.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We provide the details about initialization and dataset split.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the details about the computation resources used in the experi- ments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
25

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttp:://neurii:: http://ppp....pppp.ip.cccaaaaaaa
Answer:
[Yes]
Justification: We strictly adhere to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts in the Appendix.A.7.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
26

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models) used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly adhered to.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi paperswithcode thcode. com /atasets
has curated licenses for some datascts. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced in the paper are well documented, and the documenta- tion is provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
27

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "9rEJ8IDn2D", "reviewer": "Reviewer_WP1s", "summary": "previous works on SAM compression replace the heavy image encoder with lightweight counterparts which requires dealing with the trade-off between training cost and model performances. Thus the end performances are usually compromised. This paper proposes a sliming framework to perform pruning and knowledge distillation to maximally preserve the performances of the original SAM model while incur minimum training costs. Extensive experiments show that the proposes method is very effective compared to various baseline methods.", "questions": "1. how is the training set (0.1%) selected? are there any impacts if a different training set is sampled and used to train SlimSAM?\n\n2. since the training and evaluations are done on SA-1B, does the method generalize well to other datasets beyond SA-1B?\n\n3. due to the structure of ViT, a strong uniformity has to be enforced. have the authors thought about ways to work around it so that the model could potentially be further compressed?\n\n4. does the proposed method work on other ViT based models?", "limitations": "it will be great to see the proposed method work on larger SAM models.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper proposes a novel method to compress SAM into significantly smaller one while greatly preserving its original performances. Extensive experiments show that the proposed method uses significantly few training examples and achieves comparable (or better) performances.\n\n2. the paper is well-written and the experiments support the claims made in the paper\n\n3. the ablation studies show more interesting findings such as even less data and different pruning strategies.", "weaknesses": "1. The final quality seems to highly depend on the selected training set, see my question below in the question section.\n\n2. The results in table 1 are a little hard to read, it's better to separate them into methods that are based on different SAM models. Right now they are all mixed together, it may not be very meaning full to compare the performances across different base models.\n\n3. the evaluation seems limited which focuses on SAM-B, it will be great to see how the method works on all SAM models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: how is the training set (0.1%) selected? are there any impacts if a different training set is sampled and used to train SlimSAM?**\nThanks for the comment. The training set is a **random subset** of the SA-1B dataset, containing 11k images, with 10k used for training and 1k for validation. We trained using multiple different subsets under the same settings and found the difference in final results to be negligible (MIoU difference is less than 0.25%). Thus SlimSAM is not very sensitive to the selection of training set.\n\n## **Q2: Since the training and evaluations are done on SA-1B, does the method generalize well to other datasets beyond SA-1B?**\nThanks for the valuable comment. To demonstrate the generalization of SlimSAM on other datasets, we evaluated its zero-shot instance segmentation on the COCO2017 validation set. As shown in Table 1, SlimSAM maintains a strong zero-shot capability very close to the original model, despite a high compression ratio. Additionally, our entire compression process only requires 10k unlabeled images (0.1% of the SA-1B dataset).\n\n### Table 1: Zero-shot Instance Segmentation on COCO\n| **Model**     | **Params ↓** | **MACs ↓** | **COCO AP ↑** | \n|---------------|------|------|--------|\n| SAM-B         | 93M  | 372G |43.4% |\n| **SlimSAM-50**| 26M  | 98G  |42.8% |\n| **SlimSAM-77**| 9.1M | 23G  |41.4% |\n\n## **Q3: The results in Table 1 are a little hard to read, it's better to separate them into methods that are based on different SAM models.**\nThanks for the valuable feedback, this is indeed a question that needs further explanation and improvement.\n\nExcept for our method which is based on **SAM-B (93M)**, all other compression methods are based on **SAM-H (641M)**. Using such a large model as the teacher model means that they will need significantly more training costs than us in the knowledge distillation stage. On the contrary, our model is only based on the lighter SAM-B and uses only 10k unlabeled images as training data to achieve better compression performance than other methods, which once again proves the superiority of our method. \n\n## **Q4: due to the structure of ViT, a strong uniformity has to be enforced. have the authors thought about ways to work around it so that the model could potentially be further compressed?**\nThanks for the comment. As you mentioned, residual connections require the input and output dimensions of each ViT block to be uniform. However, because the intermediate feature dimensions in each ViT block are independent, we can apply dimension pruning at different ratios for each ViT block while keeping the overall pruning ratio consistent. In this work, we use a global ranking of importance scores to perform global structural pruning on the intermediate dimensions of each ViT block.\n\nTable 2 shows the intermediate dimensions of each ViT block after both local uniform pruning and our flexible global pruning. Our method breaks the uniformity enforcement by assigning different pruning ratios to different ViT blocks, resulting in a significant improvement in segmentation MIoU **(65.9%-->67.4%)**.\n\n### Table 2: Intermediate Dimensions of Each ViT Block after Pruning\n*Note: We demonstrate the dimension comparison on SlimSAM-77.*\n| **Method**| **Type**|ViT[0]|ViT[1]|ViT[2]|ViT[3]|ViT[4]|ViT[5]|ViT[6]|ViT[7]|ViT[8]|ViT[9]|ViT[10]|ViT[11]|\n|----|----|----|----|----|----|----|----|----|----|----|----|----|----|\n| Uniform Pruning | Attention|168|168|168|168|168|168|168|168|168|168|168|168|\n| Uniform Pruning | MLP      |696|696|696|696|696|696|696|696|696|696|696|696|\n| **Flexible Pruning**| Attention|156|156|180|156|180|180|180|192|168|180|180|204|\n| **Flexible Pruning**| MLP      |756|768|696|732|732|648|624|648|684|744|744|756|\n\n## **Q5: does the proposed method work on other ViT-based models?**\nThanks for the comment. Our method is effective with any models that have numerous cascaded ViT blocks, especially in scenarios where data availability is limited. We are currently working on applying this compression scheme to other large ViT-based models and plan to make it available to the community in the future.\n\n## **Q6: the evaluation seems limited which focuses on SAM-B, it will be great to see how the method works on all SAM models.**\nThanks for the valuable suggestion. We are actively seeking additional hardware resources to support our work on compressing the 641M SAM-H model, and we plan to accomplish this in the future. It is a common observation in model compression that larger ViTs have greater parameter redundancy. Therefore, we believe that SlimSAM will yield even better results on SAM-H.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jIvNYqH6Nc", "reviewer": "Reviewer_wJAz", "summary": "In this paper, the authors propose a novel and data-efficient SAM compression method called SlimSAM, which progressively compresses the SAM model by alternately pruning and distillation. They introduce disturbed Taylor pruning to address the misalignment between the pruning objective and the training target, ultimately achieving better performance with ten times less training data than other compression methods.", "questions": "See the weaknesses above.", "limitations": "Limitations are discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed method is highly efficient and reasonable, achieving SOTA performance among different compression methods.\n- The experimental results are comprehensive.\n- This paper is well-written and easy to follow.", "weaknesses": "- To the best of our knowledge, compressing the SAM model aims to make it more suitable for annotation scenarios in edge applications. However, the authors' experiments seem to compare only parameter count and computational cost, without testing the throughput of the compressed model in real-world scenarios. The reviewer hopes to see relevant discussions in the experiments to better demonstrate the method's practical value in real-world applications.\n- The authors achieved impressive performance using only a small amount of data, which raises curiosity about whether this method can achieve further improvements with more data or if this method quickly reaches its performance bottleneck with more training data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1:the authors' experiments seem to compare only parameter count and computational cost, without testing the throughput of the compressed model in real-world scenarios. The reviewer hopes to see relevant discussions in the experiments to better demonstrate the method's practical value in real-world applications.**\nThanks for the valuable comment. Table 1 shows the actual running latency and acceleration ratio. We compressed the large SAM model into an extremely smaller size, achieving a significant speedup while maintaining model capability. This makes SlimSAM ideal for deployment on resource-constrained devices in real applications.\n\nAdditionally, our whole compression process only requires 10,000 unlabeled images (0.1% of the SA-1B dataset). This greatly reduces the need for training data, lowering the costs associated with storing and downloading large datasets. For users who need to compress and fine-tune SAM on their customized data, which is often limited and expensive, our method provides an effective solution.\n\n### Table 1: The Evaluation of Real Running Speed.\n*Note: We measured the latency and speedup ratio of our SlimSAM on a TITAN RTX GPU. The latency refers to the time taken to infer a 1024x1024 image.*\n| **Pruning Ratio** | **Method**| **Latency ↓** |**Speed Up↑** |\n|------|------|-------|------|\n| Ratio=0%    | SAM-H  | 654ms |Faster×1.0|\n| Ratio=0%    | SAM-L  | 386ms |Faster×1.7|\n| Ratio=0%    | SAM-B  | 152ms |Faster×4.3|\n| Ratio=50%   | **SlimSAM-50 (Ours)**|94ms |Faster×6.9|\n| Ratio=77%   | **SlimSAM-77 (Ours)**|76ms |Faster×8.6|\n\n## **Q2:The authors achieved impressive performance using only a small amount of data, which raises curiosity about whether this method can achieve further improvements with more data or if this method quickly reaches its performance bottleneck with more training data.**\nThanks for the valuable comment. In Table 2, we present experimental results using 2k, 5k, 10k, and 20k training data samples. As shown in the table, the model's performance improves significantly with an increase in the amount of training data. In the future, we plan to use the entire SA-1B dataset to train a SlimSAM model and make it available to the community.\n\n### Table 2: Comparision of Training Results Using Varied Amounts of Training Data\n*Note: We evaluated the effect of increasing data on SlimSAM-77.*\n| **Pruning Ratio** | **Training Data** | **Iterations** | **MIoU↑**   |\n|-----------|----------|-----------|-------------|\n| Ratio=77% | 20k | 200k | 68.10% |\n| Ratio=77% | 10k | 200k | 67.40% |\n| Ratio=77% | 5k  | 200k | 64.47% |\n| Ratio=77% | 2k  | 200k | 61.72% |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kR1E4r6DbW", "reviewer": "Reviewer_YejR", "summary": "This paper works on compressing the Segment Anything Model (SAM). The compression method has the advantage of using much fewer training data. It has an alternate slimming framework and the label-free importance estimation criterion, i.e., disturbed Taylor pruning. The alternate slimming framework minimizes the divergence from the original model and enables the intermediate feature alignment by consistent dimensionality.", "questions": "1. In Eq. (4), the paper introduces disturbed Taylor importance, with $\\mathcal{N}$ as the Gaussian noise with zero mean and standard deviation $\\sigma$.  How will the deviation values influence the performance and why 0.01 is a reasonable setting? \n\n2. What is the rational of Eq. (4)? Why adding the Guassian noise can serve as the purpose of evaluating the importance of model weights? \n\n3. How is the 0.1\\% data sampled? Will different sampled sub-dataset influence the performance of the proposed method? If the data is randomly sampled, will better sampling methods such as K-means clustering to select representative data provide better results?\n\n4. The paper tries to reduce the training cost for the pruned model to update the parameters with a subset of the training data. Why not use the whole dataset but with fewer training epochs/itertations, which also reduces the cost? Which way can lead to a better performance? \n\n5. Can this method also be trained on whole dataset? With the entire dataset, will it result a better MIoU performance than the limited data case? \n\n6. In Table 6, is the training epoch/iterations for different data amount the same?", "limitations": "Please refer to the weakness and questions.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "SAM is an important type of image segementation model with many applications. The high computation costs associated with SAM is a huge challenge to be resovled. The paper works on faciliating the efficiency of SAM with fewer training data. It reduces the cost for the inference and pruning. The method is straightforward and not hard to follow.", "weaknesses": "1. Limited novelty. The pruning method simply follows the prediction errors introduced by prior works with an additional Guassian noise for the creation of disturbed image embedding. It's not clear why using the gradient of loss between the original image and Guassian-disturbed image embedding can serve as the importance evaluation metric. The slimming by reducing the divergence between the original and pruned models is also standard. \n\n2. The motivation of the paper is not strong, or the writing of the paper can be improved. It seems to emphasize the data-efficienct compression technique, but the methods do not contain much space to say how this data efficiency is achieved. The benefit of using less training data is also not emphasized. If the goal of data efficiency is to reduce the computation cost to update the parameters of the pruned model, why not use the whole dataset but with fewer training epochs/iterations? \n\n3. Limited evaluation. The paper only provides the results on SA-1B. How is the performance on other datasets? For instance, EdgeSAM provides results on SA-1B, COCO, and LVIS to evaluate both in-domain performance and zero-shot transferability. Can SlimSAM show better results acrooss different datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: In Eq.(4), the paper introduces disturbed Taylor importance, with the Gaussian noise with zero mean and standard deviation. How will the deviation values influence the performance and why 0.01 is a reasonable setting?**\n The noise deviation should be carefully chosen: a too-large deviation can cause the perturbed output to deviate significantly from the original distribution, while a too-small deviation can result in negligible gradients and inaccurate importance scores. Based on our experience, setting the deviation between 0.05 and 0.005 yields satisfactory pruning results.\n## **Q2: What is the rational of Eq. (4)? Why adding the Gaussian noise can serve as the purpose of evaluating the importance of model weights?**\nThanks for the comment, this is indeed a question that needs further explanation.\n\nThe main challenge we face is the misalignment between the pruning target and the distillation target when using hard labels to generate gradient-based importance scores. To address this, we need to use soft labels to compute the importance score. However, soft labels do not generate gradients by loss functions for Taylor importance estimation. To solve this, we added zero-mean Gaussian noise with a small deviation to the soft labels. The disturbed soft label creates sufficient gradients by loss function while keeping the slightly noisy output close to the original. When the batch size is large enough (>=100), the mean of the sampled Gaussian noise across the whole batch approximates zero. Now the parameter with the highest importance score is the one that causes the greatest deviation between the pruned model's output and the soft label.\n\nThe experimental results in our submission show that this method will cause less damage to the model during pruning and does not rely on accurate hard labels.\n## **Q3: How is the 0.1% data sampled? Will different sampled sub-datasets influence the performance of the proposed method? If the data is randomly sampled, will better sampling methods to select representative data provide better results?**\nThanks for the valuable comment. The training set is a **random subset** of the SA-1B dataset, containing 11k images, with 10k used for training and 1k for validation. We trained using multiple different subsets under the same settings and found the difference in final results to be negligible (MIoU difference is less than 0.25%). As you said, using a better method to select the training data may provide better results, but this will lead to more costs and place higher demands on the total amount of data available.\n## **Q4: Why not use the whole dataset but with fewer training epochs/iterations, which also reduces the cost? Which way can lead to a better performance?**\nThanks for the valuable comment. Using the entire SA-1B dataset involves significant storage and download costs. It takes more than **15 days** to download under typical network conditions and requires around **20 TB** of storage space, which is impractical for many users. Additionally, some users prefer to compress and fine-tune SAM on their customized datasets, but such data is often limited and expensive. Our method is highly user-friendly, as it requires only 10k data samples and one 24GB GPU to complete the entire compression and fine-tuning process.\n## **Q5: Can this method also be trained on whole dataset? With the entire dataset, will it result a better MIoU performance than the limited data case?**\nThanks for the valuable comment. Our method can be trained on the whole dataset and will result in an even better performance. In Table 1, we present experimental results using 2k, 5k, 10k, and 20k training data samples. As shown in the table, the model's performance improves significantly with an increase in the amount of training data. However, as explained in Q4, increasing data brings more cost and is less user-friendly.\n### Table 1: Comparision of Training Results Using Varied Amounts of Training Data\n*Note: We evaluated the effect of increasing data on SlimSAM-77.*\n| **Pruning Ratio** | **Training Data** | **Iterations** | **MIoU↑**   |\n|-----------|-----|------|---|\n| Ratio=77% | 20k | 200k | 68.10%  |\n| Ratio=77% | 10k | 200k | 67.40% |\n| Ratio=77% | 5k  | 200k | 64.47% |\n| Ratio=77% | 2k  | 200k | 61.72% |\n## **Q6: In Table 6, is the training epoch/iterations for different data amount the same?**\nYes, except for the training data, all other training settings are exactly the same for a fair comparison.\n## **Q7: The paper only provides the results on SA-1B. How is the performance on other datasets?**\nThanks for the valuable suggestion. To demonstrate the generalization of SlimSAM on other datasets, we evaluated its zero-shot instance segmentation on the COCO2017 validation set. As shown in Table 2, SlimSAM maintains a strong zero-shot capability very close to the original model. Additionally, our entire compression process only requires 10k unlabeled images (0.1% of the SA-1B dataset).\n### Table 2: Zero-shot Instance Segmentation on COCO\n| **Model**| **Training Data** |**COCO AP**| \n|-----|-----|-----|\n| SAM-B  | 11M |43.4% |\n| FastSAM| 220k|37.9% |\n| EdgeSAM| 100k |41.9% |\n| MobileSAM| 100k |41.0% |\n| SlimSAM-50| **10k** |**42.8%**|\n| SlimSAM-77| **10k** |41.4% |\n## **Q8: Difference between our method and standard technique**\nThanks for the comment. The application of standard pruning methods falls short in achieving superior performance due to the unique challenges presented by SAM's coupled structure and our data-efficiency goal. To enhance performance, we propose an Alternate Slimming Strategy to minimize divergence from the original model and enable the intermediate feature alignment by consistent dimensionality which is very effective when data availability is limited. We also propose Disturbed Taylor Pruning to address the misalignment between pruning objectives and training targets. The comprehensive ablation study strongly proves the superiority of our new methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vTiyOeyOv1", "reviewer": "Reviewer_2d2P", "summary": "The authors present a data-efficient method for SAM 39 compression, SlimSAM, comprising of alternate slimming framework and the disturbed Taylor pruning. Comprehensive experiments are conducted, demonstrating the significant superior performance and data efficiency.", "questions": "As mentioned in Weaknesses.", "limitations": "n/a", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The paper is clearly written and well-organized.\n2. The use of alternate slimming that combines pruning and knowledge distillation is innovative.\n3. Comprehensive experiments are conducted.", "weaknesses": "1. There's a typo with $w_i$  on line 112.\n2. The symbol $N$ is assigned different definitions on lines 111 and 187.\n3.Curious about the significant performance drop in Figure 3—could it be related to the setting of N? Would this decline still occur if the epochs were increased to 40?\n4. More experiments are needed to prove that 'SlimSAM preserves SAM’s robust segmentation capabilities'.\n5. How does the model perform in terms of generalization? Could the authors conduct evaluations on additional datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: There's a typo with $w_i$ on line 112.**\nThanks for the careful review. We will correct this error in the next version and check the rest of the submission.\n\n## **Q2: The symbol *N* is assigned different definitions on lines 111 and 187.**\nThanks for the valuable feedback. We apologize for the oversight. We will clarify the definition in the next version.\n\n## **Q3: Curious about the significant performance drop in Figure 3.**\nThanks for the comment, this is indeed a question that needs further explanation.\n\nThe significant performance drop in the process is due to model pruning. Our alternate slimming strategy involves a two-stage compression method. First, we prune the model's embedding dimension and align the bottleneck dimension through distillation from epoch 1 to epoch 20. After that, we prune the bottleneck dimension and align the embedding dimension through distillation from epoch 21 to epoch 40. The performance drop occurs after epoch 20 when the bottleneck pruning is implemented.\n\n## **Q4: More experiments are needed to prove that 'SlimSAM preserves SAM’s robust segmentation capabilities'.**\nThanks for the valuable feedback. To further demonstrate SlimSAM's robust segmentation capabilities, we thoroughly evaluated its zero-shot instance segmentation on the COCO2017 validation set. As shown in Table 1 below, SlimSAM maintains strong segmentation performance across small, medium, and large areas, indicating its ability to handle objects of various sizes. We also provide comprehensive qualitative results in the supplementary materials, showing that our model delivers segmentation results similar to the original model across different prompts and scenarios. Notably, SlimSAM achieves this using only 10k unlabeled images (0.1% of the SA-1B dataset). Increasing the amount of training data will yield even better results.\n\n## **Q5: How does the model perform in terms of generalization? Could the authors conduct evaluations on additional datasets?**\nThanks for the comments. To demonstrate the generalization of SlimSAM on other datasets, we evaluated its zero-shot instance segmentation on the COCO2017 validation set. As shown in Table 1, SlimSAM maintains a strong zero-shot capability very close to the original model, despite a high compression ratio.\n\n### Table 1: Zero-shot Instance Segmentation on COCO\n| **Model**| **Params ↓** | **MACs ↓** |**COCO AP ↑**|**COCO AP ↑(small area)**|**COCO AP ↑(medium area)**|**COCO AP ↑(large area)**|  \n|------|------|------|-------|------|------|-------|\n| SAM-B         | 93M  | 372G |43.4% |28.2%|47.4%|58.7%|\n| **SlimSAM-50**| 26M  | 98G  |42.8% |27.3%|46.8%|58.5%|\n| **SlimSAM-77**| 9.1M | 23G  |41.4% |26.2%|45.9%|57.4%|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NjBC5iBxE3", "reviewer": "Reviewer_5dUf", "summary": "This paper introduces SlimSAM, a novel data-efficient method for compressing SAM. The authors propose an alternate slimming framework and disturbed Taylor pruning to enhance knowledge retention and compression performance with minimal training data. The method progressively prunes and distills distinct sub-structures, achieving significant performance improvements with only 0.1% of the original SAM training data. SlimSAM is validated through comprehensive experiments, showing superior performance and efficiency compared to existing SAM compression methods.", "questions": "See weakness.", "limitations": "Not applicable.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The alternate slimming method with dynamic alignment stratey seems interesting and reasonable.\n2. Extensive experiments have been conducted to demonstrate the effectiveness of the proposed method.\n3. The design of disturbed Taylor importance criterion is inspiring.", "weaknesses": "1. It would be better to analyze the impact of different Gaussian distributions on the pruning performance.\n2. The impact of some hyperparameter settings on the pruning performance is under-explored. \n3. Since the method combines both pruning and knowledge distillation techniques, it would be better to analyze the separate contributions of the two techniques.\n4. Since the framework seems to be a progressive pruning manner and the pruning ratio of each dimension is a global preset value, I think splitting the two-stage pruning scheme into iterative pruning schemes would benefit the aggressive pruning, where the fine-tuning epochs and pruning ratio in each iteration are adjusted to make its compression cost equivalent to that of the two-stage pruning. So I am interested in the comparison of these two schemes.\n5. There are several confusing symbol definitions, e.g., $N$ with two different usages in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## **Q1: It would be better to analyze the impact of different Gaussian distributions on the pruning performance.**\nThanks for the valuable suggestions. The mean $\\mu$ of Gaussian noise must be set to 0 so that the average of the sampled noise across the entire batch approximates zero. The Gaussian noise deviation $\\sigma$ should be carefully chosen: a too-large deviation can cause the perturbed output to deviate significantly from the original distribution, while a too-small deviation can result in negligible gradients and inaccurate importance scores. Based on our experience, setting the deviation between 0.05 and 0.005 yields satisfactory pruning results.\n\n## **Q2: The impact of some hyperparameter settings on the pruning performance is under-explored.**\nThanks for the comments, this is indeed a question that needs further explanation and improvement.\n\nIn our method, the **batch size** is set to 4, constrained by GPU memory (24GB). A larger batch size would result in a smoother training curve and better outcomes.\n\nThe **learning rate** starts at 1e-4. If the model shows no performance improvement on the validation data for three consecutive epochs, the learning rate is halved.\n\nThe total **training iterations** are 100k for SlimSAM-50 and 200k for SlimSAM-77. Increasing the iterations provides little improvement for SlimSAM-50 but can slightly enhance the performance of SlimSAM-77.\n### Table 1 Effect of Different Training Iteration\n| **Pruning Ratio** | **Training Data** | **Training Iters** | **MIoU↑**   |\n|-----|-----|-----|-----|\n| Ratio=50% | 10k | 50k  | 70.83% |\n| Ratio=50% | 10k | 100k | 72.33% |\n| Ratio=77% | 10k | 100k | 64.43% |\n| Ratio=77% | 10k | 200k | 67.40% |\n\nTable 2 shows the effect of different **loss weight** settings during training. We use constant weights during stage 1 and dynamic weights during stage 2. Here, **n** represents the nth epoch, while **N** is set to half of the total training epochs. A smaller **N** leads to slower convergence, while a larger **N** results in worse performance.\n### Table 2 Effect of Dynamic Weights Setting\n| **Step** | **Model**| **Constant (α=0.5)**| **Dynamic (α=(N-n-1)/N)**|\n|--------|--------|-------|------|\n| 1 | SlimSAM-50 | **MIoU: 72.07%** | MIoU: 71.60%|\n| 1 | SlimSAM-77 | **MIoU: 66.32%** | MIoU: 65.79%|\n| 2 | SlimSAM-50 | MIoU: 70.42%     | **MIoU: 70.83%**|\n| 2 | SlimSAM-77 | MIoU: 63.63%     | **MIoU: 64.48%**|\n\n## **Q3: Since the method combines both pruning and knowledge distillation techniques, it would be better to analyze the separate contributions of the two techniques.**\nThanks for the comments. Both pruning and distillation are indispensable in our method, as using them together is necessary for achieving satisfactory compression results.\n\nWithout knowledge distillation, the final model performance is very poor (MIoU less than 30%) if the pruned model is trained with labeled data. This is because the complex coupling structure of SAM's image encoder and mask encoder requires a large amount of training data and batch size for effective training. This issue is also noted in the MobileSAM [1] paper.\n\nOn the other hand, if we only use distillation without pruning and try to train a lightweight model with the same structure from scratch using such limited data (10k), the training results do not converge at all (MIoU less than 2%), as reported in Table 1 of our submission.\n\n[1] Zhang, Chaoning, et al. \"Faster segment anything: Towards lightweight sam for mobile applications.\" arXiv preprint arXiv:2306.14289 (2023).\n\n## **Q4: I think splitting the two-stage pruning scheme into iterative pruning schemes would benefit the aggressive pruning, where the fine-tuning epochs and pruning ratio in each iteration are adjusted to make its compression cost equivalent to that of the two-stage pruning. So I am interested in the comparison of these two schemes.**\nThanks for the valuable comments. This is a quite interesting idea.\n\nWe experimented with splitting the two-stage pruning of 50% of the dimensions into iterative pruning at 10%, 20%, 30%, 40%, and 50%. We observed two key findings:\n\n1. With the same number of total training iterations, iterative pruning produces better results, especially when the number of iterations is low. However, this advantage gradually diminishes as the number of iterations increases.\n\n2. Iterative pruning takes more training time when the number of total training iterations is the same.\n\nCarefully tuning the hyperparameters for iterative pruning might yield even better results, but this requires additional effort and expertise.\n\n## **Q5: There are several confusing symbol definitions, e.g., N with two different usages in the paper..**\nThanks for the careful review. We apologize for the oversight. We will correct this error and review the rest of our submission for any similar issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers, Chairs, 

We sincerely appreciate the time and effort you have spent evaluating our submission, and we look forward to the discussion stage. We will include the review stage results in the appendix of our next version.

Meta-review: The submission introduces a novel method SlimSAM for compressing the Segment Anything Model (SAM) with an emphasis on data efficiency. SlimSAM employs an alternate slimming framework that combines progressive pruning and knowledge distillation, alongside a new disturbed Taylor pruning technique. This framework is designed to achieve competitive performance while using significantly less training data. Extensive experiments demonstrate that SlimSAM reduces parameter counts, MACs, and training data requirements by impressive margins while maintaining performance close to the original SAM.

The reviewers collectively appreciate the novelty and practical significance of SlimSAM, particularly its ability to achieve competitive performance with minimal training data. The evaluation and ablation studies are robust and provide compelling evidence to support the authors' claims. Furthermore, during the rebuttal phase, the authors effectively addressed concerns related to dataset generalization, real-world applicability, and the clarification of technical details. Given the strengths of this work and the authors' responses, SlimSAM makes a good contribution to the field of model compression and efficient segmentation models.
Decision: Accept (poster)