Adaptive Pruning of Pretrained Transformer via Differential Inclusions
Authors: Yizhuo Ding
Ke Fan
Yikai Wang
Xinwei Sun
Yanwei Fu
Venue: ICLR 2025 Poster (2025)
Keywords: Pruning
Sparse optimization.
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We adaptively pruned pretrained transformer based on a differential inclusion.

Abstract: Large transformers have demonstrated remarkable success, making it necessary to compress these models to reduce inference costs while preserving their performance. Current compression algorithms prune transformers at fixed compression ratios, requiring a unique pruning process for each ratio, which results in high computational costs. In contrast, we propose pruning of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. Therefore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization.In this paper, weintroduce such an effective pruning method, termed SPP (Solution Path Pruning). To achieve effective pruning, we segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low-rank compression to these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP.

Published as a conference paper at ICLR 2025
ADAPTIVE PRUNING OF PRETRAINED TRANSFORMER VIA DIFFERENTIAL INCLUSIONS
Yizhuo Ding, Ke Fan, Yikai Wang Xinwei Sun Yanwei Fu School of Data Science, Fudan University (yzding22, kfan21 @m. fudan. edu en; yi-kai wakkaii.wngoouulook.. com; (sunxinwel, yanwe fu) ) @ fudan. .n
ABSTRACT
Large transformers have demonstrated remarkable success, making necessary to compress these models to reduce inference costs while preserving their perfor- mance. Current compression algorithms prune transformers at fixed compression ratios, requiring unique pruming process for each ratio, which results in high com- putational costs. In contrast, we propose pruning of pretrained transformers any desired ratio within a single pruning stage, based on differential inclusion for a mask parameter. This dynamic can generate the whole regularization solution path of the mask parameter, whose support set identifies the network structure. There- fore, the solution path identifies a Transformer weight family with various sparsity levels, offering greater flexibility and customization. In this paper, introduce such an effective pruning method, termed SPP (Solution Path Pruming). To achieve effective pruning, segment the transformers into paired modules, including query-key pairs, value-projection pairs, and sequential linear layers, and apply low- rank compression these pairs, maintaining the output structure while enabling structural compression within the inner states. Extensive experiments conducted on various well-known transformer backbones have demonstrated the efficacy of SPP. Our code is available at https://gthbb.om/yyyhoobbb/oob Pattiiinn-P---PPnnnng Pruninr
INTRODUCTION
Transformers have succeeded in various tasks due to their scalability, parallel processing abilities, and capacity to learn complex data patterns. Pretrained transformers, in particular, perform well on downstream tasks by leveraging large datasets during training. These models are highly effective for transfer learning, allowing fine-tuning for different tasks and delivering strong performance in many natural language processing applications. However, their large size, often with billions of parameters, makes them difficult deploy on low-cost hardware. Despite their widespread use and impressive performance (Radford et al. 2021 Touvron et al. 202 la). running transformers on lightweight devices like phones and laptops remains challenging due computational constraints. Therefore, compressing transformers to run efficiently on affordable hardware is crucial.
Various techniques have been developed to compress transformers while preserving their performance.
These include weight sharing (Lan et al. 2019). low-rank factorization (Yu et al.) 2017). quantization
(Gong al.
[2014] Polino et al. [0188] [ lao et al.) [2022), )) knowledge distillation (Hinton al. 2015)
Yuan et al.)
2019; Touvron et al. [220 Liu et al. 2020), and pruning (Yu & Xiang 2023 Yang
ctt al. 2023 Shi al. 2023 Yin et al. 2023). Pruning methods typically involve structured pruning,
wherein entire neurons or attention heads are removed based on importance, followed by fine-tuning to
regain accuracy. Despite the array of pruning methods developed for transformers, current algorithms face a fundamental challenge: they are tailored achieve predefined pruning ratio by the end of the pruning phase.
Typically, when targeting new target. Restarting such an entire pruning process for each level of sparsity introduces high costs for model compression. For over-parameterizee models, the training cost can be substantial.
degree of sparsity, the entire pruning process is restarted meet the
Cooresponding authors. Prof. Yanwei Fu is also with Fudan ISTBI-ZJNU Algorithm Centre for Brain- inspired Intelligence, Zhejiang Normal University, Jinhua, China.

---

Published as a conference paper at ICLR 2025
Fx
fr
Upditt Siape
Finetune Stage
Finetune Stage
(e) Our method
Lasso method
Figure I: Comparison of SPP and lasso method. (a) SPP can obtain sparse models of all sparsity after search stage. which includes update stage and prune stage. (b) Lasso method can only obtain one sparse model in single search stage.
To address this challenge, we propose adaptive compression strategy through differential inclusion for mask-based pruning of pretrained transformer, named SPP (Solution Path Pruning). The dynamics where where each mask's support captures the important weights. As is shown in Figurc] along this solution trajectory, the sparsity of the projected target model incrementally increases, with important weights learned earlier in the dynamics until convergent to the fully dense network. Besides, this dynamics has a simple iterative form to implement. Therefore, after running whole iteration, we can obtain a transformer weight family with different compression ratios. Note that Fu et al. (2020) used similar pruning method for structured sparsity in neural networks trained from scratch. They also used differential inclusions with inverse scale spaces to train the network. Our focus, however, is pre-trained transformers, which are much harder to prune while maintaining performance. Unlike common pruning methods, SPP does not require restarting the search stage, rendering it cost-effective pruning technique requiring just one search stage to derive a transformer weight family with diverse sparsity levels from the uncompressed model.
efficiently generates whole regularization solution path of masks,
Our exploration of transformer structures adopts a novel and efficient approach, automatically learning compression structures throughout training, thus enjoying greater flexibility and customization. Currently, no other algorithm can theoretically guarantee optimization convergence while producing sparse models with different levels of sparsity during training. For example, the Upop (Shi al.) 2023) algorithm only makes the model structure sparse at the final step, with the mask becoming zero only at that point. The OFA (Cai et 2019) method lacks convergence guarantee and simply fixes the mask updates during training to achieve sparse network. SPP addresses this by using the solution path to obtain sparse structures without affecting convergence, providing a stronger theoretical foundation. Notably, our method can be extended to prune large language models (LLMs) with one-shot post training.
In this paper, applied SPP to the classification task dataset ImageNet (Deng et al. 2009) using the DeiT (Touvron et al., 2021a) backbone. Furthermore, we also extended the method to image and text retrieval datasets, using CLIP models (Radford et al. 2021) Our contributions are summarized as follows:
We develop differential inclusion-based method for adaptive compression of pretrained trans- former, enabling the acquisition of sparse models with different sparsity within single search stage, significantly reducing the cost of model pruning.
We introduce the novel concept of the Transformer Weight Family, obtained through a simple
iterative algorithm following the discretization of the differential inclusion.
We also prove the global convergence of the method in such non-convex optimization based on the Burdyka--ooasiewiiz framework, demonstrating that the entire iterative sequence converges to a critical point of the empirical loss function from arbitrary initializations.
We demonstrate the effectiveness of our framework across various backbones and datasets. Results show that we can significantly reduce the computational costs while preserving the prediction accuracy.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Transformer pruning. In the expanding field of Transformer-basee model research, the concept of pruning Transformer has garnered considerable interest for its potential to streamline model architectures.
Over recent years, various structured pruning methods have been developed. ViT-Slim (Chavan et al. 2022) employs single-shot training approach to search for optimal sub-structures in vision transformers. SAViT (Chen et al., 2021) collaboratively prunes all components of Vision Trans- formers, integrating structure-aware interactions and using a Taylor-based optimization function for joint importance. UP-ViTs (Yu & Wu 2023) introduces a unified pruning framework for vision transformers, focusing on comprehensive pruning of all components of VIT models and their variants.
The process of pruning a Transformer model is twofold: first targeting the MLP and then the Attention mechanism. Approaches such as WDpruning (Yu et al. 2022) employ a mask-based technique. Specifically, a mask M is defined to correspond to each column of the MLP's weight matrix, and pruning conducted by considering the gradient magnitudes of this mask. However, due to the complicated structure of the Attention mechanism, such strategies may fail.
WDpruning (Yu et al., 2022) extends its paradigm by incorporating head-level pruning within the multi-head attention framework. Concurrently, methodologies like SAVIT and Upop advocate for the pruning of the input projection matrices, retaining the structural integrity of the query. key, and value matrices. Those approaches, however, lack flcxibility and expandability. Because the matrices for the query, key, and value play distinct roles during the forward pass of the attention process.
Our methodology introduces innovative approach to pruning within the attention paradigm. It enables an asymmetric dimensionality between the query, key, and value matrices after pruning. allowing for more nuanced and efficient pruning process. This novel technique does not necessitate the uniform dimensionality across these matrices, thereby enhancing the pruning mechanism's 's flexibility and applicability diverse Transformer-based architectures.
Mirror Descent Algorithm and Bregman Inverse Scale Space. Mirror Descent Algorithm (MDA) was first proposed by Nemirovskij & Yudin (1983) solve constrained convex optimization, and can be seen as a generalized projected gradient descent (Beck & Teboulle 2003) using Bregman distance Ba(u, v) = (2())= (2(v)= (V)(v), v) induced by a convex and differentiable function $2(.).
Convergence analysis for convex loss has been extended to stochastic versions (Ghadimi & Lan 2013) and Nesterov acceleration (Krichene et al.] 20155) (Su et al. (0166). For non-convex loss deep learning, convergence to global optima for overparameterized networks has been established (Azizan al. 2019). For non-differentiable penalties, such as & penalty for sparse recovery, Osher et al. (2016a); So et al. (2008) proposed Linearized Bregman Iteration (LBI,, follows discretized solution path of differential inclusions called Bregman Inverse Scale Space (ISS). These solution paths enjoy the inverse scale space property, which means important features such as the signal, will be learned earlier than non-important ones such as noise.
Mask-based pruning for pretrained CNN Fu et al. 2020 2022 Bungert et al. (022 applied LBI to forwardly train a network from scratch, based on the lottery ticket hypothesis. By incorporating several training techniques tailored the network architecture, the sparse network achieved com- parable results to the fully dense network. In contrast, we propose a new differential inclusion for mask-based pruning, which can adaptively prune pre-trained transformer. This method generates an iterative solution path that uncovers key sparse architectures early during training. Our method can perform consistently well across various backbones and datasets. Unlike ADMM (Wahlberg et al. 2012 Boyd et al. (011). which focuses on convergence, our differential inclusion dynamics aim at generating a whole solution path with various compression ratios.
3 METHOD
Problem setup. Given the model weights W, inputs X, and the objective C, the target of pruning is minimizing the size of model and keeping the performance of the model. i.e.
min L (X.W) s.t. p(W) < p. W
(1)
3

---

Published as a conference paper at ICLR 2025
where pWW is the sparsity level of model, p E (0, 1] is the desired degree of sparsity. Straightforward unstructured pruming simply discards unimportant weights in the architecture, but this approach often fails to meet the requirements for accelerating computation reducing memory costs. In contrast, structural pruning reduces the complexity and computational cost of neural networks by removing entire structural units, making it more suitable for hardware acceleration and practical deployment. In this paper, we focus on the problem of structural pruning.
Transformer weight family. Our goal is efficiently obtain a family of neural networks with different sparsity levels. To achieve this, we propose dynamic approach based on differential inclusion induced by l penalty. This dynamic can generate whole regularization solution path from sparse to dense, with important weights learned earlier. Besides, it enjoys simple iterative form to implement, which identifies weight family with different sparsity levels. Such a weight family can be obtained from single search stage, eliminating the need to restart the search process. This method cost-effective compared to previous pruning methods.
3.1 MASK- -BASED PRUNING
Structural weight importance mask. Without loss of generality, suppose the weight matrix is W E Rmxd where is the feature dimension we want to prune. We introduce the mask matrix M = (maski.--. ) [0, 1]_xd for every weight matrix of the transformer models with the same matrix size, where mask; indicates the importance of corresponding column. This column- wise design naturally achieves structural pruning by discarding columns of the weight matrices. Subsequently, the masked network is defined by:
C(W, M) L(W 3 M).
where @ denotes the Hadamard product. Our target for the mask-based structural pruning task is to learn the M to identify the important columns while discarding the others.
Pair-wise shared mask. In transformers, adjacent layers typically coupled in the feature dimensions. When considering them separately, manual efforts like padding are required to avoid dimension mismatch issues. To address this, many pruning approaches use shared mask for an entire module, such as attention layers. While this ensures dimensional consistency within the same layer, it is conservative and lacks the potential for more fine-grained pruning.
To achieve both dimension matching and pruning flexibility, we propose pruning at the smallest pair- wise level. This approach maximizes flexibility without causing dimension mismatches. Specifically, transformers are primarily based on multi-head self-attention (MHSA) layers and feed-forwart MLPs. We suggest dividing MHSA into query-ket and value- output pairs while considering the adjacent linear layers in MLPs. Given the standard MHSA as,
A softmax (XWQ, (XWK..) /Vd), V, = A(WWV,,), OMHSA V.W projids (3)
Σ
where i is the i-th head, WK, E Ruxd WQ,+ Rmndd Wv, E gmxdd Wproj,t Rdd weight matrices of the Query, Key, Value and output projection, respectively. The m and d denotes the dimension of the input features and the hidden feature, respectively.
d Xm are
Our proposed pair-wise shared mask introduces the weight importance mask via,
Wo , WK. + WQ.+ @ M@K. , WK.* @ MoK.
(4a)
WV,,, W Wproj. + WV. O My, MY @ Wpros,
(4b)
where MoK E [0, 1] matrix are shared in the query-key pair and in value-output pair. Similarly, for the feedforward module,
Fixxd,Mo My E [0, 1|2xd and || llo & d, ||Mvllo < d1. The same mask
FFN(X)= & (X W
Woutput: W Winput E Raxd W Woutput E Rxd b is activate function, (5)
we assign shared mask MMLP E glxd to prune Winput and W Woutput:
Winput: Woutput + Winpur C MMLP, MLP C Woutput
(6)
Sparse optimization of masks. To minimize information loss in the pruned model. a common objective is to ensure that the weights being pruned gradually approach zero during the search phase.
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Transformer Weight Family
Perform searching
Input: Pretrained weight W and step size a, iteration steps in the update stage T, and prune stage Σp
Initialize sub-gradient Vo = O, mask Mo = L, sparse mask To 0.
for k = 0 to T. do
Calculate the loss
1 = L(Wo @ Mc) + 2u M. - 1'x112
# update V2 and mask Mr according to sub-gradient
Ma+1 = M4 ROa R M
V(+1 = V2 ae V
F update Ts as the proximal operator with penalty M() X|[I] (T)
PEL1 = PRoXA (Ve+1), where Pv) = argminr (IIV - /
end for
Perform pruning
for
===00 to Tp do
$ Update masks with reverse turn of P being non-zero in searching
Ma+1 = = k int(T, (k (k+1)) 1)
$ Update and save the sparse model weights
WK+1 = Wk+1 @ M4+1
4Save the checkpoint of WK+1 as the pruned model
end for
Return Weight Family for the model
Output: Transformer Weight [W/]i [0,
To achieve this, it is essential to add a regularization term, specifically, the L1 loss of the mask, to the loss function in the optimization algorithm. The updated loss function can be formulated below:
Z(W, M) C(W @ M) + X || + (M),)(MM),
where the indicator function of interval [0, 1] restrict the mask to a meaningful range. And we initialize M with all-one values. However, the Lasso optimization method insufficient for adaptive compression. This limitation arises because the solution path of Lasso only exists when the optimiza- tion function is linear. As result, Lasso can only produce model with a certain level of sparsity, dependent on the hyperparameters. Therefore, we need to find an alternative method to effectively address the sparsification problem.
3.2 DIFFERENTIAL INCLUSION FOR REGULARIZATION WEIGHT FAMILY
Given pre-trained weights Wo, aim to obtain weight family with different sparsity. To effectively achieve this, we consider the following dynamics:
Mr V Va
(8a)
A
V/2 = (M,,
(8b)
V, E a
(8c)
where V is sub-gradient of SQ(T) ||T||2 + 1 (.. 1. [[|; and is damping factor to ensure the right continuity of the solution path. Since M cannot be initialized to zeros, we introduce
augmented parameter I and enforce it be sparse. To learn important weights, we also enforce r to be close to M through an f2 penalty in the following appended loss:
(,, L(W, O M) + LII r (p > 0).
(9)
Remark 1. Eq. 8 is a special form of a mirror descent flow with Q(T). Unlike previous works for mirror descent primarily concerned with convergence analysis, main objective is to obtain regularization solution path, where each solution along this solution corresponds to a sparse mask that defines the network's sparse structure.
The differential inclusion in Eq generrates a solution path for (M,, where M, follows gradient descent flow to learn the activation values for the pre-trained weights Wo. while T2 is updated via
5

---

Published as a conference paper at ICLR 2025
mirror descent flow with the penalty Q(T)) to explore the sparse structure. During the updating process of c,, important weights are learned earlier than non-important ones. Essentially, this is
known the inverse scale space property in inverse problems (Burger et al. 2006: Osher al.
2016b).
Similarly, starting from To O, o, identifies family of network structures, where important weights will be learned in earlier epochs. Specifically, note that for each element i, [V( < 1, if) 0. and is equal that gives dense solution lim M,, re) = arg min (,,,, I), V2 that begins from Vo= 0,
when it becomes non-zeros. Therefore, driven by the gradient descent Eq. 8b
will have more of its elements reaching the boundary of making corresponding elements in r becoming non-zeros. Compared to directly solving 7 our dynamiic is more efficient in generating a network family with various sparsity levels and, hence more flexible for deployment.
The differential inclusion has been previously explored forwardly train a sparse network from scratch [Fu et al. 2020; 2022 Bungert el al.) (022). incorporating several deep learning techniques tailored the multilayer perceptron and the convolutional neural network. However, these techniques may not be applicable to the Transformer, and thus may lead to suboptimal performance. In contrast, our dynamics involves backward pruning of a pre- trained network. Furthermore, instead of weight-baun based pruning, we employ mask-based pruning, which is significantly easier to train.
Remark 2.
To implement, we consider the following iteration, beginning from Vo = To = 0, and Mo = 1:
Mg+1 = MM kak V VM (M,..
(10a)
Va+1 = V/x a (Mr,
(10b)
T&+1 = R. (Voxx(Vi++).
(10c)
where a is the step size and should be small enough to well approximate the original dynamics in Eq. In Eq. 10b. the proximal operator Proxa(-) is defined as:
{
Pvoxa(V) = arg min r
2 | - V +
(11)
which gives ((t) = (f) > 2 V()) - if < V < 2, = otherwise. Running Eq. 10
-
will obtain a solution path of P& with increasing levels, that define family of weights dubbed as Transformer weight family:
W O..
(12)
CONVERGENCE
We present theorem that guarantees the global convergence of method, i.e. from any initializa- tion, the sequence converges to a critical point of L. Our treatment extends the Block Coordinate Descent (BCD) studied in Zeng et al. (2019). with a crucial difference being the mirror descent involved in our method. Instead of the splitting loss in BCD, new Lyapunov function is developed here to meet the
property (Lojasiewicz, 1963). Let P == T), Following Huang & Yao (2018), our method algorithm can be rewritten as the following standard linearized Bregman iteration,
Pk+1 arg min PP.PP. (L (B
(13)
where
+ 1 = 12x
1
1 Im
PPP)=0A00 =
W
2x
2%
2x
Pk (V)())) and B is the Bregman divergence associated with convex function V, defined by
= v(P) P Q)
(14)
for some E (Q). Without loss of generality, consider > = 1 in the sequel. One can establish the global convergence our method under the following conditions.
Ei= Lly,, &w (x.)) is continuous differentiable and V is Lipschitz continuous with a positive constant Lip: (b) has bounded level sets; (c) L(W) lower bounded (without loss of generality, we assume that the lower bound 0): (d) S. is proper
Condition 1. Suppose that: (a) L(W) =

---

Published as a conference paper at ICLR 2025
lower semi-continuous convex function and has locally bounded subgradients. that is. for every compact set S C. IR", there exists a constant >> 0 such that for all I and all g E (T) there
holds |[9|| - C: and (e) the Lyapunov function
F(P,g) = (an]==C((M,))-) + BR, [',
(15)
a hoaitikaa--ojssiiieeefeeeteeee on any bounded set, where (T, := -()
I E (g), and Or is the conjugate of $2 defined as
( (g) == sup { - Q(U))
UER
Now we are ready to present the main theorem.
Theorem 1. [Global Convergence of SPP] Suppose that Assumption I holds. Let (Wa, be the
sequence generated by Our method (Eq. (TO)) with finite initialization. If
2
0 < ar
(16)
s(Láp Clcipp+++ C+
where C= max |Wol is value of the pretrained model then (M,, Rx) converges critical point of f. defined in Eg. 10). and {M*} converges to critical point of L(W).
5 EXPERIMENTS
In this section, we evaluated our method on three transformer models. These models were trained on two classical datasets, ImageNet-Ik, COCO and a smaller dataset, CIFAR-IO. We also conducted ablation studies to highlight the importance of introducing the mask parameters and gamma buffer. Additionally, we extended our method to large language models (LLMs) such as Llama2-7b and OPT-6.7b. More experimental details are provided in Table Table
Experimental setting. Our method begins with a pretrained model, keeping all parameters unchanged except for the mask parameters. The search stage is performed only once for all degrees of sparsity. From the solution path, we derive a sparse transformer weight family by applying early stopping and saving the model checkpoint when the desired sparsity is achieved. As described in Algorithm [ each model produces a transformer weight family with sparse architecture.
5.1
MAIN RESULTS
We reported our results pruning DeiT, Swin and CLIP model with various degrees of sparsity.
Transformer weight family results of DeiT. Table shows our transformer weight family results of DeiT on ImageNet- lk. The results show that our proposed method performed well on DeiT. For DeiT- Small, method maintained accuracy 80.2% while reducing 29.5% of the parameters. Compared to the other methods, WDpruning only achieved 78.6% accuracy with a 29.6% compression ratio. Only UPop (Shi et al. (023) achieved comparable results ours. This indicates that our method of solution path is highly effective, preserving more parameter information even at higher compression ratio.
Our method performed well on DeiT-Base. At almost the same compression ratio, our accuracy surpassed SCOP (Tang et al. 2020) and PoWER (Goyal et al. 2020), and we achieved a better com- pression ratio than IA-RED (Pan et al. 202T), the smallest model among the compared algorithms.
Transformer weight family results of CLIIP Apart from the classification task, we applied our method to image and text retrieval tasks. Using the CLIP backbone, compress the transformer module to different sparsity levels, as shown in Table 3 Our method is robust, maintaining high recall under different compression ratios. Notably, our method excels in Image-to- toxTet retrieval, with only a 1 % performance drop while using around 60% of the FLOPs of the full models.
SPP also demonstrates its capability for low-cost pruning. In the CLIP case, after 6-epoch search stage, we obtained sparse model architectures with different sparsity levels. Each sparse model only needs be retrained for 5 epochs achieve good performance.
7

---

Published as a conference paper at ICLR 2025
Table 1: The results of (Tuuroo et al. 2021b) models. We compared the results with the other advanced methods. The reduce is the reduce of parameters.
Model
Method
Top-1 %)
Top-5(%) FLOPS(B) Params(M)
Uncompressed
79.8
95.0
1.6 1.6100%
22. 100%
S ViTE-
79.2
14. 19.666..15 1%%
GOHSP
80.0
3.0g5.29 2%
14.4a5 2%
PS- PS-ViT-S
79.4
2.758.7%
22. 22.0g9.6%
ViTAS E
77.4
93.8
2.7 287.8.1%
12.657.00 0%
Upop
79.6
94.8
2.8.0.9%
13.5g1.1%
DeiT-Small
Upop
80.2
95.1
3.2g0.6%
15.771.0%
ViT-Slim
80.0
95.1
3.371.7%
15.7 15.771.0%
WDPruning
78.6
94.4
3.187.496
15.0g7.00 99
WDPruning
78.4
94.1
2.636.5%
13.3go
X-Pruner
78.9
94.2
2.452.295
OPTIN
79.2
3.2g8.495
SPP
80.2
95.1
3.47.99%
15.670.0%
SPP
78.9
94.6
2.630.0%
12.6s7.0%
SPP
77.7
94.0
1.941.395
10.4,7.19
Uncompressed
72.2
91.1
1.3
5.7
GOHSP
70.2
0.9g0.2%
4.070.2%
ViTE
70.1
1.07G.9%
4.2 7.273.7%
DeîT-Tiny
WDPruning
89.8
0.753.8%
PoWER
69.4
89.2
0.8g1.5%
UPDP
70.3
0.970.35
3.8
MCF
71.5
0.773.880
3.9
OPTIN
71.3
0.9
SPP
72.3
91.1
0.8g1.6%
4.0 70.0
Uncompressed
81.8
95.6
17.5 17.5
86.6100%
ViT-B/16
77.9
95.3
5 10.5
86.8 000
SCOP
79.7
94.5
10.2 10.238.3%
58.3 3%
DeiT-Base
IA-RED
80.3
11 4..7.4%
67 67.077.49
PoWER
80.1
94.6
10. 10.4go.49
X-Pruner
81.02
95.38
8.5g8.0%
SPP
81.9
95.7
9.8 0%
48 1s 5%
SPP
81.2
95.4
6,939. 49
239 5%
SPP
78.1
93.8
4.42s.1%
22.026.4%
Table 2: Results of Swin-Tiny and DeiT-Tiny Imagenet- 1k, together with result of evaluating DeiT-Small on Cifarl0.
Model
Method
T-1( %
T-5(%)
FLOPS(B)
Pa. (M)
Uncompressed
28.0
Swin-Tiny
ViT-Slim
80.7
95.4
3.476.6%
19.4
SPP
80.6
95.2
3.470.6%
18.5aa.
Uncompressed
72.2
91.1
1.3
5.7
GOHSP Yin et al.
2023
70.2
0.969.29
4.070.2%
DeiT-Tiny
SP
al.
1.076.9%
4.273.7%
WDPruning Yu ct al
2022
70.3
89.8
0.7
PoWER Goyal et al.
2020
69.4
89.2
0.861.5%
SPP
72.3
91.1
0.861.8%
4.070.2%
Uncompressed
98.5
4.6
22.1
ViT- ViT-Slim/Chave Chavan
2022
98.7
3.371.1%
15.670.6%
DeiT-Small
WDPruning Yu et all
2022
98.1
2.8.p.9%
14.967.4%
SPP
98.8
3.371.
15.4g9.7%
8

---

Published as a conference paper at ICLR 2025
Table 3: Ablation: The results of -lre- and CLIP-base.
Image->Text
Text=>
Model
Method
Params(M) FLOPS(B)
R@1
R@5
R@10
R@1
R@5
R@10
Uncompressed
71.5
90.8
95.4
80.7
87.6
856.0100%
395.7 100%
73.7
92.5
96.2
55.6
79.1
85.7
80794.3%
376.895.25
Large
71.9
91.6
95.6
55.5
79.3
86.3
757 4%%
353.389.3%
SPP
70.4
90.7
95.3
55.5
80.6
87.8
699
324.8g2.1
70.8
90.9
95.5
54.6
80.1
87
650
299.376.4%
70.3
90.5
95.3
52.5
78.8
86.4
532g2.1%
245. 61.9%
Uncompressed
52.5
76.4
84.2
33.0
57.9
68.7
299 100%
41.2100%
69.0
89.6
94.8
84.5
78.5
86.7
278
38.593.
CLIP-Base
65.9
88.6
94.1
50.0
77.3
86.1
257 357a6.0%
35.5g6.29
SPP
61.9
86.4
93.1
47.0
75.6
84.9
23478.3%
31.977.4%
49.8
78.5
87.3
35.4
65.5
77.2
18160.5%
22.956.69
31,4
61.3
73.7
21.2
49.1
62.5
13043.5
13.6 13.633.0%
Table 4: Comparisons with training from scratch method on pruning DeiT models.
Model
Method
Top-1(%)
Top-5(%)
FLOPS(B)
Params(M)
Uncompressed
79.8
95.0
4.6100%
22. 100%
DeiT-Small
DessiLBI
78.9
94.2
3.20p.6%
15.2os.8%
SPP
80.2
95.1
3.473.9%
15.670.4%
Uncompressed
72.2
91.1
1.3
5.7 100%
DeiT-Tiny
DessiLBI
71.8
90.8
1.076.9%
4.070.2%
SPP
72.3
91.1
0.8g1.59
4.070.2%
Uncompressed
81.2
95.5
4.5100%
28
Swin-Tiny
DessiLBI
80.4
95.2
3.475.6%
18.3a5
SPP
80.6
95.2
3.475.8%
18.566. 1%
Results of tiny models and datasets. We also extended our method to the Tiny models like Swin- Swin-Tiny Tiny and DeiT-Tiny model. Table 2 presents a comparison between our approach and the other method, ViT-Slim (Chavan et al., 2022). Notably, our method achieves only a 0.8% reduction in accuracy while compressing 5% more than ViT-Slim (Chavan et al.) 2022). The success of our method with the Swin-T model, which demonstrates its adaptability across various transformer models.
For DeiT-Tiny, at almost the same compression ratio, our accuracy surpassed SSP (Chen et al. 2021) and V'TE (Chen et al. (022)) by around 2% points, and we achieved a better compression ratio than GOHSP (Yin et al. (023), the smallest model among the compared algorithms.
On CIFAR-1O dataset, shown in Table 2 our method maintained reasonably good accuracy even at higher compression ratio, achieving better accuracy with a smaller model than ViT-Slim, demonstrating its effectiveness across various datasets.
5.2 ABLATION STUDIES
Ablation of training from scratch method. To highlight the improvements of our method over training from scratch, conducted experiments using the DessiLBI method, as shown in Table 4 Both two methods finetuned from the same pretrained model. While the DessiLBI method can be applied to transformers, there was still noticeable performance gap compared to approach. This clearly demonstrated that our method significantly enhances the differential inclusion pruning technique.
5.3 FURTHER STUDIES
The consistency among family. We plotted the solution path of as shown in Figure 2 The parameters do not return zero during training, which shows that the weights within the same family

---

Published as a conference paper at ICLR 2025
Table 5: Results of pruning LLMs. We pruned the Llama2-7B and OPT-6.7B model with 50 % sparsity, then evaluated the pruned model on datascts.
Model
Method
Calib data ARC-c(%) ARC-e(%) (oolQ(%% RTE(%) SST(%)
Uncompressed
43.52
76,26
77.71
62.82
51.95
RIA
38.40
71.59
75.60
54.51
49.77
RIA
Wikitex12
37.97
71.68
75.17
55.96
50.57
Llama2-7B Wanda
C4
37.03
69.70
74.01
55.23
53.10
Wanda
Wikitext2
37.29
69.65
74.28
57.04
51.72
SPP
38.57
71.80
75.96
55.96
49.66
SPP
Wikitext2
37.88
71.59
74.28
54.51
50.11
Uncompressed
30.46
65.57
66.06
55.23
RIA
C4
29.27
63.68
66.82
53.07
61.81
RIA
Wikitext2
29.18
64.10
63.55
52.71
76.49
OPT-6.7B
Wanda
C4
27.39
57.45
63.88
50.90
78.21
Wanda
Wikitex12
26.11
56,40
62.20
53.43
62.96
SPP
C4
28.75
63.76
63.15
52.71
74.66
SPP
Wikitext2
28.67
63.80
63.18
52.71
75.34
ATTN Parameter
MLP Gamrma Paramettrr
10
Maciso
Macks
Mocta2
bocks
Moota mb
Moot:4 mb
Mosts
Mocte7
Moots mg
2-
Nootu mb
Mocha.11
Mosta m0
Moote cimb
200
400
666
832
3200
220
400
600
660
1000
Figure 2: Visualization of solution path of DeiT-small. We show the changes of the Ll-norm of projected weight value I during the search stage. The x-axis is the iteration number during training, the y-axis is the Ll- Ll-norm of the I' parameters per layer.
remain consistent. Such consistency among weights allow us effectively analyze the performance of sparse models.
Extentions to LLMs. extended our method to large language models (LLMs) with post-training. We applied our solution path method during the LLM pruning search stage, combining it with the RIA (Zhang et al. 2024) pruming metric. The detailed algorithm is shown in Algorithm 2
We applied our method on Llama2-7B and OPT-6.7b 7b. The calibration datasets C4 and Wikitext2 were used to generate activations during the forward pass, which, along with weight magnitude, served as the pruning metric. The results were reported on 5 datasets. As shown in Table the accuracy after pruning with 50% sparsity is comparable to the advanced method RIA and Wanda (Sun et al. 2023 indicating the potential our method in pruning LLMs.
CONCLUSION
We proposed a dynamic approach based on differential inclusion, which can adaptively prune any pre- trained transformers with various compression ratios. Along this path, series of models, named the Transformer Weight Family, was derived from the masks the solution path. With just a single run of iteration,
achieve all sparsity levels of the original pre-trained model. We have demonstrated the stability and consistency of the Transformer Weight Family, showing that the solution path method is robust. We also demonstrated the potential of method for pruning large language models (LLMs). future work, we will explore this direction with a more tailored design suited to the architecture and post-training processes of L.MMs.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the Science and Technology Commission of Shanghai MunicipalityyNN.. 2451 24511103100.
REFERENCES
Hedy Attouch, Jérôme Bolte, and Benar Fux Svaiter. Convergence of descent methods for semi- algebraic and tame problems: proximal algorithms, forward-backward splitting, and regularized gauss-scidel methods. Mathematical Programming, 137:91-129, 2013.
Navid Azizan, Sahin Lale, and Babak Hassibi. Stochastic mirror descent on overparameterizee nonlinear models: Convergence, implicit regularization, and generalization. arXiv preprint MrXiv::906.038300 2019.
Amir Beck and Mare Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Reseurch Letters, 31():1677-75,, 2003.
Martin Benning, Marta M. Betcke, Matthias J. Ehrhardt, and Carola-Bibiane SchönlieB. Choose your path wisely: gradient descent in bregman distance framework. arXiv preprint arXiv:1712.04045, 2017.
Jacek Bochnak, Michel Coste, and Maric-Francoise Roy. Real algebraic geometry, volume 3. Ergeb. Math. Grenzgeb. Springer-Verlage Berlin, 1998.
Jérôme Bolte, Aris Daniilidis, and Adrian Lewis. The Lojasiewicz inequality for nonsmooth subana- lytic functions with applications to subgradient dynamical systems. SIAM Journal on Optimization, 17:1205-1223, 2007a.
Jérôme Bolte, Aris Danilidis, Adrian Lewis, and Masahiro Shiota. Clark subgradients of stratifiable functions. SIAM Journal on Optimization, 18:556-555 572, 2007b.
Stephen Boyd, Neal Parikh, Eric Chu, Borja Pelcato, Jonathan Eckstein, et al. Distributed optimizztion and statistical learning via the alternating direction method of multipliers. In Foundations and Trends@ in Machine Learning, volume 3, pp. 1-122. Now Publishers, Inc., 2011.
Leon Bungert, Tim Roith, Daniel Tenbrinck, and Martin Burger. A bregman learning framework for sparse neural networks. Journal of Machine Learning Research, 23(192): 2022.
Martin Burger, Guy Gilboa, Stanley Osher, and Jinjun Xu. Nonlinear inverse scale space methods. 2006.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize for efficient deployment. arXiv preprint arXiv: /908v: 0979/, 2019.
Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi Multi-dimension searching in continuous optimization space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4931 19911,94,,11 2022.
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. Advances Neural Information Processing Systems, 34:19974-19988, 2021.
M. Coste. An introduction to o-minimal geometry. RAAG Notes, 81 pages, Institut de Recherche Mathematiques de Rennes, 1999.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 2009. doi: 10. 009/CCPP.2000955266600
Yanwei Fu, Chen Liu, Donghao Li, Xinwei Sun. Jinshan Zeng, and Yuan Yao. Dessilbi: Exploring structural sparsity of deep networks via differential inclusion paths. In International Conference Machine Learning, pp. 3315-3326. PMLR, 2020.

---

Published as a conference paper at ICLR 2025
Yanwei Fu, Chen Liu, Donghao Li, Zuyuan Zhong, Xinwei Sun, Jinshan Zeng, and Yuan Yao. Exploring structural sparsity of deep networks via inverse scale spaces. IEEE Transactions on Patter Analysis and Machine Intelligence, 45(2):1749-1765, 2022.
Saeed Ghadimi and Guanghui Lan. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization, 23:2341-2366 2013.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115, 2014.
Saurabh Goyal, Anamitra Roy Choudhury, Saurabh Raje, Venkatesan Chakaravarthy, Yogish Sab- harwal, and Ashish Verma. Power-bert: Accelerating bert inference via progressive word-vector elimination. In International Conference on Machine Learning, pp. 3690-3699. PMLR, 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503. 0..531,,,, 2015.
Chendi Huang and Yuan Yao. unified dynamic approach to sparse model selection. In The 2 International Conference Artificial Intelligence and Statistics (AISTATS), Lanzarote, Spain, 2018.
Steven Krantz and Harold R. Parks. A primer of real 2002.
Birkhäuser, second edition,
Walid Krichene, Alexandre Bayen, and Peter L Bartlett. Accelerated mirror descent in continuous and discrete time. In Advances in neural information processing systems, pp.
2015.
Krzysztof Kurdyka. On gradients of functions definable in o-minimal structures. Annales de I l'institut Fourier, 48: 7897639783, 1998.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: /909. / /942, 2019.
Benlin Liu, Yongming Rao, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Metadistiller: Network self- boosting via meta-learned top-down distillation. In Computer Esion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 694-709. Springer, 2020.
Stanistaw Lojasiewicz. Une propriété topologique des sous-ensembles analytiques réels. In: Les Equations aux dérivées partielles. Éditions du centre National de la Recherche Scientifique, Paris, pp. 87-89, 1963.
Stanistaw Eojasiewiez. Ensembles semi-analytiques. Institut des Hautes Etudes Scientifiques, 1965.
Stanislaw tojasiewicz. Sur la geometrie semi-et sous-analytique. Annales de l'instital Fourier, 43: 1595-1595, 1993.
Boris S. Mordukhovich. Variational analysis and generalized differentiation I: Basic Theory. Springer, 2006.
Arkadij Semenovič Nemirovskij and David Borisovich Yudin. Problem complexity and method efficiency in optimization. 1983.
Stanley Osher, Feng Ruan, Jiechao Xiong, Yuan Yao, and Wotao Yin. Sparse recovery via differential inclusions. Applied and Computational Harmonic Analysis, 41(2):436-469, 2016a. arxiv
1406. 7728
Stanley Osher, Feng Ruan, Jiechao Xiong, Yuan Yao, Wotao Yin. Sparse recovery via differential inclusions. Applied and Computational Harmonic Analysis, 41(2):336-49 2016b.
Bowen Pan, Rameswar Panda, Yifan Jiang, Zhangyang Wang, Rogerio Feris, and Aude Oliva. ia - red?: Interpretability-awee aware redundancy reduction for vision transformers. Advances in Neural Information Processing Systems, 14:24488-2491 2021.
12

---

Published as a conference paper at ICLR 2025
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint 1002:v88022.0666 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
R. Tyrrell Rockafellar and Roger J-B Wets. Variational analysis. Grundlehren Math. Wiss. 317, Springer- Verlag, New York, 1998.
Dachuan Shi, Chaofan Tao, Ying Jin, Zhendong Yang, Chun Yuan, and Jiaqi Wang. Upop: Uni- fied and progressive pruning for compressing vision-language transformers. arXiv preprint arXiv:23311.3341, 2023.
Masahiro Shiota. Geometry of subanalytic and semialgebraic sets, volume 150 of Progress in Mathematics. Birkhäuser, Boston, 1997.
Anthony Man-Cho So, Yinyu Ye, and Jiawei Zhang. A unified theorem on sdp rank reduction. Mathematics of Operations Research, 33(4):110-92)) 2008.
Weijie Su, Stephen Boyd, and Emmanuel J Candes. A differential equation for modeling nesterov's 's accelerated gradient method: theory and insights. The Journal of Machine Learning Research, 17 (1):5312-5354, 2016.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. arXiv preprint 7XXi:2306.116955 2023.
Ychui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 101000966699999 2020.
Chaofan Tao, Lu Hou, Wei Zhang, Lifeng Shang, Xin Jiang, Qun Liu, Ping Luo, and Ngai Wong. Compression of generative pre-trained language models via quantization. arXiv preprint arXiv:2203.1077 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, and Hervé Jégou. Training data-efficient image transformers & distillation through attention, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers &amp; distillation through attention. In International Conference on Machine Learning, volume 139, pp. 10347-10357. July 2021a.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning, pp. 10347-10357. PMLR, 2021b.
L. van den Dries. A generalization of the tarski-seidenberg theorem and some nondefinability results. Bull. Amer. Math. Soc. (N.S.), 15:189-193, 1986.
L. van den Dries and C. Miller. Geometric categories and o-minimal structures. Duke Matheratiial Journal, 84:497-540, 1996.
Bo Wahlberg, Stephen Boyd, Mariette Annergren, and Yang Wang. An admm algorithm for class of total variation regularized estimation problems. IFAC Proceedings Volumes, 45(16):83 38,8 2012.
Yu Wang, Wotao Yin, and Jinshan Zeng. Global convergence of admm in nonconvex nonsmooth optimization. Journal of Scientific Computing, 78(1):29-6) 63, 2019.
Huanrui Yang. Hongxu Yin, Maying Shen, Pavlo Molchanov, Hai Li, and Jan Kautz. Global vision transformer pruning with hessian- hessian-aware aware saliency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18547-18557, 2023.
13

---

Published as a conference paper at ICLR 2025
Miao Yin, Burak Uzkent, Yilin Shen, Hongxia Jin, and Bo Yuan. Gohsp: A unified framework of graph and optimization based heterogeneous structured pruning for vision transformer. arXiv preprint arXiv:2301.05345. 2023.
Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width depth pruning for vision transformers. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 3143-3151, 2022.
Hao Yu and Jianxin Wu. unified pruning framework for vision transformers. Science China Information Sciences, 66(7):1-2, 2023.
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 24355-24533 24363,, 2023.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse dccomposition. In Proceedings of the IEEE conference on computer vision and patter recognition, pp. 7370-7379, 2017.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisit knowledge distillation: teacher-free framework. 2019.
Jinshan Zeng, Tim Tsz- Kit Lau, Shao-Bo Bo Lin, and Yuan Yao. Global convergence of block coordinate descent in deep learning. In Proceedings of the 36th International Conference Machine Learning. Long Beach, California, 2019.
Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug- and and-play: An efficient post-training pruning method for large language models, In The Twelfth International Conference on Learning Representations, 2024.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPERIMENTS DETAILS AND VISUALIZATION
Experiments details
As shown in Algorithm 2 instead of directly using our algorithm, we combined it with the RIA Zhang ct al. (2024) pruning method. This because post-training tasks are quite different from fine-tuning tasks. To maintain the generalization performance of LLMs, need to take the size of the weight parameters and activations into account.
We used 4 A100 GPU with memory size of 80GB for those experiments. The search stage contain an update stage and prune stage, both just need to run once for one model. All the finetune stage used AdamW the optimizer and consine scheduler the learning rate scheduler. The hyperparameters are listed in Table. and Table. g
Model
Datasets
Updating Epochs
Pruming Epochs
Finetuning Epochs Batch Size
LR
DeiT-Base
Imagenet-Il
20
300
1024
8c-4
DeiT-Small
Imagenet-I lk
300
512
8e-4
DeiT-Tiny
Imagenet-Ik Ik
10
30
300
256
8e-4
Swin-Tiny Imagenet-Ik Ik
20
300
256
8e-4
CLip-Larre COCO
5
32
1e-5
CLip-Base
COCO
32
1e-5
Table 6: The hyperparameters of experiments mentioned above.
Algorithm 2 Extention to LLMs
Perform searching
Input: Pretrained weight Wo and step size a, iteration steps in the update stage T, and prune stage T
Initialize sub-gradient Vo 0. mask Mo = 1, sparse mask To = 0. w.,
Set A A Wl. WT..T
[
(I)X, which is the pruning metric of RIA Zhang al.
(2024)
for
0 to T, do
# Calculate the loss
I. = L(W, @ M&) + 2v M - Txll
# update V/ and mask Ms according to sub-gradient
M&+1 = M - KKap V
V4+1 V/8 au V
# update L'x as the proximal operator
M(T) A ||[|| (T)
14+1 Prox (V4+1), where Prox(V) argminr {IIV -11--112 /212/2200 m(T)}
end for
Perform pruning
for k = to Tp do
# Update masks with reverse turn of T being non-zen zero in
searching
M = i h in, - (k: + +))
# Update and save the sparse model weights
W = Wx+1 @ Ma+1
#Save the checkpoint of We+1 as the pruned model
end for
Return Weight Family for the model
Output: LLM Weight Family:(Wgje E [0,Tp]}
15

---

Published as a conference paper at ICLR 2025
Model
Method
Latency Time
Params
FLOPS(B)
Uncompressed
9,273s
299 100%
11.2100%
8.675s
278g3.09
38.593.4%
CLip-Base
7.859s
257 86.%%
35.5
SPP
7.341s
23478.3%
31.977.40
6.490s
18160.5%
22.955.6%
5,461s
13043.5%
13.6
Table 7: The latency results of pruned model.
Visualization of compressed model
Figure 3: Visualization of the proportion of parameters on DeiT-Small. The three kind of color indicate three pairs of weight.
In Fig. 3 and Fig we visualized the proportion of parameters retained each layer of the DeiT- Small model at a set compression ratio of 0.3. Notably, since the query, key, and value, as well as output projection, form two pairs with equivalent parameter quantities, we treated them as unit. It's observed that in the shallower layer and deeper layer 10, the proportion of saved parameters in the QK pair is significantly higher than that the V and Project pair. This indicates that our low-rank pruning method is effective. It skillfully segregates parameters, allowing those with closer relationships to be pruned together.
A.1 GROUP LASSO
Our algorithm D enhances structural sparsity within transformer layers. aligning under a group lasso penalty framework, (T) E, [|2,, where
T
[ 12
(17)
Model
À
Epochs
LR
DeiT-Base
15
8e-4
DeiT-Small
15
10
8e-4
DeiT-Tiny
15
1,6e-3
Swin-Tiny Tiny
15
8e-4
100
3
le-5
CLip-Base
100
3
le-5
Table The hyperparameters expriments mentioned above updating stage.
16

---

Published as a conference paper at ICLR 2025
Model
Method
Latency Time
Params
FLOPS(B)
Uncompressed
9.273s
299100%
41 41.2 100%
8.675s
27893.0%
38.5g3.4 44%
CLIP-base
7.859s
25786.0%
35.5%6.2%
SPP
7.
23478.3%
31.977.4
6,490s
18160.59
22.965.05.
5.461s
13043.5%
13.633.0%
Table 9: The latency time results of compressed CLIP-basc Touvron et al. (2021b) models.
MHA-Q
100
100
100
100
100
100
1.00
100
100
100
100
100
MHA-Q
100
71
100
98
98
95
100
93
89
89
14
MHA-K
100
100
100
100
MHA-K K
71
9K
100
95
MHA-V
100
1O0
100
100
100
100
MHA-V
96
V8
10U
H5
MLP
100
100
100
100
1000
100
100
100
100
100
100
100
MLP
49
60
57
63
60
61
58
77
93
700
62
71
2
4
5
7
8
10
2
4
6
9
10
11
Compressed Ratio
0%
Compressed Ratio 30%
MHA-Q
100
98
81
84
82
MHA-Q
12
MILA-K
300
48
98
92
79
81
81
84
79
82
MHA-K
96
12
59
355
1
10
4
9
21
MHA-V
98
78
MHA-V
MLP
41
42
47
47
68
48
MLP
29
25
27
27
48
44
27
2
4
5
7
8
9
10
1
2
4
6
7
9
10
11
Compressed Ratio 50%
Compressed Ratio 70%
Figure 4: Visualization of components. The depth color shows the sparsity of the corresponding layer. The number shows the dim of the linear matrix.
and [o| represents the number of weights in each group 1 Therefore, we have clear formula for solving this equation:
1
r max(0,
v
(18)
1
A.2 MORE RELATED WORKS
Mirror Descent Algorithm (MDA) was first proposed by Nemirovskij & Yudin (1983) to solve
constrained convex optimization, and can be seen as a generalized projected gradient descent Beck Teboulle (2003) using Bregman distance (n(u, v) == Q(w) M(r) - (Vl(v), - v) induced by convex and differentiable function S2(.).
Zx+1 Zμ. aVW(
(19a)
Wx+1 (Z
(19b)
where the conjugate function of (2.) ) is (Z) == supw: 2 (W, 7) - Equation (1) optimizes
W in two steps: implemennt the gradient descent on in the dual space Z A2(W;:;
and Eq proooeesssss it back to the primal space. As step size a + 0, MDA converges to the following ordinary differential equation (ODE) dynamie<(Su et al. (2016):
21 aVC(W) W Vn
(2a)
(2b)
Compared with DessiLBI Both our method and DessiLBI use the mirror descent algorithm to get the solution path, with the sparse model obtained through early stopping. However, our method is specifically designed for pre-trained models, while DessiLBI, which trains from scratch, doesn't 't perform well on them. During the search stage, we keep the pre-trained model's 's weights fixed and only update the mask parameters, making approach suitable for tasks like pruning LLMss.
17

---

Published as a conference paper at ICLR 2025
Additionally, our method uses pair-wise wise mask architecture, which works for fully connected layers in transformers, but not applicable to CNN architectures.
B PROOF OF THEOREM
First of all, we reformulate Eq. into an equivalent form. Without loss of generality, consider ? S2 in the sequel. Denote R == (T(T), then Eq (3) can be rewritten as:
Px+1 Prox, RT (Px + x(ps - vV(())))
(20a)
Pk+1 = Ps x (Px+1 - Ps. + kP)))
(20b)
where Pk = [0, [0 E (R(Ps) and g E @ Thus Algorithm is equivalent to the following iterations.
MR+1 Ma ka WW
(2la)
T8+1 Prox, + K(t - -VrC(M,, [w)),,
(21b)
9k+1 ( k+1 F + Ka Vrā(M,,
(21c)
Exploiting the equívalent reformulation 21c), one can establish the global convergence of (M&, Tk 9&) based the Kurdyka-F.ojasiewins framework. In this section, the following extended version of Theorem is actually proved.
Theorem 2. (Global Convergence of Our method] Suppose that Assumption 7 holds. Let (Ms, Is, gh) be the sequence generated by Our method (Eq. 21a 21c with finite initializa- tion.
2 0 ag = <[Lip C + =-1)
then (M,, 9k) converges critical point of F. Moreover, {MM T&] converges stationary point of L defined in Eq. 9 and [Mx] converges to a stationary point of C(M).
Remark 3. Assumption (a)-(c) are regular the analysis of nonconvex algorithm (see, AAtoouch et al. 2013) for instance), while Assumption (d) is also mild including all Lipschitz continuous convex function over compact set. Some typical examples satisfying Assumption [Yd) are the &
norm, group & norm, and every continuously differentiable penalties. By Eq. (15) and the definition of conjugate, the Lyapunov function F can be rewritten as follows.
F(W,T,g) aC(Wr Q(T) (1) (T,
(22)
Applying the neural networks. typical examples are summarized the following corollary.
Corollary 1. Let {Ma, I', } be sequence generated by Our method (21a 21c for neural network training where (a) is any smooth definable loss function, such as the square loss (t²), exponential loss (e"), logistic loss log(1+e '), and cross cross-entropy loss; (b) a is any smooth definable activation, such as linear activation (t), sigmoid (T++)), hyperbolic tangent (5 and softplus ( log(1 + e for some > 0) as smooth approximation of ReLU; (c) Ω is the group Lasso. Then the sequence {M&} converges a stationary point of L(M) under the conditions of Theorem 7
B. SUFFICIENT DESCENT PROPERTY ALONG LYAPUNOV FUNCTION
Let P& == (Me, Tk). and Qk := (Ps. 1), & E N. In the following, we present the sufficient descent property of Q₁ along the Lyapunov function F.
Lemma. Suppose that is continuously differentiable and VL is Lipschitz continuous with a constant Lip > 0,C max Wolis the value of the pretrained model parameters Wo. Let {Qx) be a sequence generated by SLBI with finite initialization. < a n(Lip=C+u-1y then
F(QR+1) S F(Qx) - Pl Qu
where p:= a(LipxOtu-1)
18

---

Published as a conference paper at ICLR 2025
Proof. By the optimality condition of (20a) and also the inclusion pk 90, l @RR(P)), there holds
<(aVE((P)) pk+1 pk)) Pk+1 Ps =0
which implies
VVCK)) Pk) = Pall D(T
(23)
where
D(T == (9k+1 - 9k. - - Tx).
Let Wo @ M = with L(M) L(Wo G.,,
VÅ(M) = Σ VC(W) Wo
(24)
Noting that L(P) L(M)+ M - r11 L(Wo 9 M) + 22
M -11 and by the Lipschitz
continuity VL(W) with constants Lip 0,C max Wol
implies VL is Lipschitz
continuous with constant Lip * C + -- This implies
Tip C + -
((Pk+1) S ((P)) (VL(P)). P&+1 P&) +
|| Pall2-
2
Substituting the above inequality into 23 yields
aC(Px+1) D(Tx+1,T')) + p||
Px < aC(P)).
(25)
Adding some terms in both sides of the above inequality and after some reformulations implies
af((P++) + B (T*+1,Tal $ aa((P) + B
(26)
- - Pxl1 (D(T'+1,,
B 1[ DC
B (T&+1,T)))
=aC(P%) + - (T',,, (T',, p|| - Pxl1 B8++1 (T,, I'x1) Ba-11 (T=,
where the final equality holds for D(T 3+1,Tx) - By 1+ [x] = B 11 (T k: T k-1)). That
F(Qu+1) & F(2x) P|| - -P 2 Ba g (Tr, B -6- (T lb-1)
(27)
< F(Q)) P|| 1x
(28)
where the final inequality holds for B (T Ix-1) 0 and B (Tx, I 2 Thus, finish the proof of this lemma.
Based on Lemma B.I we directly obtain the following lemma.
Lemma 1. Suppose that assumptions of Lemma B. hold. Suppose further that Assumption (b)-(d)
hold. Then
(i) both a{C(Px)) and [F(Qz)) converge to the same finite value, and limg B (T (K44, O.
(ii) the sequence {MM, Tk. g is bounded,
(iii) limg- |P441
P 1 = 0 and limp-+oo D(T 0,
(iv) K Ek PA - P& 3 + at a rate of (1/KK.
Proof. By 25 [(Pt) is monotonically decreasing due D ((T 2 0. Similarly, by (28).
F(Q*) is also monotonically decreasing. By the lower boundedness assumption of L(W), both
C(P) and F(Q) lower bounded by their definitions, i.e., (9) and (15). respectively. Therefore, both (f(P)) and (F().)) converge, and it obvious that ling- F(2)) 2 limg- (L(..)
By 27
B < F(Qx) F(QR+1), 1 1=
19

---

Published as a conference paper at ICLR 2025
By the convergence of F(Qx) and the nonegativeness of Ba "I'x, [x-),, there holds
lim B '(Tky k- = 0. By the definition of F(Q2) aC((p) + -
(T ks T R- and the above equality, it yields
lim F(Qx) lim (P.).
Since ((M) has bounded level sets, then M2 is bounded. By the definition of L(M, r) and the finiteness [(M,, [x), T3 is also bounded due to Mg is bounded. The boundedness gs. is due to gi 8Q(TT)) condition (d), and the boundedness of Tλ.
By 28
summing up 28 over h = 1, K yields
(plIPttI PkI12
< aL(PP) < oc.
(29)
k=
Letting K + 3o and noting that both || - Pxl and D(T k+1, 1441,Dp P&) are nonnegative, thus lim |Pk41 Pall = 0, lim D(T k41 [x) 0.
80000
k
Again by 29
K
(pll PAll D(T++1, < <k-oo)()),
K
k=0
which implies R Σk
|| - rate of (1/K)).
B.2 RELATIVE ERROR PROPERTY
In this subsection, we provide the bound of subgradient by the discrepancy of two successive iterates. By the definition of (15).
a aVM M ((M++1, Ik+1)
Ha+1
WWLCMM+H +1, I2+1) + gk+1 - 9k
EOF(QU+),
(30)
T Tx+1
Lemma.
Under assumptions of Lemma then
Qull- for H8L1 E @FF(Qx+1), k E,N, where P1 = 2n +1 a(Lip C 1)- Moreover, K EE He& - 0 at rate of O(1KK)
|| H || S p: p1
Proof. Note that
MMLMM (PM VM
(31)
+ +VVM Ta) VMR((M, T&)) + (MM(M,
By the definition of (see 9
| L(M MM Pa)l --o1==-1[[[[ - - Txtill:
VM.(M. Px)l = [[(VC(M)+1) vL(M&)) + - - M&)ll
$ (Lip = C - -1 - Mell,
where the last inequality holds for the Lipschitz continuity of VL with constant Lip 0.and max Wol By 21a
I M C(M. [x)]l (sa) ( 1|| M&ll-
Substituting the above (in)equalities into 31 yields
MM < [(Ra]- + Lip C' C'+ -1 Mall + -ー [ 1
Txll
20

---

Published as a conference paper at ICLR 2025
Thus,
[ M.VMAMMMH Fx+1)ll & [x= + a(Tip ++ 1| Mell + tav "
-xll-
(32)
By (1c). yields
gh+1 9b x- -T's l1k+1) - aV
Noting that VrL(M& [R) ( M&), and after some simplifications yields
gk+11 gs |1 -) (Tx [K+1) av 1 Mk-1)1l [ TR+1)
S av av 1| MM+1ll + (x av av= -r+tlll
(33)
where the last inequality holds for the triangle inequality and x > av by the assumption.
By
and definition of Ha+1 (30). there holds
|
[ + a(LTp C + 2v 2=- | M&ll (N + 1)][T k+1 Txll
2n 2 <(Lip C' 2n 2- })] |Pk+1 PAll
(34)
M 12k + 1 a (Lip * C' + 2v 2v- |Qk+1 - Qxll.
By 344 and Lemmal
1 liv), Σズ || 0 at of O(1/K).
This finishes the proof of this lemma.
B.3 KURDYKA-LOUASIEVVE PROPERTY
To introduce the definition the Kurdyka-Lojase Lojasiewizz (KL) property, we need some notions and notations from variational analysis, which can be found in Rockafellar & Wets (1998).
The notion of subdifferential plays a central role in the following definitions. For each x E dom(h)= =
(x E Rº l(() < +oo). the Fréchel subdifferential of at x, written (x), is the set of vectors RP which satisfy
h(y)
((x)
(v,
lim
inf y
2 0.
x
When x & dom(h), we set ōh(x) = Ø. The introduced in Mordukhovich (006)), written ah(x) dom(h), defined by
(or simply subdifferential)
@h(x) = (v RD Ex* x. hxx*) h(x), v @h(x*) vv.
(35)
necessary (but not sufficient) condition for R2 be a minimizer of h is ōh(x). point that satisfies this inclusion is called limiting-criticnn or simply critical. The distance between point x subset S of R., written dist(x, S), is defined by dist(xx,) S) inf || represents the Euclidean norm.
s|| s E 5), where
Let R" R {+oo} be an extended-reat seaededd-rel-valueee function (respectively, R R be point-to-set mapping), its graph is defined by
Graph(h) = {(x, y) E RD x R = h(x)),
(resp. Graph(h) ((x,y) R R y e h(x))))
and its domain by dom(h) := {x E RP h(x) < } (resp. dom(h) = {x E Re h((x) + Ø}).
When h is a proper function, i.e., when dom(h) # Ø, the set of its global minimizers (possibly empty) is denoted by
argmin = {x RP h(x)
inf h}.
The KL property (Lojasiewicz 1963 1993 Kurdyka 1998 Bolte et al. 2007a b) plays central role
the convergence analysis of nonconvex algorithms (Attouch et al. 2013 Wang et al. 2019). The following definition is adopted from Bolte et al. 2007b).
21

---

Published as a conference paper at ICLR 2025
Definition 1. (Kurehyaa-ojjssiewiie.e
A function h is said to have the Kurdyka-Lojasiewts (KL) property at ū E dom( := {u E | * 0}. if there exists a constant 1) E 00), neighborhood N of u and a function o (). - R,, which is a concave function that is continuous at O and satisfies x(0) = 0, & E C* ((0, n)), i.e., & is continuous differentiable on (0, n),
and (s) > 0 for all & E (0, n). such that for all E n {u R. [h(ū)= < h((u) h((u) n). the following inequality holds
o' ('h(u) - (((())(((a)) dist(O, ah(u)) 2 1.
(36)
If h satisfies the KI. property at each point of dom((h)) h is called a Klnfunction.
KL functions include semialgebraic functions, real analytic functions, continuous subanalytic func- tions (Bolte et al.] (0077a and locally strongly convex functions, tame functions defined in some o-minimal structures (Kurdykal 1998, Bolte et al., 2007b). In the following, we provide some important examples that satisfy the Kurdyka-Lojasiewiic property.
Definition 2. /Semialgebraic)
(a) A function h R. + R LRU(H [+ (resp. point-to-set mapping h IR = RT) is called semialgebraic if its graph Graph(h) is a semialgebraic set.
(b) set D c. RP called semialgebraic (Bochnak et al. 1998) if it can be represented as
D= UN (XER: i=1j=1
D
: 0, Qx)(() 0],
where Piy. Qi are real polynomial
fum functions for sis 8, 1, si
1998) and (Shiota 1997 1.2.9, page 52), the class of semialgebraic sets are stable under the operation of finite union, finite intersection, Cartesian product complementation. Some typical examples include polynomial functions, the indicator function of a semialgebraic set, and the Fuclidean norm (Bochnak et al., 1998 page 26).
According to (Lojasiewicz) 1965 Bochnak et al.
Definition 3. [Real analytic] A function h with domain an open set U E. and range the set of either all real or complex numbers, is said to be real analytic at u if the functiin h may be represented by convergent power series on some interval of positive radius centered at u: h(xx) Σi= a - u)', for some {a C RR. The function is said to be real analytic on V C U if it is real analytic at each V (Krantz & Parks 2002 Definition 1.1.5). The real analytic function f over I. for some posilive integer p >> can be defined similarly.
According to Krantz & Parks 2002)), typical real analytic functions include polynomials, exponential functions, and the logarithm, trigonometric and power functions on any open set of their domains. One can verify whether a multivariable real function h((x) on IRP is analytic by checking the analyticity of g(t) = la(x + ty) for any x, E IRP.
B.4 KL PROPERTY IN DEEP LEARNING AND PROOF OF COROLLARY II
In the following, we consider the deep neural network training problem. Consider a l-layer feedfor- ward neural network including / - 1 hidden layers of the neural network. Particularly, let d, be the number hidden units the i-th hidden layer for == I,...,
Let do and dj be the number of units of input and output layers, respectively. Let W E Rd.xd be the weight matrix between the (i (i- 1) layer and the i-th layer for any = 1....
According to Theorem 2 one major condition is to verify the introduced Lyapunov function F property. For this purpose, we need an extension of semialgebraic set, called the o-minimal structure (see, for instance Coste (199)) van den Dries (1986). Kurdyka (1998), Bolte et al. (2007))) The following definition is from Bolte et al. (007b))
defined in (15) satisfies the
Definition& 4. [o-minimal structure] An o-minimal structure on (R, +. a sequence of boolean algebras O, of "definable" subsets of RR, such that for each n E N
'To simplify notations, we regard the input and output layers as the 0-th and the I-th layers, respectively, and absorb the bias of each layer into W
22

---

Published as a conference paper at ICLR 2025
(i) the elements of O1 are exactly finite unions of intervals and points.
(ii) On contains the family of algebraic subsets of RR" that is, every set of the form
(ii)) if A belongs to On then x R and x A belong to Oa+1i
(iv) iT [R=+1 _ I is the canonical projection onto R", then for any A in Ont1. the set [1(A) belongs to On
E :ERR:p(a; p(x) 0),
where p. P: + R is a polynomial function.
Based on the definition o-minimal structure, we can show the definition of the definable function.
Definition (Definable function] Given an o-minimal structure (over (R,+,)) :),, a function f: R + R is said be definable in O if its graph belongs to On+1-
According to van den Dries & Miller 1996); Bolte et al. (2007b), there are some important facts of the o-minimal structure, shown as follows.
(i) The o-minimal structure stable under the sum, composition, the inf-convolution and several other classical operations of analysis.
(il) The collection of semialgebraic sets an o-minimal structure. Recall the semialgebraic sets are Bollean combinations of sets of the form
{o R" p(z) 0, (x) < O,..., ........m... (2) < 0},
where p and q: 's are polynomial functions in R"
(iil) There exists an o-minimal structure that contains the sets form
{ t) E [-1, 1]" x R f(x) = t}
where is real-analytic around [|1,
(iV) There exists o-minimal structure that contains simultaneously the graph of the exponential function è x + cxp(x: ( and all semialgebraic sets.
The Kurdyka-Looasiewiicz property for the smooth definable function and non-smooth definable function were established in (Kurdyka 1998 Theorem 1) and (Bolte et al., 20076 Theorem 14), respectively. Now we are ready to present the proof of Corollary
Proof. [Proof of Corollary 1 function F satisfies Kurdyka-Lojasiewiezz inequality, this case and by (22) F be rewritten as follows
To justify this corollary, we only need to verify the associated Lyapunov
F(M,T.9) = (((M,T)
1 M - rTp) Q(T) + (g) (T, 2v
Because L. and a,'s 's are definable by assumptions, then L(M,T) are definable as compositions of definable functions.
Moreover, according to Krantz& & Parks 2002), |M - r and (T. are semi-algebraie and thus definable. Since the group Lasso S2(T') Σg [ the composition of l2 and I, norms, and the conjugate group Lasso penalty is the maximum of group no-mom, i.e. (*()') = max, 11' where the 12. l1, and 7o norms definable, hence the group Lasso and its conjugate definable as compositions of definable functions. Therefore, is definable and hence satisfies Kurdyka- Eojasiewiez inequality by (Kurdyka 1998 Theorem 1).
The verifications of other cases listed in assumptions can found in the proof of (Zeng et al. 2019 Proposition 1). This finishes the proof of this corollary.
23

---

Published as a conference paper at ICLR 2025
B.5
PROOF OF THEOREM! 2
Our analysis is mainly motivated by paper (Benning et al. 2017). as well as the influential work (Attouch al. 2013). According to Lemma 2.6 in Attouch al. (213)), there are mainly four ingredients in the analysis, that is, the sufficient descent property, relative error property, continuity property of the function. More
property of the generated sequence and the specifically, we first establish the sufficient descent property of the generated sequence via exploiting the Lyapunov function F (see, (15) in Lemma B. 1 in Section B.1 and then show the relative enror property of the sequence in Lemma B.2 in Section B.2 The confinuity property is guaranteed by the continuity of [(M,1) I") and the relation lit Bn (T 1: (x) = established in Lemma I in Section B.1 Thus, together with the Kurdyka-L.ojaseewicc assumption of F, we establish the global convergence of SLBI following by (Attouch et al. 2013 Lemma 2.6).
Let (W, g) be critical point of F, then the following holds
FM P, gg at (VL(()))))). + + [)) 0, 0r 1, g) ou av M) + (2 0 9 0,
(37)
0,F(M,I. (., - (g) 9.
By the final inclusion and the convexity of St it implies g E @N(T). Plugging this inclusion into the second inclusion yields ov (T M) 0. Together with the first equality imples
v VCM,TT r) O, VC(M) = 0.
This finishes the proof of this theorem.
24


Reviews:
[{"review_id": "UqhvyKP0LN", "reviewer": "Reviewer_MvCV", "summary": "This paper proposes a method to prune pretrained transformers at any desired ratio within a single pruning stage. The proposed method enjoys a theoretical analysis to guarantee the global convergence. Extensive experiments validate the effectiveness of the proposed method.", "questions": "1. For pruning large language models, why to combine the propsed SPP with the RIA pruning metric? However, pruning CV models (ViTs and CLIP) does not use pruning metric. What is the difference between pruning CV models and LLMs?\n2. In line 207, it seems that the dimension is d_1 rather than d.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is motivated well. The method for any desired pruning ratio is needed in many real-world applications. And the proposed method is able to achieve this.\n2. This paper identifies the limitation of Lasso and develops a differential inclusion-based method to achieve various compression ratio pruning. \n3. There is a sound theoretical analysis to guarantee the global convergence of the method. A detailed proof is included in the appendix.\n4. Experimental results are strong. Many experiments are conducted, including ViTs for image classification, CLIP, and even large language models.", "weaknesses": "1. Although the authors claim the proposed method significantly reducing the cost of model pruning. The training cost is not reported in this paper. It is better to introduce how long the search stage is. And make a comparison for training cost between different methods.\n2. The ablation studies are weak. Experiments demonstrate the strong performance of the proposed SPP. But it is hard for the reader to figure out why the proposed method is effective. More ablation studies are needed. For example, make comparison for weight-based pruning and mask-based pruning. Why does the weight-based pruning not be applicable to the Transformer? What if using Lasso for the searching stage. By optimizating Eq. 7, Lasso is able to achieve different level of sparsity during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "suiRxVSdev", "reviewer": "Reviewer_1jDf", "summary": "The paper proposes SPP (Solution Path Pruning), a novel approach to mask-based pruning for pretrained transformers. SPP leverages differential inclusions to create a dynamic solution path that produces a range of sparsity levels in a single pruning stage, eliminating the need for multiple retraining steps for different compression ratios. The method applies fine-grained, pair-wise shared masks across transformer layers, including attention heads and MLP layers, achieving high flexibility and model performance retention. SPP is tested on various datasets and transformer backbones, showing efficiency improvements with minimal accuracy loss.", "questions": "- How does the method scale to very large transformers (e.g., GPT-3 scale)?\n- What is the stability of the solution path across different random seeds?\n- How sensitive is the method to the choice of hyperparameters κ and λ?\n- Can this method be extended to dynamic/runtime pruning scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**Adaptive Solution Path for Pruning**  \n   - Unlike traditional mask-based pruning, SPP generates models with different sparsity levels in a single pruning run, allowing for a Transformer Weight Family adaptable to varying hardware or performance needs without retraining.\n\n**Flexible, Fine-Grained Pruning Strategy**  \n   - SPP’s pair-wise shared mask strategy applies pruning at the smallest functional units within transformers (e.g., query-key and value-output pairs), allowing for greater flexibility and effectiveness in compression compared to conventional methods.\n\n**Reduced Computational Cost**  \n   - The SPP method reduces the need for repeated pruning stages, resulting in significant cost savings, especially for large-scale transformer models.", "weaknesses": "**Marginal Improvement Over Existing Methods**  \n   - Although the method introduces adaptive pruning, it does not fundamentally change the mask-based pruning paradigm. The improvements, while novel in terms of execution, may appear incremental compared to existing mask-based and structural pruning strategies.\n\n**Lack of Broad Comparison with Other Mask-Based Methods**  \n   - The paper does not provide an in-depth comparison with other advanced mask-based pruning techniques, making it difficult to fully assess SPP's advantages in performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "97aYojtcVT", "reviewer": "Reviewer_P1Rj", "summary": "This paper presents a pruning method of pretrained transformers at any desired ratio within a single pruning stage, based on a differential inclusion for a mask parameter.", "questions": "1.\tThe manuscript does not describe how low-rank compression can be applied to paired modules of transformers.\n2.\tThe reviewer is concerned about the actual running time of the proposed method and how much its efficiency improves compared to methods with a single pruning rate.\n3.\tThere seem to be some errors in the details of the manuscript, for example, \"Specifically, note that for each element i, V (i) ≤ 1 if V (i) = 0, and is equal to 1 when it becomes non-zeros\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tClear motivation.\n2.\tThe theoretical proof and experiment are quite sufficient.", "weaknesses": "1.\tThe results of ablation studies are insufficient to demonstrate the effectiveness of the proposed method for the following reasons: 1) For DeiT-Small and Swin-Tiny models, the proposed SPP achieves higher accuracy with more parameters and higher or equal FLOPS, which does not indicate that SPP is superior. 2) Conducting experiments solely with the DessiLBI method lacks generalizability. \n2.\tFigure 1 and tables lack legends and comments.\n3.\t\"FLOPs\" is generally not written as \"Flops\" when used as an abbreviation for \"floating point operations\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)