Pruning Pre-trained Language Models with Principled Importance and Self-regularization
Year: 2023

Abstract:
Iterative pruning is one of the most effective compression methods for pre-trained language models. We discovered that finding the optimal pruning decision is an equalityconstrained 0-1 Integer Linear Programming problem. The solution to this optimization problem leads to a principled importance criterion which we use to rank parameters during iterative model pruning. To mitigate the poor generalization at high sparsity levels, we propose a self-regularization scheme where model prediction is regularized by the latest checkpoint with increasing sparsity throughout pruning. Our experiments on natural language understanding, question answering, named entity recognition, and datato-text generation with various Transformerbased PLMs show the effectiveness of the approach at various sparsity levels.

heading: 1 Introduction
text: Pre-trained language models (PLMs) (Devlin et al., 2019; Radford et al., 2018) have significantly advanced the state-of-the-art in various natural language processing tasks (Wang et al., 2018; Zhou and Lampouras, 2020; Dušek et al., 2020; Radev et al., 2020). However, these models often contain a vast amount of parameters, posing nontrivial requirements for storage and computation. Due to this inefficiency, the applications of PLMs in resource-constrained scenarios are still limited.
To resolve the above challenge, model compression (Sun et al., 2019; Ben Noach and Goldberg, 2020; Lan et al., 2020) has been actively studied to make PLMs meet the practical requirement. Among them, iterative pruning methods are widely adopted at only a tiny expense of model performance when adapting PLMs to downstream tasks. During the course of iterative pruning, model parameters can not only be updated but also
∗ The corresponding author.
be pruned based on the rank of their importance scores in order to satisfy the cardinality constraint. Prevalent importance criteria are based on the parameter’s magnitude (Zhu and Gupta, 2017; Renda et al., 2020) or sensitivity (Louizos et al., 2018; Sanh et al., 2020; Liang et al., 2021; Zhang et al., 2022). Parameters with low importance scores are pruned and are expected to have little impact on model performance.
Despite the empirical success, existing importance criteria for model pruning still face two major limitations: (1) they are heuristically defined and may not accurately quantify a parameter’s contribution to the learning process, e.g., absolute weight value in magnitude-based pruning and gradient-weight product in sensitivity-based pruning; (2) they determine the importance of each parameter individually without considering the effect of coinstantaneous parameter updates on model performance, e.g., sensitivity is estimated by the absolute change in training error if only a single parameter is pruned and others remain unchanged.
In this paper, we rethink the design of the importance criterion for model pruning from an optimization perspective. We begin by analyzing the temporal variation of any given learning objective based on a single-step gradient descent update under the iterative pruning setting. We show that finding the optimal pruning decision can be framed as solving an equality-constrained 0-1 Integer Linear Programming (ILP) problem, where the constraint is defined by the specified sparsity. The resulting problem is a particular case of a general 0-1 Knapsack problem in which the weight for each item is the same. The solution to this problem naturally leads to a principled importance criterion which we use to rank all model parameters and derive the optimal stepwise pruning decision.
When a high sparsity (e.g., 80%∼90%) is pursued, the limited capacity often renders the pruned model fails to retain satisfactory performance with
8995
conventional fine-tuning. To further improve the model’s generalization ability, we propose a selfregularization scheme, where the model prediction is regularized by the latest best-performing model checkpoint during pruning. We show that such a scheme eases model learning with decreasing capacity and effectively yields a tighter upper bound of expected generalization error than learning from training data alone.
To validate the effectiveness of our approach, dubbed PINS (Pruning with principled Importance aNd Self-regularization), we conducted extensive experiments with various pre-trained language models on a wide variety of tasks, including natural language understanding on GLUE (Wang et al., 2018)), question answering on SQuAD (Rajpurkar et al., 2016), named entity recognition on CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003), and data-to-text generation on WebNLG (Zhou and Lampouras, 2020), DART (Radev et al., 2020), and E2E (Dušek et al., 2020). Experimental results show that PINS provides more accurate models at different sparsity levels. Detailed analysis shed further light on some intriguing properties of models pruned by PINS. By exploiting the resulting high sparsity, we show that the storage/inference can be reduced/accelerated by 8.9x and 2.7x using CSR format and a sparsityaware inference runtime (Kurtz et al., 2020) on consumer-level CPUs 1.
In summary, our contributions are:
• We establish the equivalence between the optimal pruning decision and the solution to an equality-constrained 0-1 Integer Linear Programming problem. The solution to this problem leads to a principled importance criterion that can be used to rank parameters during iterative pruning.
• We propose a simple yet effective selfregularization scheme to enhance the model’s generalization capability, especially under a high-sparsity regime.
• Comprehensive experiments and analyses confirm the effectiveness of our approach at various sparsity levels.
1Code available at https://github.com/DRSY/ PINS

heading: 2 Background and Related Work
text: In this section, we review the necessary background on Transformer-based pre-trained language models and popular importance criteria for iterative pruning.

heading: 2.1 Transformer-based Pre-trained Language Models
text: Most existing pre-trained neural language models (Radford et al., 2018; Devlin et al., 2019; Wang et al., 2020; Clark et al., 2020) are based on the Transformer (Vaswani et al., 2017) architecture, which consists of several identical blocks of self-attention and feedforward network. After pre-training on a massive amount of unlabeled general-domain corpus in a self-supervised learning manner, these models exhibit superior performance on various downstream tasks via finetuning. However, good generalization performance comes at the cost of a vast amount of parameters. For example, the base version of BERT has 110M parameters and leads to more than 400MB of disk storage. Therefore, how to effectively reduce model size while preserving as much task accuracy as possible remains a challenging research problem.

heading: 2.2 Iterative Pruning
text: Pruning methods can be divided into two categories: one-shot pruning (Lee et al., 2018; Frankle and Carbin, 2018) and iterative pruning (Louizos et al., 2018; Sanh et al., 2020; Zhang et al., 2022). One-shot pruning removes parameters of low importance after training. It is efficient but ignores the complicated training dynamics when applied to modern large neural language models. On the contrary, iterative pruning performs training and pruning simultaneously. Therefore, the resulting sparsity pattern is aware of the complex dynamics of parameters through the course of training and delivers considerable improvement compared to one-shot pruning.
Let θ(t) = {θ(t)1 θ (t) 2 , ..., θ (t) d } denote the ddimensional model parameters at t-th training iteration, the typical updating rule of iterative pruning can be formulated as:
θ̂(t+1) = θ(t) − η(t)∇θL(θ(t)) (1) θ(t+1) = θ̂(t+1) ⊙M (t) (2)
where η(t) is the learning rate at time step t and L is the learning objective. The temporarily updated
θ̂(t+1) is further pruned by the binary mask M (t)∈ {0, 1}d, which is computed based on a given importance criterion S(t):
M (t) i =
{ 1, if S(t)i is in the top-r (t)of S(t)
0, otherwise (3)
where r(t)≤ d indicates the number of remaining parameters at time step t according to a given sparsity scheduler.

heading: 2.3 Importance Criteria for Model Pruning
text: Popular importance criteria for model pruning include parameters’ magnitude and sensitivity.
Magnitude is a simple yet effective importance criterion that is widely used for model pruning. It estimates the importance of each parameter as its absolute value, i.e., S(t)i = |θ (t) i |. Despite its simplicity, the magnitude cannot accurately gauge the importance of parameters because even parameters with small magnitude can have a large impact on the model prediction due to the complex compositional structure of PLMs.
Sensitivity is another useful importance criterion. It estimates the importance of each parameter as the absolute change of the learning objective if the parameter is pruned, i.e., set to zero. The mathematical formulation of the sensitivity of i-th parameter is given by:
S (t) i = |L(θ (t) −i)− L(θ(t))| (4)
≈ |g(t)i θ (t) i | (5)
where θ(t)−i is identical to θ (t) except that the i-th entry is set to zero and g(t)i is the gradient of i-th entry. Though taking the training dynamics into account, sensitivity still estimates the importance of each parameter individually without considering the effect of holistic parameter update.

heading: 3 Methodology
text: Instead of heuristically defining the importance criterion as in prior pruning methods, we take a step back and rethink the design of the importance criterion for model pruning from an optimization perspective. From our analysis, we draw an equivalence between finding the optimal stepwise pruning decision and solving an equality-constrained
0-1 Integer Linear Programming problem. We further show that the optimal solution to this problem leads to a new importance criterion for model pruning. Moreover, we propose a simple yet effective self-regularization scheme to facilitate the generalization ability of the sparse model. We elucidate our analysis in Section 3.1 and describe our self-regularization scheme in Section 3.2.

heading: 3.1 Rethinking Importance Criterion from the Optimization Perspective
text: Without loss of generality, we denote L as the learning objective when adapting a pre-trained language model f with parameter θ to a downstream task. At t-th training iteration, we denote the current model parameters as θ(t) and the evaluated learning objective as L(θ(t)).
The temporal variation of the learning objective L(θ(t)) at time step t is given by the second-order Taylor series expansion:
∆L(t) = L(θ(t) +∆θ(t))− L(θ(t)) (6) = ∇θL(θ(t))⊤∆θ(t)+ 1
2 ∆θ(t)
⊤ H(t)∆θ(t) + o(|∆θ(t)|2) (7)
where H(t) is the Hessian matrix at step t. It is known that the largest eigenvalue λmax of Hessian matrices in a PLM is typically small (Shen et al., 2019), i.e., ∆θ(t)
⊤ H(t)∆θ(t) ≤ λmax|∆θ(t)|22 ≈
0. Thus, we ignore the second-order term as well as the infinitesimal of higher order in Eq. (7):
∆L(t) = ∇θL(θ(t))⊤∆θ(t)
=
d∑
i=1
g (t) i ·∆θ (t) i (8)
Under the iterative pruning setting, the actual temporal variation ∆θ(t)i of i-th parameter depends on whether it is allowed to be updated or forced to zeroed out. Formally, we use a binary variable x(t)i to indicate the pruning decision of i-th parameter at time step t, i.e., x(t)i = 1 means θ (t) i is updated and x(t)i = 0 means θ (t) i is pruned. The temporal variation in Eq. (8) can now be rewritten as:
∆L(t) = d∑
i=1
g (t) i (x (t) i ∆θ̂ (t) i + (1− x (t) i )(−θ (t) i ))
(9)
where ∆θ̂(t)i = −η(t)g (t) i is the gradient descent update. Finding the optimal pruning decision that
leads to the smallest ∆L(t) is now converted to an equality-constrained 0-1 integer linear programming (ILP) problem of variables x(t):
x̃(t) = argmin x(t) ∆L(t)
s.t. d∑
i=1
x (t) i = r (t),x (t) i ∈ {0, 1} (10)
where r(t) is the number of remaining parameters at step t according to the pre-defined sparsity scheduler. If we consider each parameter θ(t)i as an item and r(t) as the total capacity, the problem that Eq. (10) defines can be treated as a special case of 0-1 Knapsack problem where the weight for each item is one and the value for each item is given by:
S (t) i = −g (t) i ∆θ̂ (t) i − g (t) i θ (t) i (11)
Contrary to the general 0-1 Knapsack problem which is known to be NP-complete, fortunately, the equal-weight 0-1 Knapsack is a P problem. Its optimal solution can be obtained by sorting items in descending order according to their values and selecting the top-r(t) ones:
x̃ (t) i =
{ 1, if S(t)i is in the top-r (t)of S(t)
0, otherwise (12)
Putting it in the context of iterative pruning, our analysis theoretically reveals the validity of: (1) selecting parameters based on the ranking of certain importance criterion; (2) using Eq. (11) as a principled new importance criterion.

heading: 3.2 Self-regularization
text: In vanilla fine-tuning, the learning objective L is defined as the training error Ler (a.k.a empirical risk in statistical learning) over the empirical data distribution. However, minimizing such training error does not translate to good generalization. Moreover, as iterative pruning proceeds, the number of non-zero parameters in the model monotonically decreases. The reduced model capacity increases the learning difficulty (Lopez-Paz et al., 2015; Mirzadeh et al., 2019) and usually leads to degenerated generalization performance of the sparsified model (Sanh et al., 2020).
Confronting the above challenges, we propose an effective self-regularization scheme tailored to
improving the model’s generalization ability during iterative pruning. Concretely, besides learning from the hard label of training data, the output of the current model with parameter θ(t) is also regularized by the output of the latest best-performing model checkpoint with parameter θ(tl), where tl ≤ t denotes the time step at which the latest checkpoint was saved. The learning objective of selfregularization is defined as:
Lsr = D(yθ(t) , yθ(tl)) (13)
where D can be any divergence metric, e.g., KLdivergence for classification tasks. Lsr is then integrated with the original learning objective, i.e., L = Ler + Lsr.
Why does self-regularization work? Our selfregularization is similar to teacher-student knowledge distillation in the sense that the model output is regularized by the output of another model. However, the most critical difference is that the “teacher” in self-regularization is instantiated by checkpoint with increasing sparsity, such that the capacity gap between “teacher” and “student” is dynamically adjusted. We theoretically justify the effectiveness of self-regularization as follows:
Theorem 1. Let ti and tj where ti ≥ tj denote the time steps at which two different checkpoints are saved; Let R(fθ(t←ti)) and R(fθ(t←tj)) denote the expected generalization error of models learned from fθ(ti) and fθ(tj); Let n denotes the size of training data; | · |C denotes a capacity measure of function class Fθ. Based on previous expositions on VC theory (Vapnik, 1998), we have the following asymptotic generalization bounds hold:
R(fθ(t←ti)) ≤ O( |Fθ(t) |C nαi ) + inf F
θ(t←ti) R(fθ(t))
︸ ︷︷ ︸ bound(f
θ(t←ti) )
R(f θ(t←tj)) ≤ O( |Fθ(t) |C nαj ) + inf F
θ (t←tj)
R(fθ(t))
︸ ︷︷ ︸ bound(f
θ (t←tj) )
Because θ(ti) is a later checkpoint with higher sparsity than θ(tj), we have the learning speed

heading: 1 ≥ αi ≥ αj ≥ 12 , then the following inequality holds with high probability:
text: bound(fθ(t←ti)) ≤ bound(fθ(t←tj))
In summary, self-regularization works by enabling a tighter generalization bound compared to learning from training data alone or a static dense teacher as in knowledge distillation. Please refer to Appendix B for detailed derivation.

heading: 3.3 The Algorithm
text: Here we formally summarize our algorithm PINS (Pruning with principled Importance aNd Self-regularization) in Algorithm 1:
Algorithm 1 PINS Input: Training set Dtr = {(xi, yi)}Ni=1; Validation set Dval; pre-trained parameters θ; maximum training steps T ; evaluation interval teval. Initialize: θ(0) ← θ, tl ← 0, best validation accuracy acctl ← −INF.
1: for t = 0 to T − 1 do 2: Sample a mini-batch (x,y) from Dtr 3: Compute current model’s output yθ(t) 4: Compute latest best-performing check-
point’s output yθ(tl) 5: Compute L based on yθ(t) , yθ(tl) and y 6: Compute S(t) via Eq. (11) 7: Compute θ(t+1) via Eq. (2) and Eq. (3) 8: if t%teval = 0 and acct>acctl then 9: acctl ← acct, θ(tl) ← θ(t)
Output: the pruned parameters θ(T ).

heading: 4 Experiments
text: In this section, We compare PINS with state-ofthe-art pruning algorithms and perform detailed analysis to understand the effectiveness of PINS.

heading: 4.1 Setup

heading: 4.1.1 Tasks
text: We conduct experiments on a comprehensive spectrum of tasks following standard data splits. Natural Language Understanding. We opt for tasks from the GLUE (Wang et al., 2018) benchmark, including linguistic acceptability (CoLA), natural language inference (RTE, QNLI, MNLI), paraphrase (MRPC, QQP), sentiment analysis (SST-2) and textual similarity (STS-B). Because the official test set of GLUE is hidden, we randomly split a small portion of training set as validation set and treat the original validation set as test set. Question Answering. We use SQuAD v1.1 (Rajpurkar et al., 2016) as a representative dataset for
extractive question answering following previous work (Zhang et al., 2022). Named Entity Recognition. We also examine our approach on CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003) for token-level named entity recognition task. Data-to-Text Generation. Besides language understanding tasks, we also extend our evaluation to data-to-text generation on three datasets: E2E (Dušek et al., 2020), DART (Radev et al., 2020), and WebNLG (Zhou and Lampouras, 2020), which involves generating a piece of fluent text from a set of structured relational triples.

heading: 4.1.2 Baselines
text: Magnitude-based. Iterative magnitude pruning (IMP) (Zhu and Gupta, 2017) is the state-ofthe-art magnitude-based approach. Sensitivity-based. l0-regularization (Louizos et al., 2018) trains masking variables via reparametrization trick with l0 penalty; SMvP (Sanh et al., 2020) uses accumulated sensitivity as importance metric; PST (Li et al., 2022) proposed a hybrid importance criterion combining both magnitude and sensitivity; PLATON (Zhang et al., 2022) uses a modified variant of sensitivity by exponential moving average and uncertainty re-weighting.

heading: 4.1.3 Implementation Details
text: We mainly conduct experiments on the pre-trained BERTbase (Devlin et al., 2019) as a pruning target for all tasks except data-to-text generation. We defer the pruning results of MiniLM12L-384H (Wang et al., 2020) and Electrabase (Clark et al., 2020) to Appendix A. For data-to-text generation, we adopt the pre-trained GPT-2 (Radford et al., 2018) following a prior study (Li et al., 2022).
During pruning, we employ the cubic sparsity scheduler (Sanh et al., 2020; Zhang et al., 2022) to gradually increase the sparsity level from 0 to the specified target sparsity. To avoid tremendous computation cost brought by hyper-parameter tuning, we only search the batch size from {16, 32} and fix the learning rate as 3e-5 for all experiments on GLUE and CoNLL. For SQuAD v1.1, we fix the batch size as 16 and the learning rate as 3e-5 following Zhang et al. (2022). We adopt AdamW (Loshchilov and Hutter, 2017) as the default optimizer. To reduce the variance induced by mini-batch sampling, we adopt a smoothing technique similar to PLATON. We run each experi-
ment five times with different random seeds and report the average results (significance tests with p-value < 0.05 are conducted for all performance gains).

heading: 4.2 Main Results

heading: 4.2.1 Comparison with Baselines
text: Natural language understanding We present the experimental results on GLUE at high sparsity, i.e., 80% and 90% in Table 1. Among all baselines, sensitivity-based methods generally achieve better results than magnitude-based IMP, which implies the importance of training dynamics when designing pruning criteria. We can see that PINS delivers more accurate sparsified models on all datasets at both sparsity levels. The advantage of PINS is more evident on small datasets. For example, PINS outperforms the previous bestperforming baseline (PLATON) by 4.1 and 2.6 points on RTE and CoLA at 80% sparsity, where there are only a few thousand training data. Under extremely high sparsity, i.e., 90%, PINS is still
able to retain 97.5% overall performance of finetuning, outperforming 95.4% of the previous best method PLATON. Notably, PINS even surpasses fine-tuning on RTE and MRPC at 80% sparsity. This can be attributed to the fact that PLMs are heavily over-parameterized and PINS can effectively identify parameters crucial to the task to realize low bias and low variance simultaneously.
Question answering Table 2 summarizes the pruning results on SQuAD v1.1. Interestingly, IMP outperforms all sensitivity-based methods except for PLATON at all considered sparsity levels, in contrast to the observations on GLUE. Our method, however, consistently yields the best performance at all sparsity settings.
Named entity recognition Table 3 demonstrates the pruning results on CoNLL 2003 dataset for named entity recognition. At 70% sparsity, our method almost matches the performance of fine-tuning, outperforming baselines on all evaluation metrics. The gain of PINS is more prominent
when further increasing sparsity.
Data-to-text generation Table 4 shows the pruning results on E2E, DART and WebNLG at 80% sparsity. PINS achieves the best performance on all three datasets in all evaluation metrics. In particular, PINS delivers performance even better than fine-tuning on the E2E dataset by 0.7 ROUGE-L and 0.4 METEOR scores, respectively. We posit that this is due to the relative easiness of E2E compared to the other two datasets.

heading: 4.2.2 Results at Medium-to-Low Sparsity
text: The typical utility of pruning is to produce a sparse yet competitive model that can benefit downstream applications in terms of efficiency without sacrificing much task accuracy. We hypothesize that PINS might also bring a regularization effect compared to vanilla fine-tuning under the medium-to-low sparsity regime.
As shown in Table 5, when specifying a medium-to-low sparsity, e.g., 50%∼30%, our method can effectively play a role of regularization and improve model performance compared to vanilla fine-tuning. With half of the parameters being pruned, the sparse model produced by PINS outperforms fine-tuning by 1 percentage point on the GLUE score. This observation suggests that appropriate pruning can effectively reduce variance without hurting model expressiveness.

heading: 4.3 Ablation Study
text: The self-regularization scheme is proposed and integrated into PINS to improve model generaliza-
tion. Here we investigate the effectiveness of selfregularization by comparing it to the conventional knowledge distillation scheme and the classical empirical risk minimization scheme.
The pruning results of using the three different learning objectives on RTE, CoLA, and MRPC are listed in Table 6. Pruning with PINS using classical empirical risk minimization still achieves performance better than existing baselines (Table 1). Learning from a densely fine-tuned BERTbase as the teacher does not always improve and sometime may even hurt performance. In contrast, our proposed self-regularization consistently boosts model performance, which echoes our theoretical justification in Section 3.2.

heading: 4.4 Analysis
text: We provide an in-depth analysis of various importance criteria to uncover more valuable insights.
Sparsity pattern of weight matrices We are interested in the sparsity pattern produced by different pruning criteria. To this end, we plot the remaining parameters’ distribution of the same weight matrix in BERTbase pruned via magnitude, sensitivity, and PINS in Figure 1. We observe
that magnitude-based pruning generates a sparsity pattern close to randomness. Sensitivity-based pruning produces a more structured pattern where the remaining parameters tend to occupy complete rows. Interestingly, the sparsity pattern produced by PINS exhibits the highest concentration on specific rows. This implies that the parameters contributing most to the end-task are preferably distributed in a structured way and PINS is more effective at extracting such patterns.
Layerwise rank distribution The highly structured sparsity pattern generated by PINS intrigues our interest to further analyze the intrinsic property of parameter matrices after pruning. Specifically, we inspect the matrix rank as it is usually associated with the complexity of matrix. To this end, we visualize the layerwise rank distribution of BERTbase pruned using different importance
criteria on SST-2 dataset. As shown in Figure 4, magnitude pruning produces sparse matrices that are still near full-rank despite containing 80% zeros. Sensitivity pruning tends to generate sparsity pattern with lower rank compared to magnitude pruning. Notably, model pruned by PINS shows consistently lower matrix rank than the other two criteria. This implies that PINS is more effective at identifying the low-dimensional task representation during adaptation, which is usually correlated with tighter generalization bounds (Arora et al., 2018; Aghajanyan et al., 2021).
Empirical validation of importance criterion In Section 3.1 we prove that the pruning decision derived by our importance criterion is theoretically optimal. Here we empirically validate this point by visualizing the change of learning objective as pruning proceeds. Figure 3 illustrates that our importance criterion indeed leads to the most significant decrease in the learning objective compared to heuristical ones like magnitude and sensitivity.

heading: 4.5 Efficiency Gain
text: We can exploit the resulting high sparsity to attain practical efficiency gain on storage and inference speed. We first apply quantization upon the pruned model and transform it into INT8 data type before saving it using Compressed Sparse Row (CSR) format. We then leverage a sparsity-aware runtime (Kurtz et al., 2020) for accelerating inference. As shown in Table 7, on the RTE dataset, the disk
space and inference time of BERTbase pruned at 80% sparsity can be reduced by roughly 8.9x and 2.7x respectively with negligible accuracy loss.

heading: 5 Conclusion
text: We present PINS, a new iterative pruning method that hinges on a principled weight importance criterion to deliver the optimal stepwise pruning decision. Integrated with a self-regularization scheme tailored to pruning-during-adaptation, PINS allows for provably better generalization ability. Empirical experiments and analyses confirm the effectiveness of our method and shed further light on the different sparsity patterns produced by PINS and other existing methods.
Limitations
Compared to the empirical risk minimization scheme, the introduced self-regularization scheme incurs certain overhead because each mini-batch of data will go through two models. For BERTbase scale pre-trained language models, the additional memory overhead is about 27% and the additional training time overhead is about 30%. Nevertheless, once pruned, the sparsified model can enjoy considerable efficiency gains in terms of storage and inference time. Therefore, this is a trade-off that future practitioners might need to consider.

heading: Acknowledgments
text: This work was generously supported by the CMB Credit Card Center & SJTU joint research grant, and Meituan-SJTU joint research grant.

heading: A Results with More PLMs on subset of GLUE
text: In addition the widely used BERT and GPT-2 models, we also perform pruning experiments upon other two pre-trained language models: Electrabase and MiniLM12L-384H to further verify the effectiveness of our method.
Due to computing resource constraint, we restrict our experiments on a subset of GLUE task, including RTE, CoLA and QNLI at 80% and 90% sparsity. We compare PINS against IMP and PLATON as two representative baselines for magnitude-based and sensitivity-based pruning methods. We fix the batch size as 32 and learning rate as 3e-5 similar to the BERT experiments. We illustrate the pruning results on Table 8 and Table 9. At both sparsity levels, PINS consistently outperforms IMP and PLATON on all three datasets, verifying the general effectiveness of PINS for language model pruning.

heading: B Proof of Theorem 1
text: Proof. Let ti and tj where ti ≥ tj denote the time steps at which two different checkpoints are saved; Let R(fθ(t←ti)) and R(fθ(t←tj)) denote the expected generalization error of models learned
from fθ(ti) and fθ(tj) ; Let n denotes the size of training data; | · |C denotes a capacity measure like VC-dimension for function class Fθ. Based on previous expositions on VC theory, the following asymptotic generalization bound holds:
R(fθ(t←ti)) = R(fθ(t←ti))−R(fθ(ti)) +R(fθ(ti))
≤ O( |Fθ(t) |C nαi ) + ϵt,ti +R(fθ(ti)) = O( |Fθ(t) |C nαi
) + inf f θ(t) ∈F θ(t←ti) R(fθ(t))
︸ ︷︷ ︸ bound(f
θ(t←ti) )
R(f θ(t←tj)) = R(fθ(t←tj))−R(fθ(tj))
+R(f θ(tj) ) ≤ O( |Fθ(t) |C nαj ) + ϵt,tj +R(fθ(tj)) = O( |Fθ(t) |C nαj
) + inf f θ(t) ∈F θ (t←tj) R(fθ(t))
︸ ︷︷ ︸ bound(f
θ (t←tj) )
where ϵt,ti is the approximation error of function class Fθ(t←ti) with respect to fθ(ti) . ϵt,tj is defined in analogy. Because: (1) θ(ti) is a later checkpoint with higher sparsity than θ(tj), we have the learning speed 1 ≥ αi ≥ αj ≥ 12 ; (2) fθ(ti) has lower generalization error than f
θ(tj) , we have the fol-
lowing inequality holds with high probability:
bound(fθ(t←ti)) ≤ bound(fθ(t←tj))

heading: C More Post-pruning Analyses
text: This section presents more visualized analyses of models sparsified by different pruning methods.
Figure 5 shows the layerwise rank distribution of BERTbase pruned using different importance criteria on the RTE dataset. The observation here is similar to what is discussed in the main body of the paper: PINS exhibits the lowest average matrix rank in the sparsified model compared to the other two criteria.
Figure 4 illustrates the weight distribution of BERTbase pruning using different importance criteria. From the left figure we can see that magnitude-based pruning tends to keep parameters with high absolute values, which is expected
based on its definition. Sensitivity and PINS produce similar weight value distribution mainly because the two methods both contain the gθ term in their importance calculation. Despite the similarity, we can still observe that PINS produces smoother distribution than sensitivity and covers more weights with larger absolute values.
The right figure shows the layerwise distribution of remaining parameters after pruning. A clear trend is that PINS tends to retain more parameters in the middle layers (4-7), which also coincided with the inter-model sparsity pattern analysis in the main body of our paper. Both sensitivity and PINS remove a large proportion of parameters in the top layers (10-12) while magnitude-based pruning has no preference for model layers.

heading: D Sparsity Scheduler
text: The proportion of remaining weights is controlled by the sparsity scheduler, here we adopt the commonly used cubic sparsity schedule to progressively reach target sparsity, i.e., r(t) at time step t within the maximum time steps T is given by:    ri t ∈ [0, ti) rf + (ri − rf )( T−tf−tT−tf−ti )
3 t ∈ [ti, T − tf ) rf otherwise
(14)
where ri = 1.0, rf is the final percent of remained parameters, ti and tf are the warmup and cooldown steps.

heading: E Accelerating Inference and Reducing Storage
text: We attain practical efficiency gain in terms of inference time and disk storage space using different sets of off-the-shelf techniques. Specifically, we use DeepSparse2, a sparsity-aware inference runtime to accelerate inference of sparse model on CPUs. We also utilize the Pytorch builtin quantization function3 and Compressed Sparse Row (CSR) format4 to achieve a much smaller disk space requirement.
2https://github.com/neuralmagic/ deepsparse
3https://pytorch.org/docs/stable/ quantization.html
4https://github.com/huggingface/block_ movement_pruning/blob/master/Saving_ PruneBERT.ipynb
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 4.1.1
3 B1. Did you cite the creators of artifacts you used? 4.1.1
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. 4.1.1;4.1.3
C 3 Did you run computational experiments? 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 4.1.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 4.1.3
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 4.2
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 4.2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Therefore, this is a trade-off that future practitioners might need to consider