Instruction-Following Pruning for Large Language Models
Authors: Bairu Hou
Qibin Chen
Jianyu Wang
Guoli Yin
Chong Wang
Nan Du
Ruoming Pang
Shiyu Chang
Tao Lei
Venue: ICML 2025 poster (2025)
Keywords: Large language model
model pruning
contextual sparsity
pre-training
fine-tuning
Primary area: deep_learning->large_language_models

Abstract: With the rapid scaling of large language models (LLMs), structured pruning has become a widely used technique to learn efficient, smaller models from larger ones, delivering superior performance compared to training similarly sized models from scratch. In this paper, we move beyond the traditional static pruning approach of determining a fixed pruning mask for a model, and propose a dynamic approach to structured pruning. In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction-following pruning'', introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both instruction-following data and the pre-training corpus. Experimental results demonstrate the effectiveness of our approach on a wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.

Instruction-Folloving Pruning for Large Language Models
Bairu Hou 12* Qibin Chen'
Jianyu Wang'
Guoli Yin Chong Wang' Nan Du'
Ruoming Pang'
Shiyu Chang? Tao Lei"
Abstract
Once the mask is optimized, the resulting mask is fixed, allowing deployment of a smaller, pruned model. However, the fixed nature of the pruned model poses challenges in real- world inference scenarios, where tasks can vary significantly, for instance, coding, mathematics, or domain-spectte re- quirements, each demanding distinct skills and knowledge from the original language model. static pruned model may struggle to balance inference efficiency with high per- formance across diverse tasks.
With the rapid scaling of large language models (LLMs), structured pruning has become widely used technique to learn efficient, smaller mod- els from larger ones, delivering superior perfor- mance compared to training similarly sized mod- els from scratch. In this paper, we move beyond the traditional static pruning approach of deter- mining a fixed pruning mask for model, and propose a dynamic approach to structured prun- ing. In our method, the pruning mask is input- dependent and adapts dynamically based on the information described in a user instruction. Our approach, termed "instruction- foollootionn-olooiingg pruming", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task. To identify and activate effective parameters, we jointly optimize the sparse mask predictor and the LLM, leveraging both cnstruction-folloviin data and the pre-training corpus. Experimental results demonstrate the effectiveness of our ap- proach wide range of evaluation benchmarks. For example, our 3B activated model improves over the 3B dense model by 5-8 points of absolute margin on domains such as math and coding, and rivals the performance of a 9B model.
Given this, we explore a paradigm shift from static pruning masks to dynamic ones, addressing the central question:
Can LLMs learn to select the most suited parameters based on the task description?
We aim automatically generate input-specific pruning masks tailored to the tasks described in user prompts. This dynamic, context-aware pruning mechanism enables the language model to perform inference using only the param- eters necessary for the task, offering a compelling balance between efficiency and expressivity compared to using a static dense model. Moreover, because the parameters are selected and fixed, our method avoids reloading new pa- rameters during the decoding process. This design choice contrasts with other dynamic methods such as contextual sparsity (Liu et al., 2023; Zhou et al., 2024) and mixture-of- experts (Lepikhin et al., 2020; Fedus et al., 2022; Dai et al., 2024), which load different parameters at each decoding step. leading to significant weight loading costs. Our design is particularly suited for on-device models (e.g. smartphones and laptops), where the inference typically samples few re- sponses given the same user query (or the same task). In this case, the same activated parameters are selected and cached as a dense model, therefore achieving the same speedup as the static pruning.
1. Introduction
Structured pruning techniques have become widely adopted method for reducing the inference cost of large lan- guage models (Wang et al., 2020; Sreenivas et al., 2024; Mu- ralidharan et al., 2024; Meta AI, 2024). These methods typ- ically optimize a binary mask over language model parame- ters minimize either language modeling task-specific loss (Xia al., 2024; Sreenivas et al., 2024; Meta AI, 2024).
To address the central question, we present Instruction- Following Pruning (IFPRUNING), method that integrates sparsity predictor with the language model to dynamically generate input-dependent pruning masks, illustrated in Figure 1. Specifically, we focus on structured pruning of the feed-forward neural network layers, where entire rows or columns of the weight matrices are pruned (Xia et al., 2024; Gunter et al., 2024; Sreenivas et al., 2024). The user
"Wor done while interning Apple. Applee AI/ML PUC Santa Barbara. Correspondence to: Bairu Hou <bbairu @ edu.. Tao Lei <ttaolei2ee <tao_lei2@apple.com @apple.com..
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Instructtin-Folllwiigg Pruning for Large Language Models
Param selection
Pruning
Evaluation
Per-Input FFN Mask
Original LLM
MATH
Sparsity Predictor
Deresz 9I
Input-specific pruning
Dense 3D
Coding
User request
Pasegl
35
Translate the English text into
Write a Python function from the given definition.
TIFT
Desse
FFN
Desse
FFN 2
Secondis)
0.3
0.6
Figure 1. Overview of IFPRUNING. (Left) For each given prompt, the sparsity predictor (much smaller than the LLM) determines which rows and columns of the FFN matrices should be activated. (Middle) The LLM then pruned accordingly and uses the selected paramcters to perform inference for that specific prompt. (Right) By pruning a 9B LLM to 3B for cach input, IFPRUNING significantly outperforms the dense 3B model and achieves performance levels close to the dense 9B model. It also achieves nearly the same inference latency as the dense 3B model, as measured by time-to no-e-to-iiee fo-fisstttoken (TTFT).
prompt is first passed into the sparsity predictor, which as- signs importance scores to the rows and columns of each feed-forward network layer. These scores are then trans- formed into differentiable masks using the SOFTTOPK op- erator (Ainslic et al., 2023), to achieve a predefined number of sparsity (e.g.. reducing a 9B language model to 3B active parameters). The resulting masks are applied to the lan- guage model, in which the feed-forward layers are pruned using the masks.
We conduct further analysis to better understand the prun- ing decisions. Specifically, we observe that instructions requiring similar skills or domain knowledge yield highly homogeneous pruning patterns. Inspired by this analysis, we explore per-task pruning in Section 4.3, where a single task prompt generates shared masks for all test instances within the same task. Results show that per-task pruning maintains robust performance while further reducing data loading overhead.
During training, the differentiable mask generation mecha- nism allows us to jointly optimize both the sparsity predictor and the language model by minimizing the next-token pre- diction loss. We employ effective training strategies that leverage both pre-training and supervised fine-tuning data. At test time, only the selected parameters are activated for inference. Parameter selection can be performed either per- imput or per-task: the input prompt can directly be used for parameter selection (Section 4.2), or predefined task prompt can be used to select parameters shared across mul- tiple inputs within the same task (Section 4.3).
We also show that IFPRUNING can significantly improve the LLM inference efficiency in Section 4.4. Compared to the full model, PFRRNINNG reduces the time-to- firststt token by up 57% and the generate time by up to 41%. In addition, the overhead introduced by dynamic pruning and parameter caching is negligible, adding less than 0.1 seconds per example and accounting for only 1-2% of the total generation time.
2. Related Work
In this section, we provide an overview of prior research that closely relates and partially motivates our work, including model pruning, contextual sparsity, and mixture-of-experts.
We validate IFPRUNING through comprehensive experi- ments across diverse tasks in Section 4. Specifically, we fine-tune pre-trained language models of varying sizes (6B, 9B, and 12B parameters) using IFPRNNING and prune them to activate only 3B parameters. In particular, IFPRUNING consistently outperforms 3B dense models across tasks such as math, coding, tool use, MMLU (Hendryeks et al., 2021a) and AlpacaEval (Dubois al., 2024). For example, when dynamically pruning the 9B model to 3B, method im- proves over the 3B dense model by 8% coding tasks and by 5% on math benchmarks, incurring only marginal perfor- mance degradation compared to the unpruned 9B model.
Model pruning Pruning has been extensively studied to compress neural networks and improve their efficiency (Han et al., 2015; Zhu & Gupta, 2017). Previous work has plored different pruning techniques for both unstructured pruning (Narang et al., 2017; Frankle & Carbin, 2018; 8; Li et al., 2020; Chen et al., 2020) and structured pruning (Wen et al., 2016; Voita et al., 2019; Louizos et al., 2018; Wang et al., 2020). As structured pruning removes entire compo-
2

---

Instruction-Followii Pruning for Large Language Models
nents in the model such as channels, attention heads, and feed-forward neural network intermediate dimensions, it is more hardware-frrindlly than unstructured pruning to com- press the large models.
Mixture-of-experts Mfxture-of-Experss (MoB) have emerged as a popular architecture for scaling LLMs while managing inference costs (Lepikhin et al., 2020; Du et al., 2022; Fedus et al.. 2022: Zhou et al., 2022: Dai et al., 2024; Liu et al., 2024b). These models organize every FFN layer into multiple large FFN blocks referred to as experts, and selectively activate a few experts for each input token via a routing mechanism (Lepikhin et al., 2020: Zoph et al., 2022: Sun et al., 2024). Our method share the same spirit as MoE by dynamically activating subset of parameters. However, our method selects the activated parameters given the input prompt, and reuses the same activated parameters during decoding. Although this choice loses the flexibility of us- ing different parameters per token, it significantly reduces weight loading costs for decoding. In this regard, our model is a sparse model designed for on-device scenarios where both memory and computational resources constrained. Another difference is that our method performs more fine- grained selection of parameters by activating or pruning each FFN dimension independently, which enhances model expressivity.
Various methods have been proposed for structured prun- ing of LLMs (Yang et al., 2024b; Kim et al., 2024; Kurtić et al., 2024; Dery al., 2024). -.AM-RRUNER (Ma al., 2023) adopt the gradient information to find unimportant components in LLMs and remove them. SLICEGPT trans- forms each weight matrix in transformer blocks into smaller one by applying orthogonal transformations to re- duce the embedding dimensions of weight matrices. SHORT- GPT (Men et al., 2024) proposes to identify and remove those less important layers, where the layer importance is measured by the similarity between inputs and outputs of that layer. In comparison, other optimization- based methods directly learn the parameter masks. For example, SHEARED LLAMA (Xia et al., 2024) use the HARDCONCRETE mask- ing (Louizos et al.. 2018; Wang et al., 2020) to generate differentiable masks and optimize the model and masks on pre-training data. Our method also directly optimize the sparsity predictor and the LLM, and we further extend static pruning input-dependent pruning.
3. Method
In this section, we elaborate on the details of IFPRUNING, including the architecture design, data mixture, and training method. We focus on pruning the feed forward blocks (FFNs) in this work, but our method can be casily extend to pruning other components such as attention heads.
Contextual sparsity Our approach is also directly moti- vated by the contextual sparsity of LLMs (Liu et al., 2023; Akhauri et al., 2024; Lee et al., 2024). Previous work has identified the existence of input-dependent sub-networks (e.g., attention heads and MLP parameters) within ReLU- based LLMs that can generate approximately the same out- put as the full model for input. By predicting such sparsity patterns at each decoding step, can achieve favorable balance between accuracy and speedup. But state- of-the-art LLMs (Dubey et al., 2024; Liu et al., 2024a; Yang et al., 2024a) design MLP blocks based on more complex non-linear activation functions such as SwiGLU (Shazeer, 2020), SiLU (Elfwing et al., 2018; Ramachandran et al., 2017) and GELU (Hendrycks & Gimpel, 2016) that do not inherently induce sparsity (Mirzadeh et al., 2023; Song et al., 2024). Therefore, directly predicting the sparsity patterns can lead to significant performance degradation (Zhou et al., 2024; Dong et al., 2024). In comparison, we co-optimize the sparsity predictor and the LLM with Ron-ReLU acti- vation functions to achieve better contextual sparsity with minimum performance degradation. Also, most contextual sparsity methods require predicting sparsity and loading different parameters at each decoding step. Our method climinates this overhead by selecting the parameters based on the input or task description before decoding starts. The selected parameters are fixed for the entire decoding process, avoiding the parameter reloading cost.
3.1. Overview of Structured Pruning
Denote the hidden dimension of the LLM as d, the interme- diate dimension of the FFN blocks as dits the input length as n, and X E Raxdd as the input of a transformer FFN block Fit (-). The goal of our structured pruning method is to reduce the FFN intermediate dimension from din to fitn- Without loss of generality, consider a standard FFN block defined as
F (X) = FF2 = (XWW))
(1)
where W, E Bdxdm W2 E Rddm xd are weight matrices, and a is the non-linear activation function. The structured pruning of the FFN block can be expressed as applying a mask variable m {0, 1 the output of the first linear transformation
Fin (X, m) = FF2 O m).
(2)
where @ is an element-wise multiplication between and each row of FF) For each dimension of m, m; O indicates that the i-th column of W1 and i-th row of W2 are pruned. This is because the output Fmn (X, m) is equiva- lent to the output of the FFN layer after we prune the i-th column of W1 and i-th row of Wz. Here m satisfies the
3

---

Instructton-Followingg Pruning for Large Language Models
sparsity constraint E,m, m. Iin- where Lin is the target intermediate dimension of the FFN blocks after pruning.
3.3. Model Training
The training of IFPRUNING incorporates two stages. We first perform continued pre- pre-training in which we initialize our model using a pretrained dense model, and then perform supervised fine-tuning (SFT) on instruction followinn data. In what follows, we elaborate on the details of the two training stages.
3.2. Architecture
As shown in Figure 1, our architecture comprises two key components: sparsity predictor and dense LLM to be dynamically pruned. For any given user prompt, the spar- sity predictor generates masks that are applied to the LLM backbone, pruning the corresponding rows and columns of the FFN blocks.
Continued pre-training Learning to select input-specific sub-networks may require lot of training data. Instead of directly training the models on the SFT data only, we first use pre-trate pre-training data to jointly optimize the sparsity predictor and masked LLM. Specifically, denoting the input text as x (x1,..., Fn)) we split it into K consecutive chunks with fixed size:
Sparsity predictor The sparsity predictor consists of two modules: O a much smaller LLM backbone to ex- tract the features of user prompts and @ a mask prediction head. Specifically, the LLM backbone takes the prompt = (21,...,23)) as input with length n, and we use the hidden states of the last token Zn in the last layer to represent the prompt. The mask prediction head is two-layer MLP, which predicts the masks given the prompt presentations. The output of the FFN mask prediction head is the masking score z E IR Lxde, where L is the number of layers of the LLM. We include more details about the architecture of the sparsity predictor in Appendix A.I.
(4) where = n/K is the fixed size of each chunk. We then use the each chunk to select parameters of the LLM for the next token predictions in the next chunk, i.e.,
x = 2(k ((k-1)s+1-..
Thes k = K,
K L.
Σ Σ (( k=1 x.En(e+1)
(5)
Given the predicted masking score 2, mask generation operator will be applied to z to convert it to the mask m E [0, 1] which contains fitn nonzero elements, In this paper, we use the SoftTopK (Lei et al., 2023; Ainslie et al., 2023) algorithm to generate differentiable m, but we also acknowledge that other algorithms such as the HardConcrete masking (Louizos et al., 2018; Wang et al., 2020) are also applicable. Particularly, given the FFN masking score 2, SofTTopK converts it to masks m via:
where O refers to the parameters of the LLM, m (*)) is the predicted mask based on chunk x(*), <x: 0 m mx)) is the next token prediction distribution from the L.M with (k) applied, and f(-) is the Cross-Entropy loss. Since the chunks are consecutive, the sparsity predictor can learn to utilize the contextual information of each chunk to predict which parameters of the LLM are best suited for the next token prediction in the next text chunk. Because both the sparsity predictor and LLM are co- co-optimized in this stage, it provides a good initialization for the fine-tuning stage.
( =(2( m
A (i) C (o,
(3)
Supervised fine-tuning Starting from the models after the first stage, we train the sparsity predictor and the LLM on a supervised fine-tuning dataset that contains several million examples. During training, the input prompts will be fed into the sparsity predictor which predicts the masks for each input. The LLM is then masked and optimized to predict the target outputs conditioned on the input prompts. Our SFT data contains a diverse set of prompts to predict the sub-netwee nebwors...... Some prompts specify the task with a task description and an input, while others include few-shot examples along with the input. Lastly, many examples only contain task description, like "write comprehensive blog post about the top 10 most eco-friendly cities in the world.",
Here =(0), (1)) and !)) represent the i-th row of each ma- trix, g(-) da. - [0, 1] 1]dra is normalization function, and Top(-, tmm) E {0, dd is an indicator function that returns a binary mask indicating the top-k values in A. The normalization function g(-) ensures that À satisfies the spar- sity constraint, i.e., Σx ( titn. where linn is the target size of the FFN layers. More details of SoftTopK can be found in the previous work (Lei et al., 2023; Ainslie et al., 2023).
Masked LLM During training, the LLM takes the masks m as an additional input and prune its FFN blocks. We use standard next token prediction loss computed over tokens within a training batch, and we co-optimize the LLM and the sparsity predictor.
For multi-turn conversational data, we only use the first human message as the prompt for sub-network selection. During training, with the exception of removing all instances of personal data, all these prompts are fed directly into the
4

---

Instruction-Folllwwl Pruning for Large Language Models
Implementation We use the AXLeam (Apple, 2023) sparsity predictor without any additional processing. This approach maximizes flexibility during inference, allowing the predictor to generate a sub-network regardless of the prompt format during inference. The training objective follows the standard SFT approach, namely minimizing the cross-entropy loss on the target outputs. Through this process, the model learns to selectively activate the most suited parameters for different input examples.
framework and JAX (Bradhury et al., 2018) for model train- ing. Following the previous work (Dubey et al., 2024), all the pre-trained model used in our experiments are achieved by performing two-stage pre-training. All models are pre- trained with a batch size of 2048 and a total number of ST tokens, except that the DENSE-3B is trained for 9T tokens. The SFT training for the baselines and our method is per- formed with a batch size of 1024 for 60k training steps. We use the same pre-train and SFT data mixture for all models.
4. Experiment
In this section, we conduct empirical evaluations to assess the effectiveness of our proposed method.
Evaluation configurations We include the following tasks for evaluation:
4.1. Experiment Setup
Instruction-folloinng We include IFEval (Zhou et al., 2023), Al AlpacaEval 2. 0 (Dubois et al., 2024), and Arena Treen-HHard-ae Burrd-Auto (Li et al., 2024) for evaluation. We report the prompt-level and instruction-level accuracy IFEval, the length-controlled win rate on AlpacaEval 2.0, and win rate on wrena-Hrd-Auus
Dataset and backbone models Our models are trained on an internal SFT dataset with several million examples. We also follow the setup used in TCLU and sample additional 800K examples from the FL.AN-V2 collection (Chung et al., 2024) to enhance task prompt diversity. The experiments conducted using series of pre-trained LLMs. Partic- ularly, for the sparsity prediction component, we initialize it with a 302M model that has been pre-trained on web- crawled data. To test the performance of our method across various model scales, we separately train three different models that use 6B, 9B, and 12B parameters, respectively, in the masked LLM component. For all these models, our approach activates 3B parameters (and prunes the rest of the parameters). More details about the model architecture are given in Appendix A.1.
Coding tasks. We evaluate the pass @ass@) I perfor- mance on HumanEval-pythen (Chen et al., 2021), mbpp (Austin et al., 2021), and MultiPL-E (Cassano et al., 2022). For MultiPL-E benchmark, we use the Swift subset for evaluation.
Math. We use GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021b) to evaluate the math capabilities of LLMs. we report the accuracy with few- shot examples on both datasets (8-shot for GSM8K and 4-shot for MATH).
. Core Text. We include a set of tasks to evaluate the model's core capabilities of natural language understand- ing, scientific knowledge, and reasoning. We report the zero-shot performance on ARC- CCCCChhalengge (Clark et al., 2018), ARC-easy (Clark et al., 2018), Hel lalwag (Zellers et al., 2019), WinoGrande (Sak- aguchi et al., 2021), PiQA (Bisk et al., 2020), LAMBADA-OpenAl (Paperno et al., 2016), and SciQ (Welbl et al., 2017).
Comparison baselines We compare method with the following models: 1 DENSE-3B. We compare with 3B dense LLM that is trained using twice as many pretraining tokens compared to our models. @ PRUNING+DISTILL We also compare our method with static pruning approaches. Similar to recent work such as Sheared LLaMA (Xia al., 2024), LLAMA 3.2 (Meta AI, 2024), and MINI- (RON(Sresnivas al., 2024), we include a baseline where a 3B dense LLM is pruned and distilled from larger pre- trained LLM. we first prune the larger LLM into a dense 3B model by learning masks on the FFN layers similar to Sheared LLaMA (Xia et al., 2024). After pruning, the model undergoes further continuous pre-training through knowl- edge distillation (Hinton, 2015), using larger dense model with 12B parameters the teacher model. Therefore, this approach serves as a stronger baseline compared to models using pruning alone. More details of the knowledge distilla- tion can be found in Appendix A.2. ① DENSE-9B. We also include a 9B dense model without pruning as a reference for upper-bount performance.
MMLU (Hendrycks et al., 2021a). We evaluate the 5-shot performance and report the multiple-choice accuracy.
Tool use. We evaluate the tool use performance on MMAU (Yin et al., 2024) and report the performance on Tool Execution and Tool Planning.
We mainly use LM- Evaluationnnnnnoeee Sateuuatioo--Haarrsssse (Gao et al., 2021) to evaluate the tasks, with the exception of instruction- following and tool-use tasks, which are based on their offi- cial implementations.
5

---

Instruction-Folllwiing Pruning for Large Language Models
Table Performance comparison between IFPRUNING and other dense M.M.s. PRUNING+DISTILL is first pruned from larger model into 3B parameters and undergoes continuous pre training with knowledge distillation using 12B LLM as the teacher model. Three versions IFPRUNING are included which activate parameters LLMs with 6B, 12B parameters. The best results are highlighted in bold and the second-best results are underlined. DENSE-9B is included for reference.
PRUNING+
IPPUNIING 9B 9B-+B
Category
Dataset
DENSE
DENSE 9B
DISTILL 3B
6B 6B-+3B
12B-3B
IFEval-Instruccee IFEval-Prompt
85.0
86.7
83.9
85.9
85.3
87.5
Instruction
77.8
78.9
78.6
81.7
Following
Alaa 2.0
27.3
30.0
29.0
31.3
32.5
38.6
Deea--Hard-Ale
15.8
16.4
18.0
19.8
HumanEval
35.2
37.1
41.0
42.4
43.3
46.5
MultiPL-E
39.0
37.9
37.6
41.8
43.0
44.0
Coding
MBPP
28.8
38.0
37.4
41.8
42.8
42.2
Average
34.3
37.7
38.7
42.0
43.0
44.2
Tool Execution
74.8
74.4
75.8
75.7
73.9
76.5
Tool Use
Tool Planning
25.2
17.5
36.6
45.4
37.5
46.5
GSM8K
69.3
70.0
72.2
72.0
70.2
75.4
Math
MATH
31.8
32.7
36.7
37.1
37.3
ARC-Challenge
47.4
46.2
50.4
50.4
51.9
53.9
ARC-Easy
79.3
79,9
81.9
81.4
82.3
83.4
HellaSwag
53.0
53,0
54.1
55.5
55.8
57.7
Core Text
LAMBDA
66.5
68.2
68.8
68.9
69.0
70.8
PIQA
77.4
77.3
77.7
78.0
78.7
79.4
SciQ
95.9
96.0
96.7
96.5
96.9
WinoGrande
69.8
69.1
67.7
67.0
68.4
74.3
Average
69.9
70.0
71.0
71.1
71.8
73.8
MMLU
MMLU
61.8
62.8
65.5
66.1
67.8
4.2. Evaluation with Input-Specific Masks
on Core Text tasks, highlighting its broad applicability. The effectivenness of our approach is further underscored by its performance relative to the 9B-parameter "upper bound" model. Notably, on coding, math, and MMLU benchmarks, our method closely approaches upper-hound performance.
In this section, we evaluate our model using input-specific masks, which align with the training scheme. For all ex- amples across the included datasets, we generate masks by feeding the testing question and few-shot examples (if appli- cable) into the sparsity predictor. The LLM is then pruned with the resulting mask and performs inference the same input. The datasets provide a diverse range of inputs for the sparsity predictor, including combinations of few-shot examples and testing questions (e.g., MATH and MMLU) and question-only formats (e.g., AlpacaEval and IFEval).
Second, IFRUNNIG demonstrates superior performance compared to the structured pruning method. Please note that the structured pruning model, PRUNING+ benefits from additional training signals due to knowledge distil- lation from larger teacher model as the teacher. Never- theless, IFRRNINNG consistently outperforms this baseline. Our method achieves higher performance across variety of benchmarks, including AlpacaEval, Arena Hard, math problems, coding tasks, tool use. MMLU, and most Core Text tasks.
Overall comparison We visualize the evaluation results in Table 1. We highlight the following observations. First, with an equivalent number of activated parameters, IFPRUN- ING significantly outperforms the dense LLM, demonstrat- ing its ability select the most relevant parameters for var- ious inputs effectively. Specifically, IFPRUNING achieves 5% and 4% higher win rate over the dense LLM Al- pacaEval and Arena Hard, respectively. Also, our method improves upon the dense baseline by 6% to 14% on coding tasks and by 3% to 5% on math benchmarks. We also ob- serve substantial improvement on Tool Use and the MMLU benchmark. Finally, IFPRUNING shows strong performance
Third, we observe a clear improvement in performance with IFPRNNING as the size of the L.M increases. As the source model size scales from 6B 9B and then to 12B, there is noticeable performance boost most of the datasets.
Scaling behavior of dense models and IFPRUNING We illustrate the scaling behavior of dense LLMs (without prun- ing) and our IFPRUNING method in relation model size

---

CntruuctionFFollwinng Pruning for Large Language Models
45
40
68
42
35
65
39
30
62
36
25
20
30
68
9B
126
38
9B
128
68
128
38
68
98
126
Coding
Math
AlpacaEval IFPruning
MMLU
Dense LLM
Figure Scaling behavior of dense models and IFPRUNING, IFPRNNNGG activates 3B parameters for each input. The x-axis represents the total number of LLM parameters for dense models and IFPRUNING, while the y-axis indicates the performance scores.
NAMLESESS
MAALESMAR
SAMLS PREN
COMMS
MORLUH
OPfcadee
oale
Cade Aias
AMtustie
Code upau
Mon
Mottidbe
oster
Gade toad
Figure 3. Sub-network overlap rates for the first layer (left), layer 16 (middle), and the last layer (right) of the LLM.
(total number of parameters) in Figure 2. In approach, we consistently activate 3B parameters across LLMs with varying numbers of total parameters. In general, increasing the size of the LLM leads to performance improvements. This trend is especially clear on Math, coding, and MMLU tasks, where IFPRNNING achieves performance levels close the upper bound. In contrast, we observe less performance gain the Al Eval dataset, suggesting opportunity for improvement and/or better understanding of the scaling behavior in future work.
history AMMLU-Histe and international law (MMLU-Law) subsets from MMLU. For general instructions, GPTeacher (Teknium et al., 2024) dataset that is not in- cluded in our training data. The overlap rates for the first, the last, and a middle layer (layer 16) of the pruned models are visualized Figure 3.
the
We highlight the following findings. First, in the lower lay- ers, especially the first layer, the LLM tends activate very similar sub-networks for different inputs. As we move to higher layers, the parameter selection becomes more diverse. Second, as shown the figure, IFPRUNING activates dis- tinct sub-networks for different domains higher layers. For instance, models pruned for MMLU-CS have substan- tial overlap with those for Code-Alpaca, significantly more than with other domains. Similarly, models for MMLU- Math, MMLU-Physics, and GSM8K share high propor- tion of activated parameters, which diverge notably from the activation patterns for MMLU-History. MMLU-Law, and G.TTeacher. Third, the overlap rate along the diagonal of the heatmap is very high, reflecting the strong similarity between models for inputs within the same domain. Finally, dataset
Interpretability parameter selection In this section, examine the parameter pruning and selection patterns across different domains and visualize the similarities be- tween them. We measure the similarity between two pruned models based on their overlap rate, defined as the propor- tion of parameters commonly activated by both models. More details of the implementation can found in Ap- pendix A.3. We perform this analysis of IFPRUNING us- ing our 6B-+3 model. We include the following datasets testing domains. For math, the college mathematics subset from MMLU (denoted as MMLU-Math). For computer science, we use the college computer science subset from MMLU (denoted as MMLU- CS) and Code-Alpaca (Chaudhary, 2023). We also include the college physics (MML.UPhys..) high school European
GSM8K and
expected, GPTTeacher is an instruction-followwe that covers wide range of domains, therefore it does not exhibit significant domain-specing characteristics, resulting in a self-overlap rate that is lower than in other domains.

---

Instruction-Folllwwii Pruning for Large Language Models
Table Performance comparison between IFPRNNIN with per-input masks and per-task masks. In the per-input setting, IFPRUNING prunes the LLMs and activates different sub-networks for each input. In the per-task setting, IFPRUNING selects single sub-network for all inputs within the same dataset based human-written written task instruction.
6B BB+BB
9B BB+BB
12B
B+BB
Category
Dataset
DENSE 3B
Per-Input
Per- Per-Task
Per-Input
Per-TTas
Per-Input
Per-TTask
HumanEval
35.2
41.0
39.0
42.4
40.9
43.3
45.3
MultiPI.-E
39.0
37.6
38.5
41.8
42.8
43.0
44.2
Coding
MBPP
28.8
37.4
39.0
41.8
38.2
42.8
40.4
Average
34.3
38.7
38.8
42.0
40.6
43.0
43.3
Math
MATH
31.8
36.2
36.6
36.7
36.8
37.1
37.8
MMLU-physics
49.6
50.8
50.7
55.5
54.2
55.7
54.4
MMLU-math
43.7
42.8
41.8
44.0
45.8
44.2
45.5
MMLU
MMLU-history
73.7
75.9
75.2
78.0
74.7
78.5
77.8
MMLU-heealth
59.7
61.9
62.0
64.4
63.4
65.1
63.7
MMLU- -MML---ussness
75.3
73.5
75.3
76.6
75.3
75.5
76.1
MML.U-cconomics
60.0
59.7
59.2
66.8
65.6
66.3
63.0
EN-DE
33.9
35.5
35.8
35.9
35.8
33.5
36.4
EN-ES
26.9
27.4
27.2
27.0
27.2
26.8
27.5
EN-FR
47.2
47.0
47.0
47.6
EN-IT
28.8
30.2
30.1
30.3
30.5
31.0
30.2
Translation
EN-PT
46.6
47.0
47.9
47.1
47.2
47.4
48.1
ZN-ZH
35.0
41.7
39.1
41.5
40.4
42.5
40.9
Average
36.0
38.2
37.8
38.1
38.0
38.1
38.7
In summary, the activated parameters are interpretable and reveal clear patterns in different domains. It aligns with the high performance of our method, the LLM dynamically activates the parameters most suitable for each input.
on the predictions made by the sub-network. The task- specific instructions used for each dataset provided in Table 6 in Appendix A.7. For the MMLU dataset, which consist of multiple subsets representing diverse domains, we group similar subsets into broader domains and assign a single instruction to each domain. For example, the subsets "astronomy," "college physics," "conceptual physics," and "high school physics" are grouped into the domain "MMLU- physics," and single instruction is written for this domain. A detailed list of the subsets included in each MMLU do- main is provided in Appendix A.6.
Ablation study We also conduct an ablation study the impact of continued pre-training in Appendix A.4. Incorpo- rating continued pre-training before SFT leads to consistent and substantial improvements across all benchmarks.
4.3. Evaluation with Task-Specific Masks
An additional noteworthy capability of IFPRUNING is task- specific pruning. While IFPRUNING can effectively prune a model on a per-input basis, the algorithm also demonstrates the ability to select one sub-network that can be used for different inputs within the same task domain. We design the following experiment demonstrate the performance of our method with task-specific pruning.
The performance is shown in Table 2. We compare the task-specific pruning capabilities our algorithm with the dense L.M with 3B parameters and the standard IFPRUN- with per-input mask. Our key findings are as follows. First, IFPRNNING can generate high-quality task-specific masks without additional training. performance improvements of the task IFPRUNING over the dense baseline across all datasets. When apply- ing per-task masks, IFPRUNING still outperforms the dense LLM by 5%-12% on coding tasks, 5%-6% on MATH, and 5% on MMLU subsets. For translation tasks, IFPRUN- ING selects the appropriate sub-networks without additional training, achieving improvement of 4.8 points in BLEU score on average. Note that these task-specific masks are directly predicted by the sparsity predictor, requiring no additional fine-tuning of either the sparsity predictor or the LLM for each task. This highlights the "zero-shot" pruning
observe substantial
Specifically, we evaluate our models on math problems, coding tasks, MMLU, and machine translation tasks from Flores- 101 (Goyal al., 2022). For each dataset, we man- ually design concise instruction that describes the task. For example, the instruction for the HumanEval dataset expert Python programmer. Write the
can be: "You code which should pass the tests." The sparsity predictor takes this instruction as input and selects the correspond- ing sub-network. This sub-network then applied consis- tently to process all testing examples from that dataset, and the model performance on the dataset is evaluated based
8

---

Cnteruction-Follwiing Pruning for Large Language Models
capability of our method. Second, compared to the standard input-specific IFPRUNING, the task-specific IFPRUNING exhibit minimal performance degradation across math prob- lems, coding tasks, and MMLU. The performance gaps are mostly under I %, indicating the robustness of our method.
Table 3. Inference latency evaluation. The source models are re- duced to 3B by pruning the FFN layers.
Parameter
Parameter Loading
Model
Prefill
TTFT
Decode
Selection
Our model
4.4. Inference Latency Evaluation
DENSE-9B DENSE-3B IFPRNINN
0.846
0.846
7.16
0.356
0.356
5.68
Given the strong performance of IFPRUNING, we further analyze and evaluate its efficiency improvement in this sec- tion. The inference pipeline of our method consists of three steps: Parameter selection: given an input, the sparsity predictor selects a subset of parameters activate. @ Pa- rameter loading: the selected parameters are loaded, and the corresponding sub-network is cached as a dense model. @ Generation: the pruned model will encode the input (pre- fill) and generate a response (decode), following the same procedure as standard dense or MoE models. Sub-network selection only involves a forward pass through the sparsity predictor, which is lightweight (300M parameters in our experiments), so the time cost is minimal. For parameter loading, maintain small base model and update its weights with the selected parameters. The generation step is identical to standard inference, allowing our method to achieve the same latency as a smaller dense model.
0.070
0.043
0.358
0.471
5,58
Llama-33 3.1
DENSE-8B DENSE-3B
0,702
0.702
5,47
0.317
0.317
3.52
IFPRUNING
0.070
0.016
0.315
0,402
3,53
pruning (DENSE-3B baseline), while offering ipput-specific adaptivity and superior accuracy. The inference latency evaluation has further verified that our method is suitable for practical deployment on edge devices, as it accelerates inference while achieving superior model performance.
Finally, we compare the inference latency of our method and an MoE model with similar number of activated parameters in Appendix A.5. We show that for the on-device inference scenario with small inference batch size, the cost of MoE is multiple times higher than dense model and our method.
To illustrate the inference speedup, we run the following simulation. Specifically, we evaluate the inference latency of the models used in our main experiments: DENSE-9B, DENSE-3B, and IFPRUNING 9B- + 3B. Additionally, we test our method on an open-source LLM, LLaMA-3.1-881 1-BBB- Instruct, where the FFN layers are pruned to reduce the model from 8B to 3B parameters, while keeping other com- ponents unchanged. Although the tests are done on GPUs, we used batch size 1 and 4 generations per query, reflect- ing on-device usage. The input length is set to 4,000 and the generation length is set to 100. We report the time-to- first-token (TTFT) and the decoding time, both measured in seconds. For dense models (8B and 3B), the TTFT consists of pre-filling only. For our method, we break down TTFT into its components: parameter selection, parameter loading, and pre-filling. The evaluation results are in Table 3.
5. Conclusion
In this paper, we extend the structured pruning for LLMs with a dynamic scheme, where the LLM is pruned into different sub-networks given the prompts. With a simple ar- chitecture and straightforward training process, our method can significantly improve the model performance compared to dense LLMs with the same number of activated param- eters. In the future, we will test our method when pruning other components of L.MS such as attention heads and hidden dimensions.
This work also opens several promising directions for future exploration. First, we focus dynamically pruning I.LMs. based on contextual information. While this approach is well suited for models on consumer-facing devices such as phones, additional challenges remain for server-side serving when the input is a batch of user requests containing differ- ent tasks. One possible solution to cluster user requests so that requests within the same batch share similar activated sub-networks. Second, the current training method relies end-to-ent optimization, which may not fully utilize the training examples. The performance could be improved by adopting more advanced training strategies. For instance, using contrastive loss could encourage higher overlap rates among sub-networte for similar inputs, making the model more robust.
We highlight the following conclusions. First, similar to standard structured pruning, our dynamic pruning scheme introduced can also significantly reduces inference latency for large-scale models. By pruning a 9B or 8B LLM to 3B, the TTFT is decreased by up to 57% and decoding time is decreased by up to 41%. total, IFPRUNING can achieve up to 1.8x speedup compared to the original dense LLMs. Second, the overhead introduced by dynamic pruning and parameter caching is negligible. For each input, the param- eter selection and loading only take less than O. 1s in total (1-2% of the total generation time). Finally, despite dynamic masking, the runtime of IFPRNNING is on par with static

---

Instruction-Following Pruning for Large Language Models
Acknowledgements
language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Shiyu Chang acknowledges support from National Sci- ence Foundation (NSF) Grant IIS 2132233825, NSF Grant IIS- 2207052, and NSF Grant IIS-2302730.
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., Necula, G., et al. Jax: composable transformations of python+ numpy programs, v0. 3.13, 2018.
Impact Statement
Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Phipps- Costin, L.,, Pinckney, D., Yee, M.-H., Zi, Y., Anderson, C. J., Feldman, M. Q., et al. Multipl-e: A scalable and extensible approach to benchmarking neural code genera- tion. arXiv preprint arXiv: 08227, 2022.
In this paper, our primary goal is to develop an algorithm that can dynamically select the most suited parameters of an LLM given an input prompt. Our method designed to improve both inference efficiency and the performance of LLMs. The training data has been carefully filtered to ensure quality and safety; for instance, all instances of personal data in the SFT data were removed to uphold privacy standards. All the data collection process strictly adheres to ethical guidelines for data use, ensuring that no private or sensitive information is included in the training or evaluation process.
Chaudhary, S. Code alpaca: An instruction Ilama model for code generation. https: //tpthub.tb///t///// com/ sah 128011 codea paca, 2023.
Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Paviov, M., Power, A., Kaiser, L., Bavar- ian, M., Winter, C., Tillet, P., Such, F. P., Cummings, D., Plappert, M., Chantzis, F., Barnes, E., Herbert-Voss, A., Guss, W. H., Nichol, A., Paino, A., Tezak, N., Tang, J., Babuschkin, I., Balaji, S., Jain, S., Saunders, W., Hesse, C., Carr, A. N., Leike, J., Achiam, J., Misra, V., Morikawa, E., Radford, A., Knight, M., Brundage, M., Murati, M., Mayer, K., Welinder, P., McGrew, B., Amodei, D., McCandlish, S., Sutskever, I., and Zaremba, W. Evaluating large language models trained on code. 2021.
Also, The sparsity- sparsity-inducing inducing mechanism proposed in this work does not introduce additional risks of bias or harm with the underlying large language model. Furthermore, our method enhances computational efficiency, potentially reducing the environmental impact of large-scale model inference. While acknowledging that any machine learning model has the potential for misuse, we focus on safe and task-specific applications, such as math, coding, and tool use. We encourage further research into mitigating biases and unintended consequences in language models and remain committed to the responsible and ethical advancement of AI technologies.
References
Chen, T., Frankle, J., Chang, S., Liu, S., Zhang, Y., Wang, Z., and Carbin, M. The lottery ticket hypothesis for pre- trained bert networks. Advances in neural information processing systems, 33:15834 15846, 2020.
Ainslie, J., Lei, T., de Jong, M., Ontanon, S., Brahma, S., Zemlyanskiy, Y., Uthus, D. C., Guo, M., Lee- Thorp, J., Tay, Y., et al. Colt5: Faster long-range transformers with conditional computation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 5085-50000 2023.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. restuuttionfinetuneed Scaling instruction finetuned language models. Journal of Machine Learning Research, 25(70):1 2024.
Akhauri, Y., AbouEElhamaeed, A. F., Dotzel, J., Zhang, Z Rush, A. M., Huda, S., and Abdelfattah, M. S. Shad- owllm: Predictor based contextual sparsity for large lan- guage models. arXiv preprint arXiv:2406.1660 16635, 2024.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A.. Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. 2018.
arXiv preprint
Apple. The axlearn library for deep learning, 2023. URL https: /estps://gitain /gtthbb..ttttt///// com/ hammmappl/////em axlearn.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 14168, 2021.
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, F., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.0772 2021.
Dai, D., Deng, C., Zhao, C., Xu, R., Gao, H., Chen, D., J., Zeng, W., Yu, X., Wu, Y., et al. Deepseekmoe: Towards ultimate expert specialization in mixture-of offuurr--o-expertss lan- guage models. arXiv preprint arXiv:240I /060666 2024.
Bisk, Y. Zellers, R., Bras, R. L., Gao, J., and Choi, Y. Piqa: Reasoning about physical commonsense in natural
10

---

CnstructionFFollowing Pruning for Large Language Models
Dery, L.,, Kolawole, S., Kagy, J.-F. Smith, V., Neubig, G., and Talwalkar, A. Everybody prune now: Structured pruning of Iims with only forward passes. arXiv preprint orXiv:2402.05406, 2024.
Advances in neural information processing systems, 28, 2015.
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), arXiv preprint A.Kv:1600.004111 2016.
Dong, H., Chen, B., and Chi, Y. Trompt-prompter mixture of experts for efficient IIm generation. arXiv preprint MMXiv:2404.01365, 2024.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a.
Du, N., Huang, Y., Dai, A. M., Tong, S., Lepikhin, D., Xu, Y., Krikun, M., Zhou, Y., Yu, A. W., Firat, O., et al. Glam: Efficient scaling of language models with mixture-of- experts. In International Conference Machine Learn- ing, pp. 5547-5569. PMLR, 2022.
Hendrycks, D., Burns, C.,, Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. NeurIPS, 2021b.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A., et al. The Ilama herd of models. arXiv preprint arXiv:2407.2178. 2024.
Hinton, G. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253/ 2015.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.028.2. 0222402834, 11, 2024.
Dubois, Y., Galambosi, B., Liang, P., and Hashimoto, B. Length-controlled alpacaeval: A simple way debias automatic evaluators. arXiv preprint (XXx:2444.044777 2024.
Kurtić, E., Frantar, E., and Alistarh, D. Ziplm: Inference- aware structured pruning of language models. Advances in Neural Information Processing Systems, 36, 2024.
Elfwing, S., Uchibe, E., and Doya, K. Sggmoid-wiihhee linear units for neural network function approximation in reinforcement learning. Neural nenworks, 107:3-11, 2018.
Lee, J.-Y., Lee, D., Zhang, G., Tiwari, M., and Mirhoseini, A. Cats: Contextually- thresholding for sparsity in large language models. arXiv preprint 2Xiv:2404.08766 2024.
Fedus, W., Zoph, B., and Shazeer, N. Switch transformers: Scaling trillion parameter models with simple and ef- ficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Lei, T., Bai, J., Brahma, S., Ainslie, J., Lee, K., Zhou, Y., Du, N., Zhao, V., Wu, Y., Li, B., al. Conditional adapters: eurameter-efficient transfer learning with fast inference. Advances in Neural Information Processing Systems, 36: 8152 81 8172, 2023.
Frankle, J. and Carbin, M. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In Interna- tional Conference on Learning Representations, 2018.
Gao, L., Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., et al. A framework for few-shot language model evaluation. Version vO. 0.I. Sept, 10:8-9, 90, 2021.
Lepikhin, D., Lee, H., Xu, Y., Chen, D., Firat, O., Huang. Y., Krikun, M., Shazeer, N., and Chen, Z. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006. 16668, 2020.
Goyal, N., Gao, C., Chaudhary, V., Chen, P.-.,, Wenzek, G., Ju, D., Krishnan, S., Ranzato, M., Guzmán, F., and Fan, A. The 101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522-538, 2022.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, E., and Stoica, I. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. /9399, 2024.
Li, Z., Wallace, E., Shen, S., Lin, K., Keutzer, K., Klein, D., and Gonzalez, J. Train big, then compress: Rethinking model size for efficient training and inference of trans- formers. In International Conference on machine learn- ing. pp. 5958-5968. PMLR, 2020.
Gunter, T., Wang, Z.., Wang, C., Pang, R., Narayanan, A., Zhang, A., Zhang, B., Chen, C., Chiu, C.-C., Qiu, D., et al. Apple intelligence foundation language models. 2024.
arXiv preprint
Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts
Han, S., Pool, J., Tran, J., and Dally, W. Learning both weights and connections for efficient neural network.

---

Instruction-Folllwiing Pruning for Large Language Models
language model. arXiv preprint arXiv:2405.04434, 2024a.
Ramachandran, P., Zoph, B., and Le, Q V. Searching for activation functions. arXiv preprint arXiv: 2017.
Liu, L.,, Kim, Y.J., Wang, S., Liang, C., Shen, Y., Cheng, H., Liu, X., Tanaka, M., Wu, X., Hu, W. al. Grin: Gradient- informed moe. arXiv preprint ar arXiv:2409. 121.:19.....1,, 2024b.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y.
Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106. 2021.
Liu, Z., Wang, J., Dao, T., Zhou, T., Yuan, B., Song, Z., Shrivastava, A., Zhang, C., Tian, Y., Re, C., al. Deja Contextual sparsity for efficient Ilms at inference time. In International Conference Machine Learning, pp. 22137-22176. PMLR, 2023.
Shazeer, N. Glu variants improve transformer. arXiv
preprint arXiv:2002.05202, 2020.
Song, C., Han, X., Zhang. Z., Hu, S., Shi, X., Li, K., Chen, C., Liu, Z., Li, G., Yang. T., et al. Prosparse: Intro- ducing and enhancing intrinsic activation sparsity within large language models. arXiv preprint arXiv:2402. 13516, 2024.
Louizos, C.,, Welling, M., and Kingma, D. Learning sparse neural networks through 10 regularization. In International Conference on Learning Representations, 2018.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. Advances in neural information processing systems, 36:21702-21720, 2023.
Sreenivas, S. T., Muralidharan, S., Joshi, R., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, P. Lim pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 1ei.:2408..199.... 2024.
Men, X., Xu, M., Zhang, Q.. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language mod- els are more redundant than you expect. arXiv preprint arXiv:2403.03885, 2024.
Sun, H., Lei, T., Zhang, B., Li, Y., Huang, H., Pang, R., Dai, B., and Du, N. Ec-dit: Scaling diffusion transform- ers with adaptive expert-choice routing. arXiv preprint arXiv:2410.0209g, 2024.
Meta AI. Llama 3 and vision: Bringing next- gen ai to edge and mobile devices, 2024.
Team, Q. Qwen2.5: A party of foundation models, Septem- ber 2024. URL https: htwp//wwwmmmmmmmmm/mm/..///m.lmm git io/
https: meta www/blll//oooooooooooooooo..ooo. URL 2-2-coonett2224-visitm 11ama-3-2-com
le devi ce f2....
Accessed: 2024-11-10.
Teknium, Machina, L. L., and Ashimine, I. E. A collection of modular datasets generated by gpt-4, 2024.
Mirzadeh, I., Alizadeh, K., Mehta, S., Del Mundo, C. C., Tuzel, O., Samei, G., Rastegari, M., and Farajtabar, M. Relu strikes back: Exploiting activation sparsity in large language models. arXiv preprint arXiv:2310.04560 2023.
Voita, E., Talbot, D., Moiseev, F,, Sennrich, R., and Titov, I. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceed- ings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5797-5808, 2019.
Muralidharan, S., Sreenivas, S. T., Joshi, R. B., Chochowski, M., Patwary, M., Shoeybi, M., Catanzaro, B., Kautz, J., and Molchanov, Compact language models via pruning and knowledge distillation. In The eiihtt Annual Conference on Neurul Information Processing Systems, 2024.
Wang, Z., Wohlwend, J., and Lei, T. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Prucessing (EMNLP), pp. 6151-6162, 2020.
Narang, S., Elsen, E., Diamos, G., and Sengupta, S. Explor- ing sparsity in recurrent neural networks, arXiv preprint 2017.
Welbl, J., Liu, N. F,, and Gardner, M. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop on Noisy User diser-generated Text, pp. 94-106, 2017.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N.-Q., Bemardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 1525-
Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. Learning structured sparsity in deep neural networks. Advances in neural information processing systems, 29, 2016.
Xia, M., Gao, T., Zeng, Z., and Chen, D. Sheared llama: Accelerating language model pre-training via structured
1534, 2016.
12

---

EestructionFFolloiing Pruning for Large Language Models
pruning. In The Twelfth International Conference
Learning Representations, 2024.
Yang. A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., et al. Qwen2 technical report. arXiv preprini arXiv:2407. 10671, 2024a.
Yang. Y., Cao, Z. and Zhao, H. Laco: Large lan-
guage model pruning via layer collapse. arXiv preprint arXiv:2402.1 187, 2024b.
Yin, G., Bai, H., Ma, S., Nan, F., Sun, Y., Xu, Z., Ma, S., Lu, J., Kong, X., Zhang, A., et al. Mmau: A holistic benchmark of agent capabilities across diverse domains. arXiv preprint arXiv:2407. 18961, 2024.
Zellers, R., Holtzman, A., Bisk. Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800, 2019.
Zhou, J., Lu, T., Mishra, S., Brahma, S.. Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-folloiinn evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
Zhou, Y. Lei, T., Liu, H., Du, N., Huang. Y., Zhao, V., Dai, A. M., Le, Q. V., Laudon, J., et al. Mixture-of- experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114, 2022.
Zhou, Y., Chen, Z., Xu, Z., Lin, V., and Chen, B. Sir- ius: Contextual sparsity with correction for efficient llms. arXiv preprint arXiv:2409003855 2024.
Zhu, M. and Gupta, S. To prune, not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.0187 2017.
Zoph, B., Bello, I., Kumar, S., Du, N., Huang, Y., Dean, J., Shazeer, N., and Fedus, W. St-moe: Designing stable and transferable sparse expert models. arXiv preprint arXiv:2202.08906, 2022.
13

---

Instroction-Followinn Pruning for Large Language Models
A. Appendix
A.1. Implementation Details
LLM architecture The LLMs used in this paper follow standard LLM design (Meta AI, 2024; Team, 2024), such as grouped -ueryy attention and RMSNorm, with no custom components. We list the detailed model architecture of the pre-trained models used in our experiments in Table 4, All models the same model dimension, attention dimensions and the number of Transformer layers. The only difference is the feed-forward dimension. Accordingly, our method learns to select 6656 FFN dimensions from the 6B and 9B model.
Table Detailed Architecture the pre-trained models used in our experiments
3B
6B
9B
Model Dimension
2048
2048
2048
FFN Dimension
6656
16384
24576
Head Dimension
128
128
128
Num query heads
16
16
16
Num key/value heads
Num layers
56
Sparsity predictor The predictor is built upon lightweight LLM with 302M parameters, built on pre-trained LM backbone. It consists of:
A small pre-trained language model with 302M parameters the feature extractor. Given input, the last hidden state of the final input token will be used the features of the input.
A Two-layer MLP the prediction head. Given the extracted feature vector with dimension d, the first layer transform the input from d to 128. The second layer transform the output of the first layer into L dinn where L is the number of layers of the source LLM being pruned and dæn is the intermediate dimension of the source LLM. Then we reshape the output vector into a I-by-dar, matrix. Each row of the matrix will be processed by the SOFTTOPK (Lei et al., 2023) operator independently to get the FFN mask for the corresponding layer.
A.2. Training of the Structured Pruning Baseline
Our pruning baseline, PRUNING+DISTILL, first prunes the given LLM into a dense 3B model then perform continuous pre-training through knowledge distillation. For knowledge distillation. we apply KL divergence between the output distributions of the student and the teacher model for each output token. The teacher distribution only keeps the highest- scoring tokens, similar to Minitron (Muralidharan al., 2024). We minimize combined loss of the standard next-token prediction and KL divergence loss.
A.3. Interpretability of Parameter Selection
We provide more details of the visualization of the parameter selection pattern in Figure 3. First, we sample 128 inputs per dataset (MMLU, GSMSK, CodeAlpaca-20K,, GPTTeacher). Inputs the sparsity predictor are formatted with in-context examples (MMLU: 5-shot, GSM8K: 8-shot) raw prompts (CodeAlpaca, GPTTeacher). After that, each input is passed through the sparsity predictor, which selects fixed number of FFN units (a binary mask) at each layer, producing 128 sub-networks per dataset. To calculate the overlap, We compare every pair of sub-networks within the 128 examples. For each pair, we compute the fraction of selected FFN units they share. The final overlap rate is the average of these pairwise overlaps.
A.4. Ablation Study on the Continued Pre-training
In this section, we conduct ablation study on the impact the continued pre- pre-training. We apply our method and activate 3B parameters of a DENSE-6B LLM. We train two variants models: one with the SFT phase ony and one with both continued pre-training and SFT phases. The experiment results are shown in Table 5. We observe consistent and notable
14

---

InstroitionnFollowinng Pruning for Large Language Models
Table Ablation study on the impact of the continued pre-training. We apply IFPRUNING dense L.M with 6B parameters and activate 3B parameters for each input.
HumanHval
MЕPР
MultiPI.-H
GSM8K
MATH
MMLJU
SFT Only
25.3
24.4
15.3
50.9
13.5
55.2
Continued pre- pre-training + SFT
31.9
35.3
22.4
61.3
20.1
59.3
gains across all benchmarks, demonstrating the effectiveness of the continued pre-training phase.
A.5. Inference Latency Comparison with MoEE
In this section, compare the latency of our method with the open-source model, parameters in total and activates 2.7B parameters per token. For our method, we prune LLaMA-3-8B to 3B parameters. The experiment configurations are exactly the same as the exper- iment in Table 3. Specifically, we evaluate the latency on a single NVIDIA RTX A6000 GPU and report toneto--irst-token (TTFT) and decoding time with input length 4k, generation length 100, and sample 4 responses for each query. The experiment results are shown in Table 4.
Figure 4. Inference latency evaluation. The source models are reduced to 3B by pruning the FFN layers.
It has 14.3B
Param. selection
Param. loading
Model
Prefill
TIFT
Decode
DENSE-9B
0.846
0.846
7.16
MOE-A2.7B
0.621
0.621
28.43
DENSE-3B
0.356
0.356
IFPRNNNGG
0.070
0.043
0.358
0.471
5.58
We can see the dense baseline and our method have significantly better latency and throughput than MoE. The main reason that MoE models are not efficient for on-device inference when generating responses given a single query, where the decoding is bottlenecked by weight loading. Since MoE requires reading many expert weights (e.g.. 2 for each token), the cost of MoE is multiple times higher than dense model and our method. This further demonstrates that our method is more suited for on-device scenarios.
A.6. Subsets included in MMLU domains
We list the subsets in each MMLU domain for the experiment in Table 2.
MMLU-physics: astronomy, college physics, conceptual physics, and high school physics.
MMLU-math: abstract algebra, college mathematics, elementary mathematic, high school mathematics, and high school statistics.
MMLU-history:
high school curopean history, high school us history, high school world history, and prehistory.
MMLU-health:
anatomy, clinical knowledge, college medicine, human aging, medical genetics nutrition, professional medicine, and virology.
MMLU-business: business ethics, management, and marketing.
MMLU-economics: econometrics, high school macroeconomics, and high school microeconomics.
A.7. Detailed Per-task Prompts
In this section, we list the full prompts for the task-specific pruning in Section 4.3 in Table 6.
A.8. License of Datasets and Use of Generative AI
We include a diverse set of datasets for evaluation, and their licenses are detailed below. The IFEval dataset is released under the Apache License 2.0. AlpacaEval 2.0 and Arena Hard are also under the Apache-2.0 License. HvmanEval, GSM8K, MATH, HellaSwag, WinoGrande, and MM--Evauation-Haare under the Creative Commons Attribution 4.0 license, while MultiPL-E's is under the BSD 3-Clause License with Machine Learning Restriction. The ARC dataset is provided under the Creative Commons Attribution Share Alike 4.0 license. and SeiQ is under the Creative Commons Attribution Non- Non-Commercial 3.0 license. PiQA is licensed under the Academic Free
are under the MIT License, The mbpp dataset is distributed
15

---

Instraction-Followinn Pruning for Large Language Models
Table Task prompt for task-specific pruning experiments.
You are a Math expert. You will be given math problem in domains such as algebra, probability, geometry and number theory. Reason and give final answer to the problem. Your response should end with "The answer is [answer]" where [answer] is the response the problem.
MATH
You expert physics. You will be given multiple choice questions in subjects such astronomy, conceptual physics, and college physics. Select the correct answer each question.
MMLUU-Physies
You expert history. You will be given multiple choice questions in subjects such european history, history and prehistory. Select the correct answer to each question.
MMI.U-History
You expert economics. You will be given multiple choice questions in subjects such econometrics, macrocconomics and microcconomics. Select the correct answer to each question.
MMLU--Economics
You skilled translator who specializes in English German translations. Your task is to accurately translate the provided English text into German while preserving meaning and context.
Translation (EN-DE)
Translation (others)
Same above. Replace the language names with the language pair being tested.
You expert Swift programmer. You will be given Swift function definition in documentation comments after Write the code to complete the function.
Here is an example input: swift
Multiple-E Swift
I|I Write swift function to count inversions in arra..
func get Inv Count(arr: [Int]) Int
You expert Python programmer. You will be given Python function definition and some test examples in triple quotes """ Write the code which should pass the tests.
Here an example input:
python
def prccaest__cmmoon covooo_divvviooor: int, b: int) int:
HumanEval-Pythan
Return greatest common divisor of two integers a and b greatest.common _ivinoor.d 5)
>>> greatest.common common divison(25, 15)
5
You expert Python programmer. You will be given Python function definition and some test examples in triple quotes """ Write the code which should pass the tests.
Here an example input:
python
Mbpp
Write function to find the similar elements from the given two tuple lists. assert similar elements((3, 5, 6).(5, 7, 4. 10)) = (4
License 3.0. Additionally, MMLU is under the MIT License, and MMAU is distributed under the Creative Commons
Attribution 4.0 license.
The usage all datasets and packages in this work aligns with their intended purposes, specifically the evaluation of LLMs.
We utilized GPT-4 to assist in checking and refining grammar and clarity across all sections. The core ideas, analyses, and textual composition remain entirely the work of the authors.
16


Reviews:
[{"review_id": "EujkXVNAxd", "reviewer": "Reviewer_Lv3b", "summary": "The paper proposes a dynamic pruning method in which a router determines the pruning strategy of the FFN layers in an LLM model using the input instruction. The sparse mask predictor and LLM weights are jointly trained using instruction-following data and the pre-training corpus. Experiments on different target benchmarks demonstrate that the proposed pruning method can generally outperform the dense baseline of similar size.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Lv3b for the support and the valuable suggestions for our paper. We will address the writing feedback, such as adding statistics of the dataset and discussing additional related work, in the next version. Please see our response to the questions and/or major comments below.\n\n\n\n**Q1: Comparison with static pruning, and latency numbers.**\n\nWe agree with the reviewer that static pruning enjoys better batch parallelism and lower memory usage compared to dynamic pruning. We discussed the downside of our work in Section 5 and will improve the clarification in our next version. Thank you!\n\nAs far as inference speed is concerned, we would like to clarify that our method is designed for on-device models (e.g. on smartphone / laptop / desktop), where the inference typically samples a few responses given the **same** user query (or the same task). In this case, the same activated parameters are selected and cached as a dense model, therefore achieving the same speedup as the static pruning and dense baseline.\n\nTo illustrate the real inference speedup, we test the inference latency by pruning the open-sourced **LLaMA-3.1-8B-Instruct** model to 3B. Although the tests are done on GPUs, we used batch size 1 and 4 generations per query, reflecting on-device usage. We report the **time-to-first-token (TTFT) and the decoding time**, both measured in seconds.\n\nFor dense models (8B and 3B), the TTFT consists of pre-filling only. For our method, we break down TTFT into its components:\n\n- Sub-network selection (via the sparsity predictor)\n- Parameter loading (load the selected parameters and cache the sub-network as a dense model)\n- Pre-filling using the 3B sub-network\n\n\n||GPU|Model|Sub-network selection|Parameter loading|Pre-filling|TTFT|Decoding Time|\n|-|-|-|:-:|:-:|:-:|:-:|:-:|\n|Input length: 4k|A6000|Llama-8b|-|-|0.702|0.702|5.47|\n|||Llama-3b|-|-|0.317|0.317|3.52|\n||| Ours 8b->3b |0.070|0.016|0.315|0.402|3.53|\n|| RTX3090 |Llama-8b|-|-|0.947|0.947|5.48|\n|||Llama-3b|-|-|0.396| 0.396 |3.18|\n||| Ours 8b->3b |0.088|0.013|0.396|0.498|3.21|\n|||||||||\n|Input length: 2k|A6000|Llama-8b|-|-|0.336| 0.336|4.11|\n|||Llama-3b|-|-|0.155|0.155|3.20|\n||| Ours 8b->3b|0.037|0.016|0.155|0.208 |3.25|\n||RTX3090|Llama-8b|-|-|0.467|0.467|3.76|\n||| Llama-3b|- |-|0.203|0.203|2.70 |\n||| Ours 8b->3b |0.045| 0.013|0.202|0.260 |2.75|\n\n\n\n\nKey takeaways:\n\n- TTFT decreased by **up to 57%**, decoding time decreased by **up to 41%. In total, we achieve 1.8x speedup compared to Llama-8b.**\n- Overhead from dynamic pruning & parameter caching is **negligible (~0.05s, ~2% of the total generation time)**\n- Despite dynamic masking, runtime of IFPruning is **on par with static pruning, while offering input-specific adaptivity and superior accuracy.**\n\nWe will include this analysis in the final version of our paper. \n\n\n\n**Q2: Choice of SoftTopK over HardConcrete.**\n\nA2: We appreciate the suggestion. Both SoftTopK and HardConcrete are mask generation operators introduced in previous work, and not a contribution of our work. We chose SoftTopK for simplicity for two reasons:\n\n1. Both methods achieve similar task performance.\n2. SoftTopK does not require another auxiliary loss, whereas HardConcrete needs some tuning (e.g. on the auxiliary loss weight and its learning rate).\n3. SoftTopk seems more stable in our preliminary study.\n\n**Q3: More details on SFT datasets**\n\nWe will expand the final version with a detailed breakdown of the SFT datasets, including data sources, instruction formats, and size per domain.\n\n\n**Q4: Missing related work.**\n\nThank you for highlighting these valuable references—we will include them in the final revision to strengthen the related work section.", "claims_and_evidence": "Yes, the main claim of the paper is that using dynamic, input-dependent sparsity can be beneficial for the pruned model's performance rather than using the same sparsity pattern for all the inputs. The experimental results support such a hypothesis.", "methods_and_evaluation_criteria": "The components of the proposed method have been introduced in previous work so the idea does not have novel elements, but it combines them in an effective manner. I think the proposed method is sound and simple yet effective in practice, so I do not complain about novelty.\n\nFor the evaluation, I think the paper can be improved in the following aspect\n\n* Although it has got a convention in Mixture of Experts literature to compare different models only based on active parameters, I think doing so does not paint a complete picture of the model's real performance in terms of inference latency. Depending on the network's topology, two 3B activated parameters models can have significant different latency values in practice on GPU/TPU. Therefore, the paper should provide at least the inference latency of the pruned model by IFPruning and the dense baselines.", "theoretical_claims": "The paper has not theoretical claims.", "experimental_designs_or_analyses": "As the main contribution of the paper is in empirical results rather than methodological development, I think the paper should be improved in the following aspects:\n\n* In Sec. 4.1, the paper only mentions that it uses an internal SFT dataset for training. I understand that they may not be able to release this dataset but they at least should provide some statistics and general structure of the dataset.\n\n* Also, the results that the higher the base model, the better the performance of the pruned model is not much surprising. There has been empirical and theoretical papers [1, 2] in the literature that indicate that overparameterization helps with training a better base model and improves the quality of the pruned model. The paper should indicate this connection in their discussion. \n\n* In Line 167, the paper admits that one can use HardConcrete trick to do pruning, yet it does not indicate why it choses not to do so. It would be nice to have a comparison between the current approach and HardConcrete as it is widely used in practical scenarios. However, I understand that doing experiments in the rebuttal period can be challenging, and I don't ask for new experiments. \n\n\n\n[1] Learning and Generalization in Overparameterized Neural Networks, Allen-Zhu et al., 2019.\n\n[2] Stronger generalization bounds for deep nets via a compression approach, Arora et al., 2019.", "relation_to_broader_scientific_literature": "The paper shows advantages compared to previous static pruning baselines in terms of performance as it makes the pruning strategy dependent on the input. However, I believe that the paper should indicate the following downsides compared to static pruning:\n\n* Static pruning enables batch-parallelism on GPUs while dynamic pruning cannot benefit from it as each sample in the batch may have a different pruning strategy. \n\n* Static pruning will enable lower memory usage for saving the model on disk and also consumes less GPU memory. In contrast, dynamic pruning cannot do so in practice.\n\n* Static pruning can achieve real inference speed up on GPUs/TPUs. It is harder to achieve inference latency reduction with dynamic pruning.", "essential_references_not_discussed": "Please check the \"Experimental Designs Or Analyses\" section about the missing references.", "other_strengths_and_weaknesses": "I cannot think of any other points other than the ones mentioned above.", "other_comments_or_suggestions": "I suggest that the authors provide inference latency of their models and the baselines. It is fine to me that the method does not beat the baselines (specially static pruning ones), but doing so will make the paper more useful for the readers and practitioners. I would be happy to raise my score if the authors provide this analysis.", "questions_for_authors": "Please check my comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1iC8WOjVQK", "reviewer": "Reviewer_epS3", "summary": "The paper proposes \"Instruction-Following Pruning\" (IFPRUNING), a novel approach to dynamic structured pruning of large language models (LLMs). Unlike traditional static pruning methods that determine a fixed pruning mask for a model, this approach generates input-dependent pruning masks that adapt based on the user's instruction. The method introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task, focusing primarily on pruning feed-forward neural network layers.\nThe architecture consists of two main components: (1) a sparsity predictor that extracts features from user prompts and generates masks, and (2) a dense LLM that gets pruned dynamically. The approach uses the SoftTopK algorithm to generate differentiable masks that activate only the most relevant parameters for specific inputs. The authors demonstrate that their method, which activates 3B parameters from larger models (6B, 9B, and 12B), outperforms dense 3B models and shows comparable performance to larger dense models in various tasks including math, coding, and general instruction-following benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer epS3 for the support and the valuable feedback. Please see our response below.\n\n**Q1: What is the reasoning behind designing a method that requires task-specific pruning for general-purpose LLMs?**\n\nWe totally agree with the reviewer that having “general-purpose LLMs and adaptability across tasks” is essential. Indeed, our method aims to **improve the general applicaiblity of smaller LMs**. Please allow us to clarify a few things:\n\n1. Given a user instruction (can be a **general-purpose question** such as a coding problem, a travel suggestion, and a knowledge-seeking question etc.), our method dynamically prunes the model and uses the most suited parameters for inference.\n2. As the pruning mask is generated by “reading” the input instruction, our model can handle questions/tasks that are **not seen during training.** See for example our evaluation results on AlpacaEval and Arena-Hard.\n\nIn other words, our approach offers the following advantages:\n\n1. Improve inference efficiency over large dense model (see Q3);\n2. Improve the model quality over static pruning;\n3. Pruning based on natural language description enables zero-shot generalization to unseen tasks and instructions, making the model still a general-purpose LLM.\n\nIn contrast, naive task-specific fine-tuning can have problems: no model available for unseen tasks during inference; memory/storage overhead for deploying many task-specific models; additional cost to collect training data for each task, etc.\n\n\n\n**Q2: Included baseline is not strong enough. Need to compare with baselines including LLM-Pruner or SliceGPT.**\n\nWe would like to clarify that our `pruning+distill` baseline reflects the **SOTA practice in large-scale model development**. It first prunes the model then continued pre-trains on trillions of tokens. This method combined with logit distillation is consistent with recent model developments like LLaMA 3.2, Nvidia Minitron, and Gemma 3.\n\nTo show the effectiveness of our method and our baseline, the table below summarizes the (relative) performance drop compared to the source model with different pruning methods:\n\n|||Sparsity|ARC-C|ARC-E|PiQA|HellaSwag|Winogrande|Average|Performance drop     |\n| -- | -- | :-: | - | - | - | - | - | - | - |\n|LLM-Pruner| Source model: 7B || 47.6| 72.8| 79.8 | 76.1| 70.1| 69.3||\n|| LLM-Pruner|20%|37.9| 63.4|76.4| 68.1| 65.1| 62.2| 7.1% (relative 10%) |\n| SliceGPT/ShortGPT | Source model: 7B | 46.3| 74.6| 79.1 | 75.9| 69.1| 69.0||\n|| SliceGPT|30%| 34.1|50.7| 67.4 | 55.7| 63.2| 54.2| 14.8% (relative 21%) |\n|| ShortGPT|31%| 40.9|56.6| 67.8 | 62.2 | 64.4| 58.4| 10.6% (relative 15%) |\n| IFPruning| Source model: 9B || 53.9  | 83.4  | 79.4 | 57.7| 74.3| 69.7||\n|| Our baseline|66%|46.2|79.9| 77.3 | 53.0| 69.1| 65.1    | 4.8% (relative 7%)|\n|| IFPruning|66%|50.4|81.4| 78.0 | 55.5| 68.4| 66.7| 3.0% (relative 4%)|\n\nOur baseline and IFPruning achieves **much higher sparsity** and **smaller accuracy degradation**, validating the effectiveness of our method.\n\n\n\n**Q3: Overhead of generating masks for each new input**\n\nWe evaluate the latency with LLaMA-3.1 8B as the source model and prune it to 3B. Due to space limit, please also see our response to Q1 by reviewer Lv3b. In brief, mask generation adds only **~0.1s overhead**. Specifically, with input length 4000, output length 100, and response sample size 4, we show:\n\n- **Time-to-first token (TTFT) decreased by up to 57%**\n- **Decoding time decreased by up to 41%**\n- **Comparable latency to static 3B models**\n\n|GPU| Model| Sub-network selection | Parameter loading | Pre-filling | TTFT  | Decoding Time |\n|--| --- | :--: | :--: | :--: | :---: | :--: |\n|A6000|Llama-8b|-|-|0.702| 0.702 |5.47|\n|| Llama-3b    |- |-|0.317| 0.317 |3.52|\n|| Ours 8b->3b |0.070|0.016|    0.315    | 0.402 |3.53|\n| RTX3090 | Llama-8b|-|-|0.947    | 0.947 |5.48|\n|| Llama-3b    |-|-|0.396 | 0.396 |3.18|\n|| Ours 8b->3b |0.088|0.013|0.396| 0.498 |3.21|\n\n\nThe overhead from sub-network selection (mask generation) & parameter loading is **negligible (~0.1s, ~2% of the total generation time).**\n\n**Q4: Concerns about JAX/AXLearn and framework portability**\n\nOur core method is **framework-agnostic** and can be easily implemented in **PyTorch**, as shown in our latency experiments using **LLaMA-3.1-8B-Instruct** and PyTorch. We will make our best effort to release the full implementation on top of an open-source model.\n\n**Q5: Sparsity predictor architecture**\n\nThe predictor consists of:\n- An LLM with 302M parameter as the feature extractor (last hidden state of the final input token)\n- Two-layer MLP:\n  - Linear(hidden_dim → 128)\n  - Linear(128 → num_layers × ffn_dim)\n\nTraining is end-to-end with the masked LLM using standard language modeling loss.\n\n**Q6: Missing citations, discussions with other dynamic pruning papers, and figure issues**\n\nThank you for your suggestion. We promise that we will address your comments in the final revision of our paper.", "claims_and_evidence": "The paper's central claim that dynamic pruning based on task descriptions leads to better performance than static pruning is supported by experimental evidence, but with limitations:\n1. The authors show that their 3B activated models outperform dense 3B models on various benchmarks, which supports their main claim.\n2. The claim that their approach avoids parameter reloading costs during decoding (compared to other dynamic methods) is reasonable but lacks direct empirical validation in terms of efficiency measurements.\n3. The claim about the interpretability of parameter selection is supported by the analysis in Section 4.2, showing that inputs requiring similar skills yield similar pruning patterns.\n4. However, the claim that their method rivals the performance of 9B models is only partially supported - while there are improvements over a dense 3B model, the gap to the 9B model remains noticeable in most benchmarks.", "methods_and_evaluation_criteria": "The proposed method makes sense for the problem of efficient LLM inference, but has several limitations:\n1. The sparsity predictor design is reasonable, using a smaller model to extract features from prompts and predict masking scores.\n2. The evaluation benchmarks are comprehensive, covering instruction-following, coding, math, NLP understanding, and tool use tasks.\n3. However, the method's practicality is questionable - if the goal is to have specialized models for specific tasks, traditional pruning followed by task-specific fine-tuning might be more straightforward and efficient.\n4. The evaluation compares against limited baselines - a dense 3B model and a pruned+distilled 3B model - but misses comparisons with competitive structured pruning methods like LLM-Pruner and SliceGPT, or other contextual sparsity approaches.", "theoretical_claims": "The paper makes limited theoretical claims and provides no formal proofs. The SoftTopK algorithm for generating differentiable masks is cited from previous work without detailed theoretical analysis of its properties in this context. The paper would benefit from theoretical analysis of why task-specific pruning works better than static pruning and under what conditions this advantage would hold.", "experimental_designs_or_analyses": "The experimental design has several issues:\n1. The use of AXLearn framework and JAX raises questions about whether the results would generalize to more commonly used frameworks.\n2. The experiments do not use widely recognized open-source models like LLaMA 3.1 or Qwen 2.5, limiting the practical applicability of the findings.\n3. The baselines are limited - there are no comparisons with state-of-the-art structured pruning methods like LLM-Pruner or SliceGPT.\n4. While the authors present per-task pruning analysis, they don't adequately address the practical overhead of generating task-specific masks for each new input, which could negate the efficiency gains from pruning.\n5. The paper lacks ablation studies on key components such as the size of the sparsity predictor or different mask generation algorithms.", "relation_to_broader_scientific_literature": "The paper adequately positions itself relative to three areas of related work:\n1. Model pruning: The authors acknowledge prior work on structured pruning techniques for LLMs, including LLM-PRUNER, SLICEGPT, and SHORTGPT.\n2. Contextual sparsity: The paper discusses how their approach differs from other contextual sparsity methods that require pruning at each decoding step.\n3. Mixture-of-experts: The authors compare their approach to MoE models, noting that while MoE activates different parameters per token, their method fixes parameters based on the task description.", "essential_references_not_discussed": "Dynamic pruning is not a novel topic. Actually, there are many previous works in this area, especially in computer vision. Some of them are listed here:\n1. Elkerdawy, S., Elhoushi, M., Zhang, H. and Ray, N., 2022. Fire together wire together: A dynamic pruning approach with self-supervised mask prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12454-12463).\n2. Gao, S., Zhang, Y., Huang, F. and Huang, H., 2024. BilevelPruning: unified dynamic and static channel pruning for convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 16090-16100).\n3. Le, Q., Diao, E., Wang, Z., Wang, X., Ding, J., Yang, L. and Anwar, A., 2025. Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing. arXiv preprint arXiv:2502.15618.", "other_strengths_and_weaknesses": "Strengths:\n1. The idea of task-specific pruning is conceptually interesting and represents a novel compromise between static pruning and token-level dynamic approaches.\n2. The analysis of activation patterns across different domains provides interesting insights into how the model specializes for different tasks.\n3. The performance improvements over a dense 3B model across different tasks are notable, demonstrating the potential of the approach.\n4. The authors explore both per-input and per-task pruning scenarios, showing the flexibility of their method.\n\nWeaknesses:\n1. The core innovation is limited and has conceptual flaws. The main differentiation from other contextual sparsity work is that pruning happens at the task level rather than per token, but this approach still requires task-specific masks to be generated for most inputs in practical scenarios.\n2. The method section lacks technical details and theoretical support. The description of the training process and objective functions is cursory.\n3. The experimental section has significant limitations, with missing comparisons to competitive baselines and experiments predominantly conducted on non-standard frameworks and models.\n4. The practical utility is questionable - for general-purpose LLMs, adaptability across tasks is essential. If task specialization is the goal, traditional pruning plus fine-tuning might be more straightforward.\n5. The presentation quality is lacking, with poor figure quality (e.g., Figure 3's small font size) and insufficient technical details in key sections.", "other_comments_or_suggestions": "1. The paper would benefit from clearer explanations of the technical details, especially regarding the sparsity predictor architecture and training.\n2. Implementation details about how the masks are efficiently computed during inference would strengthen the paper.\n3. It would be valuable to explicitly measure and report the computational overhead of the sparsity predictor.\n4. The authors should improve figure quality, particularly in Figure 3, where the text is difficult to read.\n5. For real-world applications, it would be helpful to discuss how the approach handles out-of-distribution task descriptions.", "questions_for_authors": "1. The method requires generating a new mask for each input (or task) during inference. Have you measured the computational overhead of this process compared to static pruning? This information would help clarify whether the performance gains outweigh the additional inference complexity.\n2. Why did you choose not to compare against other structured pruning methods like LLM-Pruner and SliceGPT, which would provide more meaningful baselines than just dense models and a simple distillation approach?\n3. How would your approach perform on widely-used open-source models like LLaMA 3.1 or Qwen 2.5? Results on these models would strengthen the practical applicability of your method.\n4. What is the reasoning behind designing a method that requires task-specific pruning for general-purpose LLMs? Most applications require general models that can handle diverse tasks without generating new masks for each input.\n5. For the per-task pruning scenario, how did you handle variations in task descriptions that might refer to the same underlying task? Is there a way to automatically cluster similar tasks to reuse masks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MguPtY0lwc", "reviewer": "Reviewer_Kh2x", "summary": "The paper introduces Instruction-Following Pruning (IFPruning), a dynamic structured pruning method for large language models (LLMs). Instead of using a fixed sparsity mask, IFPruning employs a sparse mask predictor that selects the most relevant model weights (specifically, rows/columns of transformer feed-forward layers) on a per-instruction basis. The model is thus pruned on-the-fly per query, using only the parameters most relevant to that instruction. The authors jointly train the mask predictor and the LLM on instruction-following data utilizing both pre-training and additional fine-tuning data. Empirically, a pruned 3B-LLM using IFPruning (activating ~3B parameters out of a larger 9B or 12B model) achieves significantly better performance on domain-specific tasks like math and coding than a static 3B dense model, even rivaling a full 9B model on those tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer Kh2x for the support and the valuable suggestions. Please see our response below.\n\n**Q1: Why does continued pre-training help and what if we remove it?**\n\nThank you for the question. We first explain our motivation followed by the ablation study on continued pre-training.\n\n- Intuition of continued pre-training. In continued pretraining, we split long text into chunks; the model predicts the next chunk after selecting a sparse sub-network from the current one. \nConsider the extreme case where we split a pretraining text into just **two chunks**. the first acts as a **prompt** to select parameters, and the second as the **target** for prediction—closely mirroring the instruction–response format of SFT. By training the model in this way across millions of natural examples, the sparsity predictor **learns to select the right sub-networks given different input contexts**.\n\n- **Empirical Results**: We ran an ablation study with and without continued pre-training (6B → 3B, train for 400B tokens)\n\n||HumanEval|MBPP|MultiPL-E|GSM8K|MATH|MMLU|\n|-|-|-|-|-|-|-|\n|No continued pre-training|25.3|24.4|15.3|50.9|13.5|55.2|\n|Continued pre-training|31.9|35.3|22.4|61.3|20.1|59.3|\n\nWe observe consistent and notable gains across all benchmarks. We will add these results and clarify this design choice in the final version.\n\n\n\n**Q2: Configuration details for the mask predictor network and the LLMs**\n\nThe predictor consists of:\n- An LLM with 302M parameter as the feature extractor (last hidden state of the final input token)\n- Two-layer MLP:\n  - Linear(hidden_dim → 128)\n  - Linear(128 → num_layers × ffn_dim)\n\nRegarding the LLMs used in this work: they follow standard LLM design, such as grouped-query attention and RMSNorm, with no custom components—similar to LLaMA. We will include these details in the final version.\n\n\n\n**Q3: Why no comparison with ShearedLLaMA or Minitron? What is the model distillation objective?**\n\nThank you for your helpful question. We clarify both aspects:\n- **On comparison**: We did not include ShearedLLaMA or Minitron due to differences in training data and model setup. Instead, we implemented a fairer pruning + distillation baseline using similar techniques in ShearedLLaMA and Minitron: structured pruning using learned masks, and logit distillation in continued pre-training. As shown below, our baseline outperforms the results in ShearedLLaMA.\n\n||ARC-C|ARC-E|PiQA|Winogrande|MMLU|Avg.|\n|-|-|-|-|-|-|-|\n|ShearedLLaMA 2.7B|41.2|67.0|75.8|64.2|26.4|54.9|\n|Our pruning baseline (3B)|46.2|79.9|77.3|69.1|62.8|67.6|\n|IFPRUNING 9B→3B|50.4|81.4|78.0|68.4|65.5|68.7|\n\nMinitron results are only partially available, but our pruning baseline already outperforms its reported MMLU score. We will add these comparisons as a reference..\n\n- **Distillation objective**: We apply KL divergence between the output distributions of the student and the teacher model for each output token. The teacher distribution only keeps the highest-scoring tokens, similar to Minitron. We minimize a combined loss of the standard next-token prediction and KL divergence loss.\n\n\n\n**Q4: Comparison with MoE.**\n\nThank you for the question. We did not include MoE models due to a fundamental difference in inference scenarios. We will improve the clarity of our paper:\n\n- Our method is designed for edge devices (e.g., smartphones) where memory and compute resources are limited, **and the inference batch size is small.**\n- While MoE models are great for server-side large batch size inference, they are not efficient for **on-device inference when generating responses given a single query**. \n- In this case, decoding is bottlenecked by weight loading. Since MoE requires reading many expert weights (e.g., 1-2 for each token), the cost of MoE is multiple times higher than a dense model and our method.\n\nTo illustrate, we compare our method with the open-source model, Qwen1.5-MoE-A2.7B. It activates 2.7B parameters per token. For our method, we prune LLaMA-3-8B to 3B parameters.\n\nWe report time-to-first-token (TTFT) and decoding time with input length = 4k, generation length = 100, and sample 4 responses for each query. For our method, we also report the latency for sub-network selection and loading parameters for selected sub-network.\n\n|GPU|Model|Sub-network selection|Parameter loading|Pre-filling|TTFT|Decoding Time|\n|-|-|:-:|:-:|:-:|:-:|:-:|\n|A6000|Llama-8b|||0.702|0.702|5.47|\n||Qwen-MoE|||0.621|0.621|28.43|\n||Llam-3b|||0.317|0.317|3.52|\n||Ours 8b->3b|0.070|0.016|0.315|0.402|3.53|\n|RTX3090|Llama-8b|||0.947|0.947|5.48|\n||Qwen-MoE|||`OOM`| `OOM` |`OOM`|\n||Llama-3b|||0.396|0.396|3.18|\n||Ours 8b->3b|0.088|0.013|0.396|0.498|3.21|\n\nWe can see the dense baseline and our method have significantly better latency and throughput than MoE.\n\nFinally, we agree with the reviewer's point that MoE and our method share the same spirit by dynamically activating parameters. In this regard, our model is a sparse model designed for on-device scenarios.", "claims_and_evidence": "The key claim is that input-dependent dynamic pruning can exceed static dense models of the same activated size. This is well supported by experiments: with an equal parameter budget, IFPruning outperforms a dense counterpart on multiple benchmarks. Notably, the 3B dynamic model nearly matches a dense 9B model’s accuracy on coding and math benchmarks (See Table 1). The authors also compare against a static structured pruning baseline (Pruning+Distill, which prunes to 3B and distills from a larger model) – IFPruning consistently beats the baseline. These results substantiate the claim that selecting weights per input yields a more effective submodel than the naive fixed mask. \\\nHowever, MoE-structured LLMs also share the same spirit: activating fewer parameters within the large parent model. The authors should compare with MoEs, which have fixed masks, in order to claim that dynamic pruning is effective.", "methods_and_evaluation_criteria": "Overall, the method is well-designed for the stated goal of task-adaptive efficiency. The mask predictor is a lightweight network that adds minimal overhead (close to linear probing), and it produces differentiable masks via a SoftTopK operator to choose top neurons per layer​. Pruning is done at the structured level, which keeps the model hardware-friendly. The evaluation spans a wide range of benchmarks: instruction following, reasoning, math, coding, tool use, and general NLP tasks.  \\\nThat said, the authors should provide a detailed configuration for the mask predictor network as pre-trained LLMs are listed in the Appendix. Also, I cannot clearly understand the purpose of continued pre-training. The paper claims it provides a good initialization, but why does utilizing similar chunks from the same context help stabilize training of the mask predictor? I would like to see the experimental results without pre-training.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "I have several concerns about the experimental setup. \\\nAs there is no codebase, it is very difficult to specify which models were used or trained, and whether the architecture is based on LLaMA or uses specific attention and normalization modules. Further, there is also no experiments regarding pruning other components, such as attention heads. Authors say it is natural to extend yet I find it should be dependent on the model choice. Moreover, the paper does not provide a clear comparison to baselines: although the authors mention distilled models following Sheared Llama and Minitron, they neither compare against these baselines nor specify which distillation objectives are employed.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "Well covered.", "other_strengths_and_weaknesses": "N/A.", "other_comments_or_suggestions": "N/A.\n\n--------------------After Rebuttal--------------------\n\nI have raised my score from 2 to 3, and lean towards acceptance, only if the authors faithfully include new experimental results in the final manuscript.", "questions_for_authors": "Regarding input-dependent pruning (opposed to the task-dependent pruning), I would like to see the analysis regarding the inference time since it needs two forward passes due to the mask prediction.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9yo6AhBYCe", "reviewer": "Reviewer_bo9C", "summary": "This article tackles the problem of letting LLMs select the most suited parameters for each prompted task and proposes a novel instruction-following pruning paradigm called IFPruning. Specifically, IFPruning uses a sparse mask predictor to predict a input-dependent mask for each context input. To train the predictor, IFPruning optimizes it together with the LLM to enable the effectiveness of the novel paradigm. Experiment results on a series of tasks demonstrate the effectiveness of IFPruning.\n\n## update after rebuttal\nI appreciate the comprehensive response that the authors managed to propose during the rebuttal phase. I decided to maintain my overall rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer bo9C for the support and the valuable suggestions. Below we address each point raised.\n\n**Q1: Clarification on actual model speedup.**\n\nSince inference speed is concerned, we would like to first clarify that our method is motivated and designed for on-device models (e.g. on smartphone / laptop / desktop), where the inference typically samples a few responses given the **same** user query (or the same task). In this case, the same activated parameters are selected and cached as a dense model, therefore achieving the same speedup as the static pruning and dense baseline. We discussed the limitations of our work and possible extensions to batch inference in Section 5. We will better clarify our limitations in the next version. Thank you!\n\nWe evaluate the speedups by pruning the open-sourced **LLaMA-3.1-8B-Instruct** model to 3B. Although the tests are done on GPUs, we used batch size 1 and 4 generations per query, reflecting on-device usage. We report the time-to-first-token (TTFT) and the decoding time, both measured in seconds.\n\nFor dense models (8B and 3B), the TTFT consists of pre-filling only. For our method, we break down TTFT into its components:\n\n- Sub-network selection (via the sparsity predictor)\n- Parameter loading (load the selected parameters and cache the sub-network as a dense model)\n- Pre-filling using the 3B sub-network\n\n||GPU|Model|Sub-network selection|Parameter loading|Pre-filling|TTFT|Decoding Time|\n|-|-|-|:-:|:-:|:-:|:-:|:-:|\n|Input length: 4k|A6000|Llama-8b|-|-|0.702|0.702|5.47|\n|||Llama-3b|-|-|0.317|0.317|3.52|\n||| Ours 8b->3b |0.070|0.016|0.315|0.402|3.53|\n|| RTX3090 |Llama-8b|-|-|0.947|0.947|5.48|\n|||Llama-3b|-|-|0.396| 0.396 |3.18|\n||| Ours 8b->3b |0.088|0.013|0.396|0.498|3.21|\n|||||||||\n|Input length: 2k|A6000|Llama-8b|-|-|0.336| 0.336|4.11|\n|||Llama-3b|-|-|0.155|0.155|3.20|\n||| Ours 8b->3b|0.037|0.016|0.155|0.208 |3.25|\n||RTX3090|Llama-8b|-|-|0.467|0.467|3.76|\n||| Llama-3b|- |-|0.203|0.203|2.70 |\n||| Ours 8b->3b |0.045| 0.013|0.202|0.260 |2.75|\n\nWe highlight the following observation:\n\n- **Practical inference efficiency Gains:** TTFT decreased by **up to 57%**, decoding time decreased by **up to 41%.** \n- **Minimal overhead from dynamic pruning & parameter caching:** Overhead from dynamic pruning & parameter caching is **negligible (~0.05s, ~2% of the total generation time)**\n- Despite dynamic masking, runtime of IFPruning is **on par with static pruning, while offering input-specific adaptivity and superior accuracy.**\n\nWe will include this analysis in the final version of our paper.\n\n\n\n**Q2: Sub-network overlap rate calculation**\n\nThank you for asking. We clarify the process in three steps:\n\n- **1: Data**: We sample 128 inputs per dataset (MMLU, GSM8K, CodeAlpaca-20K, GPTTeacher). Inputs to the sparsity predictor are formatted with in-context examples (MMLU: 5-shot, GSM8K: 8-shot) or raw prompts (CodeAlpaca, GPTTeacher).\n- **2: Sub-network Generation**: Each input is passed through the sparsity predictor, which selects a fixed number of FFN units (a binary mask) at each layer, producing 128 sub-networks per dataset.\n- **3. Overlap Calculation**: We compare every pair of sub-networks within the 128 examples. For each pair, we compute the fraction of selected FFN units they share. The final overlap rate is the **average of these pairwise overlaps**.\n\nWe will include these details in the final version of our paper.\n\n**Q3: More details on SFT datasets**\n\nWe will expand the final version with a detailed breakdown of the SFT datasets, including data sources, instruction formats, and size per domain.\n\n**Q4: Sparsity predictor architecture**\n\nThe predictor is a lightweight model with **302M parameters**, built on a pre-trained LM backbone. It consists of:\n\n- A feature extractor (last hidden state of the final input token)\n- Two-layer MLP:\n  - Linear(hidden_dim → 128)\n  - Linear(128 → num_layers × ffn_dim)\n\nThe output is a tensor of FFN importance scores per layer. The SoftTopK operator then converts the scores into structured binary masks.\n\n**Q5: Related work on SFT for efficient LLMs**\n\nThank you for the pointers. We will add these references in the final paper to strengthen the related work section.\n\n**Q6: Will the code be published upon acceptance?**\n\nWe will make our best effort to release the full implementation on top of an open-source model upon acceptance.", "claims_and_evidence": "The claims in this article are reasonable.", "methods_and_evaluation_criteria": "The design of methods is clear and satisfactory.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "1. How is the sub-network overlap rates calculated? I would like to see a complete pipeline for that calculation (i.e., data selection, forwarding, calculation metric).\n2. More details toward constructing the SFT dataset: I would like to see the proportion of each sub-domain and its corresponding source. \n3. Model speedup: The experiments have revealed the sparsity factor that IFPruning achieves. However, sparsity does not necessarily leads to inference speedup. I would like to see a comprehensive comparison on model speedup when IFPruning is applied.\n4. Design of the sparsity predictor: What is the source of applied predictors? What are their sizes?", "relation_to_broader_scientific_literature": "The findings of this study may inspire researchers to explore the issue of creating task-specific LLMs from the pre-trained base model, thereby advancing the real-world applications of LLMs.", "essential_references_not_discussed": "There is a recent-emerged field focusing on \"SFT for efficient LLMs\". Notable works of that field include [1,2]. Both [1,2] and IFPruning targets the goal of creating task-specific efficient LLMs, and should be compared and discussed in the article. There is no need for experimental results since those works are very recent.\n\n[1] TrimLLM: Progressive Layer Dropping for Domain-Specific LLMs\n\n[2] UniAttn: Reducing Inference Costs via Softmax Unification for Post-Training LLMs", "other_strengths_and_weaknesses": "Notable strengths of this article:\n\n1. Novelty: The idea of instruction-following pruning is novel and is worth investigating, since real-world applications cannot supply directly deploying large-size LLMs. Therefore, the motivation and methodology of IFPruning is promising for further research.\n2. Writing: This article is well written and structured.\n3. Strong results: IFPruning achieves significant improvements compared to existing methods.\n\nSee other fields for weaknesses.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Will the code be published upon acceptance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Instruction-Following Pruning, a dynamic pruning method where a router determines the pruning strategy for the FFN layers in an LLM model based on the input instruction. The proposed method is well-designed and technically sound. Experiments across various target benchmarks show that the pruning method generally outperforms the dense baseline of similar size. Overall, this is a solid contribution. To improve the paper, the authors should add comparisons with more state-of-the-art structured pruning methods, demonstrate the method's generalizability to commonly used frameworks, and provide additional discussions on related work, datasets, and static pruning techniques.
Decision: Accept (poster)