Multi-Objective One-Shot Pruning for Large Language Models
Authors: Weiyu Chen
Hansi Yang
Yunhao GOU
Han Shi
En-Liang Hu
Zhenguo Li
James Kwok
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Objective Optimization
Pruning
Large Language Models
Primary area: deep_learning

Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but require substantial computational resources, limiting their deployment in resource-constrained environments. While one-shot pruning methods can reduce model size without expensive retraining, they typically optimize for single objectives, ignoring LLMs' multi-faceted applications. We introduce Multi-Objective One-Shot Pruning (MOSP), which formulates LLM pruning as a multi-objective optimization problem. MOSP efficiently generates a Pareto set of pruned models representing different capability trade-offs, allowing users to select solutions aligned with their preferences. The proposed approach identifies share core support while enabling specialized support. Experiments across various LLMs and sparsity levels demonstrate MOSP's superior performance in navigating multi-objective trade-offs compared to baseline methods.

Multi-Objective One-Shot Pruning for Large Language Models
Weiyu Chen'
Hansi Yang'
Yunhao Gou 1.2
Han Shi Enliang Hu
Zhenguo Li James T. Kwok
The Hong Kong University of Science and Technology
Southern University of Science and Technology
'Huawei Noah's Ark Lab Yunnan Normal University
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but require substantial computational resources, limiting their de- ployment in resource-constrainee environments. While one-shot pruning meth- ods can reduce model size without expensive retraining, they typically optimize for single objectives, ignoring LLMs' multi-faceted applications. We introduce Multi-Ohjective One-Shot Pruning (MOSP), which formulates LLM pruning as multi-objective optimization problem. MOSP efficiently generates Pareto set pruned models representing different capability trade-offs, allowing users to select solutions aligned with their preferences. The proposed approach identifies share core support while enabling specialized support. Experiments across various LLMs and sparsity levels demonstrate MOSP's superior performance in navigating oujtii-bbecctie trade-offs compared to baseline methods.
1 Introduction
Large Language Models (LLMs) have demonstrated exceptional performance across many scenarios, such question answering, coding, and reasoning. However, their effectiveness typically requires massive model sizes and intensive computing resources, creating barriers for deployment in limited- resource settings. Network pruning 22 [7]. which aims remove redundant parameters, offers an important approach to create models with smaller computation cost while maintaining functionality.
For LLMs, traditional pruning methods [22 17 27 19 3 that involve iterative fine-tuning and/or extensive retraining are often prohibitively expensive due to the sheer scale of these models. Conse- quently, one-shot pruning techniques, which identify and remove weights without requiring retraining, have gained prominence [15]
37 30 These methods typically aim to minimize a layer-wise recon- struction error based calibration data from single, purppoe dataset.
While effective in reducing model size, this single-objective focus overlooks the multi-faceted nature of modern LLMs, which are increasingly evaluated across diverse tasks like text comprehension, mathematical reasoning, and code generation. Users prioritize these capabilities differently, requiring models that are not only sparse but also adaptable specific preferences. Current one-shot pruning methods lack mechanisms to address such multi-objective demands or tailor models to varying priorities. As discussed in Section 3.I. simple adaptations, like concatenating activation data or independently pruning and merging, offen result in suboptimal performance or limited flexibility.
To address this gap, we introduce Multi- Multi-Objective One-Shot Pruning (MOS)), novel framework designed to efficiently prune LLMS while considering multiple objectives simultaneously. Instead of producing single pruned model, MOSP generates Pareto set of solutions, where each solution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

represents a different trade-off among the objectives. This allows users to select pruned model that best aligns with their particular needs and preferences. MOSP achieves this through a multi-stage process: First, we identify a common core support of weights crucial across all tasks using Dual ADMM optimization. This involves formulating the problem as bilevel optimization with ADMM applied to both inner and outer levels, with proven convergence guarantees. Second, using the identified supports, we perform simplified ADMM for each task separately. This decoupling of shared knowledge preservation from task-specific optimization enables efficient on-the-fly generation of specialized sparse models based on user-defined preference vectors, allowing effective exploration of the Pareto front.
The main contributions of this paper are as follows:
We frame LLM pruning as multi- oujtii-b optimization problem, explicitly addressing diverse user preferences, which a novel perspective in LLM pruning.
We introduce MOSP, an efficient one-shot pruning approach that generates a Pareto set of pruned LLMs, enabling flexible trade-off' across objectives.
We provide proof of convergence for the proposed dual ADMM method, which non-trivial.
Extensive experiments on various LLMs and sparsity levels show that MOSP outperforms baselines in navigating multi-objective trade-off's and provides a superior set of pruned models.
2 Background
2.1 Network Pruning
Network pruning reduces model complexity by eliminating redundant parameters, offering benefits like smaller size and faster inference 22 17
19 3]. Traditional methods, often computational expensive and/or requiring extensive retraining, are not suitable for Large Language Models (LLMs) due to their immense scale [15] B7]. Consequently, one-shot pruning techniques, which avoid retraining, are gaining prominence for LLMs.
27
LLM pruning methods can be categorized by the granularity of weights removed. Structured pruning removes entire components like neurons or attention heads [28] 42 [ maintaining regularity for hardware acceleration but usually leads worse performance without retraining. Unstructured pruning removes individual weights [5 37 44 45), offering fine-grained control but creating irregular sparse patterns that can be harder accelerate. Semi-structured pruning, such as N:M sparsity (N out of M weights kept), offers a compromise, balancing performance with hardware efficiency [46] 20). Most unstructured LLM pruning methods are also applicable semi-structured pruning. This paper focuses on one-shot unstructured and semi-structured LLM pruning.
2.2 One-Shot Unstructured and Semi-structured LLM Pruning
One-shot unstructured and semi-structured pruning methods aim to efficiently create sparse LLMs without retraining. These methods often optimize layer-wise reconstruction error. Given an original dense weight matrix W E exd for layer (where and d are the input and output sizes, respectively), and calibration activations X E RR (from n samples each of length 1), the goal is to find a sparse surrogate W by solving:
min WeBexd
XW
XW F
s.t. | S k,
where Ill is the lo-norm, || F is the Frobenius norm, and k is the maximum number of non-zero elements.
Notable algorithms in this class include SparseGPT 15 which employs partial weight updates and adaptive mask selection to approximate the second-order information (Hessian) efficiently. Another approach, Wanda (37]. offers simplicity by pruning weights based on the product of their magnitudes and the norms of corresponding input activations. ALPS (30) is an optimization- based framework that directly solves the fo-constrained layer-wise reconstruction problem using the Alternating Direction Method of Multipliers (ADMM) 4 10]. It identifies the optimal weight support and values, which are then refined by Preconditioned Conjugate Gradient (PCG) steps. However, these methods typically

---

consider calibration data X from single, general-purpose purpose dataset. This overlooks the fact that LLMs are often evaluated across multiple criteria. Different users may prioritize these objectives differently. Current one-shot pruning techniques generally do not address this need for customization, lacking mechanisms to generate models tailored to specific user preferences.
Another line of work considers the allocation of sparsity across different layers 41 layer-wise sparsity distribution strategies can often be combined with most of the aforementioned pruning algorithms to further improve performance. These approaches are orthogonal to the proposed methods and the two can be combined in straightforward manner.
45
23 39. Such
2.3 Multi-Objective Optimization
Multi-objective optimization (MOO) 31 optimizes m objective functions simultaneously. Without loss of generality, minøge (fi fm (m))), where @ is the feasible decision space. A solution dominates b, denoted a < b, if vi E (1......) fa) I fibb) and 3j E {1....,m) m] feasible solution is Pareto-optimal when not dominated by any other feasible solution. The set of all Pareto-optimal decision vectors is called the Pareto set. The corresponding set of objective vectors, F (f(0) A Pareto-optimal]; is the Pareto front.
consider the minimization problem: mingee f(0)
(a < (b).. A
tradient-based MOO methods have been widely adopted in deep learning [8] They can be classified into three main categories: (i) Learning a single solution, with examples including MGDA 32 14
12). CAGrad [25], and Nash-MTL [33 (ii) Learning a finite Pareto set, with examples including
PMTL 24 EPO 229 MOO-SVGD 26 and GMOOAR 6 and (3) Learning an infinite set of
solutions, with examples including PHN (4) PaMaL [3] and LORPMAN (7].
All the aforementioned algorithms utilize gradient descent for optimization. However, the direct application of gradient descent is empirically ineffective in obtaining satisfactory solutions in the unstructured LLM pruning scenario, particularly when dealing with high sparsity ratios (30. Conse- quently, these algorithms are not directly amenable to modification for one-shot LLM pruning.
3 Multi-Objective LLM Pruning
As mentioned in Section 2.2 current LLM pruning methods typically rely on calibration data X from single, cenneral-puposss dataset. However, LLMs are evaluated across multiple objectives, such as performance on general text, mathematical reasoning. and code generation. Single-objective approaches fail to generate models that satisfy users with varying preferences across these objectives. We therefore propose to formulate LLM pruning as a multi-objective optimization (MOO) problem.
Formally, let (x() E Xe
Jj=1 be disjoint calibration sets, one for each objective j. Denote (WW) = | w x W The problem is then a MOO problem:
Ty=
minimize
f(W) = [J(WW,...... (W) s.t. || S h:
(2)
Relative importance of the objectives are represented by user preference À = (A1...., Am] T E RRm. where > j=1 Aラ 1.
3.1
Straight-Forward Multi-Objective Extension of ALPS
We consider two straightforward multi-objective extensions of the SOTA single-objective LLM pruning method ALPS 330 Experiments are performed on LLaMA-2-7B, For comparison, we show the performance (test perplexity) on applying ALPS to each task individually. While this per-task ALPS requires distinct models for different datasets, serves as a performance upper bound.
Extension 1:
Activation Concatenation. This stacks all calibration activations to form X = and then apply ALPS to produce pruned model.
(X. (X..)),...; (X(")) [T" E (2,
Limitation. This extension produces only one single pruned model, which cannot adapt to varying user preferences across objectives. As can be seen from Table [ dataset conflicts lead to performance
We follow the parameter settings in ALPS.
3

---

Table Test perplexities for multi-objective extensions of ALPS in Section 3.1 at 70% sparsity (i.e., 70% of the weights are pruned).
using LLaMA-2-7B
C4
Code
GSM8K
Average
Per-task
17.66
2.43
2.96
7.68
Extension
20.43
2.63
3.12
8.72
Extension 2
410.22
10.63
6.68
142.51
Core Support
Other Support
Task Support
Task 2 Support
Task
Task 1
Task
2
Stage Dual ADMM
Stage Task-Specific ADMM
Stage 3: Refinement
Figure 1: An overview of the proposed MOSP.
degradation compared to per-task pruning. While the calibration activations can be weighted by user preference, this requires running the full pruning process and storing separate sparse models for each preference vector A (() incurring significant computational and storage costs.
Extension 2: Independent Pruning then Merging. This proceeds in three steps: (i) For each objective j. ALPS runs on (i) to obtain sparse matrix W (ii) A merged matrix computed as:
Wmmergad Σド= X Wi), (iii) Since Wmerrge is generally not s-spase is re- re-pruned by retaining the l entries with the largest magnitudes and setting all others zero
Limitation. As the individual supports Supp W())'s can differ across tasks, the final re- pruning step may discard important weights, even if they are identified as important in the first step. This frequently leads to significant performance degradation across all objectives, as shown in Table
3.2 Proposed Method
The limitations in Section 3.1 highlight the need for new approach to efficiently generate pruned models tailored diverse user preferences. In this section, we introduce Callii-Ojjeetvve One-Shot Pruning (MOSP), which operates in three main stages. (i) Dual ADMM (Section 3.2.1) which uses modified ADMM algorithm to jointly learn a primary sparse model, W. and core support, W.. (ii) Task-specific ADMM (Section 3.2.2). which performs simplified ADMM procedure for each task i separately, leveraging the supports identified in the first stage. (iii) Refinement, which further refines each model using Projected Conjugate Gradient (PCG) as in ALPS [30]. An overview of the proposed MOSP illustrated in Figure [ In the following, we will also discuss inference (Section 3.2.3).
computational costs (Section 3.2.4) and the extension to N:M sparsity (Section 3.2.5).

---

3.2.1 Dual ADMM for Core Support Identification
We adapt ADMM 4 10] to simultaneously learn two nested weight supports. We identify primary weight matrix W with cardinality and a core weight matrix W. inside W with cardinality ok (where a E [0, 1). This is formulated as the following bi-level optimization problem:
W.ERex
Wello I ak, Supp(W C Supp (W), (3) XW Xw. + 7ll w W.lle.ll
subject to
W arg e X W
x + ? W
W l S k
(4)
where x= [X ; (T( is the concatenated activation, and y z is the regularization parameter.
To solve this optimization problem, we apply ADMM to both the inner and outer level problems. Besides the standard dual variables V and V.. ADMM also introduces auxiliary variables D and De for the sparsity constraints. We initialize W D(O), W! and D (0) W, and set both V and Vo!)) O. Let p be the ADMM penalty parameter and H = (X) X + 7I. At iteration the updates are performed sequentially for the weight matrices, auxiliary variables, and dual variables as follows. The detailed derivation is in Appendix A
First, the primary weight matrix (((+1) and the core weight matrix (1-1) are computed as:
(++1)
(H++P) (H+pp) (1 V +(()), W((+1) (H+pT) (HW (( (10+ + (5)
Next, D and D. are updated. Let ((+1) W(++)) + V and W W V The primary support projection yields D = Px ((+1)) by selecting the k largest-magnitude elements of W(++1) The primary support ((++) Supp (upp(DD(+1) ((+ For the support projection, ((+++) is projected onto matrices with at most ak non-zero elements, constrained such that its support is subset of g(++1), This is achieved by first masking (f+1) with g(++1) and then selecting the ak largest-magniuule elements this masked matrix: D Pak (Ww( (W((+)))
The core support is then (!--l) !) = Supp(D (++1)) Finally, the dual variables V and V. are updated as:
(++1) ( p (W(++) ((+1)), v(++1) v -
(6)
The above steps are repeated until convergence, outputting weight matrix W and core support S..
Convergence. in Appendix B
The following Theorem guarantees convergence of the algorithm. The proof is given
Theorem 1. Let {D( ( (D! J and (
( J be the sequences generated by the algorithm in Section 3.2.1 Suppose the penalty parameter {pr satisfies E 1/pt We have
[D (1+1) D ((+1)) (
((+11) 11e; (D(++)) -D |l, -((H))))D((()) [1] S C/pps
max
where C is a constant depending on X, W, and Eie 1/Pr. In particular, there exists matrix D and D. such that ((t)_ D. w(() D. ! D., and w!!) D. as co.
3.2.2 Task-Specific ADMM
After identifying the core support S, and primary weight matrix W in Section applied individually to each task. This aims to obtain task-specific weights W; = I,... m)) while
3.2.1 ADMM is
preserving the core support S... This can be formulated as the following optimization problem:
min W.
X O
XW + + Wil.
s.t.
| < k, Suppu W.) C Supp(W.), (7)
For initialization, matrix W from Section 3.2.1 is used. As this initialization is strong, we use fixed ADMM parameter p. This allows pre-computation of the Cholesky decomposition of (H; + pl) (where H, (x (0)T x (6) ID for each task i, significantly speeding up the optimization process.
5

---

10-
Grdiral
ip 1.00
50
(ppe D
08
00
0.2
7.4
0.6
888
1.0
Ag
Figure 2: Example illustrating the mapping from À to X' for m = 2. Vectors closer to ( 2) shrink toward the origin, while vectors the corners remain unchanged.
Algorithm. For each task i, ADMM sequentially updates W,, D:, and V. At ADMM iteration weight W is updated
(++1)) (H: + (H; Vi + )
Next, the auxiliary variable D; is updated. Given W (((+)) w v((1) /p, the projection Px.s. (-) first retains all elements of W corresponding to S.. Then, it selects the k [Sa)
largest- elements outside Se. The resulting matrix D(++1) = Pk.s, (W(++1) comprises
these selected elements from (++1), with all others set to zero. Finally, the dual variable is updated:
(((-) v(0 + +(W((++) -D((+)))
For efficiency, the above steps are for maximum of 10 iterations, outputting (W:)
3.2.3 Inference
During inference, given user preference vector [A....., Am] we adjust the impact of task-specific corrections (W; W} and create a tailored model with weight matrix Wx Pk.s. (W + Σ X(W, W)) on the fly. Here, Pe.s. is projection that ensures WA has at most is non-zero elements and includes all elements corresponding to the core support S.. We scale based on its & distance from a uniform distribution, normalized by the maximum possible distance:
Σ
j== A
X'
X
2(1 m
where p hyperparameter. For uniform À (i.e., A 1/m, Vj.. X(WW = as the global weight W should provide a balanced solution. For a one-hot X, we have X' = Thus, the mapping from A' smoothly interpolates: near-uniform preferences favor the global model, while task-specific preferences amplify task-specifie adjustments (Figure 2 Moreover, a larger delays task-sp adjustments until À is highly skewed, while a smaller p enables earlier transitions. In the experiments, we simply use 0.5.
3.2.4 Computation Cost
Stage (Section 3.2.1h: In this stage, the computational bottleneck is the calculation of (H+ -P) To address this, similar ALPS [30], we pre-compute and store the cigen-decomppsstton of H

---

as QMQ", where Q is the matrix of eigenvectors and M is the diagonal matrix of eigenvalues. In each iteration. the stored Q and M can then be reused to efficiently compute (H + pI) as Q(M + - Note that since the two optimization subproblems (for W and We) use the same matrix H. this eigen-decomposition needs to performed only once at the beginning of this stage, with a complexity ((c²). The subsequent per-iteration computational complexity is (c. Compared to ALPS, this stage introduces a minor overhead consisting few additional matrix multiplications and additions.
Stage 2 (Section 3.2.2): In this stage, we use fixed p. which enables the prepomaatt of the Cholesky decomposition of (H; + pl), followed by the computation of its inverse based on the decomposition, once for each task. While the Cholesky decomposition has O(c²) complexity, is significantly more computationally efficient in practice. After the inverse is computed, the per- iteration computational complexity for this stage becomes (c) For m tasks, the total per-iteration computational complexity is O
In Section 4.2,
will demonstrate that the computational overhead of the proposed method small in comparison to ALPS.
3.2.5 Extension to Semi-Structured Sparsity
Similar the other unstructured pruning techniques, the proposed method extends naturally to semi-structured sparsity by modifying the D-update projection steps.
Stage 1 (Section 3.2.1): First, partition ((++) into non-overlapping blocks of M elements. Re- tain the N largest-magnitude elements in each block via projection PN:M (-). yielding D
PN:M( (W)+))). To identify the core support. compute the absolute sum of elements in each block of (i(-1) Designate the blocks with smallest sums as "weak blocks," and define the core support S. by excluding these blocks. The result is D ((+1) PN:M (W: @ Se.
Stage 2 (Section 3.2.2): Using core support S, from Stage apply the projection PN:M,S. (-) by preserving elements in (((++) at positions in S. while maintaining the N:M structure for the remaining elements. Specifically, retain up N largest-magnitude elements in each block of M non-core blocks, resulting ((++1) PN:M.S, (WMM,.((((())) (W(++)) This ensures the N:M sparsity pattern while preserving the identified core support.
4 Experiments
In this section, we show the experimental results of the proposed Multi-Objective One-Shot Pruning (MOSP). We begin by outlining the setup in Section 4.1 Subsequently, demonstrate the effective- ness of MOSP on a two-objective pruning scenario (general language understanding and mathematical reasoning) in Section 4.2 We then extend the evaluation three objectives, incorporating code generation capabilities, in Section 4.3 Finally, we provide ablation studies in Section 4.4
4.1 Setup
We evaluate MOSP across multiple LLMs including Llama-2 [38] Llama-3 [6]. and OPT series [43
We consider three representative datasets to evaluate model performance across different domains:
(1) General Text: C4 (3). (2) Mathematical Reasoning: GSM8K 1.. (3) Code Generation: Code [5].
Following 15 30 we use calibration set of 128 segments (up to 2048 tokens) and evaluate using
perplexity (PPL) the test sets (the lower the better).
We compare MOSP with state-of-the-arr unstructured pruning methods: Magnitude Pruning (MP) 18. Wanda 37 SparseGPT [15], and ALPS (30. We use the same pruning datasets across all methods. For Wanda. SparseGPT, and ALPS, we use the activation concatenation strategy in Section 3.1. We exclude independent pruning then merging (extension 2) due to its poor performance (as shown in [T) The penalty p is adapted using the same strategy as ALPS [30]. We set a = 0.5 and p 0.5 without hyperparameter tuning. Additional experimental details are provided in Appendix
7

---

4.2 Two-Objective Pruning
We first demonstrate the effectiveness of MOSP for prun- ing Llama-2-7 2-7B on two datasets: C4 and GSMBK. The un- pruned Llama-2-78 baseline achieves test PPL of 6.97 on C4 and 2.73 on GSMSK.
Table 2: Comparison of time and GPU memory consumption between ALPS and the proposed MOSP.
Figure 3 shows the performance 70% unstructured spar- sity and 2:4 semi-structured sparsity. MP and Wanda are omitted due to their significantly inferior performance. As can be seen, the baseline algorithms produce single pruned model, whereas MOSP generates diverse Pareto set, demonstrating the trade-off between general language understanding and mathematical reasoning. This allows users select models that align with their preferences.
Method
Time
GPU Memory
ALPS MOSP
1.09h
18.70 7GB
1.36h
20.3GB
In Figure 4 vary the sparsity ratio. As be seen, MOSP generates meaningful Pareto fronts across varying sparsity settings. Notably, higher sparsity ratios amplify the trade-off region, empha- sizing the value of multi-objective optimization greater compression levels.
Table 2 examines the time and GPU memory overhead of the proposed MOSP compared with ALPS. As can be seen, both the time and memory overhead of the proposed method are small.
POPP
MAAHOPTIINN ALF CHI
42-
80%
2:4
65H8K
23-
26.
Figure 3: Test PPL on C4 and GSMSK for Llama- 2-7B pruned to 70% unstructured sparsity and 2:4 semi-structured sparsity.
Figure 4: Test PPL on C4 and GSMaK for Llama- 2-7B pruned to 50%, 60%, 70%, 80% unstruc- tured sparsity and 2:4 semi-structured sparsity.
4.3 Three-Objective Pruning
In this section, we evaluate MOSP in scenario involving three objectives: general language understanding (C4), mathematical reasoning (GSM8K), and code generation (Code). Figure shows the solutions obtained by MOSP on pruning the Alama-2-7BB model to 70% unstructured sparsity. Each point on the resulting three-dimensional surface represents a pruned model, generated using of the 36 uniform preference vectors (A). As can be seen, MOSP successfully identifies diverse set of models spanning different objective trade-offs. More figures for different models pruned to different sparsity levels provided in Appendix E
T Table provides a quantitative analysis comparing models pruned using the baselines with represen- tative points selected from the models obtained by MOSP for Llama-2-7B at 70% sparsity. These selected points correspond to different preference vectors: A represents balanced preference across all three objectives, while A X², and A heavily favor C4, GSMSK, and Code, respectively, with X representing an intermediate preference. The results clearly demonstrate MOSP's ability to effectively navigate the trade-off spaces by adjusting the preference vector X. For instance, X yields the lowest PPL on C4, while 3 achieves the lowest PPL on Code.
Table 4 compares the Hypervolume (HV) 47 21 achieved by MOSP with baseline methods for pruning various OPT and Llama models to different sparsity targets. A higher HV value indicates better approximation of the true Pareto front, reflecting solutions that are both high-performing
8

---

32
24
25
12-
31
3.2
3.
20
C4
12-
3.2
1.1
27
2.7
3.1
3.4
C4
GSPPX
Test PPL on C4, GSM8K, and Code for Llama-2-7B pruned to 70% unstructured sparsity, The top left figure shows 3D view and the other three figures show different 2D projections.
Figure 5:
Table 3: Test PPL of Llama-2-7B pruned to 70% unstructured sparsity with different prefer- ence vectors. A [0.33, 0.33, 0.33), ( = [1,0, (())) 1,, = [0,0, 1), X (0.72,0.14,0.3)
Method
C4
GSMSK
Code
MP
9839, ,28
6191.83
3285.59
Wanda
73.64
12.90
10.51
SparseGPT
30.91
3.53
3,07
ALPS
20.44
3.12
2,63
MOSP (A
20.44
3.12
2.63
MOSP (A*)
18.80
3.37
3.20
MOSP (A²)
24.33
3.05
3.06
MOSP (X3)
25.82
3.43
2.53
MOSP (X1)
19.30
3.20
2.79
(closer to the ideal point) and diverse (covering a broader range of trade-offs). The reference point is set as 1.1 times the PPL of the model obtained using SparseGPT. Note that since the reference point varies across different sparsity levels and models, HV comparisons are only meaningful within the same model-sparsity combination. Results show that MOSP consistently outperforms SparseGPT and ALPS in HV, Wanda performs far below the reference point, yielding an HV of zero.
4.4 Ablation Studies
This section studies the impact of key components in MOSP: the support sharing parameter and the preference mapping strategy. All experiments use Llama-2-7B pruned to 70% sparsity, with the hyperparameters kept consistent with the other experiments.
Effect of a. Parameter a controls the degree of support sharing, interpolating between fully task- specific supports (a 0) to completely shared support (a = 1). Figure 6al shows that neither

---

Table Comparison of HV on Llama-2, Llama-3, and OPT models across various sparsity levels
Sparsity
Method
OPT-2.7B
Llama-2-7B
Llama-3-8B
Llama-2-13B
OРT-30B
Wanda
0.00
0.00
0.00
0.00
0.00
SparseGPT
0.39
0.09
0.27
0.07
0.16
2:4
ALPS
0.99
0.38
1.39
0.21
0.28
MOSP
1.18
0.48
1.70
0.25
0,31
Wanda
0.00
0.00
0.00
0.00
0.00
SparseGPT
0.70
0.33
0.99
0.20
0.20
70%
ALPS
4.68
7.73
25.36
3.35
0.72
MOSP
5.67
9.59
30.85
4.24
0.85
Wanda
0.00
0,00
0.00
0.00
0,00
SparseGPT
4.86
4.83
9.28
2.02
0,90
80%
ALPS
168.50
770.74
1180.42
220.53
43.56
MOSP
188.26
887.52
1287.07
259.21
48.65
wofootrrrrooom
COMOX
505
305
(a) Effect of
(b) Effect of preference transformation.
Figure 6: Ablation studies for Llama-2-7B pruned to 70% sparsity.
extreme yields the optimal trade-off. An intermediate a, as used in our main experiments, allows partial overlap in supports. This enables MOSP to maintain similarity for better interpolated models while still permitting task-specific specialization.
Effect of preference transformation. Figure 6b shows the effect of the transformation from À to X' While the performance at the extreme ends remains the same, other models generally achieve better performance when the transformation is applied. This indicates that the performance transformation helps in finding superior intermediate models.
Conclusion
In this paper, consider the multi-objective nature of LLM pruning and formulate it as a MOO problem. The proposed method MOSP, with proven convergence, efficiently identifies models with varying trade-offs across different objectives. Experiments on representative models demonstrate approach's 's effectiveness in providing tailored models based on user preferences.
Limitations. We only evaluate MOSP on some representative models. Extending this evaluation to broader range of LLMs would be valuable. Additionally, considering more objectives an interesting future work.
Acknowledgment
This research was supported in part by the Research Grants Council of the Hong Kong Special Administrative Region (Grants 16202523 and HKU C7004-22G).
10

---

References
[1] Yongqi An, Xu Zhao, Tao Yu, Ming Tang. and Jinqiao Wang. Fluctuation-bast adaptive structured pruning for large language models, In AAAI Conference Artificial Intelligence. volume 38, pages 10865-10873, 2024.
[2] Salch Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. Preprint arXiv:2401.15024 2024.
[3] Davis Blalock, Jose Javier Gonzalez Ortiz, Jonathan Frankle, and John Guttag. What is the state of neural network pruning? In Proceedings of Machine Learning and Systems, volume pages 129-146, 2020.
[4] Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. Distributed opti- mization and statistical learning via the alternating direction method of multipliers. Foundations and Trends in Machine learning, 3(1):1-122, 2011.
[5] Sahil Chaudhary. Code Alpaca: An instruction-follloinng LLaMA model for code generation. https://gi thpp:bu com/ www.saaoaoaaaaaaea 10011111111111111a//aaaooa 2023.
[6] Weiyu Chen and James Kwok. Multi-objeccivee deep learning with adaptive reference vectors. Advances in Neural Information Processing Systems, 35:32723-32735, 2022.
[7] Weiyu Chen and James Kwok. Efficient pareto manifold learning with low-rank structure. In International Conference on Machine Learning, pages 7015-7032, 2024.
[8] Weiyu Chen, Baijiong Lin, Xiaoyuan Zhang, Xi Lin, Han Zhao, Qingfu Zhang, and James T Kwok. Gradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. Preprint arXiv:2501. 100.:19551..555,,.. 2025.
[9] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. Preprint 11.iv:2110.14168. 2021.
[10] Damek Davis and Wotao Yin. Convergence rate analysis of several splitting schemes. In Splitting Methods in Communication, Imaging, Science, and Engineering, pages 115-163. Springer, 2017.
[11] Lucio Dery, Steven Kolawole, Jean-François Kagy, Virginia Smith, Graham Neubig, and Ameet Talwalkar. Everybody prune now: Structured pruning of lims with only forward passes. Preprint arXiv:2402.05406, 2024.
[12] Jean-Antoine Désidéri. Multiple-gradien descent algorithm (MGDA) for multiobjective opti- mization. Comptes Rendus Mathematique, (50(5-6):31331) 2012.
[13] Nikolaos Dimitriadis, Pascal Frossard, and François Fleuret. Pareto manifold learning: Tackling multiple tasks via ensembles of single-task models. In International Conference on Machine Learning, pages 8015 8052, 2023.
[14] Jörg Fliege and Benar Fux Svaiter. Steepest descent methods for multicriteria optimization. Mathematical Methods of Operations Research, 51(3):479_49) 2000.
[15] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pages 10323-10337, 2023.
[16] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. Preprint arXiv:2407.21788 2024.
[17] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[18] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.

---

[19] Yihui He, Ji Lin, Zhijian Liu, Hanrui Wang, Li-Jia Li. and Song Han. Ame: Automl for model compression and acceleration on mobile devices. In Proceedings of the European conference on computer vision (ECCV), pages 784-800, 2018.
[20] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry. Accelerated sparse neural training: A provable and efficient method to find n: m transposable masks. Advances in neural information processing systems, 34:21099-21 2021.
[21] Joshua D Knowles, David W Corne, and Mark Fleischer. Bounded archiving using the Lebesgue measure. In Congress on Evolutionary Computation, pages 2490-2497, 2003.
[22] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
[23] Wei Li, Lujun Li, Mark Lee, and Shengjie Sun. Adaptive layer sparsity for large language models via activation correlation assessment. Advances in Neural Information Processing Systems, 37:109350-109380 2024.
[24] Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qing-Fu Zhang, and Sam Kwong. Pareto multi-task learning. In Neural Information Processing Systems, 2019.
[25] Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-averse gradient descent for multi-task learning. In Neural Information Processing Systems, pages 18878-18890, 2021.
[26] Xingchao Liu, Xin Tong, and Qiang Liu. Profiling Pareto front with multi-objective stein variational gradient descent. In Neural Information Processing Systems, pages 14721-14733, 2021.
[27] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. Metapruning: Meta learning for automatic neural network channel pruning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 3296-3305, 2019.
[28] Xinyin Ma, Gongfan Fang, and Xinchao Wang. language models. Advances in neural information processing systems, 36:21702-22 21720, 2023.
On the structural pruning of large
[29] Debabrata Mahapatra and Vaibhav Rajan. Multi-task learning with user preferences: Gradient descent with controlled ascent in Pareto optimization. In International Conference on Machine Learning, pages 6597-6607, 2020.
[30] Xiang Meng, Kayhan Behdin, Haoyue Wang, and Rahul Mazumder. Alps: Improved optimiza- tion for highly sparse one-shot pruning for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[31] Kaisa Miettinen. Nonlinear Multiohjective Optimization, volume 12. Springer Science & Business Media, 1999.
[32] Hiroaki Mukai. Algorithms for multicriterion optimization. IEEE Transactions Automatic Control, 25(2):177-186, 1980.
[33] Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as bargaining game. In International Conference on Machine Learning, pages 16428-16446, 2022.
[34] Aviv Navon, Aviv Shamsian, Ethan Fetaya, and Gal Chechik. Learning the Pareto front with hypernetwrrs.. In International Conference on Learning Representations, 2020.
[35] A Paszke. Pytorch: An imperative style, high-performance deep learning library. Preprint arXiv:1912.01703: 2019.
[36] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1-67, 2020.
12

---

[37] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. Preprint arXiv:2306. 1695, 2023.
[38] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. Preprint arXiv:2307 20000222000009888... 2023.
[39] Peng Xu, Wenqi Shao, Mengzhao Chen, Shitao Tang, Kaipeng Zhang, Peng Gao, Fengwei An, Yu Qiao, and Ping Luo. Besa: Pruning large language models with blockwise parameter-efficient sparsity allocation. Preprint arXiv:2402.16880 2024.
[40] Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. Preprint 1rXiv:2402.11187. 2024.
[41] Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yi Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, Ajay Jaiswal, Mykola Pechenizkiy, Yi Liang, et al. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning iims high sparsity. Preprint arXiv:2310.05175. 2023.
[42] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Loraprune: Structured pruning meets low-rank parameter--fffiiiet fine-tuning. Preprint MrXiv:2305.18403 2023.
[43] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. Preprint aixi::200500 001068, 2022.
[44] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-am and--aay An efficient post-training training pruning method for large language models. In The Twelfth International Conference Learning Representations, 2024.
[45] Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao. Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse no training: Training-free fine-tuning for sparse Ilms. In The Twelfth International Conference Learning Representations, 2024.
[46] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning n: m fine-grained structured sparse neural networks from scratch. Preprint 1rXiv:2102.04010 2021.
[47] Eckart Zitzler and Lothar Thiele. Multiobjective optimization using evolutionary algorithms A comparative case study. In International Conference on Parallel Problem Solving from Nature, pages 392-301, 1998.
13

---

A Detailed Derivation of Dual ADMM
We the Alternating Direction Method of Multipliers (ADMM) solve these constrained opti- mization problems. The updates for the primary weights (W, D. V) and core weights (We, De, V.) are performed sequentially within each iteration l. Let H X X +7I.
A.1
ADMM for Inner Problem
The inner problem 44 aims to find the primary weight matrix W. The problem is:
min
XW Xw|
+ W w/|
s.t.
[wl S k.
(8)
p
We introduce an auxiliary variable D = W and a scaling factor of z to simplify the resulting expressions:
+7 w w((²) (IDD),,((D) s.t. W D:
XW XW W,D
min
(9)
F
where I (I,),s(( an indicator function that if DIlo S and do otherwise. The augmented Lagrangian is:
1 XXW-xw/p +7 w WWwl1; +1 DI. (D)+ L |
(,W,D,V)
DDI
2
(10)
The ADMM iterations consist of the following updates:
1. W-update: (t-11) = arg minw (W D v(t)). This involves minimizing:
XW [Ww + W w wl()) (V w) w - D
(11)
Taking the derivative with respect to W and setting it to zero:
w w + (V W)F P W W- - - |1
WWS
XW
XW
F
XWW- XW) WW W) v + p(W ((1)) =0
X X II + W = -XX X W + pD( V
Using the definition
(X) X +
(H + pl) (((+)))+))))) HW (D (t))
(12)
Thus, the update for W is:
((++)) = (H + pl) (HW ( + (D
(13)
2. D-update: ((+1)) = arg minp L, (++1) D. v(t)). This involves minimizing:
InDD su (D) (V DD + P
W ( -D
(14)
2
This can be rewritten by completing the square for terms involving D:
(e + D (w(++)) /(9pp) const.
(15)
The solution is obtained by projecting (t+11) (() onto the of matrices with at most non-zero elements. Let ((+1) = w((+1) + (1) /p.
( Ps (W((1)))
(16)
where (A) is an operator that keeps the elements of A with the largest magnitudes and sets others zero. The primary support is S(++1) Supp D((1)).
V-update: The dual variable V is updated as:
(t-1) (()) + p( (((1+) D((+!)).
(17)
14

---

A.2
ADMM for Outer Problem
The outer problem (3) aims to find the core weight matrix We. The problem is:
mim
XW
XWo|
[|]
+7 W We
s.t. |Wello S ak, Supp W.) C s(t+1) (18)
where ((++1) Supp D(++1) is the primary support identified at iteration / from the inner problem's ADMM step.
Similarly, we introduce an auxiliary variable De W. and scaling factor of & to simplify the resulting expressions:
(1 XW XW. +7 W We [/2 F) W.
min
+ +II Sa..llo D.) (Do), s.t. W. = De-
(19)
The augmented Lagrangian is:
2 L. Wo: De, Vo) = 1 (|X - XW: -|
w:ll²)
+7 W - W. F:
I.D.lll [iD.lllsskkspp(..cccs..s (Da) + (V, (We Dz)) + P W. Dell (20)
Wudupate: W arg minw. We,
(W. D V( ). The objective function for W.
identical in form to that for W. Thus, the derivation is analogous:
(H + pl) HW + DD V(O)
(21)
The update for Wa is:
w((+1)= (H PI) v (D)
=
(22)
De-update: D(+1) argminp, (W
( De. V.()). This involves minimizing:
l D. p (w((-) V /29 2
LiD.ll.Sakk.upp(..)c.s (D.)
(23)
2
Let (:(+1) = (!(+1) + v.(') /p. The solution is obtained by first ensuring the support constraint Supp(D.) C S(++) and then projecting onto the set of matrices with at most ak non-zero elements. This is achieved by masking (++1) with the primary support S(++1) and then selecting the ak largest magnitude elements from this masked matrix:
D((-1) Pak (W S(41)).
(24)
3. V-update: The dual variable V. is updated as:
V (t-1) v(t) + +(W(tt))) D((+)))
(25)
These derivations provide the update rules for W, W.D.Y and W.. De, V. as presented in Section 3.2. of the main paper.
B Proofs of Theorem
Proof. Since we do not modify the update steps for W and D compared to ALPS (30, the con- vergence proof of ALPS remains valid for W and D. Here, we further extend the proof in 30 to analyze the convergence of W. and D.-
For the sake of conciseness, throughout the proof, we denote H = X X and G = X XW. To establish the theorem, we first present the following two lemmas. The proofs of these two lemmas given in Section B.I and B.2 respectively.
15

---

Lemma 1. Let {D!
and {v!")
(v(0)
be the sequence generated by MOSP. Then for any 2
1=0 t=
holds
HV
I
|V
G
HD
F+
(26)
Pt.
and
2
|GG
HV |
(D(+)) D
S
(27)
Pt
Lemma Let _00
[w!")
1=0 and {v!"]]
be the sequence generated by MOSP.
1=0 Suppose {pe h is non-decreasing. Then for any 2 0, holds
H
)
3
G|G
1
(28)
Pa
Ps
==0
Returning to the proof of the main theorem, combining Lemma 2 with the initialization of our method MOSP gives
[D
|F
IF
+
3
|D [1
|I
||
3
G
Pt
Ps
Pa
Ps
L==0
==0
(29)
exp
3||H||2
|
|G||
3
Ps
Ps
Let
[GG|| [[I
Σ
1
C(X, W. == 2||G|| + H||
exp 3 H
==0 Ps
Ps
(30)
be the constant depending on X, W and E. 1/ps. Lemma together with
29 leads to
HV [V(+1) F S G HD
(1)
lr
Pe
(31)
[ |F + s |G|| + H||
2 C( Wo,,,t,
and
HHV!" 1|
C(X, W, ,o. t)
D (11))
-
|| S
|
HD (4)
(32)
6
Pt
c
Pt
then follows from (++1) D (1+1) = (V(++1)
)/p that
((+1)) DD(+1)
[
F
!IV!
C(X, W. Po. tu. 7)
(33)
Therefore, we prove the desired inequality. Since E 1/0, < co, D a Cauchy sequence, and therefore there exists matrix D. such that D D.. follows from w((+1) DD (+1) 0 that (!! - D... The proof is completed.
F
16

---

B.1 Proof of Lemmal! 1
Proof. According to the W W-update rule in 5 holds
p()) (H+ (H+) (G v ) ( V!
(1)
w - (
p(()
(G - v( + v
((H + prt) -1 Pt 1) D + (H + pcl)
Pt
v!
H
H
I+
HD
(t)
I+
(G v( +
Pt
Pt
Pt
P
I+
H
-1
(G
HD
(0)
1
I
I
H
V(()
Pt
P:
Pt
Pt
1
H
-1
HV!")
(1)
I+
G
HD
(1)
Pt
Pt
Pt.
(34)
Therefore, we obtain
V!!
I H
-1
HV!"
w
D
+
$
G
HD
+
p())
Pt
Pt
Pt
2
HV
(1)
G
HD
(35)
Pt
Pt
F
HH (
|G
(0
F
Pt
Pt
Denote 7(0 = ((i.j) E [c] x
[d]
= ((.,))
x [d] (D. )ij = =0}. It follows
from the D D-updete rule and the definition of the projection operator that
(
w((+1) + v(())
D D - w((+)
min IC ICSupp D(t+)) (1--11
Σ
Pt
(LJ)EZ
Pt. i.j
(36)
(!
2
Σ
W(++1)
+
Pt
(1)))Z(11)
By definition, note that Supp D ((+ ( u( = [c] x [d] and Supp Supp(D((+1)) I(+++)) (n 1)++1) = o. then we
can write Z(()) = (7(!) -(7!)) n Supp D D((t))) U
((t) nz(-1)) Then from [[(+1]] = cd - k, we must
have (T(0) n I(+11 I cd - k. And since
( cd - 1k and ((t))
(7()) n Supp D (++1)) U
(T9) nn(+)))). we should have (700) n Supp Supp(D((+U)) ((+))) 2 (1 - 1)k. Therefore, assume 7 0
Supp (L+)() = (1 (1-1)Ks (1-())=A> with A > O, which also directly gives us _ nz(+)) < k A
we first should have:
(1) 2 W((+)) +
W((+1) ((()) 2
min ZC Supp((D++))) [1=(111jk
Σ
min I (uupp(DD [1=(111)k
Σ
Pt. / ij
Pt. i.j
(NJ)EI
(1,)EI
( 2
Σ
w(++1) (D( +
min IC (7 Suppp! (Suppp (() 11= 1ij
Pr.
(iJ)Ez
i.j
(37)
17

---

Regarding the second term in (6). we have:
+ v
2
Σ
W
= Σ
W
+
Pt ij
Pt
(Gj)etie))
i.j
2
Σ
W
+
Pr
(3))
By definition, should have (7(+1) 1 A, which also matches the number of additional elements in [T[*) Supp(D(t+)) other than the smallest (1 - -)) clements it. Then since all
elements in [(++) corresponds the smallest ed - k elements in W(++1) v should have:
De
( +
W V(I)
Σ
w((+1)
Σ
IC (Spppp (upp(( (8,5)ZI [=|=
Pt
Pt
id
bd
Thus we have:
((( + V(!)
w V(!I} 2
Σ
Σ
Pi in
(t,))2+++))
i
V!t) 2
w
max
(38)
Ig (
c
Pt. i
(K.))
=
Then combining 37 and will give us:
v(()
2
V((() 2
D
D
(++1)
((++1)
min
Σ (K.))z
w +
Pt
((1))
F
(1-=(1
i.j
w((+1) + (!))
2
Σ
Pt.
(K.)
ij
(1) 2
w((-1) { 2
Σ
w +
Σ
(39)
Pt
Pt ij
(W)) (supp (Bupp((())))))
i.
(iu))
(!)) 2
Σ W(++1) D +
c
Pt
(5))
ij
V(O)
$ W((-1) -())
Pt
Together with 35 we get
D(+D w((+1) !!!)
[HV
1
<
|
HD
(0)
(40)
Pt
Pt
Pt.
then follows from the V -pddat rule that
(t-
D (++1)
w(++1)
(t)
1 ||G HD
H
(41)
Pr
Pr
P:
Pr.
This establishes the inequality 26 triangle inequality, we verify the inequality 27
Furthermore, by summing up 355 and
(40
and applying the
D
18

---

B.2 Proof of Lemma 2
Proof. It follows from Lemma that
| |F
|V |r &
|r
Pt
(42)
|||||
V 1
H2 |
|
Pt
and
2
HH
D ([(+))]
D
| - HD 0 1
(43)
2
|||||||D
(0 |F
+
|
This further implies
D
2 1
2 2|
| (4)
|
||
2||
2
F
(44)
Pt
Pe
Combining inequalities 442 and yields
| F
1--11 D 1
11
[
(++
c
P++1
Pt
H||
3|| 3|Hl|//lV!
| ( c
+
(45)
pi
Pt
Pt
H||H||
[V:
3||
Denote a, | + [IV(!))|
|| Pro then the above inequality be rewritten
3
H||
2
3/|
ac+1 S
1
+
(46)
Pt
Pr
Therefore,
0c+1
a₁
3 3|G
oo (1 3 H pk))
[-1
(1
H
/ pk:)
Pr II.
o (1 2/2///
(47)
31 G F
0=0 (1 + 3 H 2/)b)
Pr
It then follows from telescoping that
O2
3 G
< ag
(48)
TT (1 + 33 Recalling the definition of a, completes the proof.
Pt
#=0
C Details of the Refinement Stage
Algorithm details the refinement stage, which further refines each model using Projected Conjugate Gradient (PCG), following ALPS 30
D Experimental Details
In this section, we provide more details about the experiment.
We perform all experiments on a GPU server equipped with NVIDIA A6000 GPUs, each with 48GB of memory. However, for each individual experiment, we utilize only single GPU. We use Py Torch version 2.0.0 35). The total time of execution for all reproduce all experiments around 100 hours.
19

---

Algorithm PCG with vectorization [30 for ith task.
Require: Support S. pre-conditioner M DH), initial solution W;
1: Set R (0) = H ( W
W
2: Project ( onto the support S by setting all elements outside the support to zero.
3: Set (0) = M M and p z
4: for /- = l... do
(11) sum R (4) 3 ( axis 0) sum(P()) (P() @ Q axis 0) ((++)) ( + (1) о P(()
5:
6:
7:
R(++)) = ( - ( c H,P())
8:
Project R ( onto the support by setting all elements outside the support zero.
9: ((-1) R-
10: if R (++1)) is sufficiently small then
break 12: end if
800 sum(R (++1) 3 ((+1),, axis 0)
13:
sum R(()) @ ( axis 0)
sum(R(0)
p(++)) = Z (++1) 3 0 P())
14:
15: end for
Preference Vectors. For MOSP, we sample user preferences (A) trace the Pareto front. In the two-objective scenario, we employ 11 uniform preference vectors (i.e., [1,0] (0.9,0.1], ..., [0.1, 0.9), [o, 1)). For the three-objective scenario, we utilize uniform pref- erence vectors. These vectors are generated by defining 2-simplex (an equilateral triangle where components sum to 1) and selecting points (A₁, \2,,)2,44 Ag) such that X; k/S for k; E No and E, = S. For 36 vectors, this corresponds to S 7. The meshzoo package is used to help generate the vertices a triangular mesh, which correspond to these preference vectors. Each preference vector À produces distinct pruned model on the Pareto front.
The Hypervolume (HV) indicator 21 is popular metric in multi-objective optimization (MOO), offering a measure of the quality of an obtained solution set by quantifying the volume of the dominated portion of the objective space. Formally, for a given solution set and reference point E R." (where m is the number of objectives), the HV is defined as:
Hypervolume.
HV(Prr)= = A (UGEER (UGERTIDZEZE))
(
(49)
where A denotes the Lebesgue measure, and p means dominates or is equal to q (assuming maximization of objectives). A larger HV preferable, indicating better approximation of the true Pareto front.
Hyperparameter Setting. As mentioned in the main paper, we both and p 0.5. For other hyperparameters, we follow the settings in [30] Specifically, we set the initial penalty parameter Po 0.1. We update p every iterations based on step function that depends on the current value of P: and st (upp(D (t) A ( The term & represents the number of elements in the symmetric difference between the support of D at iteration and iteration t - 3. Specifically, the update rule is:
1.3pt
if 2 0.1k, if s, 2 0.005k,
Pt+1
1.2pt
(50)
1.1pt
if &t 2
where k is the total number of elements or parameters being considered for pruning. If s, - 0. indicates that p is sufficiently large and the support has stabilized. For stage 2, the task-specific ADMM, we use fixed p = 0.5. We set y to 0.01Tr(X X). We refine each model with 10 PCG iterations.
20

---

Dataset. We use three publicly available datasets for our experiments:
ODC-BY License. This a large-scale, cleaned version of the Common Crawl dataset, containing primarily English text. Following common practice 37 15 30), we use a subset of C4 dataset. We use the same data split as [30].
C4 (Colossal Clean Crawled Corpus) 36
GSM8K (Grade School Math 9]: MIT License. A dataset comprising high-quality grade school mathematics word problems designed to evaluate the multi-step reasoning capabilities of models. We follow the official data split.
Python Code : CC-BY-4.0 License. An ceeruction-follwiing dataset tailored for Python code generation, styled after the Alpaca dataset. We follow the official data split.
Licenses for Models. The licenses for the models are as follows: for the LLaMA-2 series, the li- cense is the "LLaMA 2 Community License Agreement", ; for the LLaMA-3 series, it is the "LLaMA Community License Agreement": and for the OPT series, the license is the "OPT License Agreement."
E
Additional Experimental Results
Extension to More Objectives. In this section, in addition to the three objectives discussed in Section 4.3 we introduce a fourth objective: multilingual performance evaluated on Chinese 2.0. All other experimental settings remain the same as in Section 4.3 The results, summarized in Table 5 show that MOSP continues to effectively identify diverse Pareto-optimal solutions across all four objectives.
Table 5: Test PPL of Llama-2-7B pruned to 70% unstructured sparsity with different preference vectors.
Method
C4
Code
GSM8K
Chinese
SparseGPT
30.52
3.21
3.60
16.57
ALPS
20.57
2.67
3.17
9.53
MOSP(A(1) (1,0,0,0)
19.17
3.37
3.42
14.31
MOSP(A(2) (0.1,0,0)
26.69
2.57
3.50
17.31
MOSP(A33) 10,0,1,0)
25.07
3.25
3.07
18.30
MOSP(A 10,0,0,1)
22.52
3.26
3.51
8.17
(OSP(A 0.007,000,,00 0.07,
20.19
2.97
3.34
8.52
Efficiency Improvement. Our observed speedups are consistent with those reported in SparseGPT [5]. This is because speedups are fundamentally determined by model sparsity and hardware; the pruning algorithm used makes only minimal difference. We refer the reader to SparseGPT [15 for more comprehensive results.
Additional Visualization Results. We present additional visualization results across various models and sparsity levels. Figure 7 compares MOSP and SparseGPT when pruning Llama-2-77 7B to 50%, 60%, 70%, 80% unstructured sparsity and 2:4 semi-structured sparsity. Figures g and 9 show the performance of Llama-2-7B pruned to 2:4 semi-structured sparsity and 80% sparsity, respectively. Figures T0
12 13 14 show the performance of OPT-1.3B OPT-2.7B. Llama-3-8B. Llama-2-
and OPT-30E at 70% sparsity. These results demonstrate that the proposed method consistently provides diverse trade-off solutions.
F Further Discussion on the Motivation
Our primary motivation is to serve the diverse range of user preferences that exist between the extremes of "ure generality" and various "pure specializations." " For instance, different deployments of a model on edge devices may require different balances between capabilities: One user might prioritize language understanding (60% importance) over coding (30%) and math (10%), while another may require strong mathematical reasoning (80% importance). As the number of objectives
21

---

increases, these preference combinations grow exponentially. MOSP efficiently addresses this by obtaining set of trade-off models in a single pruning run. This allows users to select the best-fit model for their specific needs post-training, avoiding the significant computational cost of re-pruning for every new preference. For instance, handling 100 distinct user preferences with traditional methods would take over 100 hours, whereas MOSP accomplishes this in just 1.36 hours. This approach also uniquely allows for dynamic preference adjustments.
Comparison with Single-Objective Baselines. To further contextualize these trade-offs, we com- pared MOSP against single-objective optimization by running the ALPS baseline on each task individually. The results are shown in Table 6 As be seen, single-objective ALPS achieves optimal performance on its target task but shows significant degradation on others. ALPS (optimized on all 3 objectives) provides more balanced performance but offers only one single pruned model for all possible user preferences. MOSP enables users to select from various reasonable trade-offs, maintaining acceptable performance across all tasks while allowing preference-basee bassed customization.
Table 6: Comparison of MOSP and single-objective optimization using ALPS.
Method
C4
GSM8K
Code
ALPS (optimized on C4only)
17.66
18.71
5.00
ALPS (optimized on GSMBKonly)
39.63
2.96
7.85
ALPS (optimized on Codeonly)
44.88
4.63
2.43
ALPS (optimized on all 3 objectives)
20.44
2.63
3.12
MOSP () [1,0, 0))
18.80
3.37
3.20
MOSP (A = 0,)
24.33
3.05
3.06
MOSP (A 0,
25.82
3,43
2.53
MOSP (A (0.72,0.4,,..1))
19.30
3.20
2.79
Interpreting the multi-stage optimization. For efficient Pareto front exploration, we design multi-stage optimization which disentangles the shared and task-specific knowledge through the following interactions:
Stage I (Dual ADMM): Identifies a foundational "core support" representing weights broadly beneficial across all tasks. This captures common knowledge required for the multi-task problem and serves as strong shared prior.
Stage (Task-specific ADMM): Leverages the core support from Stage to guide task-specific pruning. For each task, performs a separate ADMM procedure where the core support acts as regularization, ensuring task-specific weights build upon the shared foundation while incorporating task specific adaptation.
This two-stage decomposition enables efficient Pareto front exploration, decoupling of shared knowl- edge preservation from tase-spec optimization.
G Broader Impacts
This paper proposes method for providing personalized pruned models tailored to different users. By utilizing these pruned models, users can reduce computational costs, making the approach more environmentally friendly. However, with other pruning methods, the models may produce more inaccurate outputs after pruning. Therefore, careful verification of the results is necessary during use.
22

---

SparseGPT (50%)
KOSP (60%)
HOSP (70%)
SparseGPT (70%)
HOSP 180%)
COMBC
SparseGPT (90%)
4.-
HOSP 12:4]
SparseGPT (2:4)
Figure 7: Test PPL on C4and GSM8Kfor Llama-2-7B pruned to 50%, 60%, 70%, 80% unstructured sparsity and 2:4 semi-structured sparsity.
3.0
10
2.30
110
Ootee
Figure 8: Test PPL on 04, GSM8K, and Code for Llama-2-7B pruned to 2:4 cemi-structured sparsity.
23

---

235
2.20
255
2.0
11.
115
125
325
13A
244
280
2.40
255
2.50
2.5
2.40
2.5-
2/4
2.9
2/8
135
2.5
139
3.28
12.
11.5
12.9
12.
220
280
2.8
2,55
3.2
3.6
Figure 9: Test PPL on C4, GSM8K, and Code for Llama-2-7B pruned to 80% sparsity.
12-
35
12-
150
200
300
12
Figure 10: Test PPL. on C4, GSM8K, and Code for OPT-1.3B pruned to 80% sparsity.
24

---

AUSS
60-
6.5
4.2-
4.0
232+
380
389
Figure 11: Test PPL on C4, GSMSK, and Code for OPT-2.7B pruned to 70% sparsity.
NG2
4.24
Figure 12: Test PPL on C4, GSM8K, and Code for LLaMMA-3-SB pruned to 70% sparsity.
25

---

NOW
200
2.80
275
2.9
g 1350
2.00
209
2/8
Gunne
Figure 13: Test PPL on C4, GSM8K, and Code for LLaMMA-2- A-2--B pruned to 70% sparsity.
MGSP
4.0
4.5
455
144-
3.00
3.8
3.2-
2.0
GGROX
Figure 14: Test PPL on C4, GSM8K, and Code for OPT-30B pruned 70% sparsity.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, focus on LLM one-shot pruning, and the contributions are clearly outlined the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitation in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification:
We provide the proof in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the information needed to reproduce the main experimental results of the paper in Section 4 and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
28

---

Answer: [No]
Justification: We have provided all the necessary details to reproduce the experiments in Appendix [ The code will be released later.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpoolliigguddde/oooooee http:/uuddd//oeeSuubnr ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided experimental setting in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to resource constraints, we did not report error bars but will include them if additional GPUs become available.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
29

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///aaarpp.ccpppo..//tccaaa
Answer: [Yes]
Justification:
We adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Section G
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the datasets and models and provide the license information in Appendix [
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
32

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: N/A
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
33


Reviews:
[{"review_id": "UeuFfxboCx", "reviewer": "Reviewer_PBt5", "summary": "The paper introduces Multi-Objective One-Shot Pruning (MOSP), a novel framework for pruning Large Language Models (LLMs) that optimizes for multiple objectives (e.g., general language understanding, mathematical reasoning, code generation) simultaneously. Unlike existing one-shot pruning methods (e.g., SparseGPT, ALPS), which optimize for a single objective, MOSP generates a Pareto set of pruned models representing different trade-offs.", "questions": "Na", "limitations": "Na", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe paper is well-written.\n\nMOSP is the first to frame LLM pruning as a multi-objective optimization (MOO) problem, addressing the practical need for task-specific model customization.\n\nExtensive experiments across 5 LLMs, 3 datasets (C4, GSM8K, Code), and diverse sparsity levels demonstrate MOSP’s superiority.\n\n\n\nWeaknesses\n\nThe author should report the zero-shot or few-shot performance in the main text.\n\nAre the pruning datasets the same as those of other methods? (Wanda, SparseGPT)?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and insightful comments. We address each point below.\n\n---\n\n**Q1:** The author should report the zero-shot or few-shot performance in the main text.\n\n\n**R1:** Thank you for the suggestion. We now add a zero-shot experiment on the ARC-Challenge, ARC-Easy, and PIQA datasets as in ALPS. We use Llama-2-7B and 70\\% sparsity. The experimental settings are the same as in Section 4.1 and those used for the experiment in Table 3. Since these are general language tasks with some reasoning components, we use the preference vector $\\lambda = [0.72, 0.14, 0.14]$ (as in Table 3). The zero-shot accuracies are shown below.\n\n| Method | ARC-Challenge | ARC-Easy | PIQA |\n|----|------------|--------|----|\n| ALPS | 24.2 | 47.0 | 65.5 |\n| MOSP | **25.4** | **48.2** | **66.3** |  \n\nAs can be seen, MOSP shows better performance than ALPS. Due to time constraints during the rebuttal period, we only evaluate the above three datasets, more comprehensive zero-shot and few-shot evaluations will be performed in the final version of the paper. \n\n**Q2:** Are the pruning datasets the same as those of other methods? (Wanda, SparseGPT)?\n\n**R2:** Yes, we use exactly the same pruning datasets across all methods to ensure fair comparison.  We will further clarify it in the final version.\n\n---\n\nWe hope these responses have addressed your concerns. We are grateful for the valuable suggestions and welcome any further discussion."}, {"review_id": "T10BHGGHhB", "reviewer": "Reviewer_yqiy", "summary": "This paper introduces Multi-Objective One-Shot Pruning (MOSP), a method for efficiently reducing LLM size while considering multiple performance objectives simultaneously, such as general language understanding and math reasoning. The authors propose to generate a Pareto set of pruned LLMs with proof of convergence on the dual ADMM method. The authors validate the effectiveness of its method on various LLMs like LLama and OPT.", "questions": "none", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengthes:\n\n1. The multi-objective formulation is novel. The authors propose to treat LLM pruning as multi-objective optimization, generating Pareto sets of models with different capability trade-offs rather than single solutions\n2. The proposed method uses a shared core support as proven convergence, and enables efficient on-the-fly model generation with low computational overhead\n3. Systematic evaluation on multiple model families and sparsity levels showing consistent improvements over state-of-the-art baselines\n\n\n\n### Weaknesses:\n\n1. Only three objectives are tested. The authors are expected to experiment on different objectives.\n2. The paper only evaluates models with 7B+ parameters. Would this method work for smaller models, such as those with 3B parameters?", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive feedback and insightful comments. We address each point below.\n\n---\n\n**Q1:** Only three objectives are tested. The authors are expected to experiment on different objectives.\n\n**R1:**\nThank you for this valuable suggestion. As suggested, we now add another objective, multilingual performance using ChineseWebText 2.0, to the three-objective experiment in Section 4.3 (with Llama-2-7B and 70\\% sparsity). We use the following five preference vectors: $\\lambda^{(1)} = [1, 0, 0, 0]$, $\\lambda^{(2)} = [0, 1, 0, 0]$, $\\lambda^{(3)} = [0, 0, 1, 0]$, $\\lambda^{(4)} = [0, 0, 0, 1]$, $\\lambda^{(5)} = [0.07, 0.07, 0.07, 0.8]$. The following table shows the testing PPL. As can be seen, MOSP can again identify the best trade-off solutions. Moreover, MOSP also achieves higher HV than ALPS (92.2 vs. 76.9). \n\n\n | Method       | C4      |  GSM8K  | Code  | Chinese |\n| -------------| :-----: | :----: | :----: | :-----: |\n|Wanda|73.49 | 13.48| 11.38| 136.39|\n|SparseGPT | 30.52| 3.60 |  3.21| 16.57|\n|ALPS          |  20.57  |  3.17  |   2.67 | 9.53   |\n|MOSP($\\lambda^{(1)} = [1, 0, 0, 0]$)|**19.17**| 3.42 |     3.37 | 14.31  |\n|MOSP($\\lambda^{(2)} = [0, 1, 0, 0]$)|25.07    | **3.07**   | 3.25| 18.30  |\n|MOSP($\\lambda^{(3)} = [0, 0, 1, 0]$)|26.69    | 3.50|   **2.57** | 17.31  |\n|MOSP($\\lambda^{(4)} = [0, 0, 0, 1]$)|22.52    | 3.51   |  3.26  |**8.17**|\n|MOSP($\\lambda^{(5)} = [0.07, 0.07, 0.07, 0.8]$)|20.19    |  3.34  |   2.97  | 8.52   |\n   \nDue to time constraints in the rebuttal period, we present results for four objectives. \nIn the final version, we will include more objectives (such as medical and legal domains) in the evaluation.\n\n**Q2:** The paper only evaluates models with 7B+ parameters. Would this method work for smaller models, such as those with 3B parameters?\n\n**R2:**\nThank you for this question. Our method is indeed also effective for smaller models. As shown in Table 4 and Figure 9 (in the appendix), we have evaluated MOSP on OPT-2.7B, which demonstrates strong performance comparable to larger models. To further address your question, we now add an experiment on OPT-1.3B. With 70\\% sparsity, MOSP achieves an HV value of 274.2, outperforming ALPS (211.8). Due to rebuttal rules, we cannot include figures here but will add these figures (similar to Figure 9) to the final version for a comprehensive evaluation across model sizes.\n\n---\n\nWe hope these responses have addressed your concerns. We are grateful for the valuable suggestions and welcome any further discussion."}, {"review_id": "vZZySiEs7B", "reviewer": "Reviewer_UtLz", "summary": "The authors present a novel formulation of the one-shot pruning problem for Large Language Models (LLMs) as a Multi-Objective Optimization (MOO) task, aiming to enhance generalization across multiple downstream tasks. The core methodology comprises a Dual ADMM framework for identifying shared core support and a Task-specific ADMM procedure for personalized pruning. The paper is further strengthened by rigorous theoretical convergence analysis and comprehensive empirical evaluations.", "questions": "See above", "limitations": "See above", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThis method is more in line with the real user's differentiated preferences for multiple task requirements (e.g., language comprehension, mathematical reasoning, code generation). \n\nProposed MOSP (Multi-Objective One-Shot Pruning) framework, combining Dual ADMM and Task-specific ADMM, to achieve personalized models under user preferences while maintaining core weight sharing. \n\nTheoretical proofs and experimental results are sufficient to further support the reliability of the method. \n\nWeaknesses:\n\nThe introduction should give a slight description of the design of the dual ADMM, Task-spcific ADMM technique to make the main contributions more clear to the reader. \n\nIs the fact that W^{prime} doesn't appear in the body of Eq. 4 caused by a typo? \n\nIt is difficult to distinguish between the MOSP method and other methods in Figure 4. \n\nMethodology Phase 3: Refinement is not sufficiently described and it is suggested to expand on this in the appendix. \n\nWhile the three-stage structure of MOSP is theoretically well-motivated, it lacks an interpretable explanation. The interaction between Dual ADMM and Task-specific ADMM is not clearly articulated, the structural relationship between core support and task-specific weights remains underexplored. \n\nThe experimental evaluation is limited to two- and three-objective scenarios, leaving the scalability of MOSP to higher-dimensional task settings (e.g., four or more objectives or finer-grained task divisions) unexplored. In addition, the approach lacks exploration of real-world human preferences, e.g., if the user requires that the model's language comprehension and code ability are equally important, should the hyperparameter be assigned 0.5 to both?  \n\nThe paper does not provide unpruned baseline metrics, making it difficult to assess whether the results of the pruned model are directly usable. \n\nThe efficiency improvement of the model after pruning is suggested to provide some results for reference. \n\nThe ablation study lacks thoroughness in several aspects. (1) There is no comparison against the case where core support is entirely disabled (i.e., α = 0), making it difficult to assess its actual contribution. (2) The sensitivity of MOSP to different task combinations is not analyzed, which is important for understanding its generalization behavior. (3) The impact of varying the transformation parameter p in the λ-to-λ′ mapping is not systematically examined, leaving the effect of this design choice unclear.", "was_revised": true, "final_justification": "The rebuttal addressed my question effectively. I thus maintain my score of acceptance.", "rebuttal": "Thank you very much for your positive feedback and insightful comments. We address each point below.\n\n---\n\n**Q1:** The introduction should give a slight description of the design of the dual ADMM, Task-spcific ADMM technique to make the main contributions more clear to the reader.\n\n**R1:** \nThank you for this valuable suggestion. We will revise the introduction to provide clearer insight into our technical approach. Specifically, lines 35-37 will be revised as:\n\n\"MOSP achieves this through a multi-stage process: First, we identify a common core support of weights crucial across all tasks using Dual ADMM optimization. This involves formulating the problem as bilevel optimization with ADMM applied to both inner and outer levels, with proven convergence guarantees. Second, using the identified supports, we perform a simplified ADMM for each task separately. This decoupling of shared knowledge preservation from task-specific optimization enables efficient on-the-fly generation of specialized sparse models based on user-defined preference vectors, allowing effective exploration of the Pareto front.\"\n\n**Q2:** Is the fact that $W^{\\prime}$ doesn't appear in the body of Eq. 4 caused by a typo?\n\n**R2:** Yes, it is indeed a typo. In the inner optimization problem (4), all $\\boldsymbol{W}$ on the right-hand side should be the optimization variable $\\boldsymbol{W}'$. We will fix this in the final version. Thank you for catching this typo.\n\n**Q3:** It is difficult to distinguish between the MOSP method and other methods in Figure 4.\n\n**R3:** Figure 4 shows the Pareto fronts achieved by the proposed MOSP under different sparsity levels, and does not provide comparison with other baselines. We will add a new figure (similar to Figure 4) in the final version to compare MOSP with the baseline methods.\n\n**Q4:** Methodology Phase 3: Refinement is not sufficiently described and it is suggested to expand on this in the appendix.\n\n**R4:** Thank you for your suggestion. The refinement phase follows the standard refinement procedure in ALPS. As suggested, we will add a more complete description of it in the appendix in the final version.\n\n**Q5:** While the three-stage structure of MOSP is theoretically well-motivated, it lacks an interpretable explanation. The interaction between Dual ADMM and Task-specific ADMM is not clearly articulated, the structural relationship between core support and task-specific weights remains underexplored.\n\n**R5:** Thank you for this valuable suggestion. The multi-stage optimization disentangles the shared and task-specific knowledge through the following interactions:\n\nStage 1 (Dual ADMM): Identifies a foundational \"core support\" ($W_c$) representing weights broadly beneficial across all tasks. This captures common knowledge required for the multi-task problem and serves as a strong shared prior.\n\nStage 2 (Task-specific ADMM): Leverages the core support from Stage 1 to guide task-specific pruning. For each task, it performs a separate ADMM procedure where the core support acts as regularization, ensuring task-specific weights build upon the shared foundation while incorporating task specific adaptation.\n\nThis two-stage decomposition enables efficient Pareto front exploration, decoupling of shared knowledge\npreservation from task-specific optimization. Due the rebuttal rules, we cannot include visualization results. We will include them to the final version, and will also incorporate this explanation into the methodology section.\n\n**Q6:** The experimental evaluation is limited to two- and three-objective scenarios, leaving the scalability of MOSP to higher-dimensional task settings (e.g., four or more objectives or finer-grained task divisions) unexplored. In addition, the approach lacks exploration of real-world human preferences, e.g., if the user requires that the model's language comprehension and code ability are equally important, should the hyperparameter be assigned 0.5 to both?\n\n**R6:**\nThank you for this valuable suggestion. As suggested, we now add another objective, multilingual performance using ChineseWebText 2.0, to the three-objective experiment in Section 4.3 (with Llama-2-7B and 70\\% sparsity). We use the following five preference vectors: $\\lambda^{(1)} = [1, 0, 0, 0]$, $\\lambda^{(2)} = [0, 1, 0, 0]$, $\\lambda^{(3)} = [0, 0, 1, 0]$, $\\lambda^{(4)} = [0, 0, 0, 1]$, $\\lambda^{(5)} = [0.07, 0.07, 0.07, 0.80]$. The following table shows the testing PPL. As can be seen, MOSP can again identify the best trade-off solutions. Moreover, MOSP also achieves higher HV than ALPS (92.2 vs. 76.9). \n\n | Method       | C4      |  GSM8K  | Code  | Chinese |\n| -------------| :-----: | :----: | :----: | :-----: |\n|Wanda|73.49 | 13.48| 11.38| 136.39|\n|SparseGPT | 30.52| 3.60 |  3.21| 16.57|\n|ALPS          |  20.57  |  3.17  |   2.67 | 9.53   |\n|MOSP($\\lambda^{(1)} = [1, 0, 0, 0]$)|**19.17**| 3.42 |     3.37 | 14.31  |\n|MOSP($\\lambda^{(2)} = [0, 1, 0, 0]$)|25.07    | **3.07**   | 3.25| 18.30  |\n|MOSP($\\lambda^{(3)} = [0, 0, 1, 0]$)|26.69    | 3.50|   **2.57** | 17.31  |\n|MOSP($\\lambda^{(4)} = [0, 0, 0, 1]$)|22.52    | 3.51   |  3.26  |**8.17**|\n|MOSP($\\lambda^{(5)} = [0.07, 0.07, 0.07, 0.80]$)|20.19    |  3.34  |   2.97  | 8.52   |\n   \nDue to time constraints in the rebuttal period, we present results for four objectives. In the final version, we will include more objectives (such as medical and legal domains) in the evaluation.\n\nRegarding human preferences, your interpretation is correct: the preference vector $\\lambda$ directly reflects user-defined priorities. In other words, with only two objectives (language comprehension and coding) that are equally important, the user can set $\\lambda = [0.5, 0.5]$. When language comprehension is more important, the user might use $\\lambda = [0.8, 0.2]$. We will clarify this in the revised version.\n\n**Q7:** The paper does not provide unpruned baseline metrics, making it difficult to assess whether the results of the pruned model are directly usable.\n\n**R7:** Thank you for this suggestion. In response to your suggestion, we compare the proposed method with unpruned baseline on two dataset case (i.e., C4 and GSM8K). The unpruned Llama-2-7B baseline achieves test PPL of 6.97 on C4 and 2.73 on GSM8K, while our model pruned to 50\\% sparsity achieves test PPL of 8.14 and 2.78, respectively.\nWhile some performance degradation is expected for pruned models (especially at high sparsity levels), our results show that MOSP provides superior Pareto-optimal solutions compared to existing methods, as shown in Section 4.2 and 4.3. We will inlcude the unpruned baseline results in the final version.\n\n**Q8:** The efficiency improvement of the model after pruning is suggested to provide some results for reference.\n\n**R8:** Thank you for this suggestion. For 2:4 sparsity, we observe ~1.5x speedup for linear layers on NVIDIA A6000. Due to the limited rebuttal period, we cannot provide timing results on other unstructured sparsity. As reference, SparseGPT [1] reports ~2.16x speedup at 60\\% sparsity using the DeepSparse engine.  We will include more efficiency benchmarks in the final version.\n\n[1] Frantar & Alistarh, SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot, ICML 2023\n\n**Q9:** The ablation study lacks thoroughness in several aspects. (1) There is no comparison against the case where core support is entirely disabled (i.e., $\\alpha$ = 0), making it difficult to assess its actual contribution. (2) The sensitivity of MOSP to different task combinations is not analyzed, which is important for understanding its generalization behavior. (3) The impact of varying the transformation parameter p in the $\\lambda$-to-$\\lambda'$ mapping is not systematically examined, leaving the effect of this design choice unclear.\n\n**R9:** Thank you for your comments. For the case where core support is entirely disabled (i.e., $\\alpha$ = 0), indeed we have reported this experiment in Section 4.4 and Figure 6(a). As can be seen, setting $\\alpha=0$ results in substantial performance degradation, confirming its critical contribution.\n\nFor the sensitivity of MOSP to different task combinations, first, we have added an experiment with 4-objectives as mentioned in our reply **R6** above. Besides, we also experiment with the following 2-task combinations: (C4, Code) and (Code, Chinese). Empirically, MOSP successfully identifies the Pareto sets for these two combinations. However, due to rebuttal rules, we cannot upload the corresponding figures here, but they will be included in the final version.\n\nFor the transformation parameter $p$, we have performed an ablation study in Section 4.4. As shown in Figure 6(b), models with extreme preferences (e.g., [1.0, 0]) perform similarly to those without transformation, while other models generally benefit from applying the transformation. During the rebuttal period, we have also examined two more $p$ values: $p=0.1$ and $p=1$. Due to rebuttal rules, we cannot upload the figure here, but it will be added to the final version.\n\n---\n\nWe hope these responses have addressed your concerns. We are grateful for the valuable suggestions and welcome any further discussion."}, {"review_id": "eEigCFKQ9k", "reviewer": "Reviewer_uTBb", "summary": "LLMs are powerful but computationally expensive, and traditional one-shot pruning methods fail to account for their diverse applications. This paper proposes Multi-Objective One-Shot Pruning (MOSP), which frames pruning as a multi-objective optimization problem, producing a Pareto set of pruned models that balance different task performances.", "questions": "See Weaknesses", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 2, "originality": 4, "strengths_and_weaknesses": "# Strengths\n- The manuscript is well written and easy to follow.\n- Extensive experiments are conducted.\n- Methods are well-defined and theoretically sound.\n\n# Weaknesses\n- My main concern is that the motivation of Multi-objective pruning is not fully justified. If \"generality\" is the key purpose, the necessity of the Pareto set of models is rather weak; on the other hand, if \"specialization\" is the goal, optimizing for a specific task will be better.\n- That being said, in Figure 3, MOSP's overall performance gain over ALPS is marginal. Again, if we want a model that performs task A better, one may argue that it is better to optimize solely on A, rather than select a model from the Pareto set that performs task A a bit better than task B. \n- Related to this point, it would be helpful to compare MOSP and other baselines against single-objective optimization results. This is not to say that MOSP should outperform the single-objective approaches; the single-objective results will reveal the upper bound for each task, which would be very informative for future research.\n- Furthermore, this raises a fundamental question: to what extent should we consider two tasks as \"distinct\"? Are GSM8K and C4 truly heterogeneous tasks, and why should they form a Pareto frontier? -- Maybe, improving the Pareto frontier could have been a more valuable discussion.\n- (minor) L154 : (3) --> (iii)", "was_revised": true, "final_justification": "Experiments and responses are done well, which clarified their motivation of their work.\nOverall, I think the paper is well written and I will keep my score advocating acceptance.", "rebuttal": "Thank you very much for your positive feedback and insightful comments. We address each point below.\n\n---\n\n**Q1:** My main concern is that the motivation of Multi-objective pruning is not fully justified. If \"generality\" is the key purpose, the necessity of the Pareto set of models is rather weak; on the other hand, if \"specialization\" is the goal, optimizing for a specific task will be better.\n\n**R1:**\nThank you for your insightful comment. Our primary motivation is to serve the diverse range of user preferences that exist between the extremes of \"pure generality\" and various \"pure specializations.\" For instance, different deployments of a model on edge devices may require different balances between capabilities: One user might prioritize language understanding (60\\% importance) over coding (30\\%) and math (10\\%), while another may require strong mathematical reasoning (80\\% importance). \n\nAs the number of objectives increases, these preference combinations grow exponentially. MOSP efficiently addresses this by obtaining a set of trade-off models in a  **single pruning run**. This allows users to select the best-fit model for their specific needs post-training, **avoiding the significant computational cost of re-pruning for every new preference**. For instance, handling 100 distinct user preferences with traditional methods would take over 100 hours, whereas MOSP accomplishes this in just 1.36 hours. This approach also uniquely allows for dynamic preference adjustments. We will clarify this motivation more explicitly in the revised version.\n\n**Q2:** That being said, in Figure 3, MOSP's overall performance gain over ALPS is marginal. Again, if we want a model that performs task A better, one may argue that it is better to optimize solely on A, rather than select a model from the Pareto set that performs task A a bit better than task B.\n\n**R2:** \nThanks for your comment. We believe the benefits of MOSP are significant. Recall that, as mentioned above, different users may have different preferences. Hence, as shown in Figure 3, when the user puts high priority on C4 (i.e., $\\lambda = [1, 0]$), the test PPL of MOSP is 17.25 vs 18.17 of ALPS. This advantage increases with more objectives. For example, in the three-objective experiment\n(Table 3), when the user puts high priority on C4, the test PPL of MOSP is 18.80 vs 20.44 of ALPS. This advantage also increases with higher sparsity levels. For example, with 80\\% sparsity, when the user puts high priority on C4, the test PPL of MOSP is 47.94 vs 53.88 of ALPS.\n\nRegarding optimize solely on A, we refer back to our reply **R1**: our method is designed for scenarios with diverse user preferences, many of which fall between rather than at the extremes. More specifically, as shown in our response **R3** below, optimizing solely for task A severely degrades performance on the other tasks, which is typically undesirable. Users generally prefer solutions that maintain reasonable performance across all tasks while emphasizing their priorities. We will clarify this in the revised version.\n\n**Q3:** Related to this point, it would be helpful to compare MOSP and other baselines against single-objective optimization results. This is not to say that MOSP should outperform the single-objective approaches; the single-objective results will reveal the upper bound for each task, which would be very informative for future research.\n\n**R3:** Thank you for this valuable suggestion. As suggested, we now add an experiment to compare MOSP against single-objective optimization using ALPS (i.e., running ALPS on each dataset separately). Note that this approach requires three separate pruning runs and cannot directly produce trade-off solutions by merging. The following table shows the test PPLs. For MOSP, we use four representative preferences.\n\n| Method                       | C4 | GSM8K | Code  |\n| ---------------------------- | :--------: | :----------: | :----------: |\n|ALPS (optimized on C4 only) |17.66 | 18.71| 5.00|\n|ALPS (optimized on GSM8K only) |39.63|2.96|7.85|\n| ALPS (optimized on Code  only) |44.88|4.63|2.43|\n| ALPS (optimized on all 3 objectives) |20.44|2.63|3.12 |\n| MOSP $\\lambda = [1, 0, 0]$  | 18.80 | 3.37 | 3.20   |\n| MOSP $\\lambda = [0, 1, 0]$  | 24.33 | 3.05 | 3.06|\n| MOSP $\\lambda = [0, 0, 1]$ | 25.82 | 3.43 | 2.53  |\n|MOSP $\\lambda = [0.72, 0.14, 0.14]$| 19.30 | 3.20 | 2.79|\n\nAs can be seen, single-objective ALPS achieves optimal performance on its target task but shows significant degradation on others. ALPS (optimized on all 3 objectives) provides more balanced performance but offers only one single pruned model for all possible user preferences. MOSP enables users to select from various reasonable trade-offs, maintaining acceptable performance across all tasks while allowing preference-based customization.\n\n**Q4:** Furthermore, this raises a fundamental question: to what extent should we consider two tasks as \"distinct\"? Are GSM8K and C4 truly heterogeneous tasks, and why should they form a Pareto frontier? -- Maybe, improving the Pareto frontier could have been a more valuable discussion.\n\n**R4:** Thank you for this question. In multi-objective optimization (MOO), the various objectives (tasks in our context) do not need to be fully distinct. Indeed, in MOO, one is more concerned about how to balance the conflicts from the different objectives. If there is no conflict among the tasks, then all of them can be improved simultaneously. When there are conflicts (as is typically the case, even when the tasks are not completely distinct), improving one may lead to performance degradation of the other, and MOO addresses the problem of finding a good trade-off solution. Our experimental results also demonstrate this issue. As shown in the table above, the model jointly pruned using ALPS performs worse on each task compared to the models pruned individually for those tasks using ALPS. This performance degradation confirms that a trade-off exists between these objectives.  We will incorporate this clarification into the final version.\n\n**Q5:** (minor) L154 : (3) --> (iii)\n\n**R5:** Thank you for pointing this out. We will correct it and conduct a thorough proofread of the manuscript.\n\nOverall, we emphasize that one of MOSP's key contribution lies in obtaining a  set of \ntrade-off solutions through a **single** efficient pruning run. These solutions effectively address the diverse range of user preferences between extremes, providing flexibility that traditional approaches cannot.\n\n---\n\nWe hope these responses have addressed your concerns. We are grateful for the valuable suggestions and welcome any further discussion."}]

Author rebuttal:

Meta-review: This paper introduces Multi-Objective One-Shot Pruning (MOSP), a framework for pruning LLMs by casting the problem as a multi-objective optimization task. Unlike existing one-shot pruning methods that focus on a single task, MOSP generates a Pareto set of models balancing multiple objectives (e.g., language understanding, reasoning, code), supported by a dual and task-specific ADMM framework with convergence guarantees. It provides a novel formulation and theoretical grounding of the proposed method, with systematic empirical evaluation. Reviewers also appreciated the clear motivation for handling diverse user preferences and the efficiency of generating multiple models in a single run.

However, some issues do exist: limited evaluation on small models and higher-dimensional objectives, and some clarity issues in the methodology description. The authors added more experimental results during the rebuttal, addressed most reviewer concerns. Therefore, despite some limitations, the balance of strengths and thorough rebuttal responses justify acceptance.
Decision: Accept (poster)