Title: Multi-Objective One-Shot Pruning for Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Objective Optimization

Pruning

Large Language Models
Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks but require substantial computational resources, limiting their deployment in resource-constrained environments. While one-shot pruning methods can reduce model size without expensive retraining, they typically optimize for single objectives, ignoring LLMs' multi-faceted applications. We introduce Multi-Objective One-Shot Pruning (MOSP), which formulates LLM pruning as a multi-objective optimization problem. MOSP efficiently generates a Pareto set of pruned models representing different capability trade-offs, allowing users to select solutions aligned with their preferences. The proposed approach identifies share core support while enabling specialized support. Experiments across various LLMs and sparsity levels demonstrate MOSP's superior performance in navigating multi-objective trade-offs compared to baseline methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nLLMs are powerful but computationally expensive, and traditional one-shot pruning methods fail to account for their diverse applications. This paper proposes Multi-Objective One-Shot Pruning (MOSP), which frames pruning as a multi-objective optimization problem, producing a Pareto set of pruned models that balance different task performances.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- The manuscript is well written and easy to follow.\n- Extensive experiments are conducted.\n- Methods are well-defined and theoretically sound.\n\n# Weaknesses\n- My main concern is that the motivation of Multi-objective pruning is not fully justified. If \"generality\" is the key purpose, the necessity of the Pareto set of models is rather weak; on the other hand, if \"specialization\" is the goal, optimizing for a specific task will be better.\n- That being said, in Figure 3, MOSP's overall performance gain over ALPS is marginal. Again, if we want a model that performs task A better, one may argue that it is better to optimize solely on A, rather than select a model from the Pareto set that performs task A a bit better than task B. \n- Related to this point, it would be helpful to compare MOSP and other baselines against single-objective optimization results. This is not to say that MOSP should outperform the single-objective approaches; the single-objective results will reveal the upper bound for each task, which would be very informative for future research.\n- Furthermore, this raises a fundamental question: to what extent should we consider two tasks as \"distinct\"? Are GSM8K and C4 truly heterogeneous tasks, and why should they form a Pareto frontier? -- Maybe, improving the Pareto frontier could have been a more valuable discussion.\n- (minor) L154 : (3) --> (iii)\n\nQUESTIONS:\nSee Weaknesses\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a novel formulation of the one-shot pruning problem for Large Language Models (LLMs) as a Multi-Objective Optimization (MOO) task, aiming to enhance generalization across multiple downstream tasks. The core methodology comprises a Dual ADMM framework for identifying shared core support and a Task-specific ADMM procedure for personalized pruning. The paper is further strengthened by rigorous theoretical convergence analysis and comprehensive empirical evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThis method is more in line with the real user's differentiated preferences for multiple task requirements (e.g., language comprehension, mathematical reasoning, code generation). \n\nProposed MOSP (Multi-Objective One-Shot Pruning) framework, combining Dual ADMM and Task-specific ADMM, to achieve personalized models under user preferences while maintaining core weight sharing. \n\nTheoretical proofs and experimental results are sufficient to further support the reliability of the method. \n\nWeaknesses:\n\nThe introduction should give a slight description of the design of the dual ADMM, Task-spcific ADMM technique to make the main contributions more clear to the reader. \n\nIs the fact that W^{prime} doesn't appear in the body of Eq. 4 caused by a typo? \n\nIt is difficult to distinguish between the MOSP method and other methods in Figure 4. \n\nMethodology Phase 3: Refinement is not sufficiently described and it is suggested to expand on this in the appendix. \n\nWhile the three-stage structure of MOSP is theoretically well-motivated, it lacks an interpretable explanation. The interaction between Dual ADMM and Task-specific ADMM is not clearly articulated, the structural relationship between core support and task-specific weights remains underexplored. \n\nThe experimental evaluation is limited to two- and three-objective scenarios, leaving the scalability of MOSP to higher-dimensional task settings (e.g., four or more objectives or finer-grained task divisions) unexplored. In addition, the approach lacks exploration of real-world human preferences, e.g., if the user requires that the model's language comprehension and code ability are equally important, should the hyperparameter be assigned 0.5 to both?  \n\nThe paper does not provide unpruned baseline metrics, making it difficult to assess whether the results of the pruned model are directly usable. \n\nThe efficiency improvement of the model after pruning is suggested to provide some results for reference. \n\nThe ablation study lacks thoroughness in several aspects. (1) There is no comparison against the case where core support is entirely disabled (i.e., α = 0), making it difficult to assess its actual contribution. (2) The sensitivity of MOSP to different task combinations is not analyzed, which is important for understanding its generalization behavior. (3) The impact of varying the transformation parameter p in the λ-to-λ′ mapping is not systematically examined, leaving the effect of this design choice unclear.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces Multi-Objective One-Shot Pruning (MOSP), a method for efficiently reducing LLM size while considering multiple performance objectives simultaneously, such as general language understanding and math reasoning. The authors propose to generate a Pareto set of pruned LLMs with proof of convergence on the dual ADMM method. The authors validate the effectiveness of its method on various LLMs like LLama and OPT.

STRENGTHS AND WEAKNESSES:
### Strengthes:

1. The multi-objective formulation is novel. The authors propose to treat LLM pruning as multi-objective optimization, generating Pareto sets of models with different capability trade-offs rather than single solutions
2. The proposed method uses a shared core support as proven convergence, and enables efficient on-the-fly model generation with low computational overhead
3. Systematic evaluation on multiple model families and sparsity levels showing consistent improvements over state-of-the-art baselines



### Weaknesses:

1. Only three objectives are tested. The authors are expected to experiment on different objectives.
2. The paper only evaluates models with 7B+ parameters. Would this method work for smaller models, such as those with 3B parameters?

QUESTIONS:
none

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05

comments: SUMMARY:
The paper introduces Multi-Objective One-Shot Pruning (MOSP), a novel framework for pruning Large Language Models (LLMs) that optimizes for multiple objectives (e.g., general language understanding, mathematical reasoning, code generation) simultaneously. Unlike existing one-shot pruning methods (e.g., SparseGPT, ALPS), which optimize for a single objective, MOSP generates a Pareto set of pruned models representing different trade-offs.

STRENGTHS AND WEAKNESSES:
Strengths

The paper is well-written.

MOSP is the first to frame LLM pruning as a multi-objective optimization (MOO) problem, addressing the practical need for task-specific model customization.

Extensive experiments across 5 LLMs, 3 datasets (C4, GSM8K, Code), and diverse sparsity levels demonstrate MOSP’s superiority.



Weaknesses

The author should report the zero-shot or few-shot performance in the main text.

Are the pruning datasets the same as those of other methods? (Wanda, SparseGPT)?

QUESTIONS:
Na

LIMITATIONS:
Na
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-10