Towards Efficient Adaptation of Pruning Strategy in Large Language Models

Authors: Shuqi LIU, Bowei He, Han Wu, Linqi Song
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Model Pruning, Large Language Model

Abstract:
Post-training pruning has gained increasing attention with the rapid growth of large language models (LLMs). However, significant variations in weight distributions across different LLMs make a fixed pruning strategy inadequate for multiple models. In this paper, we propose an efficient evolutionary optimization framework, \textbf{Mecon}, for adaptive LLM pruning. In particular, we design an effective search space built on our \textbf{Me}ta pruning metric to mitigate diverse weight distributions among LLMs. We then introduce model-wise re\textbf{con}struction error, a lightweight search evaluation to speed up the evaluation of each search trial. We finally leverage Non-dominated Sorting Genetic Algorithm III (NSGA-III) as our search algorithm, handling both the single-objective problem of pruning metric search and the multi-objective problem of layerwise sparsity ratio search in discovering the optimal pruning strategy. We extensively evaluate our framework on LLaMA-1/2/3 and Mistral models across multiple benchmarks. Our results demonstrate that our adaptive pruning metrics consistently outperform existing ones, and the layerwise sparsity ratios improve the effectiveness of other pruning metrics. Furthermore, we validate the cross-task and cross-model generalizability of our pruning metrics, offering a cost-effective solution to streamline the search process. We release our code in the anonymous repository: \textcolor{blue}{\url{https://anonymous.4open.science/r/Mecon-5819}}.

Decision: reject
Reviewer scores: 6, 8, 8, 3, 3

Reviews and discussion:
<thread>
creation date:
1730001190636

thread_type:
official_review

<thread_object>
creator: reviewer_mjmt

<content>
summary:
1. The paper presents MECON, an evolutionary optimization framework designed for adaptive pruning of large language models (LLMs). Recognizing that fixed pruning strategies are ineffective due to the diverse weight distributions across LLMs, the authors develop a Meta pruning metric and an efficient search space to address this challenge. 

2. The framework utilizes model-wise reconstruction error as a fast evaluation method and employs the NSGA-III algorithm to optimize both single-objective and multi-objective pruning problems. Extensive experiments on various LLaMA and Mistral models show that MECON's adaptive pruning approach outperforms existing methods, improves pruning efficiency, and demonstrates cross-task and cross-model generalizability.

strengths:
1. The paper is clearly written and free of obvious typos.
2. The proposed evolutionary computation-based algorithm for searching hyperparameters in large model pruning is tested on well-known models such as LLaMA-1/2/3 and Mistral, as well as on datasets like WikiText, GSM8K, and MMLU.
3. The authors conduct an ablation study on the evolutionary computation algorithm used, attempting to justify the choice of Non-dominated Sorting Genetic Algorithm III (NSGA-III) as their search algorithm.

weaknesses:
1.  The most critical issue with this paper is the lack of testing for the algorithm's efficiency. The authors only evaluate the accuracy under unstructured and semi-structured (N: M) pruning with a 50% mask compression rate across various datasets but do not provide results on memory efficiency or latency in GPU-CPU collaborative scenarios. Without these efficiency metrics, especially for unstructured pruning, it is unclear whether the method can be applied in real-world scenarios.

2. The paper only tests and compares baselines at a 50% compression rate and does not explore other extreme compression rates, such as below 50%.

3. The datasets used to validate the algorithm are limited, as testing is only conducted on GSM8K, MMLU, and WikiText. It would be beneficial to evaluate the model's generalizability on a broader range of datasets, such as those from the lm-evaluation-harness library.

4. The performance improvement of the algorithm appears marginal on some datasets and models. For instance, in Table-2, MECON achieves a score of 63.51 on the Mistral model, while the simpler Magnitude pruning achieves 63.34. This minor gain does not justify the significantly higher time and complexity involved in MECON’s search and pruning process compared to straightforward methods like Magnitude pruning.

5. Some relevant works on compression of LLMs using search algorithms/NAS methods are not mentioned or compared in the paper, such as: 
[1] Pruning Large Language Models via Accuracy Predictor.
[2] Tune As You Scale: Hyperparameter Optimization For Compute Efficient Training.

questions:
1. Can the authors conduct efficiency testing experiments?

2. Can the authors perform experiments at other compression rates?

3. Can the authors provide additional dataset evaluations to demonstrate the algorithm’s generalizability? Given that post-training algorithms typically do not consume significant computational resources or time, this should be feasible.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730362301484

thread_type:
official_review

<thread_object>
creator: reviewer_kwku

<content>
summary:
To address the issue of fixed pruning strategies being unable to adapt to multiple different models, the authors propose an adaptive evolutionary framework for LLMs pruning. Specifically, they introduce designated search spaces for the pruning strategy and layerwise sparsity ratios, aiming to find an optimal pruning method based on model reconstruction loss and sparsity ratio discrepancy. The search algorithm used is the Non-dominated Sorting Genetic Algorithm III (NSGA-III). Additionally, the authors conducted extensive experiments to demonstrate the effectiveness of the proposed method.

strengths:
1 . The paper designed an effective search space built on our meta pruning metric to mitigate diverse weight distributions among LLMs.
2. The authors leverage Non-dominated Sorting Genetic Algorithm III (NSGA-III) as the search algorithm.

weaknesses:
1. The motivation of this paper is unclear. Although the authors mentioned that the existing fixed pruning methods cannot adapt to models with different weight distributions (such as Llama-2 and Llama-3), they did not explain the underlying reasons for this. Specifically, Figure 1 only shows that different pruning methods have different performance on models with different weight distributions, but does not analyze the specific reasons from a theoretical perspective.
2. I believe the overhead introduced by the search algorithm in the proposed method is non-negligible.  the author compressed the search time to an acceptable range by restricting the search space to a limited discrete space.
3. This paper lacks innovation and is a combination of existing work. The proposed work and Pruner-Zero[1] are not highly distinguishable, and the extended search pruning rate is also the existing work of CV.

[1] Pruner-Zero: Evolving Symbolic Pruning Metric from Scratch for Large Language Models

questions:
1. Why didn't the authors conduct experiments on models other than LLaMA and Mistral?
2. Did the authors consider a more general representation to model the relationship between weights and activations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730430834207

thread_type:
official_review

<thread_object>
creator: reviewer_av8q

<content>
summary:
The author proposed a meta-pruning algorithm that uses evolutionary search algorithm to automatically find the right combination of pruning heuristics and sparsity ratio. This work is very novel, the experiment protocol seems rigorous and the results look solid.

strengths:
- Casting the LLM pruning problem as a meta problem is novel and thought-provoking.
- Very strong evaluation protocol — this work includes comprehensive task evaluation plus end-to-end speedup evaluation.
- Paper is well written and easy to follow.

weaknesses:
- No obvious weakness.

questions:
- What are the most important pruning metrics? I’m looking for a high-level interpretation of your A.2 results.
- As a follow-up, wonder if the optimal searched pruning configuration can shed any light on why SparseGPT/Wanda performed so poorly on some models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730545337985

thread_type:
official_review

<thread_object>
creator: reviewer_rxqx

<content>
summary:
This paper presents an adaptive and efficient pruning method for large language models (LLMs) within the framework of post-training pruning. It includes a meta pruning metric search, layerwise sparsity ratio search, and introduces a new search evaluation metric: model-wise reconstruction error. The authors first validate and highlight the suboptimal generalization performance of existing methods across different models, analyzing the underlying causes. They propose an adaptive pruning strategy tailored for various LLMs, optimizing both the pruning metric and the layerwise sparsity ratios.

In response to the time-consuming nature and poor task generalization of using perplexity as the evaluation measure under the SEARCH EVALUATION, the authors introduce the model-wise reconstruction error metric. This metric directly measures the difference between the output layers before and after pruning using the Frobenius norm, providing a more direct and effective assessment. Furthermore, the paper employs a unified NSGA-III algorithm to efficiently address both single and multi-objective search problems across the two phases.

In the experimental section, the study conducts comprehensive testing to verify the robustness and superiority of the proposed method under different models, tasks, and parameter requirements, accompanied by an in-depth analysis.

strengths:
1. The scientific problem addressed in this paper is highly significant. The development of efficient and robust pruning methods within the post-training paradigm has broad applications in the AI community. The authors provide a rigorous validation and analysis of the issues at hand.

2. The methodology presented in this paper is comprehensive, including well-designed solutions to key scientific questions such as pruning metric search, subsequent layerwise sparsity ratio search, and optimization of evaluation metrics. The experiments are solid, the logic is clear, and the paper progressively deepens the discussion, rigorously demonstrating advancements in robustness and effectiveness.

3. The writing is logically structured and clearly articulated, effectively presenting the logical flow of the methodology, the importance of the experiments, and the distinctions and improvements compared to related works in the field.

weaknesses:
1. Regarding the insufficient generalization performance of current methods across different models, while the pruning metric search presented in this paper shows promising improvements based on experimental results, I find the design of the meta metric and the candidate options in the search space somewhat confusing. Furthermore, I did not see the underlying logic and theoretical support for the pruning metric search.

2. According to the framework of this paper, the pruning metrics from similar methods can be viewed as a special case of the proposed meta metric. I am curious about the resource consumption-effectiveness curve when searching within a potentially smaller search space under similar methods with specific coefficients (α, β). It raises the question of whether simple coefficient tuning within this method could yield satisfactory results.

questions:
As mentioned in the above weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730755009993

thread_type:
official_review

<thread_object>
creator: reviewer_aamd

<content>
summary:
This paper argues there is significant variations in weight distributions across different LLMs, which make it unable to use a fixed pruning strategy for multiple models. Author further propose their framework which first search a pruning metric based on evolutionary optimization and then use Non-dominated Sorting Genetic Algorithm III which search layerwise sparsity ratio to find optimal pruning strategy. Experiments on 4 7B-level models show great improvements over related works.

strengths:
1. This paper provides insights on diverse wieght distribution of different models which would cause unstable performance for one fixed pruning strategy. And further propose their method for adaptive LLM pruning.
2. Experiments are quite solid with analysis, four 7B models show notable improvements over baselines on 9 tasks.

weaknesses:
1. It is better to provide reference for Non-dominated Sorting Genetic Algorithm III in introduction (e.g. line 76)

questions:
1. Is there any insights on why this method show great improvments on 7B level models but negligible improvments on 30B/70B model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732522817591

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to further discussions

comment:
Dear Reviewers,

We sincerely appreciate your time to review our submission and provide valuable comments. We have carefully considered all of your concerns and tried to resolve them in our rebuttal. Your constructive feedback will greatly help use improve the quality of the work.

As the deadline of discussion period is apporaching, we would really appreciate it if you have read our response and let us konw whether the previous responses have addressed your concerns accordingly. If your concerns have not been well resolved, could you please let us know your remaining concerns so that we have the opportunity to respond before the deadline? We are happy to have any follow-up discussions.

We understand you are very busy and we really appreciate your time. Looking forward to your further comments and discussions.

Best wishes,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces an evolutionary search framework that automatically finds pruning metrics and layer-wise sparsity rates for LLMs. The approach is based on the idea that fixed pruning strategies don't work well for different models. Experiments across multiple models and tasks indicate consistent performance improvements over baselines. 

In general, all reviewers agreed that the experiments and evaluations were thorough and the writing was clear. Reviewers aamd, rXqX, and Av8Q acknowledge the importance of the proposed problem. 

However, some concerns were raised. Reviewer rXqX raised questions about the design choices and wondered if the search space could be further simplified. Reviewer kWKU asked about experiments on broader datasets and models. Reviewer Av8Q raised questions about further analysis on which metrics are suitable for different models. Reviewer Mjm pointed out a lack of efficiency evaluation. 

During the rebuttal, the authors made great efforts to resolve these issues. They conducted abundant experiments, such as additional experiments on OPT models, simplified search space, and efficiency comparison. They confirmed that it's important to explore both coefficients and operations and explained how their metrics apply to different LLaMA models.

However, some concerns have not been addressed after the rebuttal. For example, reviewer kWKU pointed out a similar work Pruner-Zero, and raised concerns about the limited technical innovation. After carefully reviewing the paper, the reviewers' comments, the rebuttal discussion, and the authors' message, the AC recommended rejecting the paper. Nonetheless, AC suggests authors include additional experiments and discussions in the rebuttal to improve the paper’s quality in its future version.