EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary Search

Authors: Oliver Sieberling, Denis Kuznedelev, Eldar Kurtic, Dan Alistarh
Primary area: foundation or frontier models, including LLMs
Keywords: large language models, compression, evolutionary algorithms, quantization, pruning

Abstract:
The high computational costs of large language models (LLMs) have led to
a flurry of research on LLM compression, via methods such as quantization,
sparsification, or structured pruning. A new frontier in this area is given by
dynamic, non-uniform compression methods, which adjust the compression
levels (e.g., sparsity) per-block or even per-layer in order to minimize accuracy
loss, while guaranteeing a global compression threshold. Yet, current methods
rely on heuristics for identifying the “importance” of a given layer towards the
loss, based on assumptions such as error monotonicity, i.e. that the end-to-end
model compression error is proportional to the sum of layer-wise errors. In this
paper, we revisit this area, and propose a new and general approach for dynamic
compression that is provably optimal in a given input range. We begin from
the motivating observation that, in general, error monotonicity does not hold for
LLMs: compressed models with lower sum of per-layer errors can perform worse
than models with higher error sums. To address this, we propose a new general
evolutionary framework for dynamic LLM compression called EvoPress, which
has provable convergence, low sample and evaluation complexity. We show that
these theoretical guarantees lead to highly competitive practical performance
for dynamic compression of Llama, Mistral and Phi models: via EvoPress, we
set new state-of-the-art results for structural pruning (block/layer dropping),
unstructured sparsity, as well as quantization with dynamic bitwidths.

Decision: reject
Reviewer scores: 8, 8, 6, 3

Reviews and discussion:
<thread>
creation date:
1730661454727

thread_type:
official_review

<thread_object>
creator: reviewer_x4pg

<content>
summary:
This paper proposes a new evolutionary framework for dynamic LLM compression.

strengths:
This paper proposes an evolutionary algorithm within an optimization framework, for sparse structure selection for LLMs.

weaknesses:
1. The theoretical justification in Theorem 1 appears irrelevant to the algorithm proposed in the paper. Theorem 1 is established for linear fitness functions; however, it’s difficult to see how this applies, as LLMs likely do not have linear fitness functions.

2.  The comparison lacks systematic rigor. For instance, Table 2 presents results only at the sparsity level 70%. What about results at other sparsity levels?

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664576422

thread_type:
official_review

<thread_object>
creator: reviewer_wuqk

<content>
summary:
This work proposes an algorithm to perform dynamic compression (different layers can be compressed to various levels) of large language models. Previous works assign importance to each layer and assume end-to-end model compression error is proportional to the sum of layer-wise errors. This work observes that error monotonicity does not hold for LLM, and proposes an evolutionary search algorithm to achieve better compression quality.

In EvoPress, multiple heuristics, including mutation operation and the selection step are proposed to achieve efficient sampling for LLM. Experimental results show better compression quality compared to previous work on multiple applications.

strengths:
LLM compression is an important topic. This work proposes a new optimization algorithm, and better performance has been presented on multiple applications, including pruning, introducing sparsity, and quantization.

weaknesses:
1. It would be good to introduce an algorithm overview figure to illustrate the steps in EvoPress. In addition, the algorithm contains mulitple heuristics to make the sampling efficient. It would be good to also summarize these heuristics in the figure.
2. It's not clear whether the hyperparameters used in the heuristics (mutation operation & multi-step selection) can transfer well across tasks/datasets/models. Further illustration would be very useful.

questions:
Please see above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679554438

thread_type:
official_review

<thread_object>
creator: reviewer_jzst

<content>
summary:
The paper introduces a meta-heuristic for non-uniform model compression that is largely independent of specific model architectures and compression methods. The method is demonstrated on three compression approaches—depth pruning, unstructured sparsity, and quantization—with ample numerical results and a supporting analytical convergence analysis.

strengths:
The strength of the approach is its simplicity and the fact that it is somewhat agnostic to model architecture and compression techniques.
While it doesn’t introduce entirely novel algorithms, its design effectively minimizes computational costs when working with larger models. Overall, the paper is quite well-written and presents compelling arguments for the proposed approach.

weaknesses:
The manuscript does not, however, discuss much about the scalability of the approach in more complex scenarios, and to what extent the evolutionary search is useful compared to more basic derivative-free optimization techniques or the like.
While presented as evolutionary, the algorithm leans heavily on an elitist, exploitation-focused strategy. This approach could limit exploration, potentially trapping the algorithm in local minima and risk of poor generalization in more complex compression scenarios.

This approach appears to be effective as long as the manifold remains relatively convex. However, with multi-modal compression techniques or non-convex landscapes, there are concerns about whether this computationally efficient method will generalize well beyond the controlled benchmarking environments, especially when handling mixed or complex compression strategies.

questions:
1) The scheme is presented as an evolutionary search, but the update mechanism uses an elitist strategy (focused on exploitation) based on single-offspring selection. The number of mutations is also kept minimal, with one mutation shown to be nearly optimal, as demonstrated in SI B. In this regime, the scheme resembles a form of random coordinate descent or derivative-free optimization, with the added twist that perturbation occurs on a manifold with constant overall compression (referred to as switching). Could you clarify which properties of an evolutionary algorithm are essential to the results achieved, as opposed to a simpler perturbative approach?

2) In the current setting, how much variability is there in the optimal compression profile found after one run of the evolutionary search? Does it appear the search get stuck in local minima in any of the benchmarks? 

3) Can you predict if the hypothesis that smooth dynamic model compression results in a smooth fitness landscape with few local optima still hold in multi-modal cases? 

Additional Questions/comments:

4) How is the step size in a single dimension determined (e.g., the 1M weights for unstructured sparsity)? Is the step size fixed or adaptive?

5) In the selection process, only two stages are shown (see SI B.2). Was this approach tested with more stages, and if so, were additional stages found to be ineffective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730709318434

thread_type:
official_review

<thread_object>
creator: reviewer_yr9o

<content>
summary:
This paper introduces a new general framework for dynamic LLM compression. Building on the observation that error monotonicity does not generally hold for LLM compression, the paper proposed evolutionary search approach. The proposed approach is provably convergent and has low sample and evaluation complexity when the fitness function is linear. Experiments show that the proposed methods improve dynamic compression of Llama, Mistral, and Phi models.

strengths:
The proposed method shows strong empirical performance. It is flexible, as it can be applied to unstructured pruning, structured pruning/layer dropping, and quantization. Theoretical analysis of the proposed methods is also provided,

weaknesses:
1. I think it would be better if the average and standard deviation of the performance metrics across different runs were reported, as it can demonstrate the significance of the improvement. But I understand it could be expensive to do and might not be possible for baseline methods.


2. The theoretical results are somewhat disconnected from other parts. I think having the theoretical results is a bonus and it can be of interest beyond the LLM context. But some exploration of the function form of $f$ in real experiments or some simple simulation example might better demonstrate the significance of the results

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1732370852702

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion reminder

comment:
Dear Reviewers, 

As the discussion period is soon drawing to a close, we wanted to ask if you could please examine our detailed responses and let us know if they addressed your concerns.

Best regards,\
The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes EvoPress, an evolutionary framework for dynamic LLM compression. The authors claim their method outperforms existing approaches in structural pruning, unstructured sparsity, and quantization for Llama, Mistral, and Phi models. They argue that error monotonicity does not hold for LLMs and present an approach with claimed convergence and low sample complexity.

The paper's main strength lies in its empirical performance across multiple compression approaches and models, as well as the flexibility of the proposed method. However, these strengths are overshadowed by significant weaknesses. The theoretical justification, based on Theorem 1, appears disconnected from the non-linear nature of LLM compression. Moreover, the paper lacks sufficient exploration of hyperparameter sensitivity and transferability, and the experimental comparisons are limited in their comprehensiveness.

The primary reason for rejection is the substantial disconnect between the theoretical justification and the practical application of the method. The theoretical results based on linear fitness functions do not adequately support the method's effectiveness in the non-linear landscape of LLM compression. This, combined with the insufficient exploration of hyperparameter transferability and limited experimental comparisons, raises concerns about the method's generalizability and the robustness of the reported results.