EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Authors: Oliver Sieberling
Denis Kuznedelev
Eldar Kurtic
Dan Alistarh
Venue: ICML 2025 poster (2025)
Keywords: large language models
compression
quantization
pruning
evolutionary algorithms
Primary area: deep_learning->large_language_models
TL;DR: We propose an evolutionary optimization procedure for heterogeneous compression of large language models via pruning, quantization, or layer dropping.

Abstract: The high computational costs of large language models (LLMs) have led to a flurry of research on LLM compression, via methods such as quantization, sparsification, or structured pruning. A new frontier in this area is given by dynamic, non-uniform compression methods, which adjust the compression levels (e.g., sparsity) per-block or even per-layer in order to minimize accuracy loss, while guaranteeing a global compression threshold. 
Yet, current methods rely on estimating the "importance" of a given layer, implicitly assuming that layers contribute independently to the overall compression error. 
We begin from the motivating observation that this independence assumption does not generally hold for LLM compression: pruning a model further may even significantly recover performance.
To address this, we propose EvoPress, a novel evolutionary framework for dynamic LLM compression. By formulating dynamic compression as a general optimization problem, EvoPress identifies optimal compression profiles in a highly efficient manner, and generalizes across diverse models and compression techniques. Via EvoPress, we achieve state-of-the-art performance for dynamic compression of Llama, Mistral, and Phi models, setting new benchmarks for structural pruning (block/layer dropping), unstructured sparsity, and quantization with dynamic bitwidths.

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Oliver Sieberling' Denis Kuznedelev Eldar Kurtie 34
Dan Alistarh
Abstract
or activations, e.g. (Frantar al., 2022; Lin et al., 2023;
Dettmers & Zettlemoyer, 2022; Tseng et al., 2024), pruning- based, which sparsify the weight matrices, e.g. (Frantar & Alistarh, 2023; Yin et al., 2024), or structured pruning layer dropping, which drop entire model components, e.g. (Kim et al., 2024; Men et al., 2024). While improvements are still being made, existing methods are reaching diminishing returns in terms of accuracy-vs-comeee compression (Dettmers al., 2023; Tseng et al., 2024).
The high computational costs of large language models (LLMs) have led to a flurry of research on L.LM compression, via methods such as quan- tization, sparsification, or structured pruning. A new frontier in this area is given by dynamic, non- uniform compression methods, which adjust the compression levels (e.g., sparsity) per- per-block or even per-layer in order to minimize accuracy loss, while guarantecing a global compression thresh- old. Yet, current methods rely on estimating the "importance" of given layer, implicitly assuming that layers contribute independently the overall compression error. We begin from the motivating observation that this independence assumption does not generally hold for LLM compression: pruning a model further may even significantly recover performance. To address this, we pro- pose EvoPress. novel evolutionary framework for dynamic LLM compression. By formulating dynamic compression as a general optimization problem, EvoPress identifies optimal compres- sion profiles in a highly efficient manner, and generalizes across diverse models and compres- sion techniques. Via EvoPress, we achieve state- of-the-art performance for dynamic compression of Llama, Mistral, and Phi models, setting new benchmarks for structural pruning (block/layer dropping), unstructured sparsity, and quantization with dynamic bitwidths. Our code is available at http:///hhubbbooomSSSSAAAAAaaaaa
In this context, new direction is dynamic, or non-uniform, layer-wise compression, in which different layers can be compressed to various levels, according to their "sensitiv- ity" relative to the model output. Dynamic compression allows to maximize model accuracy while satisfying a given compression requirement, c.g. target model size. Instance- specific solutions for this problem have already been pro- posed for essentially every compression type: sparsity (Yin et al., 2024), quantization (Frantar & Alistarh, 2022), or layer dropping (Kim et al., 2024; Men et al., 2024). Broadly, these approaches work by assigning an error/sensitivity score to each layer and compression level, which measures the impact of its compression on output loss increase. Then, calculates compression assignment which minimizes the sum of error scores, while still satisfying the global compression constraint. Thus, such approaches inherently assume error monotonicity: i.e., that a lower sum of error scores taken over layers implies lower compression error for the entire model.
Our work starts from the observation that error monotonicity does not hold generally for LLM compression: specifically, there are instances where compressed models with lower sums of per-layer errors can perform worse than models with higher error. We illustrate this fact in Table 1, which shows an instance of a layer dropping configuration where pruning more blocks leads to significantly better perplexity than an instance which prunes strictly fewer blocks.
1. Introduction
Compression has become a standard way of reducing the deployment costs of large language models (L.LMs). Current post-training techniques can be roughly categorized into quantization-based which reduce the bit-width of weights
This refutation of error monotonicity im- plies that most prior approaches, which are based on this assumption, can lead to sub-optimal solutions. Thus, it motivates our investigation of alternatives towards optimal non-uniform compression. For this, we propose a new evolu- tionary search approach called EvoPress, which provably convergent, and is also sample and iteration efficient. Thus,
Contribution.
'ETH Zürich "Yandex Research 'IST Austria "Red Hat AI. Correspondence to: Dan Alistarh edan.alistarh@istte @ts..ac.at>.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 1. Depth pruning is not monotone. In this example (llama-3-88), removing strictly more blocks (depicted in orange) can improve perplexity across sources. The left half of a block corresponds to the attention layer. the right half to the MLP.
Model
Configuration (Each block contains Attention + MLP)
Wiki2
C44
FWJ
5.54
8.80
7.72
Llama-3-8B
188.01
147.25
70.46
24.39
35.53
26.24
EvoPress is the first non-uniform compression method with guarantees; its two efficiency properties are critical for prac- ticality in the context of L.MMs, where the cost of evaluating single models ("offspring"" is exceedingly high. We vali- date the approach across all three popular approaches for post-training LLM compression: layer dropping, one-shot sparsification, and quantization. We find that EvoPress con- sistently improves upon existing techniques, with major improvements at higher compression ratios.
meaning they can be optimized through highly local explo- ration, akin to gradient descent in continuous search space. EvoPress is specifically designed as an efficient hill climber, though it is capable of optimizing a broad class of fitness environments. Notably, our algorithm guarantees conver- gence: specifically, any linear fitness function defined on the n-dimensional hypercube will be maximized in expected Olk(n /) generations under the constraint ||æ||| k, where À is the number of offspring. The proof is quite non-trivial, as it needs to adapt stochastic drift analysis tech- niques to the case where multiple offspring are examined in each sub-step. In Figure I, we illustrate the algorithm's fast convergence and high efficiency on a practical example with correlated block dropping on Llama-3-8B, where we determined the optimum via (expensive) exhaustive search: EvoPress is able to reach the optimum in only 6 genera- tions, using a total of only 56 model evaluations. A key advantage of our approach is that it is agnostic of the model architecture and compression type. We illustrate this via ex- periments, which are the first to span all three compression methods, across different LLM families.
In more detail, we assume a setting where we are given a pre- trained model, a compression constraint such as the target model size, a set of compression options (e.g., 10 possible sparsity options per layer), and aim to identify a per-layer assignment which satisfies the constraint, while minimizing accuracy loss, measured in perplexity or in-context learning accuracy degradation. As is standard, e.g. (Frantar & Alis- tarh, 2022), from the compression options we build level database, where each layer is compressed independently to each compression option. During the candidate search, our offspring are models stitched together from the level database, and our fitness function will be the difference ge.g., in KL-divergence) between the outputs of the offspring and the original model, on set of calibration samples.
Results show that EvoPress significantly improves upon all prior work on depth pruning in terms of accuracy-vs. compression, especially at medium levels, and also outper- forms the prior best methods - OWL and dynamic program- ming, respectively - for non-uniform pruning and quanti- zation. Moreover, it can do so efficiently: the full version of EvoPress, applied at high compression granularity, will converge in few hours on single RTX 3090 GPU, and we also present lightweight version which utilizes fewer samples and converges in ~ I hour in the same setting, on an 8B-parameter model.
At each step, our search algorithm starts with a single search point (candidate model), and generates a constant À 2 1 ad- ditional offspring, by applying mutation operation which preserves the compression constraint. The selection stage is composed of multiple steps, where we iteratively evaluate the offspring and parent increasingly many randomly chosen samples. For instance, we may start to evaluate the parent and À = 64 offspring on less than a single sample on the first sub-step. but progressively multiply the number of calibration samples as we sift through candidates, reducing variance as we obtain more competitive offspring. We found this trade-off between exploration and evaluation variance essential for efficiency on LLMs, it drastically reduces our total number of evaluations relative to the case where all the initial offspring must be evaluated on full batch.
2. Related Work
To our knowledge, we are the first to present unified ap- proach which covers all types of post-training LLM com- pression (i.e., layer dropping / depth pruning and non- uniform pruning / quantization) - so far, these problems have generally been approached independently.
Our approach builds on the observation that the partial ef- fectiveness of prior work, which assumes error linearity, suggests that the fitness landscape induced by compression allocation has properties similar to those of a linear function. Such functions are particularly well-suited to hill climbing.
Depth Pruning. Recently, there has been a lot of interest in compression by removing entire transformer blocks, both for efficiency and to gain insights about the language model
2

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
plicitly relies on monotonicity and linearity assumptions the dependency between the per-layer errors and model output error, which we find not to hold on large models, es- pecially in the tigh-coopression regime (e.g., below 3 bits per parameter). Relative to SPDY, EvoPress provides guar- antees for a broader class of input functions, and focuses on efficiency for LLM compression.
300
EvoPress
Hamming Dist. to Opt.
Optimum
5
250
Perplexity
200
3
150
2
The recent OWL method (Yin al., 2024) focuses on non- uniform pruning of LLMs, and provides consistent improve- ments over uniform profiles via layer scoring system which analyzes the activation outlier structure. Experimentally, we find that OWL is effective especially for faaiily mod- els (Touvron et al., 2023) and at moderate sparsities, but observe significant gaps in favor of EvoPress across all mod- els and compression levels.
100
1
50
0
2
3
4
Generation
Figure 1. Removing twelve transformer blocks from LJama-3-8B under the constraint that only pairs of consecutive blocks be removed. EvoPress finds the optimal configuration from the 8008 possible removal combinations in generation 6.
NAS and Structural Pruning. Random search is also popular in the context of structural pruning and Neural Ar- chitecture Search (NAS) (Chen et al., 2020; Dong et al., 2021; Wang et al., 2020; Xu et al., 2021; Yin al., 2021; Molchanov al., 2022; Kurtić et al., 2024). However, such methods rely re-training and have notoriously high costs, which limits their applicability to post-training compression. Thanks to its low sample complexity, we believe that Evo- Press could be extensible to lightweight NAS, and plan to investigate this in future work.
itself. Most methods are based on scoring the importance of each block, and then maximizing the importance of the re- sulting model by removing the blocks of lowest importance. Therefore, they assume error linearity, meaning that each block contributes independently to the total compression error. Weight Subcloning (Samragh et al., 2023) proposed multi-step process to find good initializations for an un- trained smaller model given an already trained larger one, where the importance of each block is scored based on the ratio of &g norms between the output embeddings of the block with and without the residual connection. Shortened Llama (Kim et al., 2024) proposes scoring each block by measuring the perplexity after ritoviin the respective block from the full model. ShortGPT (Men et al., 2024) uses the cosine similarity between the input and output embeddings of each block to assess its importance. By contrast, Gromov et al. (2024) restrict to removing consecutive blocks and score each configuration by cosine similarity.
3. Method
All applications of EvoPress are grounded in a unified frame- work, where the objective is to identify the optimal model that adheres to a specified compression method and con- straint. Formally, given a base model M, we seek to maxi- mize the performance of the compressed model while satis- fying the compression constraint:
M = arg max ((M) subject to g(M) S C,
M
Non-Uniform Pruning and Quantization. He al. (2018); Ashok et al. (2018) were among the first con- sider automatic optimization of non-uniform compression, specifically for the case of pruning, where they developed Reinforcement Learning (RL)-based approaches. However, both approaches suffer from high tuning complexity and would be very hard to scale to large models. Follow-up work (Hubara al., 2021; Yao et al., 2021: Li et al., 2021) considered a similar problem specifically for quantization, but explore computationalll-exppee -expenssve solvers (e.g. ILPs) which rely on the fact that quantization has only a small number of choices (precision levels) per layer. SPDY (Fran- tar & Alistarh, 2022) considered a unified framework which reduces the problems to knapsack-type instances, and solves them optimally modulo discretization. However, SPDY ex-
where f(ār) quantifies the performance of the compressed model M and g(M) represents the compression constraint. For simplicity, we will define g as the model's total size (in terms of parameters); however, the method can be adapted to other constraints, such as inference speed.
We approach this optimization problem using evolutionary search, which is a specific form of randomized search. The feasibility of such an approach heavily depends on two fac- tors: the time required to evaluate the fitness of candidate solution and the number of such function evaluations needed until a satisfying result is achieved. This poses a particular challenge in our case, as assessing the performance of an L.M involves substantial computational costs.
3

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Level Database. As a first step, we compress the model to different levels. It is crucial that the units we search over specifically layers or blocks are compressed indepen- dendl; otherwise, we risk losing performance when stitch- ing together the compressed model. Ideally, the difference between two compression levels should be consistent across layers. This uniformity simplifies the optimization process, allowing for the free exchange of compression levels, as we will demonstrate for unstructured sparsity. However, this restriction is not essential for the search procedure to be ef- fective. In the context of quantization, we will demonstrate a relaxation of this requirement, where compression steps are uniform only across layers of the same size.
Algorithm 1: EvoPress: A (1 + Ao-ouuuution
Algorithi with Level-Switch Mutation and Multi-
Step Selection for Maximizing f [m]]" + R.
Initialization:
candidates <
for i <- to www.iinnaandddaaeeeeaeeaaaaaaaee
candidate < sampleUniffommyy(
cardlidttesappee((cccddaaee)
x0)) t o0lecctooopP1111essccccc...o initialnokens, K 1);
Optimization: for 1 <- to co do
offspring < D:
Mutation: for i 1 to do
yi + ++ z())
yi Le offapring.append((j))
Fitness Environment. Given the specified database, any compressed model is characterized by its compression level per unit (e.g. per layer). With n units, each available in m compression levels, our objective is to find
Selection: for step +- to seiectSteps do
Elitism: step select.Steps then
L Lfffapringaapeeddde"
offs. <- sts.eeettooooiieeeeeffff...o
tokens K survivors(ttep)) ((+1) offspring(0);
M arg max (AZg) subject to DE(m]
g(Ale) SC,
where we are searching over the set of n--tuples over [m]] Assessing the performance of a model in practice typically involves benchmark tasks, which have limited scope and require lengthy evaluation. We address these challenges by using the base model as the gold standard and focusing solely on the relative degradation of our compressed models. To quantify this degradation, we measure the Kullback- Leibler (KL) divergence between the two models, as it has proven particularly robust with limited data. Empirically, we observed that already around 64K tokens of calibration data (corresponding to 8 full sample sequences for Llama- 3-8B) are sufficient to reliably determine the quality of the lightweight model. To avoid confusion, we will refrain from inverting the fitness function and from now on consider the minimization problem
randomly selecting one unit and increasing its compression level. Next, a second unit is sampled until one with a match- ing level step size is found, and its compression level is decreased. This approach ensures that 1) the compression constraint is preserved, and 2) the offspring model maintains high similarity to the parent model - an important feature for achieving rapid convergence.
The second modification is that we employ a very aggres- sive form of multi-step selection. In the first stage, all À offspring are evaluated using only a fraction of full sam- ple. From this, only a small subset of the fittest offspring are selected to compete in the next stage, where they are evaluated on a significantly larger sample size. This pro- cess is repeated once more, and in the final stage, the few remaining offspring are evaluated against the parent using a "full" minibatch, consisting of approximately 20-50 times the number of tokens used in the first stage.
M = arg min DKL(PM II Qxa.) subject to g(Mr) S C, "E[m]=
where we speak of higher fitness whenever the KL- Divergence is lower.
For initialization, we apply the target level directly if it matches an available setting (c.g.. all layers at 70% sparsity for an average of 70% sparsity). If the target falls between two compression levels (e.g., for block dropping), we ini- tialize by randomly sampling candidates with some units compressed the next lower level, and others to the next higher level, selecting the fittest among them. A high level overview of this procedure can be found in Algorithm 1.
Algorithm. EvoPress starts from upon the classic (1 + A)- evolutionary algorithm, which maintains a single search point at any given time. In each generation, À offspring are generated by copying the parent and then applying a muta- tion operator to each copy. The offipring are then evaluated on the fitness function, and the fittest one is selected. As an elitist evolutionary algorithm, the (1 + X)-EA replaces its parent only if the best offspring has superior fitness.
We change this standard algorithm in two important ways. The first is by introducing level-switch mutation, a simple mutation operator that ensures high locality while preserv- ing the compression constraint. The operator involves first
Design Considerations. Randomized search heuristics are heavily influenced by the exploration-exploitaiie dilemma, i.c. the trade-off between exploring a broader solu- tion space and intensifying the search around the currently-
4

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
best solutions. Many applications utilize sophisticated search procedures, such as genetic algorithms, to enhance exploration, that often maintain a large population, intro- duce crossover operations, and adopt non-elitist strategies, where parents have no chance of survival into the next gen- cration. However, implementing these approaches for LLM compression would come with significant computational costs.
to two compression levels, while each weight of the linear function corresponds to the "salieney". The l f1-constriint now equivalent to compression constraint.)
Theorem 3.1. Let n, k E NN with k S n. and consider the (1 + A)-EA with X E OOn/ log(n)) and level-switch mutation. Then any linear fitness function J {x x E [0, ||x||| = n -k + R is optimized in expected
o I: (n (m b)) x) generations.
(n:
Crossover, for instance, is only effective if population di- versity is preserved, for example measured by the sum of pairwise Hamming distances between individuals (Jansen & Wegener, 2002; Opris et al., 2024). While this promotes a more thorough exploration of the search space, it requires allocating resources to less promising regions, which slows down the progress toward optimal solutions. Similarly, non- elitist algorithms, despite their ability to escape local optima (Dang et al., 2021; Jorritsma et al., 2023; Lengler et al., 2024), also incur costs by frequently discarding potentially useful individuals. Consequently, these approaches should be reserved for situations where the fitness landscape is rugged, and escaping local optima is critical to finding bet- ter solutions.
The proof is based stochastic drift analysis and can be found in Appendix A. Notably, by increasing the number of offspring per generation, we can reduce the number of generations required for convergence, with the reduction scaling proportionally to À up to a reasonably large value. Since our approach uses a highly aggressive form of multi-step selection, the benefit not simply a zero- sum tra Evaluating many offspring in each generation incurs a significantly lower per-offspring computational cost, leading to a substantial speedup in convergence time. This makes the algorithm highly efficient in relatively smooth fitness environments.
Discussion.
Convergence. Contrary to many real-world problems, dy- namic model compression with a carefully designed level database induces notably smooth fitness environment, where small changes in the compressed model tend to lead to small changes in performance. A key insight into the effectiveness of evolutionary approaches is that, although the search space expands exponentially with the number of units considered, the maximum Hamming distance between any two search points in the search space increases only linearly. Therefore, long as we receive a "signal" indi- cating the direction of improvement, even with scemingly limited progress per generation, we can converge rapidly to a high-quality solution.
4. Experiments
We now validate the effectiveness of EvoPress for deter- mining the optimal layer-wise compression across three approaches: (I) layer dropping, where the goal is to isolate the "optimal" set of blocks to drop given target ratio, (2) non-uniform unstructured sparsity and (3) non-uniform quantization, where we are given a set of compression options per layer (sparsities or bit-widths), and the goal is to find the "optimal" configuration that matches a certain model size. We focus on LLM compression, given the major interest in the reduction of their model size and inference latency, but our method is general and can be applied to any neural network architecture and application domain.
To illustrate this, we consider the problem of removing pairs of consecutive blocks of Llama-3-8B. We perform brute-force search over all possible 8008 block removal configurations, where six pairs of blocks are removed. Our method identifies the optimal configuration by the 6th gener- ation, having evaluated only 16 candidates for initialization and 8 candidates per generation. Figure illustrates how the algorithm approaches the optimum in Hamming distance.
Experimental Setup. We consider base models from the Llama-2 and Llama-3 (Touvron et al., 2023) families, Mistral-v0.3 (Jiang et al., 2023), and the instruction-tuned Phil-Medium--instrue---- 128k model (Abdin et al., 2024). We adopt KL-divergence as our fitness function as it provides a stronger and more robust signal compared to perplexity, re- flecting the predictive distribution of the original model. We present ablations to validate this choice Appendix B.3.
Consequently, EvoPress is heavily exploitation-foeet focuused:: we rely on elitism, introduce minimal mutation, maintain only single offspring, and therefore employ zero population diversity. We present ablations and a short discussion on these choices in Appendix B. EvoPress excels at optimizing smooth fitness environments, a capability we theoretically support by proving rapid convergence under an & -coonstrai for the class of linear functions. (Here, one bit corresponds
Concretely, our algorithm works as follows: Initially, for the case of quantization between available bit widths (e.g. 2.5 bit) and block dropping, we produce number of initial configurations (around 32), evaluate them on a few data sam- ples, and take the fittest one. For quantization with available target bitwidth and unstructured sparsity, we simply initial- ize using the uniform configuration. Then, we generate
5

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
new offssring in each generation by making small number of random switches in compression levels, where the num- ber of switches is sampled from min (randint (1, 3), randint ()) and compression levels are exchanged in such a way that the overall compression ratio is main- tained. We perform selection in multiple steps by iteratively choosing only the best configurations for survival, where each round uses progressively more tokens and has fewer survivors. To ensure elitism, we add the current parent to the candidate pool in the last stage of selection. Finally, after two or three of such stages, we take the last remaining configuration and adopt it as the population for the next round. We selected the number of generations, offspring, and tokens based on the search space size but found the search to be highly robust. Even drastic changes in hyper- parameter settings yield similar results (see, e.g., Figure 3). The detailed parameter setting is described in Appendix C.1. For our main results we used a fixed number of generations for optimization, which was chosen conservatively to better understand convergence behavior. In practice, stopping the search carly can significantly reduce runtime requirements. We diseuss simple stopping criteria and provide runtime comparisons in Appendix G.
2024), which scores blocks on the perplexity change after re- moval; (2) ShortGPT (Men et al., 2024), where blocks scored based on the average cosine similarity between input and output embeddings, including the residual stream; (3) Weight Subeloning (Samragh et al., 2023), where blocks are scored using the ratio is the input embedding and f()) is the block's output, ex- cluding the residual stream; (4) Sliding Window Cosine Similarity (Gromov et al., 2024), where sets of consecutive blocks are scored based on the cosine similarity between em- beddings before and after the blocks, including the residual stream. While Gromov et al. (2024) directly score entire re- moval configurations, the other approaches determine block removals based on their isolated scores.
+ æ||, where æ
Mistral-7B-vO.
180
EvoPress
160
EvoPress (Attn.+MLP)
140
Cosine Sim. (Window)
ShortGPT
Perplexity
120
Shortened Uama
100
Weight Subcloning
80
60
To perform per-layer compression via unstructured spar- sity and quantization, we adopt the data-aware compres- sion methods SparseGPT (Frantar & Alistarh, 2023) and GPTQ (Frantar et al., 2022). For this purpose, we use Fineweb-Edu (Penedo et al., 2024) as a source of clean and diverse calibration data. Following Egiazarian et al. (2024), we fix the total number of calibration tokens to 8 million (8M). For a fair comparison, all competitive meth- ods employ the same calibration data.
20
0-
30
50
Sparsity (%)
Figure 2. Depth pruning results, Mis. Relative all prior methods, EvoPress shows significantly lower PPL gap relative to the uncompressed model, with remarkably large gaps medium compression rates.
Search Space. In our approach, attention and MLP mod- ules are treated independently rather than as single unit For each module, there are two options: either retain it or remove it. To achieve a target sparsity/depth, we initially remove an equal number of attention and MLP modules. During mutation, we allow compression level adjustments only between modules of the same type. We leave open for future research to remove this constraint to allow flexibility in the number of removed attention and MLP modules.
Evaluation. We follow a standard evaluation proto- col (Frantar et al., 2022), measuring perplexity on the WikiText-2 (Merity et al., 2016) and C4 (Raffel et al., 2019) datasets for language performance and accuracy on zero-shot evaluations on standard benchmarks: Wino- Grande (Sakaguchi et al., 2021), PiQA (Tata & Patel, 2003), HellaSwag (Zellers et al., 2019), ARC-easy and ARC- challenge (Clark et al., 2018) via the LM Eval Harness (Gao et al., 2021).
Figure 2 compares our method with baselines from previous work on Mistral-7B-v0.3. For a better com- parison, we also included results where only entire trans- former blocks are removed (Att.+MLP).. EvoPress consis- tently outperforms all previous methods, showing significant improvements even at medium sparsity levels. While all baseline methods fail entirely beyond 31.25% sparsity, Evo- Press identifies functional submodels even when removing half of the model. To our knowledge, this is the first method to achieve such results. We observed similar collapses in Llama-2-7B, 1lam--3-88 and Llama-3. 3.1-BB....BB........
Results.
4.1. Application 1: Depth Pruning
We first apply EvoPress on Depth Pruning. Although re- moving entire transformer blocks generally results in large accuracy losses, this approach recently attracted attention in the context of initializing smaller models, as it guarantees speedups proportional to the sparsity (Samragh et al., 2023; Kim et al., 2024). Additionally, block dropping provides insights into the capabilities of transformer models, mak- ing it relevant for interpretability. We will compare against the following baselines: (1) Shortened LJama (Kim et al.,
1-8B. Overall, Evo-
6

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Llama-2-7B 70% Sparsity
-lam---77 70% Sparsity
26
Perplexity (Test)
1.4
Perplexity (Test)
24
KL Divergence (Train)
24
KL Divergence (Train)
Perplexity
22
GPU Hour (RTX 3090)
1.2
(L-Divergence
Perplexity
22
GPU Hour (RTX 3090]
1.2
KL--Divergecc
20
20
1.0
16
0.8
0.8
14-
0.6
0.6
12
50
100
150
200
250
300
350
400
50
100
150 200
250
300
350
400
Generation
Generation
Figure 3. Left: The convergence of EvoPress vs. number of generations and wall-clock time (on single RTX 3090 GPU with 24GB RAM) for Llama-2-7B. observe convergence close to optimum in 5-6h; Right: Convergence of the "super-fast" version which reduces the number of tokens used for each evaluation. It converges to similar accuracy in little over one hour, in the same setting. The KL-Divergence corresponds to the fitness of the survivor in each generation, which is measured on a random minibatch the entire training dataset. The perplexity is computed the entire test dataset.
Press consistently outperforms baselines across all tested models and sparsities (see Appendix D.1 for full results), and does in matter of minutes (Appendix D.2). We pro- vide runtime comparisons as well as additional comparisons to the iterative search methods SLEB (Song et al., 2024) and BlockPruner (Zhong et al., 2024) in Appendix G.
Search Space. Sparsity levels are generated as follows: For each layer, we first produce the base level correspond- ing the targeted average sparsity. Then, both higher and lower compression levels, where the differ- ence between two levels corresponds to a fixed number of weights. In our experiments, we used a "step size" of IM weights uniformly. This approach enables the mutation of compression levels across layers, independently of their size. We adopt SparseGPT (Frantar & Alistarh, 2023) for layer pruning. We provide results of EvoPress with Wanda pruning (Sun et al., 2024) in Appendix G.
generate
All four previous methods rely on human-crafted scoring methods to identify the optimal combination of transformer blocks remove. This not only suboptimal, but also prone to bias, as their results may reflect the characteristics of the method itself rather than the model's true behavior. Specifically, we found that most scoring methods tend to favor deeper blocks, resulting in highly similar removal con- figurations across different prior scoring methods (Appendix Table 13). This likely occurs because methods that bias to- wards deeper blocks generally perform better than those that focus earlier blocks, although neither may be optimal. In contrast, EvoPress employs an unbiased approach, offering more accurate and meaningful insights into the model.
Table 2. Performance various methods at 70% average sparsity. EvoPress outperforms prior methods both in terms of validation perplexity (PPL) and zero-shot accuracy.
Model
Method
Wiki2|
C41
Task Avg."
Dense
4.82
7.72
68.7
Uniform
23.08
30.03
49,9
Mistral-7B-v0.3
OWL
17.22
21.66
51.9
EvoPress
14.42
16.46
53.8
4.2. Application 2: Unstructured Sparsity
Dense
5.54
7.10
68.6
Next, examine performance for unstructured sparsity, which offers more fine-grained compression. The standard approach is to allocate sparsity uniformly across layers. However, some layers may be more sensitive to sparsity, which can significantly impact the model's output. To ad- dress this, OWL (Yin et al., 2024) introduces the Layer Outlier Distribution (LOD) metric a measure of layer saliency, and computes sparsity profile that is weighted by LOD. We compare EvoPress with both uniform sparsity and OWL. For OWL we used the same hyperparameter grid as the original work and took the configuration yielding the best perplexity for each model.
Uniform
85.84
98.35
44.1
Llama-3-8B
OWL
48.07
52.32
48.4
EvoPress
28.76
33.72
50.8
Dense
4.02
8.31
73.2
Uniform
16.66
24.73
56.5
Phi-3-Medium-
OWL
15.66
23.38
55,4
EvoPress
13.83
19.13
59.8
Experimental Results. We compare different methods for pruning to 50%, 60% and 70% unstructured sparsity. We report the 70% results in Table 2; the 50% and 60% results can be found in Appendix Tables 14 and 15, respectively.

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
As illustrated in Table 2, EvoPress successfully finds better profiles than uniform sparsity and noticeably outperforms competitive methods on PPL and zero-shot average accuracy by large margins on all models.
Llama-3-80
8.6
Uniform EvoPress
8.5
8.4
Examining sparsity profiles (Appendix Figures 12 and 13), we observe that EvoPress prunes the first blocks less ag- gressively, blocks in the beginning of the second half of the model more aggressively while keeping the even deeper
Perplexity
8.3
8.1
blocks relatively dense. Notably, EvoPress assigns high
importance to the v-proj matrices, reducing its sparsity to below 45%, compared to an overall average of 70%.
4.0
4.5
5.0
5.5
6.0
6.5
7.0
7.5
8.0
Bit
Running Time. EvoPress is also iieeefffiient. Figure 3 illustrates the rapid convergence of our method vs. itera- tions and time, with steady improvements in test perplexity. Moreover, by reducing the number of tokens used in the multi-step selection evaluation, by 4x in the first step and 8x in the last step, and making each generation have fewer offspring, we can significantly speed up the search. This "super-fast" version converges in little over one GPU hour to similar test PPL (Figure 3, right), demonstrating the ro- bustness of EvoPress, which can lead to further gains.
Figure 4. EvoPress enables non-uniform quantization with end- to-end speedups in vLLM. vLLM supports only 4-bit and 8-bit quantization, while requiring the query/key/value matrices as well as the up/gate projections to share the same bitwidth. The displayed average bitwidths include the overhead from groupwise scales.
the original model output at a layer, and Y the output of the compressed model. Then, the optimal compression allocation can be determined via a dynamic programming (DP) approach. The full SPDY method applies second iterative random search step, which is very expensive to implement at LLM scale, and is therefore omitted.
Table Performance of various profiles at quantization, for PPL and avg. zero-shot accuracy.
Model
Method
Wiki2|
C44
Task Avg.t
Dense
4.82
7.72
68.7
Search Space. For each linear layer, we produce differ- ent configurations via GPTQ (Frantar et al., 2022) with a standard group size of 128. In each step of the evolutionary search, the bitwidth of some layers is increased while the bitwidth of others is decreased. To facilitate uniform tran- sitions between compression levels, quantization options differ by integral bits (1 bit in the following). Since differ- ent layers may have different sizes, we allow swaps only between projections with the same number of weights.
Uniform
5.54
8.57
66.3
Mistral-7B-v0.3
DP
5.79
8.84
66.0
EvoPress
5,21
8.42
67.1
Dense
5.54
7.10
68.6
Uniform
12.19
15.76
60.2
Llama-3-8B
DP
29.00
20.03
61.3
EvoPress
7.49
12.03
64.3
Dense
4.02
8.31
73.2
Uniform
5.18
9.05
70.0
Experimental Results. To validate the effectiveness of EvoPress, we consider the challenging problem of quantiza- tion to bits and below. For this compression rate, uniform GPTQ quantization faces significant performance drops, motivating dynamic quantization bitwidth allocation. We produce quantization levels at 2, 3, 4, 5, and 6 bits and search for an optimal compression profile with respect to the fitness function. The results in Table 3 indicate that non-uniform quantization with EvoPress produces superior models compared the baseline methods. The improve- ments are even more pronounced at 2.25 bit and 2.5 bit quantization, detailed in Appendix F.1. To show the real- world applicability of EvoPress, we additionally tested a restricted search limited to VLLM-supported options. vLLM allows only 4-bit and 8-bit quantization, and requires uni- form bitwidth for query/key/values matrices and up/gate projections within block. Figure 4 demonstrates strong
Phi-3-Medium-14B
DP
5.72
9.71
69.1
EvoPress
5.09
9.00
70.8
4.3. Application 3: Quantization
Finally, we apply EvoPress to the more challenging task of non-uniform neural network quantization, where the widely adopted baseline is uniform per-layer quantization (Frantar al., 2022; Lin et al., 2023; Chee al., 2023). Additionally, we consider Da-based approach for comparison. (While OWL has also been applied to quantization, the authors found that it underperforms even relative to uniform per- layer guantization (Yin et al., 2024).) The DP search is very similar to SPDY (Frantar& & Alistarh, 2022), where the goal to minimize the Normalized Mean Squared Error (NMSE), defined as NMSE || Y | where Y represents
8

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
mixed precision quantization even under these constraints.
A., Modi, H., Nguyen, A., Norick, B., Patra, B., Perez- Becker, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C., Roy, S., Ruwase, O., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Tanaka, M., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Yadav, S., Yang, F. Yang, Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L.,, Zhang, Y., Zhang. Y., Zhang. Y., and Zhou, X. Phi-3 technical report: A highly capable language model locally on your phone, 2024.
We visualize the configurations found by EvoPress for Llama-3.1-88 in Appendix Figures 15 and 16. We observe that the second and final blocks are compressed less aggres- sively, while the first block undergoes the highest compres- sion. This contrasts with unstructured sparsity, where the first block is among the least compressed. Therefore, dy- namic compression allocation must account for the specific compression method used, which underscores the impor- tance of automated compression allocation.
Overall, we observe that EvoPress yields significant accu- racy improvements (e.g.. 4.1 on the zero-shot averages for Llama-3-8B), compared to the uniform profile. Moreover, the improvement over the next-best method is always signif- icant, both in terms of perplexity and zero-shot accuracy.
Ashok, A., Rhinehart, N., Beainy, F., and Kitani, K. M. N2N learning: Network to network compression via policy gra- dient reinforcement learning. In International Conference on Learning Representations (ICLR), 2018.
Bengio, Y., Léonard, N., and Courville, A. Estimating or propagating gradients through stochastic neurons for con- ditional computation. arXiv preprint arXiv: 2013.
5. Conclusion
We have presented EvoPress, an optimization framework for non-uniform compression. EvoPress is based on a new evolutionary search algorithm with low sample and itera- tion complexity, especially well-suited loss landscapes in LLM compression. Specifically, we have shown that EvoPress can converge extremely fast to accurate configura- tions for various non-uniform LLM compression problems, and is also fast to execute in practice. We also emphasize the breadth of our study, our method was implemented and tested on three different compression approaches, relative to prior work which largely focused on a single application. Interesting directions did not investigate are 1) combin- ing different compression approaches into the same search space, and 2) finer-grai grained structured pruning. We plan to investigate this in future work.
Chee, J., Cai, Y., Kuleshov, V., and Sa, C. D. Quip: 2-bit quantization of large language models with guarantees, 2023.
Chen, D., Li, Y., Qiu, M., Wang. Z., Li, B., Ding. B., Deng. H., Huang, J., Lin, W., and Zhou, J. Adabert: Task- adaptive bert compression with differentiable neural ar- chitecture search. In Bessiere, C. (ed.), Proceedings of the Twenty-Ninth International Joint Conference on Artifi- cial Intelligence, IJCAI-20. pp. 2463-2469, International Joint Conferences on Artificial Intelligence Organiza- tion, 2020. doi: 000224963/jcaii2222023333 URL https: 10 //
2020/ /441
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint /80: 054557, 2018.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Dang, D.-C., Eremcev, A., and Lehre, P. Escaping local optima with non-elitist evolutionary algorithms. Proceed- ings of the AAAI Conference on Artificial Intelligence, 35: 05 2021. doi: 1.609/aai..33514..1....
References
Dettmers, T. and Zettlemoyer, L. The case for 4-bit pre- cision: k-bit inference scaling laws. arXiv preprint arXiv:2212.09770, 2022.
Abdin, M., Jacobs, S. A., Awan, A. A., Aneja, J., Awadallah, A., Awadalla, H., Bach, N., Bahree, A., Bakhtiari, A., Behl, H., Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V. Chopra, P., Giorno, A. D., de Rosa, G., Dixon, M., Eldan, R., Iter, D., Garg, A., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lee, Y. T., Li, Y., Liang, C., Liu, W., Lin, E., Lin, Z., Madan, P., Mitra,
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D.,, Frantar, E., Ashkboos, S., Borzunov, A., Hoefler, T., and Alistarh, D. Spqr: A sparse-quantized representation for near-lossless Ilm weight compression. arXiv preprint 2023.
Doerr, B. and Goldberg, L. A. Drift analysis with tail bounds. In Schaefer, R., Cotta, C.,, Kolodziej, J., and Rudolph, G.

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
(eds.), Parallel Problem Solving from Nature, PPSN XI, pp. 174-183, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg. ISBN 978-3-642-15844-55
He, Y., Lin, J., Liu, Z., Wang, H., Li, L.-J,, and Han, S. AMC: AutoMI. for model compression and acceleration mobile devices. In European Conference Computer Vision (ECCV), 2018.
Doerr, B., Doerr, C., and Lengler, J. Self-adjusting mu- tation rates with provably optimal success rules. In Proceedings of the Genetic and Evolutionary Compu- tation Conference, GECCO '19, pp. 1479-1487, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450361118. 3321777.33211333. URL https://doi.org 1145/3321707
Hevia Fajardo, M. A. and Sudholt, Self-adjusting popula- tion sizes for non-elitist evolutionary algorithms: why success rates matter. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '21, pp. 1151-1159, New York, NY, USA, 2021. Associa- tion for Computing Machinery. ISBN 9781 9781450383509, doi: 10. 0111144444499.45553333 URL https: :ttp:/// /doi. org 2tg/10.10 10, 1ee..114433444
doi: 10.1145/
3459338.
Dong. C., Wang. Xu, H., Peng, J., Ren, X., and Liang, X. Efficientbert: Progressively searching multilayer percep- tron via warm-up knowledge distillation. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1424-1437, Association for Computational Linguistics, 2021. URL https:// acl anthology........ - org/oo////oo 2tgg/021..r emml p-mmal--mm. 1--m-miin 108.
Hubara, I., Chmiel, B., Island, M., Banner, R., Naor, S., and Soudry, D. Accelerated sparse neural training: A provable and efficient method to find N:M transposable masks. In Conference on Neural Information Processing Systems (NeurlPS), 2021.
Jansen and Wegener. The analysis of evolutionary algorithms-a a proof that crossover really can help. Al- gorithmica, 34(1):47-66 September 2002. ISSN 0178- 4617. doi: 10. 10.10077/000053--022-00000222220 URL https: //doi orgg//or//i 10tp:/1000000//0005000000222---0
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E.,
Babenko, A., and Alistarh, D. Extreme compression of large language models via additive quantization. arXiv preprint arXiv:2401.06111, 2024.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, d. I., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.., al. Mistral 7b, arXiv preprint arXiv:2310.06822 2023.
Frantar, E. and Alistarh, D. SPDY: Accurate pruning with speedup guarantees. arXiv preprint arXiv:2201.13096 2022.
Frantar, E. and Alistarh, D. Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2301.00774 2023.
Johannsen, D. Random Combinatorial Structures and Ran- domized Search Heuristics. PhD thesis, Universität des Saarlandes, 2010
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Jorritsma, J., Lengler, J., and Sudholt, D. Comma se- lection outperforms plus selection on onemax with ran- domly planted optima. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '23, pp. 1602-1610, New York, NY, USA, 2023. Associa- tion for Computing Machinery. ISBN 9798400701191 doi: 10. 111553581133,,,554488,, URI. https: /ooi. org 4 35904488.........
Gao, I.,, Tow, J., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., McDonell, K., Muennighoff, N., Phang, J., Reynolds, L., Tang, E., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot lan- guage model evaluation, September 2021. URL https: 5371628.
Kaufimann, M.. Larcher, M.. Lengler, J., and Zou, X. Self-adjusting population sizes for (1, (), mono- tone functions. In Parallel Problem Solving from Na- ture PPSN XVII: 17th International Conference, PPSN 2022, Dortmund, Germany, September 10-14, 2022, Proceedings, Part II, pp. 569-585, Berlin, Heidelberg. 2022. Springer- ISBN 978-3-031-14720-- doi: 10. 10.100779788-3-31114470000 40. URL https::/t////e.. org 00111.1000/// 3- 100--3311111-----11 0_400
/ //doi. org 10. /281/
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers, 2024.
He, J. and Yao, X. A study of drift analysis for estimat- ing computation time of evolutionary algorithms. Nat- ural Computing: An International Jourmal, 3(1):21-35, March 2004, ISSN 1567-7818. 8. doi: 10.1023/B.NACO. 0000023411..11993.1,, URL htdpo 1023/B NACO. 00023//8H800.00000331311..33333
Kern, S., Müller, S. D., Hansen, N., Büche, D., Oce- nasck, J., and Koumoutsakos, Learning probabil- ity distributions in continuous evolutionary algorithms-
10

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
comparative review. Natural Computing: An Inter-
pp. 1605-1613, New York, NY, USA, 2024. Associa- tion for Compuring Machinery. ISBN 9798400704949,
national Journal, 3(1):77-112, March 2004. ISSN 1567-7818, doi: 10. (0100310:NAAC0.0000231163333
doi: 10.1 10.114553633522936544100 URL attps://doi. org/1C.1145/3638223 365753363552936552208 3654120.
4e. URL https:://// http://.....r1111112121112 00000234166 6. 49..9.4e.
Penedo, G., Kydliček, H., allal, L. B.. Lozhkov, A., Mitchell, M., Raffel, C., Werra, I.. V., and Wolf, The fineweb datasets: Decanting the web for the finest text data at scale, 2024. URL https:/ aarxiv.vvvv//////vv///aa/////aaa wwweiiiiii..o.iee....o.....ooooe oor.oorg/ooo......ooo.......o.be / 2406 17557.
Kim, B.-K., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: A simple depth pruning for large language models, 2024.
Kurtié, E.,, Frantar, E., and Alistarh, D. Ziplm: Inference- aware structured pruning of language models. Advances in Neural Information Processing Systems, 36, 2024.
Raffel, C.,, Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, J. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv: 1910. 0. 10683, 2019.
Lengler, J. Drift analysis. In Theory of Evolutionary Com- putation, pp. 1311 Springer, 2020.
Sakaguchi, K., Bras, R. L., Bhagavatula, C.,, and Choi, Y. Winogrande: an adversarial winograd schema chal- lenge at scale. Commun. ACM, 64(9):99-106, 2021. doi: 10.1445/3474381 URL https: / htt//doioooooo org//....o...oooo
Lengler, J., Schiller, L.. and Sieberling, O. Plus strate- gies are exponentially slower for planted optima of random height. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '24, pp. 1587-1595, New York, NY, USA, 2024. Associa- tion for Computing Machinery. ISBN 9798400704949. doi: 10. 10.1144/3666 1111114/3332299666655553 org/10. 10l..1145/0000 1111111114//333529333333 3654088.
114 1145/34744
Samragh, M., Farajtabar, M., Mehta, S., Vemulapalli, R., Faghri, F,, Naik, D., Tuzel, O., and Rastegari, M. Weight subcloning: direct initialization of transformers using larger pretrained ones, 2023.
URL https: //doi.
Li, Y. Gong, R., Tan, X., Yang. Y., Hu, P., Zhang. Q., Yu, E,, Wang, W., and Gu, S. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference Learning Representations (ICLR), 2021.
Song, J., Oh, K., Kim, T., Kim, H., Kim, Y., and Kim, J.-J. Sleb: Streamlining lims through redundancy verification and elimination of transformer blocks. In Proceedings of the 41st International Conference on Machine Learning, 2024.
Lin, J. Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-awan weight quantization for IIm compression and acceleration. arXiv preprint (xXi::2306.00978, 2023.
Sun, M., Liu, Z., Bair, A., and Kolter, Z. A simple and effective pruning approach for large language models, 2024.
Ma, X., Fang, G., and Wang, X. LIm-pruner: On the struc- tural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
Tata, S. and Patel, J. M. PiQA: An algebra for querying pro- tein data sets. In International Conference on Scientific and Statistical Database Management, 2003.
Men, X., Xu, M., Zhang, Q. Wang, B., Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect, 2024.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint 23..i::2302.19777. 2023.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.0 07843, 2016.
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and Sa, C. D. Quipi:: Even better Ilm quantization with hadamard inco- herence and lattice codebooks, 2024.
Molchanov, P., Hall, J., Yin, H., Kautz, J., Fusi, N., and Vahdat, A. Lana: latency aware network acceleration. In European Conference on Computer Vision, pp. 137-156. Springer, 2022.
Wang. H., Wu, Z., Liu, Z., Cai, H., Zhu, L., Gan, C., and Han, S. Hat: Hardware-award transformers for efficient natural language processing. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 7675-7683 Association for Computational Linguistics, 2020. URL https: aalaattnoooooooooo org/20200 a
Opris, A., Lengler, J., and Sudholt, D. A tight o(4k/pc) runtime bound for a (μ+1)ga on jumpk for realistic crossover probabilities. In Proceedings of the Genetic and Evolutionary Computation Conference, GECCO '24
684.

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Xu, J., Tan, X., Luo, R., Song, K., Li, Qin, T., and Liu, T.-.. Nas-bert: Task-agnostic and adaptive-size bert compression with neural architecture search. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 1933-1943. ACM, 2021. URL https: /t///dlaaee acm. org/doi/10. 1145/34 344334447445 3467404.
Xu, P., Shao, W., Chen, M., Tang, S., Zhang, K., Gao, P., An, F,, Qiao, Y., and Luo, P. Besa: Pruning large lan- guage models with blockwise parameter-efficient sparsity allocation, 2024. URL httpp:://rxiv..r org/ abs/ 2402. 1.....6880 16880
Yao, Z., Dong, Z., Zheng, Z., Gholami, A., Yu, J., Tan, E., Wang, L., Huang. Q., Wang, Y., Mahoney, M., et al. H3wq-v3: Dyadic neural network quantization. In Inter- national Conference on Machine Learning, pp. 11875- 11886. PMLR, 2021.
Yin, L., Wu, Y., Zhang, Z., Hsieh, C.-Y., Wang, Y., Jia, Y., Li, G., Jaiswal, A., Pechenizkiy, M., Liang, Y., Bendersky, M., Wang. Z., and Liu, S. Outlier weighed layerwise sparsity (owl): A missing secret sauce for pruning Ilms to high sparsity, 2024.
Yin, Y., Chen, C., Shang, L., Jiang. X., Chen, X., and Liu, Q. Autotinybert: Automatic hyper-parameter optimization for efficient pre-trained language mod- els. In Proceedings of the 2021 International Joint Conference Natural Language Processing, pp- 5146-5157. Association for Computational Linguistics, 2021. URL https: /aaclaaattlloyyy 2r2/2021. Ajanlp-main.. 428.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can machine really finish your sentence? In Korhonen, A., Traum, D. R., and Mårquez, L. (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, Italy, July 28- August 2, 2019, Volume Long Papers, pp. 4791-4800. Association for Computational Linguisties, 2019. doi: 10.18653/ 1472. URL https:// doi. waii.org/11 /00.////./00 186118663331 1ttp://..111533///1111111111
Zhong, L.. Wan, F., Chen, R., Quan, X., and Li. L. Block- pruner: Fine-grained pruning for large language models. arXiv preprint ar 2rXhv:246..05500 2024.
12

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
A. Convergence Proof of EvoPress
A.I. A Warm-Up Argument for a Single Offspring
The overall goal of this section is to prove Theorem 3.1. As the main argument quite complex, relying heavily on stochastic drift analysis, we begin with warm-up, namely by presenting simpler proof for the restricted case where
Unlike the practical application of Algorithm 1, this section assumes that each fitness evaluation returns the exact, 'true,' fitness value, ignoring any noise introduced by minibatching. Additionally, our results hold for any initialization. To align with standard notation in the runtime analysis of evolutionary algorithms, we will count generations starting from zero (i.e., using 0- b-based indexing).
Theorem A.1 (Single offspring). Let n, k E N with lo S and consider the (1 + 1)-EA with level-switch mutation. Then any linear fitness function f {x E {0, 1)", ||x|| = k) - IR. is optimized in expected
O (n k)) generations.
Proof. Let R" be the weights associated with the linear function such that f(z) Di=1 24- W₁. To derive upper bound, we can assume that no two weights are equal'. Furthermore, assume without loss of generality that these weights are sorted increasingly, meaning wi < W2 < < Wn. and that k S (n k), as the other case follows from symmetry. Since f is defined on the bit strings with exactly O's its unique optimum is now given by Tapt = 1*1n-= n- Denote by (*() the search point at step and let
T = in ( 2 0 x(()) = Topt
be the number of generations required until the optimum is found. the random variable that captures the number of I's in the first & bits of the search point step t. We observe the following:
Define (+) プラ=
(1)
1. x()) = 0 <> 19 Topei
2. (() is monnonn-eee non-increasing;
3. x²) - ((+11 1
4. X(0) Σド 2
It follows that given the initial search point (0))) we can decompose T into
Σイ E
T2,,., (0) stages Tı, T2,T2,-. ..., Te, where
T int 2 0 X(t) = -1)) -in((ft 2 0 (*) j}} captures the number of generations spent at stage j. By the linearity of expectation, we have
E[T] x-()) = T
It remains to bound the expected time spent at each stage. Each offspring is generated by copying the parent, selecting a 1- bit uniformly at random, selecting a O-bit uniformly at random, and finally flipping both bits. At stage j exactly j of the O-bits are among the last positions and exactly of the n k 1-bits are among the first positions. Hence, j2 out of the total k(n k) (1-bit position, O-bit position)- -pairs advance the optimization to the next stage, yielding
j²
(+ = (+) = = k(nn - k)
Therefore, T ~ Geometric 8
and
k(-kk
E T k(n k)
j2
FFrrmall, this he shown using stochastic domination, which involves coupling the potentials in both and proving that, given the same randomness, one always at least as large as the other.
13

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
To obtain an upper bound, we can make worst-case assumption by setting X(x!0)) = k. We conclude
k
E[T] S B =k] = Σ
B[T]] = k(n k)
E (O(k(n ())) 1:).
j=1
Discussion. Observe that, under the assumption that the probability of initializing at the optimum is sufficiently small, the proof is tight to a constant factor of 2.
is important to note that the above proof relies on the key assumption that whenever one of the j² "good" pairs is selected during mutation, the resulting offspring the fittest among all candidates. This condition holds naturally when there is only a single offspring, as the offspring produced by flipping one of the j² pairs will have higher fitness than the parent. However, in the case of multiple offspring, this approach breaks down, as offspring produced by flipping of the j² "good" pairs might still have lower fitness than another offspring that was not generated by flipping of these j² "good" pairs.
A.2. The Main Argument
Drift analysis, originally developed study all kinds Markov chains, has become the most widely used technique for analyzing the runtime of evolutionary algorithms in recent years. It works by first defining potential function X that measures the progress over each step t of the optimization. By estimating how this potential changes at each step in expectation, i.e., computing the drift in X one can then make probabilistic statements about the number of steps required until the potential reaches a certain threshold, also called the hitting time. To this end, a variety of drift theorems have been established, two of which will be employed in our proof. For a more thorough introduction to drift analysis, we refer to Lengler (2020).
First of all, we will utilize the the Multiplicative Drift Theorem, more specifically tail bound introduced by Doerr and Goldberg, which is applicable when the potential decreases by constant fraction in each step.
Theorem A.2 (Multiplicative Drift, Tail Bound (Doerr & Goldberg, 2010)). Let (X(1)) J120 be a sequence of non-negative random variables over finite state space S C IRJ. Assume that (0) < b and let T be the random variable that denotes the first point in time t E for which (t) I a, for some a S b. Suppose that there exists > O such that for all t < T,
E| x((+1) Xx0] 2 8x(())
Then,
t log(b/a)
PTT
Se
8
Additionally, we will employ Johannsen's Variable Drift Theorem. This theorem provides more flexibility compared to the Multiplicative Drift Theorem, as it can be applied when the drift is bounded by any increasing function of the potential. This often occurs naturally, as optimization typically becomes more difficult approaching the optimum.
Theorem A.3 (Variable Drift Theorem (Johannsen, 2010).. Let (X(())20 be a sequence of non-negative random variables over a finite state space S C RØ. Let 8min == min(S) \ (0)), ler T inf t 2 x 0}, and for s S let A (1) () ((+1) x s). there an increasing function
R - IR + such that for all E S (0) and
all t 2 0.
A ( Z h(s)),
then
x (0)
8min
E T
<
+
do
h( h(Smin)
h(o)
where the expectation on the latter term is over the random choice of (0).
14

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
We will first prove an auxiliary lemma, which will play a central role in bounding the drift. For this purpose, we define an inversion in bit string x: E (0, 1 pair of indices (i such that i < and x; z-j- The distance between these indices, j i, will be referred to as the spread of this inversion.
Lemma A.4. Let E (0, 1)th be arbitrary bit string with k 0-bits and denote by the number of inversions in x. Then, the average spread of these inversions is at least v8/16.
Proof: Consider the bit string 1a-k containing all 1-bits of z. We can now generate an arbitrary bit string {0, 1) with k O-bits and inversions by adding k 0-bits in such way that inversions are generated. Observe that adding 0-bit after the j'th 1-bit results in exactly j additional inversions, regardless of the other 0-bits. This means that the order in which the O-bits are added does not affect the outcome. We proceed by a case distinction depending on how the inversions are generated.
Case 1: at least s/2 inversions are generated by adding 0-bits after the /s'th 1-bit.
For each O-bit that is added after the Vs'th th 1-bit, at least half of the resulting inversions have spread at least v8/2. Consequently, this implies that there are at least 8/4 inversions having spread at least / in total.
Case 2: fewer than s/2 inversions are generated by adding 0-bits after the /itth 1-bit.
It follow that more than s/2 inversions generated by adding 0-bits not after the min(n k, vs) thh 1-bit. Observe that each 1-bit can participate in at most j inversions with spread at most j. More specifically, each 1-bit can be part of at most / inversions with spread at most v Because all of the s/2 inversions that are added contain one of the first min(n- - ki V 1-bits, at most 8/4 of these inversions can have spread at most v8/4. Therefore, we conclude that the average spread of all inversions must be at least /5/16.
We continue to prove the final result.
Proof of Theorem 3.1
Proof. As in the proof of Theorem A.1 let E R" represent the weights associated with linear function of the form f(x) = Ei= Zi W/. To establish an upper bound, we can again assume that no two weights are equal. Additionally, without loss of generality, assume that the weights are ordered in increasing value, i.e., Wi < W2 < < w,, and that k < k; as the other case follows by symmetry. Let (,()) denote the search point at step I, and define
T= = inf(t 2 0 = 1m
as the number of generations required to reach the optimal solution. Consider the potential function
(0)) 1- k. ( 1)+1
X² Σ 1
2
which captures the number of inversions at step t. Since Topt 0 1n-k is the only bit string with k 0-bits without inversions, we have (t) 0 if and only if (1)) Topt. At the same time, no bit string with & 0-bits has more than k(n k) inversions, hence, (4) < k(n - k) at all times. During mutation, each of the À offspring is generated independently by copying the parent (1), choosing uniformly at random one of the 1-bits, choosing uniformly at random one of the 0-bits and finally flipping both bits. This flipping can also be viewed as switching both bits, so that bits "move" across the search point in consecutive generations. We will use this abstraction in a later step of the proof.
As we assume the weights to be ordered increasingly, an offspring fitter than its parent if and only if the chosen 1-bit was the left of the chosen 0-bit, meaning, the chosen pair during mutation was an inversion. Since there are k(nn k)) possible pairs in total, we have for each offspring yı, yX
P (f > f()(t)) (t) = 1:n1 k)
15

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
At the same time, switching two bits corresponding an inversion decreases the number of inversions by the difference in their positions, which we call the spread of an inversion. This implies that any offspring fitter than its parent must have fewer inversions than its parent and therefore, X I x( for all f. Note that we cannot make the same statement about the entire group of offspring, meaning, the fittest offspring is not guaranteed to have the fewest inversions. Since (*) is non-increasing we can decompose T into the number of steps required until for the first time the current search point x:(+) has at most k - / inversions and the number of steps required from there until the optimum is found. By linearity of expectation
E = [[]] + [[T2],
where
Ti = in X v k(n - kk)
x
and
T3 inf(t 2 0 x 00 -on inf 2 01 x00 = k(n - k)).
In the remainder of this proof we will demonstrate that each of these two phases requires only an expected O(k(n k))/2) generations.
We begin by bounding the expected number of steps until the search point has at most k(n - k)/A inversions. As computed previously, a single offspring fitter than its parent with probability œςn Since any fitter offspring has fewer inversion than its parent, the potential decreases in a given step, if and only if, at least one of the offspring is fitter. By using that each offspring is generated independently and that s > k((=k) for this phase we get that
A
P[x(++)) < x | x = =1== (1 k(n2 k))
)
RT 21-
This means, in phase we have a constant probability of decreasing the potential every step. However, the resulting constant drift only provides upper bound of Okk(n - k)) via the Additive Drift Theorem (He & Yao, 2004). Improving this constant drift bound is challenging because we must establish lower bound on the expected reduction in the number of inversions, given the existence of fitter offspring. The number of inversions in an offspring is not independent of its fitness, and there is no guarantee that fitter offspring will have fewer inversions than a less fit one. This issue is mitigated when there is only a single fitter offspring (as demonstrated in the proof of phase 2), but becomes problematic when multiple offspring are fitter than the parent with high probability. For example, consider the bit string 1=000 with corresponding weights wi = 1, W12 1002, W3 1003, .... W112 = 1112. If is reasonably large becomes very likely that least one of the children will have the first 1-bit chosen in mutation. This offspring is guaranteed to be the fittest one, but at the same time (assuming the chosen 0-bit is not the last onc) it decreases the number of inversions very little compared to sampling one of the inversions for mutation uniformly at random. We will resolve this difficulty by separate drift argument.
Let Bc be the event that, within the next
2C knnn k)
steps, the number of inversions in (+()) falls below the threshold of
k(n k)
Here, C is chosen such that X < C then the proof of the first phase is established, as Be is expected to occur after a constant number of repetitions.
log(n) If we can demonstrate that Be occurs with a probability of at least some constant,
Henceforth, we will implicitly condition on 8 2 k(n k X, since otherwise, the conclusion follows immediately. By the Chernoff bound over round events, the probability that the potential decreases at most
k(n k)
A
16

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
times within the next
2C knn k)
rounds is sub-constant. We will condition on the event that the potential decreases at least
kn k)
times, and from now on, we will only consider such potential-reducing generations.
If we regard mutation as swapping the 1-bit with the 0-bit, we can enumerate all O-bits from 1 to k and denote by in the current position of the j'th O-bit, which will be referred to as 0y. Note that this enumeration stays fixed across generations, meaning that the relative order can change and 0y is not necessarily the j'th O-bit in x(1), Now define
2
as the random variable that captures the number of 1-bits before Og plus one, or in other words, one plus the number of inversions this specific 0-bit is part of. Let Sj denote the event that the fittest offspring was generated by a mutation that selected Oj and this offspring is fitter than the parent. We continue to show that
Biz 1 s, 2'
We achieve this by systematically revealing the randomness in each generation. First, uncover which 0-bit flip produced the fittest offspring?. Assume this bit Og. Next, reveal all offssring that were generated by flipping other 0-bits than Oy. Let m be the number of offspring that were not uncovered yet, i.e., the number of offspring where 0g was switched. Now enumerate all 1-bits to the left of 0, in 2(1) from right left (here, relative order matters). Let be the smallest integer such that when switching the l'th 1-bit to the left of 0j with 0, the resulting offspring of 2:(() has higher fitness than all A - m previously uncovered offspring. Denote by Dj the corresponding event. Such must exists, since we condition on the event that some offspring with bit 0j flipped (switched) is the fittest among all offspring. Because the weights are sorted increasingly must hold that switching the + 1'th 1-bit with 0, will also result in an offspring with higher fitness than the other X m offspring, while switching the 1'th 1-bit with 0j will result in an offspring with lower fitness than the other offspring. Next, uncover all offspring where bit 0, was switched with one the first up to (I - 1) bit the left of Oj. Let m' denote the number of yet uncovered offspring. Now each of the remaining m' offspring is generated by flipping 0j with one of the I'th to (s 1)tth 1-bits to the left of 0j. Observe that the fittest among them will be the one with the leftest 1-bit chosen. Therefore,
B|z
(2,00 s, S,, Dr, m' offspring not uncovered] E max
D'
U2
i=_.......m
where Uj- ~ Uniform(1, - 1). Given that we are conditioning on Sj. we know that the fittest offspring was produced by flipping 0y. which implies m' 2 1. As/ 2 1 it follows that
E E| 2 = S,52 8/2.
Denote by T the number of steps required until Z, reaches 1, only counting steps where Z, is decreased. Using tail bound for the Multiplicative Drift Theorem (Theorem A.2) we have that
1 P > 2(log(n) + log(n - k))] < -
As l: < (n - k) we conclude by a union bound that with probability at least 1/2 each potential Z₁ will reach 1 within at most 4 log(n) steps. Therefore, with probability at least 1/2, after 4k log(n) generations where some offspring is fitter than
2 MMore precisely, we must uncover which O-bit flip resulted in the offspring selected during the selection process. This accounts for scenarios where multiple offspring have the same highest fitness, in which case one of the fittest candidates is typically chosen uniformly random. As the occurrence of multiple equally fit offspring mere technicality, have largely omitted further discussion of this case.
17

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
the parent, there must be 0 inversions in s0 Note that in practice, there will actually be inversions in 4i, as the condition 2 k(n- kt) / is violated earlier, leading the optimization process enter the second phase. Using the fact that C log(nj and k 2 n/2 we obtain
4k log(n) 8k(n kt) log(n)
ck(n kn(nk) I <C
71
X
Finally, as the probability of having less than Ck(n / "successful" generations in the considered time period is sub-constant, we conclude via another union bound that there exists a constant C' such that event Bc occurs with probability least 1/C". Consequently, we have
E T SCC C² k(n - k) EO
k(n k) (Km kl) A
A
To compute R[[b] we first bound the probability that exactly one of the generated offspring is fitter than the parent. Denote by
A {K (I....,, f(y,) > f(a!1) }
the event that exactly i of the offspring fitter than the parent (+))) As shown earlier, the probability that given offspring is fitter than its parent is exactly k(n-Ry, where represents the number of inversions in (f()), Given that each offspring generated independently, we have for 8 S k(n - k)/A
A-1
P|A, x = 8) = X
8
8
kn - k)
k - k)
kln-k)
X
k(n k)
k: k)
8
2 A k(n k)
Lemma A.4 indicates that when selecting an offspring uniformly at random from all those with higher fitness than the parent (i.e., those generated by flipping an inversion), the expected number inversions in that offspring is at least vs/16 fewer than in the parent. We can now reveal the randomness two steps. First, we only uncover how many the generated offspring are fitter than the parent. Given that there is only a single fitter offspring, i.e., conditioned on A,, we then uncover its number of inversions. Clearly, this single fitter offspring is now sampled uniformly at random from all offspring with higher fitness than (f); thus, for I k(n /)/
(1) (s) = E[x (1+1) (1+1)
(1)) x(1) = =s
E E[x x((1) x((1 = =, AR] IP[A x(() = s]
(((++)))
x(() x 8, A1] P[A X s]
v8
16
k(n k)
Finally, applying Johannsen's Variable Drift Theorem (Johannsen, 2010) (Theorem A.3) yields
k(n
k)
16e k(n k)
ET4 < 16e
+
E
3/2 do
k( k)
k(n - k)
1
< 16e
1+
3/2 do
k(n - k))
E
A
18

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
B. Evolutionary Search Parameter Ablations
B.1. Mutation Rate (Depth Pruning)
The mutation rate plays a crucial role in balancing exploration and exploitation. A higher mutation rate allows for broader exploration of the search space; however, this space grows exponentially with the number of mutations. As a result, when trying approach the optimum in terms of Hamming distance, the proportion of "good" offspring decreases significantly with increasing mutation rate. Consequently, in smooth fitness landscape, expect significantly faster optimization with lower mutation rate.
To provide some mathematical intuition, consider optimizing over the 200-dimensional hypercube {0, 1,200, current search point is z²(() = 0200 and the global optimum is Fopt = 1 1 For this illustration we use a mutation operator that randomly selects a subset of k bits to flip. Flipping bits corresponds to selecting a bitstring from the k'th Hamming layer of x²(t) 0200 uniformly at random, where the 4'th Hamming layer consists of all bitstrings with a Hamming distance of k from (t()), Similarly, the Hamming ball of radius Is includes all bitstrings with a Hamming distance at most k. Assume that any bitstring closer to the optimum in terms of Hamming distance has higher fitness than our current search point'. The probability of improving the fitness via a mutation of k bits equals the fraction of points in the k'th Hamming layer of (e()) that are also in the Hamming ball of radius 19 around Topt. When the current search point is reasonably close to the optimum, this ratio is maximized for k = 1. For the described setting, we can compute the probabilities of each event Ag, where Aş represents the event that mutating & bits of (+(() results in a decrease in the Hamming distance from Togt. These probabilities are given by:
200, where the
220
1P[A1]
0.1
IP/ [2]
RJ 0.0095
(20) (20)
(100)
(20) (180)
P[Aa]]
FC 0.0269
P[A,]
RC 0.0032
(2)))
2000
(30 (3?)
(110)
(39)
(180)
20
(0
(150)
(20)
(180)
PP[ [P[As]
R 0.0076
[P[A]] =
2 0.0010
(200)
(200)
Note that accounting for the potentially greater Hamming distance gained for higher mutation rates (i.c., calculating the drift in the Hamming distance) has only marginal effect. This is because for an odd number of mutations, most of the conditional probability mass is concentrated on the case where the Hamming distance is reduced by just one bit. Similarly, for an even number of mutations, most of the conditional probability mass is concentrated cases where the reduction in the Hamming distance is only two bits. The advantage of a low mutation rate becomes even more pronounced as the search process nears the optimum. For instance, when the Hamming distance between (*) and Tept is 5, mutating single bit results in 16-fold greater drift in Hamming distance compared to any other mutation rate.
To study the empirical impact of the mutation rate on our search process, we tested various distributions from which the number of mutations sampled. Table 4 illustrates the effects of these distributions for selecting the optimal twelve blocks to drop for Mistral-7B--0.33 The results confirm our intuition: higher mutation rates generally reduce performance. However, sampling from the minimum of two uniform distributions ensures reasonably high probability of choosing low number of mutations. These offspring, with fewer mutations, then drive the optimization process, yielding comparably lower performance drops. Conversely. when we eliminate this sampling and instead use a high, constant mutation rate, we lose the locality that is crucial for evolutionary algorithms, leading to a significant drop in performance.
low mutation rate carries the risk of getting trapped local optima. However, as discussed in Section 3, we expect dynamic model compression to exhibit a smooth fitness landscape with few local optima. Moreover, fitness evaluations in our context are relatively expensive. Increasing the mutation rate would only be beneficial if the smaller search space had already been thoroughly explored. In our case, though, even a small neighborhood cannot be fully explored within a feasible time frame.
A widely used strategy for balancing the advantages and disadvantages of different mutation rates involves self-adjusting mutation rates, which have been shown to be effective both theoretically and in practice (Kern et al., 2004; Doerr et al., 2019). These methods decrease the mutation rate when progress is relatively "easy", and increase it when progress becomes difficult, offering a greater chance of escaping local optima.
Whiie this assumption does not hold in practice, it serves as a useful intuition in reasonably smooth fitness landscape.
19

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 4. Effect of varying the distribution determining the number of mutations.
Number of Mutations
Wiki2
C44
FW4
min(U1, U2), min(U1, U2),
U2,U2 (11,3)
17.52
21.60
16.79
U,,02~ U(1,7)
21.49
22.41
17.65
min(U1, U2),
U1,02~00 (11,1)
18.65
22.67
17.63
I
18.12
21.12
16.33
3
22.09
25.42
19.25
7
25.06
26.52
19.65
15
27.01
28.19
22.03
B.2. Multi-Step Selection (Unstructured Sparsity)
We will use this subsection to ablate the impact of hyperparameters for the multi-step selection, namely, the number of tokens and survivors. As discussed earlier Section 4.2, the default hyperparameters we chose for our unstructured sparsity search were quite conservative. The following experiments will be conducted based on the "super-fast" version, which uses two steps of selection. It first generates 16 offssring, evaluates them 512 tokens, and compares only the fittest one with the parent another 8192 tokens.
Table 5 shows the impact of adapting the number of tokens in the first selection step. Note that reducing tokens is only reasonable up to certain degree, as fitness evaluation has constant overhead independent of the number of tokens (e.g., for loading the levels). Table 6 ablates the number of offspring in each generation. All perplexities were measured after 400 generations.
Table 5. Effect of varying the number of tokens in first preselection step.
Offspring Stage 1: Tokens Stage 2: Tokens
Wiki2,
C44
FWL
16
1024
8192
16.22
17.93
12.26
512
8192
15.87
18.28
12.38
16
256
8192
17.25
18.51
12.52
128
8192
16.01
18.99
12.72
16
64
8192
15.89
19.35
12.98
Table 6. Effect of varying the number of offspring.
Offspring
Stage Tokens
Stage Tokens
Wiki2|
C41
FW]
64
512
8192
16.35
18.27
12.36
32
512
8192
16.65
18.22
12.44
16
512
8192
15.87
18.27
12.38
8
512
8192
16.37
18.74
12.64
512
8192
17.87
18.97
12.72
In similar vein the discussion in Appendix B.I, the number of offspring can also be dynamically adapted. Ideally, the number of offspring should increase the point where the computational effort compensated by the number of generations, outlined in Theorem 3.1. Methods such the belf-Adjusting A)EA have recently gained significant theoretical interest and have been shown to automatically determine "ideal" offspring sizes on specific problems (Hevia Fajardo & Sudholt, 2021; Kaufmann et al., 2022).
B.3. Fitness Environment (Quantization)
We explored an alternative fitness function by testing perplexity opposed to KL--Divergence. One advantage of using perplexity is the reduced memory requirement, as it does not necessitate storing the logits, which can be particularly burdensome for large vocabularies. However, perplexity relies solely on the information from the ground truth token, while KL-Divergence takes into account the entire distribution. This distinction is significant only the selection decisions vary between the two metries. Generally, we expect Kl.-Divergence to perform at least as well as perplexity; however, in many
20

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
instances, their performances similar. This observation could indicate that KL-Divergence might be using more tokens than necessary assess fitness effectively. Although in the context of quantization KL.-Divergence yielded slightly better results (Table 7, Figure 5 left), both metrics showed comparable performance when applied to unstructured sparsity (Figure 5 right).
Lamm---78-
Liam a-m--2278. Sparsity
6.90
Perplexity
Perplexity
6.88
Divergence
24
Divergence
22
6.86
Perplexity
Perplexity
20
6.84
18
6.80
14
12
6.78
40
80
100
100
150
200
250
300
350
400
Generation
Generation
Figure Convergence EvoPress unstructured sparsity (left) and quantization (right) for different fitness functions.
Table 7. Comparison of using KL- KL-Divergence Perplexity fitness function.
Model
# Bits
Method
Wikiz
C44
FW4
Uniform
12.19
15.76
11.47
EvoPress (PPL)
8.17
12.15
9.64
EvoPress (KL)
7.49
12.03
9.56
Llama-3-8B
Uniform
6.48
9,50
8.46
EvoPress (PPL)
5,86
9,46
8.23
EvoPress (KL)
5.86
9.44
8.22
Uniform
6.16
7.96
6.86
EvoPress (PPL.)
5.74
7.90
6.79
EvoPress (KI.)
5.70
7.87
6.76
Llama-2-7B
Uniform
5.48
7.10
6.40
EvoPress (PPL)
5,25
7.09
6.37
EvoPress (KL)
5.22
7.07
6.34
Uniform
5.54
8.57
6.96
EvoPress (PPL)
5.23
8.45
6.87
EvoPress (KL)
5.21
8.42
6.86
Mistral-7B-v0.3
Uniform
5.10
7.87
6.50
4
EvoPress (PPL)
4.85
7.86
6.49
EvoPress (KL)
4.84
7.84
6.48
21

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
C. Experimental Setup
C.1. Hyperparameter Setting
Here, we provide an overview of the hyperparameters used in our experiments. As shown in Table 8, we employed different choices for the number of tokens, offspring, and generations for different applications to account for the size of the respective search space. However, the search is very robust with respect to these choices, and using one set of hyperparameters for another application yields similar results. This is also demonstrated through the "super-fast" version, which uses drastically different hyperaarameters, but achieves comparable performance (see Figure 3 in main text).
Across all upplications. we sampled the number of mutations from the distributions min(U1, U2) with U1, U2 ~ 31,3, which closely mimics the behavior of using only one mutation (see the ablation study Appendix 4).
For Depth Pruning, where each block has only two choices and significantly fewer blocks are present compared to layers in other methods, we leveraged the insight from Theorem 3.1, which suggests that the number required generations scales proportionally to k(n - k), where A represents the number of removed blocks and the total number of blocks.
For Unstructured Sparsity, the search space is considerably larger, with more than 10 choices per layer'. As a result, more generations are necessary to converge because each generation only makes small improvement in terms of Hamming distance from the optimum.
For Quantization, the search space somewhat smaller since fewer "natural" compression levels are available. However, the fitness landscape less smooth, with significantly larger step sizes in compression levels, motivating the use of a higher number of tokens.
For all these applications, we adopted a conservative approach to the number of generations to better understand convergence. In practice, we need significantly fewer generations to converge close to optimum, as demonstrated in Section 4.2, Appendix D.2, Appendix F.2, and Appendix B.3. Additionally, we showed much faster version (in terms of time per iteration) that uses significantly less tokens and has fewer offssring.
Table 8. Employed hyperparameters different applications.
Application
Generations
Offspring Survivors (1)
Tokens (1)
Survivors (2)
Tokens (2)
Survivors (3)
Tokens (3)
Depth Pruning
k(n /1.5
32
2
2048
32768
N/A
N/A
Unstr. Sparsity
400
64
8
2048
16384
1
65536
Quantization
150
128
16
2048
16384
1
131072
Super-Fast
400
16
1
512
8192
N/A
N/A
C.2. Robustness to Random Seed
To evaluate the robustness of EvoPress, we conducted 16 independent runs with different random seeds. Specifically, we used the "super-fast" variant to determine the optimal compression allocation for Llama-3-8B at 70% sparsity, assessing perplexity scores on the C4, Wikitext2, and hold-out Fineweb-Fdu datasets. The results indicate that EvoPress is highly robust, as reflected by the low standard deviation observed across the hold-out metrics (Figure 7). For example, after 1000 generations of the "super-fast" variant, the configurations found achieve mean C4 perplexity of 33.82 with standard deviation of 0.61, compared to 52.32 for the next best method, OWL. This improvements achieved by EvoPress therefore statistically relevant. Furthermore, as shown in Figure 6, the configurations identified across different runs are highly similar, which is expected to improve further with additional generations.
*If needed, one could increase the step size and reduce the number compression levels load.
22

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Llama-3-8B 70% Sparsity
Seed:
Seed:
Seed: 3
0
1.0
10
10
10
0.9
20
20
20
0.8
30
30
30
Sparsity
d
d
d
0.7
Seed:
Seed:
Seed:
0
0
0.6
0.5
10
10
10
0.4
0.3
20
20
20
0.2
30
30
30
g
9
9
9
Figure Configurations identified by EvoPress on Llama-3-88 after 1000 generations show high similarity across different seeds. The y-axis represents the depth of the respective transformer block, while the x-axis denotes the corresponding layer (q: query, k: key, value, o: output, u: MLP up. g: MLP gate, d: MLP down).
Llama-3-8B 70% Sparsity
60
Wikitext2
C4
Fineweb-Edu
Perplexity
30
200
400
600
800
1000
Generations
Figure 7. Convergence behavior of the "super-fast" variant across 16 independent runs. The extremely low standard deviation (shaded area) demonstrates the robustness the method, which suggests that local optima not pose significant challenges to the search.
23

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
D. Additional Depth Pruning Results
D.I. Full Results
Here, we present our additional results for depth pruning experiments on Mistral-7B-vO.3 (Table 12), Llama-2-7B (Table 9), Llama-3-8B (Table 10), and Llama-3.1-8 1--BB (Table 11). Across all levels of sparsities, EvoPress consistently outperforms previous methods. Additionally, Table 12 includes results where only entire transformer blocks removed by EvoPress. This showcases that the significant gains not primarily due this relaxation, and that our method performs better than baselines even when dealing with this coarser search space.
Table 9. Depth pruning of Llama-2-7B.
Sparsity
Method
Wiki2
C4.
FWJ
0%
Dense
5.21
6.93
6.40
EvoPress
6.42
8.60
7.54
ShortGPT
8.86
10.78
9.30
12.5%
Cosine Similarity (Window)
7.53
9.82
8.51
Weight Subeloning
9.09
11.06
9.60
ShomtenedLlama
7.68
10.44
8.57
EvoPress
9.15
11.46
9.69
ShortGPT
23.41
30.30
21.16
25%
Cosine Similarity (Window)
16.60
21.04
17.37
Weight Subcloning
23.41
30.30
21.16
Shortened Llama
13.86
14.08
11.81
EvoPress
17.98
18.91
15.53
ShortGPT
70.94
63.51
54.07
37.5%
Cosine Similarity (Window)
192.07
212.60
151.10
Weight Subcloning
70.94
63.51
54.07
Shortened Ilama
35.37
26.07
20.37
EvoPress
48.84
42.29
33.57
ShortGPT
226.14
171.04
180.51
50%
Cosine Similarity (Window)
4570.15
2876.83
1861.06
Weight Subcloning
226.14
171.04
180.51
Shortened Llama
145.78
87.40
68.79
24

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table Depth pruning of Llama-3-8B.
Sparsity
Method
Wiki2|
C44
FWJ
0%
Dense
5.54
8.80
7.62
EvoPress
7.72
12.61
10.15
ShortGPT
13.21
19.56
14.25
12.5%
Cosine Similarity (Window)
9.54
14.87
11.64
Weight Subcloning
13.21
19.56
14.25
Shortened Llama
9.42
15.09
11.57
EvoPress
13.99
22.83
15.84
ShorGPT
5527.54
11589.93
2346.13
25%
Cosine Similarity (Window)
5519.95
11629.61
2342.91
Weight Subcloning
5527.54
11589.93
2346.13
Shortened Llama
16.59
20.81
16.28
EvoPress
27.56
35.70
26.77
ShorGGPT
64281.36
13836.12
3789.09
37.5%
Cosine Similarity (Window)
64627.29
13890.14
3784.72
Weight Subcloning
64381.36
13836.13
3789.09
Shortened Llama
50.20
61.56
37.40
EvoPress
84.99
87.86
66.41
ShortGPT
1663.97
1740.04
1588.20
50%
Cosine Similarity (Window)
2053.19
1116.47
694.00
Weight Subcloning
1663.97
1740.04
1588.20
Shortened Llama
724.86
666.41
210.30
Table 11. Depth pruning of Llama-3.1-88 1--8B.
Sparsity
Method
Wiki24
C41
FWL
0%
Dense
5.61
8.90
7.67
EvoPress
7.58
12.24
10.00
ShortGPT
12.54
19.21
13.76
12.5%
Cosine Similarity (Window)
12.54
19.21
13.76
Weight Suhcloning
12.54
19.21
13.76
Shortened Llama
9.27
14.80
11.21
EvoPress
11.59
17.84
13.96
ShorGGPT
4278.39
6754.92
1512.39
25%
Cosine Similarity (Window)
4278.39
6754.92
1512.39
Weight Subcloning
4278.39
6754.92
1512.39
Shortened Llama
20.41
20.33
16.12
EvoPress
24.98
35.77
25.93
ShortGPT
123044.19
22071.51
6059.03
37.5%
Cosine Similarity (Window)
123044.19
22071.51
6059.03
Weight Subcloning
123044.19
22071.51
6059.03
Shortened Llama
41.34
43.53
31.00
EvoPress
105.84
110.69
61.25
ShorGGP
1630.11
1680.21
1698.64
50%
Cosine Similarity (Window)
1881.54
1196.63
683.24
Weight Subcloning
1630.11
1680.21
1698.64
Shortened Llama
454.96
309.42
153.96
25

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 12. Depth pruning of Mistral-7B-vO.3.
Sparsity
Method
Wiki2
C4
FW
0%
Dense
4.82
7.72
6.41
EvoPress
6.06
9.00
7.42
EvoPress (Att)+MLL))
6.33
9.44
7.80
12.5%
ShortGPT
7.19
10.18
8.46
Cosine Similarity (Window)
7.19
10.18
8.46
Weight Subcloning
7.19
10.18
8.46
Shortened Llama
6.64
9.71
7.94
EvoPress
8.66
12.04
9.92
EvoPress (Attn++MP))
9.46
13.02
10.59
25%
ShortGPT
43.26
40.16
29.54
Cosine Similarity (Window)
33.75
54.07
36.26
Weight Subcloning
43.26
40.16
29.54
Shortened Llama
14.94
19.30
14.73
EvoPress
17.52
21.60
16.90
EvoPress (Attn+MML))
21.62
25.17
18.97
37.5%
ShorGPT
2898.98
2722.66
981.99
Cosine Similarity (Window)
1034.09
2471.86
1050.56
Weight Subcloning
2898.98
2722.66
981.99
Shortened Ilama
440.20
442.09
486.15
EvoPress
61.75
54.15
43.23
EvoPress (Attn+MMLP)
108.91
99.74
69.07
50%
ShorGGPT
2422.72
2134.92
1083.51
Cosine Similarity (Window)
3411.47
1934.16
1740.91
Weight Subcloning
2422.72
2134.92
1083.51
Shortened Llama
5241.76
3595.71
1953.14
26

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
D.2. Practical Convergence
EvoPress identifies superior compression profiles in highly efficient manner. Figure 8 displays that the evolutionary search produces better compressed models than previous techniques in matter of minutes, with full convergence in around half an hour.
Mistral-7B-v0.3 Blocks Dropped
Mistral-7B-v0.3 -66 Blocks Dropped
800
2.5
Perplexity (Test)
Perplexity (Test)
KL Divergence (Train)
-0.8
700
WWW
KL Divergence (Train)
2.4
16-
2.3
Perplexity
GPU Hour (RTX 3090)
-0.
KL--Divergence
Perplexity
500
GPU Hour (RTX 3090)
2.2
Goverger
14
400
WM
2.1
Mmn www M
300
2.0
00.5
200
1.9
100
10
0.4
1.8
20
40
60
80
100
120
25
50
75
100
125
150
175
Generation
Generation
Figure 8. Convergence of EvoPress when removing 8 transformer blocks (left) and 16 transformer blocks (right) of Mistral-7B-v0.3.
D.3. Locality of Dropped Blocks
Prior research indicates that deeper layers, aside from the final ones, are generally less effective (Gromov et al., 2024; Men al., 2024). Figure 9 illustrates the optimal removal configurations identified by EvoPress. For comparison, Table 13 displays the removal order of prior scoring methods. While EvoPress indeed removes some deeper layers across all sparsities, also observe that certain shallow layers appear to be less important. Notably, "two hills" pattern emerges in many cases, where blocks before and after the midpoint are pruned, yet the central blocks remain intact. Meanwhile, the first two blocks are never pruned. However, in contrast to heuristic proposed by Ma et al. (2023), we find that, in some instances, it is effective to prune the final block as well.
Table 13. First blocks in removal order of ShortGP,, Weight Subeloning and Shortened Llama on different models.
Model
Method
Removal Order (Leff to Right)
ShortGPT
26, 25, 24, 27, 23, 22, 28, 30, 21, 29, 20, 19, 13, 17, 18, 12
Mistral-7B-v0.3
Weight Subcloning
26, 25, 24, 27, 23, 28, 22, 30, 21, 29, 20, 19, 13, 17, 12, 18
Shortened Llama
10, 12, 13, 11, 08, 09, 14, 15, 07, 06. 04, 27, 24,
16, 25, 05
ShortGPT
27, 25, 26, 28, 29, 24, 23, 22, 21, 30, 20, 19, 18, 17, 15, 14
Llama-2-7B
Weight Subcloning
27, 28, 29, 26, 24, 23, 22, 21, 19, 30, 20, 18, 17, 14, 15
Shortened Llama
11, 12, 08, 09, 10, 06, 24, 25, 07, 14. 23, 13, 22, 21, 15. 27
ShortGPT
25, 26, 27, 24, 28, 23, 22, 29, 20, 21, 19, 18, 30, 17, 16, 11
Llama-3-8B
Weight Subcloning
25, 25,27. 27, 26, 24, 28, 23, 22, 29, 20, 21, 19, 18, 30, 17, 16, 11
Shortened Llama
10, 08, 09, 11, 26, 25, 12, 22, 24, 23, 14, 13, 28, 06, 19,
ShortGPT
25, 26, 24, 27, 23, 28, 22, 29, 20, 21, 19, 18, 17, 30, 16, 10
Llama-3.1-88
Weight Subcloning
25, 27. 26, 24, 28, 23, 22, 29, 20, 21, 19, 18, 30, 17, 16, 10
Shortened Ilama
10, 09, 11, 08, 26, 25, 12, 24, 22, 23, 14, 28, 06, 13, 19,
27

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Dropped
Dropped
Uama-3-88
Lama-3-88
Mistral-8--00.3
Mistral-7B-0.3
Subblocks Drapped
Subblocks Dropped
Depth
Depth
Dropped
Dropped
2
Liama-2
2
Liama
Llama 8B
Subblocks Dropped
Mistral-7B.. 78
Subblocks Dropped
Mistral 78.vD.3
Depth
Depth
Figure Optimal removal configurations identified by EvoPress for difffrrnnt models.
D.4. Correlation of Scores with Perplexity
In this experiment, we first calculated the cosine similarity and squared error for each block by comparing activations before and after the block. Next, we randomly removed subsets of blocks (excluding the first and last two) and for each configuration, computed the average cosine similarity and squared error. The results shown in Figure 11. Initially, the average squared enror exhibited negative correlation, as the f2-orrm of the activations increased with depth. This led configurations with early blocks removed having small average error. To mitigate this, we normalized the activations prior to computing the squared error, which significantly improved the correlation, resulting in performance comparable to cosine similarity. However, as sparsity increased, the correlation degraded significantly for both methods, offering insight into why removal techniques based on scoring fail even at moderate levels of sparsity. Meanwhile, when removing only small number of blocks, the average perplexity when removing each block separatly is strong predictor of the performance after removing an entire set of blocks, depicted in Figure 10. We conclude that monotonicity holds at smaller compression levels, but de- cays rapidly at medium sparsity. The experiments were done using 131,072 tokens from the Fineweb-Edd calibration dataset.
28

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Figure 10. Effect of removing random subsets of 2, 4, and 6 blocks for 1-8ma-3-8B.. The x-axis depicts the average perplexity when removing each block separately, while the y-axis depicts the perplexity after removing the entire of blocks. Error monotonicity holds for smaller compression levels, but decays with incrcasing sparsity.
Llama-3- 3-88
Blocks Dropped
Llama-3-88 Blocks Dropped
Llama-3-88 Blocks Dropped
100
13
25
80
Perplexity
Perplexity
Perplexity
20.
60
11-
10-
20
8.8
9.0
922
9.4
9.0
9.2
9.4
9.0
9.1
9.2
9.3
9.4
9.5
Average Perplexity
Average Perplexity
Average Perplexity
Llama-3-8B Blacks Drapped
Llama-3-8B 10 Blacks Dropped
Llama-3-88 12 Blocks Dropped
Perplexity
Perplesity lexiity
Perplexity
1.0
10t-
08c 0.50
Average Similarity
Average Similarity
Average Similarity
Llama-3-8B- 8 Blocks Dropped
Llama-3-88 10 Blocks Dropped
Llama-3-88 12 Blocks Dropped
Perplexity
Perplexity
Perplexity
10
200
1.0
150
100
1/0
130
150
Average Squared Error
Average Squared Error
Average Squared Emor
Llama-3-8B- 8 Blocks Dropped
1lama-3-8B- 10 Blocks Dropped
Llama-3-88 12 Blocks Dropped
10a
Perplexity
Perplexity
Perplexity
10a
1.7-
302
10a-
11
13 14
12
já
12
14
Average Squared Eror (normalized)
Average Squared Error (normalized)
Average Squared Error (normalized)
Figure 11. Effect of removing random subsets of 8, 10, and 12 blocks for Llama-3-8B. The x-axis depicts the average cosine similarity (first row), average squared error (second row), and the average squared error after normalization (third row) when removing cach block separately. while the y-axis depicts the perplexity after removing the entire set of blocks. Error monotonicity breaks at medium sparsity rates, providing an explanation the failure of scoring. based methods at these sparsity levels.
29

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
E. Additional Unstructured Sparsity Results
E.1. 50% and 60% Sparsity
In the main text, we focused on results at 70% sparsity, where the performance differences are more pronounced. However, since 50% and 60% sparsity levels are more practical and frequently referenced in the literature, we present the corresponding results in Tables 14 and 15. We have also included Llama-2-7B in these tables for legacy purposes. Even these lower sparsity levels, EvoPress demonstrates significant improvements uniform sparsity and consistently outperforms OWL.
Table 14. Performance of various sparsity profiles at 50% sparsity.
Model
Method
Wiki2
C44
AreCt
ArcEt
HSt
PiQAt
WG1
Avg
Dense
4.82
7.72
48,9
79.6
60.9
80.3
73.9
68.7
Uniform
5.68
8.93
43.7
76.7
55.7
78.4
71.0
65.1
Mistral-7B-v0.3
OWL
5.69
8.94
43.9
76.9
55.4
78.5
70.3
65.0
EvoPress
5.49
8.70
45.7
77.3
56.5
78.9
71.2
65.9
Dense
5.12
6.93
43.4
76.3
57.1
78.1
69.0
64.8
Uniform
6.40
8.87
41.3
73.4
52.8
75.7
68.8
62.4
Llama-2-7B
OWL
6.38
8.77
41.1
73.2
53.2
76.5
70.2
62.9
EvoPress
6.22
8.52
41.5
74.2
54.0
76.7
69.6
63.2
Dense
5.54
7.10
50.4
80.1
60.2
79.7
72.6
68.6
Uniform
8.05
13.07
43.6
75.7
54.2
76.
71.7
64.3
Llama-3-8B
OWL
8.13
13.12
43.8
75.8
54.0
75.7
72.2
64.3
EvoPress
7.63
12.53
43.9
77.5
54.5
76.8
72.2
65.0
Dense
5.61
8.90
51.2
81.4
60.0
80.1
73.9
69.3
Uniform
8.06
13.03
44.5
76.7
54.0
76.7
71.5
64.7
1lama-3.1-88
OWL
8.02
12.99
44.2
76.5
53.8
76.8
72.5
64.8
EvoPress
7.51
12.31
46.6
77.7
54.9
77.6
71.7
65.7
30

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 15. Performance of various sparsity profiles 60% sparsity.
Model
Method
Wiki2.
C44
AreC1
ArcEt
HSt
PiQAt
WG1
Avg
Dense
4.82
7.72
48.9
79.6
60.9
80.3
73.9
68.7
Uniform
7.78
11.86
38.0
72.3
49.4
75.0
69.3
60.9
Mistral-7B-v0.3
OWL
7.50
11.34
38.5
71.9
49.6
75.1
70.2
61.1
EvoPress
7.08
10.27
40.5
72.8
51.9
76.9
68.8
62.2
Dense
5.12
6.93
43.4
76.3
57.1
78.1
69.0
64.8
Uniform
9.3
12.37
35.8
69.5
45,9
72.4
65.9
57.9
Llama-2-7B
OWL
8.35
11.00
36,0
69.1
47.5
73.2
66.2
58.4
EvoPress
8.21
10.34
37.1
70.6
49.3
74.4
67.6
59.8
Dense
5.54
7.10
50.4
80.1
60.2
79.7
72.6
68.6
Uniform
13.86
21.43
35.2
69.7
45.6
72.2
68.0
58.2
Llama-3-8B
OWL
12.37
18.53
38.0
70.3
47.7
72.1
68.5
59.3
EvoPress
11.02
16.37
39.0
71.9
48.6
74.0
69.1
60.5
Dense
5.61
8.90
51.2
81.4
60.0
80.1
73.9
69.3
Uniform
13.43
21.46
36.4
69.7
46.2
72.3
67.7
58.5
Llama-3.1-8B
OWL
12.08
18.25
38.9
71.1
47.7
73.1
68.8
59.9
EvoPress
10.58
15.96
40.0
72.5
49.0
74.6
69.5
61.1
E.2. Sparsity Profiles
Below, we visualize sparsity profiles determined by EvoPress and baseline approaches. Notably, EvoPress prunes the initial blocks less aggressively than the middle and later blocks. Additionally, the q proj projection attains higher sparsity levels, whereas the v-proj projection is pruned to significantly lower sparsity average. Although Figure 12 may suggest that OWL and EvoPress produce similar sparsity profiles, this is misleading OWL enforces uniform sparsity at block level, as their original per-layer approach underperformed (Yin et al., 2024).
EvoPress
0.80
OWL
0.75
unfform
0.75
Average sparsity
Sparsity
0.70
0.65
0.60
0.65
0.55
0.60
0.50
0.45
0.55
5
10
15 Depth
25
30
0,40
q_pro)
k_pro]
v.prol
0_pro)
gate_pro
up_proj
down_pro)
Figure 12. Comparison of different block-level sparsity profiles for Llama-3.1-88 70% sparsity.
Figure 13. Average sparsity per projection type for Llama-3.1 70% sparsity for EvoPress.
F. Additional Quantization Results F1. 2.25 Bit and 2.5 Bit
In addition to the bit results presented in Section 4.3, we further evaluated EvoPress under extreme quantization conditions. specifically testing it at 2.25 bit and 2.5 bit levels. As a baseline, we generated 32 random configurations combining bit and 3 bit layers and selected the best performing setup. The results, as shown in Table 16, demonstrate that EvoPress significantly outperforms this baseline, highlighting ability to achieve extreme quantization levels.
31

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 16. Performance of EvoPress 2.25 bit and 2.5 bit quantization.
Model
Bits
Method
Wikiz,
C44
AreCt
AreEt
HS1
PiQAt
WGt
Avgt
Best
11.53
18.32
30.1
59.6
69.4
56.8
52.1
2,25
EvoPress
13.47
36.2
66.0
49.3
74.2
63.5
57.8
Mistral-7B-vO.3
Best of 32 EvoPress
7.50
11.76
37.0
68.0
51.7
75.0
63.5
59.0
2.5
6.60
10.40
39.8
71.7
54.0
77.1
65.8
61.7
Best of 32 EvoPress
13.18
18.19
24.8
50.2
40.3
66.8
56.1
47.7
2.25
9.82
9.93
29.5
61.8
46.2
70.3
59.4
53.4
Llama-2-7B
Best of 32
9.42
9.01
29.1
58.6
46.9
70.1
62.6
53.5
2.5
EvoPress
8,03
7.33
35.3
68.4
50.8
73.9
64.2
58.5
Best of 32 EvoPress
149.85
432.96 23.93
21.2
29.1
28.1
55.6
49.8
36.8
2.25
43.17
23.6
46.9
39.3
63.6
56.5
46.0
Llama-3-8B
Best EvoPress
21.65
23.92
47.6
41.2
56.2
47.1
2.5
13.93
18.15
31.7
61.5
47.9
71.7
64.3
55.4
Best of 32 EvoPress
259.61
181.36
20.7
31.9
30.6
57.0
51.9
38.4
2.25
22.75
33.58
26.7
48.9
40.2
63.4
55.7
47.0
Llama-3.1-8 1-88
Best of 32 EvoPress
35.33
37.09
24.1
48.4
41.7
62.7
54.5
46.3
2.5
11.73
19.03
32.2
63.3
47.5
71.8
62.3
55.4
Best of 32 EvoPress
14.20
18.19
28.9
46.8
40.0
61.8
53,1
46.1
2.25
10.48
14.60
36.2
62.0
46.6
66.2
55.6
53.3
Phi-3-Medium
Best of 32 EvoPress
8.26
12.65
40.5
69.3
50.3
70.9
61.9
58.6
2.5
7.12
11.23
44.1
75.9
54.1
73.5
64.6
62.4
E2. Practical Convergence
Similar to unstructured sparsity, EvoPress also demonstrates rapid convergence when applied to quantization. As shown in Figure 14, the majority of improvements occur within two GPU hours, with full convergence achieved after approximately eight GPU hours. If needed, this optimization time could be further shortened by tuning the hyperparameters, similarly to the "super-fast" version for unstructured sparsity discussed in Section 4.2. However, we observed that the convergence dynamics are less smooth compared to unstructured sparsity, likely due to the limited number of quantization levels available (practically only and 4 bit are used), which results a less smooth fitness landscape.
Llama-3.1-8B 2.25 bit
Llama-3-88 bit
120
2.4
11.5
0.325
Perplexity (Test)
Perplexity (Test)
100
KL-Divergence (Train)
2.2
KL-Divergence (Train)
0.300
-2.0
KL-Diiergent Divergente
11.0
GPU Hour (RTX 3090)
0.275
KL-Divergence
Perplexity
GPU Hour (RTX 3090)
1..8
snivergen
Perplexity
0.250
1.6
10.5
60
0.225
1..
40
-1.2
10.0
0.200
-1.
0.175
9.5
20
40
60
80
100
120
140
25
50
75
100
125
150
Generation
Generation
Figure 14, Convergence of EvoPress for 2.25 hit quantization on Tlama-3.1-88 3-B (left) and 3 bit quantization on Llama-3-8B (right).
F.3. Quantization Profiles
In this section, we visualize a quantization profile determined by EvoPress. As shown, EvoPress maintains a relatively uniform quantization bitwidth allocation across the model. Interestingly, while the second and the two last blocks are less compressed, the first block undergoes significantly higher compression. This suggests that "saliency" does not directly
32

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
equate block importance but rather depends the specific compression method used. Additionally, similar the profiles for unstructured sparsity, observe transfer of capacity proj.
3.3
3.6
EvoPress
uniform
3.4
Average bitwidth
3.2
3.0
Bits
2.8
2.9
2.6
2.8-
2.4
2.2
Depth
2.0
v_Drcl
o.pral
gate_pro)
down_end
Figure 15. Block-level quantization profiles for Llama-3.1-80 at 3 bit compression on average.
Figure 16. Average bitwidth per projection type for 1lama-3.1- 8B at bit compression on average.
G. Running Time Comparisons
In main results, did not include stopping criteria into the optimization process, but instead continued the search well beyond convergence to observe the convergence behavior. Here, we include simple stopping criteria to our search, which terminates the search whenever the perplexity on the training set has not improved in the last 20 generations. Based on this version we compare the runtime requirements of EvoPress with baseline methods. However, it should be noted that the amount of data used impacts both runtime and compression quality (with specific trade-off's being methood-eppndent), meaning one can generally trade-off both objectives, which makes runtime comparison difficult.
In Table 17 compare EvoPress for block dropping to various baseline methods Llama-2-7B, where we, additionally the main results, also compare against the iterative search methods BlockPruner (Zhong et al., 2024) and SLEB (Song et al., 2024). As the results show, EvoPress achieves the best compressed models across all sparsity ratios, while being quicker than other search methods. Scoring-basee methods are considerably faster than the search-baae based methods, but produce significantly worse models.
Table 18 we provide a comparison of EvoPress with BESA (Xu al., 2024), method for non-uniform sparsity allocation. BESA performs gradient-descent based search for rowwise sparsity allocation using straight- strough-estimation (Bengio al., 2013), with uniform sparsity per-block level. Therefore, both methods are not fully comparable, as EvoPress searches for layerwise sparsities globally (across the entire model, not just blockwise), while keeping per-row sparsities uniform. We demonstrate that both methods are complementary by performing a two-stage search. first, for blockwise sparsities using EvoPress, then, for rowwise sparsities with fixed per-block sparsity using BESA. This merged version produces best results on 50% and 60% sparsity, while performs slightly worse than pure EvoPress on 70% sparsity. For fair comparison, we search on top a Wanda (Sun al., 2024) layer database for all methods. The runtime requirements for both BESA and EvoPress are similar, as there an additional hyperparameter sweep for BESA, which is not provided in the author's implementation (authors provide tuned coefficients). Similarly, have not included OWL in this comparison, as the authors have not provided instructions how hyperparameters were obtained.
33

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 17. Comparison of various depth pruning methods different sparsities. We report the runtime single NVIDIA RTX 4090
GPU. Note that some methods perform forward passes on a strongly depth-pruned model, which explains the runtime difference for EvoPress 50% and 12.5% sparsity.
Sparsity
Method
Fineweb.
Wiki2
C4
Runtime
Fwd. Passes (Tokens)
0%
6.40
5.12
6.99
12.5%
ShortGPT
9.30
8.86
10.78
46s
65K
12.5%
Cosine Similarity (Window)
8.51
7.53
9.82
65K
12.5%
Weight Subcloning
9.60
9.09
11.06
49s
65K
12.5%
ShortenedLlama
8.57
7.68
10.44
8min
2.1M
12.5%
SL.BB
7.71
6.51
8.78
70min
18.8M
11.4%
BlnckPruner
7.88
7.88
8.86
100min
26.6M
12.5%
EvoPress
7.61
6.41
8.67
58min
12.4M
25%
ShorGGPT
21.16
23.41
30.30
44s
65K
25%
Cosine Similarity (Window)
17.37
16.60
21.04
65K
25%
Weight Subcloning
21.16
23.41
30.30
49s
65K
25%
ShortenedLlana
11.81
13.86
14,08
8min
2.1M
25%
SLEB
10.21
9.45
12.03
122min
35.2M
25%
BlockPruner
11.31
12.49
13.32
152min
42.8M
25%
EvoPress
10.06
10.31
11.99
44min
10.0M
37.5%
ShorGPT
$4.07
70.94
63.51
65K
37.5%
Cosine Similarity (Window)
151.10
192.07
212.60
45s
65K
37.5%
Weight Subcloning
54.07
70.94
63.51
49s
65K
37.5%
ShortenedLlama
20.37
35.37
26.07
Smin
2.1M
37.5%
SLEB
17.29
19.57
20.91
159min
49.1M IM
36.4%
BlockPruner
19.41
28.75
185min
54.8M
37.5%
EvoPress
16.09
17.44
19.87
53min
19.7M
50%
ShonGPT
180.51
226.14
171.04
44s
65K
50%
Cosine Similarity (Window)
3611.06
4570.15
2876.83
45s
65K
50%
Weight Subcloning
180.51
226.14
141.04
49s
65K
50%
ShortenedLlama
68.79
145.78
87.40
8min
2.1M
50%
SLEB
51.30
117.21
58.56
184min
10
98.9%
BlockPruner
52,28
97.23
60.47
208min
64.9M
50%
EvoPress
34.20
47.15
40.10
31min
12.4M
34

---

EvoPress: Accurate Dynamic Model Compression via Evolutionary Search
Table 18. Comparison of EvoPress with uniform pruning and BESA on Llama-2-7B. Both EvoPress as well BESA (layerwise) operatc on a layer database generated via Wanda. BESA (rowwisc) additionally learns per-output-channl sparsities, which target different dimension of the compression space than EvoPress's layerwise allocation. EvoPress+BESA applies BESA using the average per-block sparsities found from running EvoPress, showing that both methods can work complementary. Here, use the lightweight version EvoPress with early stopping.
Sparsity
Method
Non- Non-Uniformity
Fineweb.
Wiki2
C41
Runtime
50.00
Uniform
N/A
7.60
6.41
8.97
49.79
BESA
Layerwise
7.55
6.32
8.93
16min
50.00
EvoPress
Layerwise
7.48
6.28
8.79
87min
49.70
BESA
Rowwise
7.39
6.19
8.72
33min
49.72
ExoPress+BESA
Rowwise
7.37
6.16
8.71
120min
60.00
Uniform
N/A
10.22
9.56
12.95
59.46
BESA
Layerwise
9.82
8.80
12.38
16min
58.91
EvoPress
Layerwise
8.75
8.03
11.94
102min
59.99
EvoPress
Layerwise
9.01
8.66
10.96
94min
58.90
BESA
Rowwise
9.15
8.11
11.88
3
59.12
EvoPress+BESA
Rowwise
8.59
7.58
10.58
135min*
70.00
Uniform
N/A
64.02
136.53
97.53
69.97
BESA
Layerwise
52.65
66.51
67.58
16min
68.52
EvoPress
Layerwise
13.53
15.61
16.90
11 1mim
69,99
EvoPress
Layerwise
15.55
24.04
19.54
120min
68.51
BESA
Rowwise
16.53
17.79
24.09
33min
68.03
EvoPress+BESA
Rowwise
13.83
14.19
19.52
144min*
Sparsity values in red are more than 0.5% below the target. They result of BESA only enforcing soft constraint the effective sparsity.
BESA runtimes include the hyperparameter sweep over '-d-coef'.
35


Reviews:
[{"review_id": "zzq2Pj7rlO", "reviewer": "Reviewer_Q9Fo", "summary": "The paper introduces EvoPress for dynamic LLM compression, which optimizes compression levels across different model components to minimize accuracy loss while satisfying a global compression threshold. By formulating dynamic compression as an optimization problem, EvoPress efficiently determines optimal compression profiles. Experiments on multiple models demonstrate SOTA results for LLM compression.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. The questions are addressed below:\n\n> To quantify model degradation, KL divergence was used in Section 3. KL divergence is popular to do it, but there are other ways, for example max absolute value, or we can even use a small calibration dataset to measure perplexity, etc. It’ll be great if authors could explain this design choice. \n\nIn our work we considered both KL-Divergence and Perplexity on the calibration set as fitness functions in the evolutionary search. We provide an ablation on these choices in Appendix B.3. Both choices show similar performance, but we decided to stick to KL-Divergence as it performed slightly better. The comparison between these two options is provided in **Table 1**.\n\n**Table 1. Comparison between KL-Divergence and Perplexity as the fitness function**\n\n| Model      \t| # Bits | Method    \t| Wiki2↓ | C4↓   | FW↓   |\n|----------------|--------|---------------|--------|-------|-------|\n| Llama-3-8B \t| 3  \t| Uniform   \t| 12.19  | 15.76 | 11.47 |\n|            \t|    \t| EvoPress (PPL)| 8.17   | 12.15 | 9.64  |\n|            \t|    \t| EvoPress (KL) | **7.49**   | **12.03** | **9.56**  \n| Llama-2-7B \t| 3  \t| Uniform   \t| 6.16   | 7.96  | 6.86  |\n|            \t|    \t| EvoPress (PPL)| 5.74   | 7.90  | 6.79  |\n|            \t|    \t| EvoPress (KL) | **5.70**   | **7.87**  | **6.76**  |\n| Mistral-7B-v0.3| 3  \t| Uniform   \t| 5.54   | 8.57  | 6.96  |\n|            \t|    \t| EvoPress (PPL)| 5.23   | 8.45  | 6.87  |\n|            \t|    \t| EvoPress (KL) | **5.21**   | **8.42**  | **6.86**  |\n\n\nWe believe this is because KL-Divergence measures relative degradation to the base model, which is more informative than a pure next token loss. This is especially important when using as little calibration data as we do in our multistep selection process. We decided to adopt KL-Divergence to quantify the distance between the two predictive distributions, as it is a well-established metric, with a grounding in information theory.\n\n> The paper has a high level problem definition. However, the proposed approach was only applied to single compression approach one by one (pruning or sparsity or quantization). It will be great if authors could show or explain the results of applying EvoPress on multiple compression approaches simultaneously.\n\nEvoPress is compatible with multiple compression approaches simultaneously. We conducted an experiment with joint depth pruning (with 25% blocks dropped) and quantization (with 4 bits on average given 2, 3, 4, 5, 6 bit width options). Specifically, on each step of the evolutionary algorithm we perform an alternating optimization between block dropping and quantization. Firstly, we select the optimal depth pruning and configuration and then exchange quantization bit-width between “alive” blocks. Our experimental results (see [Wikitext-2 perplexity](https://anonymous.4open.science/r/EvoPress-Data-DF25/figures/multimodal_search_wikitext2.pdf) and [C4 perplexity](https://anonymous.4open.science/r/EvoPress-Data-DF25/figures/multimodal_search_c4.pdf)) suggest that EvoPress manages to find a better solution given some starting point and exhibits relatively stable convergence.", "claims_and_evidence": "Yes. Experiments demonstrate that EvoPress has SOTA performance for dynamic compression on various models.", "methods_and_evaluation_criteria": "The proposed methods and evaluation make sense. Here are some questions for the proposed method: \n\n(1) To quantify model degradation, KL divergence was used in Section 3. KL divergence is popular to do it, but there are other ways, for example max absolute value, or we can even use a small calibration dataset to measure perplexity, etc. It’ll be great if authors could explain this design choice. \n\n(2) The paper has a high level problem definition. However, the proposed approach was only applied to single compression approach one by one (pruning or sparsity or quantization). It will be great if authors could show or explain the results of applying EvoPress on multiple compression approaches simultaneously.", "theoretical_claims": "Briefly read the proofs in Section 3 and appendix.", "experimental_designs_or_analyses": "The overall experiments are valid. Various models and different compression methods are tested.", "relation_to_broader_scientific_literature": "LLM model compression under a threshold is very important. This paper proposed an efficient method to solve this problem, and has the potential to inspire future works on dynamic model compression.", "essential_references_not_discussed": "Most related works are discussed in paper.", "other_strengths_and_weaknesses": "Please see the comments above.", "other_comments_or_suggestions": "Please see the comments above.", "questions_for_authors": "Please see the comments above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VPn33MhWr8", "reviewer": "Reviewer_eQfQ", "summary": "This paper aimed at LLM compression and motivated by the observation that depth pruning LLM further may improve the performance. Evolutionary search algorithm is applied to search pruned model with compressed size and performance constraint. It also applied to layer/block-wise non-uniform unstructured pruning and quantization. The results show the improvement for these three compression methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback. The concerns are addressed below:\n\n**Experimental design and analyzes**\n\n*Runtime of the method* \n\nSimilar as for unstructured pruning in the main text, we have indicated the runtime of EvoPress for block dropping and quantization within convergence plots (see Appendix Figure 8 for Block Dropping, and Appendix Figure 14 for Quantization). We provide runtime measurements in the additional baseline comparisons further down.\n\nYou correctly observed that the super-fast version depicted in Figure 3 does not fully reach the model performance of the more heavy-weight search. However, the hit in performance is rather small, and we wanted to demonstrate that it can be obtained for a fraction of the search time. This also suggests a practical strategy: one could first run the faster search to quickly reach good performance, and only then switch to the more heavy-weight search for best results.\n\n**Comparison with Baselines**\n\nWe acknowledge your point; below, we provide additional comparisons with existing methods. \n\n1. Depth pruning\n\nWe acknowledge that SLEB is a relevant baseline and will include it as related work in the revision. Below, we have included a comparison with EvoPress on Llama2-7B, where we also included BlockPruner as a baseline, following the suggestion of Reviewer **zC2u**. Both SLEB and BlockPruner iteratively remove blocks and can thus be interpreted as highly restricted search procedures. In such methods, once a block is removed, it cannot be recovered in later iterations, which explains the worse performance compared to EvoPress. We report the runtime of EvoPress when using a simple stopping criteria, that terminates the search whenever the perplexity on the training data has not improved within the last 20 generations. The comparison of different depth pruning methods is detailed in [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/Comparison_depth_pruning.pdf).\n\n2. Unstructured sparsity\n\nIndeed, BESA is a relevant work and will be added to the related work section in the revised edition. For a fair comparison, we ran the fast version of EvoPress on a Wanda layer database (thus, the results are slightly worse than presented in the paper, where we used SparseGPT). Still, EvoPress finds notably improved layerwise sparsity allocations. \n\nFor completeness, we also included the row-wise version of BESA, which adapts the Wanda Mask within each layer. We show that this is complementary to EvoPress by using the average per-block sparsity found by EvoPress as the per-block sparsity for BESA (EvoPress+BESA), which mostly yields better perplexities than BESA and EvoPress individually. However, this is still very much experimental, and there might be better methods to merge both approaches, like producing a layer database using BESA with row-wise non-uniform sparsity allocation, and then searching this database with EvoPress.\n\nThe experimental results with BESA in the original setup as well EvoPress applied on top of BESA and Wanda are provided in [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/Besa_wanda_table.pdf).\n\n3. Quantization\n\nSlim-LLM and CWMQ perform mixed precision quantization on a per-channel level compared to the per-layer precision allocation adopted in our method. Therefore, we could adopt Slim-LLM or CWMQ as an alternative to GPTQ, and run EvoPress on top of this method. A direct comparison between EvoPress and these two works is not possible. However, we believe that EvoPress would synergize very well with such intra-layer mixed precision quantization methods, as it allows to produce layers with non-integer bitwidth, which is beneficial for the search procedure. According to your suggestion, we will reference these two works in the updated revision.\n\n*Slim-LLM*\n\nWe conducted Slim-LLM experiments using the open-sourced implementation on GitHub. We considered two setups: \n- Original calibration data (Wikitext2, 128k samples of length 2048)\n- Our calibration data (FinewebEdu, 8M tokens)\n\nEvoPress shows significant improvement over SlimLLM in the paper's original setup in terms of 0-shot and MMLU evaluations. When comparing methods with the same calibration data, both methods show similar performance. However, the runtime of Slim-LLM increases dramatically with more data (~50 hours on single RTX3090 GPU compared to ~10 hours for EvoPress). That said, EvoPress and SlimLLM are complementary and hence, running EvoPress on top of SlimLLM will lead to further gains. We present results in [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/Slim-LLM_vs_EvoPress.pdf).\n\n*CWMQ*\n\nThe algorithm implementation of CMWQ is not open-sourced, making it difficult to provide a comparison in a comparable setup.", "claims_and_evidence": "The paper claims SOTA performance on depth pruning, unstructured pruning and quantization but there are some related works not compared in this paper.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "Yes. The evolutionary search algorithm is checked.", "experimental_designs_or_analyses": "Only application 2 (unstructured sparsity) presents the running time for the proposed method and running time for depth pruning and quantization are not clear.\nIn Figure3, the “super-fast” version seems not reach out the same model performance compared with the normal version.", "relation_to_broader_scientific_literature": "Layer doping, pruning and quantization are all efficient methods for LLM memory problem as related works referenced in the paper.", "essential_references_not_discussed": "- For depth pruning, SLEB[1] and Slicegpt[2] should be compared.\n- For unstructured pruning, BESA[3] should be compared. \n- For quantization, other mixed-precision quantization works should be discussed and compared, such as SliM-LLM[4], CWMQ[5].\n\n[1] SLEB: streamlining llms through redundancy verification and elimination of transformer blocks.\n\n[2] Slicegpt: Compress large language models by deleting rows and columns.\n\n[3] BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocation\n\n[4] SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models\n\n[5] Channel-Wise Mixed-Precision Quantization for Large Language Models", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zKv0V9aUM9", "reviewer": "Reviewer_vouD", "summary": "This paper proposes EvoPress, a general framework for LLM compression. The authors observe that error monotonicity does not hold for LLM, and proposes an evolutionary search approach to improve performance. Experimental results demonstrate that on three compression approaches, depth pruning, unstructured sparsity, and quantization, the proposed methods achieve state-of-the-art performance for dynamic compression.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. Below, we address the two weaknesses:\n\n> The theoretical proof does not fully explain its effectiveness in nonlinear LLM compression. It is unclear whether EvoPress maintains optimization capability in nonlinear regions.\n\nFundamentally this is true – we cannot expect the linearity assumption to fully hold in practice. Notably, assuming linearity is standard in prior work, such as OWL, ShortGPT, and DP-based methods such as SPDY. In contrast, while our theoretical results rely on this assumption, EvoPress can optimize a much larger class of fitness environments. In this sense, EvoPress operates under significantly weaker assumptions than prior approaches. That said, our main contribution is empirical, with EvoPress showing strong performance across various settings.\n\nAdditionally, we conducted experiments to verify the capabilities of EvoPress:\n* In Figure 1, we consider the problem of removing twelve transformer blocks of Llama-3-8B, with the additional constraint that only pairs of consecutive blocks can be removed. We brute forced the entire search space (16 choose 6 = 8008 configurations) using significant compute in order to identify the global optima. Then, we ran our evolutionary approach on this problem, and found that it detects the global optima within 6 generations.\n* We performed a robustness analysis for pruning Llama-3-8B to 70% sparsity. We found that over 16 independent runs the resulting perplexity on hold-out data has very little variance (Figure 7), and the final configurations show high similarity (Figure 6). This indicates that local optima do not pose a problem for the search process.\n\n> The proposed strategy may get trapped in local optima in high-dimensional and multimodal compressed spaces. Its performance in highly constrained or multi-objective settings remains uncertain.\n\nWe would like to point out that the search space is relatively high-dimensional - the dimensionality of space is several dozens for the case of block dropping and several hundreds for quantization and unstructured sparsity. For a high enough compression ratio the loss landscape is nonlinear (as the example in Table 1 shows), yet the evolutionary search procedure successfully converges to a good optimum. EvoPress is compatible with multiple compression techniques simultaneously. Specifically, we conducted an experiment with joint depth pruning (with 25% blocks dropped) and quantization (with 4 bits on average given 2, 3, 4, 5, 6 bit width options) (see details in response to reviewer **Q9Fo**).", "claims_and_evidence": "Yes, the evidence provided by the authors supports the claims.", "methods_and_evaluation_criteria": "Yes, I think it makes sense.", "theoretical_claims": "The mathematical symbols, variables, and equations are generally well defined and mathematically correct.", "experimental_designs_or_analyses": "I think the experimental designs in this paper are reasonable.", "relation_to_broader_scientific_literature": "EvoPress greatly expands the literature by combining evolutionary optimization with dynamic LLM compression, challenging long-held heuristic assumptions, and providing a flexible framework.", "essential_references_not_discussed": "I'm not familiar with this area, so I'm not sure.", "other_strengths_and_weaknesses": "Strengths:\n\nThe proposed method is applicable to depth pruning, unstructured sparsity, and quantization while achieving strong performance.\n\nWeaknesses:\n1. The theoretical proof does not fully explain its effectiveness in nonlinear LLM compression. It is unclear whether EvoPress maintains optimization capability in nonlinear regions.\n2. The proposed strategy may get trapped in local optima in high-dimensional and multimodal compressed spaces. Its performance in highly constrained or multi-objective settings remains uncertain.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the weakness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cWHnKRzBX", "reviewer": "Reviewer_zC2u", "summary": "The paper proposes Evopress, a novel pruning approach for the dynamic compression of LLMs based on evolutionary computation. The authors identify a critical issue in current compression algorithm approaches: error monotonicity does not apply to LLM compression. Aiming to resolve such drawbacks, they propose a (1 + λ)-evolutionary algorithm for compressing LLMs, derived from the theory that the dynamic compression of LLMs has a linear fitness function. Hence, (1 + λ) sounds like an ad-hoc algorithm due to its hill-climbing properties, which are established to be optimal for linear fitness functions. The main novelty is that Evopress can be applied to different compression techniques: structured and unstructured pruning as well as quantization.\nIt also modifies the standard (1 + λ) by integrating level-switch mutation (to maintain a fixed sparsity ratio) and multi-step selection to overcome the high evaluation time required for the offspring.\nThe experimental results show how Evopress outperforms the selected baselines in terms of perplexity over the Language Modeling task and accuracy over Zero-Shot tasks.\n\n## Update after rebuttal\n\nThe authors included in the rebuttal the required pruning runtime comparison. I stick to my final comment that the comparison should be done with the same calibration data size to have a fair comparison. All my other concerns were addressed. Hence, I changed my score from 2 to 4 and supported the acceptance of the paper. However, from my perspective, all the new results provided in the rebuttal should be included in the main paper. Also, a clear statement about the pruning runtime complexity should be highlighted in the main paper w.r.t. some more \"pruning time\" efficient baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for a thorough and detailed view with a lot of insightful comments and suggestions. The concerns are addressed below:\n\n**Stopping Criteria**\n\nWe ran the search for more generations than necessary to get insights about the convergence behavior. We agree that in practice one should determine the number of generations dynamically, and as suggested, we have implemented a stopping criteria: Every 20 generations the perplexity on the training set is computed, and if it hasn’t improved, the search terminates. We present timings and results of EvoPress including early stopping below.\n\n**Runtime vs. baselines**\n\nDetermining meaningful runtime results for baselines is challenging due to several reasons: \n\n* For block dropping, 3 out of 4 baseline methods neither shared their code nor specified data usage, making runtime comparisons difficult. Generally, scoring methods are faster than EvoPress. We compare the runtime of EvoPress with baseline methods below.\n* For quantization and unstructured pruning, runtime depends on the specific method, while our approach works independently of the layerwise compression method. OWL, for instance, requires an extensive hyperparameter sweep (20 configurations) and a larger layer database than EvoPress, leading to similar runtime constraints in our SparseGPT setup, dominated by layer database generation.\n* Additionally, for pruning, the sweep in OWL is not provided by the authors, and it is not clear how the “best” model is chosen (in particular, over how much data). We decided (in their favor) to evaluate all 20 models on the entire evaluation data, and choose the model with lowest perplexities. \n\nWe will make this more transparent in the revised version. To address the reviewer’s concern, we have included runtimes under common parameter values below.  \n\n**Comparison with baselines**\n\n*Block dropping*\n\nWe provide comparisons with SLEB and BlockPruner in [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/Comparison_depth_pruning.pdf). We also included the runtimes of other baseline methods, where we used 64K calibration tokens of Fineweb Edu consistently. One can observe that EvoPress outperforms these additional baselines, especially at higher compression, while being faster. We report the results of EvoPress with early stopping as outlined above.\n\n*Unstructured sparsity*\n\nPlug and Play and Wanda are relevant to unstructured pruning, and we'll include them in the new revision. Our work primarily focuses on finding the optimal sparsification profile on top of a layerwise compression method. We adopted SparseGPT due to its popularity and good performance. We demonstrate that EvoPress also functions effectively on top of a layer database generated using Wanda in [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/Besa_wanda_table.pdf), where we also report runtime measurements. As requested by Reviewer **eQfQ**, the table contains results for BESA.\n\n*Quantization*\n\nAWQ, BiLLM, and OmniQuant will be discussed in our updated version, acknowledging their relevance. To test EvoPress's compatibility with various quantization methods, we applied it to a database generated by AWQ. Due to AWQ's constraints, the search space requires the same bitwidth within transformer block. Despite this, EvoPress achieved an improved compression profile (see [Table](https://anonymous.4open.science/r/EvoPress-Data-DF25/tables/EvoPress-AWQ.pdf)).\n\n**Human-Crafted Scoring**\n\nWe wanted to emphasize the fact that the baseline methods do not evaluate the fully pruned model, instead, they make strong assumptions on how the fully pruned model performs based on another metric, which is inherent to scoring methods. In contrast, EvoPress (and [1,2] in a more restricted setting) evaluate the fully pruned models directly. We will rephrase this in the revised version.\n\n**Questions**\n\n* *Use of SparseGPT*. Regarding SparseGPT, we used it for its reliable performance at high compression, superior to Wanda at 60%-70% sparsity, which are our targets. Wanda, however, allows re-use of weight configurations across sparsities with separate masks.\n* *Mixed-quantization baselines*. At the time of submission, we were not aware of any baselines that address the problem of mixed-precision quantization in a comparable setting. Reviewer **eQfQ** suggested the Slim-LLM paper, which we compared but found it complementary due to its channel-wise approach. OWL's layerwise non-uniform quantization performed worse than uniform quantization, so we did not include it.\n* *Convergence Plots of EvoPress*. The train accuracy refers to the KL-Divergence of the survivor of the selection process, i.e. the parent of the subsequent iteration. It is measured on a random subset of the full training data (Fineweb-Edu), which is why the line is noisy. The test accuracy is measured on the full set of hold-out Fineweb-Edu data and does not impact the search process. We will clarify this in the revision.", "claims_and_evidence": "Partially.\n\nThe motivation for the linear landscape in dynamic compression is supported. Also, the numerical results over the tested task support the claims of the paper with respect to the selected baselines. A claim that, in my opinion, is not supported is \"iteration efficient.\" While I agree with the authors that the multi-step selection improves the overall runtime of EvoPress, with respect to the baselines, the method is not efficient in terms of runtime and resources required to obtain the final sparse model. While this may not be considered a major problem, I personally don’t think that, in its current state, the paper properly highlights such limitations.\nAnother aspect that does not completely support the claims is the selection of baselines. While it is true that the paper tackles three different compression schemes, the baselines selected for each compression level do not provide a clear picture with respect to the recent literature for each compression scheme.\n\n(See Relation To Broader Scientific Literature)", "methods_and_evaluation_criteria": "Yes, the selected models are in line with the latest public LLMs. The datasets and tasks are consistent with recent works on LLM compression.", "theoretical_claims": "I checked the Theorem 3.1 in the main text, not the full analysis in the Appendix.", "experimental_designs_or_analyses": "Yes. My main concern is the limited baselines for each compression scheme (see below)", "relation_to_broader_scientific_literature": "The paper focuses on dynamic compression of both structured and unstructured pruning as well as quantization. It tackles the problem of monotonic error in current pruning approaches, and at the same time focuses on dynamic compression, which is a crucial topic at the moment in the pruning community.", "essential_references_not_discussed": "The paper does not discuss several works.\n\nRegarding structured pruning, it does not mention or use [1,2] as baselines. For unstructured pruning, they apply Evopress to SparseGPT without mentioning [3,4]. On the quantization side, no recent weight-only quantization algorithms, such as [5,6,7], have been discussed or included in the baselines.\n\n[1] Zhong, Longguang, et al. \"Blockpruner: Fine-grained pruning for large language models.\" arXiv preprint arXiv:2406.10594 (2024). \\\n[2] Song, Jiwon, et al. \"SLEB: Streamlining LLMs through Redundancy Verification and Elimination of Transformer Blocks.\" Forty-first International Conference on Machine Learning. \\\n[3] Zhang, Yingtao, et al. \"Plug-and-play: An efficient post-training pruning method for large language models.\" The Twelfth International Conference on Learning Representations. 2024. \\\n[4]  Sun, Mingjie, et al. \"A Simple and Effective Pruning Approach for Large Language Models.\" The Twelfth International Conference on Learning Representations. 2024 \\\n[5] Lin, Ji, et al. \"Awq: Activation-aware weight quantization for on-device llm compression and acceleration.\" Proceedings of Machine Learning and Systems 6 (2024): 87-100. \\\n[6] Huang, Wei, et al. \"BiLLM: Pushing the Limit of Post-Training Quantization for LLMs.\" Forty-first International Conference on Machine Learning \\\n[7]  Shao, Wenqi, et al. \"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models.\" The Twelfth International Conference on Learning Representations. \\", "other_strengths_and_weaknesses": "## Strengths\n\n- The investigation of the linear landscape for dynamic compression is novel and well-supported.\n- Applying (1 + λ) for model compression in a linear landscape is novel and effective. The proposed level-switch and multi-step selection modifications allow (1 + λ) to be effective in compressing LLMs.\n- Using KL-divergence instead of perplexity, as in [1,2], provides a smoother evaluation of the sparse model.\n- The application of a single framework for different compression schemes is the main novelty and a key strength of the paper.\n- The results across the selected baselines show that EvoPress achieves better performance.\n\n\n## Major Weaknesses\n\n- The main weakness, in my opinion, is the time required to obtain the sparse model. Figures 3 and 8 clearly show that EvoPress requires hours to complete its evolutionary process. In particular, Figure 3 (left) indicates that 13 hours are required. On the other hand, approaches like OWL only require a single forward pass of the calibration data through the model to obtain outlier values from which to compute block-wise sparsity. Similarly, structured pruning methods like ShortGPT and Sliding Window Cosine Similarity are time-efficient in computing block scores. While this may not be a reason to reject the paper, I believe a fairer comparison in terms of pruning runtime, along with clearer details on the time and resources needed to run EvoPress, is necessary.\n- While I highly appreciate EvoPress for its ability to apply dynamic compression to different compression schemes, the evaluation of each scheme (in terms of baselines) is too limited.\n- Regarding structured pruning, the paper claims, “All four previous methods rely on human-crafted,” however, Shortened Llama and [1,2] also rely on the perplexity metric to evaluate the sparse model, similar to the proposed KL-divergence approach.", "other_comments_or_suggestions": "I appreciated the idea of EvoPress and agree with the authors that it is the first framework that can be used across different compression schemes. I think the idea of using (1 + λ) as hill-climbing over a linear landscape, such as that of dynamic compression, is novel.\nHowever, I would like to see a fairer comparison in terms of pruning runtime and resources required w.r.t. the baselines in order to increase my score. Even though the advantage of obtaining a sparse model compensates the time required to obtain it, I think it is fair to report the actual runtime (seconds) and resources (possibly FLOPs) required (as numerical values and not as vertical lines inside a plot) and especially w.r.t. the baselines. Also, provide a detailed explanation of why no other baselines are included, especially in the quantization experiments. Also, provide information about train and validation data; see the question below.", "questions_for_authors": "- In unstructured pruning, why apply EvoPress to SparseGPT instead of Wanda, which would have been faster, given that SparseGPT requires weight reconstruction differently for each level in the database?\n- Is there a specific reason why there are no other baselines for quantization besides DP-pruning?\n- Regarding the runtime plots, the authors state, “We observe convergence close to optimum in 5-6h” (Figure 3). However, when applying EvoPress, do you still run for all the generations specified in Table 8? I also have some doubts about the plot—does the train accuracy refer to the KL of the offspring at generation t (computed as an average across all individuals)? Is the test accuracy computed on the same calibration data used for KL-divergence? Please provide more information about it.\n- Since (1 + λ) retains only one individual after evaluation, if the optimum is reached at generation t << num_generations, shouldn’t the evolutionary process stop? As you mention, EvoPress usually converges close to the optimum before reaching num_generations, why not implement a stopping condition?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposed a method called EvoPress, an evolutionary search-based framework for dynamic LLM compression, capable of optimizing depth pruning, unstructured sparsity, and quantization under global accuracy constraints. Experimental results looked strong. All reviewers agreed that this is a good contribution to the community and authors have addressed the concerns during the rebuttal phase.
Decision: Accept (poster)