EvoPress: Towards Optimal Dynamic Model Compression via Evolutionary
  Search
Authors: name: Oliver Sieberling, name: Denis Kuznedelev, name: Eldar Kurtic, name: Dan Alistarh

The high computational costs of large language models (LLMs) have led to a
flurry of research on LLM compression, via methods such as quantization,
sparsification, or structured pruning. A new frontier in this area is given by
dynamic, non-uniform compression methods, which adjust the compression
levels (e.g., sparsity) per-block or even per-layer in order to minimize
accuracy loss, while guaranteeing a global compression threshold. Yet, current
methods rely on heuristics for identifying the "importance" of a given layer
towards the loss, based on assumptions such as error monotonicity, i.e.
that the end-to-end model compression error is proportional to the sum of
layer-wise errors. In this paper, we revisit this area, and propose a new and
general approach for dynamic compression that is provably optimal in a given
input range. We begin from the motivating observation that, in general,
error monotonicity does not hold for LLMs: compressed models with lower
sum of per-layer errors can perform worse than models with higher error
sums. To address this, we propose a new general evolutionary framework for
dynamic LLM compression called EvoPress, which has provable convergence, and
low sample and evaluation complexity. We show that these theoretical guarantees
lead to highly competitive practical performance for dynamic compression of
Llama, Mistral and Phi models. Via EvoPress, we set new state-of-the-art
results across all compression approaches: structural pruning (block/layer
dropping), unstructured sparsity, as well as quantization with dynamic
bitwidths. Our code is available at https://github.com/IST-DASLab/EvoPress.