ReplaceMe: Network Simplification via Layer Pruning and Linear
  Transformations
Authors: name: Dmitriy Shopkhoev; name: Ammar Ali; name: Magauiya Zhussip; name: Valentin Malykh; name: Stamatios Lefkimmiatis; name: Nikos Komodakis; name: Sergey Zagoruyko
Published: 2025-05-05 17:47:42

Abstract: We introduce ReplaceMe, a generalized training-free depth pruning method that
effectively replaces transformer blocks with a linear operation, while
maintaining high performance for low compression ratios. In contrast to
conventional pruning approaches that require additional training or
fine-tuning, our approach requires only a small calibration dataset that is
used to estimate a linear transformation to approximate the pruned blocks. This
estimated linear mapping can be seamlessly merged with the remaining
transformer blocks, eliminating the need for any additional network parameters.
Our experiments show that ReplaceMe consistently outperforms other
training-free approaches and remains highly competitive with state-of-the-art
pruning methods that involve extensive retraining/fine-tuning and architectural
modifications. Applied to several large language models (LLMs), ReplaceMe
achieves up to 25% pruning while retaining approximately 90% of the original
model's performance on open benchmarks - without any training or healing steps,
resulting in minimal computational overhead (see Fig.1). We provide an
open-source library implementing ReplaceMe alongside several state-of-the-art
depth pruning techniques, available at this repository.