Title: ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Pruning

Depth Pruning

Transformers

LLMs

Training-free
Abstract: We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast to conventional pruning approaches that require additional training or fine-tuning, our approach requires only a small calibration dataset that is used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seam- lessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-free approaches and remains highly competitive with state-of-the-art pruning methods that involve extensive retraining/fine-tuning and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25% pruning while retaining approximately 90% of the original model’s performance on open benchmarks—without any training or healing steps, resulting in minimal computational overhead. We provide an open- source library implementing ReplaceMe alongside several state-of-the-art depth pruning techniques, available at https://github.com/mts-ai/ReplaceMe.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pruning method that is both data- and computation-efficient. The proposed ReplaceMe substitutes selected Transformer blocks with linear layers, guided by an importance metric. This strategy aims to reduce model complexity while minimizing performance degradation, serving as an alternative to directly removing blocks. The authors present experimental results indicating that ReplaceMe can match or even outperform structured pruning methods that require retraining.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength #\n\n* The paper addresses a practical and important topic in model compression. Structural pruning is generally more hardware-friendly but often more challenging in terms of preserving model performance. Thus, research that advances this area is both timely and valuable. Furthermore, the emphasis on pruning efficiency aligns well with real-world deployment constraints.\n\n* The paper is clearly written and well-organized, making it easy to follow the proposed method and its motivation.\n\n* The proposed method is both effective and efficient. The key idea of compensating for pruning-induced errors by inserting linear layers is compelling. These linear layers are data-efficient to train due to their convex nature and can be integrated into existing MLP layers without increasing inference cost, which enhances the practicality of the approach.\n\n* The experiments span both NLP and CV tasks, demonstrating that the proposed method performs on par with or better than existing structured pruning techniques, including those that require additional tuning or retraining.\n\n# Weakness #\n\nI did not find a very strong weakness. Here are some minor suggestions.\n\n* The first two subfigures in the introductory figure convey similar concepts and could potentially be merged to reduce redundancy and improve visual conciseness.\n\n* It would be helpful to include an ablation study analyzing the distribution and quantity of skipped blocks. Specifically, showing how many layers are skipped at once and how this distribution affects performance would clarify how the redundancy lies.\n\nQUESTIONS:\nI am curious about the size of the added linear transformations used for error compensation. It would be useful to clarify whether the number of additional parameters introduced by these layers has a significant impact on overall performance or model size. A discussion or ablation on this aspect could enhance the understanding of the method’s trade-offs.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces ReplaceMe - a depth pruning method that prunes transformer blocks from the base models. Pruned layers are determined by the distance between the activation outputs of different transformer blocks. In particular, they select a predefined number of layers to be pruned and seek for cut block index that minimises the distance between hidden states before and after the cut. To compensate for pruning effects, they utilise a small calibration dataset to learn a linear transformation that bridges the gap between activations before and after the cut. The authors propose to use L2 and cosine distance and find that cosine distance consistently produces better results. Besides, they propose regularisation into the calibration of linear transformation and also extend this method to non-consecutive replaced blocks. Experimental results validate the effectiveness and generality of ReplaceMe across various model families, including LLMs and Vision transformers.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper is well written and easy to follow\n2. ReplaceMe shows better performance on various experimental settings from LLM families on NLP tasks to Vision tasks with transformer architectures with moderate compression ratios.\n3. The authors validate their method with very detailed ablation studies\n4. ReplaceMe significantly reduces the computational overhead in terms of latency and environmental impact.\n\n## Weaknesses\n\n1. This method lacks novelty since fundamental motivation and pruning technique (cosine distance) are proposed in previous work UIDL.\n2. The effectiveness is only on a low/moderate compression ratio 25%.\n\nQUESTIONS:\n1. Could the authors provide more ablation studies with other similarity measures like CKA [1], CCA [2]?\n2. The linear transformation is just like a feature alignment method, does it have the same role as adapters? Could we replace the linear transformation matrix with PEFT methods like Adapter or LoRA? \n3. Which layers/blocks are often pruned? Could the authors provide more analysis on this? \n\n[1] Kornblith et al., 2019. Similarity of Neural Network Representations Revisited.\n\n[2] Morcos et. al., 2018. Insights on representational similarity in neural networks with canonical correlation.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ReplaceMe, a simple and efficient method to compress transformer-based models by replacing selected layers with linear transformations. The approach is post-hoc and training-free, requiring only a small “calibration dataset” to optimize the replacement layers via regression. The method is evaluated across a variety of language and multimodal models, demonstrating competitive performance compared to existing pruning techniques, with lower computational and environmental costs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n-\tThe paper is well-written, well-structured, and easy to follow.\n-\tSimple yet effective idea: Replacing layers with linear transformations is a straightforward yet practical strategy that reduces computational overhead.\n-\tThe experiments are broad, covering multiple tasks and models. The results on multi-modal models highlight the transferability of the method.\nWeaknesses\n-\tLimited compression ratio evaluation: The experiments primarily focus on a 25% compression ratio. While this is acknowledged in the limitations section, evaluating a wider range of ratios would provide a more complete picture of the method’s robustness.\n-\tThe primary comparison is with UIDL. Including more pruning baselines would make the comparison more convincing.\n-\tMissing reference: The core idea, replacing layers with linear transformations, has been explored in prior work [1, 2], although not specifically in the pruning context. These works are missing from the related work discussion.\n-\tThe calibration dataset, a key component of the method, is only clearly explained in Section 3.3.1. Introducing this concept earlier would improve clarity and understanding.\n[1] Hernandez, et al. \"Linearity of relation decoding in transformer language models.\" ICLR 2023\n[2] Wang, et al. \"Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models.\" ACL 2025\n\nQUESTIONS:\n-\tCan the authors comment on using non-linear transformations instead of linear ones? Would this further improve performance?\n-\tHow is the “environmental impact” calculated?\n-\tIs the learned linear transformation task-specific? That is, does each task/dataset require a new calibration and transformation? How does this compare with other pruning methods in terms of transferability?\n-\tThe choice of calibration dataset seems to have a large impact on performance. Do the authors have any insights on the reason of this sensitivity?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a structured pruning approach that removes transformer blocks and replaces them with a learned linear transformation. The authors argue that this method is training-free unlike other compression methods which require some fine-tuning (healing). The linear layers are estimated using a calibration dataset by matching the activations of before and after the blocks to be removed. L2 distance and cosine distance metrics are used to estimate the linear transformation from a small dataset. Experiments are comprehensive, and show that for compression ratios up to 25% the proposed method compares favourably with other methods.\n\nSTRENGTHS AND WEAKNESSES:\n* The idea of replacing structures within a neural network with learned linear transformations is a simple but elegant idea. \n\n* The justification of which blocks to prune based on the distance between activation maps is reasonable. \n\n* The distance metrics used are relatively straightforward. Both L2-norm and cosine distance seem to work quite well for \"reasonable\" compression ratios.\n\n* Reduction in memory achieved by using Eq. 8 is a nice addition to the contribution.\n\n* Experiments are comprehensive; both language and vision transformers are assessed. Many recent baseline methods are also reported. \n\n* While the compression gains are modest, the strength of the method as the authors argue lies in two factors: use of linear layers, and no fine-tuning after compression. \n\n### Weaknesses\n\nI have some clarifying questions. \n\n* **Justification for removing entire blocks**: The prune and replace strategy empirically seems to work for reasonable compression ratios. How is it that these entire complex blocks so redundant? The authors point to UIDL paper as the justification. But, I would have also liked to see this argument laid out in the paper. Is it because some of the deeper layers not very useful? If so, why? Can this not be done already from the outset? Can this prune and replace strategy be used during training also? What makes this so unique to transformer blocks, and can it be applied to CNNs? If not, why?\n\n* **Choice of blocks**: In Eq. 4 authors mention that _n_ blocks can be removed and the optimal index is obtained by minimizing the distance between transformer blocks. Why do these blocks have to be contiguous? The authors do report some of this analysis in the Appendix (Fig. 5) but I would have liked to get some justification for this in the main text too. \nIt would also be interesting to perform some form of a greedy search over the different transformer blocks. It would then be a sequence selection problem which would output the set of indices that need to be pruned. I can see that replacing blocks with a linear transformation is more practical, but this could also be done when disconnected blocks are removed, if the linear layer is still used just after the last removed block. \n* **Difficult results tables**: I found it hard to parse the results tables. It could simply be because the authors do not elaborate the metrics being reported. For instance, what is the last column in Table 3? It just says %. I initially read it as compression ratio which you would like  to increase but then the baseline is at 100%. \n \n* **Performance at high compression ratio**: In L189, the authors point out that at higher compression ratios their method is able to recover some performance after fine-tuning. I do not immediately see where these results are reported. Are these the results still in Fig. 3? If so, I do not understand this. \nI mention this because, this would be useful to see. My hypothesis is that at higher compression ratios even large fine-tuning cannot recover the performance as linear transformations are not as expressive as something like LoRA.\n\n* **Related work**: I find this work more closely related to the class of model compression methods that use factorization, and less to structured pruning literature. Perhaps the authors should consider describing works like [1] in their related work.\n\n### Other Comments\n\n* While I usually like a summary figure on page 1, I do expect it to be fully described. The figure 1 is placed even before the abstract and none of the abbreviations are introduced at this point; so the figure is misplaced. I would have preferred the method overview figure 2 in its place as it is quite self-contained.\n\n* If reporting carbon emissions, make sure if it is CO2 or CO2 equivalent, as the latter is the more common measure.  \n\n### References\n\n1. Ren et al. \"Exploring Extreme Parameter Compression for Pre-trained Language Models\" (2022)\n\nQUESTIONS:\n1. Why is this strategy well suited for transformer blocks. Why can a similar linear transformation not be used for other deep CNNs? \n\n2. I would like the authors to speculate on the importance of choosing contiguous blocks versus optimizing the block sequence itself. \n\n3. Clarification on performance at high compression ratio (see details under weaknesses)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}