ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Authors: Dmitriy Shopkhoev
Ammar Ali
Magauiya Zhussip
Valentin Malykh
Stamatios Lefkimmiatis
Nikos Komodakis
Sergey Zagoruyko
Venue: NeurIPS 2025 poster ()
Keywords: Pruning
Depth Pruning
Transformers
LLMs
Training-free
Primary area: deep_learning
TL;DR: Training-free depth pruning method for transformer based models via approximating blocks with linear transformation

Abstract: We introduce ReplaceMe, a generalized training-free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast to conventional pruning approaches that require additional training or fine-tuning, our approach requires only a small calibration dataset that is used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seam- lessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-free approaches and remains highly competitive with state-of-the-art pruning methods that involve extensive retraining/fine-tuning and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25% pruning while retaining approximately 90% of the original model’s performance on open benchmarks—without any training or healing steps, resulting in minimal computational overhead. We provide an open- source library implementing ReplaceMe alongside several state-of-the-art depth pruning techniques, available at https://github.com/mts-ai/ReplaceMe.

ReplaceMe: Network Simplification via Depth Pruning and Transformer Block Linearization
Dmitriy Shopkhoev MWS AI, ITMO University
Ammar Ali MWS AI, ITMO University
Magauiya Zhussip MWS AI
Valentin Malykh
Stamatios Lefkimmiatis
Nikos Komodakis University of Crete, IACM-Forth, Archimedes Athena RC
MWS AI, ITMO University, IITU University
MWS AI
Sergey Zagoruyko Polynome
Abstract
We introduce ReplaceMe, a generalized training-free free depth pruning method that effectively replaces transformer blocks with a linear operation, while maintaining high performance for low compression ratios. In contrast conventional pruning approaches that require additional training or fine-tuning, our approach requires only small calibration dataset that used to estimate a linear transformation, which approximates the pruned blocks. The estimated linear mapping can be seam- lessly merged with the remaining transformer blocks, eliminating the need for any additional network parameters. Our experiments show that ReplaceMe consistently outperforms other training-fire frregg-feeeeee approaches and remains highly competitive with oftte-of-the-art art pruning methods that involve extensive retraining/fine teniettunngg and architectural modifications. Applied to several large language models (LLMs), ReplaceMe achieves up to 25% pruning while retaining approximately 90% of the original model's performance on open benchmarks-withon healing steps, resulting in minimal computational overhead. We provide open- source library implementing ReplaceMe alongside several state-of-the-a depth pruning techniques, available at htlt:p//giiiub..ommmmmm////llees
any training or
Introduction
In recent years, transformers have achieved unprecedented success across wide range of tasks on both computer vision and natural language processing. Modern large language models (LLMs) typically scale up to billions even hundreds of billions of parameters, significantly increasing the computational and memory requirements for both training and inference stages. This substantial resource demand poses a critical challenge for their wider practical deployment and usability.
Due to the excessive size of modern LLMs, there has been significant research effort to make such models accessible users with limited hardware capabilities. These efforts primarily focus three key strategies: quantization, distillation, and pruning. Pruning, which is the focus of the current work, involves identifying and removing less important parameters entire structural components to streamline the model, thereby reducing computational overhead without significantly compromising the performance. Structured pruning different from unstructured pruning in that it focuses entire groups of parameters or layers, allowing their complete removal. This approach not only
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

enhances hardware utilization efficiency but also potentially achieves greater reductions in resource consumption. Importantly, operates independently of the hardware type used.
In this work, we focus on structural depth pruning, operating under the hypothesis that a contiguous set of transformer blocks can be effectively approximated by single linear transformation. To validate this idea, we propose ReplaceMe, a novel training-free pruning method that replaces selected blocks with linear transformation estimated from a small calibration dataset. should be noted that most existing pruning methods require a post-pruning retraining phase, often referred to as "healing process", recover lost performance. This retraining stage can be tume-consuming and computationally expensive. In contrast, ReplaceMe preserves the majority of the model performance without any retraining for reasonable compression ratio scenarios. ReplaceMe generalizes depth pruning methods by introducing a simple yet effective linear transformation that compensates for the error caused by block removal. This transformation is subsequently fused with one of the remaining model weights, enabling seamless integration without adding parameters. The contributions of this work can summarized as follows:
1. We propose ReplaceMe, generalized method for depth pruning that can maintain model performance without requiring any healing process, for reasonable compression ratios;
2. We conduct a detailed study on the estimation of the linear transformation with both analytical and numerical methods and under differrnnt objectives;
3. We provide detailed ablation studies for different calibration data, solvers, and LLM archi- tectures;
4. We validate the effectiveness and generality ReplaceMe across diverse model families, including LLMs and vision transformer architectures like ViT 7
This paper organized as follows: Section 2 presents the core methodology behind our training- free depth pruning approach. It introduces the framework for identifying prunable layers in large language models (LLMs) and estimating the corresponding linear transformations that compensate for the removed components. This section also discusses the selection of appropriate loss functions, regularization strategies to ensure generalizability, and the potential extension to multiple linear transformations for more flexible pruning. Section 3 then provides comprehensive experimental results and ablation studies, demonstrating the effectiveness and robustness of our method, and analyzing the key factors that influence its performance.
Method
Next, we introduce ReplaceMe, a novel depth-wise neural network pruning method that balances simplicity and effectiivnees to optimize model performance. Our approach is based on the idea of pruning multiple layers in transformer models and replacing them with a single linear transformation. It consists of the following key steps: First, we identify layers suitable for pruning by targeting those with minimal impact on performance, in line with prior research (Section I. Next, we compute optimal linear transformation (LT) to compensate for the contributions of the pruned layers. Notably, this transformation is seamlessly integrated into the preceding layer, preserving model performance without introducing additional parameters (Section 2.2 Furthermore, we study the effect of regularization methods on the estimation of the transformation and show that this can be helpful strategy to maintain balance between model performance and perplexity (section 2.3). Finally, we outline how to extend our framework to support multiple linear transformations, enabling fiexible and informed pruning decisions (section 2.4p. Together, these components establish ReplaceMe as practical and robust advancement in training-free neural network pruning.
2.1 Layers selection
Let X, Nxd be the input to the i-th transformer block, where N denotes the number of tokens and d the hidden dimension of the transformer model. Then, typically, the conventional transformer

---

Norm
Allention
Mu Mult-HHaa Head
MLP
Transform
Norm
Attinntion
Mult- sad
Norm
MLP
Attention Norm
Mulli-Head Fead
Norm
MLP
Norm
Allention
Multi- Mulli-Hoad Head
Norm
MLP
X* block
(8' +*++++ 1) block
skipped transformer blocks
Figure ReplaceMe compresses and accelerates LLMs by bypassing contiguous sequence of transformer blocks-illustrated by the red whh-e preserving model performance. This is achieved by inserting an estimated linear transformation matrix that maps the MLP output of the i"-th block directly to the input space expected by the (i* + 1 + 1)-th block, effectively replacing all blocks between.
block can be expressed in the following way:
Y: X, (HA, (X.))
(1)
M, = MTP, (TN (Y,))
(2)
L = Y, M,
(3)
where MHA; and MLP; denote the multi-head attention (MHA) and MLP layers, respectively, while IN(" LN(?) correspond the layer normalization operators before the MHA layer and after the MLP, respectively. The output the attention sub-block is denoted as Y, while M, and L; represent the output of the MLP layer and the transformer block, respectively.
The layer selection strategy based on the significance of each layer, which is determined by the distance between the activation outputs of different transformer blocks. Formally, for a predefined number layers be pruned, denoted as n, the optimal cut index is determined by minimizing the distance between hidden states before and after the cut:
arg min h (Li, Litn).
We evaluated various distance metries h(-) using activations computed small calibration dataset, with the impact of the dataset explored in Section 3.3.1 For each candidate cut, the distance between hidden states before and after the is calculated and then averaged across the calibration dataset obtain a robust layer-importance estimate. We found that the cosine distance is particularly effective in identifying nearly optimal layers for pruming. This observation aligns with findings reported recent studies [13]. In the supplementary material A0 we present results of an exhaustive brute-foree layer selection, which confirm that cosine distance consistently identifies optimal near-optimal layers for removal. Additionally, we provide comparative analysis of the L2 distance metric to further validate our choice.
2.2 Linear Transform Estimation
To compensate for the pruned transformer blocks. we leverage the same set of calibration data to compute activations directly before and after the removal point. Using these activations, we estimate an appropriate linear transformation that accurately approximates the final output of the pruned blocks. Depending on the selected criterion, this estimation can be performed using either analytical numerical approaches, enabling precise modeling of the omitted layers.
As illustrated in Fig. [ linear transformation is applied following the MLP and prior to the residual summation. The goal of our method is estimate optimal linear transformation matrix T so that:
T = arg min h (M, T Y, L
T
3

---

where h (+) corresponds to a distance function (e.g., . Lg distance, cosine distance, etc.) between two input tensors and denotes the optical cut index estimated using Eq. 4 Once the transformation matrix is estimated, the transformer blocks from i + 1 to i + n (inclusive) are removed.
L2 Lz-Distance. classical solution for Eq. 5 arises when setting h(-) as the L2-distance and is obtained by solving least-squares (LS) problem:
T arg min (M, T T+
1 (MF M,) M (Litn Y.).
(6)
For detailed derivation of the above result we refer to the supplementary material A.2
Cosine Distance. As discussed in Section 2.1 our ablation study on various distance functions to assess the importance of transformer blocks, revealed that the cosine distance is the most effective in identifying the least significant blocks. Motivated by these results, we have further used the cosine distance the objective function to estimate the optimal linear transformation. In this case, the optimization problem takes the form:
T arg min cos (M; T + Y,, LiLn)
T
N E(
(M. T Yi,) LLin,k | T Y. [Ltn,
arg min
T 1
where cos(:;)) denotes the cosine distance between the two input vectors. We use the notation M2, R to denote the k-th of matrix M; E RNxd which then represent as column vector. Here, the cosine distance is calculated per token and aggregated over all N tokens. Unlike the Ly-distance formulation, this objective does not admit closed-form solution, and thus numerical optimization approach is required. In our experiments, we have utilized the Adam [19] optimization algorithm. Furthermore, an ablation study involving various alternative numerical solvers can be found in the supplementary material A.6
To solve the optimization problem in E it would be necessary to store the hidden states M,, Y., and Litn. To improve memory efficiency, we instead optimize the following simplified formulation:
T arg min cos (M; T, L Y₁)-
This alternative formulation requires us to store only M, and the difference Litn Y,, instead of keeping all three matrices. In supplementary material A.I1I we empirically demonstrate that this simplification has a negligible effect on performance while improving memory efficiency.
Fusing the Linear Transformation Once the optimal transformation T has been estimated, our approach allows it to incorporated into the MLP layer of the i-th transformer block. Since the learned transformation and the pre-cut MIP block represent a sequential linear operators, they can be algebraically composed into a single equivalent linear transformation, allowing T to be fused with the weight matrix of the down projection linear layer of the MLP block. Consequently, the overall architecture of the model remains unchanged, except for the removal of the "non-effective" transformer blocks.
2.3 Regularization
We further consider estimating the linear transformation by imposing additional constraints the matrix T through regularization. Specifically, we reformulate the optimization problem as follows:
T arg min h(M; T Y;; Litn) + a R(T),
T
where R (-) denotes the regularizer and a controls its strength. To promote sparsity in the transforma- tion matrix and encourage more balanced distribution of feature importance, we use both L, and L2 regularization terms when we use the cosine distance our objective. When we instead utilize L2 as our objective, we derive the analytical solution under L2 regularization. Empirical analysis shows that the considered regularization approaches improve the ability of the pruned model to generate accurate predictions, as reflected by the accuracy-basee based benchmarks (see Sec. 3. Though, such improvement comes the cost of increased perplexity.

---

2.4 Multiple Linear Transforms
The proposed ReplaceMe method can be easily extended to be applied on multiple non-overlapping groups of blocks within the model, estimating separate linear transformation for each group (multi- LT). This approach provides flexibility in achieving the desired performance metrics, even under high compression ratios. Furthermore, if some of the selected groups of blocks are consecutive, they can be merged into single block with one corresponding linear transformation. We refer to this method as non-consecutive Multi- multi-linear transformations (Multi LT_ [C).. Experimental analyses further validating Multi LT LTCNC are detailed in the Appendix A.16
3 Experiments
In this section, first describe our experimental setup and then provide a systematic comparison of our training-free pruning method against existing structured pruning approaches, including those relying healing mechanisms. In particular, performance without requiring additional training. To further analyze the factors influencing our approach, we conduct ablation studies on several key aspects: the calibration dataset (used for choosing the layers to be pruned and for estimating the linear transformation), the choice of distance function in Eq. and the impact of regularization in the linear transform estimation.
show that our method achieves competitive
3.1 Experimental setup
In our experiments, we have focused primarily on LLaMA-2-7B and LLaMA-3-888 8B-Instruct models, while also reporting results for Qwen2.5-7B and Falcon-11B for a comparative analysis. Results for additional models are provided in the supplementary material A.7 For numerical estimation of the linear transform, we used Adam optimizer with LRR 1e-4 and batch size 1024, iterating for 10 epochs over the calibration data. In Table D we present results on different benchmarks that have been widely used in previous research. These benchmarks have been introduced in the following works: CMNLI [56). HellaSwag [57]. PIQA (2). CHID [59]. WSC (24. MMLU [6). CMMLU 255, Race- Bice-High/Midde (22), C3 46). Additionally, we benchmarked ReplaceMe using well-established public datasets, namely Winogrande [4I, BoolQ [4) OpenBookQA [I], SciQ [55], and Lambada OpenAl [35]. For all benchmarks except Lambada OpenAl, we report accuracy as the evaluation metric, along with the average accuracy across all benchmarks. For Lambada OpenAI, we report perplexity. To calculate the environmental impact we used CodeCarbon 6 a well-known tool to estimate CO2 emissions and power consumption
3.2 Comparison with other prnnetured-pruningg methods
In this section, we report our key findings from applying ReplaceMe across various model architec- tures and benchmarks. To ensure the statistical stability of our results, all experiments are executed multiple times. As demonstrated in Fig. 2 we conduct a comparative analysis between ReplaceMe and UIDL [13], evaluating key metrics including time-to-get comparable accuracy, environmental impact, and final model performance. Notably, for UIDL's healing process, we restricted fine-tuning to Low-Rank Adaptation (LoRA) applied exclusively the MLP layers, whereas alternative ap- proaches incur significantly higher computational costs. Our proposed method exhibits substantially reduced computational demands and achieves a markedly faster recovery compared to other methods. In Table we compare ReplaceMe with other state-of-the-arr aat structured depth pruning approaches. should be noted that all competing methods rely on healing mechanisms and require extensive retraining, whereas our method remains completely training-free free (no healing is applied). Despite this, as shown in Table T our method consistently outperforms all baselines on average and achieves 92.5% of the performance of the uncompressed Llama 2 7B model at a 25% compression ratio.
In Table 2 we further compare our method against state-of-the-art structured pruning approaches on the more recent Llama 3 8B Instruct model, under the setting where no healing applied. We note that SVD-LLM [53] employs low-rank approximation of the weights, while LLM--runer [29] combines both width and depth pruning. Despite these differences, as shown in Table our method again outperforms these baselines. All results are reported at 25% compression ratio. The Multi LT results correspond to the application of the method described in Section 2.4 While the identification of multiple groups of blocks typically yields consecutive blocks in most cases, we additionally evaluate the scenario where con-conseee blocks are enforced. This configuration demonstrates an improvement in perplexity but Ieads to performance degradation across benchmark tasks.
5

---

Environmental Normalized Comparison
Relative Accuracy
Relative Accuracy
100
150
Rel. to LS
100
92.5
59.9
90.3
50
90
3.0
6.6
85
Ours (Cosine)
(LS)
UIDL
Emissions
Time
Ours (Cosine) D
Figure 2: Comparison of the proposed LLM compression method with state- of-the-art UIDL. Subplots illustrate (a) compression time and environmental impact (CO2 emissions), and (b) performance accuracy relative to the uncompressed baseline. Our approach attains the shortest compression time and reduced emissions, while achieving the highest accuracy, demonstrating superior efficiency, sustainability, and effectiveness over existing methods.
Method
Truin-Free
C3
CMNLI
WSC
PIQA
Race-M
Race-H
MMLU
CMMLU
AVG
RP
(lesi)
Swag
Liura 2 7B (buseline)
43.8
33.0
41.6
37.5
71.3
78.1
33.1
35.5
46.8
31.8
45.3
100.0%
LLM S
43.3
33.0
24.1
36.5
61.1
71.5
34,8
37.0
15.5
29.4
11.6
92.0%%
29.7
33.4
28,4
72.0
22.9
22.0
25.0
7W.25
SliceGPT
I LaCo*
39.7
34.4
36.1
40.4
15.7
69.8
23.6
22.6
26.5
25.2
37.4
82.7%
UIDL*
40.2
34.4
21.5
40.4
59.7
69.0
35.2
34.7
44.6
28.9
40.9
90.3%
Ours (Cosine)
42.5
33.0
25.2
38.5
59.4
71.1
35.4
36.7
46.4
30.4
41.9
92.5%
33.0
Table 1: Comparing other pruning methods after healing and our training free approach ReplaceMe, * indicates that the numbers are taken from streamline paper (3). After compressing Llama 7B with 25% compression ratio. signifies that the model was trained following pruning, whereas indicates that the model training-free. We also report the relative performance to the original, unpruned model (RP).
In Figure we also compare our training-free free approach, ReplaceMe, against UIDL various models and different amounts of layer pruning. Our method consistently outperforms UIDL both benchmark scores and perplexity evaluations, while also exhibiting greater stability.
across
Finally, we note that at high compression ratios, applying a healing process becomes necessary, as linear transformations alone insufficient to fully recover performance. Details provided in Table Although ReplaceMe continues to outperform UIDL under these conditions, healing phase required at extreme compression levels maintain model effectiveness.
3.3 Analysis
Up this point, we have outlined our primary goal: replacing a series of transformer blocks with simpler, estimated linear transformation using calibration data. The nature of this calibration data critical, as we demonstrate in Section B.3.I, where the type of text (instructional vs. plain) and the amount of data significantly influence results. Furthermore, in Section 3.3.2] analyze the impact of regularization on results, revealing a trade-off between performance metrics such perplexity and accuracy. In the supplementary material further explore structured linear transformations (e.g., diagonal or orthonormal matrices) A.3
3.3.1 Ablation on calibration data
Our pruning method eliminates the need for additional training by leveraging small calibration datasets place of conventional training data. These calibration datasets serve core purposes: 1) assessing block importance to identify transformer blocks for removal (Section 2. and 2) capturing hidden states before and after the pruned blocks to solve the optimization problem in Eq. [5] which leads to the estimated linear transformation. The quality and characteristics of these calibration subsets are critical to the accuracy the estimation. To understand the influence of calibration data, we conducted ablation studies exploring the impact of sample size and dataset type- sype-innluding plain text, instruction-tuned data, and self-generated content. Our primary experiments utilized datasets such Arcee [5), Fin [36], and SlimOrca (26). consistent with prior work like UIDL [3].

---

Method
Linear transform
Lambada OpenAl ppl 4
Avg-acet
RP
Llama 8B Instruct
3.11
0.7
100%
SVD-LIM 153
None
29.90
0.59
85.3%
LLMPruner 29
None
12.31
0,60
85.3%
UIDI 13
Identity
2216.96
0.58
82.5%
ReplaceMe(ours)
Linear (I.S)
20.23
0.63
89.9%
ReplaceMc(ours)
Linear (Cosine)
15,88
0.63
90.9%
ReplaceMe(ours)
Multi Multi_I.T__ __ (Cosine)
13.95
0.63
90.0%
Table 2: Results of pruning Llama 3 8B instruct for 25% using different methods, without any healing finetuning. Avg-acc is the average performance across the Race, Winogrande, PIQA, BoolQ, OpenBookQA and SciQ benchmarks. Perplexity is measured on the Lambada OpenAl dataset. We also report the performance relative the original, unpruned model (RP). Multi LT _C denotes the non-consecutive blocks case when applying the method described in Section 2.4
Llama 3 8B instruct
Falcon 2 11B
Qwen 2.5 7B instruct
0.7
0.7
0.7
Accuracy
0.65
0.65
0.65
0.6
0.6
0.6
0.55
0.55
0.35
2
6
5
10
15
2
UIDL
ReplaceMe Adam
Figure 3: Comparison between our method and UIDL with different number of pruned layers and different models including, Llama 3 8B, Mistral 3 7B, and Qwen 2.5 7B. We show the average accuracy across the Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks.
In particular, we investigated three key factors: (1) the effect of dataset type and its source, (2) the minimum amount of data required to produce stable and accurate estimates, and (3) the efficacy of masking as a lightweight data augmentation strategy when working with limited calibration samples. As shown in Table [ we apply method prune LLaMA 3 8B [8] by 25%, using ReplaceMe across all ablation settings. We evaluate three distinct calibration datasets: Fine a plain- plain-text web corpus; SlimOrca, curated instruction dataset generated with ChatGPT; and orca generaeee,,,, synthetic dataset where responses are generated by the baseline model (targeted for pruning) using prompts from SlimOrca.
Impact of Calibration Dataset Type As shown in Table 4 calibration with instruction datasets leads to better performance benchmark evaluations than using plain text, particularly for instruction- tuned models. While self self-generate generated data yields improved perplexity scores, it tends to underperform on downstream benchmarks. This observation consistent for both the optimization objectives that we utilize in Eq. S] that is the L2 and cosine distances.
We have also explored combining SlimOrca with other datasets, such FineWeb and Aya 43 The latter is a multilingual corpus covering 66 languages. Interestingly, these mixed datasets performed on par with SlimOrca, suggesting that high-quality instruction data have the biggest impact.
Furthermore, in the supplementary material (Section 1.12), we present a comprehensive analysis of the impact of calibration dataset size, including the minimum number of samples required to achieve stable model performance. We also introduce a masking- based augmentation technique designed to
Compression ratio
12.5%
25%
37.5%
50%
UIDL
59.9
50.4
41
30.4
Ours
64.6
60.7
51
38.6
Table 3: Average accuracy for method and UIDL at higher compression ratios compared across the Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks.
7

---

Method
Objective
Calibration Data
Avg-ace 1
Perplexity
%
Baseline Model
0.70
3.11
100.00
ReplaceMe
LS
fineweb 8k
0.56
26.74
80.47
ReplaceMe
LS
slim_orca 8k
0.62
21.21
89.59
ReplaceMe
LS
urca_generated 8k
0.61
13.58
87.40
ReplaceMe
Cosine
fineweb 8k
0.58
25.07
83.16
ReplaceMe
Cosine
slim orca 8k
0.63
15.90
90.67
ReplaceMe
Cosine
4K SlimOrea 4K Fineweb
0.63
15.85
90.51
ReplaceMe
Cosine
Mix of 66 languages
0.63
15.72
90.64
ReplaceMe
Cosine
orea urea__enerated 8k
0.61
13.24
87.33
Table 4: Pruning results for Llama 3 8B instruct using ReplaceMeby 25%. We estimate the linear
transformation using different data, including plain text data such as Fineweb [36] and self generated data using Slim Orca instructions.
maintain robust performance when only limited subset of data samples is available, ensuring computational efficiency under resource-constraing conditions. In addition, Section [A.17 [ presents results with task-specific calibration datasets, showing that using data from the same task as the evaluation target (e.g., SciQ) further improves accuracy compared to general-purpose calibration data such SlimOrca.
3.3.2 Regularization effect
We have also investigated how regularization impacts the linear transform estimation. We have applied ridge regularization for the L2 objective and observed that for 0 < a 10sh in Eq. 9 we notice a slight improvement in perplexity, while the average accuracy on benchmarks remains the same. Conversely, increasing further to 10 tends to enhance benchmark accuracy, though at the cost of higher perplexity. Thus, one can consider a as a tradeoff parameter between perplexity and accuracy of the pruned model. Regarding the cosine distance objective, L₁ regularization with = 10-1 gives higher boost to accuracy at the cost of perplexity performance. Similar results we obtain for T.2 regularization but compared to In regularization, the performance gain benchmarks smaller.
Model
Method
Avg-acc
Perplexity
RP
Llama3 8B
0.697
3.1
100.0
ReplaceMe
LS
0
0,624
21.2
89.6
ReplaceMe
LS L2 reg
0.1
0.625
21.2
89.6
ReplaceMe
LS + L2 reg
0.5
0.625
21.2
89.6
ReplaceMe
LS L2 reg
10
0.624
21.2
89.5
ReplaceMe
LS 1.2 reg
100
0.624
21.1
89.6
ReplaceMe
LS 1.2 reg
1000
0.624
20.8
89.5
ReplaceMe
LS L.2 reg
10000
0.626
22.9
89.8
ReplaceMe
Cosine
0
0.634
15.9
90.9
ReplaceMe
Cosine + L. reg
0.01
0.635
20.7
91.1
ReplaceMe
Cosine L1 reg
1 10
0.638
22.1
91.6
Table 5: The affect of regularization on LS and cosine methods in terms accuracy and perplexity
3.4 Vision Transformers pruning
So far, we have focused on the application of ReplaceMe exclusively the decoder transformer architecture, specifically within LLMs. This raises an important question: how well does this method generalize to other tasks beyond text generation, particularly when the transformer acts as an encoder. To answer this, we have applied ReplaceMe on the CLIP model for compression ratios of 13% and 25%. We utilized 8,000 samples from the MIMIC dataset and using the same evaluation procedure as in (7), we considered well-known benchmarks, namely MS-COCO (27]. Cifar-10 21), EuroSAT [5), VTAB (58), and Pascal VOC-2007 [9] For comparison purposes with state-of-the-art method we also report results when UIDL B is applied the same model.

---

Compres. ratio
MS-COCO Captions (retrieval)
Cifar10
VOC2007 Multilabel (zero-shot) mean _ag__
Model
VTAB/EuroSAT
(zero-shot)
text r
vision @
acel
uce5
neel
acc5
CLIP-L/I4 (37
0.794
0.611
0.956
0.996
0.790
0.625
0.960
UIDL
13%
0.745
0.609
0.927
0.996
0.781
0,490
0.931
ReplaceMe (1.S)
13%
0.767
0.620
0,939
0.996
0.552
UIDL
25%
0.515
0.418
0.693
0.971
0.597
0.381
0.814
ReplaceMe (LSS)
25%
0,556
0.471
0,780
0.971
0.688
0.395
0,823
Table Pruning CLIP vision encoder using ReplaceMe. The model performance after compressing by 13% is almost good as the original one, while in both scenarios method outperforms UIDL.
As shown in Table 6 ReplaceMe retains strong performance CLIP-ViL [37 even 13% com- pression ratio, closely matching the original model's 's accuracy and without requiring any additional training. While performance declines higher compression ratios, this degradation is expected and consistent across benchmarks. Despite this, ReplaceMe consistently outperforms the training- -ree state-of-the-an method UIDL [13) Finally, we note that the performance can be further improved by utilizing a lightweight post-pruning "healing" procedure.
4 Related Work
Model pruning [23 [4] has been at the frontier of deep-learning research since the early developments of this field. It has found practical applications not only in model size reduction but also in enhancing the interpretability of the models under study. Earlier studies on transformer- based language models, The same observation
such as BERT, demonstrated that these models are highly compressible 42 40) holds for pruning LLMs.
A significant number of studies focuses unstructured pruning, where individual weights within matrices throughout the model are zeroed out, resulting in sparse connections. SparseGPT [TT] tackles the challenge of layer-wise reconstruction in pruning by leveraging approximations of the inverse Hessian matrix. Wanda [47] improves the SparseGPT idea of reducing computations via simplification of the Hessian approximation. The LLM Surgeon [52] uses Kronecker-factong curvature approximations to perform pruning of LLMs. Despite maintaining high model quality post-pruning, computational savings from unstructured pruning requires specialized hardware support for sparse computations, limiting its wide applicability.
In contrast, structured pruning involves the complete elimination of certain structures inside the network. this context, removing entire attention heads or MLP channels is referred to as width pruning. LLM-Pruner (29] suggested to calculate an importance metric based on the difference in the loss when this computed with and without pruned group of weights, respectively. FLAP [1 proposed training free approach that is based fluctuation pruning metric and an adaptive compression ratio.
Another typical strategy within structured pruning is depth pruning. Methods in this category remove entire transformer layers of the network. In Shortened llama [8], the authors suggested identifying the significance of each decoder layer using perplexity analysis and Taylor metric. This metric is based on similar idea with the LLM-Pruner importance metric, that is, it measures the difference of the model loss when computed with and without pruned layer. After pruning, the authors [18] further propose healing via LoRA fine-tuning, continual pre-training, or their combination.
The authors ShonGGT (30 introduced the Block Influence (BI) metric to quantify the contribution of each network layer. This metric corresponds the cosine distance between the hidden states before and after the layer. After pruning, they optionally recommend retraining the model to recover the model's performance. In contrast, UIDL [13] suggested computing the importance of fixed-length sequence of layers instead of computing this metric for each layer individually. They calculate the cosine distance between the input and the output of the sequence and then the distance below a pre-defined threshold they remove the entire sequence of layers completely. Post-removal healing Strrtalee paper [3] the authors propose to
with LoRA A on the MLP is applied. In the recent replace a fixed-length sequence of layers with a lightweight network, which can be either transformer layer or a Feed-Forward Network (FFN). This network then trained using the MSE loss and the
LLM loss with LoRA. Prior work [38) has demonstrated that certain transformer blocks exhibit

---

linear characteristics, with empirical analyses revealing near- pear-perfect linear relationships between embedding transformations across layers. They suggest to substitute highly linear transformer blocks with linear layers, which facilitates efficient model compression through direct knowledge distillation. Furthermore, the study proposes incorporating a seiine-simllaritybbassee during pretraining to mitigate excessive linearity. Recently, the Minitron L.M family [35) and its pruned variants were introduced, demonstrating an effective balance between depth and width pruning to mitigate performance degradation. The approach estimates the relative importance of depth and width dimensions to achieve an optimal trade-off that minimizes performance loss. However, a notable drawback is its reliance substantial amount of data approximately 100B tokens.
regularization mechanism
5 Limitations
While the effectiveness of our proposed method is mathematically justified and experimentally validated, it is important to note that due to the training- free nature of ReplaceMe, such effectiveness is primarily evident within certain compression ratio ranges. As demonstrated in Figure ReplaceMe performs well without fine-tuning for what we refer to as "reasonable" compression ratios, which are dependent both the size and architecture of the original model. For higher compression ratios, some degree of retraining or parameter adjustment becomes necessary. However, an advantage of our approach is that such finetuning full-model fine-tuning. This selective healing preserves the computational efficiency of our method, as further detailed in the supplementary material, and contributes its overall efficiency compared to alternative approaches.
be limited to only the linear transformation, rather than requiring
6 Conclusion
In this work, we introduced a novel method named ReplaceMe, which is training-free depth pruning method. Our proposed strategy involves substituting certain transformer blocks with a linear transform, which is estimated using calibration data. ReplaceMe requires retraining or fine-tuning, yet it consistently outperforms existing pruning techniques in training-free settings and remains competitive even when compared to approaches that rely post-pruning "healing" stages. We have conducted extensive experiments and outlined methodologies to accurately estimate linear transformations under different optimality criteria using both analytical and numerical techniques. The proposed method has been evaluated across a range of transformer architectures - including large language models and vision transformers demonstrating its robustness, adaptability, and effectiveness. These results represent a significant step toward a truly training-free pruning strategy.
References
[1] Yongqi An. Xu Zhao, Tao Yu, Ming Tang, and Jinqiao Wang. Fluctuation- adaptive structured pruning for large language models. In Proceedings of the Thirty-Eighth AAAI Conference on Artificial Intelligence and Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances Artificial Intelligence. AAAI Press, 2024.
[2] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence, 2019.
[3] Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Stream- lining redundant layers to compress large language models. The Thirteenth International Conference on Learning Representations, 2025.
[4] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Humun Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers), pages 2924-2936. Association for Computational Linguistics, 2019.
10

---

[5] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc. the ai2 reasoning challenge. arXiv preprint arXiv: 1803.054377777 2018.
[6]
Benoit Courty, Victor Schmidt, Sasha Luccioni, Goyal- Kamaal, MarionCoutarel,, Boris Feld, Jérémy Lecourt, LiamConnell, Amine Saboni, Inimaz, supatomic, Mathilde Léval, Luis Blanche, Alexis Cruveiller, ouminasara, Franklin Zhao, Aditya Joshi, Alexis Bogroff, Hugues de La- voreille, Niko Laskaris, Edoardo Abati, Douglas Blank, Ziyao Wang. Armin Catovic, Marc Alencon, Michal Stęchty, Christian Bauer, Lucas Otávio N. de Araújo, JPW, and MinervaBooks. 2024.
mlco2/codecarbon: 2024.
[7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[8] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang. Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino. Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Frank Zhang, Gabriel Synnaeve. Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jeimer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang. Jiawen Liu, Jie Wang. Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The llama 3 herd of models. CoRR, 0oos/2407.21783. 2024.
[9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
[10] Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open Ilm leaderboard v2. 11m_ leaderboard 2024.
co/
11m hagpae//orrppne-n---aaaaaaarrrrr
[11] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, pages 10323 10337. PMLR, 2023.
[12] Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 2021.
[13] Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers. CoRR, 1tbs/2403.17887, 2024.
[14] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances Neural Information Processing Systems. Curran Associates, Inc., 2015.
[15] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
11

---

[16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03:000 2020.
[17] Dan Hendrycks, Collin Burss, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.
[18] Bo- Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kye Kyu Song. Shortened L.LMA: simple depth pruning for large language models. 2024.
[19] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 14/2.6980, 2014.
[20] Simon Korbblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019.
[21] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar- 10 (canadian institute for advanced research).
[22] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard H. Hovy. RACE: large-scale reading comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 785-794. Association for Computational Linguistics, 2017.
[23] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In Advances in Neural Information Processing Systems.
Kaufmann, 1989.
[24] Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
[25] Haonan Li. Yixuan Zhang, Fajri Koto, Yifci Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023.
[26] Wing Lian, Guan Wang. Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification, 2023.
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[28] Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming, 45(1-3):503-52 528, 1989.
[29] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems 36: Annual Conference on Neurul Information P Processing Systems 2023, NeurlPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[30] Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers large language models are more redundant than you expect. arXiv preprint arXiv: 2.03303853. 2024.
[31] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 2381-2391. Association for Computational Linguisties, 2018.
12

---

[32] Ari S. Morcos, Maithra Raghu, and Samy Bengio. Insights on representational similarity in neural networks with canonical correlation, 2018.
[33] Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi. Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. CoRR, abs/2407.14679 2024.
[34] Jorge Nocedal and Stephen J. Wright. Numerical Optimization. Springer, New York, NY, 2 edition, 2006.
[35] Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset, 2016.
[36] Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. The fineweh datasets: Decanting the web for the finest text data scale. arXiv preprint arXiv: 2406 1 17557, 2024.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning, pages 8748-8763, PMLR, 2021.
[38] Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Nikolai Gerasimenko, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov. Your transformer is secretly linear. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 5376-5384, Bangkok, Thailand, 2024. Association for Computational Linguisties.
[39] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
[40] Yuxin Ren, Benyou Wang, Lifeng Shang, Xin Jiang, and Qun Liu. Exploring extreme parameter compression for pre-trained language models, 2022.
[41] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: an adversarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106, 2021.
[42] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020.
[43] Shivalika Singh, Freddie Vargus, Daniel Dsouza, Börje F. Karlsson, Abinaya Mahendiran, Wei-Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura OMahony, Mike Zhang, Ramith Hettiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzemiñski, Hakimeh Fadaci, Irem Ergün, Ifeoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafcai, Vu Minh Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Üstün, Marzich Fadace, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning, 2024.
[44] Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. Musr: Testing the limits of cf oftthuught with multistep soft reasoning, 2024.
[45] Trond Steihaug. The conjugate gradient method and trust regions in large scale optimization. SIAM Journal on Numerical Analysis, 20(3):626-633) 637, 1983.
[46] Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension. Transactions of the Association for Computational Linguistics, 8:141-155, 2020.
13

---

[47] Mingjie Sun, Zhuang Liu, Anna Bair. and J. Zico Kolter. A simple and effective pruning approach for large language models. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
[48] Qi Sun, Edoardo Cetin, and Yujin Tang. Transformer-squaree:: Self-adaptive Ilms, 2025.
[49] Qi Sun, Mare Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers painters, 2025.
[50] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoe V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challeng- ing big-bench tasks and whether thain-of-thoughl can solve them, 2022.
[51] S. Umeyama. Lqaares estimation of transformation parameters between two point patterns. IEFE Transactions on Pattern Analysis and Machine Intelligence, 13(4)3376-880 1991.
[52] Tycho A. van der Ouderaa, Markus Nagel, Mart van Baalen, and Tijmen Blankevoort. The LLM surgeon. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.nec, 2024.
[53] Xin Wang, Yu Zheng, Zhongwei Wan, and Mi Zhang. SVD-LLM: Truncation-aware singular value decomposition for large language model compression. In The Thirteenth International Conference on Learning Representations, 2025.
[54] Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. Mmlu-pru pro: A more robust and challenging multi-tass language understanding benchmark, 2024.
[55] Johannes Welbl, Nelson Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. In Proceedings of the 3rd Workshop Noisy User-generated Text, NUT@ EMNLP 2017, Copenhagen, Denmark, September 7, 2017, pages 94-106. Association for Computational Linguistics, 2017.
[56] Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark. arXiv preprint arXiv:2004.
2020.
[57] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv:1905.07833 2019.
[58] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv: 1910.04867, 2019.
[59] Chujie Zheng. Minlic Huang, and Aixin Sun. Chid: A large-scale chinese idiom dataset for cloze test. arXiv preprint 1XXv::106..01255 2019.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The claims mentioned the abstract and introduction are elaborated Sec. (Method) and further supported by experimental results in Sec. 3 Furthermore, extensive ablation studies and detailed explanation of experimental setup can be found Appendix.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the proposed method are discussed Section (Limitations) and also mentioned throughout the manuscript.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [Yes]
Justification: We provide a mathematical explanation for each step of the proposed method in Moreover, one find step-by-step derivations in the supplementary material.
Sec. 2.1 2.3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide publicly available library facilitate the reproduction of results. In the main text, we discuss the statistical stability of the proposed method, while the supplementary material presents detailed analysis of its statistical significance. To validate consistency, we ran the method with different random seeds and observed identical results for the analytical solution, with only negligible variations in the numerical solution (standard deviation of approximately 0.08).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility.
16

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide a library that supports our proposed method and the state-of-the-ar UIDL method. Library can be easily installed to reproduce any experiment with a single command. All data used are accessible and publicly available, except for the self-model generated data, which we provide with the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
tablic//uudees/Coeetttiiii wwtpaaumiissiioooooooooooo onoliicy))) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips. ptaips..c/pulicc//uiice guw.///ieee//ee// Codeeubmii ssi asionPolicy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: our method is training-free, we do not require traditional training details. However, for the calibration data splits and hyperparameters used in the numerical estimation. we provide full details in the experimental setup sections, namely Sec. Sec.B.] [nd and Sec. 3.3.1 Additionally, the supplementary material includes a comparison of performance using different optimizers.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
17

---

Justification: As a training free method, the Statistical Significance information are need only for the numerical estimation, which we provide on the appendix but not on the main paper text.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots. The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: These details are provided in the experimental setup (see Sec. 3.1 Addition- ally, we present comparative analysis environmental impact of the proposed method against existing approaches.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The current work follows with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
18

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: none observable negative societal impact is feasible with the proposed method
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper proposal is a training-free pruning method, the limitation of any model is transferable to the pruned version of that model.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: This work will be published under an open-license such MIT or Apache 2.0, but the models and datasets license follow the originals.
19

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:[Yes
Justification: all new assets introduced in our paper are thoroughly documented. For every pruned model released, we provide comprehensive documentation and evaluations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This is not relevant to the paper.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
20

---

Answer:
[NA]
Justification:
This work does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
21

---

A Appendix
In this section, we revisit the central aspects of our research, starting with the closed-fome solution
presented in A.2 This provides foundation for examining structured linear transformations (LTs) as
discussed in A.3 leading to key experimental findings.
We evaluate multiple numerical solvers to minimize the cosine objective (A.6) and investigate the behavior of structured LTs. We then assess the applicabilityy of our method on diverse model architectures, demonstrating the effect of the applied "healing" process on the proposed ReplaceMe. In addition, we include a computational comparison between ReplaceMe and competing approaches, as those introduced in our initial study.
Further, we dissect the role of integrating linear layer as standalone block between full layer activations. This is in contrast to our approach of merging LTs with MLP activation in the primary architecture. The obtained results reveal that both methods yield remarkably similar results.
Finally, to validate the robustness and generalization ability of our method, we evaluate several models wide range of benchmarks. These experiments demonstrate the stability and adaptivity of our approach in varying conditions.
A.1 Terminology and Definitions
Relative Performance (RP): normalized metric quantifying performance relative to the baseline model, computed as the ratio of benchmark results between the evaluated model and the baseline.
Compression Ratio: The proportional reduction in model parameters expressed as a percent- age, calculated as:
N Npraned
100
(10)
Noriginal
where Npraned and Noriginal represent the number parameters in the compressed and original models, respectively.
ReplaceMe: Our proposed method for model compression, which replaces complete trans- former blocks with optimized linear transformations.
Average Accuracy (Avg-acc): The arithmetic mean of model accuracy scores across multiple benchmark datasets.
Perplexity: A metric for evaluating language model performance, defined as the exponential of the cross-entropy loss on the Lambada-openai benchmark.
A.2 Closed-form solution for L2 Distance
The optimization problem for the linear transform matrix estimation can be expressed as follows:
T arg min [MM; T Y,) - L1+11
The objective is an Fuclidean norm and thus we can expand it to:
R 2 = tr (M, T + Y. L.n)) (M. T Y, - Liin)) =t (T (M. M,J + L44h) MM TM, T L Iitn)
(11)
To minimize the objective, take the gradient with respect to T and set it to zero:
Vr R 0
(12)
2M M, T 2M, L Litn) 0
M. T M. (Litn - Y,,
T T"= (M, (M) MM (Litn Y.)
22

---

A.3 Structured LT Matrix
To improve the interpretability our approach, we further investigate additional constraints that can be imposed on the structure of the linear transformation T. The results of all the different constrained transformations are presented in See A.4 Motivated by the Transformens-squaree work [48], we also consider the case where we condition T to be a diagonal matrix. This constraint is meaningful under the assumption that an adequate mapping of the activations is possible through only scaling of the hidden states. In this case, the optimization problem is of the form:
T arg min (M, T Y.)) Litall2
TEDexu
where D denotes the space of diagonal matrices of dimensions d x d. The corresponding closed-form solution is given by:
T (M,) M.) 1) 1 ((M. (Litn ol),
(13)
where denotes the Hadamard product.
Another constraint that can be imposed on ' is the requirement for it to represent an orthogonal transformation [5I:
T min (M, T Yi) L,4 s.t. T
This problem admits an analytical solution via singular value decomposition. Specifically, let SVD(M, (Li+n Y,)) U U.E. V then the optimal orthonormal matrix is given by:
T U.V..
A.4 Results of Structured Linear Transformations
Our evaluation protocol. summarized Table 7 explores constrained forms of the linear trans- formation matrix T E ddd Inspired by the architectural design of Transformer-square we first examine the case where T is restricted to a diagonal matrix, i.c., T (fi..(1...... ta). Although this parameterization successfully restored model functionality, the resulting perplexity P remained suboptimal, with Paing > Peenerics where Pgenerie represents the baseline achieved using an unconstrained full matrix.
[48).
Model
LT Structure
Multi-linear Transform
Avg-acc
Perplexity 4
% 1
Llama3 8B
0.70
3.11
100.00
UIDL
0.58
2216.96
82.5
ReplaceMe
Generic
X
0.62
21.21
89.59
ReplaceMe
Orthonormal
X
0.60
700.57
85.67
ReplaceMe Diagonal
X
0.62
89.09
88.42
ReplaceMe Generic
non-consecutive
0.62
16.07
89.60
Table Ablation study on (un)constrained LT matrix structure (Generic, Diagonal, Rotational) and multi-linear transforms, The base model is set to Llama 3 8B with 25% pruning ratio.
Subsequent experiments with orthogonal transformations T (where T T I) demonstrated limited efficacy, yielding only marginal performance recovery. This indicates that actually scaling is much more important to recover model performance than rotations or reflections.
More promising results emerged from employing multiple linear transformations (T.) We examined the following approach: Non-consecutive transformations: For disjoint parameter subsets, we learned independent transformations [T.iijjiin }
1--1- The experimental results indicate that applying multiple transformations yields consistent improvements both perplexity (P) and accuracy (AA) across both analytical and numerical solutions. For the analytical approach, these gains are achieved in single computation step, whereas the numerical solution requires iterative optimization, leading to significantly higher computational costs.
23

---

A.5 Statistical Significance
To ensure reproducibility and stability of our method we ran our method multiple times for Llama- 3-8B- 3-8B-Instre Instruct with 8 pruned layers. While the analytical solution always leads to the same results, numerical methods might have different results. Howewer, we were able get almost the same results: (1) the mean average accuracy over all benchmarks equals to 60.71 71 and the standartd deviation (std) is 0.08; (2) the mean lamhada-openai perplexity equals to 15.90 and the standartd deviation (std) is 0.05.
A.6
Comparative Analysis on the Performance of Numerical Solvers
This section investigates the effects of different optimization solvers when minimizing the cosine distance objective function. The previously reported results were obtained using the Adam optimizer; here, perform systematic comparison using alternative solvers, including: 1) Non-Linear Conjugate Gradient (NICG), Newton-GG, and Limited-memory BFGS (L-BFGGS). Additionally, we compare the results obtained when minimizing the Mean Squared Error (MSE) either via numerical optimization or using the Least Squares (LS) closed-form solution. These comparisons provide insights about the solver efficiency and the solution accuracy for different objective functions. As
Solver
Avg-acc
RP
Percentage
Baseline 19
0.697
3.106
100
Adam 19
0.634
15,875
90.940
NLCG
0.630
15,960
90.412
L-BFGS 28
0.634
16.200
90,921
Trust-NCG
0.620
47,000
88.945
LS
0.624
21,206
89.588
MSE Adam
0.624
20.633
89.536
Table 8: Comparison between different solvers to estimate the linear transform for Llama 8B after 25% compression with cosine distance objective, and sanity check between MSE with Adam solver as a numerical solution and LS analytical solution
demonstrated Table nearly all solvers converge to similar point, with results on both benchmarks and perplexity metrics being very close. An exception is the Trust-NCG solver, which seems to got trapped in local minimum. Furthermore, there is clear indication that miminizing the MSE, either numerically or analytically, yields nearly identical outcomes. Nevertheless, the analytical solution is much faster and has much less hardware requirements.
A.7 Generalization Across Model Scales
While the primary results presented in this work focus on a subset model architectures, these findings may fully capture the scaling behavior across a broader parameter spectrum, where model sizes can vary from (1) billion to (70) billion parameters or beyond. To rigorously investigate the
scaling properties of our approach,
conduct extensive experiments across the following benchmark
suite: (Winogrande 41. BoolQ 4 OpenBookQA (1), SciQ (55), Race (2 and PIQA (2).
Our scaling analysis involves the Llama architecture family, from the compact Llama 3.2 (1B
parameters) to the largest available variant, Llama 3 (70B parameters). As evidenced in Table 9 we
1.ama-3.2-1B-Imms
Mama--.1-8B-Insteee
1lama-3-70B-Instaen
Model
Baseline
ReplaceMe
Baseline
ReplaceMe
Baseline
ReplaceMe
ReplaceMe
Avg-Ace
0.6098
0.5348
0.7118
0.6537
0.728
0.7036
0.6596
1.0
0.8771
1.0
0.9184
0.9664
0.9060
Compression Ratio
0%
25%
0%
25%
0%
25%
37.5%
Table 9: Results of utilizing ReplaceMe on various model sizes, characterized by the number of parameters, for the Llama 3 Families, For Llama-3-70B Instruct we show ReplaceMe results with 25% and 37.5% compression ratios.
observe strong correlation between model size and achievable compression ratio n. Specifically, for

---

large-scale models (70B parameters), we achieve optimal compression ratios of 7max = 37.5% while maintaining performance retention RP 2 90% of the original model's capability.
A.8 Computational Efficiency Analysis
This section presents a quantitative comparison between our training-free method and the baseline UIDL framework, with focus computational overhead, energy consumption, and associated CO2 emissions. As demonstrated in the Table [1] our approach achieves competitive performance despite being training- free,, whereas UIDL requires fine-tuning a limited dataset. To ensure equitable comparison, we adopt this configuration to rigorously evaluate the computational advantages of our method. Furthermore, healing and the complete model healing approach. As demonstrated in Section[A. our methodology requires only the healing of the LT component while maintaining competitive performance relative to the full-model healing. For the UIDL baseline, we adhere to the original implementation of the authors, which employs LoRA-based healing exclusively on the MLP layers, As demonstrated in Fig.
conduct comparative analysis between our proposed method with partial
Normalized Performance Comparison (Relative to Best in Each Category)
170.1x
Methods
ReplaceMe
107.7x
107.2x
107.1x
ReplaceMe Cosine
Relative Performance (log scale)
102
77.5x
ReplaceMe with bealing only LT
UTDL with Healing
10t
6.6%
3.5x
3.5x
1.0x
1.0x
1.0x
10"
Duration
Emissions
Energy
Performance Metric
Figure 4: Comparison between ReplaceMe and UIDL in terms of computation and environmental impact.
a comparative analysis of our two proposed methodologies is presented. The L. approach offers significant advantages in terms of cost-efficiency, speed, and the reduced demand for computational resources and memory. However, this method incurs a slight decline in performance metrics. Thus, the choice between these methods should be informed by the specific requirements and available hardware resources of the user.
A.9 Mergable LT vs. LT as an Independent Block
LT
Objective
Avg-accuracy
Perplexity RP
Fusable into MLP
Cosine
0.634
15.875
0.905
Separate LT block
Cosine
0.640
13.481
0.913
Table 10: Compressing Llama 8B by 25% using a cosine distance objective, without applying healing. This involves applying LT the MI.P output the full output activation of the transformer block.
25

---

In our work, we propose integrating the linear transformation (LT) matrix so that can be merged with the down-projection component of the MLP layer, preserving architectural compatibility with standard I.LM designs. However, this approach raises critical question regarding the comparative efficacy of injecting the I.T' as a mergable layer versus incorporating it as an independent, non-mergable block within the transformer architecture. In Table TO] we present the outcomes of integrating the linear transform (LT) directly into the Multi-Layer Perceptron's ' (MLP) down-projection through matrix fusion. This approach ensures parameter efficiency in contrast employing standalone LT Block, where the LT functions as an independent layer between transformer blocks, thereby introducing additional computational overhead. The results demonstrate marginal improvement; however, the extent this improvement is contingent upon user speciscciiiic requirements.
A.10 Block Selection Analysis
In this section, we conduct systematic ablation study to investigate the relationship between inter-layer activation distance metrics and model performance under sequential layer removal. We employ a sliding window approach, iteratively pruning contiguous blocks of eight layers starting from the initial layer index and incrementally shifting the window by single layer position. For each configuration, computed between intermediate activations of the original and pruned models. Subsequently, for method estimate the linear transformation (LT) and evaluate the pruned architectures standardized subset benchmark tasks. As demonstrated Fig. 5 we observe a strong inverse
quantify activation dissimilarity using both cosine distance and L2 norm,
Accuracy vs Distance
Cosine Dist Adam
Cosine Dist LS
0.45
Cosine Dist UIDL
Trendline Adam (Cosine)
Trendline LS (Cosine)
Cosine Distance
0.40
Trendline UIDL (Cosine)
L2 Dist Adam
L2 Dist LS
0.35
L2 Dist UIDL
Trendline Adam (L2)
Trendline LS (L2)
Trendline UIDL (L2)
0.30
0.25
0.30
0.35
0.40
0.45
0.50
0.55
0.60
Accuracy
Figure 5: Comparative analysis of distance metrics and predictive accuracy across layer pruning configurations. Trendlines illustrate the inverse correlation between cosine distance and accuracy, contrasted with the positive correlation between L2 and accuracy degradation Results are shown for ReplaceMe with LT estimation via Cosine/LS metrics and UIDL baselines.
correlation between cosine distance reduction and accuracy improvement across all methodologics. Specifically, ReplaceMe with cosine-based LT estimation achieves peak performance at the lowest cosine distance values, Conversely, configurations exhibiting lower L2 correspond to significant accuracy degradation suggesting that L2 lacks power for optimal layer selection.
In Fig. 6 we present the detailed results sequential layer removal and compare cosine distance with other similarity measures: CKA ( and CCA (32]. We observe that the highest accuracy corresponds to the lowest cosine distance, which confirms the validity of using cosine distance as block selection criterion.
26

---

It is also important to note that the optimal pruning range depends on both the architecture and the number layers removed. Previous works have also observed that layers form natural groups 449 We conducted a similar experiment and plotted the cosine distance across all layers in Fig.
0.6
0.5
Metric Value
0.4
Avg. acc.
Cosine dist.
CCA
0.3
1 CKA
0.2
0.1
1-9
2-10
3-11
4-12
5-13
6-14
7-15
8-16
9-17
10-
11-19
12-28
13-21
14
4-22
15-23
18-24
17-25
18-2
19-27
20
28
21.29
22
22-30
23-31
24-32
Pruned Layers
Figure 6: Comparison of cosine distance, CCA, and CKA across all 8-layer pruning configurations in
Llama3 8B.
A.11 Cosine Distance Approximation
In this section, we discuss further our proposed approximation the loss, where we subtract attention output from the activation after the cut, when using the cosine distance. First, revisit the original Eq. 14)
T arg min cos (M, T Y,, L(+n),
(14)
T
where we observe that in order to estimate T numerically we need to store 3 activations (M, Y,, Litn) for each token. This is not effective and requires considerable memory and time to compute. To overcome this issue, we apply the cosine distance after subtracting the attention output from the full activation at the end-block of the cut as formulated in Eq. (15):
arg min cos (M, T,L Y.).
(15)
As evidenced by the experimental results presented in Table II the approximated cosine formulation
Model
Method
Pruned Layers
Calibration
Training Perplexity
Avg-acc
Lluma 8B instruct
approx. loss orig. loss
8
slim slim prea
training no training
15.88
0.634
Llama 8B instract
8
16.63
0.630
Qwen 2.5 7B instruct
approx. loss orig. loss
7
slim orea
slim_orca
no training
7.92
0,591
Qwen instruct
orea
training
Llama 8B instruct
Multi A approx. loss Multi_A A orig. loss
8 slim, slim_orea
training
13.95
0.628
Llama 8B instruct
8
no training
13.12
0.630
Table 11: Comparison between the original loss and the approximated version. The approximated version makes use of the cosine distance between the transformed MLP output and the residual of the activation of (i + n) block and the attention of the i-th block.
achieves comparable performance to the exact loss computation, with marginal improvements observed. This approximation yields memory efficiency gains, requiring only two stored activations per token (M, L Y,). thereby reducing memory usage by approximately 66% relative the original implementation.
27

---

Cosine Distance Confusion Matrix
-0.4
6
8
10
Sample Index
12
0.3
14
16
18
0.2
20
22
24
0.1
26
28
30
4
6
10
12
14
16
18
20
22
24
26
28
30
Sample Index
Figure 7: Average cosine distance between hidden states of layers for Llama 3 8B. Each cell (i shows the cosine distance between the output vectors of layer, and layer computed across 256 samples of SlimOrca.
A.12 Data ablation
LS
Cosine
Llama3 8B (Baseline)
Average Accuracy vs. Calibration Samples
Perplexity vs. Calibration Samples
Average Accuracy
0.8
Perplexity
20
0.6
0.4
10
0.2
0
0.1
0.4
0.8
1.2
1.6
0.1
0.4
0.8
1.2
1.6
Calibration Samples
-104
Calibration Samples -10
Figure 8: Pruning Llama 8B by 25% using different number of samples to estimate the linear
transform
Impact of Calibration Dataset Size Figure 8 illustrates how the size of the calibration dataset affects the linear transform estimation, for both the L2 and the cosine distance objectives. Although increasing the number of calibration samples does not significantly improve benchmark accuracy, does substantially reduce perplexity, especially when the cosine distance objective is considered.
The linear transformation matrix has shape d x d, requiring approximately N = d2 tokens for reliable estimation. With a per-sample sequence length of this translates to least d samples. For example, in LLaMA 3 8B, where d = 4096 and S 1024, about 16,000 samples are theoretically
28

---

needed. However, as seen in Fig 8 accuracy remains stable even with as few as 1,000 samples, suggesting robustness sample size. That said, perplexity continues to improve with more data, indicating better model confidence and predictive quality.
Impact of Masking for Data Augmentation We have also investigated random token masking as lightweight data augmentation technique for scenarios with limited calibration data (e.g., 1,000 samples). As shown in Table 12] masking improves the stability of the numerical optimization and leads to better convergence when estimating the linear transformation. This masking strategy proves especially beneficial in low-data regimes, where it reduces overfitting and enhances generalization. However, when more data are available, the impact of masking becomes negligible.
In summary, calibration dataset with approximately de tokens ensures stable and accurate estimation. When working with fewer tokens, random masking can mitigate overfitting and improve estimation quality. For instruction-tuned models, instruction-style calibration data consistently leads to better pruning outcomes. While self-generated data can reduce perplexity, it may degrade benchmark accuracy, highlighting trade-off between confidence and task-specific performance.
Model
Masking Calibration Data
Avg-acc
Perplexity
% t
Llama3 8B
0,697
3.10
100.00
ReplaceMe
x
lk
0.632
24.96
90.62
ReplaceMe
/
1k
0.634
21.08
90.91
ReplaceMe
X
8k
0.633
15.88
90.76
ReplaceMe
8k
0.632
15.69
90.73
Table 12: Random token masking during the estimation of the linear transformation contributes to more stable solution, particularly when working with small datasets.
A.13 Extra Model Evaluation
To further validate the efficacy of our pruned models, we conducted an additional evaluation on a new subset benchmarks derived from the Huggingface Leaderboard (10), utilizing modified version of Eval-Harness [1]. This benchmark set encompasses Big Bench Hard (BBH) comprising 23 complex
and diverse tasks [50]; High-school-level mathematical competition problems [7]: PhD-level domain
expertise assessments across disciplines [39]; Algorithmically generated intricate problem sets 44]
refined version of (MMLU) benchmark 54
As shown in Fig. 9 the normalized results demonstrate that the Llama2 model preserves performance across nearly all benchmarks post-compression. Notably, Llama3 variant has a marginal performance degradation on the mathematical benchmark, potentially attributable to task-specific sensitivity to parameter reduction. Depending on the user's specific needs, additional healing process can be considered to overcome this issue.
A.14 Healing Experiments
Here, we present a comparative evaluation of our method in three experimental configurations: (1) the baseline implementation without any healing mechanism, (2) partial healing only the learned linear transformation (LT) layer, and (3) complete model fine-tuning using subset of the C4 dataset.
Method
Training
Winogrande
PIQA
BoolQ
OpenBookQQ
SciQ
Perplexity
Avg.
Llama 7B
None
0.3952
0.7427
0.7900
0.7783
0.4400
0.9050
3.3973
0.6752
ReplaceMe
None
0,3656
0.6977
0,7051
0.7263
0.3460
0.8370
22,3586
0.61295
ReplaceMe
Only LT
0.3799
0.7182
0.7350
0.7728
0.3760
0.8840
5.3965
0.64432
ReplaceMe
Full model
0.3847
0.7056
0.7416
0.7835
0.3920
0.8720
4,9138
0.64657
Table 13: Performance comparison of ReplaceMe with/without healing and with different trainable parameters.
As demonstrated in Table 13 the model variants which involve healing yield performance improve- ments. Notably, the LT-only healing approach achieves comparable accuracy to the full model
29

---

Inormalized baseline)
GPO4
GPO4
1.0
1.0
88
BBH
B8H
MUSR
MUSA
MMLU-PRO
KMLU-PRO
Figure 9: Comparison between our proposed methods and UIDL on a different set of benchmarks. Results are normalized relative the baseline model performance.
fine-tuning while offering substantially reduced computational costs. This selective healing strategy demonstrates superior efficiency compared to alternative healing approaches documented in prior work.
A.15 Computation environment
All experiments were conducted using an NVIDIA A100-SXM4-40GB GPU with an AMD EPYC 7742 64-Core Processor, running Ubuntu 22.04 and Python 3. The software environment was based on the official NVIDIA PyTorch container nver weo/ii/viii a apppt//cttth23333 23..1tttttr.....pppp.... 10-py3.
For additional testing and validation, models were also tested on P100 GPU using the Kaggle environment, which imposes stricter compute and memory constraints.
The computational setup and training configurations summarized in tables: Table 14 details the hyperparameters for ReplaceMe(cosingg and Table [5 compares the two healing experiments: training only LT versus full-model fine-tuning.
Parameter
Value
Optimizer
Adam
Learning Rate
0.0001
Batch Size
1024
Epochs
10
Loss Function Weight Initialization
Cosine Distance Loss
Identity Matrix
Bias
False
Table 14: Hyperparameters and Configuration for ReplaceMe(Cosiie-baeed Training)
A.16 Multi-Linear transformations
To isolate the impact of number of LT's, we fix the total pruned layers at eight and vary how many of those are implemented as LT: 1, 4, or 8, then benchmark both Llama-3-8B-Innsee and Mistral-7B. Instruct-v0.3 on the same benchmarks. We approximate using ReplaceMe((.S) method. Table T6 summarizes the results.
30

---

Parameter
Only LT Training
Full Model Training
Optimizer
Adam
Adam
Context Length
2048
2048
Learning Rate
3e-4
le-5
Number of GPUs
4
Batch Size (per device)
1
Max Steps
80,000
20,000
Gradient Accumulation Steps
1
1
Gradient Checkpointing
True
True
Unfrozen Weights
LT Only
Full Model
Trainable Parameters
16.8M
6.3B
Table 15: Hyperparameters for Healing Experiments (full transform full model)
Method
Number of L.Ts
Race
Winegrande
PIQA
BoolQ
OpenBonkQA
SciQ
Perplexity
Acc
Ilama-3-33 3-8B----B---- 4B-Instruct
ReplaceMe
0,3694
0.7167
0.6844
0.8061
0.3300
0.8400
21.2061
0.6244
ReplaceMc
0,3885
0.7261
0.6872
0.7798
0.3380
0,8580
18,9853
0.6296
ReplaceMe
0,375
0,7277
0.6953
0.7661
0,3240
0,8590
16,0669
0.6245
ReplaceMe
Distral-7B-Inttrcctta
ReplaceMe
0.3400
ReplaceMe
2
0.4134
0.6156
0.7486
0.6985
0.3580
0.9020
7.11199
0.6227
ReplaceMe
0,4182
0.7119
0.7040
0,8287
0.3640
0:9140
5,9130
0.6568
ReplaceMe
0.4077
0.6551
0.7318
0.8174
0.3820
0.9120
5.9897
0.6510
Table 16: Performance comparison of ReplaceMe(LSS) with the different number of L.Ts. We fix
total of 8 pruned layers and vary how many LTTs we insert. Reported are task accuracies on the same set of benchmarks for both Llama-3-8B Bomma-3-8BIIntruuee IB-Instruct and ---nal-----nnnttt--l
For both models, using two or four LTT modules tends to offer the best trade-off between task performance and language modeling quality. However, the differences across configurations are relatively small, and results are largely comparable. Fewer LT modules may under-adapt the pruned model, while more modules can introduce unnecessary complexity. The adaptation method appears robust to moderate variation in the number modules.
A.17 Task specific LT
We ran additional experiment to see how the choice of calibration data affects results. In this test, we used the training part of the SciQ dataset as calibration data and then measured performance on its test set. For comparison, we also applied our method using part of the general SlimOrca dataset but still evaluated it on SciQ. Using the Llama3 8B model, we found that calibration data from the same task leads to much better accuracy than calibration with general-purpose data. This shows that tailoring calibration data to the target task can significantly improve compressed model performance. See the results in Table [7]]
Model
Calibration Data
Compression
Sciq Accuracy
Llama3 8B instruct
0.93
UIDL
Sciq- Task specific
25%
0.687
Ours (LSTSQQ)
Sciq- Task specific
25%
0.89
Ours (LSTSQ)
Orca General
25%
0.858
Table 17: Performance comparison of Llama3 8B models calibrated with task-specific (SciQ train split) versus diverse open-source (Orca subset) datasets on the SciQ test split
31


Reviews:
[{"review_id": "NiiyAXm9KT", "reviewer": "Reviewer_MdC9", "summary": "This work proposes a structured pruning approach that removes transformer blocks and replaces them with a learned linear transformation. The authors argue that this method is training-free unlike other compression methods which require some fine-tuning (healing). The linear layers are estimated using a calibration dataset by matching the activations of before and after the blocks to be removed. L2 distance and cosine distance metrics are used to estimate the linear transformation from a small dataset. Experiments are comprehensive, and show that for compression ratios up to 25% the proposed method compares favourably with other methods.", "questions": "1. Why is this strategy well suited for transformer blocks. Why can a similar linear transformation not be used for other deep CNNs? \n\n2. I would like the authors to speculate on the importance of choosing contiguous blocks versus optimizing the block sequence itself. \n\n3. Clarification on performance at high compression ratio (see details under weaknesses)", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "* The idea of replacing structures within a neural network with learned linear transformations is a simple but elegant idea. \n\n* The justification of which blocks to prune based on the distance between activation maps is reasonable. \n\n* The distance metrics used are relatively straightforward. Both L2-norm and cosine distance seem to work quite well for \"reasonable\" compression ratios.\n\n* Reduction in memory achieved by using Eq. 8 is a nice addition to the contribution.\n\n* Experiments are comprehensive; both language and vision transformers are assessed. Many recent baseline methods are also reported. \n\n* While the compression gains are modest, the strength of the method as the authors argue lies in two factors: use of linear layers, and no fine-tuning after compression. \n\n### Weaknesses\n\nI have some clarifying questions. \n\n* **Justification for removing entire blocks**: The prune and replace strategy empirically seems to work for reasonable compression ratios. How is it that these entire complex blocks so redundant? The authors point to UIDL paper as the justification. But, I would have also liked to see this argument laid out in the paper. Is it because some of the deeper layers not very useful? If so, why? Can this not be done already from the outset? Can this prune and replace strategy be used during training also? What makes this so unique to transformer blocks, and can it be applied to CNNs? If not, why?\n\n* **Choice of blocks**: In Eq. 4 authors mention that _n_ blocks can be removed and the optimal index is obtained by minimizing the distance between transformer blocks. Why do these blocks have to be contiguous? The authors do report some of this analysis in the Appendix (Fig. 5) but I would have liked to get some justification for this in the main text too. \nIt would also be interesting to perform some form of a greedy search over the different transformer blocks. It would then be a sequence selection problem which would output the set of indices that need to be pruned. I can see that replacing blocks with a linear transformation is more practical, but this could also be done when disconnected blocks are removed, if the linear layer is still used just after the last removed block. \n* **Difficult results tables**: I found it hard to parse the results tables. It could simply be because the authors do not elaborate the metrics being reported. For instance, what is the last column in Table 3? It just says %. I initially read it as compression ratio which you would like  to increase but then the baseline is at 100%. \n \n* **Performance at high compression ratio**: In L189, the authors point out that at higher compression ratios their method is able to recover some performance after fine-tuning. I do not immediately see where these results are reported. Are these the results still in Fig. 3? If so, I do not understand this. \nI mention this because, this would be useful to see. My hypothesis is that at higher compression ratios even large fine-tuning cannot recover the performance as linear transformations are not as expressive as something like LoRA.\n\n* **Related work**: I find this work more closely related to the class of model compression methods that use factorization, and less to structured pruning literature. Perhaps the authors should consider describing works like [1] in their related work.\n\n### Other Comments\n\n* While I usually like a summary figure on page 1, I do expect it to be fully described. The figure 1 is placed even before the abstract and none of the abbreviations are introduced at this point; so the figure is misplaced. I would have preferred the method overview figure 2 in its place as it is quite self-contained.\n\n* If reporting carbon emissions, make sure if it is CO2 or CO2 equivalent, as the latter is the more common measure.  \n\n### References\n\n1. Ren et al. \"Exploring Extreme Parameter Compression for Pre-trained Language Models\" (2022)", "was_revised": true, "final_justification": "The authors have satisfactorily addressed my concerns.", "rebuttal": "### **Justification for Removing Entire Blocks:**\n**Comment (1):** How is it that these entire complex blocks so redundant? ... Is it because some of the deeper layers not very useful? If so, why? \n\n**Response (1):** The reviewer raises a valid and interesting question. While we don't have a definite answer, we hypothesize that large transformer models are often over-parameterized. This is supported by the fact that older models (like Llama 1 and 2 7B) are more compressible than newer ones (like Llama 3 8B), with the newer models being trained on significantly more data. Same findings extend to Qwen models (trained on 18T tokens), which are less compressible than Llama 3 (15T tokens). This suggests that better-trained models use their capacity more efficiently. Additionally, Huge models (e.g., Llama 3 70B) can be pruned more aggressively (up to 35%) with minimal performance loss. \n\n\n### **Why Not Remove Redundant Blocks from the Start?**\n**Comment(2):** Can this not be done already from the outset? Can this prune and replace strategy be used during training also? \n\n**Response (2):** While our findings in this work suggest that it is possible to achieve the same performance with smaller models, trying to train from the start such a small model that matches the performance of the original larger model can be very challenging. The main reason is that the objective function being minimized during training is highly non-convex and there is always the possibility of landing to a \"bad\" local minima. Prior work has demonstrated that over-parameterization can help during training, makes the overall loss more amenable to optimization and improves convergence (we refer to the work on ExpandNets [Guo, Shuxuan and Alvarez, Jose M. and Salzmann, Mathieu, ExpandNets: Linear Over-parameterization to Train Compact Convolutional Networks]). Therefore, while it might be possible to design more efficient architectures upfront, current training methods seem to benefit from more parameters.\n\n\n### **Applicability to CNNs:**\n**Comment (3):** Why is this strategy well suited for transformer blocks. Why can a similar linear transformation not be used for other deep CNNs? What makes this so unique to transformer blocks, and can it be applied to CNNs? If not, why?\n\n**Response (3):** The main reason we have focused on transformers is that they involve significanlty more parameters than CNNs, which casts pruning more impactful. While we don't believe that there is any particular reason why the same strategy couldn't be applied also to CNNs, their smaller size will most probably translate to less significant gains. However, exploring this direction could be an interesting future research direction.\nFurthermore, CNNs have different dimensions in between blocks due to the pooling operation that is typically used. In such case the learned linear transformation cannot be merged as in transformer architectures and will introduce additional parameters in the network. This will lead to smaller compression ratios given that CNNs rely on structured matrices that already use significantly less parameters than generic weight matrices.\n\n### **Choice of Blocks**\n**Comment (4):** I would like the authors to speculate on the importance of choosing contiguous blocks versus optimizing the block sequence itself, Why do these blocks have to be contiguous? ... the linear layer is still used just after the last removed block.\n\n**Response (4):** We thank the reviewer for his suggestion. In our experiments, we have explored both consecutive and non-consecutive block removal (Appendix Table 15). While contiguous blocks are often optimal (as cosine distance tends to favor them), the best grouping can vary by model. For example, Llama 3 performs better with 2 groups of 4 continuous blocks, while Mistral works best with 4 groups of 2 consecutive blocks. \nIn practice, we have observed that even when we split into groups, in most cases the chosen groups tend to also be consecutive to each other. In addition, contiguous groups allow the merged linear transformations (LTs) to be combined into a single LT, simplifying the overall process. This is also the main reason that in the main paper we report only results with a single LT. Nevertheless, our library supports both strategies, with the greedy/non-contiguous search under the $\\ell_2$ loss (LSTSQ method) being fast. We will add a discussion of this in the revised version of the paper.\n\n\n### **Difficult Results Tables**\n**Comment (5):**  what is the last column in Table 3? It just says %.\n\n**Response (5):** We thank the reviewer for highlighting this lack of clarity. The \"%\"  in Table 3 denotes the model’s performance relative to the original (uncompressed) model, where 100% is the baseline. Therefore, the values present in that column indicate the percentage of the original model's performance preserved after compression. We will revise the table headers and include a brief explanation in the caption to avoid any confusion to the readers.\n\n\n### **Performance at high compression ratio**\n**Comment (6):** Clarification on performance at high compression ratio.\n\n**Response (6):** We thank the reviewer for raising this important point. For reasonable compression ratios (e.g., 25% for 7B models), our results (Appendix Table 12) show that finetuning the compressed model can recover some additional performance. In fact, an interesting finding from these results is that either training the entire model or only the Linear Transformation (LT) will lead to almost the same performance, with the latter being significantly more computationally efficient than the former. The recovery of additional performance can be attributed to the fact that our initial optimization (using $\\ell_2$/cosine loss) does not directly align with the LLM training objective.\nFor even higher compression ratios, we agree with the reviewer's comment and we also recommend finetuning with LoRA or full-model training, as LTs alone may lack expressiveness. While our method provides a strong starting point, the optimal approach depends on the particular model and the available computing resources and budget. Below we attach a table with the results (confirming the superiority of our approach even for larger compression ratios). We will also include a corresponding figure that depicts these results in the final version of the paper.\n\nClarification on performance at high compression ratio\n\n| Model | Compression Ratio | Layers Removed | Perplexity (WikiText) ↓| Average Accuracy ↑|\n|-------|------------------|---------------|-----------------------|------------------|\n| UIDL  | 12.5%            | 24-28         | 21.51                 | 59.9             |\n| Ours  | 12.5%            | 24-28         | **14.31**                 | **64.6**             |\n| UIDL  | 25%              | 20-28         | 112.8                 | 50.4             |\n| Ours  | 25%              | 20-28         | **24.8**                  | **60.7**             |\n| UIDL  | 37.5%            | 18-30         | 58308                 | 41               |\n| Ours  | 37.5%            | 18-30         | **192.18**                | **51**               |\n| UIDL  | 50%              | 3-19          | 140114                | 30.4             |\n| Ours  | 50%              | 3-19          | **5932**                  | **38.6**             |\n    \nIn this Table (1) we report the average performace accuracy for (Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks), and the perplexity on wikitext. for Llama 3 8B instruct model.\n\n\n### **Related Work**\n**Comment (7):** Perhaps the authors should consider describing works like [1] in their related work\n\n**Response (7):** We thank the reviewer for bringing to our attention this work. The method in ([1]) is indeed relevant, as it shares similarities with SVD-LLM method,  and we will expand our related work section to include it.\n\n\n### **Clarification (1)**\n **Comment (8):** While I usually like a summary figure on page 1, I do expect it to be fully described. The figure 1 is placed even before the abstract and none of the abbreviations are introduced at this point; so the figure is misplaced. I would have preferred the method overview figure 2 in its place as it is quite self-contained\n\n**Response (8):** We thank the reviewer for his suggestion. We agree with his assessment and we will follow his advice to move Figure 2 in the position of Figure 1. \n\n\n### **Clarification (2)** \n**Comment (9):** reporting carbon emissions, make sure if it is CO2 or CO2 equivalent, as the latter is the more common measure\n\n**Response (9):** We thank the reviewer for his comment. We used the publicly available tool codecarbon from github which reports emissions in CO₂ equivalent (CO₂eq), not just pure CO₂. We will make this point clear in the revised version of the manuscript."}, {"review_id": "yh4uVJ86ta", "reviewer": "Reviewer_wAe8", "summary": "This paper proposes ReplaceMe, a simple and efficient method to compress transformer-based models by replacing selected layers with linear transformations. The approach is post-hoc and training-free, requiring only a small “calibration dataset” to optimize the replacement layers via regression. The method is evaluated across a variety of language and multimodal models, demonstrating competitive performance compared to existing pruning techniques, with lower computational and environmental costs.", "questions": "-\tCan the authors comment on using non-linear transformations instead of linear ones? Would this further improve performance?\n-\tHow is the “environmental impact” calculated?\n-\tIs the learned linear transformation task-specific? That is, does each task/dataset require a new calibration and transformation? How does this compare with other pruning methods in terms of transferability?\n-\tThe choice of calibration dataset seems to have a large impact on performance. Do the authors have any insights on the reason of this sensitivity?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n-\tThe paper is well-written, well-structured, and easy to follow.\n-\tSimple yet effective idea: Replacing layers with linear transformations is a straightforward yet practical strategy that reduces computational overhead.\n-\tThe experiments are broad, covering multiple tasks and models. The results on multi-modal models highlight the transferability of the method.\nWeaknesses\n-\tLimited compression ratio evaluation: The experiments primarily focus on a 25% compression ratio. While this is acknowledged in the limitations section, evaluating a wider range of ratios would provide a more complete picture of the method’s robustness.\n-\tThe primary comparison is with UIDL. Including more pruning baselines would make the comparison more convincing.\n-\tMissing reference: The core idea, replacing layers with linear transformations, has been explored in prior work [1, 2], although not specifically in the pruning context. These works are missing from the related work discussion.\n-\tThe calibration dataset, a key component of the method, is only clearly explained in Section 3.3.1. Introducing this concept earlier would improve clarity and understanding.\n[1] Hernandez, et al. \"Linearity of relation decoding in transformer language models.\" ICLR 2023\n[2] Wang, et al. \"Lost in multilinguality: Dissecting cross-lingual factual inconsistency in transformer language models.\" ACL 2025", "was_revised": true, "final_justification": "Author addressed my concern on testing on different compression rate and adding more baselines.", "rebuttal": "### **Limited Compression Ratio**\n**Comment (1)** Limited compression ratio evaluation: The experiments primarily focus on a 25% compression ratio. While this is acknowledged in the limitations section, evaluating a wider range of ratios would provide a more complete picture of the method’s robustness.\n\n**Response (1)** We appreciate the reviewer’s suggestions and we thank the reviewer for his constructive comments which will strengthen our paper.\nCompression Ratio Evaluation: While our main focus was on a 25% compression ratio (as models below ~90% original performance often cannot be successfully used in practice), we agree that evaluating a wider range of compression ratios can be insightful. We include here a comparison up to 50% and we will also include this in the final version of the paper.\n\n| Model | Compression Ratio | Layers Removed | Perplexity (WikiText) ↓| Average Accuracy ↑|\n|-------|------------------|---------------|-----------------------|------------------|\n| UIDL  | 12.5%            | 24-28         | 21.51                 | 59.9             |\n| Ours  | 12.5%            | 24-28         | **14.31**                 | **64.6**             |\n| UIDL  | 25%              | 20-28         | 112.8                 | 50.4             |\n| Ours  | 25%              | 20-28         | **24.8**                  | **60.7**             |\n| UIDL  | 37.5%            | 18-30         | 58308                 | 41               |\n| Ours  | 37.5%            | 18-30         | **192.18**                | **51**               |\n| UIDL  | 50%              | 3-19          | 140114                | 30.4             |\n| Ours  | 50%              | 3-19          | **5932**                  | **38.6**             |\n    \n### **More Pruning Baselines**\n**Comment (2)** The primary comparison is with UIDL. Including more pruning baselines would make the comparison more convincing.\n\n**Response (2)** In our initial experiments targeting LLMs, we have compared against multiple state-of-the-art model compression strategies, including SVD-LLM, LaCo, LLM-Pruner, SliceGPT, and LLM-Streamline. Please refer to tables (1, 2) in the main paper. \n\nFor Vision Transformers the reviewer is correct as we indeed provide comparisons only against UIDL. The main reason for this is that almost all of the aforementioned methods have considered and provided comparisons exclusively on LLMs. In addition, our experiments on vision tasks are not by any means exhaustive, and their role is mainly to highlight that our strategy can also be successfuly deployed on vision models. Nevertheless, we agree with the reviewer that including more comparisons would be more convincing and will better highlight the benefits of our strategy. For this reason, in the updated version of our manuscript we will include additional comparisons on vision models with the LLM-streamline baseline. We include these new results below.\n\n| Model | Compression ratio | mscoco_captions (retrieval) | | cifar10 (zero-shot-classification) | | \"voc2007_multilabel<br>(zero-shot-classification)\" |\n|-------|-------------------|-----------------------------|---|------------------------------------|---|--------------------------------------------------|\n|       |                   | text recall@5 | vision recall@5 | acc1 | acc5 | mean_avg_p |\n| clip-vit-large-patch14 | | 0.794 | 0.611 | 0.9558 | 0.9963 | 0.7902 |\n| UIDL | 25% | 0.515 | 0.418 | 0.6933 | **0.9708** | 0.597 |\n| ReplaceMe lstsq | 25% | 0.556 | **0.471** | **0.7804** | 0.9706 | **0.688** |\n| LLMStreamline | 25% | **0.566** | 0.46 | 0.649 | 0.9515 | 0.6531 |\n\n### **Missing Reference** \n**Comment (3)** The core idea, replacing layers with linear transformations, has been explored in prior work [1, 2], although not specifically in the pruning context. These works are missing from the related work discussion.\n\n**Response (3)** We thank the reviewer for bringing to our attention the works of ([1, 2]), which we will refer to in the related work of the revised paper.\n### **Calibration Data**\n**Comment (4)** The calibration dataset, a key component of the method, is only clearly explained in Section 3.3.1. Introducing this concept earlier would improve clarity and understanding\n\n**Response (4)** We thank the reviewer for his very helpful comment. We agree that introducing the calibration dataset earlier would improve the clarity of the paper. For this reason, in the revised manuscript we will make explicit mention to the importance of the calibration data in Section 2 where we describe our method.\n\n### **NonLinear Transform**\n**Comment (5)** Can the authors comment on using non-linear transformations instead of linear ones? Would this further improve performance?\n\n**Response (5)** \n* While non-linear transformations could potentially improve performance, they a) require significant fine-tuning (\"healing\") on large datasets and b) unlike our method, the parameters used to represent the non-linear transform cannot be directly fused with the model's existing parameters. Our key contribution is a training-free approach that achieves strong results at reasonable compression ratios (as shown in our main results tables), where we outperform methods like LLM-Streamline (that uses non-linear transformations) without requiring any healing phase.\n\n* For applications where substantial training is feasible, non-linear transformations could in principle lead to further improvements, but such a strategy would represent a different approach from our training-free method. For this reason, we have decided to focus on a practical zero-training solution, which despite its simplicity and minimal requirements can maintain a very competitive performance.\n\n### **Enviromental Impact**\n**Comment (6)** How is the “environmental impact” calculated?\n\n**Response (6)** We calculated the environmental impact using the CodeCarbon library, which estimates energy consumption and carbon emissions during computation. We will add an explicit mention of this in the revised version of our manuscript.\n\n### **Task Specific LTs**\n**Comment (7)** Is the learned linear transformation task-specific? That is, does each task/dataset require a new calibration and transformation? How does this compare with other pruning methods in terms of transferability?\n\n**Response (7)** We thank the reviewer for raising a very interesting point. In the context of this paper we chose open source diverse data for calibration and we didn’t explore the task-specific recovery. However, the reviewer’s comment prompted us to conduct a preliminary investigation into whether, and to what extent, performance would improve if a task-specific calibration dataset were used. To explore this question, we conducted the following experiment:\n\n1- Use the train split of the Sciq benchmark as our calibration data.\n\n2- Evaluate the model on the test part\n\n3- Use a subset of Orca open-source dataset as calibration data\n\n4- Evaluate on the same benchmarks and compare the performance for the different calibration datasets.\n\nThe results are reported for Llama3 8B and we observe that indeed task-specific calibration data can further improve the performance of the compressed model. We plan to include these findings in the revised version of the paper. \n\n| Model  | Calibration Data | Compression | Sciq Accuracy |\n|--------|------------------|---------------|------------------|\n|    Llama3 8B instruct   |            -     |      -         |     0.93       |\n|    UIDL   |            Sciq- Task specific      |      25%         |     0.687       |\n|    Ours (LSTSQ)   |            Sciq- Task specific      |      25%         |     **0.89**       |\n|     Ours (LSTSQ)  |            Orca General      |        25%       |      0.858      |\n\n### **Calibration Data Sensitivity**\n**Comment (8)** The choice of calibration dataset seems to have a large impact on performance. Do the authors have any insights on the reason of this sensitivity?\n\n**Response (8)** In Table 3 of the main manuscript, we present an evaluation of model sensitivity to the calibration data. The results indicate a dependence of model performance on data type. For example, FineWeb, which consists of raw text, leads to a degradation in performance when used for instruction-tuned models, in contrast using instruction-finetuned data such as orca or mixed-domain corpora, including a multilingual composition of 66 languages, yields more stable and consistent performance across evaluated benchmarks. On the other hand, self generated data demonstrates superior performance in terms of language modeling efficacy, as measured by perplexity. However, this advantage can be accompanied by a decline in reasoning-oriented tasks likely related to the presence of noise, factual inaccuracies, or hallucinated content within the self-generated data. \n\nIn summary, our analysis reveals two principal observations regarding calibration data sensitivity:\n(1) instruction-tuned models achieve better performance when calibrated with instruction-aligned data;  (2) while self-generated data can enhance language modeling capabilities, their potential contamination with spurious or incorrect content may adversely affect performance on downstream reasoning tasks. \nThese insights are solely based on our experimental results. Further investigation of this topic is very interesting and we will consider it for our future work."}, {"review_id": "fCtBmelp7t", "reviewer": "Reviewer_ME16", "summary": "This paper introduces ReplaceMe - a depth pruning method that prunes transformer blocks from the base models. Pruned layers are determined by the distance between the activation outputs of different transformer blocks. In particular, they select a predefined number of layers to be pruned and seek for cut block index that minimises the distance between hidden states before and after the cut. To compensate for pruning effects, they utilise a small calibration dataset to learn a linear transformation that bridges the gap between activations before and after the cut. The authors propose to use L2 and cosine distance and find that cosine distance consistently produces better results. Besides, they propose regularisation into the calibration of linear transformation and also extend this method to non-consecutive replaced blocks. Experimental results validate the effectiveness and generality of ReplaceMe across various model families, including LLMs and Vision transformers.", "questions": "1. Could the authors provide more ablation studies with other similarity measures like CKA [1], CCA [2]?\n2. The linear transformation is just like a feature alignment method, does it have the same role as adapters? Could we replace the linear transformation matrix with PEFT methods like Adapter or LoRA? \n3. Which layers/blocks are often pruned? Could the authors provide more analysis on this? \n\n[1] Kornblith et al., 2019. Similarity of Neural Network Representations Revisited.\n\n[2] Morcos et. al., 2018. Insights on representational similarity in neural networks with canonical correlation.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is well written and easy to follow\n2. ReplaceMe shows better performance on various experimental settings from LLM families on NLP tasks to Vision tasks with transformer architectures with moderate compression ratios.\n3. The authors validate their method with very detailed ablation studies\n4. ReplaceMe significantly reduces the computational overhead in terms of latency and environmental impact.\n\n## Weaknesses\n\n1. This method lacks novelty since fundamental motivation and pruning technique (cosine distance) are proposed in previous work UIDL.\n2. The effectiveness is only on a low/moderate compression ratio 25%.", "was_revised": true, "final_justification": "After carefully reviewing all the authors' replies, all my concerns have been addressed. Therefore, I have increased my rating from 3 to 4 and updated my assessments.", "rebuttal": "### **Lack of Novelty**\n**Comment (1):** This method lacks novelty since fundamental motivation and pruning technique (cosine distance) are proposed in previous work UIDL.\n\n**Response (1):** We thank the reviewer for his comment but we respectfully disagree with his assessment. The problem of model compression is a very important one with numerous practical applications and has gathered a wide research attention since several years ago. There are many different strategies that have been proposed including pruning methods (structured, unstructured, depth-pruning), and thus we don't believe that additional motivation for investigating this problem is needed. We agree with the reviewer that the cosine distance that we use for identifying the blocks to be removed was previously expored in UIDL and ShortGPT, but we never claimed it to be part of our contribution. Our main contribution is the replacement of a sequence of layers with a linear transformation and the computation of the specific form of this transformation using calibration data. More importantly, our approach achieves model compression without a significant drop in performance while requiring a fraction of the computational cost of competing strategies, as it eliminates the need for the expensive healing process present in prior methods. Furthermore, we have conducted a detailed analysis of linear operators that feature specific properties and assessed their impact on the approximation accuracy of the original model. Finally we have performed several comparisons against competitive state-of-the-art methods both in NLP and Vision tasks and we have shown that our approach compares favorably in all cases. \n### **Higher Compression**\n**Comment (2):** The effectiveness is only on a low/moderate compression ratio 25%.\n\n**Response (2):** The reviewer is correct that our main focus was on 25% compression ratio. The main reason for this is that with higher compression rates the compressed LLM models' performance typically drops below ~90% of the original performance. In such cases the resulting compressed model doesn't have much of practical value. Moreover, we believe that 25% is a significant pruning ratio, especially when we consider the modern LLMs who employ billions of parameters. In these  cases the number of removed parameters is in the order of hundred of millions or even billions. Nevertheless, we agree with the reviewer that evaluating a wider range of compression ratios can still be insightful. In the revised version of the paper, we will include comparisons with UIDL without healing at compression ratios up to 50%. These results are also provided below, showing the superiority of our approach even for larger compression ratios.\n\n| Model | Compression Ratio | Layers Removed | Perplexity (WikiText) ↓| Average Accuracy ↑|\n|-------|------------------|---------------|-----------------------|------------------|\n| UIDL  | 12.5%| 24-28| 21.51| 59.9|\n| Ours  | 12.5%| 24-28| **14.31**| **64.6**|\n| UIDL  | 25%| 20-28| 112.8| 50.4|\n| Ours  | 25%| 20-28| **24.8**| **60.7**|\n| UIDL  | 37.5%| 18-30| 58308| 41|\n| Ours  | 37.5%| 18-30| **192.18**| **51**|\n| UIDL  | 50%| 3-19| 140114| 30.4|\n| Ours  | 50%| 3-19| **5932**| **38.6**|\n\nIn this Table we report the average performace accuracy for (Race, Winogrande, PIQA, BoolQ, OpenBookQA, and SciQ benchmarks), and the perplexity on wikitext. for Llama 3 8B instruct model.\n\n### **CKA, CCA Ablations**\n**Comment (3):** Could the authors provide more ablation studies with other similarity measures like CKA, CCA?**\n\n**Response (3):** We agree with the reviewer that the choice of distance metric is important, and we thank him for this valuable suggestion. In response, we have conducted additional experiments using CCA and CKA distances, as presented below. These are similar to those reported in Section A.10, where we record the model's average accuracy when we prune a fixed number of layers (eight in this case) but at diffferent locations of the original model, Llama 3 8B Instruct. The last three columns of the table contain the respective values for the cosine, CCA and CKA distances.\n    \n    \n| Pruned layers   |   Average accuracy |   Cosine distance |   1 - CCA |   1 - CKA |\n|:----------------|-------------------:|------------------:|----------:|----------:|\n|1-9 | 33.35 |0.41 |    0.4382 |    0.6627|\n|2-10 |  34.16 |  0.4  |    0.4336 |    0.4389|\n|3-11 |  31.79 |  0.38 |    0.43   |    0.2138|\n|4-12 |  46.3  |  0.38 |    0.423  |    0.1228|\n|5-13 |  50.35 |  0.37 |    0.419  |    0.1214|\n|6-14 |  52.74 |  0.37 |    0.4143 |    0.097 |\n|7-15 |  53.48 |  0.36 |    0.4105 |    0.089 |\n|8-16 |  53.67 |  0.36 |    0.4062 |    0.1152|\n|9-17 |  54.35 |  0.37 |    0.4016 |    0.1714|\n|10-18 |   52.6  |    0.38 |    0.3964 |    0.2431|\n|11-19 |   53.04 |    0.38 |    0.39   |    0.3551|\n|12-20 |   51.38 |    0.39 |    0.3833 |    0.3445|\n|13-21 |   51.59 |    0.38 |    0.3716 |    0.3341|\n|14-22 |   54.23 |    0.37 |    0.3558 |    0.3124|\n|15-23 |   54.23 |    0.35 |    0.3362 |    0.2791|\n|16-24 |   56.15 |    0.33 |    0.3173 |    0.1893|\n|17-25 |   60.39 |    0.31 |    0.2956 |    0.1221|\n|18-26 |   58.45 |    0.28 |    0.2795 |    0.0679|\n|19-27 |   58.37 |    0.25 |    0.2685 |    **0.0484**|\n|20-28 |   **62.04** |        **0.24** |    0.2643 |    0.0529|\n|21-29 |   60.64 |    0.25 |    0.2602 |    0.0643|\n|22-30 |   61.57 |    0.25 |    **0.2592** |    0.0775|\n|23-31 |   55.16 |    0.28 |    0.2697 |    0.1226|\n|24-32 |   57.65 |    0.42 |    0.2815 |    0.1014|\n\nAs we can see from the results, CCA and CKA show similar patterns to cosine distance. Their values are also high in the first layers while their minima are closer to the end of the model. However, cosine distance continues to yield superior performance: the highest accuracy is observed at the lowest cosine distance, while the accuracies corresponding to the lowest CKA and CCA distances are lower. These findings provide additional evidence that cosine distance is a reliable criterion for layer removal. We will include these results in the final version of the paper.\n\n ### **LTs VS LoRA**\n**Comment (4):** The linear transformation is just like a feature alignment method, does it have the same role as adapters? Could we replace the linear transformation matrix with PEFT methods like Adapter or LoRA?\n\n**Response (4):** We thank the reviewer for raising this important point about the relationship between our linear transformation (LT) and parameter‑efficient fine‑tuning (PEFT) adapters. Whether one labels LT a “PEFT adapter” it ultimately depends on how one defines an adapter. In LLM Streamline, for example, pruned layers are replaced by a small MLP, what one would traditionally call an adapter, or even an entire Transformer block that is first initialized via an MSE loss and then fine‑tuned on next‑token prediction. Likewise, UIDL incorporates LoRA modules during its healing phase.\n\nOur LT could, in theory, be viewed through the same lens: it is a feature‑alignment layer that sits where a removed block once was. However, it differs fundamentally in both initialization and deployment. We compute the LT in closed form or numerically on a small calibration dataset, without backpropagating through the entire model. Once LT is calibrated, it is merged into the previous layer’s down‑projection and introduces no extra parameters at inference. Although we do explore a subsequent healing step (see Section A.14).\nPerhaps most compellingly, even without any healing, our method outperforms the more complex adapters in LLM Streamline and the LoRA‑based modules in UIDL, as shown in Table 1 in the paper. We fully acknowledge that, under very aggressive compression ratios, a simple linear map may not possess sufficient expressive power to capture all dependencies. In such extreme settings, LoRA or deep non-linear adapters might achieve a better approximation. Yet those scenarios typically incur catastrophic quality drops and require prolonged healing, which runs counter to our goal of minimal overhead and predictable, one‑shot calibration.\n\nWe hope this clarifies the conceptual distinction and practical benefits of our LT approach compared to existing PEFT adapters. We would be happy to elaborate more on this if the reviewer requires more details.\n\n### **Analysis of the Pruned Layers**\n**Comment (5):** Which layers/blocks are often pruned? Could the authors provide more analysis on this?\n \n**Response (5):** In the table included in response (3) above we report accuracy and distance metrics for all combinations of eight removed layers. \n\nIn general, the most redundant layers lie in the middle and closer to the last blocks of the network, although the exact pattern varies by model. E.g., for Llama 3 8B, skipping layers 20–28 (1‑indexed) out of 32 yields the best results. We provide additional information in the following table:\n\n| Model    | Number of pruned layers | Range of pruned layers |\n| -------- | --------                | --------|\n| Llama 3 8B Instruct|2 out of 32| 24-26|\n| Llama 3 8B Instruct|4 out of 32| 24-28|\n| Llama 3 8B Instruct|8 out of 32| 20-28|\n| Falcon 2 11B|5 out of 60| 47-52|\n| Falcon 2 11B|10 out of 60| 36-46|\n| Falcon 2 11B|15 out of 60| 36-51|\n| Qwen2.5 7B Instruct|2 out of 28| 15-17|\n| Qwen2.5 7B Instruct|4 out of 28| 13-17|\n| Qwen2.5 7B Instruct|8 out of 28| 10-18|\n| Mistral 7B Instruct v0.3|2 out of 32| 22-24|\n| Mistral 7B Instruct v0.3|4 out of 32| 22-26|\n| Mistral 7B Instruct v0.3|8 out of 32| 20-28|\n\n\nIt is also important to note that the optimal pruning range depends on both the architecture and the number of layers removed. Previous work has also observed that layers form natural groups [1], which we confirm using confusion matrices based on cosine similarity and CKA. While we cannot include these figures in the rebuttal due to the new NeurIPS policy, we will include them in the revised manuscript.\n\n[1] Sun et al. \"Transformer Layers as Painters\" (2024)"}, {"review_id": "ouEpcftFIZ", "reviewer": "Reviewer_PYR3", "summary": "This paper proposes a pruning method that is both data- and computation-efficient. The proposed ReplaceMe substitutes selected Transformer blocks with linear layers, guided by an importance metric. This strategy aims to reduce model complexity while minimizing performance degradation, serving as an alternative to directly removing blocks. The authors present experimental results indicating that ReplaceMe can match or even outperform structured pruning methods that require retraining.", "questions": "I am curious about the size of the added linear transformations used for error compensation. It would be useful to clarify whether the number of additional parameters introduced by these layers has a significant impact on overall performance or model size. A discussion or ablation on this aspect could enhance the understanding of the method’s trade-offs.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength #\n\n* The paper addresses a practical and important topic in model compression. Structural pruning is generally more hardware-friendly but often more challenging in terms of preserving model performance. Thus, research that advances this area is both timely and valuable. Furthermore, the emphasis on pruning efficiency aligns well with real-world deployment constraints.\n\n* The paper is clearly written and well-organized, making it easy to follow the proposed method and its motivation.\n\n* The proposed method is both effective and efficient. The key idea of compensating for pruning-induced errors by inserting linear layers is compelling. These linear layers are data-efficient to train due to their convex nature and can be integrated into existing MLP layers without increasing inference cost, which enhances the practicality of the approach.\n\n* The experiments span both NLP and CV tasks, demonstrating that the proposed method performs on par with or better than existing structured pruning techniques, including those that require additional tuning or retraining.\n\n# Weakness #\n\nI did not find a very strong weakness. Here are some minor suggestions.\n\n* The first two subfigures in the introductory figure convey similar concepts and could potentially be merged to reduce redundancy and improve visual conciseness.\n\n* It would be helpful to include an ablation study analyzing the distribution and quantity of skipped blocks. Specifically, showing how many layers are skipped at once and how this distribution affects performance would clarify how the redundancy lies.", "was_revised": true, "final_justification": "I keep my rating since I did not pose too many questions towards this work, and the authors solved my concerns about specific experimental settings and provided additional experiments.", "rebuttal": "### **Figure Modification**\n**Comment (1):** The first two subfigures in the introductory figure convey similar concepts and could potentially be merged to reduce redundancy and improve visual conciseness.\n    \n**Response (1):** We thank the reviewer for his helpful comment. In the revised version of the paper we will combine and merge the two sub-figures.\n    \n### **Analysis Distribution and Quantity of Skipped Blocks**\n**Comment (2):** It would be helpful to include an ablation study analyzing the distribution and quantity of skipped blocks. Specifically, showing how many layers are skipped at once and how this distribution affects performance would clarify how the redundancy lies.\n\n**Response (2):** Below, we report experimental results for a fixed setting of eight pruned layers to illustrate performance variation similar to what we did in section A.10 of the appendix. The following table shows how average accuracy and cosine distance vary when fixing the number of pruned layers at 8.\n    \n\n| Pruned layers   | Average acc.|   Cosine distance |\n|:----------------|-------:|--------------:|\n| 1-9             |  33.35 |          0.41 |\n| 2-10            |  34.16 |          0.4  |\n| 3-11            |  31.79 |          0.38 |\n| 4-12            |  46.3  |          0.38 |\n| 5-13            |  50.35 |          0.37 |\n| 6-14            |  52.74 |          0.37 |\n| 7-15            |  53.48 |          0.36 |\n| 8-16            |  53.67 |          0.36 |\n| 9-17            |  54.35 |          0.37 |\n| 10-18           |  52.6  |          0.38 |\n| 11-19           |  53.04 |          0.38 |\n| 12-20           |  51.38 |          0.39 |\n| 13-21           |  51.59 |          0.38 |\n| 14-22           |  54.23 |          0.37 |\n| 15-23           |  54.23 |          0.35 |\n| 16-24           |  56.15 |          0.33 |\n| 17-25           |  60.39 |          0.31 |\n| 18-26           |  58.45 |          0.28 |\n| 19-27           |  58.37 |          0.25 |\n| 20-28           |  62.04 |          0.24 |\n| 21-29           |  60.64 |          0.25 |\n| 22-30           |  61.57 |          0.25 |\n| 23-31           |  55.16 |          0.28 |\n| 24-32           |  57.65 |          0.42 |\n\n\nAdditionally, the following table details the relationship between different number of pruned layers, their distributions in the original network and their corresponding average accuracy:\n\n| pruned_layers   |   Average accuracy |\n|:----------------|-------------------:|\n| 25-26           |              69.47 |\n| 24-26           |              66.47 |\n| 23-26           |              66.39 |\n| 24-28           |              64.62 |\n| 23-28           |              64.01 |\n| 22-28           |              63.34 |\n| 22-29           |              62.98 |\n| 20-28           |              60.67 |\n| 20-29           |              59.87 |\n| 19-29           |              58.97 |\n| 19-30           |              53.28 |\n| 18-30           |              51    |\n| 18-31           |              51.09 |\n| 5-19            |              46.87 |\n| 4-19            |              42.22 |\n| 3-19            |              38.66 |\n| 3-20            |              38.01 |\n| 3-21            |              38.61 |\n| 3-22            |              34.07 |\n| 3-23            |              31.78 |\n| 3-24            |              31.14 |\n| 1-23            |              29.95 |\n| 1-24            |              30.42 |\n| 1-25            |              30.22 |\n\nWe hope these results bring more clarity and successfully address the reviewer's concerns. We will add this table in the Appendix.\n\n### **Additional Parameters** \n**Comment (3):** I am curious about the size of the added linear transformations used for error compensation. It would be useful to clarify whether the number of additional parameters introduced by these layers has a significant impact on overall performance or model size. A discussion or ablation on this aspect could enhance the understanding of the method’s trade-offs.\n\n**Response (3):** We thank the reviewer for his comment. The outputs of the MLP in the $i$-th block and of the $(i+n+1)$-th block, both have shape $B \\times L \\times M$, where $B$ is the batch size, $L$ the sequence length, and $M$ the hidden dimension. The linear transformation that maps the MLP output of block $i$ to the output of block $i+n+1$ is therefore an $M \\times M$ weight matrix. As shown in Section 2.2, this transformation **can be fused into the MLP’s down‑projection layer**, since they are two consecutive linear operations. Based on this observation **our method introduces no additional parameters**. In the revised version of the paper we will try to make this aspect clearer, so that to avoid any confusion to the readers."}]

Author rebuttal:

Meta-review: This work introduces a compression method for transformer-based models that replaces selected layers with learned linear transformations. Their approach requires only a small calibration data set compared to some compression based schemes that require retraining the entire network.

Strengths:  the reviewers felt that experiments were comprehensive and that the approach yields significant performance improvements in practice.  Many reviewers also noted that, by and large, the paper was easy to read.

Weaknesses:  the reviewers identified a number of places where the exposition could be improved as well as suggested additional experimental studies that could be performed.  Most of these are quite minor, but should be addressed.

Overall:  the reviewers had a positive view on this work both before and after the discussion.  The results presented in the author feedback addressed most of the reviewers concerns.
Decision: Accept (poster)