DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
Authors: Tianteng Gu
Bei Liu
Bo Xiao
Ke Zeng
Jiacheng Liu
Yanmin Qian
Venue: NeurIPS 2025 poster ()
Keywords: Large language model
Model compression
Pruning
Primary area: deep_learning

Abstract: Pruning is a widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation—especially under semi-structured sparsity constraints. Existing pruning methods primarily focus on estimating the importance of individual weights, which limits their ability to preserve critical capabilities of the model. In this work, we propose a new perspective: rather than merely selecting which weights to prune, we first redistribute parameter importance to make the model inherently more amenable to pruning. By minimizing the information entropy of normalized importance scores, our approach concentrates importance onto a smaller subset of weights, thereby enhancing pruning robustness. We instantiate this idea through DenoiseRotator, which applies learnable orthogonal transformations to the model’s weight matrices. Our method is model-agnostic and can be seamlessly integrated with existing pruning techniques such as Magnitude, SparseGPT, and Wanda. Evaluated on LLaMA3, Qwen2.5, and Mistral models under 50% unstructured and 2:4 semi-structured sparsity, DenoiseRotator consistently improves perplexity and zero-shot accuracy. For instance, on LLaMA3-70B pruned with SparseGPT at 2:4 semi-structured sparsity, DenoiseRotator reduces the perplexity gap to the dense model by 58%, narrowing the degradation from 8.1 to 3.4 points.

DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
Tianteng Gu
Bei Liu" HKUST beiliudust.hr hk
Bo Xiao Meituan xiaobo090meituan.co com
Shanghai Jiao Tong University 995999277ssjtu. edu.
Ke Zeng Mcituan zengke028meituuan com
Jiacheng Liu HKUST j jiachengliuusst.tt
Yanmin Qian
Shanghai Jiao Tong University yanmingian@sj juu edu. cn
Abstract
Pruning is widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation- ceceudatioo-esecciaally under stri-structured sparsity constraints. Existing prun- ing methods primarily focus estimating the importance of individual weights, which limits their ability to preserve critical capabilities of the model. this perspective: rather than merely selecting which weights to prune, we first redistribute parameter importance to make the model inherently more amenable to pruning. By minimizing the information entropy of normal- ized importance scores, our approach concentrates importance onto a smaller subset of weights, thereby enhancing pruning robustness. We instantiate this idea through DenoiseRotator, which applies learnable orthogonal transformations to the model's weight matrices. Our method can be seamlessly integrated with existing pruning techniques such as Magnitude, SparseGPT, and Wanda. Evaluated on LLaMA3, Qwen2.5, and Mistral models under 50% unstructured and 2:4 semi- structured sparsity, DenoiseRotate consistently improves perplexity and zero-shot accuracy. For instance, on LLaMA3-700 pruned with SparseGPT 2:4 semi- structured sparsity. DenoiseRotator reduces the perplexity gap to the dense model by 58%, narrowing the degradation from 8.1 to 3.4 points. Codes are available hittps://ght http//gttttttt...///////e com/ Axel 1eelllgu/Dua ggt/DeooiseRotator
work, we propose
Introduction
Recent advancements in large language models (LLMs) 33 10 16] have significantly improved their performance in complex reasoning, multimodal processing, and extended context handling. However, their substantial model sizes and computational demands present practical challenges for deployment and inference efficiency. To mitigate these issues, a variety of model compression
techniques have been explored, including quantization 14 29
39 40 28 knowledge distillation
20. and pruning 31 18
38
Among these, pruning stands out effective method for reducing parameter count and com- putational cost by eliminating less important weights. Traditional pruning methods [18] rely on importance score metrics-sudl and prune those with the lowest scores. Notably, recent methods like SparseGPT [13 and Wanda
weight magnitude output sensitivity-to rank parameters
"qqual contribution Corresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

estimate the impact of removing individual parameters using Taylor approximations, justifying pruning parameters with minimal estimated impact and approximates the output deviation of the layer by summing the importance scores of the pruned parameters.
38
Despite their effectiveness, these methods operate within the fixed parameter space of pretrained models and focus solely on selecting which weights to prune. This paradigm does not modify the un- derlying distribution of parameter importance, which limits its flexibility and partsuulss--atiuullaaly under semi-structured (c.g.. 2:4) sparsity constraints where pruning choices are restricted.
To overcome this limitation, propose fundamentally different perspective. Rather than merely selecting weights to prune, we aim to reshape the distribution of importance score prior to pruning. Specifically. learnable orthogonal transformations to reparameterize weight matrices, leveraging the computational invariance 2 of Transformer architectures. As illustrated in Figure these transformations trained to rotate the weight matrices in a way that concentrates the importance scores into smaller subset of parameters, thereby enhancing pruning robustness.
instantiate this idea through DenoiseRotator, framework that applies
The term Denoise reflects the principle that minimizing the information entropy of normalized importance scores-viewed as discrete probability distribution- a theoretically grounded, differentiable, and cornunttaionnnvvaiiin objective for importance concentration. Moreover, the norm- peem-preerving property of orthogonal transformations ensures that the total importance of each layer remains invariant, allowing importance to be redistributed across parameters without being artificially introduced lost. This property contributes the overall stability of the algorithm.
Our key contributions are summarized as follows:
Entropy- Importance Concentration: We propose enhancing pruning robustness by con- centrating parameter importance into small subset through minimizing information entropy of discrete distribution consisting of normalized parameter importance.
DenoiseRotator Framework: We introduce Rotaioe concrete instantiation of our im- portance concentration idca, implemented via learnable orthogonal transformations that leverage the computational invariance of Transformer 4 architectures.
Plug-and- Compatibility:
DenoiseRotator decouples the importance concentration process from the actual pruning step. The learnable transformations are optimized independently prior to pruning, and can be plugged into any existing pruning pipeline.
Extensive Empirical Evaluation:
We demonstrate the effectiveness of DenoiseRotator on range of open-source LLMs, including Mistral (7B) 24] LLaMA3 (8B, 70B) [7.. and Qwen2.5 (7B, 14B, 32B, 72B) [42] Our method consistently and significantly improves pruning performance across unstructured and semi-structured sparsity patterns, reducing perplexity and improving accuracy compared to baseline pruning methods.
By combining entropy-guided importance concentration with orthogonal transformations, Denois- eRotator offers a novel and principled direction for robust model pruning.
Background
Neural network pruning a widely used model compression technique that reduces model size and computational cost by eliminating parameters deemed less important. Existing approaches can be broadly categorized into two paradigms. The first class relies on heuristic metries to estimate parameter importance and prunes those ranked lowest [1 8 13 38 44 9 26
19 45 III. The second class formulates pruning as a continuous optimization problem by relaxing the binary pruning mask into a differeeniable form, allowing end-to-end training at the cost of significantly increased
computational overhead 30 12
Computational invariance refers to the property of transformer architecture proposed in SliceGPT 2 that allows certain orthogonal transformations to applied to their weight matrices without altering the model's output. Building upon this principle, QuaRot (3] and SpinQuant (29) apply orthogonal rotations to weight matrices and activations to facilitate low-bit quantization by redis- tributing outlier values more evenly. RotPruner 7 attempts to leverage this property for pruning and

---

Aurg
Us
Rocate for concentration of
parsamter importance
SuPE
Merge
Soyfence
Sturts
XCT
Orthogonal
Mergeable combination
Pruned matrix
Figure 1: Overview the DenoiseRotator framework. The top row illustrates a Transformer layer architecture-usee used in mainstream models such as LLaMA (17), Mistral [24). and Qwen that consists of RMSNorm, attention, and feed-forward blocks. the middle, learnable orthogonal matrices are inserted to rotate the weight matrices, concentrating parameter importance before pruning. The rotated weights are then merged and pruned in the bottom row. In this illustration, linear layers are represented in the Y XW format.
reports promising empirical results; however, lacks theoretical analysis or formal justification for why rotation improves pruning robustness.
3 Method
In this section, we present the motivation and formulation of our proposed framework. We begin by formalizing the problem setting and introducing the concept of enhancing pruning robustness through concentrating parameter importance. Next, we describe how reducing the information entropy of normalized importance scores naturally leads to such concentration. Finally, we introduce a practical instantiation of this idea, termed DenoiseRotator, which implements entropy- importance concentration using learnable orthogonal transformations.
3.1 Enhance pruning robustness via parameter importance concentration
Pruning aims reduce model size and computational requirements by removing parameters deemed less important. critical aspect of effective pruning is accurately estimating the importance of each parameter. As mentioned earlier, recent methods such as SparseGPT [13] and Wanda (38] leverage Taylor series to approximate the change in a linear layer output when a parameter is removed, serving as importance metries identify and eliminate the least significant parameters.
For instance, the Wanda method computes the importance score 5 Waanda for each weight Wiy of
linear layer parameterized by W e Rdd X by considering both the weight's magnitude and the corresponding input activation norm:
S
Wanda
|Wel ||X
This metric is equivalent to applying the Optimal Brain Damage (OBD) (26] approach to the optimization problem of pruned weight W to minimize the change in the lincar layer's output after

---

Orggett lase set summ wiigens mortorre
Aacered map set yoh, werpes inpottacc
000
180
drcitalinnd lapi MT1A
Coar drtitit of bypickuel atou iqatave
(c)
(d)
Figure 2: Visualization of OBD importance in Eq 2 for the output projection in the first layer of LLLMA-3-8B before and after orthogonal rotation. (a) and (b) show the 3D heatmaps of importance scores of the weight matrix before and after applying DenoiseRotator, respectively. (c) and (d) display the corresponding importance distributions, highlighting parameters before pruning (in blue) and after pruning (in orange) by the Wanda method. After rotation, importance becomes more concentrated.
pruning. The OBD importance score is the second-order term of Taylor approximation:
min | - 2
|W,,1 (XXT)) )5s
W
Consequently, optimizing the change caused by pruning could be approximated by minimizing the sum of importance scores over the pruned parameters whose index in P, defined the set of indices corresponding to weights selected for removal, according to Talyor approximation property:
min ||W - WXI Ro min P
Σ
GORD 1
(L.))P
This insight motivates minimizing the total importance of the pruned parameters, which underlies the algorithmic design of existing methods - sorting and removing the weights with the lowest importance scores. However, reducing this sum is not limited to selecting the least important parameters- we can also reshape the distribution of importance itself, offering a new perspective for enhancing pruning robustness.
To this end, we propose applying transformations to minimize total importance score of pruned weight. Formally, the optimization objective can be expressed as:
min T.Tw.Tx
[wx(W)xx(X) (w
RO min P.T (KJ))P
Σ Ts(S)
where (Tw. Tx) is a pair of transformation applied to the layer's weight and input (e.g., orthogonal transformation), and Ts the corresponding transformation applied to importance scores S.j- Here, we omit the specific form of the OBD score to emphasize that our method is compatible with pruning metries based on heuristic importance estimation, where parameters with lower estimated importance removed.

---

By decreasing the total importance of pruned weights, the deviation in the linear layer's output is minimized, thus preserving model performance and enhancing robustness against degradation caused by pruning. Furthermore, reducing the magnitude of importance score of pruned weights improves the accuracy of "Taylor-series-aaae -hased importance approximations, leading more precise importance estimation and compensation strategies, such as those used in SparseGPT [13].
3.2 Reducing information entropy of normalized importance score
Although the objective in Equation provides a principled formulation for importance concentration, is generally diffieult to optimize practice. On the left-hand side, the selection of the pruning indices P a combinatorial problem and is known to be NP-hard [6. On the right-hand side, sorting- based strategies-commonti commonly used identify low-importance parameters- are non-differeentabll and thus incompatible with gradient-bass basedd optimization.
To tackle these challenges, we propose optimizing learnable transformation to minimize the information entropy the normalized importance distribution prior to pruning as a surrogate objective for concentrating parameter importance. Given that importance scores are non-negative, they can be normalized to form a valid probability distribution, facilitating a probabilistic interpretation of parameter importance. Entropy, being permutationninvaaiinn,, remains unaffected by the index ordering of parameters. Furthermore, as a concave function over the probability simplex, entropy, when minimized, naturally promotes sparsity by concentrating mass onto fewer elements. This strategy independently addresses importance redistribution, separate from the pruning process itself, allowing enhanced flexibility across various pruning metrics and sparsity patterns.
Formally, S } denote the importance scores, and Tg learnable transformation (e.g.. orthogonal rotation applied indirectly through weight and input transformation that reshapes the distribution of importance. We first compute transformed scores = Ts (S), and normalize them over specified group G S {(t,j)) (such row, column, or the entire matrix):
Ts(S) Pij E(.nee Ts(S) H(P)
Σ (b.))
Pij log Pij
where P {pty denotes the normalized importance scores computed within group 9, which could be seen as a discrete probability distribution, its entropy H(P) is then given by the above formula.
Our objective is to learn shared transformation Ts that minimizes the entropy across all normaliza- tion groups:
min Σ (P)) Ts
(6)
G
The summation is taken over all groups that share the same transformation Ts. By reducing the entropy of the transformed importance distribution, we effectively transfer importance toward smaller subset of parameters. This results in more skewed and robust importance profile, making the model inherently more amenable pruning.
3.3 DenoiseRotator
In this section, introduce DenoiseRotator, practical instantiation of the entropy-based im- portance concentration framework described earlier. DenoiseRotator implements the learnable transformation Ts using orthogonal matrices, enabling lightweight yet effective mechanism to reshape importance distributions without altering the model's functionality.
DenoiseRotator is designed to be modular and can be easily combined with existing pruning pipelines. The following pseudocode outlines an example workflow integrating RotoiseRotator (highlighted in blue) with layer-wise pruning methods:
5

---

Algorithm DenoiseRotator Integration Pipeline
Require: Dense model M. pruning method P. calibration data Deal
Ensure: Pruned model M'
Merge RMSNorm's 's weight into adjacent linear layers
Compute Hessian for pruning
X.X {X is from Deat)
Integrate and train orthogonal matrices R1, R2
Merge orthogonal matrices R1. R2
Apply pruning method P to convert M into M'
return Pruned model M'
3.3.1 Integration of orthogonal matrices
As shown in Figure | DenoiseRotator integrates two pairs of learnable orthogonal matrices into each Transformer layer.
Layer-Level Rotations (Rı): A pair of orthogonal matrices, Ri and R of shape (dhidden y dhidden). are applied at the beginning and end of each Transformer layer, respectively. Specifically, Ri is inserted before the RMS normalization and residual addition, and R applied after these operations. Ri effectively maps activations and weights into rotated spaces, allowing the network to redistribute parameter importance. Since orthogonal transformations preserve vector norms and inner products, the inputs to non-linear functions remain consistent. Thus, the model's output remains invariant.
Attention-Level Rotations (R2): Within the self-attention, another pair of orthogonal matrices R₂ and RJ are applied to the Value (V) and Output (O) projections. These rotations adjust the internal representations, further concentrating importance and enhancing the model's resilience pruning.
To elucidate the effect of these transformations, consider the Output (O) projection as an example. Let W denote its weight matrix of shape (dhidden; (aiudden and X its input of shape (daaddens len x bsz) before transformation. After applying the orthogonal transformations, the weight and input are transformed as follows:
W' To (W) W.W. WR, X' Tx(X)= R W'X R. W
The corresponding transformation of the OBD importance score in Equation 2 can be expressed
Tss)) = | (X'X D (R W R (R. XX B2),
This formulation reveals how importance scores are indirectly redistributed through orthogonal transformations applied to inputs and weights. Detailed transformation for other linear layer and pruning method could be found in Appendix
After training, the orthogonal matrices, except for those the beginning and end of the Transformer layer, are merged into the model's 's weight matrices as illustrated in Figure I Pruning is then applied the merged weight matrices.
This integration leverages structural properties of the Transformer architecture specifically, the pres- ence of residual connections, layer normalization (which could be transformed RMS normalization 2)), and attention mechanisms-and is suitable for any model that exhibits similar characteristics.
3.3.2 Invariance of total importance
Orthogonal transformations preserve the total importance of a linear layer's weights due to their property of maintaining vector norms, i.e., || = ||=|||. Formally, this implies (s) where Ts denotes the orthogonal transformation applied to the importance scores. This invariance ensures that importance redistributed among parameters without being artificially introduced or eliminated, thereby making the algorithm stable. For a full derivation and proof of the invariance property under orthogonal transformation, Appendix B
S,
To align the importance concentration mechanism with invariance, normalization in the entropy computation 5 is performed based on the position of the orthogonal matrix relative to the weight

---

matrix. Specifically, if the orthogonal matrix is applied on the right side (e.g. W R), normalization is conducted over each row. Conversely, if the orthogonal matrix is applied on the left side (e.g., RW), normalization is performed over each column. In cases where orthogonal matrices are applied on both sides (e-g., R2 W R2); both row-wise and column-wise entropies are calculated and their sum used. This means that the normalization group is determined by the position of the orthogonal matrix.
3.3.3 Optimization of the orthogonal matrix
During training, only the orthogonal matrices are optimized, while the weights of the original model and input features kept frozen, thus the hessian matrix H XXX for pruning could be reused. Each orthogonal matrix is initialized as the identity matrix to ensure that the model's '' behavior remains unchanged at the start of training.
Since all linear layers within a Transformer layer share the same set of orthogonal transformations R and R2, the overall training objective within a Transformer layer is defined as the sum of the entropy values across all normalization groups from all affected linear layers. The total loss is:
Loss ( R2,i)
ΣEH(PO) FEC. GEE
where C; denotes the set of all linear layers within the i-th Transformer layer, and indexes the normalization groups within each linear layer. Note that we do not apply any additional weighting to the entropy of each group. This is because the number of normalization groups per linear layer is inherently tied to the weight's 's shape (e.g., rows or columns), and the value of discrete entropy naturally scales with the number of categories.
Maintaining the orthogonality of matrices RS and R2 during backpropagation can be challenging. as direct gradient descent on these matrices does inherently preserve orthogonality. To address this issue, we employ a reparameterization strategy using QR decomposition. For each orthogonal matrix, we define an unconstrained matrix A and compute its QR decomposition: A QR, where Q is an orthogonal matrix and R is an upper triangular matrix. During training, only the orthogonal component Q is utilized in the forward pass, while gradients are applied to the underlying matrix A.
The theoretical justification for the feasibility of this method is provided in Appendix C
4 Experiment
Models and Tasks We evaluate our method on several recent open-source large language models, including Mistral 7B (24), LLaMA (8B and 70B) [17. and Qwen2.5 (7B, 14B, 32B, and 72B) [42)
Our evaluation encompasses both language generation, measured by perplexity on the WikiText-2 dataset (32), and five widely-used zero-shot tasks: PIQA 5). WinoGrande [36), HellaSwag 43
ARC-e, and ARC-c (8) Following established practices, utilize the LM Evaluation Harness [5 with default settings for all evaluations.
Baselines We integrate DenoiseRotator with three pruning methods: the classic Magnitude pruning and two advanced techniques, Wanda and SParseGPT. Both Wanda and SparseGPT require calibration data to estimate input statistics. For consistency, we use the same calibration dataset as in prior work, consisting of 128 sequences with a context length of 2048 tokens, sampled from the C4 training set [35) We apply uniform sparsity level across all decoder layers and evaluate two types of sparsity: unstructured 50% sparsity and semi-structured 2:4 sparsity.
Setup We trained the orthogonal matrices of DenoiseRotator using the Adam optimizer with a learning rate 0.01 over 2000 steps. All computations, except for QR decomposition, were performed in 1 precision to enhance efficiency. The computation resource cost scales linearly with the number of model parameters. For instance, training on LLaMA 70B with SparseGPTT took approximately 28 hours and utilized around 30 GB of GPU memory a single NVIDIA A100 GPU. Notably, the training duration independent of the calibration dataset size, as the optimization process reuse the hessian matrix for pruning. No recovery finetuning performed after pruning.
7

---

4.1
Main result
In Table we evaluate the generation capabilities of both dense and pruned models with and without the integration of DemoiseRotator-uue -uing perplexity on the WikiText-2 validation set. Under both unstructured (50%) and comi-structured (2:4) sparsity settings, DenoiseRotator consistently and significantly reduces perplexity across all model families and scales.
Table I: Perplexity 4 results WikiText-2 of various models under different sparsity settings.
Mistral
LLaMA3
Qwen2.5
Sparsity
Method
7B
8B
70B
7B
14B
32B
72B
0%
Dense
5.95
6.14
2.86
6.85
5.29
5.02
3.88
Magnitude
30.39
30.39
10.58
198.88
22.94
19.22
734.04
+DenoiseRotator
7.30
14.43
7.00
9.27
8.78
6.97
5.37
50%
Wanda
6.92
9.86
5.80
8.61
7.31
6.30
5.22
*DenoiseRotator
6.52
7.82
4.73
7.93
6.72
5,99
4.94
SparseGPT
6.94
5.99
8.46
7.27
6.35
4.94
+DenoiseRotator
6.38
7.60
4.61
7.60
6.51
5,86
4.78
Magnitude
141.96
141.96
18.17
559.87
58.93
24.27
287.70
+DenoiseRotator
9.52
75.23
11.53
11.97
13.51
8.61
8.81
Wanda
10.18
25.19
9.39
15.01
11.66
8.08
6.69
+DenoiseRotator
7.80
11.41
6.60
10.13
8.71
7.90
6.16
SparseGPT 9.71
17.67
10.97
11.35
10.20
7.92
7.19
+oenoiseRRtator
7.30
10.01
6.25
8.88
7.86
6.75
5.85
We report zero-shot task results for dense and pruned models in Table Across both unstructured (50%) and semi-structured (2:4) sparsity settings, DenoiseRotator consistently improves zero-shot accuracy, often recovering performance close to or surpassing the dense model, especially on larger models like Qwen2.5-72B and LLaMA3-70B.
Table 2: Average zero-shot accuracy (%) five benchmark tasks (PIQA, WinoGrande, HellaSwag, Swag, ARC-e, ARC-c).
Mistral
LLaMA3
Qwen2.5
Sparsity
Method
7B
8B
70B
7B
14B
32B
72B
0%
Dense
74.21
72.72
80.05
72.18
75.81
75.12
78.69
Magnitude
57.78
57.78
64.05
39.09
57.52
63.81
41.27
+DenoiseRotator
69.79
60.44
70.37
66.83
62.45
69.19
75.54
50%
Wanda
71.21
65.77
76.23
67.28
73.36
73.82
78.04
RotnoiseRotator
72.66
69.28
78.37
71.35
75.18
76.39
78.32
SparseGPT
71.76
66.88
76.66
68.95
74.00
74.06
78.25
#DenoiseRotator
73.46
69.58
78.54
72.18
75.71
76.02
78.42
Magnitude
48.04
48.04
59.04
40.51
53.89
59.79
41.41
+DenoiseRotator
60.57
41.10
61.94
62.41
58.89
66.45
69.42
2:4
Wanda
63.28
51.03
70.09
61.24
64.66
71.09
74.94
DDenoiseRotator
69.61
64.22
75.74
67.85
70.23
73.37
77.66
SparseGPT
66.18
55.95
69.16
64.67
66.76
71.10
75.43
+DenoiseRotator 70.98
66.11
76.37
69.56
72.38
73.25
77.16
Although we do not explicitly optimize for the 2:4 semi-structured sparsity constraint, DenoiseRotator still performs remarkably well-even making previously unusable models viable under this constraint. This is likely because the orthogonal matrices act as form of random permutation, increasing the chance that crucial weights align with the strisstructured sparsity pattern.

---

The impact of rotation is illustrated in Figure 2 which visualizes the OBD importance for the output projection in the first layer of LLaMA-3-S88 before and after applying DenoiscRotator, revealing how importance becomes more concentrated after transformation.
For detailed performance results (including the LLaMA 1 and 2 families), please refer Appendix G
4.2 Effectiveness of Entropy Reduction on Pruning Robustness
To validate the efficacy of entropy re- duction in enhancing pruning robust- conduct ablation study on LLaMA-3-8P using SparseGPT with 50% unstructured sparsity calibrated by 128 sequences of 2048 tokens sam- pled from C4 training set (generation task). We record model performance various entropy reduction steps of DenoiseRotator. Table 3
Table 3: Impact of entropy reduction on pruning robustness.
Results are reported on LLaMA-3-SB with SparseGPT (un- structured 50% sparsity) calibrated by 128 sequences of 2048 tokens sampled from C4 training set.
Step
Entropy
Time (8)
Zero-shot ( (%) f Wikitext2 PPL
457280
1181
66,88
9.567
100
396992
1603
70.54
7.701
400
387904
2760
70.12
7.619
2000
384128
9120
69.58
7.597
presents the average entropy value per layer, time consumed, average zero-shot accuracy, and perplexity on the WikiText2 validation set over different optimization steps. step refers the SparseGPT baseline.
As shown in Table 3 reducing the entropy of the normalized importance distribution significantly improves model's performance. At step 100, where average entropy per layer is reduced by 13%, we observe a substantial boost in zero-shot accuracy from 66.88% to 70.54%, along with a decrease in perplexity from 9.567 to 7.701. Further optimization continues reduce entropy and improve perplexity. A comprehensive hyperparameter analysis, detailing optimization steps and learning rates, available in Appendix D
These results empirically support our hypothesis: concentrating importance via entropy reduction enables more robust pruning, leading to reduced performance degradation.
4.3 Inference speedup and overhead analysis
DenoiseRotator introduces pair of orthogonal matrices at the beginning and end of each layer. Since the orthogonal matrices of adjacent layers can be merged, the actual overhead is limited to storing an additional shape matrix (hidden_sine siseeessseeeeeeee coidee__izze per layer, along with matrix multiplication during inference. For example, in L.LMMA3-8B,, this results in approximately 0.5 billion additional parameters (i.e., 4096 x 4096 32). This overhead is relatively small, accounting for only about 6.7% of the original model size. To further address overhead concerns, we explore the potential of block diagonal orthogonal matrices as a reduction strategy in Appendix B
Table 4: Average inference time and speedup per layer on LLaMA3-SB for 32 sequences of length 2048 on A100 GPU
Configuration
Dense Layer
2:4 Sparse Layer
OOthogonal Matrix
Time (ms)
5.80 1.00x (haseline)
4.37
4.69
Speedup (%)
1.33x
1.24x
We evaluate the average inference time of single Transformer layer LLaMA3-8B on 32 sequences of length 2048 using an NVIDIA A100 GPU. The batch size is set to 1, and we utilize PyTorch's 's semi- structured sparsity framework, specifically the torch. spwrse. ses_sparr__sseeee semee_ss_______ wwt.ssstruttttussssseeeeeesssste stt.ucuurre function. Timing measurements are conducted using torch. utils. benchmark. Timer. As shown Table dense layer takes 5.80 ms on average, while a 2:4 semi-structured sparse layer reduces this to 4.37 ms achievinn 1 1.33x speedup. The additional cost introduced by DenoiseRotator's orthogonal matrix is only 0.32 ms per layer, which minor compared to the overall computation time and maintains practical efficiency for deployment.

---

Conclusion
This paper introduces DenoiseRotator, a alay-a framework that improves the robustmess of model pruning through importance concentration. By applying learnable orthogonal transformations optimized via entropy reduction, DenoiseRotator reshapes the importance landscape ensure critical informations and abilities are preserved during pruning. Our theoretical analysis and empirical studies validate effectiveness of this strategy: concentrating importance distributions results in more informative retained parameters and enhances the model's robustness to pruning. Extensive evaluations on multiple LLMs show consistent gains across perplexity and zero-shot accuracy under both unstructured and semi-structured sparsity. Moreover, DenoiseRotator achieves these improvements with minimal overhead and without modifying the original pruning pipeline. These results demonstrate the promise of entropy-guided importance reshaping robust and efficient model sparsification. While our experiments focus on post-training pruning, the underlying idea of concentrating parameter importance may also benefit other stages of the model lifecycle, such as being integrated into pretraining or continual training.
general principle for
Limitations
Semi-structured sparsity patterns offer hardware acceleration benefits but also impose structural constraints. Future work is needed to explore how such constraints can be explicitly integrated into the learnable transformation's 's optimization process.
10

---

7 Acknowledgements
This work was supported by the China STI 2030-Major Projects under Grant No. 20212D0201500, and it was also supported in part by the Beijing Nova Program.
References
[1] anthropic. Claude 3.7 sonnet and claude code, Feburary 2025.
[2] Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations, 2024.
[3] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L.. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[4] Gary Bécigneul and Sctavian-Eugen Ganea, Riemannian adaptive optimization methods. arXiv preprint 1810.0000)0 2018.
[5] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432-7439. AAAI Press, 2020.
[6] Thomas Blumensath and Mike Davies. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications, 14(5-6):6296655 2008.
[7] Haoxian Chen and Limin Wang. Rotpruner: Large language model pruning in rotated space, 2025.
[8] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv: /00.://033.0555v1.. 2018.
[9] Rocktim Jyoti Das, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models, 2024.
[10] DeepSeek-AI, Daya Guo, Dejian Yang. et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[11] Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In Forty-forsi International Conference on Machine Learning, 2024.
[12] Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. MaskLLM: Learnable semi-structured sparsity for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[13] Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in onc-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10323-10337 PMLR, 2023.
[14] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post- post-training compression for generative pretrained transformers. arXiv preprint arXiv: 20..::2221.113333 2022.
[15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for shot language model evaluation, 07 2024.

---

[16] Google. Gemini 2.5: Our most intelligent ai model, March 2025.
[17] Aaron Grattafioni, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama herd of models, 2024.
[18] Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016.
[19] Babak Hassibi, David G. Stork, and Gregory J. Wolff. Optimal brain surgeon: Extensions and performance comparison. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector, editors, Advances in Neural Information Processing Systems 6. [7th NIPS Conference, Denver, Colorado, USA, 1993], pages 263-2773777 270. Morgan Kaufmann, 1993.
[20] Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. CoRR, abs/1503.0255 1,033, 2015.
[21] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations, 2022.
[22] Xing Hu, Yuan Cheng, Dawei Yang. Zhixuan Chen, Zukang Xu, Jiangyong XUCHEN, Zhi- hang Yuan, Zhe jiang. and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirleenth International Conference on Learning Representations, 2025.
[23] loan Mackenzie James. The topology of Stiefel manifolds, volume 24. Cambridge University
Press, 1976.
[24] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut
Lavril, Thomas Wang. Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.
[25] Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt:
Riemannian optimization in
pytorch, 2020.
[26] Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, 1989.
Advances in Neural Information Processing Systems, volume 2.
[27] Jun Li, Fuxin Li, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. In International Conference on Learning Representations, 2020.
[28] Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In The 2024 Conference on Empirical Methods in Natural Language Processing, 2024.
[29] Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quamtization with learned rotations. In The Thirteenth International Conference on Learning Representations, 2025.
[30] Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through lo regularization. In International Conference on Learning Representations, 2018.
[31] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems, 2023.
[32] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017.
[33] OpenAl, Josh Achiam, Steven Adler, et al. Gpt-4 technical report, 2024.
[34]
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary De Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In Advances in Neural Information Processing Systems 32: Anmual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 8024-8035, 2019.
[35] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine I Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1-14000 2020.
12

---

[36] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula. and Yejin Choi. Winogrande: An adversarial winograd schema challenge scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirly-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732-8740. AAAI Press, 2020.
[37] Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 11296-11300, 2024.
[38] Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[39] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QulPSwS: Even better LLM quantization with hadamard incoherence and lattice codebooks. Forty-first International Conference Machine Learning. 2024.
[40] Albert Tseng, Qingyao Sun, David Hou, and Christopher De Sa. QTIP: Quantization with trellises and incoherence processing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages
2017.
[42] An Yang, Baosong Yang. Beichen Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412. 151 2024.
[43] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 4791-4800. Association for Computational Linguistics, 2019.
[44] Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Pruning meets low-rank parameter-efffcient fine-tuning, 2023.
[45] Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations, 2024.
13

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
The abstract and introduction accurately reflect the paper's scope and con- tributions, including the proposal of entropy-guided importance concentration, the design of DenoiseRotator, and its compatibility and effectiveness across pruning methods. These claims are supported by both theoretical analysis and experimental results.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper discusses the limitations in terms of the additional training cost and overhead introduced by learnable orthogonal transformations and the lack of explicit adaptation to semi-structured sparsity patterns like 2:4.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
14

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:The paper provides clearly stated mathematical formulations and protf-particularlly for the entropy-based importance concentration and invariance total importance-wii with all assumptions explicitly described and derivations validated through ablation studies and empirical evidence.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides detailed descriptions of model settings, datasets, pruning methods, training configurations, and evaluation metrics. It also outlines a reproducible workflow and presents full integration pseudocode for the DenoiseRotator framework, facilitating reproduction of the main results.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
(b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
15

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The model and data used in this paper are open source, and the code has been submitted to ensure faithful reproduction of the main experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.cer
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https for more details.
/ //nips. t/tips.cc/pub.l.... wwtpppblic/guidese.eeeeeespeeeee huwbliicguuddds/oeeeuuuiiiii wwweeesuummsssoooooo...
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
The paper provides detailed experimental settings including model types, sparsity configurations, pruning methods, and evaluation datasets. Other implementation details such as the number of calibration samples and training steps for DenoiseRotator are also described main text and appendix.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
16

---

The paper reports consistent improvements across a wide range of models and tasks, but it does not include enror bars formal statistical significance tests. Future revisions may include these to strengthen empirical claims.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative етror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper specifies the number of calibration tokens, training steps, and runtime for entropy optimization.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work adheres to the NeurlPS Code of Ethics. It does not involve human subjects or sensitive data.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
17

---

10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper proposes general-purpose model compression technique focused on enhancing the robustness of pruning algorithms for large language models. The method operates the algorithmic level without targeting specific applications domains, and thus does not raise direct societal impact concerns such as fairesss, privacy, or misuse. As result, the broader societal impact minimal and not explicitly addressed in the paper.
Guidelines:
The answer NA means that there no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any data or pretrained models that pose a high risk of misuse. It focuses on general pruning framework applicable existing models without introducing new datasets or deployable systems. Therefore, no specific safeguards are necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All third-party assets used in this paper-including pretrained language models (LLaMA-3, Qwen2.5, Mistral), datasets (WikiText-2, C4), and pruning baselines (SparseGPT, Wanda, Magnitude) properly cited and credited in the main paper. Their usage complies with the licenses and terms specified by the original authors.
Guidelines:
existing assets. The authors should cite the original paper that produced the code package or dataset.
The answer NA means that the paper does
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The paper introduces new assets, including implementation code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
19

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This work does not involve any research with human subjects crowdsourcing. and therefore no IRB approval is required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: Large language models were not used as core or original component in the methodology of this research; their use was limited to standard writing assistance and does not impact the scientific contributions of the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, ntn-standard components.
Please refer to our LLM policy hactpp/:/maarrpps.cc////feeeeeee for what should or should not be described.
20

---

A Details of Orthogonal Transformations and Importance Score Formulations
In this section, we present how orthogonal transformations affect the importance scores under three commonly used pruning methods: Magnitude pruning, Wanda, and SPArseGPT. We also show how modify the importance formulas to maintain the Taylor expansion property under transformation.
Throughout this discussion, denote the original weight matrix as W Romxda, the input matrix as X E I and approximate the Hessian as H = XX The output of a linear layer is given by = WX.
Depending on which sides the orthogonal transformations are applied, three typical cases arise:
Case 1: Right-side rotated weight and left side rotated input (e.g., Q, K, Up, Gate).
For layers such as Query (Q), Key (K), Up. and Gate, the weight rotated on the right side and the input on the left. Specifically, the transformed weights and inputs are given by W = W Ri and X" R X, respectively. The corresponding changes in importance scores under different pruning methods are summarized below:
Table 5: Ridht-side rotated weight and left-side rotated input (e.g., Q. K, Up. Gate)
Method
Original Importance Score Sij
Transformed Importance Score Ts (S()))
Magnitude
|Wal
(W R1) (W Ru 32 (R. HR)) (W R (R) H R1)55
Wanda
W X
SparseGPT
W
17
Case 2: Left-side rotated weight and unrotated input (e.g., Down)
For layers like Down, only a left-side rotation is applied the weights, while the inputs remain unrotated. In this case, W = Ri W and X' = X. The modified importance scores are given by:
Table 6: Left-side rotated weights and unrotated inputs (e.g., Down)
Method
Original Importance Score Sij Transformed Importance Score Tss (Sig)))
Magnitude
[Wol Wel | 2
(RTW) (R. W)? Hgj
Wanda
SparseGPT
W H
(R. W) H
Case 3: Two-side rotated weight and left-side rotated input (e.g., O, V)
For components like Output (O) and Value (V) projections in the self-attentin mechanism, rotations are applied on both sides.
For Output (O): W" =R WR and X" RTJ X. The transformed importance scores are:
Table 7: Transformation for Output (0)
Method
Original Importance Score Siz
Transformed Importance Score (S
Magnitude
|Wyl W ||X
(R. RRWWR WRRR
Wanda SparseGPT
(R. (RJRRR) (R HR))
W H 1j
(R. WR) (R) H -
For Value (V): W R, W Rs and X' R. X. The corresponding transformation is similar that of Output (O).
21

---

B Proof of invariance of total importance
As established in Equation 3 the total importance of parameters can be approximated by the Frobenius norm of the layer output, corresponding to the scenario where all weights are pruned (set to zero). Specifically, according to optimal brain damage [2). have:
Σ O ||
Σ [SX|I |
(10)
Since the magnitudes of Weyl are very small, this approximation is highly accurate.
Case 1: Right side rotated weights and left side rotated inputs (e.g., Q, K, Up, Gate).
When both the weight matrix and the input are transformed via orthogonal matrices:
(T)(S) (W))x((X) = |
= || R ES
(11)
Case 2: Left side rotated Weights and unrotated input (e.g., Down).
When only the weight is rotated:
ET((S) s (W)Txx(x( -I W tr(W T R WX) [[X]]] R ES
(12)
This equality holds because orthogonal matrices satisfy R R-1 and hence preserve both vector norms and inner products.
Case 3: Two side rotated weight and left side rotated input (e.g., O).
Components such O and V, which affected by both left and right orthogonal transformations, can be viewed as a combination of the two cases above. Thus, their total importance is also preserved.
C QR decomposition reparameterization
Parameter
Loss Function
Loss Function
Intermediate
Variable
GR Decomposition Reparamteriiaaioo
Orthogonal Matrix
A
R
Upper Triangular Matrix
Unconetrained Matrix
Figure Illustration of the QR decomposition reparameterization process. The diagram shows how an unconstrained matrix A is optimized indirectly to achieve constrained optimization of the orthogonal matrix Q, while interfacing with the loss function during forward and backward passes. The process leverages QR decomposition to preserve orthogonality and integrate seamlessly into gradient-based optimization methods.
Standard gradient descent algorithms and their variants encounter significant challenges when dealing with the orthogonality constraint of matrices. In conventional settings. these algorithms directly update matrix parameters without ensuring that the orthogonality condition is preserved post-update. This can lead to loss of orthogonality, compromising model integrity and stability. address this, previous research 29 22 7 has frequently adopted Stiefel manifold optimization techniques 23 27 25
These methods are specifically tailored handle the complex geometric constraints associated with orthogonal matrices, necessitating specialized optimizers and additional computational resources.
22

---

QR decomposition provides an elegant solution to this problem by exploiting the properties of linear algebra to indirectly parameterize orthogonal matrices. This approach sidesteps the orthogonality constraint by using the QR decomposition, thus enabling the use of conventional gradient descent methods for optimization tasks without requiring complex manifold optimizers. As a result, QR decomposition not only maintains the orthogonality of matrices but also simplifies the optimization process, effectively mitigating the difficulties posed by orthogonality constraints.
In Figure B we start with an unconstrained matrix A. During the forward pass of the neural network, QR decomposition performed, splitting A into two matrices: an orthogonal matrix Q and an upper triangular matrix R. The orthogonal matrix Q becomes the immediate subject of optimization in the loss function, even though we never directly modify Q itself. Instead, we focus on optimizing the unconstrained matrix A, which indirectly alters Q due to the nature of QR decomposition.
The process of QR decomposition is composed of basic matrix operations like dot products, vector normalization, and projection. These are mathematically smooth operations, meaning they can be differentiated, which key for gradient-basee optimization. Each step in QR decomposition involves operations that smoothly vary, making it possible to calculate how small changes in the input A affect the output and R.
Because of these properties, QR decomposition can seamlessly integrate into the computation graph used by automatic differentiation tools. PyTorch (34) provides automatic differentiation support for QR factorization through torch. qr. When QR decomposition is performed, Py ToTchh automatically manages and propagates gradients with respect the input matrix A. This capability enables gradient-based methods, such as gradient descent, to efficiently update A, automatically respecting the orthogonality constraint imposed on Q.
QR decomposition reparameterization therefore transcends the limitations of direct parameterization by introducing flexible and effective mechanism to handle orthogonality in optimization tasks. As the network trains, the orthogonal matrix Q adapts continuously to minimize the loss function, while the underlying matrix A undergoes refinement through regular gradient-bace based updates.
D Hyperparameter Sensitivity Analysis
thorough analysis of hyperparameter sensitivity essential for understanding how different hyperparameter choices impact the final performance of model. In the main paper, we relied on fixed set of hyperparameters, such using 2000 steps the rotation training process. To enhance the rigor and reproducibility of our research, we conducted comprehensive experiments on the LLaMA-3-88 model, exploring the effects of various hyperparameter configurations on the final results.
In our experiments, each presents the perplexity and average zero-shot accuracy corresponding to different learning rates and training steps. The setup involves utilizing SparseGPT with 50% unstructured pruning. using calibration dataset consisting of 128 samples of length 2048 from the C4 dataset. All experiments were performed on an A100 80G GPU.
Table 8: Results across various learning rates and training steps
Steps
0.1
0.01
0.001
0.0001
Perplexity
Zero-shot
Perplexity
Zero-shot
Perplexity
Zero-shot
Perplexity
Zero-shot
100
7.75
69.52
7.70
70.54
7.96
69.95
8.52
67.83
200
7.70
70.31
7.64
68,93
7.82
70.04
8.30
68.59
400
7.63
69.84
7.62
70.12
7.66
70.31
8.16
69.37
800
7.61
70.25
7.61
70.21
7.72
69.55
7.94
70.77
2000
7.64
69.82
7.60
69.58
7.67
70.36
7.85
70.
4000
7.60
70.46
7.60
69.59
7.63
70.37
7.78
69.59
These results indicate: When the learning rate is less than or equal 0.001, the efficiency is low due to the small learning rate. 2. With a learning rate of 0.01, training converges around 2000 steps, and excessive steps increase training costs. 3. Compared to the baseline without the importance concentration mechanism, all configurations show significant improvements training progresses. 4. Zero-shot task accuracy fluctuates around 70%, with a variation of approximately 1%, not showing a clear linear relationship with perplexity.
23

---

E Trade-off with Block Diagonal Orthogonal Matrices
Block diagonal orthogonal matrices provide an effective approach reduce both training and inference overhead of DenoiseRotator. By leveraging the structural efficiency of block diagonal matrices, this method achieves balanced optimization of performance and resource utilization.
In our experiments with the LLaMA-3-80 model, we evaluated these benefits using SparseGPT with 50% unstructured pruning over 2000 steps and learning of 0.01. Dense orthogonal matrices were replaced with block diagonal matrices, consisting of block_ ______nmm___ orthogonal submatrices arranged along the diagonal. Each submatrix has dimensions of hidden size This approach reduces both spatial and computational complexity to block_pum of the original requirements.
block_ num
Table 9: Performance and Computational Costs with Different Block Configurations
Block Number Perplexity
Zero-Shot Accuracy Time Cost per Step (s)
Entropy
7.597
69.58
0.124
384128
2
8.024
68.68
0.088
410816
4
8.544
68.47
0.076
428160
8
8.882
67.51
0.072
440512
These results highlight the trade-offs between performance and computational demands across different block configurations, illustrating the capacity of block diagonal orthogonal matrices to optimize these aspects effectively.
Looking ahead, potential enhancements could involve: 1. Varying the dimensions of each block within the block diagonal matrix (as opposed to maintaining uniform block sizes in this section's experiments) to further improve performance. 2. Incorporating permutation matrices, which have low computational complexity, to combine with block diagonal orthogonal matrices, thus emulating dense orthogonal matrices. This approach might involve arranging highly correlated dimensions or parameters adjacently for enhanced efficacy.
Compatibility with LoRA for Fine- TuninT Pruned Models
In this section, explore the of LoRA 21 to fine-tune pruned models. By freezing rotation matrices during fine-tuning, we maintain consistent gradient flow, aligning with scenarios where rotations are not applied, thus ensuring stability throughout the training process.
DenoiseRotator Configuration
Steps: 100
Learning Rate: 0.01
Model: LLAMA-3-8B
Pruming Method: Wanda, 2:4 semi-structured
Fine-tuning Configuration
Method: LoRA
Alpha: 32.0
Dropout: 0.1
LoRA-r: 8
Dataset: 4096 samples of length 2048 from the WikiText2 train set
Learning Rate: 2e-4
Weight Decay: le-2
Optimizer: Adam
Learning Rate Scheduler: Linear
24

---

Warm-up up Steps: 400
Note: Rotation matrices are kept frozen during fine-tuning
Table 10: Performance of Fine-Tuning with LoRA on Pruned Models
Method
Perplexity Zero-Shot Accuracy
Dense
6.14
72.72
Wanda
25.19
51.03
+ DenoiseRotator
14.31
60.67
+ Finetune
9.32
61.08
These results illustrate that additional fine-tuning with LoRA further reduces perplexity and im- proves accuracy. This suggests that fine-tuning post-denoising an effective strategy for boosting performance, while confirming that the rotation matrices introduced do not destabilize the training process.
G Performance of pruned model
Table 11: Pruning LLaMA 3.2 1B
Model
Zero-shot accuracy (%)
Perplexity
LLaMA-22 3.2 1B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
c4
Dense
36.77
60.52
63.65
74.43
60.3
59.14
9.748
14.034
Magnitude
22.35
28.87
30.51
55.6
48.38
37.14
410.099
$35.319
#DenoiscRotator
24.32
34.68
35.4
58.0
52.41
40.96
55.735
79.761
Wanda
26.02
47.77
44.1
64.64
54.85
47.48
23.881
35.242
50%
#DenoiseRotator
29.78
49.62
54.28
70.67
56.35
52.14
13.98
20.213
SparseGPT
29.61
47.52
50.63
68.06
57.06
50.58
19.961
28.117
DDtnoiseRotator
33.02
52.48
57.01
71.33
59.04
54.58
12.857
18.611
Magnitude
24.57
26.52
26.92
52.39
49.49
35.98
7417.287
6169.094
DennoiseRotator
22.18
31.23
29.75
55.77
51.07
37.99
458.362
524.964
Wanda
22.53
36.32
31.53
57.62
50.36
39.67
90.658
131.275
2:4
#DenoiseRotator
25.34
46.25
42.72
52.33
46.42
25.896
36,962
SparseGPT
25.0
40.57
38.06
61.32
53.91
43.77
36,233
48.101
-DenoiseRotator
28.67
48.74
48.62
57.46
50.35
18.088
25,091
Magnitude
23.89
28.24
27.77
$2.88
47.99
36.18
1257.331
1334.822
RotnoiseRotator
23.29
29.55
30.59
55.11
52.17
38.14
245.479
259.474
Wanda
22.87
39.77
35.59
62.35
49.41
42.0
48.225
69.767
4:8
+DenoiscRotator
29.01
50.34
48.68
68.39
55.09
50.3
18.199
26.491
SparseGPT
25.51
45.33
43.37
65.18
56.04
47.09
25,458
35.121
:DenoiseRotalor
31.14
50.63
52.49
56.27
52.17
15.019
21.369
25

---

Table 12: Pruning LLaMA -B
Model
Zero-shot accuracy (%)
Perplexity
LLaMA 3- 8B
ARC-c
ARC-e
Hellaswag
Piqu
Winogrande
Average
wikitext2
c4
0%
Dense
52.73
77.78
79.19
80.96
72.93
72.72
6.188
9.443
Magnitude
35.92
58.46
59.33
72.2
62.98
$7.78
30.393
36.197
otatoiseRoaator
38.65
60.02
65.51
73.12
64.88
60.44
14.425
22.761
Wanda
44.88
67.97
68.8
76.71
70.48
65.77
9.864
15.243
50%
otato
48,12
72.73
74.82
78.89
71.82
69.28
7.816
12.251
SparseGPT
44.45
69.44
72.34
76.88
71.27
66.88
9.567
14.223
*DenoiscRotator
46.67
74.45
75.67
78.94
72.14
69.58
7.597
11.819
Magnitude
30.03
42.68
46.25
66.38
54.85
48.04
141.962
183.124
otenoiseRotator
23.38
36.45
37.11
57.51
51.07
41.1
75.233
107.18
Wanda
30.38
50.63
47.78
67.9
58.48
51.03
25. 189
37.082
2:4
+DenoiseRotator
41.64
70.16
66.38
74.1
68.82
64.22
11,411
17.657
SparseGPT
55.72
65.19
55.95
25.027
+DenoiseRotator
41.89
70.12
69.67
76.88
71.98
66.11
10.008
15.011
Magnitude
32.42
49.92
53.67
70.08
59.91
53.2
47.863
59.894
+DenoiseRotator
27.73
44.78
$3.92
66.43
$9.67
50.51
23.971
34.597
Wanda
36.69
59.13
59.28
71.76
66.14
58.6
14.453
21.78
4:8
+DenoiseRotator
44.97
72.69
71.09
76.66
71.19
67.32
9.274
14.569
SparseGPT
37.12
61.53
63.87
74.32
67.56
60.88
12,836
18.443
+DenoiseRotator
44.62
68.9
73.14
77.64
73.48
67.56
8.557
13.151
Table 13: Pruning LLaMA-3-70B
Model
Zero-shot accuracy (%)
Perplexity
I.IaMA-3-70B
ARC-e
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitex12
0%
Dense
64.25
85.9
84.87
84.55
80.66
80.05
2.857
7.167
Magnitude
43.6
68.06
70.4
73.78
64.4
64.05
10.578
17.681
+DemoiseRRnator
48.38
73.02
76.9
78.73
74.82
70.37
6.998
11.951
Wanda
58.87
81.48
80.97
82.64
77.19
76.23
5.798
9.861
50%
-DtnoiseRotator
60,58
84.39
83.24
83,62
80.03
78.37
4.727
8,561
SparseGPT
$7.85
82.15
80.73
81.88
80.66
76.66
5.986
9,764
+DenoiseRotator
61.35
84.68
83,46
83.68
79.56
78.54
4.607
8.363
Magnitude
40.44
60.51
73.39
57.85
59.04
18.169
+DenoiscRotator
40.19
64.27
66.05
73.99
65.19
61.94
11.528
20.073
Wanda
48.89
75.46
74.01
80.36
71.74
70.09
9.388
2:4
+DenoiscRotator
56.57
81.9
80.12
81.56
78.53
75.74
6.597
10.907
SparseGPT
47.27
74.37
69.7
78.94
75.53
69.16
10.972
+DenoiseRotator
57.85
81.73
81.1
81.99
79.16
76.37
6.253
10.113
Magnitude
44.54
67.42
66.27
75.24
62.75
63.25
12.653
19.851
#DenoiseRotator
45.73
73.48
60.87
75.14
66.85
64.42
10.579
Wanda
52.9
79.71
78.55
81.94
74.43
73.51
7.158
11.746
+DenoiseRotator
59.13
82.04
82.26
78.37
76.94
5.714
9,709
SparseGPT
54.44
80.26
76.51
81.12
77.43
73.95
8.011
12.736
+DemoiseRotator
60.75
83.59
82.52
82.81
79.32
77.8
5.453
9.175
26

---

Table 14: Pruning Mistral-7B
Model
Zero-shot accuracy (%)
Perplexity
Mistral-7B
ARC-c
ARC-
Hellaswag
Piga
Winogrande
Average
wikitext2
c4
0%
Dense
56.14
76.89
83.68
80.63
73.72
74.21
5.947
9.741
Magnitude
35,92
58.46
59.33
72.2
62.98
$7.78
30.393
36.197
+DenniseRotator
51.02
70.2
77.65
78.24
71.82
69.79
7.303
11.611
Wanda
51.88
75.63
78.36
79.76
70.4
71.21
6.916
10.914
50%
+DenoiscRotttor
54.69
76.09
80.52
79.6
72.38
72.66
6.516
10.385
SparseGPT
51.96
74.33
79.53
80.47
72.53
71.76
6.938
10.783
+DenoiseRotator
54.52
76.09
81.94
81.18
73.56
73.46
6.376
10.228
Magnitude
30.03
42.68
46.25
66.38
54.85
48.04
141.962
183.122 124
DDtnoiseRotator
40.44
60.65
73.72
62.67
60.57
14.718
Wanda
43.6
66.16
65.79
75.73
65.11
63.28
10.176
15,572
+DenoiseRotator
73.61
75.65
79.38
70.32
69.61
7.796
12.005
SparseGPT
46.84
70.2
69.64
76.5
67.72
66. 18
9.714
13,974
+DenoiseRotator
51.02
74.58
80.09
71.82
70.98
7.301
11.229
Magnitude
32.42
19.92
$3.67
70.08
$9.91
53.2
47.863
59.893
+DenoiseRotator
46.25
65.53
73.34
78.18
67.56
66.17
8.061
12.678
Wanda
49.15
72.56
72.85
77.75
69.93
68.45
8.108
12.551
4:8
DDemoiseRotator
52.9
74.54
78.77
78.84
72.06
71.42
7.056
11.019
SparseGPT
50.17
71.63
74.27
77.97
70.17
68. 84.844
8.062
12,031
+DenoiscRotator
53.24
74.71
79.47
71.27
71.86
10.617
Table 15: Pruning Qwen-2.5-7B
Model
Zero-shot accuracy (%)
Perplexity
Qwcn-2.5-7B
ARC-
ARC-e
Hellaswag
Piga
Winogrande
Average
wikitex12
c4
0%
Dense
51.37
77.61
78.92
80.14
72.85
72.18
6.846
11.881
Magnitude
26.37
40.91
30.02
50.11
48.07
39.09
198.879
217.043
#DenoiscRotator
45.56
73.19
70.75
76.99
67.64
66.83
9.272
15.702
Wanda
44.62
71.97
71.61
77.8
70.4
67.28
8.614
14,248
50%
RDttoisecotator
52.47
79.38
74.7
78.84
71.35
71.35
7.932
13.277
SparseGPT
47.95
74.12
73.58
77.75
71.35
68.95
8.455
13.587
+DenoiseRotator
53.33
79.0
76.18
79.92
72.45
72.18
7.599
12.789
Magnitude
24.74
34.97
38.2
55.6
49.01
40.51
$59,866
948 148.652
+DetoiseRotator
42.15
63.99
74.76
62.04
62.41
11,966
19,712
Wanda
41.55
69.65
59.19
71.87
63.93
61.24
15,008
23,414
2:4
-DenoiseRotator
48.38
74.75
68.21
77.2
70.72
67.85
10.133
16.472
SparseGPT
42.92
72.05
64.3
74.86
69.22
64.67
11.349
17.179
+DenoiseRotator
50.0
77.95
71.41
77.8
70.64
69.56
8,878
14.334
Magnitude
24.49
37.88
32.81
$1.47
50.28
39.38
1188. 284
1095.696
#DenoiseRotator
44.88
70.4
67.54
75.84
65.98
64.93
10.379
17.446
Wanda
45.73
73.27
67.02
75.68
69.85
66.31
10.472
16,973
4.8
DDenoiseRotator
51.19
78.54
71.53
77.31
70.48
69.81
8.832
14.612
SparseGPT
48.89
76.39
68.81
76.82
70.17
68.22
9.612
15.088
*DenoiseRotator
50.94
76.68
73.66
78.29
70.64
70.04
8.168
13.435
27

---

Table 16: Pruning Qwen-2.5-14B
Model
Zero-shot accuracy (%)
Perplexity
Qwen-2.5-14B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
Dense
58.96
79.34
82.95
82.1
75.69
75.81
5.294
Magnitude
40.36
$9.81
54.07
70.4
62.98
$7.52
22.936
32.145
+DenoiseRotator
41.21
65.11
63.34
75.9
66.69
62.45
8.781
15.919
Wanda
$5.12
82.45
76.24
79.98
73.01
73.36
7.313
12.429
50%
+DenoiseRntator
$7.42
82.91
79.56
80.96
75.06
75.18
6.719
11.634
SparseGPT
55.29
77.71
80.09
74.66
74.0
+DenoiseRotator
59.22
83.84
79.92
80.9
74.66
75.71
6.505
11.325
Magnitude
36.43
56.69
49.49
69.15
57.7
53.89
58.927
68.898
+DenoiseRotator
38.48
67.13
55.14
71.44
62.27
58.89
13.513
22.581
Wanda
41.38
70.12
66.15
75.19
70.48
64.66
11.658
18.569
+DenoiseRotator
48.63
76.56
73.83
78.02
74.11
70.23
8.705
14.419
SparseGPT
43.43
72.18
69.08
76.82
72.3
66.76
10.196
15.678
+DenoiseRotator
52.9
79.34
75.84
78.78
75.06
72.38
7.858
12.875
Magnitude
37.97
60.19
50.75
71,82
60.22
56.19
29,907
42.944
RDttoiseRotator
43.43
70.71
60.85
74,86
66.06
63.18
9.871
17.492
Wanda
49.91
78.58
72.18
77.53
72.14
70.07
8.829
14.446
4:8
#DenoiseRotator
55.03
82.74
77.41
79.76
74.35
73.86
7.533
12.703
SparseGPT
48.89
78.07
73.18
78.89
73.09
70.42
8.398
13.539
*ennoiseRotator
55.38
81.61
78.17
80.47
74.51
74.03
7.127
11.997
Table 17: Pruning Qwen-2.5-32B
Model
Zero-shot accuracy (%)
Perplexity
Qwen-2.5-32B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
Dense
55.72
84.12
82.05
75.69
75.12
Magnitude
46.93
66.62
63.1
72.96
69.46
63.81
19.221
33.582
*DenoiseRotator
47.7
73.7
73.63
79.43
71.51
69.19
6.971
12.845
Wanda
54.44
77.36
80.63
81.45
75.22
73.82
6.297
11.359
50%
+DenoiseRotator
58.62
83.38
82,34
82.15
75.45
76.39
5,993
10.998
SparseGPT
53,5
78.62
81.25
81.34
75.61
74.06
6.348
11.179
+DenoiseRotator
57.76
81.69
82.56
81.94
76.16
76.02
5.858
10.76
Magnitude
43.6
62.42
57.82
69.75
65.35
59.79
24.272
45.718
+DenoiscRotator
46.08
71.17
68.25
78.07
68.67
66.45
8.613
14,929
Wanda
48.89
76.64
75.52
79.65
74.74
71.09
8.077
13.819
+DenoiseRotator
53.41
80.51
78.02
79.6
75.3
73.37
7.901
13.632
SpurseGPT
48.63
74.62
76.45
79.92
75.85
71.1
7.919
13.047
+DenoiseRotator
51.88
78.32
79.74
80.69
75.61
73.25
6.754
11.695
Magnitude
45.22
65.28
61.51
72.03
68.43
62.5
21.363
36.348
#DenoiseRotator
46.84
72.22
71.65
78.4
69.22
67.67
7.619
13,565
Wanda
53.92
79.42
78.25
80.2
75,69
73.5
7.073
12.314
4:8
DDenoiseRotator
57.0
81.06
80.37
81.56
77.03
75.4
6.619
11.696
SparseGPT
52.56
79.17
78.63
80.52
76.32
73.44
7.099
12.015
DDenoiseRotator
55.03
81.06
81.42
81.56
75.61
74.94
6.324
11.177
28

---

Table 18: Pruning Qwen-2.5-72B
Model
Zero-shot accuracy (%)
Perplexity
Qwen-2.5-72B
ARC-c
ARC-e
Hellaswag
Piqu
Winogrande
Average
wikitext2
c4
0%
Dense
62.46
83.42
86.01
83.73
77.82
78.69
3.875
9.26
Magnitude
30.46
46.17
28.73
47.33
53.67
41.27
734.042
557.275
otanosseRotator
57.08
81.02
80.83
82.37
76.4
75.54
5.369
10.695
Wanda
60.92
84.68
83.5
82.43
78.69
78.04
5.218
10.229
50%
+DenoiseRotator
61,86
83.46
84.42
83.19
78.69
78.32
4,937
9.956
SparseGPT
61.52
85.19
83.29
82.15
79.08
78.25
4.937
9.957
+DenoiseRotator
62.2
84.51
84.8
83.08
77.51
78.42
4.778
9.761
Magnitude
28.75
47.01
28.32
49.67
53.28
41.41
287.701
314,286
+DenoiscRotator
50.43
74.41
72.32
79.22
70.72
69.42
8.81
14.647
Wanda
56.66
81.02
79.01
81.61
76.4
74.94
6.692
11.883
+DenoiscRotator
60.75
84.81
81.96
82.1
78.69
77.66
6.164
11,258
SparseGPT
56,31
82.41
78.59
81.28
78.53
75.43
7.187
11.947
+DenoiseRotator
59.81
84.22
82.29
82.26
77.19
77.16
5,851
10.715
Magnitude
30.29
47.14
25.38
48.04
52.01
40.57
456,802
397,805
+DenoiscRotator
52.39
76.35
78.62
82.05
74.03
72.69
6.138
11.434
Wanda
60.92
85.4
81.45
82.48
77.43
77.53
5,939
10.938
+DenoiseRotator
83.5
83.45
83.08
78.14
77.56
5.511
10.538
SparseGPT
61.09
84.72
81.33
82.32
79.79
77.85
6.158
10.962
+DenoiseRotator
58.19
83.71
83.81
82.48
76.
76.87
5.295
10.164
Table 19: Pruning Llama-2-7B
Model
Zero-shot accuracy (%)
Perplexity
Llama-2-7B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
c4
0%
Dense
46.25
74.54
75.98
79.11
69.14
69.0
5.472
7.263
Wanda
42,83
68.69
70.65
76.88
66.85
65.18
6,9
9,231
+DenoiseRotator
42.83
71.83
77.2
68.11
65.98
6.519
8.817
50%
SparseGPT
41.3
67.38
71.11
77.04
69.3
65.22
7.003
9,254
+DenoiseRotator
42.92
72.26
72.09
76.99
68.59
66.56
6.273
8.409
Wanda
31.57
57.11
55.13
71.22
62.59
55.52
12.265
15.856
+DenoiscRotator
34.47
62.84
61.79
74.43
65.67
59.83
9.876
12,99
SparseGPT
33.36
59.05
57.99
71.76
65.43
57.51
11.291
14.301
+DenoiseRotator
37.37
64.98
64.64
74.37
66.61
61.59
8.696
11.156
Wanda
37.71
63.51
64.32
74.59
67.01
61.42
8.613
11.418
#DenoiseRotator
38.23
65.28
66.92
75.41
66.61
62.48
7.645
10.286
4:8
SparseGPT
38.57
63.17
65.19
75.41
67.48
61.96
8.689
10.946
+DenoiseRotator
37.29
65.61
68.46
76.55
68.59
63.3
7.157
9.543
29

---

Table 20: Pruning Llama-2-15
Model
Zero-shot accuracy (%)
Perplexity
Llama-2-13B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
0%
Dense
49.15
77.48
79.37
80.52
72.14
71.73
4.883
6.727
Wanda
46.5
70.66
76.07
79.38
71.43
68.8
5.965
8.297
+DenoiseRotator
47.53
73.48
75.52
78.56
71.82
69.38
5.779
7.957
50%
SparseGPT
45.73
71.25
75.16
79.33
71.11
68.51
6.058
8.261
+DenoiseRotator
47.44
75.72
75.94
79.0
70.72
69.76
5.515
7.582
Wanda
37.54
64.35
62.54
75.52
67.4
61.47
9.036
12.564
+DenoiscRotator
38.14
63.34
64.92
73.07
68.19
61.53
8.772
12.46
2:4
SparseGPT
38.48
64.44
75.68
69.46
62.37
9.058
+DenoiseRotator
41.64
70.66
70.77
76.17
72.06
66.26
6.811
9.322
Wanda
43.17
68.35
70.88
77.31
69.3
65.8
6.96
9.72
+DenoiseRotator
43.0
71.68
70.9
76.77
71.27
66.72
6.999
10.041
4:8
SparseGPT
43.09
69.07
70.48
76.71
70.56
65.98
7.222
9.705
+DenoiseRotator
43.6
71.46
73.95
78.02
72.53
67.91
8.335
Table 21: Pruning Alama-2-70B
Model
Zero-shot accuracy (%)
Perplexity
Llama-2-70B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitex12
c4
0%
Dense
57.34
80.98
83.84
82.7
77.98
76.56
3.319
5.709
Wanda
55.63
78.91
81.28
82.59
77.74
75.23
4.218
6.5
*ennoiseRotator
55.55
80.22
82.01
82.48
77.51
75.55
3.894
6.174
50%
SparseGPT
55.55
79.67
81.27
82.21
78.14
75.36
4.256
6.459
RDttoiseRotator
55.2
79.5
82.45
82.48
75.33
3.81
6.075
Wanda
50.85
77.44
75.96
79.76
75.37
71.87
5.468
8.118
+D +DenoiseRntator
$2.9
78.03
79.82
80.74
77.66
73.83
4.845
7.231
SparseGPT
50.68
76.77
75.9
79.16
75.22
71.54
5.727
8.16
+DenoiseRotator
53.16
78.32
79.48
81.28
77.27
73.9
4.643
6.936
Wanda
53.24
79.0
78.77
81.18
75.93
73.62
4.764
7.156
+DenoiseRotator
54.18
79.0
80.97
81.72
74.54
4.333
6.62
4:8
SparseGPT
53.84
78.7
78.68
80.79
76.56
73.71
4.926
7.201
+DenoiseRotator
54.01
81.41
81.61
77.03
74.71
4.247
6.467
Table 22: Pruning Llama-1-7B
Model
Zero-shot accuracy (%)
Perplexity
Llama---77
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitex12
c4
0%
Dense
44.62
72.94
76.22
79.16
69.93
68.57
5.677
7.343
Wanda
40.61
64.65
69.96
77.42
66.46
63.81
7.243
9.324
+DenoiseRotator
41.47
69.23
71.74
78.35
68.59
65.87
6.553
8.599
50%
SparseGPT
39.85
65.28
69.47
77.48
69.22
64.25
7.247
9.347
+DenoiseRotator
41.3
69.23
72.02
77.26
70.17
65.99
6.379
8.317
Wanda
31.31
55.01
56.63
70.73
62.12
55.15
11.613
14.668
#DenoiscRotator
34.3
60.98
62.78
75.35
68.67
60.41
9.214
11.899
2:4
SparseGPT
32.51
56.78
57.7
72.2
65.11
56.85
11.341
13.847
+DenoiscRotator
35.41
60.56
65.54
74.48
67.09
60.61
8,233
Wanda
34.39
63.55
74.21
64.33
59.23
8.595
11.268
*DenoiseRotator
37.12
63.68
67. 67.45
74,65
68.82
62.34
7.506
9,868
SparseGPT
35.32
64.26
74.37
59.86
10.914
+DenoiseRotator
39.25
65.28
68.78
76.66
68.6 68.67
63.72
7.136
9.254
30

---

Table 23:
Pruning Llama-1-13B
Model
Zero-shot accuracy (%)
Perplexity
Llama-1-13B
ARC-c
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitext2
0%
Dense
47.61
74.75
79.04
80.2
72.69
70.85
5.09
6.798
Wanda
43.86
70.33
74.75
78.67
71.43
67.8
6.134
8.134
+DenoiscRotator
45.73
72.43
76.18
78.89
71.59
68.96
50%
SparseGPT
40.61
66.2
74.25
78.35
72.06
66.29
6.229
8,145
+DenoiscRotator
46.16
71.46
76.62
79.16
73.32
69.34
Wanda
36.43
60.61
63.66
73.88
68.43
60.6
12.125
+DenoiseRotator
39.76
66.33
67.86
76.12
69.61
63.93
7.631
12.271
SparseGPT
36.09
65.6
75.19
69.22
61.38
9.084
11,385
+DenoiseRotator
42.32
69.44
71.16
76.71
71.19
66.16
6.902
8,993
Wanda
40.78
66.37
69.75
77.04
71.35
65.05
7.384
9,538
#DenoiseRotator
42.41
70.66
72.98
77.8
70.96
66.96
6.383
8.538
4:8
SparseGPT
41.21
67.68
69.75
76.61
71.43
65.33
7,513
9,458
+DemoiseRotator
43.6
70.83
73.99
78.67
73.16
68.05
6.221
8.146
Table 24: Pruning Llama-1-30B
Model
Zero-shot accuracy (%)
Perplexity
Llama-1-30B
ARC-c
ARC-c
Hellaswag
Piqu
Winogrande
Average
wikitex12
c4
Dense
52.99
78.91
82.65
82.21
75.77
74.5
6.129
Wanda
51.79
77.78
79.97
79.87
72.49
5.242
7.274
RDenoiseRotator
49.83
77.36
79.82
80.03
75.06
72.42
4.815
6.823
50%
SparseGPT
50.85
76.77
79.27
80.47
74.43
72.35
5.35
7.351
*DenoiseRotator
49.57
75.67
80.34
81.12
73.88
72.11
4.7
6.683
Wanda
45.31
71.17
73.32
78.56
71.35
67.94
6.929
9.505
+DenoiseRotator
44.54
72.73
75.17
78.24
75.53
69.24
6.17
8.442
2:4
SparseGPT
43.94
72.1
72.61
78.24
71.9
67.75
7.244
9.542
RntaoiseRntator
45.31
74.07
76.21
79.
73.72
69.68
5,822
7.888
Wanda
49.06
75.38
76.58
79.38
72.93
70.66
5.967
8.162
Rotaoor
49,23
73.74
77.83
79.38
75.06
71.04
5.467
7.482
4:8
SparseGPT
48.12
75.17
75.52
79.71
73.72
70.44
6.239
8.26
+DenoiseRotator
48.89
76.43
78.26
80.58
75.85
72.0
5.267
7.216
Table 25: Pruning Llama-1-657
Model
Zero-shot accuracy (%)
Perplexity
Llama-1-65B
ARC-e
ARC-e
Hellaswag
Piqa
Winogrande
Average
wikitex12
c4
0%
Dense
55.63
79.76
84.13
82.26
77.35
75.82
3.532
5.811
Wanda
53.58
77.78
81.98
81.94
76.8
74.41
4.616
6.694
+ DDttoiseRotator
53.84
78.03
81.4
81.39
76.01
74.13
4,223
6.347
50%
SparseGPT
54.18
76.56
81.89
81.72
77.19
74.3
4.601
6.661
+DenoiseRotator
54.78
79.76
82.43
81.83
77.11
75.18
4.094
6.229
Wanda
46.84
75.17
76.04
79.16
75.3
70.5
6.234
8.829
+DenoiseRotator
47.01
74.12
77.4
78.73
74.98
70.44
5.584
7.776
2:4
SparseGPT
48.81
75.63
76.11
79.49
76.95
71.39
6.263
8.427
*DenoiseRotator
51.96
76.64
79.48
79.76
77.82
73.13
5.061
7.222
Wanda
51.11
77.4
79.04
81.23
76.56
73.06
5.297
7.506
+DenoiseRotator
51.45
78.03
80.45
79.92
75.93
73.15
4.741
6.871
SparseGPT
51.19
76.22
79.04
80.47
77.66
72.91
5.308
7.404
+DenoiscRotator
52.99
77.78
80.58
81.23
77.03
73.92
4.558
6.689
31


Reviews:
[{"review_id": "yxsmejxk8r", "reviewer": "Reviewer_5XD8", "summary": "This paper introduces DenoiseRotator, a novel framework designed to enhance the robustness of LLM pruning. The core thesis is to shift the paradigm from simply selecting which weights to prune to first preparing the model to be more prunable. This is accomplished by using learnable orthogonal transformations to actively concentrate parameter importance into a smaller subset of weights. The optimization process is guided by minimizing the information entropy of normalized importance scores. DenoiseRotator is presented as a plug-and-play module that is compatible with existing pruning techniques like Wanda and SparseGPT. Extensive experiments on models like LLaMA3 and Qwen2.5 demonstrate significant improvements in perplexity and zero-shot accuracy under both unstructured and 2:4 semi-structured sparsity.", "questions": "1. The results in Table 3 hint at a potential trade-off between perplexity and zero-shot accuracy as training progresses. Could you provide more insight into this behavior? Does it suggest a risk of overfitting to the calibration set, and is there an optimal stopping point for the rotation training?\n2. Could the authors elucidate whether solving eq(6) is also NP-H and how does this corresponds to eq(4)? Does other variance concentration techniques works, such as PCA on the centralized covariance of S?\n3. How does the importance of weight indices corresponds to the transformed scores?\n\nI am willing to adjust my ratings after seeing the authors' responses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n1.Novelty of the Core Idea: The idea of importance concentration as a preprocessing step is a significant and insightful departure from traditional pruning pipelines, offering a new and promising direction for model compression research.\n\n2.High Practicality and Compatibility: By decoupling the importance reshaping from the pruning algorithm itself, DenoiseRotator is designed as a versatile, plug-and-play module. This significantly lowers the barrier to adoption for researchers and practitioners already using established pruning workflows.\n\n3.Strong and Comprehensive Empirical Results: The authors provide compelling evidence across multiple model families (LLaMA3, Qwen2.5, Mistral) and sizes. The consistent and significant performance gains in both perplexity and zero-shot tasks, especially under the hardware-relevant 2:4 sparsity, strongly validate the method's effectiveness.\n\nWeaknesses:\n1.Indirect Mechanism for Structured Sparsity: The paper's justification for its strong performance on 2:4 sparsity is presented as a hypothesis—that the rotation acts like a \"random permutation\" improving alignment. This suggests the entropy-minimization objective is not explicitly tailored for structured pruning, and the precise mechanism behind its success in this crucial scenario is not fully understood.\n\n2.Potential Overstatement of \"Model-Agnostic\" Claim: While tested on several decoder-only models, the paper's claim of being \"model-agnostic\" may not fully extend to other architectures like encoder-decoder or Mixture-of-Experts (MoE) models without further investigation.\n\n3.Limited Analysis of Hyperparameter Sensitivity: The paper reports results using a fixed set of hyperparameters for training the rotations (e.g., 2000 steps). A more thorough analysis of how sensitive the final performance is to these choices would strengthen the paper's conclusions and improve its reproducibility.", "was_revised": true, "final_justification": "I think all my concerns have been addressed and I would like to maintain my current recommendations.", "rebuttal": "Thank you for taking the time to review our paper and for providing such detailed and constructive feedback. We greatly appreciate your recognition of the novelty and practical implications of our work, DenoiseRotator, as well as the comprehensive empirical results we have presented. Your insights into the strengths of our approach, particularly the innovative concept of importance concentration and its compatibility with existing pruning techniques, are invaluable to us.\n\nPlease find below our point-by-point response regarding your feedback:\n\n### Weakness 1:\n\nThe notable performance in 2:4 semi-structured pruning is not solely due to the rotation acting as a \"random permutation matrix.\" Instead, the \"random permutation\" is an integral part of the rotation matrix-based importance concentration mechanism, both of which contribute to the enhancement of performance.\n\n2:4 semi-structured pruning represents a constrained version of unstructured pruning. In the context of importance score-based pruning, the primary difference between the two lies in the selection of comparison groups. Specifically, 2:4 semi-structured pruning eliminates the two least important parameters from every four consecutive parameters, whereas unstructured pruning chooses parameters to prune from a larger group (SparseGPT sets its pruning group to 256 columns, Wanda to 1 column).\n\nThe importance concentration mechanism is designed to intensify the focus of importance within the normalization group (1 column or 1 row). For 50% unstructured pruning, half of the low-importance parameters need to be removed from the entire comparison group (containing multiple normalization groups). In contrast, for 2:4 semi-structured pruning, the normalization group is divided into multiple blocks of four consecutive parameters, from which half of the low-importance parameters are removed. If each block retains two parameters from the 50% unstructured pruning and removes two, the 2:4 semi-structured pruning will mirror the effect of 50% unstructured pruning, making the importance concentration mechanism equally effective. However, in practice, there may be blocks with three or four important parameters, which could lead to inferior performance in 2:4 pruning compared to 50% unstructured pruning.\n\nTo ensure that parameters retained in 50% unstructured pruning are more evenly distributed across the blocks of 2:4 semi-structured pruning, a permutation matrix can be utilized to swap parameters among these blocks. According to probability theory, a randomly generated permutation matrix can effectively achieve this task. Since permutation matrices are orthogonal, multiplying them with another orthogonal matrix results in an orthogonal matrix. Additionally, permutation does not affect entropy, allowing for numerous equivalent rotation matrices, with gradient descent randomly optimizing to one of them.\n\nIn summary, the synergy between the importance concentration mechanism and random permutation collectively enhances the performance observed in 2:4 semi-structured pruning in this paper.\n\n### Weakness 2:\n\nThank you very much for pointing out the potential overstatement regarding the \"model-agnostic\" claim in this paper. We have indeed tested on several decoder-only models, but we recognize that this claim may not yet fully extend to other architectures such as encoder-decoder or Mixture-of-Experts (MoE) models. If this paper is accepted, we will promptly revise the text to ensure alignment with the experimental results and scope presented in the paper. Your feedback is greatly appreciated and will help us enhance the rigor and accuracy of our research.\n\n### Weakness 3:\n\nThank you very much for your suggestions regarding the analysis of hyperparameter sensitivity. We completely agree with your viewpoint that a thorough analysis of how hyperparameter choices affect final performance will help strengthen the conclusions of the paper and improve its reproducibility. The current paper indeed uses a fixed set of hyperparameters for experiments, such as 2000 steps in rotation training. To enhance the rigor of our research, we conducted a more comprehensive analysis on LLaMA-3-8B to assess the impact of different hyperparameter choices on the final results.\n\nIn our experiments, the left column represents perplexity on the WikiText2 test set, while the right column shows average zero-shot accuracy. The experimental setup involves SparseGPT with 50% unstructured pruning, using a calibration dataset consisting of 128 samples of length 2048 from C4. All experiments were performed on an A100 80G GPU.\n\nThe table below presents the results across various learning rates and training steps:\n\n| Steps\\Learning Rate | 0.1         | 0.01        | 0.001       | 0.0001     |\n|---------------------|-------------|-------------|-------------|------------|\n| 100                 | 7.75\\69.52  | 7.70\\70.54  | 7.96\\69.95  | 8.52\\67.83 |\n| 200                 | 7.70\\70.31  | 7.64\\68.93  | 7.82\\70.04  | 8.30\\68.59 |\n| 400                 | 7.63\\69.84  | 7.62\\70.12  | 7.66\\70.31  | 8.16\\69.37 |\n| 800                 | 7.61\\70.25  | 7.61\\70.21  | 7.72\\69.55  | 7.94\\70.77 |\n| 2000                | 7.64\\69.82  | 7.60\\69.58  | 7.67\\70.36  | 7.85\\70.11 |\n| 4000                | 7.60\\70.46  | 7.60\\69.59  | 7.63\\70.37  | 7.78\\69.59 |\n\nThese results indicate:\n\n1. When the learning rate is less than or equal to 0.001, the efficiency is low due to the small learning rate.\n2. When the learning rate is 0.01, the training converges around 2000 steps, and excessive steps increase training costs.\n3. Compared to the baseline without the importance concentration mechanism, all configurations show significant improvement as training progresses.\n4. The zero-shot task accuracy fluctuates around 70% with a variation of approximately 1%, not showing a clear linear relationship with perplexity.\n\n### Question 1:\n\nCertainly, there appears to be a trade-off between perplexity and zero-shot accuracy as training progresses. The training of the rotation matrix can be viewed as a form of continuous pretraining on the calibration dataset. The observed trade-off between different tasks in the later stages of training aligns with observations in these domains, and the risk of overfitting to the calibration set can be referenced from studies in these fields. Unfortunately, due to the lack of open-source pretraining datasets for large models and the absence of a clear theoretical analysis on neural network interpretability, we cannot provide a theoretical framework to determine the optimal stopping point.\n\nMoreover, detailed experimental results analyzing hyperparameter combinations in Response to Weakness 3 show that zero-shot task accuracy exhibits slight fluctuations and does not have a strict linear relationship with perplexity. Therefore, the most effective approach is to use grid search to determine the training steps that optimize model performance in practical application scenarios.\n\n### Question 2:\n\nThank you for your insightful questions. Addressing whether solving equation (6) is NP-hard requires considering different scenarios. In Scenario 1, if the transformation $T_s$ for importance scores is continuously differentiable, then all components in equation (6) are continuously differentiable, allowing optimization via gradient descent according to the chain rule. In Scenario 2, if $T_s$ is not continuously differentiable, gradient descent cannot be used, and whether it is NP-hard depends on the number of possible $T_s$. DenoiseRotator uses continuously differentiable orthogonal matrix transformations for  $T_s$, falling under Scenario 1, thus it is not NP-hard. \n\nEquation (6) serves as a continuously differentiable substitute for equation (4), and optimizing equation (6) can optimize equation (4) empirically. For details, please refer to lines 127-135.\n\nRegarding the effectiveness of other variance concentration techniques, such as PCA on the centralized covariance of S, this is an excellent question. PCA is a well-known variance concentration technique primarily used for dimensionality reduction of high-dimensional data, which is assumed to be sampled from a high-dimensional random variable. However, parameters are a set of fixed constants, and the formula for parameter importance involves the parameters themselves and the expected value of corresponding inputs, lacking the essence of random sampling. Therefore, PCA should target the hidden dimension of layer's inputs that can be viewed as samples from high-dimensional random variables. Columns of the weight matrix corresponding to the inputs' pruned hidden dimensions are removed. Thus, PCA inherently implies structured constraints, requiring complete removal of entire columns. In fact, SliceGPT[1] employs this approach using PCA for structured pruning. The importance concentration mechanism proposed in this paper, based on entropy minimization, does not have structured constraints, offering greater flexibility and a higher performance ceiling than PCA. Experimental results also demonstrate that our proposed method achieves superior performance. \n\n### Question 3:\nThank you for your inquiry regarding the relationship between the importance of weight indices and transformed scores. The importance of weight indices is assessed based on the contribution of parameters to the model's predictive capabilities. Transformed scores, however, represent the importance of weight indices after they have been modified through rotation or other transformation techniques to facilitate more robust pruning. In this paper, transformations are implemented using rotation matrices, and the transformed importance can be calculated from the original weights and the current rotation matrix using the formula provided in Appendix A.\n\n[1]\"SliceGPT: Compress Large Language Models by Deleting Rows and Columns\", ICLR 2024"}, {"review_id": "lV9TwQzSme", "reviewer": "Reviewer_BWCC", "summary": "This paper proposes **DenoiseRotator**, a novel framework to improve the robustness of pruning in large language models (LLMs) by concentrating parameter importance *prior* to pruning. Unlike previous methods that only select which weights to prune, DenoiseRotator applies **learnable orthogonal transformations** to redistribute importance scores, guided by entropy minimization. This makes the model more amenable to pruning. The method is compatible with existing pruning algorithms (e.g., SparseGPT, Wanda, Magnitude) and is extensively evaluated on multiple LLMs (LLaMA3, Qwen2.5, Mistral) across different sparsity types (50% unstructured and 2:4 semi-structured). Experiments show consistent improvements in perplexity and zero-shot accuracy with minimal inference overhead.", "questions": "1. **Can this method be applied to structured pruning?** If yes, please provide results. If no, clarify why this is not feasible. This affects practical applicability on hardware with structured sparsity support.\n2. **What is the optimal number of entropy reduction steps?** As shown in Table 3, over-training might hurt generalization (e.g., zero-shot accuracy drops after 2000 steps). Would an adaptive early stopping criterion help?", "limitations": "Yes, but it's not enough.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. **Clear writing and explanation**: The paper is clearly written, and the methodology—including Figure 1 and the entropy-based formulation—is intuitive and easy to follow.\n2. **Strong empirical results**: The method is extensively validated across different model scales, benchmarks, and pruning schemes. Visualization and ablations further support the effectiveness of the approach.\n3. **Original idea**: The use of orthogonal rotations—originally popularized in quantization—for pruning is novel. Applying entropy-guided transformations to improve robustness is both theoretically grounded and practically effective.\n\n---\n\n### Weaknesses\n1. **Limited practical efficiency in real deployment**: While effective under unstructured and semi-structured sparsity, the actual *inference speedup* under 2:4 patterns is limited. The method has not been shown to work under *structured pruning* (e.g., heads or channels), which is typically more impactful for latency reduction.\n2. **Additional training cost**: DenoiseRotator requires a dedicated optimization phase for learning the orthogonal transformations. Although the cost is manageable, it adds extra complexity, potentially limiting usability in low-resource or deployment-constrained settings.", "was_revised": true, "final_justification": "While the rebuttal addresses many of my concerns, some practical aspects—particularly around structured pruning and adaptive optimization—remain somewhat open in my view. I will therefore maintain my original score.", "rebuttal": "Thank you for your thorough review and insightful feedback on our paper. We are grateful for your recognition of the clarity and originality of our work, DenoiseRotator, and the strong empirical results we have presented. Your appreciation of our approach, particularly the novel application of orthogonal rotations and entropy-guided transformations to enhance pruning robustness, is highly encouraging.\n\nWe acknowledge the concerns regarding the limited practical efficiency in real deployment, particularly the inference speedup under 2:4 sparsity patterns and the applicability of our method to structured pruning. These are important areas that we are keen to address. Additionally, we understand the concern about the additional training cost associated with learning orthogonal transformations and its potential impact on usability in resource-constrained environments. Your questions about the feasibility of applying our method to structured pruning and the optimal number of entropy reduction steps are crucial for further enhancing the practical applicability of our approach. We will provide detailed responses and insights to clarify these aspects.\n\nPlease find below our point-by-point response regarding your feedback:\n\n### Weakness 1:\nThank you for your insightful comments regarding the practical efficiency of our method in real deployment scenarios. We recognize that achieving inference speedup through model compression often involves trade-offs, where some model performance may be sacrificed. While structured pruning can more readily provide acceleration benefits, it typically results in more severe performance degradation compared to unstructured and semi-structured pruning. On the other hand, thanks to the specialized circuit design in NVIDIA GPUs, semi-structured sparsity can also effectively reduce inference time. We believe it is fairer to compare acceleration effects under similar levels of performance loss.\n\nThe approach proposed in SliceGPT[1], which involves performing principal component analysis on model inputs to achieve structured pruning, serves as a valuable comparison point.\n\nExperiments comparing with SliceGPT:\nConfiguration: calibration dataset consisting of 1024 samples of length 2048 from WikiText2, evaluating perplexity on the WikiText2 test set. SliceGPT's results are referenced from the paper.\n\n| Method                        | LLaMA-2-7B | LLaMA-2-13B | LLaMA-2-70B |\n|-------------------------------|------------|-------------|-------------|\n| Dense                         | 5.47       | 4.88        | 3.32        |\n| SparseGPT 2:4                 | 8.69       | 7.07        | 4.98        |\n| SparseGPT 2:4 + DenoiseRotator| 7.35       | 6.16        | 4.36        |\n| SliceGPT 10%                  | 5.89       | 5.21        | 3.69        |\n| SliceGPT 20%                  | 6.64       | 5.81        | 4.25        |\n| SliceGPT 25%                  | 7.24       | 6.30        | 4.60        |\n| SliceGPT 30%                  | 8.12       | 6.99        | 5.05        |\n\nThe performance of the semi-structured sparse model pruned using the SparseGPT 2:4 + DenoiseRotator combination is comparable to that of the 25% structured pruning achieved by SliceGPT. However, the former requires less memory, only 60% of the original model, compared to 80% for the latter (SliceGPT adds two additional orthogonal matrices to each layer of the model). Furthermore, according to the Inference Time section in SliceGPT's main text, and Appendices A.6 and A.7, the latency of 2:4 semi-structured pruning is lower than that of 25% structured pruning.\n\nIn summary, the semi-structured sparse model pruned using the combination of SparseGPT and DenoiseRotator achieves comparable performance to the structured sparse model pruned using SliceGPT 25%, while offering faster inference speed.\n\n### Weakness 2:\nThank you for highlighting the concern regarding the additional training cost associated with DenoiseRotator. We would like to clarify that the training resources required by DenoiseRotator are minimal, substantially less than those needed for actual deployment. Therefore, large language models capable of operating in resource-constrained or deployment-limited environments can also utilize DenoiseRotator for pruning within the same settings.\n\nDue to the independence of rotation matrices across layers, the training of each layer's rotation matrix can be conducted separately. The actual necessary memory consumption is calculated as 16 (comprising 4 for parameters, 4 for gradients, and 8 for optimizer states) multiplied by the hidden size squared, equivalent to the memory required for deploying one single layer. In our straightforward, uncompiled implementation, optimizing rotation matrices for LLaMA-3-8B requires only 10GB of memory, while LLaMA-3-70B requires only 30GB. Furthermore, the ablation study on training steps in Section 4.2, lines 253-273, demonstrates that even a small number of optimization steps can significantly enhance the performance of the pruned model.\n\nAdditionally, model compression is a one-time process conducted before deployment, and the computational environment for compression does not need to match the deployment environment. Thus, the associated cost can be considered negligible.\n\n\n### Question 1:\nCertainly, DenoiseRotator can be applied to structured pruning. However, it has been demonstrated in SliceGPT[1] that performing PCA on layer's input can yield the optimal rotation matrix suitable for structured pruning. Using DenoiseRotator to optimize the rotation matrix will ultimately converge to this optimal matrix, thus repeating previous research findings.\n\nFor a comparison of the performance of semi-structured pruning using DenoiseRotator versus structured pruning, please refer to our response to Weakness 1.\n\n### Question 2:\n\nYou are absolutely right; employing an adaptive early stopping criterion can be an effective way to determine the appropriate number of optimization steps. However, due to the lack of interpretative analysis between model performance and intermediate hidden states, we are currently unable to provide a theoretical framework and must rely on experimental methods to determine this.\n\nTo address this, we conducted a more rigorous grid search for hyperparameters (step, learning rate) on LLaMA-3-8B. Our findings revealed that zero-shot task accuracy does not exhibit a strict linear relationship with perplexity; rather, it appears to fluctuate randomly.\n\nIn our experiments, the left column represents perplexity on the WikiText2 test set, while the right column shows average zero-shot accuracy. The experimental setup involves SparseGPT with 50% unstructured pruning, using a calibration dataset consisting of 128 samples of length 2048 from C4. All experiments were performed on an A100 80G GPU.\n\nThe table below presents the results across various learning rates and training steps:\n\n| Steps\\Learning Rate | 0.1         | 0.01        | 0.001       | 0.0001     |\n|---------------------|-------------|-------------|-------------|------------|\n| 100                 | 7.75\\69.52  | 7.70\\70.54  | 7.96\\69.95  | 8.52\\67.83 |\n| 200                 | 7.70\\70.31  | 7.64\\68.93  | 7.82\\70.04  | 8.30\\68.59 |\n| 400                 | 7.63\\69.84  | 7.62\\70.12  | 7.66\\70.31  | 8.16\\69.37 |\n| 800                 | 7.61\\70.25  | 7.61\\70.21  | 7.72\\69.55  | 7.94\\70.77 |\n| 2000                | 7.64\\69.82  | 7.60\\69.58  | 7.67\\70.36  | 7.85\\70.11 |\n| 4000                | 7.60\\70.46  | 7.60\\69.59  | 7.63\\70.37  | 7.78\\69.59 |\n\nThese results indicate that the zero-shot task accuracy fluctuates around 70% with a variation of approximately 1%, not showing a clear linear relationship with perplexity.\n\nSummary: The optimal number of hyperparameters and the calibration dataset should be selected based on the model's intended application empirically. For example, a model designed to assist with coding should choose the optimal entropy reduction steps and calibration dataset to maximize coding capabilities. A model intended for chat should aim to maximize user experience. The required optimization steps vary across specific tasks and evaluation criteria; some may be fewer than 2000, while others may exceed 2000. Conducting a grid search for optimization steps is the simplest and most effective approach.\n\n\n[1]\"SLICEGPT: COMPRESS LARGE LANGUAGE MODELS BY DELETING ROWS AND COLUMNS\", ICLR 2024"}, {"review_id": "MMPvkX2thI", "reviewer": "Reviewer_3pAp", "summary": "This paper introduces DenoiseRotator, a novel framework that enhances the robustness of neural network pruning for large language models (LLMs) by concentrating parameter importance prior to pruning. Unlike traditional pruning methods that focus solely on selecting which weights to remove, DenoiseRotator takes a fundamentally different approach by first redistributing parameter importance to make models inherently more amenable to pruning. The core innovation lies in applying learnable orthogonal transformations to weight matrices while minimizing the information entropy of normalized importance scores, effectively concentrating importance onto a smaller subset of parameters. The method leverages the computational invariance property of Transformer architectures and is designed to be model-agnostic, allowing seamless integration with existing pruning techniques such as Magnitude, SparseGPT, and Wanda.", "questions": "See weakness.", "limitations": "Yes. The paper addressed limitations in the \"Limitations\" section.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n1. The paper presents a genuinely novel perspective on pruning by focusing on importance redistribution rather than weight selection. The entropy-guided importance concentration approach is theoretically well-motivated and represents a significant departure from existing paradigms.\n\n2. The framework's design allows easy integration with existing pruning pipelines without requiring modifications to the underlying pruning algorithms, enhancing its practical utility.\n\n3. The paper cleverly employs QR decomposition reparameterization to train orthogonal matrices, which elegantly solves the challenge of maintaining orthogonality constraints during gradient-based optimization.\n\n**Weaknesses**\n1. Unlike quantization, pruning intuitively targets weights with low importance scores as the primary candidates for removal. Therefore, a transformation that induces a non-uniform distribution (where some weights are naturally pushed toward smaller magnitudes) would seem more appropriate for pruning. However, it is not immediately clear why enforcing a more uniform distribution via entropy minimization would yield better pruning performance. A more thorough explanation is needed to clarify how lower-entropy weight distributions, which typically concentrate probability mass on a subset of values, influence the effectiveness of pruning.\n\n2. While the paper presents QR decomposition as a solution for training orthogonal matrices, it does not clearly explain the specific training advantages this approach offers compared to alternative methods. For instance, SpinQuant[1] demonstrates that Cayley SGD can also effectively maintain orthogonality during backpropagation for rotation matrix optimization. The paper lacks a comparative analysis of why QR decomposition is preferable to other orthogonal training schemes such as Cayley transforms, making it unclear what unique benefits QR decomposition provides in the training context.\n\n3.  The core contribution of integrating QR decomposition to enforce orthogonality in rotation-based pruning appears as a straightforward extension of existing rotation methods rather than a fundamentally novel technique. Without deeper theoretical insights or comparative analysis demonstrating why QR decomposition offers unique advantages over other manifold optimization schemes, this addition risks being perceived as incremental.\n\n[1] \"SpinQuant: LLM quantization with learned rotations\", ICLR 2025", "was_revised": true, "final_justification": "My concerns regarding the entropy-based reasoning have been fully addressed, and my questions about QR decomposition versus Cayley SGD were clearly resolved. Based on these clarifications, I have updated my score to 4.", "rebuttal": "Thank you for your detailed and thoughtful review of our paper. We appreciate your recognition of the novel perspective our framework, DenoiseRotator, brings to the field of neural network pruning for large language models. By focusing on importance redistribution rather than traditional weight selection, we aim to provide a significant departure from existing paradigms, supported by a theoretically well-motivated entropy-guided approach.\n\nWe understand the concerns raised regarding the intuitive nature of pruning, which typically targets weights with low importance scores, and the need for a more thorough explanation of how entropy minimization contributes to improved pruning performance. We also acknowledge the need for a clearer comparative analysis of QR decomposition versus other orthogonal training methods, such as Cayley transforms, to elucidate the specific advantages offered by our approach.\n\nYour feedback is invaluable in helping us refine our explanations and strengthen the theoretical insights of our contributions. We are committed to addressing the weaknesses and questions you've highlighted to ensure the robustness and clarity of our work. Thank you again for your constructive comments and for guiding us toward enhancing the presentation and impact of our research.\n\n### Weakness 1:\nThank you for your thoughtful comments. The estimation of importance scores is crucial as they predict the change in output when parameters are removed, and they possess additive properties. Thus, the expected change in a layer's output due to pruning can be approximated by the sum of the importance scores of all pruned parameters. To minimize the change in output caused by pruning, it is essential to minimize the sum of the importance scores of the pruned parameters.\n\nIn lines 182 to 195 and Appendix B of our paper, we provide a detailed explanation showing that the sum of importance scores for all parameters remains consistent before and after rotation. A lower-entropy weight distribution, compared to a uniform distribution, results in higher importance for top-ranked parameters and lower importance for bottom-ranked parameters (which are likely to be pruned). This leads to a smaller sum of importance scores for the pruned parameters. Consequently, the change in model output is minimized, making the model's output closer to the pre-pruning state and thus more amenable to pruning.\n\nWe hope this explanation clarifies the rationale behind using entropy minimization to enhance pruning performance. Thank you for your feedback, which helps us improve the clarity and depth of our work.\n\n### Weakness 2 & 3:\n\nThank you for raising this important point regarding the comparative advantages of QR decomposition versus other orthogonal training schemes such as Cayley SGD. To address this, we conducted comparative experiments to evaluate the effectiveness of different optimizers:\n\nConfiguration: LLaMA-3-8B, unstructured 50%, 2000 steps, evaluated by WikiText2 test set perplexity. Experiments are conducted on an A100 GPU, where the average time per step for the QR-based optimizer was 0.124 seconds, while for Cayley SGD it was 0.118 seconds.\n\n| Optimizer - Learning Rate \\ Method |SparseGPT| Wanda.    |\n|------------------------------------|---------|-----------|\n| QR decomposition - 0.01                          | 7.597   | 7.816     |\n| Cayley SGD - 1                     | 7.727   | 9.695     |\n| Cayley SGD - 0.1                   | 8.097   | 9.827     |\n| Cayley SGD - 0.01                  | 8.832   | 9.834     |\n\nThe results indicate that QR decomposition with a learning rate of 0.01 consistently outperforms Cayley SGD across different learning rates, in terms of perplexity. \n\nThe superior performance of QR decomposition may be attributed to the fact that the actual optimization takes place in  $\\mathbb{R}^{n \\times n}$. In contrast, Cayley SGD operates within the Stiefel manifold, which is a subset of $\\mathbb{R}^{n \\times n}$. Therefore, QR decomposition allows for a broader exploration of the parameter space.\n\nWe hope this comparative analysis clarifies the unique benefits of QR decomposition in the training context. Thank you for your feedback, which has helped us strengthen the clarity and depth of our research findings."}, {"review_id": "dyobCDPgEe", "reviewer": "Reviewer_PoQx", "summary": "The paper introduces DenoiseRotator, a plug‑and‑play pre‑pruning procedure for large language models (LLMs). Instead of only selecting low‑importance weights, the main contribution of this paper is to introduce an importance score redistribution framework by inserting two pairs of learnable orthogonal matrices per Transformer block. These matrices are trained to minimise the information entropy of the normalised importance scores, so that the importance scores concentrate on fewer weights, making any subsequent pruning step more robust. Extensive experiments on Mistral‑7B, LLaMA‑3 (8 B & 70 B), and Qwen‑2.5 (up to 72 B) show large perplexity and zero‑shot‑accuracy gains at both 50 % unstructured and semi-structured sparsity compared with the underlying pruners.", "questions": "- Parameter Overhead Analysis. For 70 B models, the rotations add >7 B parameters (~10 %) before merging. Memory spikes during training might be a concern. Could low‑rank or block‑diagonal orthogonals work? An ablation on rank‑k Householder products would clarify the trade‑off. I noticed you discussed it in lines 232-234, but a more detailed elaboration regarding the training timing and memory costs would be appreciated.\n\n- Baseline breadth. Your work's high-level idea is quite similar to RotPruner. Would you please provide a detailed comparison of these two works? It would also be appreciated if the performance of RotPruner is included in the baseline.\n\n- Downstream fine-tuning compatibility. How would you recommend handling $R_1/R_1^\\top$ before LoRA/4-bit quant? A short empirical check (e.g., 1-epoch LoRA on 8 B) would reassure users.\n\n- Structured-mask awareness. In lines 245-248, you mentioned DenoiseRotator works remarkably well for 2:4 sparsity even without an explicit constraint. Have you tried explicitly learning rotations under a 2:4 mask, e.g., adding a differentiable surrogate for block occupancy? Results suggest good luck, but a principled variant might push performance further.\n\n- Clarify Fig. 2 (c) (d). Why those low-importance weights are unpruned? Did you mislabel?", "limitations": "Partially addressed.\nThe paper notes that semi-structured masks impose constraints not yet modelled, but omits:\n- Peak memory and time cost of rotation training on very large models.\n- Possible risks when further fine-tuning a model that still contains one orthogonal pair per block.\n\nAdding brief quantitative statements on these points would make the limitations section complete.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n### Quality\n- This paper is technically sound, with a clear mathematical formulation linking entropy to pruning robustness.\n- Solid implementation details (QR re‑param, invariance proof).\n- Large-scale evaluation across three types of models, various sizes, three baselines of importance scoring methods, and three sparsity patterns. \n\n### Clarity\n- Writing is generally good. Organisation and notation are easy to follow. Fig. 1 & Alg. 1 are helpful.\n- Appendix lists transformed score formulas, aiding reproducibility.\n\n### Significance\n- Clear improvement over baselines at both 50 %, 2 : 4 and 4:8 masks.\n- The method is agnostic to the pruner; practitioners can add it cheaply before any favourite pruning tool.\n\n### Originality\n- Introduces entropy-minimised importance shaping, which is new in the pruning literature.\n\n## Weaknesses\n### Quality\n- Recovery /downstreaming fine-tuning (FT) is not discussed. Since the pruned model still contains one orthogonal pair per block, discussing the potential risk/instability of light FT would be beneficial.\n\n### Clarity\n- Some images are confusing (e.g., Fig. 2 (c) & (d)). See Questions.\n\n### Significance\n- Some closely related baselines are not included. See Questions.\n\n### Originality\n- High-level idea (“learn orthogonal rotations, then prune”) overlaps with RotPruner [1]. Without a head-to-head, the novelty margin is hard to judge.\n\n[1] Haoxian Chen and Limin Wang. Rotpruner: Large language model pruning in rotated space, 2025.", "was_revised": true, "final_justification": "I think all my concerns have been addressed and I would like to maintain my current recommendations.", "rebuttal": "Thank you for your comprehensive review and constructive feedback on our paper. We are grateful for your recognition of the technical soundness and originality of our work, DenoiseRotator, which introduces an innovative importance score redistribution framework for enhancing the robustness of pruning in large language models. Your acknowledgment of the clarity in our writing and the significant improvements demonstrated through extensive experiments is highly encouraging.\n\nWe understand the concerns raised regarding recovery and downstream fine-tuning, as well as the need for clearer images and a broader baseline comparison, particularly with RotPruner. Additionally, your questions about parameter overhead, downstream fine-tuning compatibility, and structured-mask awareness are crucial for further refining our approach.\n\nYour feedback is invaluable in helping us address these points and improve the clarity and depth of our work. We are committed to providing detailed responses and insights to the questions and weaknesses you've highlighted. Thank you again for your thoughtful comments, which guide us in enhancing the presentation and impact of our research.\n\n### Question 1:\n\nThank you for your insightful comments!\n\nWe would like to clarify that the training resources required by DenoiseRotator are minimal, substantially less than those needed for actual deployment. Therefore, large language models capable of operating in resource-constrained or deployment-limited environments can also utilize DenoiseRotator for pruning within the same settings.\n\nDue to the independence of rotation matrices across layers, the training of each layer's rotation matrix can be conducted separately. The actual necessary memory consumption is calculated as 16 (parameters 2 + gradients 4 + optimizer states 8) times hidden times hidden, equivalent to the memory required for deploying one layer. In our straightforward, uncompiled implementation, optimizing rotation matrices for LLaMA-3-8B requires only 10GB of memory, while LLaMA-3-70B requires only 30GB.\n\nRegarding the use of low-rank matrices for orthogonal transformations, it is important to note that orthogonal matrices must be full-rank square matrices. Thus, matrices derived from low-rank approximations cannot maintain orthogonality, and due to the presence of RMSNorm, they cannot be merged into the model's weights.\n\nRank-k Householder products can only reduce the computational complexity during training but do not reduce the inference overhead.\n\nBlock-diagonal matrices offer a promising trade-off by reducing both training and inference overhead. To validate their effectiveness, we conducted experiments on LLaMA-3-8B:\n\nSetup: SparseGPT, unstructured 50%, 2000 steps, learning rate 0.01\n\n| Block Number | Perplexity | Zero-Shot Accuracy | Average Time Cost per Step (s) | Entropy |\n|----|---|-----|---|--|\n| 1   | 7.597  | 69.58   | 0.124   | 384128  |\n| 2   | 8.024  | 68.68| 0.088   | 410816  |\n| 4  | 8.544   | 68.47   | 0.076  | 428160  |\n| 8  | 8.882  | 67.51   | 0.072   | 440512  |\n\nThese results demonstrate the trade-offs in performance and computational cost associated with different block configurations. Thank you for your feedback, which has guided us in exploring these options further.\n\n### Question 2:\n\nThe core contribution of our paper lies in the concept of \"enhancing model robustness to pruning by concentrating importance onto a smaller subset of parameters.\" Learning orthogonal rotations followed by pruning is one specific implementation of this idea, tailored to post-train pruning, but it is not the only approach. Additionally, although not experimentally verified in our paper, we propose the notion of incorporating the information entropy of importance scores as a regularization term in the objective function for sparse pre-training. RotPruner, on the other hand, focuses solely on the implementation steps of learning orthogonal rotations followed by pruning without providing theoretical motivation.\n\nRegarding the differences in implementation:\nIn the RotPruner framework, the training of rotation matrices (continuous) and parameter pruning (discontinuous) are not independent but alternate. Due to the presence of pruning masks, pruned parameters do not participate in the training of rotation matrices, leading to potential information loss. This alternating implementation method is incompatible with pruners that incorporate parameter reconstruction mechanisms, such as SparseGPT. Moreover, RotPruner requires end-to-end optimization of the entire model, simultaneously optimizing rotation matrices for all layers, resulting in high memory requirements. Furthermore, RotPruner adds two non-mergeable rotation matrices per layer—one for self-attention and one for MLP—adding additional overhead.\n\nIn contrast, our proposed DenoiseRotator decouples the training of rotation matrices from parameter pruning. This approach avoids information loss due to discontinuous pruning and offers excellent compatibility with many existing pruners. DenoiseRotator adds only one non-mergeable rotation matrix per layer, and as mentioned in the response to Question 1, the training of each layer's rotation matrix can be conducted independently, significantly reducing the memory required for training.\n\nSince RotPruner has not released their code and has only conducted experiments on models of 8B and below, we provide a comparison using their paper's configuration and data for reference. The calibration dataset consists of 128 samples, each with a length of 2048, sourced from WikiText2. The models' performances are evaluated based on perplexity on the WikiText2 test set. For the optimization of the rotation matrix in DenoiseRotator, the configuration includes a learning rate of 0.01 and 2000 steps.:\n\nUnstructured 50% Sparsity:\n| Method  | LLaMA-2-7B | LLaMA-3-8B |\n|-----|----|----|\n| Dense | 5.47| 6.13  |\n| SparseGPT | 6.46  | 8.29       |\n| SparseGPT + DenoiseRotator | 6.09   | 7.25       |\n| Wanda  | 6.72| 9.40|\n| Wanda + DenoiseRotator  | 6.27       | 7.56       |\n| RotPruner | 6.42  | 8.50  |\n\nSemi-Structured 2:4 Sparsity:\n| Method| LLaMA-2-7B | LLaMA-3-8B |\n|----|---|-----|\n| Dense | 5.47   | 6.13|\n| SparseGPT | 10.37 | 14.65 |\n| SparseGPT + DenoiseRotator | 7.81  | 8.84 |\n| Wanda | 11.34 | 21.21 |\n| Wanda + DenoiseRotator  | 9.37 | 10.68|\n| RotPruner | 9.20 | 11.65 |\n\nDenoiseRotator achieved superior performance with reduced overhead. We hope these comparisons provide clarity on the distinctions and advantages of our approach. Thank you for your feedback, which has guided us in presenting a more comprehensive analysis.\n\n### Question 3:\nThank you for your question regarding downstream fine-tuning compatibility, specifically in relation to handling before LoRA/4-bit quantization.\n\nSince quantization fixes the model's weights, it is essential that the training and merging of rotation matrices occur before quantization. Additionally, as pruning removes some of the model's weights, inevitably causing information loss, LoRA should be applied before pruning to preserve as much information as possible.\n\nDuring training, freezing the rotation matrices can be beneficial. This approach ensures that the gradient flow remains consistent with scenarios where rotations are not applied, thus maintaining training stability.\n\n**DenoiseRotator Configuration:**  \n- Steps: 100  \n- Learning Rate: 0.01  \n- Model: LLaMA-3-8B  \n- Pruning: Wanda, 2:4 semi-structured  \n\n**Fine-tuning Configuration:**  \n- Method: LoRA  \n- Alpha: 32.0  \n- Dropout: 0.1  \n- LoRA-r: 8  \n- Dataset: 4096 samples of length 2048 from WikiText2 train set  \n- Learning Rate: 2e-4  \n- Weight Decay: 1e-2  \n- Optimizer: Adam  \n- Learning Rate Scheduler: Linear  \n- Warm-up Steps: 400  \n- Note: Rotation matrices are not fine-tuned  \n\n**Results on WikiText2 test set:** \n|Method| Perplexity| Zero-Shot Accuracy |\n|---|---|------|\n| Dense | 6.14  | 72.72  |\n| Wanda | 25.19  | 51.03  |\n| + DenoiseRotator | 14.31 | 60.67   |\n| + LoRA Finetune  | 9.32  | 61.08   |\n\nThe experiment demonstrates that applying DenoiseRotator improves performance compared to Wanda alone when considering both perplexity and zero-shot accuracy. Additional recovery finetuning further enhances the model, significantly reducing the perplexity and improving zero-shot accuracy. \n\n### Question 4:\nThank you for your insightful suggestion, which highlights an exciting direction for future research!\n\nWe have experimented with some straightforward strategies to explicitly incorporate the constraints of 2:4 semi-structured pruning into the optimization of rotation matrices. For instance, we tried replacing the normalization groups used for entropy calculation from entire columns or rows to groups of four consecutive parameters. However, the results were not favorable; perplexity on WikiText2 increased from 11.41 to 17.53 (LLaMA-3-8B, Wanda). We suspect this is due to the focus on local information at the expense of global context.\n\nSubsequently, we attempted to incorporate global information by multiplying the entropy of each normalization group by the sum of the importance scores within that group. This approach improved performance to a perplexity of 14.96, but it still fell short of the method that does not explicitly integrate the constraints of 2:4 semi-structured pruning into the optimization of rotation matrices. \n\nThis remains a very promising idea worth exploring further.\n\n### Question 5:\nThank you for your observation regarding Figure 2(c) and 2(d). The phenomenon you noted is due to the comparison group used by Wanda, which considers only a single column of the weight matrix. Figures 2(c) and 2(d) depict the statistical analysis results of all parameters contained within the weight matrix. The importance of parameters retained by Wanda within a single column of the weight matrix may not necessarily rank among the highest when considering the importance of all parameters."}]

Author rebuttal:

Meta-review: This paper proposes a learnable rotation on the weights using entropy minimization, allowing important parameters to concentrate into small subsets of weights, which improves pruning.

Strengths
- Interesting idea of rotating the weight matrices.
- Can be integrated with other pruning methods.
- Solid empirical results.

Weaknesses
- Potentially limited to semi-structured pruning.
- Additional training cost for learning the rotation.
- Possible overlap with the idea of RotPruner.

During the rebuttal, the authors presented additional experiments, including:
- Ablation studies on the hyperparameters.
- Demonstration of gains from semi-structured pruning.
- Clarification that the additional training introduces minimal overhead.
- Comparison with other orthogonal training schemes.
- Use of other structured matrices for rotation.
- Comparison with RotPruner.
- Further experiments on fine-tuning.

I agree with the reviewers that the paper should be accepted.
Decision: Accept (poster)