Title: DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large language model

Model compression

Pruning
Abstract: Pruning is a widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation—especially under semi-structured sparsity constraints. Existing pruning methods primarily focus on estimating the importance of individual weights, which limits their ability to preserve critical capabilities of the model. In this work, we propose a new perspective: rather than merely selecting which weights to prune, we first redistribute parameter importance to make the model inherently more amenable to pruning. By minimizing the information entropy of normalized importance scores, our approach concentrates importance onto a smaller subset of weights, thereby enhancing pruning robustness. We instantiate this idea through DenoiseRotator, which applies learnable orthogonal transformations to the model’s weight matrices. Our method is model-agnostic and can be seamlessly integrated with existing pruning techniques such as Magnitude, SparseGPT, and Wanda. Evaluated on LLaMA3, Qwen2.5, and Mistral models under 50% unstructured and 2:4 semi-structured sparsity, DenoiseRotator consistently improves perplexity and zero-shot accuracy. For instance, on LLaMA3-70B pruned with SparseGPT at 2:4 semi-structured sparsity, DenoiseRotator reduces the perplexity gap to the dense model by 58%, narrowing the degradation from 8.1 to 3.4 points.

DenoiseRotator: Enhance Pruning Robustness for LLMs via Importance Concentration

Abstract

Pruning is a widely used technique to compress large language models (LLMs) by removing unimportant weights, but it often suffers from significant performance degradation-especially under semi-structured sparsity constraints. Existing pruning methods primarily focus on estimating the importance of individual weights, which limits their ability to preserve critical capabilities of the model. In this work, we propose a new perspective: rather than merely selecting which weights to prune, we first redistribute parameter importance to make the model inherently more amenable to pruning. By minimizing the information entropy of normalized importance scores, our approach concentrates importance onto a smaller subset of weights, thereby enhancing pruning robustness. We instantiate this idea through DenoiseRotator, which applies learnable orthogonal transformations to the model's weight matrices. Our method can be seamlessly integrated with existing pruning techniques such as Magnitude, SparseGPT, and Wanda. Evaluated on LLaMA3, Qwen2.5, and Mistral models under 50% unstructured and 2:4 semistructured sparsity, DenoiseRotator consistently improves perplexity and zero-shot accuracy. For instance, on LLaMA3-70B pruned with SparseGPT at 2:4 semistructured sparsity, DenoiseRotator reduces the perplexity gap to the dense model by 58%, narrowing the degradation from 8.1 to 3.4 points. Codes are available at https://github.com/Axel-gu/DenoiseRotator .

1 Introduction

Recent advancements in large language models (LLMs) [33, 10, 1, 16] have significantly improved their performance in complex reasoning, multimodal processing, and extended context handling. However, their substantial model sizes and computational demands present practical challenges for deployment and inference efficiency. To mitigate these issues, a variety of model compression techniques have been explored, including quantization [14, 29, 3, 39, 40, 28], knowledge distillation [20], and pruning [31, 18, 13, 38, 37].

Among these, pruning stands out as an effective method for reducing parameter count and computational cost by eliminating less important weights. Traditional pruning methods [18] rely on importance score metrics-such as weight magnitude or output sensitivity-to rank parameters and prune those with the lowest scores. Notably, recent methods like SparseGPT [13] and Wanda

[38] estimate the impact of removing individual parameters using Taylor approximations, justifying pruning parameters with minimal estimated impact and approximates the output deviation of the layer by summing the importance scores of the pruned parameters.

Despite their effectiveness, these methods operate within the fixed parameter space of pretrained models and focus solely on selecting which weights to prune. This paradigm does not modify the underlying distribution of parameter importance, which limits its flexibility and robustness-particularly under semi-structured (e.g., 2:4) sparsity constraints where pruning choices are restricted.

To overcome this limitation, we propose a fundamentally different perspective. Rather than merely selecting weights to prune, we aim to reshape the distribution of importance score prior to pruning . Specifically, we instantiate this idea through DenoiseRotator , a framework that applies learnable orthogonal transformations to reparameterize weight matrices, leveraging the computational invariance [2] of Transformer architectures. As illustrated in Figure 2, these transformations are trained to rotate the weight matrices in a way that concentrates the importance scores into a smaller subset of parameters, thereby enhancing pruning robustness.

The term Denoise reflects the principle that minimizing the information entropy of normalized importance scores-viewed as a discrete probability distribution-provides a theoretically grounded, differentiable, and permutation-invariant objective for importance concentration. Moreover, the norm-preserving property of orthogonal transformations ensures that the total importance of each layer remains invariant, allowing importance to be redistributed across parameters without being artificially introduced or lost. This property contributes to the overall stability of the algorithm.

Our key contributions are summarized as follows:

Entropy-Guided Importance Concentration: We propose enhancing pruning robustness by concentrating parameter importance into a small subset through minimizing information entropy of a discrete distribution consisting of normalized parameter importance.

DenoiseRotator Framework: We introduce DenoiseRotator , a concrete instantiation of our importance concentration idea, implemented via learnable orthogonal transformations that leverage the computational invariance [2] of Transformer [41] architectures.

Plug-and-Play Compatibility: DenoiseRotator decouples the importance concentration process from the actual pruning step. The learnable transformations are optimized independently prior to pruning, and can be plugged into any existing pruning pipeline.

Extensive Empirical Evaluation: We demonstrate the effectiveness of DenoiseRotator on a range of open-source LLMs, including Mistral (7B) [24], LLaMA3 (8B, 70B) [17], and Qwen2.5 (7B, 14B, 32B, 72B) [42]. Our method consistently and significantly improves pruning performance across unstructured and semi-structured sparsity patterns, reducing perplexity and improving accuracy compared to baseline pruning methods.

By combining entropy-guided importance concentration with orthogonal transformations, DenoiseRotator offers a novel and principled direction for robust model pruning.

2 Background

Neural network pruning is a widely used model compression technique that reduces model size and computational cost by eliminating parameters deemed less important. Existing approaches can be broadly categorized into two paradigms. The first class relies on heuristic metrics to estimate parameter importance and prunes those ranked lowest [31, 18, 13, 38, 44, 9, 26, 19, 45, 11]. The second class formulates pruning as a continuous optimization problem by relaxing the binary pruning mask into a differentiable form, allowing end-to-end training at the cost of significantly increased computational overhead [30, 12].

Computational invariance refers to the property of transformer architecture proposed in SliceGPT [2] that allows certain orthogonal transformations to be applied to their weight matrices without altering the model's output. Building upon this principle, QuaRot [3] and SpinQuant [29] apply orthogonal rotations to weight matrices and activations to facilitate low-bit quantization by redistributing outlier values more evenly. RotPruner [7] attempts to leverage this property for pruning and

RoPE

dn

RMSNorm

RMSNorm

Softmax

RoPE

[x]

Gate

Swish

Rotate for concentration of

parameter importance

R

aQ

RoPE

R-

αUp

Softmaz

αK

RoPE

Down

R

aV

R2

R

aGate

Swish

Merge and prune

RoPE

Up

Softmax

RoPE

Gate'

Swish

R

R2

Orthogonal matrix

forrotation

R-

Mergeablematrix

combination

Up'

Pruned matrix

reports promising empirical results; however, it lacks theoretical analysis or formal justification for why rotation improves pruning robustness.

3 Method

In this section, we present the motivation and formulation of our proposed framework. We begin by formalizing the problem setting and introducing the concept of enhancing pruning robustness through concentrating parameter importance. Next, we describe how reducing the information entropy of normalized importance scores naturally leads to such concentration. Finally, we introduce a practical instantiation of this idea, termed DenoiseRotator , which implements entropy-based importance concentration using learnable orthogonal transformations.

3.1 Enhance pruning robustness via parameter importance concentration

Pruning aims to reduce model size and computational requirements by removing parameters deemed less important. A critical aspect of effective pruning is accurately estimating the importance of each parameter. As mentioned earlier, recent methods such as SparseGPT [13] and Wanda [38] leverage Taylor series to approximate the change in a linear layer output when a parameter is removed, serving as importance metrics to identify and eliminate the least significant parameters.

For instance, the Wanda method computes the importance score S Wanda ij for each weight W ij of a linear layer parameterized by W ∈ R d out × d in by considering both the weight's magnitude and the corresponding input activation norm:

This metric is equivalent to applying the Optimal Brain Damage (OBD) [26] approach to the optimization problem of pruned weight ˆ W to minimize the change in the linear layer's output after

(a)

(b)

(c)

(d)

Original layer.0.self_attn.o_proj.weights' importance

0.200

0.175

0.150

0.125

Importance

0.100

0.075

+0.050

0.025

0.000

4000

3500

3000

2500

0

500

2000

1000

1500

1500dim

Output dimension

2000

2500

3000

500

1000P

3500

4000

0

Count distribution of original layer.0.self_attn.o_proj.weights' importance

400000

350000

Unpruned

Pruned

300000

250000

200000

150000

100000

50000

0

0.000

0.002

00'0

0.006

0.008

0.010

Importance

Rotated layer.0.self_attn.o_proj.weights' importance

0.200

0.175

0.150

0.125

Importance

0.100

0.075

+0.050

0.025

0.000

4000

3500

3000

2500

0

500

2000

1000

1500

1500dim

ousuawip gndno

2000

2500

3000

500

1000nP°

3500

4000

0

Count distribution of rotated layer.0.self_attn.o_proj.weights' importance

400000

350000

Unpruned

Pruned

300000

250000

200000

150000

100000

50000

0.000

0.002

00'0

0.006

0.008

0.010

Importance

pruning. The OBD importance score is the second-order term of Taylor approximation:

Consequently, optimizing the change caused by pruning could be approximated by minimizing the sum of importance scores over the pruned parameters whose index is in P , defined as the set of indices corresponding to weights selected for removal, according to Talyor approximation property:

This insight motivates minimizing the total importance of the pruned parameters, which underlies the algorithmic design of existing methods - sorting and removing the weights with the lowest importance scores. However, reducing this sum is not limited to selecting the least important parameters-we can also reshape the distribution of importance itself, offering a new perspective for enhancing pruning robustness.

To this end, we propose applying transformations to minimize total importance score of pruned weight. Formally, the optimization objective can be expressed as:

where ( T W , T X ) is a pair of transformation applied to the layer's weight and input (e.g., orthogonal transformation), and T S is the corresponding transformation applied to the importance scores S ij . Here, we omit the specific form of the OBD score to emphasize that our method is compatible with pruning metrics based on heuristic importance estimation, where parameters with lower estimated importance are removed.

By decreasing the total importance of pruned weights, the deviation in the linear layer's output is minimized, thus preserving model performance and enhancing robustness against degradation caused by pruning. Furthermore, reducing the magnitude of importance score of pruned weights improves the accuracy of Taylor-series-based importance approximations, leading to more precise importance estimation and compensation strategies, such as those used in SparseGPT [13].

3.2 Reducing information entropy of normalized importance score

Although the objective in Equation 4 provides a principled formulation for importance concentration, it is generally difficult to optimize in practice. On the left-hand side, the selection of the pruning indices P is a combinatorial problem and is known to be NP-hard [6]. On the right-hand side, sortingbased strategies-commonly used to identify low-importance parameters-are non-differentiable and thus incompatible with gradient-based optimization.

To tackle these challenges, we propose optimizing a learnable transformation to minimize the information entropy of the normalized importance distribution prior to pruning as a surrogate objective for concentrating parameter importance. Given that importance scores are non-negative, they can be normalized to form a valid probability distribution, facilitating a probabilistic interpretation of parameter importance. Entropy, being permutation-invariant, remains unaffected by the index ordering of parameters. Furthermore, as a concave function over the probability simplex, entropy, when minimized, naturally promotes sparsity by concentrating mass onto fewer elements. This strategy independently addresses importance redistribution, separate from the pruning process itself, allowing enhanced flexibility across various pruning metrics and sparsity patterns.

Formally, let S = { S ij } denote the importance scores, and T S be a learnable transformation (e.g., orthogonal rotation applied indirectly through weight and input transformation) that reshapes the distribution of importance. We first compute transformed scores S ′ = T S ( S ) , and normalize them over a specified group G ⊆ { ( i, j ) } (such as a row, column, or the entire matrix):

where P = { p ij } denotes the normalized importance scores computed within group G , which could be seen as a discrete probability distribution, its entropy H ( P ) is then given by the above formula.

Our objective is to learn a shared transformation T S that minimizes the entropy across all normalization groups:

The summation is taken over all groups that share the same transformation T S . By reducing the entropy of the transformed importance distribution, we effectively transfer importance toward a smaller subset of parameters. This results in a more skewed and robust importance profile, making the model inherently more amenable to pruning.

3.3 DenoiseRotator

In this section, we introduce DenoiseRotator , a practical instantiation of the entropy-based importance concentration framework described earlier. DenoiseRotator implements the learnable transformation T S using orthogonal matrices, enabling a lightweight yet effective mechanism to reshape importance distributions without altering the model's functionality.

DenoiseRotator is designed to be modular and can be easily combined with existing pruning pipelines. The following pseudocode outlines an example workflow of integrating DenoiseRotator (highlighted in blue) with layer-wise pruning methods:

Algorithm 1 DenoiseRotator Integration Pipeline

Require:

Dense model M , pruning method P , calibration data D cal

Ensure:

Pruned model M ′

Merge RMSNorm's weight into adjacent linear layers

Compute Hessian for pruning H ← XX ⊤ { X is from D cal}

Integrate and train orthogonal matrices R 1 , R 2

Merge orthogonal matrices R 1 , R 2

Apply pruning method P to convert M into M ′ return Pruned model M ′

3.3.1 Integration of orthogonal matrices

As shown in Figure 1, DenoiseRotator integrates two pairs of learnable orthogonal matrices into each Transformer layer.

Layer-Level Rotations ( R 1 ) : A pair of orthogonal matrices, R 1 and R ⊤ 1 of shape ( d hidden , d hidden ) , are applied at the beginning and end of each Transformer layer, respectively. Specifically, R 1 is inserted before the RMS normalization and residual addition, and R ⊤ 1 is applied after these operations. R 1 effectively maps activations and weights into rotated spaces, allowing the network to redistribute parameter importance. Since orthogonal transformations preserve vector norms and inner products, the inputs to non-linear functions remain consistent. Thus, the model's output remains invariant.

Attention-Level Rotations ( R 2 ) : Within the self-attention, another pair of orthogonal matrices R 2 and R ⊤ 2 are applied to the Value ( V ) and Output ( O ) projections. These rotations adjust the internal representations, further concentrating importance and enhancing the model's resilience to pruning.

To elucidate the effect of these transformations, consider the Output ( O ) projection as an example. Let W denote its weight matrix of shape ( d hidden , d hidden ) and X its input of shape ( d hidden , len × bsz ) before transformation. After applying the orthogonal transformations, the weight and input are transformed as follows:

The corresponding transformation of the OBD importance score in Equation 2 can be expressed as:

This formulation reveals how importance scores are indirectly redistributed through orthogonal transformations applied to inputs and weights. Detailed transformation for other linear layer and pruning method could be found in Appendix A.

After training, the orthogonal matrices, except for those at the beginning and end of the Transformer layer, are merged into the model's weight matrices as illustrated in Figure 1. Pruning is then applied to the merged weight matrices.

This integration leverages structural properties of the Transformer architecture-specifically, the presence of residual connections, layer normalization (which could be transformed to RMS normalization [2]), and attention mechanisms-and is suitable for any model that exhibits similar characteristics.

3.3.2 Invariance of total importance

Orthogonal transformations preserve the total importance of a linear layer's weights due to their property of maintaining vector norms, i.e., ∥ Rx ∥ = ∥ x ∥ . Formally, this implies ∑ T S ( S ) = ∑ S , where T S denotes the orthogonal transformation applied to the importance scores. This invariance ensures that importance is redistributed among parameters without being artificially introduced or eliminated, thereby making the algorithm stable. For a full derivation and proof of the invariance property under orthogonal transformation, see Appendix B.

To align the importance concentration mechanism with invariance, normalization in the entropy computation 5 is performed based on the position of the orthogonal matrix relative to the weight

matrix. Specifically, if the orthogonal matrix is applied on the right side (e.g., WR ), normalization is conducted over each row. Conversely, if the orthogonal matrix is applied on the left side (e.g., RW ), normalization is performed over each column. In cases where orthogonal matrices are applied on both sides (e.g., R 1 WR 2 ), both row-wise and column-wise entropies are calculated and their sum is used. This means that the normalization group is determined by the position of the orthogonal matrix.

3.3.3 Optimization of the orthogonal matrix

During training, only the orthogonal matrices are optimized, while the weights of the original model and input features are kept frozen, thus the hessian matrix H = XX ⊤ for pruning could be reused. Each orthogonal matrix is initialized as the identity matrix to ensure that the model's behavior remains unchanged at the start of training.

Since all linear layers within a Transformer layer share the same set of orthogonal transformations R 1 and R 2 , the overall training objective within a Transformer layer is defined as the sum of the entropy values across all normalization groups from all affected linear layers. The total loss is:

where L i denotes the set of all linear layers within the i -th Transformer layer, and G indexes the normalization groups within each linear layer. Note that we do not apply any additional weighting to the entropy of each group. This is because the number of normalization groups per linear layer is inherently tied to the weight's shape (e.g., rows or columns), and the value of discrete entropy naturally scales with the number of categories.

Maintaining the orthogonality of matrices R 1 and R 2 during backpropagation can be challenging, as direct gradient descent on these matrices does not inherently preserve orthogonality. To address this issue, we employ a reparameterization strategy using QR decomposition. For each orthogonal matrix, we define an unconstrained matrix A and compute its QR decomposition: A = QR , where Q is an orthogonal matrix and R is an upper triangular matrix. During training, only the orthogonal component Q is utilized in the forward pass, while gradients are applied to the underlying matrix A . The theoretical justification for the feasibility of this method is provided in Appendix C.

4 Experiment

Models and Tasks We evaluate our method on several recent open-source large language models, including Mistral 7B [24], LLaMA 3 (8B and 70B) [17], and Qwen2.5 (7B, 14B, 32B, and 72B) [42]. Our evaluation encompasses both language generation, measured by perplexity on the WikiText-2 dataset [32], and five widely-used zero-shot tasks: PIQA [5], WinoGrande [36], HellaSwag [43], ARC-e, and ARC-c [8]. Following established practices, we utilize the LM Evaluation Harness [15] with default settings for all evaluations.

Baselines We integrate DenoiseRotator with three pruning methods: the classic Magnitude pruning and two advanced techniques, Wanda and SparseGPT. Both Wanda and SparseGPT require calibration data to estimate input statistics. For consistency, we use the same calibration dataset as in prior work, consisting of 128 sequences with a context length of 2048 tokens, sampled from the C4 training set [35]. We apply a uniform sparsity level across all decoder layers and evaluate two types of sparsity: unstructured 50% sparsity and semi-structured 2:4 sparsity.

Setup We trained the orthogonal matrices of DenoiseRotator using the Adam optimizer with a learning rate of 0.01 over 2000 steps. All computations, except for QR decomposition, were performed in torch.bfloat16 precision to enhance efficiency. The computation resource cost scales linearly with the number of model parameters. For instance, training on LLaMA 3 70B with SparseGPT took approximately 28 hours and utilized around 30 GB of GPU memory on a single NVIDIA A100 GPU. Notably, the training duration is independent of the calibration dataset size, as the optimization process reuse the hessian matrix for pruning. No recovery finetuning was performed after pruning.

4.1 Main result

In Table 1, we evaluate the generation capabilities of both dense and pruned models-with and without the integration of DenoiseRotator-using perplexity on the WikiText-2 validation set. Under both unstructured (50%) and semi-structured (2:4) sparsity settings, DenoiseRotator consistently and significantly reduces perplexity across all model families and scales.

We report zero-shot task results for dense and pruned models in Table 2. Across both unstructured (50%) and semi-structured (2:4) sparsity settings, DenoiseRotator consistently improves zero-shot accuracy, often recovering performance close to or surpassing the dense model, especially on larger models like Qwen2.5-72B and LLaMA3-70B.

Although we do not explicitly optimize for the 2:4 semi-structured sparsity constraint, DenoiseRotator still performs remarkably well-even making previously unusable models viable under this constraint. This is likely because the orthogonal matrices act as a form of random permutation, increasing the chance that crucial weights align with the semi-structured sparsity pattern.

The impact of rotation is illustrated in Figure 2, which visualizes the OBD importance for the output projection in the first layer of LLaMA-3-8B before and after applying DenoiseRotator, revealing how importance becomes more concentrated after transformation.

For detailed performance results (including the LLaMA 1 and 2 families), please refer to Appendix G.

4.2 Effectiveness of Entropy Reduction on Pruning Robustness

To validate the efficacy of entropy reduction in enhancing pruning robustness, we conduct an ablation study on LLaMA-3-8B using SparseGPT with a 50% unstructured sparsity calibrated by 128 sequences of 2048 tokens sampled from C4 training set (generation task). We record model performance at various entropy reduction steps of DenoiseRotator. Table 3 presents the average entropy value per layer, time

Table 3: Impact of entropy reduction on pruning robustness. Results are reported on LLaMA-3-8B with SparseGPT (unstructured 50% sparsity) calibrated by 128 sequences of 2048 tokens sampled from C4 training set.

consumed, average zero-shot accuracy, and perplexity on the WikiText2 validation set over different optimization steps. 0 step refers to the SparseGPT baseline.

As shown in Table 3, reducing the entropy of the normalized importance distribution significantly improves model's performance. At step 100, where average entropy per layer is reduced by 13%, we observe a substantial boost in zero-shot accuracy from 66.88% to 70.54%, along with a decrease in perplexity from 9.567 to 7.701. Further optimization continues to reduce entropy and improve perplexity. A comprehensive hyperparameter analysis, detailing optimization steps and learning rates, is available in Appendix D.

These results empirically support our hypothesis: concentrating importance via entropy reduction enables more robust pruning, leading to reduced performance degradation.

4.3 Inference speedup and overhead analysis

DenoiseRotator introduces a pair of orthogonal matrices at the beginning and end of each layer. Since the orthogonal matrices of adjacent layers can be merged, the actual overhead is limited to storing an additional shape matrix ( hidden_size , hidden_size ) per layer, along with a matrix multiplication during inference. For example, in LLaMA-3-8B, this results in approximately 0.5 billion additional parameters (i.e., 4096 × 4096 × 32 ). This overhead is relatively small, accounting for only about 6.7% of the original model size. To further address overhead concerns, we explore the potential of block diagonal orthogonal matrices as a reduction strategy in Appendix E.

We evaluate the average inference time of a single Transformer layer in LLaMA3-8B on 32 sequences of length 2048 using an NVIDIA A100 GPU. The batch size is set to 1, and we utilize PyTorch's semistructured sparsity framework, specifically the torch.sparse.to_sparse_semi_structured function. Timing measurements are conducted using torch.utils.benchmark.Timer . As shown in Table 4, a dense layer takes 5.80 ms on average, while a 2:4 semi-structured sparse layer reduces this to 4.37 ms-achieving a 1.33× speedup. The additional cost introduced by DenoiseRotator's orthogonal matrix is only 0.32 ms per layer, which is minor compared to the overall computation time and maintains practical efficiency for deployment.

5 Conclusion

This paper introduces DenoiseRotator, a plug-and-play framework that improves the robustness of model pruning through importance concentration. By applying learnable orthogonal transformations optimized via entropy reduction, DenoiseRotator reshapes the importance landscape to ensure critical informations and abilities are preserved during pruning. Our theoretical analysis and empirical studies validate the effectiveness of this strategy: concentrating importance distributions results in more informative retained parameters and enhances the model's robustness to pruning. Extensive evaluations on multiple LLMs show consistent gains across perplexity and zero-shot accuracy under both unstructured and semi-structured sparsity. Moreover, DenoiseRotator achieves these improvements with minimal overhead and without modifying the original pruning pipeline. These results demonstrate the promise of entropy-guided importance reshaping as a general principle for robust and efficient model sparsification. While our experiments focus on post-training pruning, the underlying idea of concentrating parameter importance may also benefit other stages of the model lifecycle, such as being integrated into pretraining or continual training.

6 Limitations

Semi-structured sparsity patterns offer hardware acceleration benefits but also impose structural constraints. Future work is needed to explore how such constraints can be explicitly integrated into the learnable transformation's optimization process.

References

anthropic. Claude 3.7 sonnet and claude code, Feburary 2025.

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns. In The Twelfth International Conference on Learning Representations , 2024.

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Pashmina Cameron, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. Quarot: Outlier-free 4-bit inference in rotated LLMs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Gary Bécigneul and Octavian-Eugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.00760 , 2018.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 7432-7439. AAAI Press, 2020.

Thomas Blumensath and Mike E Davies. Iterative thresholding for sparse approximations. Journal of Fourier Analysis and Applications , 14(5-6):629-654, 2008.

Haoxian Chen and Limin Wang. Rotpruner: Large language model pruning in rotated space, 2025.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1 , 2018.

Rocktim Jyoti Das, Liqun Ma, and Zhiqiang Shen. Beyond size: How gradients shape pruning decisions in large language models, 2024.

DeepSeek-AI, Daya Guo, Dejian Yang, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.

Peijie Dong, Lujun Li, Zhenheng Tang, Xiang Liu, Xinglin Pan, Qiang Wang, and Xiaowen Chu. Pruner-zero: Evolving symbolic pruning metric from scratch for large language models. In Forty-first International Conference on Machine Learning , 2024.

Gongfan Fang, Hongxu Yin, Saurav Muralidharan, Greg Heinrich, Jeff Pool, Jan Kautz, Pavlo Molchanov, and Xinchao Wang. MaskLLM: Learnable semi-structured sparsity for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 202 of Proceedings of Machine Learning Research , pages 10323-10337. PMLR, 2023.

Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323 , 2022.

Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024.

Google. Gemini 2.5: Our most intelligent ai model, March 2025.

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The llama 3 herd of models, 2024.

Song Han, Huizi Mao, and William J. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, 2016.

Babak Hassibi, David G. Stork, and Gregory J. Wolff. Optimal brain surgeon: Extensions and performance comparison. In Jack D. Cowan, Gerald Tesauro, and Joshua Alspector, editors, Advances in Neural Information Processing Systems 6, [7th NIPS Conference, Denver, Colorado, USA, 1993] , pages 263-270. Morgan Kaufmann, 1993.

Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR , abs/1503.02531, 2015.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations , 2022.

Xing Hu, Yuan Cheng, Dawei Yang, Zhixuan Chen, Zukang Xu, JiangyongYu, XUCHEN, Zhihang Yuan, Zhe jiang, and Sifan Zhou. OSTQuant: Refining large language model quantization with orthogonal and scaling transformations for better distribution fitting. In The Thirteenth International Conference on Learning Representations , 2025.

Ioan Mackenzie James. The topology of Stiefel manifolds , volume 24. Cambridge University Press, 1976.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023.

Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch, 2020.

Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky, editor, Advances in Neural Information Processing Systems , volume 2. Morgan-Kaufmann, 1989.

Jun Li, Fuxin Li, and Sinisa Todorovic. Efficient riemannian optimization on the stiefel manifold via the cayley transform. In International Conference on Learning Representations , 2020.

Yifei Liu, Jicheng Wen, Yang Wang, Shengyu Ye, Li Lyna Zhang, Ting Cao, Cheng Li, and Mao Yang. Vptq: Extreme low-bit vector post-training quantization for large language models. In The 2024 Conference on Empirical Methods in Natural Language Processing , 2024.

Zechun Liu, Changsheng Zhao, Igor Fedorov, Bilge Soran, Dhruv Choudhary, Raghuraman Krishnamoorthi, Vikas Chandra, Yuandong Tian, and Tijmen Blankevoort. Spinquant: LLM quantization with learned rotations. In The Thirteenth International Conference on Learning Representations , 2025.

Christos Louizos, Max Welling, and Diederik P. Kingma. Learning sparse neural networks through l 0 regularization. In International Conference on Learning Representations , 2018.

Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems , 2023.

Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations , 2017.

OpenAI, Josh Achiam, Steven Adler, et al. Gpt-4 technical report, 2024.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, highperformance deep learning library. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. , 21:140:1-140:67, 2020.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 8732-8740. AAAI Press, 2020.

Hang Shao, Bei Liu, and Yanmin Qian. One-shot sensitivity-aware mixed sparsity pruning for large language models. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 11296-11300, 2024.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations , 2024.

Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. QuIP$\#$: Even better LLM quantization with hadamard incoherence and lattice codebooks. In Forty-first International Conference on Machine Learning , 2024.

Albert Tseng, Qingyao Sun, David Hou, and Christopher De Sa. QTIP: Quantization with trellises and incoherence processing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA , pages 5998-6008, 2017.

An Yang, Baosong Yang, Beichen Zhang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Màrquez, editors, Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pages 4791-4800. Association for Computational Linguistics, 2019.

Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. Pruning meets low-rank parameter-efficient fine-tuning, 2023.

Yingtao Zhang, Haoli Bai, Haokun Lin, Jialin Zhao, Lu Hou, and Carlo Vittorio Cannistraci. Plug-and-play: An efficient post-training pruning method for large language models. In The Twelfth International Conference on Learning Representations , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction accurately reflect the paper's scope and contributions, including the proposal of entropy-guided importance concentration, the design of DenoiseRotator, and its compatibility and effectiveness across pruning methods. These claims are supported by both theoretical analysis and experimental results.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations in terms of the additional training cost and overhead introduced by learnable orthogonal transformations and the lack of explicit adaptation to semi-structured sparsity patterns like 2:4.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification:The paper provides clearly stated mathematical formulations and proof-particularly for the entropy-based importance concentration and invariance of total importance-with all assumptions explicitly described and derivations validated through ablation studies and empirical evidence.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper provides detailed descriptions of model settings, datasets, pruning methods, training configurations, and evaluation metrics. It also outlines a reproducible workflow and presents a full integration pseudocode for the DenoiseRotator framework, facilitating reproduction of the main results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The model and data used in this paper are open source, and the code has been submitted to ensure faithful reproduction of the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper provides detailed experimental settings including model types, sparsity configurations, pruning methods, and evaluation datasets. Other implementation details such as the number of calibration samples and training steps for DenoiseRotator are also described in the main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: The paper reports consistent improvements across a wide range of models and tasks, but it does not include error bars or formal statistical significance tests. Future revisions may include these to strengthen empirical claims.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The paper specifies the number of calibration tokens, training steps, and runtime for entropy optimization.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This work adheres to the NeurIPS Code of Ethics. It does not involve human subjects or sensitive data.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper proposes a general-purpose model compression technique focused on enhancing the robustness of pruning algorithms for large language models. The method operates at the algorithmic level without targeting specific applications or domains, and thus does not raise direct societal impact concerns such as fairness, privacy, or misuse. As a result, the broader societal impact is minimal and not explicitly addressed in the paper.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper does not release any data or pretrained models that pose a high risk of misuse. It focuses on a general pruning framework applicable to existing models without introducing new datasets or deployable systems. Therefore, no specific safeguards are necessary.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All third-party assets used in this paper-including pretrained language models (LLaMA-3, Qwen2.5, Mistral), datasets (WikiText-2, C4), and pruning baselines (SparseGPT, Wanda, Magnitude)-are properly cited and credited in the main paper. Their usage complies with the licenses and terms specified by the original authors.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduces new assets, including implementation code.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve any crowdsourcing or research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This work does not involve any research with human subjects or crowdsourcing, and therefore no IRB approval is required.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Large language models were not used as a core or original component in the methodology of this research; their use was limited to standard writing assistance and does not impact the scientific contributions of the paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Details of Orthogonal Transformations and Importance Score Formulations

In this section, we present how orthogonal transformations affect the importance scores under three commonly used pruning methods: Magnitude pruning, Wanda, and SparseGPT. We also show how we modify the importance formulas to maintain the Taylor expansion property under transformation.

Throughout this discussion, we denote the original weight matrix as W ∈ R d out × d in , the input matrix as X ∈ R d in × n , and approximate the Hessian as H = XX ⊤ . The output of a linear layer is given by Y = WX .

Depending on which sides the orthogonal transformations are applied, three typical cases arise:

Case 1: Right-side rotated weight and left side rotated input (e.g., Q, K, Up, Gate).

For layers such as Query (Q), Key (K), Up, and Gate, the weight is rotated on the right side and the input on the left. Specifically, the transformed weights and inputs are given by W ′ = WR 1 and X ′ = R ⊤ 1 X , respectively. The corresponding changes in importance scores under different pruning methods are summarized below:

Case 2: Left-side rotated weight and unrotated input (e.g., Down)

For layers like Down, only a left-side rotation is applied to the weights, while the inputs remain unrotated. In this case, W ′ = R ⊤ 1 W and X ′ = X . The modified importance scores are given by:

Case 3: Two-side rotated weight and left-side rotated input (e.g., O, V)

For components like Output (O) and Value (V) projections in the self-attention mechanism, rotations are applied on both sides.

For Output (O): W ′ = R ⊤ 2 WR 1 and X ′ = R ⊤ 1 X . The transformed importance scores are:

For Value (V): W ′ = R ⊤ 1 WR 2 and X ′ = R ⊤ 2 X . The corresponding transformation is similar to that of Output (O).

B Proof of invariance of total importance

As established in Equation 3, the total importance of parameters can be approximated by the Frobenius norm of the layer output, corresponding to the scenario where all weights are pruned (set to zero). Specifically, according to optimal brain damage [26], we have:

Since the magnitudes of | W ij | are very small, this approximation is highly accurate.

Case 1: Right side rotated weights and left side rotated inputs (e.g., Q, K, Up, Gate).

When both the weight matrix and the input are transformed via orthogonal matrices:

Case 2: Left side rotated Weights and unrotated input (e.g., Down).

When only the weight is rotated:

This equality holds because orthogonal matrices satisfy R ⊤ = R -1 , and hence preserve both vector norms and inner products.

Case 3: Two side rotated weight and left side rotated input (e.g., O, V).

Components such as O and V , which are affected by both left and right orthogonal transformations, can be viewed as a combination of the two cases above. Thus, their total importance is also preserved.

C QR decomposition reparameterization

Parameter

LossFunction

LossFunction

Intermediate

Variable

QR Decomposition

Orthogonal Matrix

Reparameterization

Forward

A

R

UpperTriangularMatrix

Backward

Unconstrained Matrix

Standard gradient descent algorithms and their variants encounter significant challenges when dealing with the orthogonality constraint of matrices. In conventional settings, these algorithms directly update matrix parameters without ensuring that the orthogonality condition is preserved post-update. This can lead to a loss of orthogonality, compromising model integrity and stability. To address this, previous research [29, 22, 7] has frequently adopted Stiefel manifold optimization techniques [23, 27, 4, 25]. These methods are specifically tailored to handle the complex geometric constraints associated with orthogonal matrices, necessitating specialized optimizers and additional computational resources.

QR decomposition provides an elegant solution to this problem by exploiting the properties of linear algebra to indirectly parameterize orthogonal matrices. This approach sidesteps the orthogonality constraint by using the QR decomposition, thus enabling the use of conventional gradient descent methods for optimization tasks without requiring complex manifold optimizers. As a result, QR decomposition not only maintains the orthogonality of matrices but also simplifies the optimization process, effectively mitigating the difficulties posed by orthogonality constraints.

In Figure 3, we start with an unconstrained matrix A . During the forward pass of the neural network, a QR decomposition is performed, splitting A into two matrices: an orthogonal matrix Q and an upper triangular matrix R . The orthogonal matrix Q becomes the immediate subject of optimization in the loss function, even though we never directly modify Q itself. Instead, we focus on optimizing the unconstrained matrix A , which indirectly alters Q due to the nature of QR decomposition.

The process of QR decomposition is composed of basic matrix operations like dot products, vector normalization, and projection. These are mathematically smooth operations, meaning they can be differentiated, which is key for gradient-based optimization. Each step in QR decomposition involves operations that smoothly vary, making it possible to calculate how small changes in the input A affect the output Q and R .

Because of these properties, QR decomposition can seamlessly integrate into the computation graph used by automatic differentiation tools. PyTorch [34] provides automatic differentiation support for QR factorization through torch.qr . When QR decomposition is performed, PyTorch automatically manages and propagates gradients with respect to the input matrix A . This capability enables gradient-based methods, such as gradient descent, to efficiently update A , automatically respecting the orthogonality constraint imposed on Q .

QR decomposition reparameterization therefore transcends the limitations of direct parameterization by introducing a flexible and effective mechanism to handle orthogonality in optimization tasks. As the network trains, the orthogonal matrix Q adapts continuously to minimize the loss function, while the underlying matrix A undergoes refinement through regular gradient-based updates.

D Hyperparameter Sensitivity Analysis

A thorough analysis of hyperparameter sensitivity is essential for understanding how different hyperparameter choices impact the final performance of a model. In the main paper, we relied on a fixed set of hyperparameters, such as using 2000 steps in the rotation training process. To enhance the rigor and reproducibility of our research, we conducted comprehensive experiments on the LLaMA-3-8B model, exploring the effects of various hyperparameter configurations on the final results.

In our experiments, each row presents the perplexity and average zero-shot accuracy corresponding to different learning rates and training steps. The setup involves utilizing SparseGPT with 50% unstructured pruning, using a calibration dataset consisting of 128 samples of length 2048 from the C4 dataset. All experiments were performed on an A100 80G GPU.

These results indicate: 1 . When the learning rate is less than or equal to 0.001, the efficiency is low due to the small learning rate. 2 . With a learning rate of 0.01, training converges around 2000 steps, and excessive steps increase training costs. 3 . Compared to the baseline without the importance concentration mechanism, all configurations show significant improvements as training progresses. 4 . Zero-shot task accuracy fluctuates around 70%, with a variation of approximately 1%, not showing a clear linear relationship with perplexity.

E Trade-off with Block Diagonal Orthogonal Matrices

Block diagonal orthogonal matrices provide an effective approach to reduce both training and inference overhead of DenoiseRotator. By leveraging the structural efficiency of block diagonal matrices, this method achieves a balanced optimization of performance and resource utilization.

In our experiments with the LLaMA-3-8B model, we evaluated these benefits using SparseGPT with 50% unstructured pruning over 2000 steps and a learning rate of 0.01. Dense orthogonal matrices were replaced with block diagonal matrices, consisting of block_num orthogonal submatrices arranged along the diagonal. Each submatrix has dimensions of hidden_size block_num . This approach reduces both spatial and computational complexity to 1 block_num of the original requirements.

These results highlight the trade-offs between performance and computational demands across different block configurations, illustrating the capacity of block diagonal orthogonal matrices to optimize these aspects effectively.

Looking ahead, potential enhancements could involve: 1 . Varying the dimensions of each block within the block diagonal matrix (as opposed to maintaining uniform block sizes in this section's experiments) to further improve performance. 2 . Incorporating permutation matrices, which have low computational complexity, to combine with block diagonal orthogonal matrices, thus emulating dense orthogonal matrices. This approach might involve arranging highly correlated dimensions or parameters adjacently for enhanced efficacy.

F Compatibility with LoRA for Fine-Tuning Pruned Models

In this section, we explore the use of LoRA [21] to fine-tune pruned models. By freezing rotation matrices during fine-tuning, we maintain consistent gradient flow, aligning with scenarios where rotations are not applied, thus ensuring stability throughout the training process.

DenoiseRotator Configuration

Steps: 100

Learning Rate: 0.01

Model: LLaMA-3-8B

Pruning Method: Wanda, 2:4 semi-structured

Fine-tuning Configuration

Dataset: 4096 samples of length 2048 from the WikiText2 train set

Method: LoRA

Alpha: 32.0

Dropout: 0.1

LoRA-r: 8

Learning Rate: 2e-4

Weight Decay: 1e-2

Optimizer: Adam

Learning Rate Scheduler: Linear

Warm-up Steps: 400

Note: Rotation matrices are kept frozen during fine-tuning

These results illustrate that additional fine-tuning with LoRA further reduces perplexity and improves accuracy. This suggests that fine-tuning post-denoising is an effective strategy for boosting performance, while confirming that the rotation matrices introduced do not destabilize the training process.

G Performance of pruned model

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DenoiseRotator, a plug‑and‑play pre‑pruning procedure for large language models (LLMs). Instead of only selecting low‑importance weights, the main contribution of this paper is to introduce an importance score redistribution framework by inserting two pairs of learnable orthogonal matrices per Transformer block. These matrices are trained to minimise the information entropy of the normalised importance scores, so that the importance scores concentrate on fewer weights, making any subsequent pruning step more robust. Extensive experiments on Mistral‑7B, LLaMA‑3 (8 B & 70 B), and Qwen‑2.5 (up to 72 B) show large perplexity and zero‑shot‑accuracy gains at both 50 % unstructured and semi-structured sparsity compared with the underlying pruners.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n### Quality\n- This paper is technically sound, with a clear mathematical formulation linking entropy to pruning robustness.\n- Solid implementation details (QR re‑param, invariance proof).\n- Large-scale evaluation across three types of models, various sizes, three baselines of importance scoring methods, and three sparsity patterns. \n\n### Clarity\n- Writing is generally good. Organisation and notation are easy to follow. Fig. 1 & Alg. 1 are helpful.\n- Appendix lists transformed score formulas, aiding reproducibility.\n\n### Significance\n- Clear improvement over baselines at both 50 %, 2 : 4 and 4:8 masks.\n- The method is agnostic to the pruner; practitioners can add it cheaply before any favourite pruning tool.\n\n### Originality\n- Introduces entropy-minimised importance shaping, which is new in the pruning literature.\n\n## Weaknesses\n### Quality\n- Recovery /downstreaming fine-tuning (FT) is not discussed. Since the pruned model still contains one orthogonal pair per block, discussing the potential risk/instability of light FT would be beneficial.\n\n### Clarity\n- Some images are confusing (e.g., Fig. 2 (c) & (d)). See Questions.\n\n### Significance\n- Some closely related baselines are not included. See Questions.\n\n### Originality\n- High-level idea (“learn orthogonal rotations, then prune”) overlaps with RotPruner [1]. Without a head-to-head, the novelty margin is hard to judge.\n\n[1] Haoxian Chen and Limin Wang. Rotpruner: Large language model pruning in rotated space, 2025.\n\nQUESTIONS:\n- Parameter Overhead Analysis. For 70 B models, the rotations add >7 B parameters (~10 %) before merging. Memory spikes during training might be a concern. Could low‑rank or block‑diagonal orthogonals work? An ablation on rank‑k Householder products would clarify the trade‑off. I noticed you discussed it in lines 232-234, but a more detailed elaboration regarding the training timing and memory costs would be appreciated.\n\n- Baseline breadth. Your work's high-level idea is quite similar to RotPruner. Would you please provide a detailed comparison of these two works? It would also be appreciated if the performance of RotPruner is included in the baseline.\n\n- Downstream fine-tuning compatibility. How would you recommend handling $R_1/R_1^\\top$ before LoRA/4-bit quant? A short empirical check (e.g., 1-epoch LoRA on 8 B) would reassure users.\n\n- Structured-mask awareness. In lines 245-248, you mentioned DenoiseRotator works remarkably well for 2:4 sparsity even without an explicit constraint. Have you tried explicitly learning rotations under a 2:4 mask, e.g., adding a differentiable surrogate for block occupancy? Results suggest good luck, but a principled variant might push performance further.\n\n- Clarify Fig. 2 (c) (d). Why those low-importance weights are unpruned? Did you mislabel?\n\nLIMITATIONS:\nPartially addressed.\nThe paper notes that semi-structured masks impose constraints not yet modelled, but omits:\n- Peak memory and time cost of rotation training on very large models.\n- Possible risks when further fine-tuning a model that still contains one orthogonal pair per block.\n\nAdding brief quantitative statements on these points would make the limitations section complete.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DenoiseRotator, a novel framework that enhances the robustness of neural network pruning for large language models (LLMs) by concentrating parameter importance prior to pruning. Unlike traditional pruning methods that focus solely on selecting which weights to remove, DenoiseRotator takes a fundamentally different approach by first redistributing parameter importance to make models inherently more amenable to pruning. The core innovation lies in applying learnable orthogonal transformations to weight matrices while minimizing the information entropy of normalized importance scores, effectively concentrating importance onto a smaller subset of parameters. The method leverages the computational invariance property of Transformer architectures and is designed to be model-agnostic, allowing seamless integration with existing pruning techniques such as Magnitude, SparseGPT, and Wanda.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The paper presents a genuinely novel perspective on pruning by focusing on importance redistribution rather than weight selection. The entropy-guided importance concentration approach is theoretically well-motivated and represents a significant departure from existing paradigms.\n\n2. The framework's design allows easy integration with existing pruning pipelines without requiring modifications to the underlying pruning algorithms, enhancing its practical utility.\n\n3. The paper cleverly employs QR decomposition reparameterization to train orthogonal matrices, which elegantly solves the challenge of maintaining orthogonality constraints during gradient-based optimization.\n\n**Weaknesses**\n1. Unlike quantization, pruning intuitively targets weights with low importance scores as the primary candidates for removal. Therefore, a transformation that induces a non-uniform distribution (where some weights are naturally pushed toward smaller magnitudes) would seem more appropriate for pruning. However, it is not immediately clear why enforcing a more uniform distribution via entropy minimization would yield better pruning performance. A more thorough explanation is needed to clarify how lower-entropy weight distributions, which typically concentrate probability mass on a subset of values, influence the effectiveness of pruning.\n\n2. While the paper presents QR decomposition as a solution for training orthogonal matrices, it does not clearly explain the specific training advantages this approach offers compared to alternative methods. For instance, SpinQuant[1] demonstrates that Cayley SGD can also effectively maintain orthogonality during backpropagation for rotation matrix optimization. The paper lacks a comparative analysis of why QR decomposition is preferable to other orthogonal training schemes such as Cayley transforms, making it unclear what unique benefits QR decomposition provides in the training context.\n\n3.  The core contribution of integrating QR decomposition to enforce orthogonality in rotation-based pruning appears as a straightforward extension of existing rotation methods rather than a fundamentally novel technique. Without deeper theoretical insights or comparative analysis demonstrating why QR decomposition offers unique advantages over other manifold optimization schemes, this addition risks being perceived as incremental.\n\n[1] \"SpinQuant: LLM quantization with learned rotations\", ICLR 2025\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nYes. The paper addressed limitations in the \"Limitations\" section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes **DenoiseRotator**, a novel framework to improve the robustness of pruning in large language models (LLMs) by concentrating parameter importance *prior* to pruning. Unlike previous methods that only select which weights to prune, DenoiseRotator applies **learnable orthogonal transformations** to redistribute importance scores, guided by entropy minimization. This makes the model more amenable to pruning. The method is compatible with existing pruning algorithms (e.g., SparseGPT, Wanda, Magnitude) and is extensively evaluated on multiple LLMs (LLaMA3, Qwen2.5, Mistral) across different sparsity types (50% unstructured and 2:4 semi-structured). Experiments show consistent improvements in perplexity and zero-shot accuracy with minimal inference overhead.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. **Clear writing and explanation**: The paper is clearly written, and the methodology—including Figure 1 and the entropy-based formulation—is intuitive and easy to follow.\n2. **Strong empirical results**: The method is extensively validated across different model scales, benchmarks, and pruning schemes. Visualization and ablations further support the effectiveness of the approach.\n3. **Original idea**: The use of orthogonal rotations—originally popularized in quantization—for pruning is novel. Applying entropy-guided transformations to improve robustness is both theoretically grounded and practically effective.\n\n---\n\n### Weaknesses\n1. **Limited practical efficiency in real deployment**: While effective under unstructured and semi-structured sparsity, the actual *inference speedup* under 2:4 patterns is limited. The method has not been shown to work under *structured pruning* (e.g., heads or channels), which is typically more impactful for latency reduction.\n2. **Additional training cost**: DenoiseRotator requires a dedicated optimization phase for learning the orthogonal transformations. Although the cost is manageable, it adds extra complexity, potentially limiting usability in low-resource or deployment-constrained settings.\n\nQUESTIONS:\n1. **Can this method be applied to structured pruning?** If yes, please provide results. If no, clarify why this is not feasible. This affects practical applicability on hardware with structured sparsity support.\n2. **What is the optimal number of entropy reduction steps?** As shown in Table 3, over-training might hurt generalization (e.g., zero-shot accuracy drops after 2000 steps). Would an adaptive early stopping criterion help?\n\nLIMITATIONS:\nYes, but it's not enough.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DenoiseRotator, a novel framework designed to enhance the robustness of LLM pruning. The core thesis is to shift the paradigm from simply selecting which weights to prune to first preparing the model to be more prunable. This is accomplished by using learnable orthogonal transformations to actively concentrate parameter importance into a smaller subset of weights. The optimization process is guided by minimizing the information entropy of normalized importance scores. DenoiseRotator is presented as a plug-and-play module that is compatible with existing pruning techniques like Wanda and SparseGPT. Extensive experiments on models like LLaMA3 and Qwen2.5 demonstrate significant improvements in perplexity and zero-shot accuracy under both unstructured and 2:4 semi-structured sparsity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.Novelty of the Core Idea: The idea of importance concentration as a preprocessing step is a significant and insightful departure from traditional pruning pipelines, offering a new and promising direction for model compression research.\n\n2.High Practicality and Compatibility: By decoupling the importance reshaping from the pruning algorithm itself, DenoiseRotator is designed as a versatile, plug-and-play module. This significantly lowers the barrier to adoption for researchers and practitioners already using established pruning workflows.\n\n3.Strong and Comprehensive Empirical Results: The authors provide compelling evidence across multiple model families (LLaMA3, Qwen2.5, Mistral) and sizes. The consistent and significant performance gains in both perplexity and zero-shot tasks, especially under the hardware-relevant 2:4 sparsity, strongly validate the method's effectiveness.\n\nWeaknesses:\n1.Indirect Mechanism for Structured Sparsity: The paper's justification for its strong performance on 2:4 sparsity is presented as a hypothesis—that the rotation acts like a \"random permutation\" improving alignment. This suggests the entropy-minimization objective is not explicitly tailored for structured pruning, and the precise mechanism behind its success in this crucial scenario is not fully understood.\n\n2.Potential Overstatement of \"Model-Agnostic\" Claim: While tested on several decoder-only models, the paper's claim of being \"model-agnostic\" may not fully extend to other architectures like encoder-decoder or Mixture-of-Experts (MoE) models without further investigation.\n\n3.Limited Analysis of Hyperparameter Sensitivity: The paper reports results using a fixed set of hyperparameters for training the rotations (e.g., 2000 steps). A more thorough analysis of how sensitive the final performance is to these choices would strengthen the paper's conclusions and improve its reproducibility.\n\nQUESTIONS:\n1. The results in Table 3 hint at a potential trade-off between perplexity and zero-shot accuracy as training progresses. Could you provide more insight into this behavior? Does it suggest a risk of overfitting to the calibration set, and is there an optimal stopping point for the rotation training?\n2. Could the authors elucidate whether solving eq(6) is also NP-H and how does this corresponds to eq(4)? Does other variance concentration techniques works, such as PCA on the centralized covariance of S?\n3. How does the importance of weight indices corresponds to the transformed scores?\n\nI am willing to adjust my ratings after seeing the authors' responses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}