OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition
Authors: Stephen Zhang
Vardan Papyan
Venue: ICLR 2025 Poster (2025)
Keywords: network pruning
low-rank
compression
sparsification
large language models
outlier features
Primary area: foundation or frontier models, including LLMs

Abstract: The recent paradigm shift to large-scale foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consumption and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit a steady drop in model performance as the compression increases. In this paper, we present a novel approach to compressing large transformers, coined OATS, that compresses the model weights by approximating each weight matrix as the sum of a sparse matrix and a low-rank matrix. Prior to the decomposition, the weights are first scaled by the second moment of their input embeddings, so as to ensure the preservation of  outlier features recently observed in large transformer models. Without retraining, OATS achieves state-of-the-art performance when compressing large language models, such as Llama-3 and Phi-3, and vision transformers, such as Google's ViT and DINOv2, by up to $60\\%$, all while speeding up the model's inference on a CPU by up to $1.37\times$ compared to prior pruning methods.

Published as a conference paper at ICLR 2025
OATS: OUTLIER-AWARD PRUNING THROUGH SPARSE AND Low RANK DECOMPOSITION
Stephen Zhang
Vardan Papyan
University of Toronto stephenn. zhang@mai i ctorontoo.ca
University of Toronto vardan, www.aa.ppapaaeerreeeeeooo.ee.eee wwp.yaa@tttooootoooccc
ABSTRACT
The recent paradigm shift to scaee foundation models has brought about a new era for deep learning that, while has found great success in practice, has also been plagued by prohibitively expensive costs in terms of high memory consump- tion and compute. To mitigate these issues, there has been a concerted effort in post-hoc neural network pruning techniques that do not require costly retraining. Despite the considerable progress being made, existing methods often exhibit steady drop in model performance as the compression increases. In this paper, we present novel approach to compressing large transformers, coined OATS, that compresses the model weights by approximating each weight matrix as the sum of sparse matrix and a low-rank matrix. Prior the decomposition, the weights are first scaled by the second moment of their input embeddings, so as to ensure the preservation of outlier features recently observed large transformer models. Without retraining, OATS achieves state-of-the-ar performance when compress- ing large language models, such as Llama-3 and Phi-3, and vision transformers, such Google's ViT and DINOv2, by up to 60%, all while speeding up the model's inference on a CPU by up to 1.37 x compared to prior pruning methods. Our code is available at: https//gttaabccomttehhen//0/111
INTRODUCTION
Large scale transformer-based models have found great success in a number of domains ranging from image classification (w et al. 2020), language modeling (Devlin al. 2019), and question answering (Brown et al. 2020). However, these models contain billions of parameters making them computationally expensive to train and deploy, which has lead to an increased demand for resource- saving techniques like model quantization (Dettmers et al. 2022 Egiazarian et al. (024), parameter efficient fine-tuning (Hu et al. 2022 Zhao et al. 2024b), and, most relevant to this work, neural
network pruning (Frantar & Alistarh 2023).
Pruning has been key focus for model compression since the early days of deep neural networks Mozer & Smolensky 1988 [eCun et al. 1989 Hassibi & Stork 1992). Various pruning techniques
have since emerged, introducing sparsity in the model parameters either before training (Lee et al.,
2019 Wang et al. 2020 Tanaka al.] [2000] de Jorge et al. (202)), during training (Zhu & Gupta
2018 Evei et al. 2020). or post-training (Benbaki et al. 2023). In the context of large foundation models, post-training pruning methods, particularly those requiring minimal (Xia al. 2022
Ma
et al. (223) or no re-training (Frantar & Alistarh 2023 Sun et al. 2024b; Ashkboos et al. 2024 Zhang al. 2024b) are preferred for their computational efficiency. These techniques, when com- pressing models by 50%, have demonstrated the ability to accelerate end-to-end CPU inference by up to 1.8x (Yin et al. 2024b) and GPU inference by up to 1.63x using structured N:M sparsity MMishra et al. 2021). highlighting their potential in reducing costs during deployment.
Despite the significant advancements in pruning techniques, was recently shown that current meth- ods suffer from a consistent degradation in model performance as compression levels increase (Yin) et al. 2024a). Moreover, although structured pruning offers greater potential for acceleration com- pared unstructured pruning, it often imposes much steeper trade-off in terms of model accuracy and effectiveness (Chen et al. 2022). These challenges underscore the need for more sophisticated pruning strategies that can achieve better performance as compression increases.

---

Published as a conference paper at ICLR 2025
1.1 CONTRIBUTIONS
To mitigate these issues, we introduce Outlier-Awann ware Pruning Through Sparse and Low Rank De- composition (OATS): novel retraining-free method for compressing large transformers that ap- proximates the model's weight matrices as a sum of sparse matrix and a low-rank matrix. In order to emphasize the outliers recently observed in large transformer models and preserve model perfor- mance (Kovaleva et al. 2021 Dettmers et al., 2022 Darcet et al. 2024 [un et al., (2024)), OATS first scales the weights by the second moment their corresponding input embeddings.
We evaluate OATS on recent large language models (LLMs)) Phi-3 (Abdin et al. (2024) and Llama- 3 (Dubey et al. 2024) and vision transformers Google's ViT (Wu et al.J (020) and DinoV2 (Oquab et al.) 2023) demonstrating that OATS achieves new state-of-the-art performance across a wide range commonly employed performance metrics. Furthermore, by combining structured pruning with unstructured pruning. OATS accelerates CPU inference across all levels of compression when compared to models that utilize just unstructured pruning.
To gain deeper understanding of the sparse and low-rank terms found by OATS, split the compressed vision transformers (Wu et al. 2020) into two separate models, sparse model and a low-rank model, and visualize their respective attention heat maps utilizing attention rollout (Abnar & Zuidema (020).. These reveal a complementary relationship between the two models, with each focusing on different key areas of the image, effectively segmenting into distinct regions.
2 THE OATS ALGORITHM
The key observation behind the OATS algorithm is that the weight matrices, W E Rd Xdos, in transformer model can be faithfully approximated as summation of sparse and low-rank matrix by solving the following optimization problem, commonly known Robust PCA (Chandrasekaran
al.
2009 2011 Candès et al. 201T):
min
W S D s.t. Rank L)) Sr, || k.
S, LE Xden
2.1
ALTERNATIING THRESHOLDING
To solve Equation T OATS leverages the alternating thresholding algorithms proposed by Zhou & Tao (201T). ). Netrapalli et al. (2014) and Bertsimas al. (2024) that iteratively alternates between solving for the low-rank term L, through singular-value thresholding, and for the sparse term S, through bard-thresholding Given matrix A E ggmxn, singular-value thresholding, also known as truncated SVD, is defined as:
TRUNCATEDSVD(A.r) U,E,,
where U,, E,, V correspond to the matrices formed by retaining only the top-r singular vectors and singular values from the full SVD of A. Hard-thresholding, which succeeds the singular-value thresholding step, is defined as:
HARDTHRESHOLD(AAK.
M A
where M E mm is a binary matrix with & non-zero entries coinciding with the & largest entries in magnitude in A. These steps are summarized in Algorithm [ on the right. To optimize memory usage, the low-rank term L is stored through its two low-rank components:
Algorithm ALTEERNATINGTTHRESHODDDN
1:
Inputs:
2:
Weight Matrix: W E Bdout
3:
Iterations: N
4:
Rank:
U, and E,
5: Nonzeros: k
6: Procedure:
2.2 ALTERNATIVE SPARSITY PATTERNS
7:
S
8:
for = 1 to N do
When performing the hard-threshold step, var- ious restrictions can be enforced on the spar- 10:
9:
L TRUNCATEDSVDD(W' S.))
S HARDTTIRESHOLD(WW L,k)
sity pattern of the sparse term of the decompo- 11:
end for
sition for enhanced performance or speed-up.
12:
return: S, L
The following important cases:
2

---

Published as a conference paper at ICLR 2025
Row-Wise Thresholding The hard-thresholding can be performed row-wise rather than layer- wise in which case M would be binary matrix with m non-zero entries coinciding with the k largest entries in magnitude in each row of A. Sun et al. (024b) have shown this leads to better performance.
The sarddthresholding can be applied at sparsity, where only the N largest entries by magnitude in every group of entries matrix are nonzero. Recently, NVIDIA's sparse tensor cores have been able to exploit such sparsity patterns for acceleration (Mishra et al. 2021).
N:M Sparsity
more granular level using N:M
2.3 INCORPORATING OUTLIER INFORMATION
The alternating thresholding on its own yields suboptimal results because the activations of large- scale transformers exhibit a small number of turge-mannitud features and altering these (for exam- ple, through the sparse and low-rank approximation) negatively impacts model performance (Koval- eva et al., [202]] Dettmers et al. [022] Darcet et al. 2024] [Snn et al., (024a). OATS takes inspiration from Wanda (Sun et al., 2024b) and computes a diagonal scaling matrix D Rd:. Xdin that captures
the second moment of the input activations
D V diag X XTX);
where X E Bxd... and B is the product of the batch size and sequence length. This diagonal matrix, containing large magnitudes for the outlier features, is used to amplify their significance in the reconstruction error of Equation [
leading to the following alternative optimization problem:
min
IWD L s.t. Rank(L) I
|| S
SE Mou X
The solution of the problem is given by:
S.L ALTERNATINGTTRREESOLDDN (WD.N,r,K k.)
which gives sparse plus low-rank approximation WD S L. OATS then applies the inverse transformation to reach the final compressed weight:
W compresseed (S L)D-1
where it leverages the fact that D diagonal so that it both preserves the sparsity pattern of S and is easy to invert. The original weight matrix is replaced with three matrices: the sparse matrix SD-1 and two matrices coinciding with the low-rank factorization of LD Aligned with Frantar & Alistarh (2023); Sun et al. (2024b), and Zhang et al. (2024b), the activations are calculated through calibration set that is propagated through the compressed layers.
2.4 OATS PARAMETERS
To determine the rank r and the number of nonzeros k. OATS takes in as input two hyperparameters: the compression rate, p (0, 1), and the rank ratio, E (0. 1). The compression rate coincides with the sparsity rate required by existing pruning algorithms and is defined as:
of nonzero parameters compressed layer
r(dout + dgn)
p =
# of parameters original layer
dout dig
The rank ratio represents the proportion of nonzero parameters that appear in the low-rank term:
# of parameters in low-rank term
r(dout + din). - ---p)dou- din
# of nonzero parameters in compressed layer
Given fixed compression rate p and rank ratio K, the two equations above can be solved to obtain the rank and nonzeros k::
dout den dout + din The complete OATS algorithm pseudocode can be found in Algorithm below.
r = k p)-
p).
k 1= - n²)
dout dan
(2)

---

Published as a conference paper at ICLR 2025
Algorithm 2 OATS
1:
Inputs:
2:
Layer Inputs Propagated Through Prior Compressed Layers: X & Bxde
3:
Layer Matrix: W E RRduutddi
4:
Compression Rate: p
5:
Rank Ratio:
6:
Iterations: N
7: Procedure:
8:
r
K (1 - D)
dout then
k + 101 - -k) - dout dinl
9:
+ V diag(X
10:
L.S & ALTERNATING TWRESHOLDDNG
W (L+S)
12:
returm: X 2+l X
EXPERIMENTS ON LARGE LANGUAGE MODELS
3.1 EXPERIMENT SETUP
Models and Tasks evaluate OATS on two state-of-the-art families of LLMs: Phi-3 (Ab- din et al. 2024) and Llama-3 (Dubey et al. 2024). To gauge the algorithm's performance under various model sizes, we select Phi-3 Mini, a 3.8B parameter model, Phi-3 Medium, a 14B parameter model, Ilama-3 8B, 8B parameter model, and Ilama-3 70B, 70B pa- rameter model. We utilize LM Harness developed by Gao et al. (2024) to evaluate five- shot performance on the Massive Multitask Language Understanding benchmark by Henddryck ett al. (2021), zero-shot performance on eight tasks, and language generation on WikiText-2.
Pruning Benchmarks As OATS does not require costly retraining after model compression, opt to benchmark it with three current state-of-the-art algorithms that similarly do require such overhead: SparseGPT by Frantar & Alistarh (2023). Wanda by Sun al. (2024b), and DSNoT by Zhang et al. (2024b). The parameters utilized for OATS are depicted in Table
Parameters
Phi-3
Llama-3
Iterations
80
80
Rank Ratio
25%
30%
Table 1: Hyperparameters utilized for OATS across
model families. Both parameters are further ablated
in Section 3.3
Calibration Data Remaining consistent with Frantar & Alistarh 2023) Sun et al. (2024b), and Zhang et al. (2024b), our calibration data consists of 128 sequences length 2048 sampled from the first shard of the C4 training set (Raffel et al. 2020). To ensure consistency, utilize the same calibration data for all pruning algorithms that we benchmark.
Layer- Layer-Wise Compression Rates We benchmark algorithm across wide range of compres- sion rates: (0.3, 0.4, 0.5, 0.6). For compression rates at or below 0.5, we compress all transformer blocks uniformly. At the higher compression rate of 0.6, we utilize Outlier Weighed Layerwise Sparsity Ratios (OWL. proposed by Jyi et al. (2024b) which were shown to lead to significant per- formance improvements higher compression rates. All linear layers in transformer block pruned uniformly to achieve the desired sparsity rate. We exclude pruning any linear layers that are present in the model head and embeddings which conforms with prior works by Frantar & Alistarh 2003
Sun et al. 2004b). and ZZang et al. (2024b).
Hardware Speedup We benchmark the CPU speedup of OATS over its competitors using the DeepSparse Inference Engine developed by Magial (2021). For GPU speed-up. we include structured N:M sparsity experiments where the rank ratio is varied to measure the trade-off between compression and performance.
DDNNoT experiments are with both SparseGPT and Wanda. We report the best results across the two. Further details are in Appendix A.14
4

---

Published as a conference paper at ICLR 2025
3.2 RESULTS
Five-shot MMLU 2 below, reports the MMLU accuracy of OATS relative to current state- of-the-art pruning algorithms. OATS is able outperform all prior methods, across all compression rates, with an increasing gap as the compression rate increases. Notably, at 50% compression, OATS surpasses previous pruning algorithms by margin of 5.42% on Phi-3 Mini, 2.52% on Phi-3 Medium, 2.86% on Llama-3 8B, and 2.03% on Llama-3 70B.
Phi-3
Llama-3
Compression
Method
Mini (3.8B)
Medium (14B)
8B
70B
0%
Dense
70.34
76.78
64.97
79.63
SparseGPT
68.31
74.12
64.25
78.28
Wanda
67.63
75.18
63.67
79.15
30%
DSNoT
68.02
75.13
63.72
79.00
OATS
68.84
76.15
65.22
78.47
SparseGPT
63.47
72.42
60.91
76.29
Wanda
64.15
73.34
60.33
77.16
40%
DSNoT
63.57
73.20
59.99
77.70
OATS
65.75
74.99
62.46
77.89
SparseGPT
53.22
67.63
53.60
72.47
Wanda
54.57
69.76
49.83
72.04
50%
DSNoT
54.28
68.65
49.20
72.76
OATS
59.99
72.28
56.46
74.79
Table 2: Comparison of average five-shot accuracies (%) on MMLU under different compression rates.
Zero-shot Tasks Table below, reports the zero-shot accuracy of OATS relative current state- of-the-art pruning algorithms averaged across the following eight commonly used tasks: PIQA (Bisk)
al. (2020): HellaSwag (Zellers al. 2019): Winogrande (Sakaguchi al. 2021): OpenBookQA
Mihaylov et al. 2018); RTE (Wang et al. 2018); BoolQ (Clark et al.) [219);: ARC-e and ARC-
(Clark al. 2018). Mirroring the trend observed in the five-shot results, the improvement of
OATS over prior pruning algorithms increases with compression, culminating in 2.05% advantage
over prior methods when compressing Phi-3 Mini 50% of its size.
Phi-3
Llama-3
Compression
Method
Mini (3.8B)
Medium (14B)
8B
70B
0%
Dense
71.99
74.27
69.79
75.27
SparseGPT
70.63
74.53
69.08
75.07
Wanda
70.66
74.05
68.63
75.19
30%
DSNoT
71.20
74.03
68.98
75.54
OATS
71.48
74.04
69.34
75.24
SparseGPT
69.18
74.40
67.58
74.63
Wanda
68.80
73.01
67.04
74.10
40%
DSNoT
69.08
72.90
66.65
74.29
OATS
70.04
74.46
68.68
74.88
SparseGPT
66.36
73.25
64.66
73.17
Wanda
65.03
70.96
63.27
72.85
50%
DSNoT 65.33
71.12
62.74
72.91
OATS
68.41
73.39
65.71
73.30
Comparison of average zero-shot accuracies (%) under different compression rates. Task- specific scores can be found in Appendix A.13
Table 3:

---

Published as a conference paper at ICLR 2025
Generation Task Table 4 below, reports the WikiText-2 perplexity of OATS relative to current state-of-the-art pruning algorithms. At 50% compression, OATS results in an 8.49% reduction in perplexity on the larger Phi-3 Medium model, and an even larger 8.99%, 9.04%. and 9.30% reduc- tion on Phi-3 Mini, Llama-3 8B, and Llama-3 70B respectively.
Phi-3
Llama-3
Compression
Method
Mini (3.8B)
Medium (14B)
8B
70B
0%
Dense
9.50
6.21
10.17
2.68
SparseGPT
11.19
7.48
9.71
3.24
Wanda
10.71
7.28
9.39
3.28
30%
DSNoT
10.51
7.11
9.36
3.27
OATS
10.27
6.85
9.59
3.07
SparseGPT
13.03
8.52
10.01
3.99
Wanda
12.59
8,49
9.74
4.08
40%
DSNoT
12.17
8.24
9.60
4.10
OATS
11.53
7.70
9.24
3.68
SparseGPT
16.80
9.89
11.95
5.27
Wanda
17.23
10.12
12.36
5.38
50%
DSNoT 16.68
9.96
12.41
5.58
OATS
15.18
9.05
10.87
4.78
Table 4: Comparison of perplexity (lower is better) WikiText-2 under different compression rates.
Performance Under High Compression Table 5 on the right, is the 5-shot MMLU accuracy of models compressed to higher compression rate of 60%, utilizing OWL ratios (Yin et al. 2024b). Following the trend above, OATS continues to outper- forms prior methods by a margin of 6.39% on Phi-3 Mini, 5.61% % Phi-3 Medium, and 4.98% on Llama-3 8B.
Phi-3
Method
Llama-3 8B
Mini
Medium
SparseGPT
46.20
57.91
39.48
Wanda
44.22
58.49
31.20
DSNoT 44.75
58.20
33.28
OATS 52.59
64.10
44.46
Table 5: MMLU accuracy (%) of models compressed by 60% using OWL ratios.
3.3 STUDIES AND HYPERPARAMETER EXPLORATION
We conduct ablation studies for OATS, on Phi-3 Mini at 40% compression rate with a rank ratio of 20%, quantify the impact the following design choices:
Scaling the weights by the second moment of the input activations, D, versus not scaling.
Pruning the weights per each output row in the matrix versus pruning layer-wise.
The results are shown Table bl
Ablation
MMLU (1)
Zero-shot (t)
Perplexity (L)
Layer-Wiss Wise
62,46
67.58
19.21
No Scaling
Row-Wit Wise
65.31
68.22
18.34
Layer- Wise
64.44
70.52
11.68
Scaling by D
Row-Wise
65.84
70.71
11.50
Table 6: Ablation results of OATS on Phi-3-Mine at 40% compression rate, with rank ratio of 20%. Scaling the weights by the second moment of the input activations and pruning row-wise significantly improves the performance of OATS.

---

Published as a conference paper at ICLR 2025
In addition to the ablations, perform additional experiments examine the impact of the rank ratio and the number of iterations on the performance OATS. Figure below shows the results.
70
Accuracy
60
O-shot
MMLU
50
40
0.1
0.2
0.3
0.4
0.5
0.6
0.7
1
20
40
60
80
100 120 140 160
Rank Ratio
Number of Iterations
Figure 1: The effect of varying the rank ratio and number of iterations on zero-shot and five-shot accuracy.
The experiments reveal that rank ratio between 25% to 30% leads to the best performance, with degradation occurring higher rank ratios. For the number of iterations, performance improves sharply in the first 20 iterations, before leveling off and saturating at around 80 iterations.
3.4
HARDWARE SPEEDUP
CPU Speedup We benchmark, using the DeepSparse engine by NNuralMagic (2021), the CPU throughput induced by OATS compared to models pruned with unstructured sparsity. We run end-to- end inference on compressed Phi-3 Medium 15B model for single token on an Intel Xeon Gold 6148 CPU @ 2.40GHz with 32 cores (for higher token-counts sec Appendix [A.6). The achieved throughput and speedup (over dense model) are shown Table By trading unstructured sparsity for structured sparsity through the low-rank terms, OATS achieves greater CPU speed-up compared to methods that rely solely on unstructured pruning. Notably, at 40% compression, OATS is 1.37 faster than unstructured pruning.
Compression
Method
Throughput
Speedup
0%
Dense
4.03
1.00x
Unstructured Pruning
4.32
1.07x
30%
OATS
5.58
1.38x
Unstructured Pruning OATS
5,08
1.26x
40%
6.86
1.73x
Unstructured Pruning
7.16
1.78x
50%
OATS
8.31
2.06x
Table 7: Comparison throughput (tokens/second) and speedup achieved through OATS and un- structured pruning methods relative to their dense counterparts.
N:M Performance We compare the performance of state-of-the-ard pruning algorithms, using 2:4 structured sparsity pattern, with the performance of OATS, using a 2:8 structured sparsity pattern on the sparse term. OATS employs sparser N:M pattern compensate for its low-rank term that remains dense. We experiment with rank ratios of (0.25, 0.3, 0.35, 0.4, 0.45, 0.5). Unlike previous pruning methods, where N:M structured sparsity enforces fixed compression rate of M OATS allows for flexible trade-off between compression and model performance by adjusting the rank ratio. Figure [ below, illustrates the compression ratio against the 5-shot MMLU accuracy for various compression algorithms.

---

Published as a conference paper at ICLR 2025
Phi-3 Mini
Llama-3 3 8B
50
Avg. Accuracy
OATS SparseGPT
MMLU
40
Wanda
30
DSNoT
20
50
54
58
61
64
66
50
61 Compression Rate
54
58
64 66
Compression Rate
Experiments evaluating OATS with 2:8 structured sparsity on the sparse terms against 2:4 sparsity of state-of-the pruning algorithms. The rank ratio for OATS is varied capture the performance across different compression rates.
Figure 2:
Despite having a sparser structured sparsity pattern of 2:8, OATS is able to recover the model perfor- mance through the presence of its low-rank term. Specifically,
compression rate of 50%. OATS able to outperform all prior state-of-the-art by 6.34% on Phi-3 Mini. In the case of Llama-3 8B, OATS only surpasses previous methods by 9.2% 50% compression, but also outperforms them by 2.86% at an even higher compression rate of 54%.
EXPERIMENTS ON VISION TRANSFORMERS
We run experiments on Google's s ViT-Base (Wu et al. 2020), an 86.6M parameter model trained
in supervised manner ImageNet-21k (Ridnik et al. (02T] and fine-tuned ImageNet 2012 (Russakovsky et al. 2015), and Dino V2-Giat (Oquab et al. 2023), 1.14B parameter model that was trained through seff-supervised learning.
We benchmark OATS against the same three pruning algorithms: SparseGPT, Wanda, and DSNoT, by evaluating top-1 accuracy on the validation set of ImageNet (Russakovsky al. (2015). A subset of 2048 images from the training set of ImageNet is used for calibration and is maintained consis- tent across all pruning experiments. All OATS experiments use rank ratio of x=20% and N-80 iterations. We exclude from compression the embedding and the classifier layers.
The results are shown in Table 8 below. Compared LLMs, vision transformers show greater resilience to pruning, with DinoV2 experiencing only 0.41% drop in top-1 accuracy when com- pressed by 50% using OATS.
Compression
Method
ViT-Base
Dino V2-Giant
0%
Dense
80.33
86.55
SparseGPT
80.21
86.46
Wanda
80.28
86.47
30%
DSNoT
80.
86.46
OATS
80.15
86.52
SparseGPT
79.58
86.39
Wanda
79.34
86.32
40%
DSNoT
79.46
86.37
OATS
79.86
86.46
SparseGPT
78.44
86.04
Wanda
76.19
85.81
50%
DSNoT 76.90
85.93
OATS
78.77
86.14
Table 8: ImageNet validation accuracy (%).
8

---

Published as a conference paper at ICLR 2025
5 VISUALIZING AND INTERPRETING THE DECOMPOSITION
To develop a better understanding of how the sparse and low rank components individually con- tribute to the flow of information through the model, we compute and visualize the attention rollout Abbar & Zuidema, 2020) of the compressed vision transformers when:
All low-rank terms are set to zero and inputs are propagated through only the sparse terms.
All sparse terms are set to zero and inputs are propagated through only the low-rank terms.
Figure 3
below provides a visualization of how the information would flow through standard transformer block for both settings.
FEN
We Coegresial
Activaicn
Tlasatio
Anwadze
W. Coegresse
W: Canpreseed
%
Muasi Head
Mansllld
Iapab
(a) Sparse terms only
(b) Low-rank terms only
Figure 3: A visualization of how the attention rollout is computed to isolate the contribution of the sparse terms versus low-rank terms given by the OATS algorithm.
Figure depicts the attention rollout various images in the Microsoft COCO dataset (L.in al,, 2014] passed to VIT-B that was compressed by 50%, with rank ratio of 20%.
Original:
Sparse:
Low-Rank:
Figure 4: Attention rollout visualization applied to various images the Microsoft COCO dataset.
The rollout visualizations show that the sparse and low-rank terms capture distinct areas of the image, effectively segmenting it. A careful analysis reveals three distinct partitioning patterns. The first. which is also commonly exhibited in the classical setting (Candès et al.) 2011). is when one component (commonly sparse) captures the subject(s), while the other component (commonly low- rank) captures the background. The second is when both components focus different parts of the

---

Published as a conference paper at ICLR 2025
same subject, each capturing distinct features. The third behavior arises when the image contains multiple subjects, with each component isolating a different subject. While these patterns provide initial insights into how the components process visual information, further investigation is needed to fully understand the mechanisms driving these behaviors.
6 RELATED WORKS
Connection with Wanda OATS utilizes the same outlier scaling as the the one that is employed by Wanda (Sun al. 2024b). In fact, Wanda be seen special case of OATS when the rank ratio k=0. Indeed. in such case, according to Equation [2] the low-rank term would become the zero matrix and OATS would perform a single hard thresholding step that equivalent to the pruning step described by Wanda: W compressed = HAARDHHRESHOLDWWW (WDLL D
Sparse and Low-Rank Approximation in Transformers The emergence of sparse and low-rank structures in transformers has recently become an area of both theoretical and practical interest. On the theoretical front, Zhao et al. (2024c) showed that the logits of LLMs trained utilizing next token prediction converge to low rank and sparse structure. On the practical front, Scatterbrain proposed by Chen et al.] (2021a) shows that it is possible to approximate the entire attention mechanism with single sparse and low rank decomposition. Pruming-wise LoRAP by Li et al. (2024) performs structured pruning on the feed-forward linear layers and apply a low-rank decomposition to the attention matrices using scaling technique similar OATS.
Structured Pruning and Low-Rank Adaptation Recent works, such as LoSparse (Li et al., 2023]. LoRAPrune (Zhang et al., 2024a), and APT (Zhao et al.) 2024a), propose variations of ap- plying structured pruning on the weights while incorporating low-rank adapter that is trained via gradient descent. These are markedly different than OATS, which does not employ any fine-tuning with low-rank adapters, nor does it perform structured pruning (but rather sparse plus low-rank decomposition which can be thought of as a combination of structured and unstructured pruning).
Robust PCA Algorithms The search for Robust PCA algorithms has been a key area of interest since the inception of the problem. Examples of other approaches include applying a convex relax- ation, where the sparsity and low-rank constraints are replaced by L and nuclear norm surrogates (hoou et al. 2010). or parameterizing the low-rank matrix as L = UV and applying gradient descent on U and (Yi al. 2016 Tong et al. 2021). While OATS utilizes the alternating thresh- olding approach for its simplicity, future work might want to investigate the use of other algorithms.
Pruning and Interpretability An active area of research is understanding what pruming is pruning and how it impacts model performance. PPaaanii| (2020) show that pruning has disproportionate negative effect on underrepresented classes. In similar vein, Yin et al. al. (202a)) showed that pruning LLMs can irreversibly harm model performance on tasks that are more challenging. We postulate that the low-rank term present in OATS might be able to mitigate the negative impacts of pruning. Indeed, Tables 2 and show that the gap between OATS and prior methods is larger at higher com- pression, suggesting that the low-rank term plays a critical role in mitigating the loss in performance.
7 CONCLUSION
We have introduced OATS, an algorithm that without any re-training, compresses the model's weight matrices through a sparse and low-rank decomposition. Taking inspiration from prior works on the emergence of outlier features, OATS first scales the weights by the second moment of their input embeddings prior to applying an alternating thresholding algorithm. comprehensive evaluation shows that OATS is able to consistently outperform prior state-of-the-art on various performance metrics across multiple compression rates, models, and modalities, while also improving on CPU speed-up. Beyond just model compression, our visualizations models exhibit sparse and low-rank structures that capture different segments of the image. This work is the first to reveal the potential of sparse and low-rank decompositions for large- scale trans- formers, setting the stage for future innovations that can hamness this structure to improve model efficiency, performance, and interpretability.
vision transformers indicate that
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC). This research was supported in part by the Province of Ontario, the Government of Canada through CIFAR, and industry sponsors of the Vector Institute (www. vectorinstitute ai ht/partnerships/ htt.p/pp/ccureee--aatteteee port provided by Compute Ontario (oa (ht::h::ttttttthothooo www. wwwp://toottooooooooooooooo
This research was also enabled in part by sup- ca and the Digital Re-
search Alliance of Canada (https::/all
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Ben- haim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Yen-Chun Chen, Yi- Ling Chen, Parul Chopra, Xiyang Dai, Allie Del Giorno, Gustavo de Rosa, Matthew Dixon, Ronen Eldan, Victor Fragoso, Dan Iter, Mei Gao, Min Gao, Jianfeng Gao, Amit Garg, Abhishek Goswami, Suriya Gunasekar, Emman Haider, Junheng Hao, Russell Hewett, Jamie Huynh, Mojan Javaheripi, Xin Jin, Piero Kauffmann, Nikos Karampatziakis, Dongwoo Kim, Mahoud Khademi, Lev Kurilenko, James R. Lee, Yin Tat Lee, Yuanzhi Li, Yunsheng Li, Chen Liang, Lars Liden, Ce Liu, Mengehen Liu, Weishung Liu, Eric Lin, Zeqi Lin, Chong Luo, Piyush Madan, Matt Mazzola, Arindam Mitra, Hardik Modi, Anh Nguyen. Brandon Norick, Barun Patra, Daniel Perez-Becker, Thomas Portet, Reid Pryzant, Heyang Qin, Marko Radmilac, Corby Rosset, Sam- budha Roy, Olatunji Ruwase, Olli Saarikivi, Amin Saied, Adil Salim, Michael Santacroce, Shi- tal Shah, Ning Shang, Hiteshi Sharma, Swadheen Shukla, Xia Song, Masahiro Tanaka, Andrea Tupini, Xin Wang, Lijuan Wang, Chunyu Wang. Yu Wang, Rachel Ward, Guanhua Wang, Philipp Witte, Haiping Wu, Michael Wyatt, Bin Xiao, Can Xu, Jiahang Xu, Weijian Xu, Sonali Yadav, Fan Yang, Jianwei Yang. Ziyi Yang. Yifan Yang, Donghan Yu, Lu Yuan, Chengruidong Zhang, Cyril Zhang, Jianwen Zhang. Li Lyna Zhang, Yi Zhang, Yue Zhang, Yunan Zhang, and Xiren Zhou, Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL https: /wwpivaiivvv./v////vv/eeaaeavwvv org/ abs /204. 14219
Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Dan Ju- rafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th An- mual Meeting of the Association for Computational Linguistics, pp. 4190-4197, Online, July URL
2020. Association for Computational Linguistics. doi: 101186633//22220almmmmm htt.p:://ccllantioooo...oo//////
385
Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. SliceGPT: Compress large language models by deleting rows and columns.
In The Twelfth International Conference on Learning Representations, 2024. URL https:
Riade Benbaki, Wenyu Chen, Xiang Meng, Hussein Hazimeh, Natalia Ponomareva, Zhe Zhao, and
Rahul Mazumder. Fast as CHITA: Neural network pruning with combinatorial optimization. In Proceedings of the 40th International Conference on Machine Learning, pp. 2031-2049) 2023.
Dimitris Bertsimas, Ryan Cory-Wright and Nicholas A. G. Johnson. Sparse plus low rank matrix decomposition: discrete optimization approach. J. Mach. Learn. Res., 24(1), mar 2024. ISSN 1532-4435.
Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. Proceedings of the AAAI Conference on Artificial Intelligence. 34005):4432-7439 Apr. 2020. doi: 10.1609/aaai, w33t:/aa....34.....6. v34i05.6239. URL https://
ojs a wwrg//nddes http://inex.pp///aaaa.. art cle/iie 6239
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec
11

---

Published as a conference paper at ICLR 2025
Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc.,
2020. URL. https: /tttps://rroceeee/e ngs. www.rnnnn..nnnnueeeee seeurrsss.. wwtp:pcr_.appppeeeeeeeeee w2w.ee424.eeaaa...aapem watprrffleepppprreeaee/
File/1 45
Emmanuel J. Candès, Xiaodong Li, Ma, and John Wright. Robust principal component analysis?
J. ACM, 58(3), jun 2011. ISSN 0004-5411. doi: 091144//1970392.1970399.. URL https:
//doi. olgoi.org/00. 4 01255/97739922 1970395
Venkat Chandrasekaran, Sujay Sanghavi, Pablo A. Parrilo, and Alan S. Willsky. Sparse and
low-rank matrix decompositions. IFAC Proceedings Volumex, 12(10):1493-149) 2009. ISSN 1474-6670. doi:
URL /tnttps://wwat 15th IFAC
com/ ora.aacccc.....aaaaa...aaaaaaaa.
Symposium on System Identification.
Venkat Chandrasekaran, Sujay Sanghavi, Pablo A. Parrilo, and Alan S. Willsky. Rank-sparsity incoherence for matrix decomposition. SIAM Joumnal on Optimization, 21(2):572-566, 2011.
doi: 10.1137/090761799.
Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric Xing. Vision transformer slimming: Multi multi-dimension searching in continuous optimization space. 2022.
Beidi Chen, Tri Dao. Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scat- terbrain: Unifying sparse and low-rank attention.
M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural In- formation Processing Systems, volume 34, pp. 17413-17426. Curran Associates, Inc.,
2021a. URL https::/prrcced ngs nwwrins wttpppss.pccppeeeee wwtppper..ppp..e.pp fi /esss//ppp///2////
Ei Elee//2855e550007777444466666653
Tianlong Chen, Yu Cheng, Zhe Gan, Lu Yuan, Lei Zhang, and Zhangyang Wang. Chasing sparsity in vision transformers: An end-to-end exploration. In A. Beygelzimer, Y. Dauphin, Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b. URL net/ wwt./eettufnnoeeee worumoiriiioooooo -oMLLLL-TTTTTCCCC
Tianlong Chen, Xuxi Chen, Xiaolong Ma, Yanzhi Wang, and Zhangyang Wang. Coarsening the granularity: Towards structurally sparse lottery tickets. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.). Proceedings of the 39th Inter- national Conference on Machine Learning, volume 162 of Proceedings of Machine Learning mml.
Research, pp. 9025-3039, PMLR, 17-23 Jul 2022. URL
press/v162. wwwp:/e.ennnaac.eme
Christopher Clark, Kenton Lee, Ming-Wos Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Techmologies, Volume (Long and Short Pupers), pp. 2924-2936. Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ URL wwg.//............ Marg/NN19-10 -100
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the A12 reasoning challenge. CoRR, abs/1803.05457, 2018.
owg. 1803. 05457
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. In The Twelfth International Conference on Learning Representations, 2024. URL
httts::/
net/ www.:neeeetteeneeeeeee /tt/:/ommmm2dd2n3iiiii
Pau de Jorge, Amartya Sanyal, Harkirat Behl, Philip Torr, Grégory Rogez, and Puneet K. Dokania. Progressive skeletonization: Trimming more fat from network at initialization. In International Conference on Learning Representations, 2021. htllhttps://ppezeeeee net//orum?? forum?
net/
id= 9GFFOUUPP
12

---

Published as a conference paper at ICLR 2025
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. GPT3.int80): 8-bit matrix multiplication for transformers at scale. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: fttt:s://cperreviiie
net / forum? d=dxi GWq3oxaD
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Com- orgg/
putational Linguistics. doi: 10.18653/11/N1-- 1423. URL
1199-1223
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshomn, Aobo Yang. Archi Mitra. Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic. Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Rearyla-Yeary, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy. Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang, Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy. Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaelle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld. Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg. Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti. Bran- don Spence, Brani Stojkovic, Brian Gamido. Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai,
13

---

Published as a conference paper at ICLR 2025
Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Fleonora Presani, Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Elena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean- Baantttttste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang. Kai Wu, Kam Hou U, Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang. Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang. Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelli, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong, Ning Zhang. Norman Cheng. Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez, Raf Ayub, Raghotham Murthy, Raghu Nayani. Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun I. Lindsay, Shaun Lind- say, Sheng Feng. Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shugiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury. Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailesou, Vladimir Ivanov, Wei Li. Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang. Xiaojian Wu, Xiaolan Wang. Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary Vito,, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024. URL https:
abs/ 2407 21783
Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, and Dan Al- istarh. Extreme compression of large language models via additive quantization. In Forty-first International Conference on Machine Learning, 2024. URL ptstps:://penreview..co net/ forum? Forum?1d=5mCa 5mmaa IRTMO.
Utku Evei, Trevor Gale, Jacob Menick, Pablo Samuel Castro, and Erich Elsen. Rigging the lottery: Making all tickets winners. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. pp. DIess/v119/evci20a. html
PMLR, 13-18 Jul 2020. URL https::/proceeeee .mlr.....innnnm.................
Elias Frantar and Dan Alistarh. SparseGPT: Massive language models can be accurately pruned in one-shot. arXiv preprint arXiv:2311.00722 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen-
14

---

Published as a conference paper at ICLR 2025
nighoff, Chris Ociepa, Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL
12608602
Jacob Gil.
github. Vision transformer explainability.
attps://jacobgil. eappaaaaorr
attention- roniiot 2021. Accessed: 2024-09-18.
Babak Hassibi and David Stork. Second order derivatives for network pruning: Op- timal brain surgeon. In S. Hanson, J. Cowan, and C. Giles (eds.), Advances in Neurul Information Processing Systems, volume 5. Morgan-Kaufmann, 1992. URL
https::/pxrcceeinngg newrips. htwp://cc.ceiioooooooo..o...ooo. cc/pppr files/ FEles/paper//992/E1I 339edce986a3332333333576-6----23
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. In International Confer- ence on Learning Representations, 2021. URL
net wttp:/wueeeeotmmmei..e
d7KB jmI 3GmQ.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022. URL https: openrevieww www. forum?
id=nZeVKeeFYF3
P.J. Huber. Robust statistics. Wiley New York, 1981.
Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers, and Anna Rumshisky. BERT busters: Out- lier dimensions that disrupt transformers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 202 pp. 3392-3405, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 v1/2021. findings-acl..300 URL htt.:o.....ttttss////////aa/ /tt////r../////llatttoooooooooo0 findings acl.
300
Yann LeCun, John Denker, and Sara Solla. Optimal brain damage. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 2. Morgan-Kaufmann, 1989. URL
wgw. nwwrips httpr//rpssppppppppaaeeee/e////p //lls//ppper1999// File fllp://282222020000333338-137773
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip Torr Snip: Single-shot network pruning based on connection sensitivity. In International Conference on Learning Representations, 2019. URL
htttps://openreviewy net http://wwww...ee/trte/eaaaaaaaaa
Guangyan Li, Yongqiang Tang, and Wensheng Zhang. LoRAP: Transformer sub-layers deserve
differentiated structured compression for large language models. Forty-first International Conference on Machine Learning, 2024. URL https:
hattps://openret /ttp:////ee/eeeieeeeee net forum? iid=
50155nc50X
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoSparse: Structured compression of large language models based on low-rank and sparse ap- proximation. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference Ma- chine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 20336-203500 URL pttt:s:/pproceeiiieg mwr..rr.......mmmmmm.. wwwsss/v22/11002222
PMLR, 23-29 Jul 2023.
html
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars (eds.), Computer Vision ECCV 2014, pp- 740-755, Cham, 2014. Springer International Publishing. ISBN 978-3-319-10602-11
15

---

Published as a conference paper at ICLR 2025
Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-pruner: On the structural pruning of large language models. In Shirty-seventh Conference on Neural Information Processing Systems, 2023. URL http:/tss//eeeeeeeeeeentt//////s
EBA E9WEX
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? new dataset for open book question answering. In EMNLP, 2018.
Asit K. Mishra, Jorge Albericio Latorre, Jeff Pool, Darko Stosic, Dusan Stosic, Ganesh Venkatesh, Chong Yu, and Paulius Micikevicius. Accelerating sparse deep neural net- works. ArXiv, abs/2104.08378, 2021. URL http::///ap..semmmnicccchhhoooe org/
CorpusID:233296228
Mohammad Mozaffari and Maryam Mehri Dehnavi. Slim: One-shot quantized sparse plus low-rank approximation of Ilms, 2024. URL htttp:://aaii..ooott htaegg/bbs/244 /2400. 09615
Mohammad Mozaffari, Amir Yazdanbakhsh, Zhao Zhang, and Maryam Mehri Dehnavi. Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of Ilms, 2024. URL https:// arxiv.orgi abs 2405. 11245....55555
Michael C Mozer and Paul Smolensky, Skeletonization: A technique for trim- ming the fat from network via relevance assessment. In D. Touretzky (ed.), Advances in Neural Information Processing Systems, volume 1. Morgan-Kaufmang Kaufmann, 1988. URL https::/proceedd.nngg www.://eeedddno................. wwtpccprppppp.eeeeeeeee fi (tiles/pppper/9833 hatpa//7777771ccccPaae.rrrp..
Paper.
Praneeth Netrapalli, Niranjan U N. Sujay Sanghavi, Animashree Anandkumar, and Prateek Jain. Non-convex robust pea.
Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger (eds.). Advances in Newral Information Processing Systems, volume 27. Curran wwwpiiinnnnneeuuureeeeeeee neurips wtwppedepp.pppppapppp.pp..
Associates, Inc., 2014. URL
http://ssseeeirirooopppeeeeer wwtpcppppppppeeeeeeeeee files/
NeuralMagic. Deepsparse: A cpu runtime for sparse inference of neural networks. https://
github. htc.uu..ccmm/// aeteurrlmmgic/deeeeeee
2021. Accessed: 2024-09-19.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo. Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Michela Paganini. Prune responsibly. ArXiv, abs/2009.09936 2020.
URL https://api.
Qwen Team. Qwen2.5; A party of foundation models, September 2024. URLpttts:://qwenn.
github. http://///gggge......o.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
html
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses, 2021.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy. Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJV), 1 115(3):211225 2015. doi: 10.1007/sl 1263-015-0816-)
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: ad- versarial winograd schema challenge at scale. Commun. ACM. 64(9)99-106, aug 2021. ISSN 0001-0782. doi: 10.1145/3474381 URL htt.hhttpp//..o..ooo 0.t.///...1444441133331
16

---

Published as a conference paper at ICLR 2025
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. In First Conference on Language Modeling, 2024a. URL htttps://openreviiett
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A simple and effective pruning approach for large language models. In The Twelfth International Conference on Learning Representations, 2024b.
net forum?
Hidenori Tanaka, Daniel Kunin, Daniel L Yamins, and Surya Ganguli. Pruning neural networks without any data by iteratively conserving synaptic flow. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural In- formation Processing Systems, volume 33, pp. 6377-6389. Curran Associates, Inc., neurips........ppoooo.o....nisie ca wacp/papr__ffeeeeeeeeeeee fi files//aper//220// 7
2020. URL 22 Paperr.apppppppp.
Tian Tong. Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Mach. Learn. Rex., 22(1), jan 2021. ISSN 1532-4435.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/11/118-5333 Bttps://aclantholoogy /wtoo/////-///// -
URL
Chaoqi Wang, Guodong Zhang, and Roger Grosse. Picking winning tickets before training by preserving gradient flow. In International Conference Learning Representations, 2020. URL
htttps:/cpeeneeviee net/ / forum?ideeiooo....... Sd-SummAskSK
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick VOn Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-ar natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Confer- ence Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38- 45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/ emnlp- demo0.... URL httlttttt//acaaataaaaa
org. 2020. emn1p-demos.6
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda. Visual transformers: Token-based image representation and processing for computer vision, 2020.
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. In Association for Computational Linguistics (ACL), 2022.
Xinyang Yi, Dohyung Park, Yudong Chen, and Constantine Caramanis. Fast algorithms for ro- bust pea via gradient descent. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp. 4159-4167, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Lu Yin, Shiwei Liu, Ajay Kumar Jaiswal, Souvik Kundu, and Zhangyang Wang. Junk DNA hypothesis: A task-centric angle of LLM pre-trained weights through sparsity, 2024a. URL
https://openreviie wwt forum 1ddEmUmpfrXN
Lu Yin, You Wu, Zhenyu Zhang, Cheng-Yu Yu Hsieh, Yaqing Wang, Yiling Jia, Gen Li, AJAY KUMAR JAISWAL, Mykola Pechenizkiy, Yi Liang, Michael Bendersky, Zhangyang Wang, and Shiwei Liu. Outlier weighed layerwise sparsity (OWL): A missing secret sauce for pruning LLMs to
high sparsity. In Forty-first International Conference on Machine Learning, 2024b. URL hctps:
//openreview net//forum forum? d=mmnmm3aammmmmmm 2P 6w
Fang Yu, Kun Huang, Meng Wang, Yuan Cheng, Wei Chu, and Li Cui. Width and depth pruning for vision transformers. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3): 3143-3151, Jun 2022a. doi: 20.1609/aaii 20222. URL https:: :/oa:://ojs. wtw.oo..aaaaa.r.............o iinexx Leidex.ppp/AAATI
17

---

Published as a conference paper at ICLR 2025
Lu Yu and Wei Xiang. X-pruner: explainable pruning for vision transformers. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24355 24363, 2023. doi: 10.1109/CVPR52722.222 202.3222322....0233..
Shixing Yu, Tianlong Chen, Jiayi Shen, Huan Yuan, Jianchao Tan, Sen Yang. Ji Liu, and Zhangyang Wang. Unified visual transformer compression. In International Conference on Learning Repre- sentations, 2022b. URL httt.htts://ooto /otppr/vv/peeevvviiiee ew wtw.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. On compressing deep models by low rank and sparse decomposition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), July 2017.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Compulational Linguisties, 2019.
Mingyang Zhang, Hao Chen, Chunhua Shen, Zhen Yang, Linlin Ou, Xinyi Yu, and Bohan Zhuang. LoRAPrune: Pruning meets low-rank parameter-efffcient fine-tuning, 2024a. URL https: openreview wwt.:eet//oeoeee 2i-oummddd-KK
Yuxin Zhang, Lirui Zhao, Mingbao Lin, Sun Yunyun, Yiwu Yao, Xingjia Han, Jared Tanner, Shiwei Liu, and Rongrong Ji. Dynamic sparse training: Training-free fine-tuning for sparse LLMs. In The Tweifth International Conference on Learning Representations, 2024b. URL https:
//openzeview.. wwt.
Bowen Zhao, Hannaneh Hajishirzi, and Qingqing Cao. APT: Adaptive pruning and tuning pretrained
language models for efficient training and inference. In Forty-first International Conference on Machine Learning, 2024a. URL /ttttss::/ppeeeeeeeeee
net/ wwtp:/eeteeo/ototooee www.roorruuudoooo
X15GJG
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient LLM training by gradient low-rank projection. In 5th Workshop on practical for limited//ow resource settings, 2024b. URL WRLhhttss://peeettt http::///ppeeeiieeeee htt.ireeeieeweeee/e
Forum? 2erum?idd=224Py02225
Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. In First Conference on Lan- guage Modeling, 2024c. URL hitps: http//ppeeevvieeeee / httppp/wwwweeeeeeeeeoeooorrraaaa
Tianyi Zhou and Dacheng Tao. Godec: randomized low-rank & sparse matrix decomposition in noisy case. In Proceedings of the 28th International Conference on International Confer- ence on Machine Learning, ICML'11, pp. 33 40, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Zihan Zhou, Xiaodong Li, John Wright, Emmanuel Candès, and Yi Ma. Stable principal component pursuit. In 2010 IEEE International Symposium Information Theory, ISIT 2010 Proceed- ings, IEEE International Symposium on Information Theory- Proceedings, pp. 1518-1522, 2010. ISBN 97814244488 1781444469604.
doi: 10. 109/1STT:2010.555 2010 IEEE International Symposium on Information Theory, ISIT 2010; Conference date: 13-06-2010 Through 18-06-2010.
Michael H. Zhu and Suyog Gupta. To prune, or not prune: Exploring the efficacy of praning for model compression, 2018. URL httpap/pee/pppeerrrreeeeeeereeea
Mingjian Zhu, Kai Han, Yehui Tang, and Yunhe Wang. Visual transformer pruning. CoRR, 00s/2104.08500, 2021. URRp::tts://aaae
aat//ab///0 /abs//14..... 08500
A APPENDIX
A.1 ADDITIONAL RELATED WORKS
Sparse and Low-Rank Decomposition for Pruning et et al. (2017) introduced a method for sparse and low-rank decomposition of CNNs, including AlexNet and GoogLeNet, by solving the
18

---

Published as a conference paper at ICLR 2025
following optimization problem:
min S. Xdii
|
(S L)X s.t. | - -(S -S+ D)ll S Rank(L) S 1 S k
where = =WX, In contrast, OATS employs a different approach, solving:
min S. Kden
I S 1 s.t. Rank(L) rr ||||| S k.
key distinction between these methods lies their objectives: the former directly minimizes reconstruction eгror, while OATS adopts a simpler formulation. One might question why not follow the approach of minimizing reconstruction error. As noted in DSNoT (Zhang et al. 2024b), pruning methods that prioritize minimizing reconstruction emor can degrade model performance in large transformers, particularly in the presence of outlier features. Their findings highlight the importance of avoiding pruning weights within outlier channels. Since feature outliers a phenomenon unique to large transformer models (Dettmers et al. (022), this issue would not have been relevant to the work of Yu et al. (017), which predates the transformer era.
Pruning Algorithms for Vision Transformers There are number of pruning approaches that have been specifically catered towards pruning vision transformers (Zhu al.] [02T]] Chen et al.] 2021b Chavan et al.) 2022: Yu et al.) 2022a bj Yu & Xiang 2023). However, much of the pruning literature developed on vision transformers involved models of much smaller scale than the large language models employed in this study, almost all of the prominent pruning algorithms require some form of training on the model parameters. As OATS was designed to require no training, OATS and the aforementioned pruning algorithms would not be comparable.
Low-Rank Adapters during Pre-Training In Mozaffari et al. (2024). the authors propose SLOPE, a novel method for accelerating the pre-training phase LLMS by incorporating N:M sparsity and adding low-rank components to the model weights to enhance model capacity. Similar to OATS, SLOPE leads a sparse plus low-rank structure in the model's weight matrices, however, the low-rank terms are introduced during the final phase of pre-training and are actively trained on the model loss function. In contrast, OATS is designed as a lightweight method to accelcrate in- ference. OATS does not require any training or fine-tuning, but instead approximates pre-trained weight matrices by solving the Robust PCA problem.
independent and concurrent work with OATS proposes SLIM (Mozaffari & Dehnavi 2024), novel pipeline that combines pruning and quantiza- tion. To restore lost performance from compression, SLIM derives low-rank term using singular- value thresholding and adopts scaling technique akin to OATS. However, instead of the L norm, SLIM utilizes the average absolute value across the batch and sequence dimensions. As a further deviation from OATS, SLIM is also not performing alternating thresholding algorithm. Instead, they perform a single quantization and pruning step to initialize the quantized and sparse terms, followed by single singular value thresholding step establish the low-rank term.
Quantized Sparse Low-Rank Approximation
A.2 TIME COMPLEXITY WALL-CLOCK TIME OATS
The time complexity for OATS is O(LNa) where L is the number of transformer blocks, N is number of iterations, and
a = max dW d. r
W
where the max taken over the weight matrices, W E gd xdW transformer block and w the rank of the low-rank term for that weight matrix. The value represents the time complexity needed to perform the singular value thresholding in OATS.
Table below reports the wall-clock time needed perform single iteration of the alternating threshold algorithm for a single transformer block for the different models that were compressed. All experiments utilized a single NVIDIA A40 with 48GB of GPU memory.
19

---

Published as a conference paper at ICLR 2025
Phi-3
Ilama-3
Mini (3.8B)
Medium (14B)
8B
70B
8.85
26.02
17.10
152.80
Table Wall-clock time (in seconds) needed to perform a single iteration of the alternating projec- tion algorithm in OATS.
While OATS does require more wall-clock time than prior pruning algorithms, in practice, model compression would only need be performed once before deployment. This trade-off is therefore worthwhile given the substantial performance improvements, particularly on more challenging tasks like MMLU (see Table 2) Furthermore, like prior pruning algorithms, compressing the layers within a single transformer block can be done in parallel. For example, the time needed per transformer block of Llama-3 70B can be reduced 71.10 seconds compressing in parallel across four NVIDIA A40 GPUs.
The total wall-clock time also be reduced by lowering the number OATS iterations. Presented in Table is an exploratory experiment compressing Llama-3 70B by 50% with a rank ratio of 0.3 with only 20 iterations. Even with only a quarter of the iterations, OATS still able to outperform all prior pruning algorithms across all performance metrics.
MMLU (t)
Zero-shot (t) Perplexity (1)
74.02
73.41
4.95
Table 10: Exploratory experiment measuring the performance of OATS Llama-3 70B with only 20 iterations.
A.3 USING A ROBUST SCALING MATRIX
To explore whether the scaling matrix D is truly related to the outlier information, we run the following two experiments:
Scaling by the square root of the features' second moments, as is currently done in OATS.
Scaling by the median the features' absolute values (computed along batch and sequence dimensions):
Drobust median( [X)
The second experiment estimates the square root of the second moment of features in manner that robust (insensitive) to outliers akin to the Median Absolute Deviation estimator from robust statistics literature (Huber 1981). The results of the two experiments presented in Table []]
below:
Scaling Matrix MMLU (t)
Zero-shot (t)
Perplexity (L)
D Drobust
55.54
65.77
18.59
59.99
68.41
15.18
Table 11: Results of OATS Phi-3-Mini, 50% compression rate, with rank ratio of 25% using different scaling matrices.
The findings show that using the robust scaling method results in significantly performance. Hence, the scaling matrix D that is sensitive to the outlier features and captures their scale leads to better compression.
20

---

Published as a conference paper at ICLR 2025
A.4 SWITCHING THE ORDER OF THRESHOLDING
OATS opts to perform the singular-value thresholding first followed by the hard thresholding sim- ilar to Zhou & Tao ( However, one might consider whether the alternative order could lead to faster convergence or a better approximation. Presented in Table [2] below is an extension of the ablation studies presented in Section 3.3 reporting the performance of OATS where the hard-
thresholding is performed first:
First Thresholding Operation
MMLU (t)
Zero-shot (1)
Perplexity (L)
Hard-Thresholding
65.51
70.54
11.72
Singular Value Thresholding (OATS)
65.84
70.71
11.50
Ablation results of switching of the order between the two thresholding operations. Ex- periments were run Phi-3-Mini, 40% compression rate, with rank ratio of 20%.
Table 12:
While the performance still remains competitive, across all performance metrics, the switched order falls short matching the original order presented in Algorithm |
A.5 MAGNITUDE- BASED PRUNING FOR THE SPARSE COMPONENT
Another question that we explored is whether it is sufficient to capture the outlier information en- tirely in the low-rank term and determine the sparse term through hard-thresholding that does not depend on the scaling:
S HARDTHRESHOLD(WVD - D)D-1.k).
Presented in Table 13 below are the results:
Outlier Scaling
MMLU (1)
Zero-shot (t)
Perplexity (4)
Bow-Rank Term Only Both Terms (OATS)
65.22
71.01
12.49
65.84
70.71
11.50
Table 13: Ablation results of OATS Phi-3-Mini, at 40% compression rate, with rank ratio of 20% testing whether the outlier information can be entirely captured by the low-rank term.
A.6 ADDITIONAL CPU SPEEDUP EXPERIMENTS
We provide additional experiments measuring the CPU speedup of OATS compared to dense mod- els. Following the same setup as Section 3.4 we run end-to-end inference on compressed Phi-3 Medium 15B model, but instead with a sequence consisting of 256 tokens, 6148 CPU @ 2.40GHz with 32 cores. The results are presented in Table 14 below:
Intel Xeon Gold
Compression
Method
Throughput
Speedup
0%
Dense
0.37
1.00x
Unstructured Pruning
0.40
1.08x
30%
OATS
0.40
1.08x
Unstructured Pruning
0.43
1.16x
40%
OATS
0.43
1.16x
Unstructured Pruning OATS
0.49
1.32x
50%
0.49
1.32x
Table 14: Comparison of throughput (tokens/second) and speedup achieved through OATS and unstructured pruning methods relative to their dense counterparts on sequence consisting of 256 tokens.
21

---

Published as a conference paper at ICLR 2025
Although OATS and unstructured pruning now deliver comparable CPU speed-ups, expect further gains are achievable by designing optimizations that explicitly exploit OATS' sparse plus low-rank structure.
A.7 ADDITIONAL HYPERPARAMETER TESTS FOR OATS
Presented in Table 15 below includes more hyperparameters that we experimented with for the Phi-3
Mini and Llama-3 models.
Model
Compression
Rank Ratio
MMLU (1)
Zero-Shot (1)
Perplexity ())
0.1
68.70
71.65
10.24
30%
0.2
68.02
71.81
10.21
0.3
69.28
72.07
10.28
0.1
65.75
69.94
11.57
Phi-3 Mini
40%
0.2
65.84
70.71
11.50
0.3
66.81
70.54
11.60
0.1
57.96
67.37
13,48
50%
0.2
59.12
68.02
15.13
0.3
58.68
68.63
15.47
0.1
63.62
68.99
9.35
30%
0.2
63.09
69.54
9.09
0.1
61.44
68.23
9.23
Llama-3 8B
40%
0.2
61.97
68.43
9.09
0.1
56.46
65.33
10.85
50%
0.2
56.07
65.51
10.70
Table 15: Further experiments testing different hyperparameter configurations for OATS on the Phi- 3 Mini and Llama-3 8B models.
A.8 PERFORMANCE GAP BETWEEN OATS AND WANDA
To better understand the increase in performance induced by the addition of the low-rank term in OATS, we have compiled in Table 16 below the performance gaps between OATS and Wanda.
Model
Compression
MMLU (t)
Zero-Shot (1)
Perplexity ())
30%
+1.21%
+0.82%
-0.44
Phi-3 Mini
40%
+1.60%
+1.24%
-1.06
50%
+5.42%
+3.38%
-2.05
30%
+0.97%
-0.01%
-0.43
Phi-3 Medium
40%
+1.65%
+1.45%
-0.79
50%
+2.52%
+2.43%
-1.07
30%
+1.55%
+0.71%
+0.20
Llama-3 8B
40%
+2.13%
+1.64%
-0.50
50%
+6.63%
+2.44%
-1.49
30%
-0.68%
+0.05%
-0.17
Llama-3 70B
40%
+0.73%
+0.78%
-0.40
50%
+2.74%
+0.45%
-0.60
Table 16: The impact of including low-rank term in OATS compared to Wanda.
A.9 QWEN 2.5 EXPERIMENTS
Presented in Table 17 below additional experiments benchmarking OATS against prior pruning algorithms on the Qwen 2.5 3B Instruct model (Qwen Team 2024). All OATS experiments utilize rank ratio of (.2 and 80 iterations.
22

---

Published as a conference paper at ICLR 2025
Compression
Method
MMLU (t)
Zero-Shot (f)
Perplexity (4)
0%
Dense
65.99
68.49
11.02
SparseGPT
65.65
67.91
11.55
Wanda
65.46
68.08
11.66
30%
DSNoT
65.65
68.21
11.67
OATS
65.36
68.74
11.45
SparseGPT
63.04
67.64
12.56
Wanda
61.88
67.14
12.89
40%
DSNoT
62.26
67.42
12.91
OATS
64.30
68.76
12.31
SparseGPT
57.43
64.36
14.92
Wanda
55.39
64.10
16.27
50%
DSNoT
55.78
64.77
16.43
OATS
58.78
65.74
14.91
Table 17: Benchmarks for OATS on the Qwen 2.5 3B Instruct model.
A.
MMLU SUBJECTS
We evaluate on the following MMLU subjects:
Abstract Algebra
Business Ethics
College Computer Science
College Mathematics
Conceptual Physics
Formal Logic
Machine Learning
Miscellaneous
Philosophy
Global Facts
which aligns with the subset utilized the codebase of Ashkboos et al. (2024) that can be found here:
A.11 ATTENTION ROLLOUT: DETAILS
To generate the attention rollout visualizations depicted in Section 5 we average the attention ma- trices across the attention heads and discard the bottom 40% attention pixels. The act of discarding the lowest value attention pixels was inspired by the following blog post by Gil (202)).
A.12 ZERO-SHOT TASK-SPECIHIC PERFORMANCE
Table 18 below, shows the task specifcciiic performance for the zero-shot evaluation results presented in Section 3.2 and Appendix A.14
23

---

Published as a conference paper at ICLR 2025
Compression
PIQA
HellaSwng
OpenRookQ4
RealQ
D%
Dense
81.25
77.50
73.36
46,80
75.81
85.32
78:45
57.25
SpaseCPT
78:35
76.34
09385
15.00
73.25
64.11
76.39
55.89
Wanda
79.65
76.27
71.59
18.00
73.65
83.70
77.23
55,20
SparseGiPT
72.22
77.55
DSNoT w/ Wanda
80.4
75.52
72.06
47.60
74.37
84.53
79.55
$5.55
OATS
30.03
77.07
72.61
17.60
74.37
84,92
77.44
$7.76
SpansCPP
78.35
15.07
68.59
47.00
72.20
83.67
75.29
53.24
Wanda
78.35
SpareeGPP
DSNoT w/ Wando
79.33
73.06
70.88
44.00
70.76
83.99
77.40
53.41
OATS
79.38
75,86
70.01
46,60
72.56
83.98
76.85
55.12
SpansePTT
77.20
70.63
66.46
15.20
70.76
83.06
70.58
47.01
Wanda
50%
DSNoT SparseGPT
76.28
67.18
65.90
42.20
63.90
81.56
72,90
48.04
DSNoT w/ Wanda
75.52
66.54
67.64
13.00
65.34
82.54
73:48
48.55
OATS
77.26
71,64
69.53
44.80
73.65
81.28
77.10
$2.05
SpansePTT
$1.39
82.02
75.77
50.80
77.26
87,80
80.05
61.18
Warda
$1.39
80.88
76.01
19.40
76.90
87.74
79.59
60.49
30%
DSNoT SpursoGPPT
81.90 94.94
80.76
76.95
48
75.81
87.65
79.25
59.81
NoT Wanda
76.53
OATS
SPanseGPT
30.4
80.70
75.53
51.20
77.26
88.32
81.23
60.58
Phi-3 Medium
40%
DSNoT SpurseGIT
Wanda
79.82
79.87
78.07
78.15
75.37
73.45
17.00
18.00
77.26
76.53
87.98
87.71
77.31
78.11
$8.96
58.19
Wanda
ONTS
$1.39
75.06
77.62
60.84
SpansCPPT
79.71
78.21
73.61
30.30
75.45
87.09
82.05
39.39
Wansaa
78.29
74.07
74.03
45,00
75.81
85.72
77.44
57. 34
DSN6T SparseGiPT
79.27
74.59
44.40
76.90
OATS
81.07
79.18
76.09
50.20
74.73
87.77
80.05
58.02
D9%
Dense
80.74
79.16
73.40
45.00
67.87
90.18
77.69
$3.50
SpanseGPT
50.35
78.58
73.24
66.79
81.38
76.81
51.11
Wanda
79.98
30%
DSNoT SparseGPT
80.20
78.12
73.30
14.40
65.70
82.20
75.72
$1.71
DSNoT w/ Wanda
79.82
77.99
73.09
14.80
63. 63.18
81.80
77.06
$1.37
OATS
8003
78.75
73.64
45.20
66.06
81.13
76. 94
$2.99
Llama-3 SB
40%
DSNoT w/ SparseGPT
Wanda
78.29
75.92
73.32
42.60
$8.48
80.86
73.11
49.15
47.70
DSN w/ Wando
78.51
75.52
73.24
43,80
61.73
90.70
72.01
47.70
OATS
79.71
77.18
74.19
43,80
67.51
74.92
SpansePTT
Wanda
77.53
69.34
70.24
10.00
61.73
76.57
66.96
43.77
50%
DSNoT w/ SparseGPT
76.88
69.45
69 69.30
39.60
$9.21
77.25
67.93
43.32
DSNoT w/ Wanda
77.09
68.57
69.77
38 60
$7.76
76.27
67.34
43.43
45.05
Dons
31.33
84,89
18.60
65.
85.26
86.05
61.31
SpineCiPT
$4.65
84.63
80.35
48.00
60:31
83.26
83.12
63.31
Wanda
84.39
83.97
80.58
48.40
70.04
85.29
85.06
63.82
30%
DSN6T SparseGPPT
$4.05
84,49
80.11
69.68
85.57
$5.10
63.82
OATS
$4.28
84.40
80.66
48.40
69.31
85.32
85.90
63.65
SpaneCPT
83.62
83.71
80103
17.80
69.68
85.69
81.47
61.95
Llama-3 70B
Wanda
SparseGPT
83.57
$2.37
83.03
78.93
47.40
68.
85.06
84.34
62.29
DSNoT Wanda
83.79
83.35
79.72
16.80
67.87
85.57
84. 89
62.37
OATS
84.44
83.69
80.11
48.60
70.40
84.56
84.55
62.71
SparseCPP
83.13
81.68
79.32
46:20
71.12
85.17
R1.27
37.31
Wanda
83.08
78.22
48.00
69.31
84.22
81.61
$7.25
SparseGPT
$1.34
80.98
DSNoT w/ Wanda
85.24
81.64
78.45
46.80
69.31
85.23
81.52
57.76
OATS
83.11
82.16
79.01
17.10
68.59
85.47
82.11
58.28
Table 18: Task-Specific Zero-Shot Results
A.13 WIKITEXT-2 PERPLEXITY
We utilized LM Harness (Gao et al.) 2024 to measure the Wikitext-2 perplexities for the Phi-3 Mini, Medium, and Llama-3 8B models in Table However, prior works (Sun et al. 2024bb Zhang et al.) 2024bb have also opted to measure Wikitext-2 perplexities utilizing a different normalization and dataset segmentation. We report the Wikitext-2 perplexities utilizing the latter approach in Table
19 below:
We the implementation sourced from the code hase of Sun et
al.
2024b
24

---

Published as a conference paper at ICLR 2025
Phi-3
Llama-3
Compression
Method
Mini (3.8B)
Medium (14B)
8B
0%
Dense
5.64
4.02
5.54
SparseGPT
6.32
4.56
6.04
Wanda
6.19
4.46
6.03
30%
DSNoT
6.11
4.40
5.99
OATS
5.96
4.30
5.87
SparseGPT
7.05
5.02
6.75
Wanda
6.99
4.97
6.69
40%
DSNoT
6.81
4.89
6.67
OATS
6.51
4.64
6.39
SparseGPT
8.55
5.65
8.30
Wanda
8.84
5.73
8.79
50%
DSNoT
8.58
5.68
8.84
OATS
7.98
5.28
7.77
Table 19: Comparison of perplexity (lower is better) utilizing a different normalization and dataset segmentation.
A. PRUNING IMPLEMENTATION AND HYPERPARAMETERS
Our code and pruning method implementations are based on the following codebases:
SliceGPT (Ashkboos al. 2024):
SparseGPT (Frantar & Alistarh
2023
Wanda (Sun et al.
2024b)
http://giihbb.commmocssbbbbaaaaa
DSNoT Zhhng et al.
2024b):
https//giihbb..cm//yyxxm//////
OWL Yin et al.
2024b
http://gtthhb.com/mnnji///WW
We utilize Huggingface's Transformers library to implement the large language models and vision transformers for our experiments (Wolf et al. 2020).
A.14.1 SPARSEGPT HYPERPARAMETERS
We utilize a blocksize of 128 across all experiments and Hessian dampening of 0.01 and 0.1 where the ladder is utilized only when faced with non-positive definiteness issues related with the Cholesky decomposition.
A.14.2 DSNOT HYPERPARAMETERS
We run experiments utilizing DSNoT where the initial masks generated by SparseGPT and Wanda. All DSNoT experiments were run with 50 iterations and an update threshold of 0.1. Table
20 below, shows the results distinguishing between the two initial methods that were utilized.
25

---

Published as a conference paper at ICLR 2025
Model
Compression
Method
MMLU (1)
Zero-Shot(t)
Perplexity())
DSNOT w/ SparseGPT
67.01
70.81
10.55
30%
DSNoT w/ Wanda
68.02
71.20
10.51
DSNoT w/ SparseGPT
62.94
68.86
12.29
Phi-3 Mini
40%
DSNoT w/ Wanda
63.57
69.08
12.17
DSNoT w/ SparseGPT
53.99
64.74
16.71
50%
DSNoT w/ Wanda
54.28
65.33
16,68
DSNoT SparseGPT
74.89
73.82
7.11
30%
DSNoT w/ Wanda
75.13
74.03
7.11
DSNoT w/ SparseGPT
73.15
72.54
8.24
Phi-3 Medium
40%
DSNoT w/ Wanda
73.20
72.90
8.27
DSNoT w/ SparseGiTT
68.65
71.12
9.96
50%
DSNoT w/ Wanda
68.12
71.10
10.02
DSNoT w/ SparseGPT
62.99
68.98
9.37
30%
DSNoT w/ Wanda
63.72
68.64
9.36
DSNoT w/ SparseGPT
58.97
66.28
9.60
Llama-3 8B
40%
DSNoT w/ Wanda
59.99
66.65
9.68
DSNoT w/ SparseGPT
49.15
62.74
12.41
50%
DSNoT w/ Wanda
49.20
62.35
12.42
DSNoT w/ SparseGPT
78.76
75.13
3.28
30%
DSNoT w/ Wanda
79.00
75.54
3.27
DSNoT SparseGPT
76.39
73.26
4.16
Llama-3 70B
40%
DSNoT w/ Wanda
77.70
74.29
4.10
DSNoT SparseGPT
72.18
72.02
5.87
50%
DSNoT w/ Wanda
72.76
72.91
5.58
Table 20: LLM performance metrics DSNoT with different initial methods.
Table 21
below, shows the analogous results but for our vision transformer experiments:
Model
Compression
Method
Accuracy (%)
DSNoT w/ SparseGPT
80.01
30%
DSNoT w/ Wanda
80.16
DSNoT w/ SparseGPT
79.12
ViT-Base
40%
DSNoT w/ Wanda
79.46
DSNoT w/ SparseGPT
75.83
50%
DSNoT w/ Wanda
76.90
DSNoT w/ SparseGIT
86.46
30%
DSNoT w/ Wanda
86.45
DSNoT w/ SparseGPT
86.37
DinoV2-Giant
40%
DSNoT w/ Wanda
86.30
DSNoT w/ SparseGPT
85.87
50%
DSNoT w/ Wanda
85.93
Table 21: ImageNet Validation Accuracy of DSNoT with different initial methods.
26


Reviews:
[{"review_id": "1lDbd62yTE", "reviewer": "Reviewer_Gg5U", "summary": "The paper introduces OATS (Outlier-Aware Pruning Through Sparse and Low-Rank Decomposition), a method designed to compress large transformer models without the need for retraining. The central concept involves representing weight matrices as the sum of a sparse and low-rank matrix.\n\nThe authors propose an iterative alternating thresholding technique to compute the joint sparse and low-rank decomposition of a matrix. They also focus on preserving outliers by scaling weights according to input embeddings prior to decomposition. The results indicate that OATS performs effectively on both language and vision tasks, outperforming other pruning methods proposed for transformers across various compression levels and tasks. Additionally, OATS offers speed improvements on the CPU.", "questions": "One potential extension of this work could involve incorporating quantization into the proposed framework. Although this addition may be a long shot, integrating quantization could make the model a unified approach to transformer compression. Could the authors provide insights on whether quantization can be integrated with their current framework, or if not, what are the main challenges?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The concept of compressing a model as a sum of a sparse and low-rank matrix is very promising. Unlike most prior methods, which focus on one approach, OATS leverages both to potentially enhance performance.\n\n- OATS is retraining-free, which is crucial for practical applications where even a single backpropagation pass can be computationally prohibitive.\n\n- The framework has been tested on state-of-the-art models like Llama and ViT, demonstrating competitive performance.\n\n- The Alternating Thresholding technique in OATS heuristically finds an effective combination of low-rank and sparse components and accommodates different sparsity patterns.\n\n- By scaling the weight matrix $W$ with a diagonal rescaling matrix $D$, OATS emphasizes outliers, enabling outlier-aware compression that avoids performance degradation.", "weaknesses": "One concern is that the method relies on multiple calls to truncated SVD, which can be computationally intensive. Specifically, finding the top-$r$ singular values of an $m \\times n$ matrix has a time complexity of $O(mnr)$. Given that compression speed is a significant factor for practical applications, it would be helpful if the authors could clarify the time complexity and wall-clock time spent on the compression process of the overall algorithm. This would offer a more concrete understanding of its practicality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "neU1UP1gnz", "reviewer": "Reviewer_BUra", "summary": "This paper introduces OATS, a novel compression technique for large-scale transformers that combines sparse and low-rank approximations to reduce memory and compute costs without requiring costly retraining. OATS scales model weights by the second moment of input embeddings to preserve essential outlier features in transformers, ensuring model performance is maintained during compression. This approach addresses the typical performance degradation seen with increasing compression in existing pruning methods.", "questions": "See the Weakness. Additionally, why use inverse transformation to reach the compressed weight? What is the actual speedup when using N:M sparsity on GPUs.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "+ Low-rankness plus sparsity is a good fit for compressing LLMs without retraining.\n+ By separating the model into a sparse and a low-rank part, the approximation error can be theoretically reduced as the two parts can compensate for each other. \n+ This paper provides measurements for practical speedups on CPUs with existing sparse computation frameworks.", "weaknesses": "- The novelty is limited. The combination of low-rankness and sparsity is an old topic that has been explored for many years [R1, R2]. Applying the well-established approximation techniques to decompose/compress the large matrices in LLMs has little technical contribution. Besides, compressing DNN models using low-rank and sparse decomposition has already been well explored in [R3]. This paper just scales it to larger models and matrices. Authors are encouraged to specify the unique difference from existing approaches and why this difference is also unique for LLMs.\n- The proposed Truncated SVD and Threshold strategies to achieve low-rankness and sparsity are too trivial. It is unknown how to decide the rank and number of zeroes. Besides, the order of applying SVD and thresholding has a significant impact on the approximation errors. Authors are encouraged to clearly explain why using such decomposition strategies.\n- This paper claims \"outlier information\" in this paper. However, I have not seen any analysis or explanation for the \"outlier information,\" and the proposed solution is not related to the \"outlier information.\" Instead, this paper seems to directly apply the pruning approaches proposed in Wanda. The authors are encouraged to provide explanations of why the diagonal matrix is related to \"outlier information\" and why it is good for compression.\n- Many works have been proposed to compress LLMs with low-rankness and sparsity [R4-R6]. The authors have not presented the main differences among them and the unique contributions that stand out from those works.\n- Even though theoretical analysis may not have a practical guarantee of accuracy, the authors are encouraged to provide.\n- The paper presentation could be improved, especially the math equations. \n\n[R1] Sparse and Low-Rank Matrix Decompositions, Forty-Seventh Annual Allerton Conference, 2009.\n\n[R2] Godec: Randomized low-rank & sparse matrix decomposition in noisy case. ICML 2011.\n\n[R3] On compressing deep models by low rank and sparse decomposition, CVPR 2017.\n\n[R4] Slope: Double-pruned sparse plus lazy low-rank adapter pretraining of llms\n\n[R5] LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation\n\n[R6] SLiM - One-shot Quantized Sparse Plus Low-rank Approximation of LLMs", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JSfChjWjj3", "reviewer": "Reviewer_PLVG", "summary": "This paper proposes a relatively novel model compression technique aimed at Transformer architectures, in which weight matrices are approximated as a sum of a sparse and a low-rank matrix. To control for the (previously documented) outlier feature problem, weights are also scaled by the second moment of their input.", "questions": "It would be nice to see a bigger hyperparameter selection section with more architerctures considered.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The method proposed by this paper is well-explained and well-justified. The actual practical algorithm is easy to follow. \n\nReal-time speedup is shown in the CPU setting.\n\nThe experiments cover a range of model sizes (3.8-14B parameters). I especially liked seeing results on fairly small models, since those may be harder to compress. \n\nSection 5 is an interesting way to look at the problem, which I think can lead to interesting further work in the direction of interpretability.", "weaknesses": "The choice of the rank ratio parameter could have been better explored (in particular, looking at multiple architectures/tasks). \n\nTypos:\nLine 18: “approximating each weight” -> “approximating each weight matrix”\nLine 142: “the activations are calculated through a calibration set that is propagated through the compressed layers” - should be uncompressed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KHnWpA8TLM", "reviewer": "Reviewer_rXxS", "summary": "In this paper, the author presents a novel approach to compressing large transformers, coined OATS, that utilizes the second moment information in the input embeddings to decompose the model weights into a sum of sparse and low-rank matrices. The author also conducts a lot of experiments showing that OATS is able to consistently outperform prior state-of-the-art across multiple benchmarks and compression rates while also improving on speed-up.", "questions": "1. The author has clarified the differences between Wanda and OATS. However, an ablation study would further illustrate the impact of the low-rank term on performance. How much does the low-rank term contribute to the performance boost compared to Wanda alone?\n2. The hardware speedup on GPU with N:M sparsity patterns can be shown and discussed in Section3.4.\n3. The paper lacks details on how the sparsity pattern is defined for a matrix composed of a sparse matrix S plus a dense matrix LLL, especially within the context of N:M sparsity and structured pruning. Providing more detailed explanations would enhance clarity. \n4. The baseline for models are all originally designed for LLMs. It would be valuable to compare against pruning metrics specifically designed for ViTs. This would present a stronger baseline to effectively showcase the proposed method’s advantages.\n5. The motivation behind the choice of outlier information for the sparse term S is not entirely clear. Given that there are various methods for selecting the sparse components, such as using magnitude-based or gradient-based criteria, it would be helpful to know if the authors experimented with these or other selection methods.\n6. Given that most pruning studies report results on 70B-parameter models, has the author tested their method on larger language models, such as Llama2-70B or the newer Llama3-70B?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.This paper propose a novel method for large transformers compression that utilizes the second moment of the input embeddings to approximate the model’s weight matrices as a sum of a sparse matrix and a low-rank matrix.\n\n2.Extensive experiments on recent large language models demonstrate the effectiveness of the proposed method, which also generalizes well to vision Transformers.", "weaknesses": "Please see the questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)