SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs
Authors: Mohammad Mozaffari
Amir Yazdanbakhsh
Zhao Zhang
Maryam Mehri Dehnavi
Venue: ICLR 2025 Poster (2025)
Keywords: sparse training
low rank adapter
LLM
optimization
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose SLoPe, a double-pruned sparse plus low-rank adapter method improves sparse LLM accuracy while accelerating pretraining and inference, offering up to 1.25x and 1.54x training and inference speedup with reduced memory usage.

Abstract: We propose SLoPe, a Double-Pruned **S**parse Plus **L**azy L**o**w-rank Adapter **P**r**e**training method for LLMs that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footprint. Sparse pretraining of LLMs reduces the accuracy of the model, to overcome this, prior work uses dense models during fine-tuning. SLoPe improves the accuracy of sparsely pretrained models by adding low-rank adapters in the final 1% iterations of pretraining without adding significant overheads to the model pretraining and inference. In addition, SLoPe uses a double-pruned backward pass formulation that prunes the transposed weight matrix using N:M sparsity structures to enable an accelerated sparse backward pass. SLoPe accelerates the training and inference of models with billions of parameters up to 1.25× and 1.54× respectively (OPT-33B and OPT-66B) while reducing their memory usage by up to 0.63× and 0.61× for training and inference respectively.

Published as a conference paper at ICLR 2025
SLOPE: DOUBLE-PRUNED SPARSE PLUS LAZY Low- RANK ADAPTER PRETRAINING OF LLMS
Mohammad Mozaffari
Amir Yazdanbakhsh Google DeepMind Mountain View. USA
Department of Compute Science
University of Toronto
mmoza (cs. www.tto.ooo...o.oo
ayazdan@goog e com
Zhao Zhang
Maryam Mehri Dehnavi
Department of Electrical and Computer Engineering Rutgers University
Department of Compute Science
University of Toronto
zhao zhangerut edu edu
mnehride@css.o www.:ccssstooooooooooooo
ABSTRACT
We propose SLoPE, Double-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMS that improves the accuracy of sparse LLMs while accelerating their pretraining and inference and reducing their memory footprint. Sparse pretraining of LLMs reduces the accuracy of the model, to overcome this, prior work uses dense models during fine-tuning. SLOPE improves the accuracy of sparsely pretrained models by adding low-rank adapters in the final 1% iterations of pretraining without adding significant overheads to the model pretraining and inference. In addition, SLOPE uses a -punned backward pass formulation that prunes the transposed weight matrix using N:M sparsity structures to enable an accelerated sparse backward pass. SLOPE accelerates the training and inference of models with billions of parameters up to 1.25 x and 1.54x respectively (OPT-33B and OPT-66B) while reducing their memory usage by up to 0.63 and 0.61 x for training and inference respectively!
INTRODUCTION
Large Language Models (LLMs) demonstrate significant potential for natural language understanding and generation; however, they are expensive to train and execute because of their extensive parameter count and the substantial volume of training data required. The training process of LLMs include a pretraining (45) and a fine-tuning stage. In the pretraining phase, the model is trained on a large high-quality text (17: and then fine-tuned on different downstream tasks (57) 48). Both phases require significant amounts of computation, memory, and communication.
Model sparsity, in which the less important parts of the model pruned, can reduce the computation and memory overheads of LLM pretraining (24). Sparsity is unstructured if elements are removed from arbitrary locations in the tensors. Unstructured sparsity is hard accelerate due non-existing hardware/softww support (58). To resolve this, structured sparsity imposes constraints on where the zero elements can appear 28 33 creating dense blocks of nonzeros in the matrix to leverage dense compute routines. The drawback of the structured sparse methods is that they limit the choice for sparsity patterns leading to a reduction in accuracy in the sparse model when compared to dense (9. NVIDIA has recently introduced sparse tensor cores (43) to their hardware that accelerate more flexible structured sparsity patterns, i.e. 2:4 sparsity; hardware support for N:M sparsity where at most N out of M consecutive elements are zero is not yet available but machine learning practitioners are developing algorithms for these patterns 29
Applying N:M sparse masks to model leads to accuracy loss because of their limited choice of sparsity patterns. Changing the sparsity mask dynamically throughout pretraining is one of the approaches proposed to address this issue (I). Zhou et al. (61) proposes novel metric for finding the N:M sparsity patterns that lead to higher accuracy in each iteration.
(29) suggest the use of
decaying masks to further improve the accuracy of the models. STEP (34 proposes new optimizer
"ode and data for SLoPE available at: httpp:///itte 1yt../////soppo/ 11m

---

Published as a conference paper at ICLR 2025
that improves the convergence of models with adaptive masks. While the adaptive methods can improve the accuracy of the models, they require storing the dense weights and possibly additional metrics for updating the new sparsity patterns, while wasting portion of the training computations to train the weights that will be pruned in later iterations. SPDF (55) and Sparse-Dense Pretraining (FST) (26), can compensate for the loss imposed by sparsity with dense fine-tuning. But the dense fine-tuning stage will disable the memory and compute savings of sparse methods at inference. Inspired by this, we introduce additional non-zeros the weight in the last steps of pretraining, To avoid storing a dense model during inference while getting the same capabilities of dense weight, we add the non-zeros in the form of low-rank adapters (25). Our experiments show that using low rank adaptors leads to noticeably faster convergence compared to when the same number of learnable parameters are added to the sparse weights.
The use of N:M sparsity in LLM pretraining is limited to accelerating the forward pass in the training loop because the row-wise N:M structure in the weight sparsity pattern will lost when the weights transposed in the backward pass. Prior work (27 60 26) attempt to leverage sparsity in both forward and backward passes by finding transposable masks through various methods: greedy search algorithms, searching among random permutations, and searching among the results of convolution. However, these transposable masks reduce model accuracy and add significant runtime overheads (26), often resulting to slow-downs (up 8 x. To address these issues, we propose double-pruned backward pass formulation with theoretical convergence guarantees. Instead of enforcing the weight transpose to be N:M sparse, our approach transposes the N:M weight matrix first and then imposes N:M sparsity. This allows the weight matrices to exhibit a wider range of sparsity patterns, leading to improved accuracy.
Our method, SLOPE, is a Pouble-Pruned Sparse Plus Lazy Low-rank Adapter Pretraining method for LLMs. employs static N:M sparsity mask with double- bouble-pruned backward pass formulation to accelerate both the forward and backward passes. Key contributions of SLOPE are:
Double-Pruned backward pass -> We propose to transpose an already sparsified N:M weight matrix (forward pass) before imposing another round of N:M sparsity (backward pass), improving model quality and reducing mask search overheads.
Lazy Low-Rank adapters + We introduce additional parameters with minimal compute and memory overheads, merely for the last 1% iterations of pretraining, improving model capacity (see Figure T.
Optimized CUDA kernels + We jointly optimize Nvidia 2:4 sparse kernels and low-rank calls through efficient tiling and scheduling. Our highly-optimized CUDA kernels result to 1 25x end-to-end training speedup and 1. 54 x inference speedup on LLMs with billions of parameters, while reducing training and inference memory footprint by up to 63 x and 0. 61x, respectively,
SPARSE PLUS LOW-RANK PRETRAINING OF LLMS
Equation I and 3 depict the formulas for the forward and backward pass of the i-th linear layer in neural network. Here, the weight tensor is denoted as W. BRd Xdhn. and the input tensor is denoted as X; E b The forward pass generates an output tensor represented as ) E Rxxdout, In all equations, din and dout refer to the input and output dimensions of the respective layer.
FWD + V1- = X.,
BWD H Vw, C V Cv,
(2)
BWD - 2 Vx Vy, LW;
The dimension along which N:M pruning occurs corresponds to the reduction dimension in Matrix- Matrix multiplication. Without this restriction, the sparse Matrix-Matrix operation can not be accelerated on GPU (1.. With this restriction in mind, to leverage weight sparsity in forward and backward pass, needs to prune elements along the columns of WT in Equation (FWD) and W, in [nquuation 3] To satisfy this requirement, it is necessary to prune elements of the weight tensor W; along both row and column dimensions.
2

---

Published as a conference paper at ICLR 2025
N:M Sparse Pretraining with Lossy Backward Pass (99% iterations)
Sparse Lazy Low-Rank Pretraining
(1% iterations)
Forward Pass
Forward Pass
y4+
)/1
X
X
WAR
Backwwardd Pizca
Vw, f
Vy; L
X
Backward Pass
Vw,C
Vr
X
& Gradients for Low-Rank Tensors
Vx.C
Vy; L
Vv
Vy, C
C
W.C BLC
W.R.C. Cakurovakke Prunsd
Row- and Column-wtte
Figure 1: The sparse training pipeline SLOPE. Here, X, J/, and W denote the input, output, and the weight tensors for specific layer, respectively. V.L represents the gradient of the loss function. and R are the low-rank terms that are introduced only in the final 1% iterations. Superscript R shows row-wise pruning using N:M scheme and R, C' shows both column and row-wise N:M sparsification, leading to extra imposed zeros. Blue elements represent non-zero values, while white elements represent pruned values, and red elements indicate additional zeros introduced during the backward pass.
2.1 DOUBLE-PRUNED BACKWARD PASS
Various approaches be used exploit N.M sparsity during both the forward and backward passes. For example, one may prune the activation tensor X; in FWD along the row dimension and W; in BWD-2 along the column dimension. Although diverse combinations exist for pruning, our focus in this study is primarily on the sparsification of weight tensors for two reasons: (a) the sparsification of weight tensors directly impact the resource required for model storage and serving, and (b) our initial findings indicate that pruning weight tensors during both forward and backward passes has comparatively lesser adverse impact on the overall end-to-end model quality. More details on our experiments can be found in J As such, we posit double- backward pass formulation that can productively accelerate FWD and BWD-2 computations.
In addition, prove that such materialization of pruned weight tensors, despite being lossy convergence properties. For the rest this paper, we represent the weight tensor subjected to row- wise pruning W while the concurrent row-wise and column-wise pruning (double (double-pruned) is presented as R.R.C. We rewrite the training equations to accommodate these modifications, with
exhibits
proposed changes highlighted in blue:
FWD + Y = X.W.RT
(4)
BWD - 1 1 Vw; C = Cy;
(5)
BWD 2 v Vy, CW
(6)
Using this formulation for training, can accelerate both forward and backward passes owing the existence of N:M sparsity along both dimensions of weight tensors.
Memory footprint analysis. Inducing N:M structured sparsity not only improves computational efficiency of GEMM operations but also reduces the memory footprint for storing sparse tensors.
We term this formulation "lossy" because the weight matrix undergoes information loss during the backward pass compare its state the forward pass.

---

Published as a conference paper at ICLR 2025
It is noteworthy, however, that the storage of auxiliary meta-data becomes necessary, containing information about the locations of non-zero elements in a supporting matrix. E delincates the requisite number of bits for storing the indices in the N:M sparsity format, where [.] denoting the ceiling function. We present the detailed results on the memory footprint reduction in section 3
7
log (N)
A
N-M indeen
n
Convergence analysis. Theorem 2.1 (proof in subsection T.I shows the additional sparsity resulting from double pruning to an initially row-wise NIM pruned matrix. Following this lemma, we quantify the increased sparsity induced by double pruning with 1:2, 2:4, and 2:8 sparsity patters 12.5%, 9.375%, and 3.39%, respectively. This observation underscores that as the value of M in N:M increases, the surplus of zero elements in double-pruned matrix diminishes. This reduction in zero elements consequently implies a decrease in computational errors, enhancing the robustness of the computations. We expound further insights into this phenomenon in Appendix I.
Lemma 2.1. Consider randomly initialized matrix A. Following our notations, we denote the row-wise pruned version of A by A and the joint column- and row-wise pruned version of A by AR.C We use D()) to present the density ratio of a matrix. Equation 8 shows the additional zero elements in matrix A that are introduced by double prruiig. where
N
M
) D(AR
= Σ j=N+1
M
8) M-J N (1)
( 8² (
(8)
M
Theorem 2.2
states that the dynamic alteration of the column- mask in Equation 5 during each training iteration does not exert a detrimental impact on the convergence of the optimizer. This phenomenon can be attributed to the equivalence between the left-hand side of of 9| which corresponds to Equation 3] [BDD-2], and the averaging effect achieved through multiple training iterations of backpropagation with distinct sparsity mask. However, for arbitrary values of N and M. 4 and [ can be used in the training with convergence guarantee (proof in (subsection T.Tb. The sparsity mask is chosen randomly initialization, i.e. all the weights have the same probability of being zero or non-zero. This is because at initialization the location of weights with larger magnitude arbitrary. After choosing the sparsity mask at initialization, we keep the mask fixed throughout the entire training process. This policy ensures that each element in the weight has the same probability of being non-zero at initialization and satisfies the assumption in Lemma 2.1
Theorem 2.2. Assuming a loss function L(W,, for random sample X and considering random mask M,. Equation Tholds, where El.] is the expectation operator and the element-wiss multiplication.
M Ex (W, OW)II
Ex, Vx, N
2.2 LAZY LOW-RANK ADAPTERS
Pruning weight tensors in FWD and BWD-2 computations is desirable for computational efficiency but may have detrimental impact on quality. To mitigate this adverse impact on model quality, we augment the doubly-prun prunnd weight matrix with a low-rank matrix. The decomposition of the doubly- pruned weight matrix, combined with the low-rank matrix, maintains the computational efficiency of spare Matrix-Matrix multiplication during forward and backward passes. Simultaneously, this approach holds promise in alleviating the adverse effects of double pruning on overall model quality,
Considering the dense weight matrix, denoted by W Weense E Xddn Equation 10 illustrates the proposed matrix decomposition. In this expression, W Weparse E dadt xdin signifies a ddoubly-prunc matrix and L Igd and R E rxd are components of the low-rank approximation. The variable r denotes the rank of this low-rank approximation. r functions as a hyperpaaaaeeter that controls the trade-offs between memory footprint, computational efficiency, and model quality.
Wdense Waparse CR
(10)
4

---

Published as a conference paper at ICLR 2025
The matrix decomposition of doubly-prumd matrix combined with a low-rank matrix approximation reduces the memory footprint of W from dyndout to dindout N. + (din + dout)r. where r << M
mnin doue). The computational complexity of dense Matrix-Matrix multiplication, however, changes from dou to dod N. in comparison to b, din. and dout, our formulation effectively reduces both memory footprint and computational complexity of Natrix-Matris multiplication by factor of x.
M + b(din + )r. Given the substantially smaller value of r
We empirically show that the convergence rate of low-rank adapters surpasses that of sparse weights. We attribute this behavior the notably lower parameter counts inherent in low-rank adapters. Leveraging this observation, we incorporate low-rank adapters exclusively during the final 1% of the training iterations. This confined usages of low-rank adapters results in additional reduction of training cost, specifically in terms of total number of operations. We term the proposed usage of low-rank adapters the final steps of the training as lazy low-rank adapters.
2.3 SPARSE KERNELS
cuSPARSELI is CUDA library designed explicitly for sparse Matrix-Matrix multiplication, where one operand undergoes pruning with the 2:4 sparsity pattern. However, this library does not offer APIs for other algebraic routines such addition and assignment for sparse tensors. We now delve into the details of different kernels for training and overview our implementation methodology.
Algorithm shows the training process of a single linear layer taken from attention based model. We assume the use of weight decay in the optimizers, and subsequently design the requisite sparse APIs to facilitate the optimizer operations. The training starts with matrix initialization (line 2) and setting up sparse formats to store weight tensors and their corresponding transpose (line 3 and 4). Then, for every mini-batch in the training set, we compute the forward pass following Eqquation 4 (line 8). As part of the back ward pass, the derivative of the loss function with respect to the output activation is computed (line 10). Subsequently, the gradients of the loss function with respect to the input activation (line 11) and the weight tensor (line 12) are computed using Equation 3 and Equation 2 respectively. In order to cireumvent the necessity of updating weights with zero values and mitigate the associated memory footprint overhead, we employ a strategy wherein we mask the gradients for pruned weights. The computed values are stored order to implement weight decay in the optimizer and mitigate the impact of gradient scaling, we compute the value of V C + aW (line 15). Here, a is the weight decay applied in the optimizer, while y denotes the gradient scaling factor for numerical stability during the half-precision backward pass. The updated values for the weight tensor are calculated according to the optimizer update rule (line 16). Finally. the value of weight tensor and its transpose are updated directly in a sparse format (line 17 and line 18). More details about the implementation of the custom kernels used in Algorithm Ican be found in Appendix K
sparse format (line 13). Next, in
2.4 SLOPE RUNTIME OPTIMIZATION
While SLOPE improves the training and inference of LLMs by introducing sparse weights and low-rank adapters, naïve implementation can hinder its full performance improvement. Specifically, cuSPARSELt (40) SpMM kernels exhibit sensitivity to input and weight tensor shapes, and intro- ducing low-rank adapters at inference increases can increase the number of calls during the forward pass of each linear layer. This section covers our approach to optimize SLOPE's implementation and further improve model performance.
Efficient tiling of upsample tensors. Figure 3 (a) showcases the speedup achieved by the cuS- PARSELt backend across a range of tensor shapes commonly used in LLMs. While the speedup of SpMM in downsample tensors increases gradually as their sizes increase, the speedup of upsample tensor drops off at around hidden dimension = 4000. overcome this limitation, we tile the upsample tensor into multiple smaller matrices of equal size, cach of which benefits from improved speedup when multiplied by the input using 2:4 sparsity. By tuning the size of the tiles, we figured that the best performance can be achieved by using square tiles. The results of these multiplications are then concatenated. This optimization, as detailed in Appendix E. leads to a 1 12% improvement in inference speed and 4% increase in training speed with SLOPE.
Efficient kernel for combined SpMM+low-rrnm adapters. A straightforward implementation of low-rank adapters requires four kemel calls: one for sparse matrix multiplication, two for low-
5

---

reliable measure of central Published as a conference paper at ICLR 2025
Algorithm 1 Accelerated Sparse Pretraining Algorithm for a Linear Layer
Input: Weight: W. Training Set: D. Weight Decay: a, Gradient Scaling Factor: ?
2: backend.init()
3: WSparseTTrasspsee backendd.etuppp..raappsss.
4: WSparse = backend.setup(()
5: sparseMask = (WSparse 0) // Element-wise
6: for (X, Y) do
7: // Forward Pass
8: == =backeddspmmm(X WSparse WSparseTransposse Transpose)
9:
// Backward Pass
10: gradOutput=Vy-.
gradInput backend. Sompaadssssmggggaotuuuuuu WSparse)
12:
gradWeight = backend. matkenddmmttmlllcccccc ( wrw.adouutut..ranppsssse X)
gradWeightSparse backend.
http:/awwwwaaaccrcouuooooooooooe pruneAndCCompesssgaddweee sparseMask)
14:
// Optimizer with Weight Decay
g == sacckendssareeeddddgaaaWiiggggip
WSparse, 1 5. a)
WNew = optimizeer.pdduueWeggggggg
17:
backend. SeteeaaaiddaaeeppsserrrWsssSSSS WNew)
backend..ppaaeeSprreeeeee Waakleuunppasspeeeaaaaaaas/sarrn
WNew.rranpooee()
19: end for
rank computations, and one for adding the results. addition, our experiments demonstrate that multiplying matrices with low-rank adapters does not scale proportionally with the adapter's rank, leading to significant overheads due to their low arithmetic intensity (see Appendix C To address this, introduce two optimizations: (/) concatenating the downsample tensor to the sparse weight tensor, reducing kernel calls and increasing arithmetic intensity as in Equation Heft, and (2) leveraging a cuBLAS fused matrix multiplication and addition kernel, minimizing cache access and kernel calls as in Equation rright As demonstrated in Appendix D these optimizations collectively contribute speedup improvement of up to 6% in the end-to-end inference speed.
[i X4WTI4]:
=-22R + +
(11)
EXPERIMENTAL RESULTS
This section evaluates the efficacy of SLOPE accelerating the pretraining while achieving memory savings. Due to the substantial computational resources required for LLM pretraining, accuracy evaluation is primarily focused on smaller-scale LLMs up to 774M parameters. However, the speedup and memory reduction results extend to wider range of models, from 2.6B up 66B parameters.
3.1 END-TO-END SPEEDUP AND MEMORY SAVING: PRETRAINING AND INFERENCE
We evaluate the speedup and memory reduction by SLoPE during pretraining and inference across LLMs with different model parameter sizes. To demonstrate the scalability and efficiency of our method, we conducted extensive benchmarking on OPT (2.6 B to B) and LLaMA-3-8B and Mistral-v0.3-7B models. In all the experiments, we have enabled F (8) (Appendix M presents detailed ablation study on the impact of FlashAttention). To mitigate the impact of outliers, conducted 1,000 iterations for each speedup experiment and reported the median value. For the memory reduction experiments, we performed five independent runs and similarly reported the median outcome. These methodologies were chosen to provide tendency in our results
'It is noteworthy that for benchmarking speedup and memory savings, which require comparatively fewer computational resources than comprehensive pretraining accuracy experiments, we utilized the OPT, LLaMMA- 3, and Mistral-v0.3 model families. These families were selected due to their diverse range of model parameter sizes, allowing for thorough study of performance different scales.

---

Published as a conference paper at ICLR 2025
Table I: Comparative analysis of end-to-end pretraining and inference speedup (x) comparison between SLOPE and the latest work (FST) on accelerating pretraining with 2:4 sparsity (ICML 2024) (26). Note that the lack of inference speedup FST because of the final dense pretraining during the final iterations, resulting in dense model for inference. E-SR-STE stands for Extened SR-STE.
MODEL
METHOо
TRAINING No ADAPTER (r =
INFERENCE 1.56% ADAPTER
No ADAPTER (r=
6.25% ADAPTER
SLoPE
1.20
1.46
1.43
1.40
66B
FST
1.06
1.00
SLoPE
1.22
1.53
1.53
1.50
3PT-30B
FST
1.07
1.00
1.00
SLOPE
1.25
OPT-13B
FST
1.10
1.00
1.00
1.00
SLOPE
1.21
1.46
0.T-6.6B
TST
1.11
1.00
1.00
1.00
SLoPE
1.13
1.31
1.25
1.18
OPT-2.6B
FST
1.09
1.00
1.00
1.00
SLOPE
1.16
1.35
1.33
1.32
LLAMA-3-8B
IST
1.09
1.00
1.00
1.00
SLOPE
1.15
1.34
1.32
1.31
MISTRAL-VO.3-7B
PST
1.07
1,00
We compared our method against dense pretraining and inference directly in PyTorrh, which uses efficient cuBLAS backend. As the sparse pretraining benchmark, we compare our work against Sparse-Dense Pretraining (FST) (26), the state-of-the-art 2:4 pretraining method and the only semi- structured sparse pretraining work that provides end-to-end speedups. Note that methods targeting LLM pretraining with N:M sparsity often suffer from inefficiency due to mask search overheads and/or compression setup. Appendix H and Appendix B detail the profiling in Bi-Mask (60) and FST (26), which similarly use N:M sparsity on both forward and backward passes.
Notably, our approach, SLOPE, diverges significantly from recent work Fully Sparse Training (FST) (26) in two key aspects. Firstly, comprehensively prune all weights in the model, encompassing both MLP and Self-Artention modules, whereas FST only prunes weights in the MLP modules. Secondly, FST employs dynamic transposable weights, which introduce additional computation and memory overhead during training. Finally, FST necessitates dense fine-tuning (~17% of pretraining), thereby negating their speedup advantages during inference. In contrast, our approach achieves efficient and accurate large language models during both training and inference without such limitations.
SLOPE speedup for pretraining and inference. Table summarizes the speedups achieved by method during both training and inference. Since over 99% of training occurs without low-rank adapters, the training speedup is largely independent of the adapter rank. Conversely, inference speedup is directly influenced by the adapter rank. Given the varying hidden dimensions actoss differ- model sizes, report the inference speedup for various adapter rank ratios:
edopter- rank hidden dimension
Figure (a) illustrates that cuSPARSELt achieves higher speedups for large matrices until it reaches its maximum performance capacity (2x). A similar trend is observed in the pretraining and inference speedups of the models. For small matrices used in low-rank adapters, the lower arithmetic intensity of low-rank adapter multiplication results in higher overhead relative to sparse multiplication. This is because low arithmetic intensity limits the full utilization of GPU resources, leading inefficiencies.
SLOPE memory reduction in prefraining and inference. For training, the memory consumption of a dense model includes weights, gradients, and optimizer states, amounting to 4 x 16 bits for weights, x 16 bits for gradients, and x x 32 bits for optimizer states. The sparse model, however, stores non-zero weights and indices twice (for both weights and transposed weights), along with binary mask, gradients, and reduced optimizer states. This adds up to 2 x (16 + 3) bits (weights and transposed weights), 4 8 bits (binary mask), x 16 bits (gradients), and 2 2 x 32 bits (optimizer states). Consequently, the memory footprint during training is reduced by 68%. For inference, a dense model requires storing weights with total memory cost of 4 x 16 bits. In contrast, sparse
7

---

Published as a conference paper at ICLR 2025
Table 2: Comparative analysis of end-to-end memory reductions (x) during training and inference between SLOPE and the latest work (FST) on accelerating pretraining with 2:4 sparsity (ICML 2024) (26). Values greater than 1.00x show memory overhead.
MODEL
METHOD
TRAINING
INFERENCE 1.56% ADAPTER
No ADAPTER (r=
No ADAPTER (r
6.25% ADAPTER
SLOPE
0.67
OPT-66B
FST
1.27
1.00
1.00
1.00
SLoPE
0.67
0.61
0.63
0.69
OPT-30B
FST
1.17
1.00
1.00
1.00
OPT-13B
SLOPE
0.68
0,51
0.62
0.68
IST
1.16
1.00
1.00
1.00
SLOPE
0.68
0.60
0.62
0.68
OPT-6.6B
FST
1.19
1.00
SLoPE
0.67
0.62
0.64
0.70
OPT-2.6B
PST
1.18
1.00
SLOPE
0.63
0.66
0.69
0.71
LLAMA-3-8B
FST
1.17
1,00
1.00
1.00
SLOPE
MISTRAL--VO.-78
FST
1.15
1,00
1,00
1,00
model optimizes memory usage by storing only the non-zero weights and their indices, resulting in 2 x 16 bits for non-zeros and three bits for indices (see equation 7 This leads to a 54% reduction in memory usage during inference.
Table 2 presents the memory reduction for different low-rank adapter ranks and OPT, LLaMA-2, and Mistral model variants. The memory reduction is slightly less than the theoretical expectation, primarily because of additional memory usage from other model components, such layer norms, and dense model parameters.
3.2 PRETRAINING ACCURACY RESULTS
To assess the impact of SLOPE on model accuracy, we conducted pretraining experiments across various models and datasets (details in Appendix (O). In all experiments, the classifications heads and the first linear layer following the input are dense.
GPT2 (Small/Large). We pretrained both the small (117 M parameters) and large (774 M parameters) variants of GPT2 (46) on the Open WebText dataset (1. For fair comparison, we evaluate the models on MMLU (23) Are Challenge (6), and OpenBookQA (35) zero-shot tasks implemented in Language
Model Evaluation Harness (8). Additionally, we evaluate the validation perplexity of the models following the same experimental settings described in FlashAttention (IO 8). We compare SLOPE against two state-of-the-arr sparse pretraining methods, including (a) Wanda (5T) + a one-shot pruning technique, (b) Extended SR-STE 661 26 dynamic mask pretraining method for N:M sparsity, which serves as the foundation of follow-up work (27) 60 26). Please note that SR-STE only supports stochastic gradient descent optimization, and FST extended it other optimizers. We use the extension provided by FST in our work, and call it Extended SR-STE. The difference between Extended SR-STE and FST is that FST requires dense pretraining (fine-tuning) in the last 17% of pretraining and only prunes the MLP layers of the model, while SR-STE is fully sparse and prunes both the MLP and the Self-Attention layers of the model.
Figure compare the validation perplexity and zero-shot accuracy of GPT2-Small and GPT2-Large across a range of sparse pretraining methods with different hyperparameters. We have additionally added lazy low-rank adapters to Extended SR-STE (61 to show the effectiveness of our approach in other methods and also compare both methods with more similar settings. While gap perplexity consistently exists between sparse and dense models, SLOPP achieves lower perplexity compared to Wanda (51) and Extended SR-STE. Additionally, Table summarizes the achieved accuracy of the models on zero-shot tasks, showing that SLOPE is consistently achieving higher accuracy in comparison to Extended SR-STE. Moreover, adding lazy low-rank adapters can benefit both static and dynamic training methods. This improved accuracy stems from SLoPE's efficient allocation of the training budget. Specifically, Extended SR-STE, with its dynamic pruning masks, expends significant portion of its training budget (e.g. gradient updates) updating weights that may be
8

---

Published as a conference paper at ICLR 2025
GPT2 Small Perplexity
GPT2 Large Perplexity
40
Dense:
2:4 StoPe
70
Donne
Wands
60
2:4 Stal's
Perpientts
30
Wanda
60
SR-STE
-26
40
SR-STE
Tre 20-4
15
20
10
Berruion Ox 1000)
Bieration (x 1000)
Figure 2: Validation perplexity of GPT2-Small and JPT22-Large on OpenWebText. You shows the value of the decay factor parameter in Extended SR-STE (FST).
Table GPT2-Small accuracy results zero-shot tasks. Adapter rank is the ratio of the low-rank adapter to the hidden dimension of the model. For Extended SR-STE, we have used a decay factor of 6,-6, since it resulted the lowest perplexity in Open WebText. The best performing sparse configuration is highlighted in bold.
ADAPTER
ARC CHALLENGE"
OPEN- BOOKQAt
WINO- GRANDE
HELLA- SWAGt
METHOD
MMLU f
MATHQAT
PIQAT
RACET
DENSE
N/A
22.9
20.7
16.2
50.6
28.5
21.8
59.8
28.4
2.1%
23.0
19.3
16.4
50.8
27.5
20.8
57.6
27.2
SL.PP
0.05%
23.0
27.4
57.5
27.1
0
23.0
19.3
16.0
50.1
27.5
20.8
57.4
27.1
EXTENDED
2.1%
24.2
18.3
14.2
47.5
26.9
21.4
55.2
24.2
SR-STE
0.05%
24.1
26.8
54.5
24.2
24.1
18.3
12.6
47.5
26.9
21.2
54.8
24.0
ultimately pruned and not used at inference, leading wasted resources. Appendix A provides further details and supporting evidence for this observation. Additional validation results for GPT experiments on GLUE dataset are also provided in Appendix Q
and
WERT-Large-Uncasee We pretrain BERT-Large- Uncased (3) (355 M parameters) and fine-tune it for various question-answering and text classification tasks, following similar approach to (42) 36 (4) for both pretraining and fine-tuning Appendix G provides details on the pretraining and fine-tuning process. We evaluate the performance of BERT- Largee--eeUe Uncased the SQuAD v1.1 (48 and GLUE (57) tasks. We report the average metric score for GLUE and present the task-specific metrics in Appendix L Please note that in all the experiments corresponding BERT-Large Lare---arg- Uncased, when using Wanda, we have fine-tuned the model after pruning to improve the accuracy of the models, since using Wanda alone led to extremely low accuracy results.
Effects of low-rank adapters. To understand the impact of low-rank adapters on pretraining performance, we conducted ablations using low-rank adapter ranks of 4, 16, and 64 for 1% of the total number of iterations. These ranks represent up 6 25% the model's hidden dimension. Table shows the results of these settings on SQuAD and GLUE downstream tasks. We present per-task metrics for GLUE in Appendix [L] As expected, adding low-rank adapters improve the model's final accuracy across all tasks. Additionally, higher ranks improve the model's performance at the cost of increased computational requirements. is also worth to note that incorporating low-rank adapters only in the final iterations (1% of total iterations) sufficient recover pretraining accuracy.
Convergence rate of low-rank adapters. We hypothesized that low-rank adapters would converge faster due to their significantly fewer learnable parameters. To test this, we introduced low-rank adapters in the second phase BERT-Large- Uncased pretraining and monitored their convergence rate. Figure 3 shows the cosine similarity of the adapters, with the downsample adapter converging rapidly within 100 iterations and the upsample adapter converging slightly slower. Despite this, limiting training to 100 iterations still yields comparable results on downstream tasks.
Effects of mixed N:M sparsity. To study the sensitivity of different blocks to varying sparsity ratios and to assess their relative importance, we experiment across a range of configurations: (a) [2:4-2:4] uniformly applying 2:4 sparsity across all layers (b) [2:4-2:8] applying 2:4 sparsity

---

Published as a conference paper at ICLR 2025
Table 4: SQuADv1.1 and GLUE results BERT--Large Uncased with different adapter ranks. denotes the ratio of the low-rank adapter to the hidden dimension (1024).
DATASET
DENSE
r 0.39%
1.56%
y= 6.25%
SQUAD
90.44
89.1
89.1
89.2
89.5
GLUE
80.22
77.4
77.7
77.8
78.2
coSPARSELE SpMM Speedup
Low-Rank Adayter Similarity with Converged Weight
2.00
1.0
1.75
1.50
Coltae Stmllarit
2.5
1.000
Allcation
0.4
Rey
Upsomple
Valte
0.50
Upoumple Tiligg (Ours)
Downarrppe
0.2
Prujactio
Upeamplle
0.23
50
4000
6000
1900
Hiddlen Dimezsion
Eeratious
(a)
(b)
Figure 3: (a) The speedup achieved using cuSPARSELI backend in PyTorch for Attention (dout din), Upsample (diut = 4d(.)) and Downsample (dout = dun) matrices with batch size of 2048. (b) The cosine similarity of the low-rank adapters and the converged adapters for different layers in the model. The cosine similarities are averaged among the 24 layers BERT-Large-Uncann
pattern to the first 12 blocks and a 2:8 sparsity pattern to the last 12 blocks and (c) [2:8-2:4] + we reverse the sparsity ratios for the first and last 12 blocks. Note that, to reduce computational costs, we use the same dense checkpoint for Phase-1 in all settings and a low-rank adapter of rank 40 for all models. We also replicate this experiment using Wanda (51) and report the comparison results.
Table 5: SQuADv1.1 results on BERT-Large- Uncased for different sparsity settings.
SPARSITY (Fiest 12 BLOCKS LAST 12 BLOCKS)
SQUAD SLOPE
SQUAD WANDA
GLUE
SLOPE
WANDA
2:4-2:4
90.17
89.93
79.08
78.84
2:4-2:8
89.85
89.55
79.03
77.24
86.57
75.92
Table 5 summarizes the GLUE and SQuAD results for these settings. As the results show, increasing the sparsity ratio reduces the accuracy of the model on all tasks. But when the first blocks of the model are pruned, the accuracy drop is significantly higher, especially on the GLUE dataset. We conclude that the first blocks of the model are more sensitive to sparsity during pretraining, but one sparsify the last blocks of LLMs more aggressively. We observe similar pattern in Wanda results as well, but Wanda performs consistently worse than SLOPE in these cases.
Effects of sparsification on different modules. Each block in LLMs consists of self-attention module and MIP module, each containing multiple linear layers. We have analyzed the sensitivity of SLOPE to pruming each of those modules. Our results in Appendix demonstrate that SLOPE can sustain competitive quality results while pruning all modules in the model.
CONCLUSION
In conclusion, SLOPE improves both pretraining and inference times while reducing memory footprint with negligible impact on model performance. SLOPE achieves these benefits by effectively using N:M sparsity and lazy low-rank adapters in both forward and backward passes, supported by efficient design of CUDA kernels. Additionally, the use of lazy low-rank adapters allows for balancing memory footprint and model accuracy across a wide range models. The results show that SLoPE achieve up to 1.25x x and 1.54x speedup for pretraining and inference, respectively. These speedups achieved while our method reduces the effective memory footprint by up to 0.63 (pretraining) and 0.61 x (inference).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was also supported part by NSERC Discovery Grants (RGPIN-06516, DGECRO0303), the Canada Research Chairs program, Ontario Early Researcher award, the Canada Research Chairs program, the Ontario Early Researcher Award, and the Digital Research Alliance of Canada (www alliancecan ca). Work of Zhao Zhang was supported by National Science Foundation OAC- 2401246. We also acknowledge Texas at Austin for providing HPC resources that have contributed to the research results reported within this paper (ht.tp://wa) /wwww//wwww.. tacc. utexas s edu). We extend our gratitude towards David Fleet, Karolina Dziugaite, Suvinay Subramanian, Cliff Young, and David Anugraha for reviewing the paper and providing insightful feedback. We also thank the extended team Google DeepMind who enabled and supported this research direction.
Texas Advanced Computing Center (TACC) at The University of

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Ellie Pavlick Aaron Gokaslan, Vanya Cohen and Stefanie Tellex. OpenWebbTen Corpus, 2019.
[2] Dimitris Bertsimas, Ryan Cory-Wright, and Nicholas AG Johnson. Sparse Plus Low Rank Matrix Decomposition: A Discrete Optimization Approach. JMLR, 2023.
[3] Beidi Chen, Tri Dao, Eric Winsor, Zhao Song, Atri Rudra, and Christopher Ré. Scatterbrain: Unifying Sparse and Low-rank Attention Approximation. arXiv preprint 10.13:3110, 2021.
[4] Stanley F Chen, Douglas Beeferman, and Roni Rosenfeld. Evaluation Metries for Language Models. Carnegie Mellon University, 1998.
[5] Zhaodong Chen, Zheng Qu, Yuying Quan, Liu Liu, Yufei Ding, and Yuan Xie. Dynamic N:M Fine-grained Structured Sparse Attention Mechanism. In PPoPP. 2023.
[6] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:/803.05457, 2018.
[7] Compute Canada. Compute Canada. https::/ccomuueecaadd..c ca/
[8] Tri Dao. Flashattention-2:: Faster Attention with Better Parallelism and Work Partitioning, arXiv preprint arXiv:2307.0869) 2023.
[9] Tri Dao, Beidi Chen, Kaizhao Liang, Jiaming Yang, Zhao Song, Atri Rudra, and Christopher Re. Pixelated Butterfly: Simple and Efficient Sparse Training for Neural Network Models. arXiv preprint arXiv:21 12.00029, 2021.
[10] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré, FlashAttention: Fast and Memory-Effieieen Exact Attention with Awarenesss In NeurlPS, 2022.
[11] Tim Dettmers and Luke Zettlemoyer. Sparse Networks from Scratch: Faster Training without Losing Performance. arXiv preprint arXiv: (007::49000044 2019.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMss. arXiv preprint arXiv:2305. 14314, 2023.
[13] Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv: 1810.04805. 2018.
[14] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin, Linear Mode Connectivity and the Lottery Ticket Hypothesis. In ICML, 2020.
[15] Elias Frantar and Dan Alistarh. SparseGPT: Massive Language Models can be Accurately Pruned in One-shot. In ICML,, 2023.
[16] Trevor Gale, Erich Elsen, and Sara Hooker. The State of Sparsity in Deep Neural Networks. arXiv preprint arXiv: (ttp::902.0957.. 2019.
[17] Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The Pile: 800GB Dataset of Diverse Text for Language Modeling. arXiv preprint trXiv:2101.0002 2020.
[18]
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika. Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A frame- work for few-shot language model evaluation, 07 2024. URL https: zenodo. zenodo.org/ Records/1220884
12

---

Published as a conference paper at ICLR 2025
[19] Han Guo, Philip Greengard. Eric P Xing, and Yoon Kim. LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning, arXiv preprint arXiv:2311.12023 2023.
[20] Song Han, Huizi Mao, and William J Dally, Deep Compression: Compressing Deep Neu- ral Networks with Pruning, Trained Quantization and Huffman Coding. arXiv preprint 1rX.v11510.00149 2015.
[21] Song Han, Jeff Pool, John Tran, and William Dally. Learning both Weights and Connections for Efficient Neural Network. NeurIPS, 2015.
[22] Babak Hassibi and David Stork. Second Order Derivatives for Network Pruning: Optimal Brain Surgeon. NeurIPS, 1992.
[23] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300 2020.
[24] Torsten Hoefler, Dan Alistarh, Tal Ben-Nun, Nikoli Dryden, and Alexandra Peste. Sparsity in Deep Learning: Pruning and Growth for Efficient Inference and Training in Neural Networks. JMLR, 2021.
[25] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank Adaptation of Large Language Models, arXiv preprint arXiv:2106.09688 2021.
[26] Yuezhou Hu, Kang Zhao, Weiyu Huang. Jianfei Chen, and Jun Zhu. Accelerating Transformer Pre-Tran Tre-Training with 2:4 Sparsity. In ICML, 2024.
[27] Itay Hubara, Brian Chmiel, Moshe Island, Ron Banner, Joseph Naor, and Daniel Soudry, Accelerated Sparse Neural Training: A Provable and Efficient Method find N:M Transposable Masks. NeurIPS, 2021.
[28] Yu Ji, Ling Liang, Lei Deng, Youyang Zhang, Youhui Zhang, and Yuan Xie. TETRIS: Tile- matching the Tremendous Irregular Sparsity. NeurIPS, 2018
[29] Sheng-Chie Chun Kao, Amir Yazdanbakhsh, Suvinay Subramanian, Shivani Agrawal, Utku Evci, and Tushar Krishna. Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask. arXiv preprint arXiv:2209.07617 2022.
[30] Yann LeCun, John Denker, and Sara Solla. Optimal Brain Damage. NeurlPS, 2, 1989.
[31] Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation. arXiv preprint a30 1222, 2023.
[32] Hong Liu, Sang Michael Xie, Zhiyuan Li, and Tengyu Ma. Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models. In ICML, 2023.
[33] Zechun Liu, Haoyuan Mu, Xiangyu Zhang, Zichao Guo, Xin Yang, Kwang-Ting Cheng, and Jian Sun. MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning. In ICCV, 2019.
[34] Yucheng Lu, Shivani Agrawal, Suvinay Subramanian, Oleg Rybakov, Christopher De Sa, and Amir Yazdanbakhsh. STEP: Learning N:M Structured Sparsity Masks from Scratch with Precondition. arXiv preprint arXiv:2302.01172, 2023.
[35] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? new dataset for open book question answering. arXiv preprint arXiv: 1809.02789, 2018.
[36] Mohammad Mozaffari, Sikan Li, Zhao Zhang, and Maryam Mehri Dehnavi. MKOR: Momentum Lamentum-Enabled Factcct Optimizer Using Rank-1 Updates. In NewrlPS, 2023.
13

---

Published as a conference paper at ICLR 2025
[37] Tan Nguyen, Vai Suliafu, Stanley Osher, Long Chen, and Bao Wang. FMMformer: Efficient and Flexible Transformer via Decomposed Near-ficth field and Far-field Attention. In NeurIPS, 2021.
[38] Mahdi Nikdan, Soroush Tabesh, and Dan Alistarh. RoSA: Accurate Parameter-Efficient Fine- Tuning via Robust Adaptation. arXiv preprint arXiv:2401.0467 2024.
[39] NVIDIA, Péter Vingelmann, and Frank H.P. Fitzek. CUDA, release: 10.2.89, 2020. URL https://developpe nviiia com/ wttp:/ccccccooooeee cuda-toolkttt
[40] NVIDIA Corporation. NVIDIA cuSPARSELt.
cusparsselt/indx..noe
[41] NVIDIA Corporation. NVIDIA cUSPARSELI Functions. https://dbcs.nvidia.
cuda/cousarreelt/Emeae hww.aiiittnnnnniinnoooo .tmll.
[42] NVIDIA Corporation. NVIDIA Deep Learning Examples.
https://github.com
WWIDIA//eeglleeannnnggBBaa les.
[43] NVIDIA Corporation. NVIDIA Ampere Architecture In-Depth. https://developet
[44] J Gregory Pauloski, Qi Huang, Lei Huang, Shivaram Venkataraman, Kyle Chard, Ian Foster, and Zhao Zhang. KAISA: An Adaptive Second-order Optimizer Framework for Deep Neural Networks. In 2021.
[45] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving Language Understanding by Generative Pre-training. OpenAl, 2018.
[46] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Hya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl blog. 188):9, 2019.
[47] Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng- Chun Kao, Shivani Agrawal, Utku Evei, and Tushar Krishna. Progressive Gradient Flow for Robust N:M Sparsity Training Transformers. arXiv e-prints, 2024.
[48] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv preprint 1rXiv:1606.052550 2016.
[49] Victor Sanh, Thomas Wolf, and Alexander Rush. Movement Pruning: Adaptive Sparsity by Fine- Tinn-TTning.. NeurlPS, 2020.
[50] Scongjin Shin, Sang-Woo Woo Lee, Hwijeen Ahn, Sungdong Kim, HyoungSeok Kim, Boseop Kim, Kyunghyun Cho, Gichang Lee, Woomyoung Park, Jung-Wor Woo Ha, et al. On the Effect of Pretraining Corpora In-context Learning by a Large-scale Language Model. arXiv preprint WwXiv:2204.13509 2022.
[51] Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. A Simple and Effective Pruning Approach for Large Language Models, arXiv preprint arXiv:2306. 11695, 2023.
[52] Wei Sun, Aojun Zhou, Sander Stuijk, Rob Wijnhoven, Andrew Nelson, Henk Corporaal, et al. DominoSearch: Find Layer-wise Fine-grained N:M Sparse Schemes from Dense Neural Networks. In NeurIPS, 2021.
[53] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at practical size. arXiv preprint 40X.0:2408.00118. 2024.
[54] Texas Advanced Computing Center. Lonestar
https://taco.
systems/ wooesttreettt/oooooo
[55] Vithursan Thangarasa, Abhay Gupta, William Marshall, Tianda Li, Kevin Leong, Dennis DeCoste, Sean Lie, and Shreyas Saxena. SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models. arXiv preprint arXiv:2303, 10464, 2023.
14

---

Published as a conference paper at ICLR 2025
[56] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wrxiv:230.009288 2023.
[57] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A Multi-Taas Benchmark and Analysis Platform for Natural Language Understanding. arXiv preprint arX4v: 01..804..0461,222 2018
[58] Lucas Wilkinson, Kazem Cheshmi, and Maryam Mehri Dehnavi. Register Tiling for Unstruc- tured Sparsity in Neural Network Inference. PLDI, 2023.
[59] Samuel Williams, Andrew Waterman, and David Patterson. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 2009.
[60] Yuxin Zhang, Yiting Luo, Mingbao Lin, Yunshan Zhong, Jingjing Xie, Fei Chao, and Rongrong Ji. Bi-directional Masks for Efficient N:M Sparse Training, arXiv preprint arXiv:2302.000 30.00008........ 2023.
[61] Aojun Zhou, Yukun Ma, Junnan Zhu, Jianbo Liu, Zhijie Zhang, Kun Yuan, Wenxiu Sun, and Hongsheng Li. Learning N:M Fine-gras graingr Structured Sparse Neural Networks from Scratch. arXiv preprint arXiv:2102.04010, 2021.
15

---

Published as a conference paper at ICLR 2025
Appendix
Table of Contents
Comparison with Dynamic Sparsity: SR-STE
B cuSPARSELA Initialization Overhead: Static vs. Dynamic Sparsity
17
C Low-Rank Adapter Performance: Scaling and Arithmetic Intensity
18
D Efficient low-rank adapter implementation
18
Efficient weight tiling implementation
18
SLoPE sensitivity to pruning different module in transformer
19
BrRI-LLarg-Unnaased: Pretraining and Downstream Evaluation
19
H
Performance overhead of bidirectional mask
20
Sparsity ratio analysis of double-pruned backward pass
21
Sensitivity to the choice of pruning matrix
21
K Implementation details
22
L. Task-specific GLUE results
23
Integration with Flash Attention
23
Additional related work
23
0 Experiment setup, hyperparameters, compute resources
24
Comparison with dense models
24
Zero-shot GLUE results for GPT
25
Extended SR-STE and FST implementation details
25
Comparison Depth and Width Pruning
27
Proofs
27
T.1
Lemma 2.1
27
T2
Theorem 2.2
29
A COMPARISON WITH DYNAMIC SPARSITY: SR-STE
We pretrained GPT2- Small (Section 3.2) using the SR-STE method (61) and reported the perplex- ity results in Figure 2 SR-STE aims to mitigate the Sparse Architecture Divergence (SAD) by dynamically adjusting the sparsity mask throughout training. We have tested various decay factor hyperparameters to find the optimal optimization strategy for SR-STE.
To understand the performance gap between SR-STE and SLOPE (our method) for the same training budget, analyzed the mask dynamies in SR-STE. We plotted the average number of mask elements
16

---

Published as a conference paper at ICLR 2025
changes during training compared to the final converged mask sparsity pattern. High mask change values indicate that training resources are spent on updating weights that ultimately get pruned and do not necessarily contribute the final model accuracy.
Figure S shows this average mask difference per iteration relative to the converged model. As training progresses, the mask difference decreases, demonstrating SR-STE's convergence specifie sparsity pattern. However, in SLOPE, where all resources are dedicated to optimizing weights under a static mask SR-STE's dynamic approach leads wasted computation (represented by the area under the curve in Figure 4). Consequently, for the same training budget, SLOPE achieves a lower perplexity in comparison to SR-STE due to its static mask approach.
GPT2-Small SR-STE Average Mask Change
0.5
Difference
0.2
0.1
0.0
100
150
200
250
300
Iteration (x 1000)
Figure 4: Average mask difference between each iteration and the converged sparsity pattern in GPT2-Small pretraining using SR-STE. The highlighted area shows the ratio the resources used for updating weights that pruned and not used in the inference of the model.
B CUSPARSELT INITIALIZATION OVERHEAD: STATIC VS. DYNAMIC SPARSITY
This section analyzes the time breakdown of the cuSPARSELI SpMM pipeline, highlighting the significant overheads associated with dynamically changing sparsity masks. The cuSPARSEL: SpMM operation consists of two main phases: (1) Setup and (2) Matrix Multiplication. The setup phase involves initializing matrix handles and compressing the 2:4 sparse matrix. This compression copies non-zero values into a contiguous memory layout and generates indices for those values. The matrix multiplication phase leverages this metadata to perform the sparse maarix-matrii multiplication.
Figure shows the setup and multiplication time for square matrices using the cuSPARSEL SpMM backend. As evident from the figure, the setup overhead is significantly larger than the actual matrix multiplication time. For SLOPF, which employs static sparsity masks, setup cost incurred only once and becomes negligible compared to the numerous matrix multiplications performed during training and inference. However, for dynamic sparsity patterns, such as Sparse-Denns Pretraining (26), Bidirectional Masks (60), and other similar methods(27] 522 34; 61. this setup overhead can be substantial, leading to reduced speedup (as observed Section 3. for Sparse- slowdowns in some configurations (as discussed in Appendix
Deaase-Dense Dense Pretraining) or
"We determine the pruning mask at the very first iteration and maintain it for the rest of training. 'A recent work observed similar overhead using dynamic sparsity in cuSPARSEL: SpMM pipeline
17

---

Published as a conference paper at ICLR 2025
CuSPARSELI SpMM Time Breakdown
Setup
Multiplicasion Time
768
1024
2560
4006
5120
7168
9216
Figure 5: The setup and multiplication time for square matrices using the cuSPARSELE Sp.M backend.
C LOW-RANK ADAPTER PERFORMANCE: SCALING AND ARITHMETIC INTENSITY
As discussed in Section 2.4 the computation time of low-rank adapters does not scale linearly with their rank. This section provides experimental results to illustrate this behavior in more detail. The computational complexity of low-rank matrix multiplications is (brd), where b, r, and d represent the batch size, low-rank, and input/output dimensions of the layer, respectively. Based on this complexity, we expect the computation time be linear function of r. other words, reducing by factor of a should result in a corresponding a-fold reduction in computation time. However, in practice, this linearity does this expectation that matrix multiplication is compute-bound - is not always true. Specifically, the arithmetic intensity of the operation can fall below the machine's s balance point, as described in the Roofline model (59). Figure 6 shows the speedup achieved for different low-rank values using PyTorc'ss matrix multiplication function, which relics on the CUBLAS backend (39). The figure demonstrates that the achieved speedups
hold. This deviation arises because the assumption underlying
significantly lower than the ideal linear scaling, particularly when reducing the rank. Moreover, is evident that as the matrix dimensions increase, the gap between the ideal speedup and the observed speedup diminishes. This behavior can be attributed to the increased arithmetic intensity for larger matrices, leading to better utilization of tensor cores.
D EFFICIENT LOW-RANK ADAPTER IMPLEMENTATION
As discussed in Section 2.4 naïve implementation of low-rank adapters lead significant performance overheads due to the increased number kernel launches and the low arithmetic intensity of their multiplications. To address these issues, introduced two key optimizations: (1) concatenating one of the low-rank adapters with the sparse weights, and (2) fusing the multiplication of the other low-rank adapter with the subsequent result addition. These optimizations reduce kernel calls and increase arithmetic intensity, leading to more efficient utilization of GPU resources. Table summarizes the speedup improvements achieved with these optimizations, demonstrating an inference speedup increase of up to 6%.
E EFFICIENT WEIGHT TILING IMPLEMENTATION
We observed that the dimensions and aspect ratios of matrices significantly influence system speedup (Section 2.4). To mitigate this, implemented matrix tiling strategy, dividing upsample matrices
18

---

Published as a conference paper at ICLR 2025
Low-Rank Matrix Multiplication Speedup
100
Speedup
Dim: 768
Dim: 5120
Dim: 1024
Dim: 71G8
Dim: 2060
Speestup up
60
0
Rank
Figure The speedup achieved by low-rank adapters in comparison to a dense malrix-multiliication..
Table End-to-end speedup (x) before (left) and after (right) efficient implementation of low-rank
adapters.
MODEL
INFERENCE 1,56% ADAPTER
INFERENCE
6.25% ADAPTER
OPT-66B
1.15-1.20
1,12-1.19
OPT-30B
1.13-1,18
1.10-1.16
OPT-13B
1.11-1.10
1.09-1.10
OPT-6.6B
1.07-1.12
1.06-1.11
OPT-2.6B
1.01-1.06
0.97-1.00
into multiple square matrices. This approach significantly improves performance, as shown in Table 7 Our results demonstrate that matrix tiling enhance training speed by up to 4% and inference speed by up to 12%, highlighting effectiveness optimizing system performance.
F SLOPE SENSITIVITY TO PRUNING DIFFERENT MODULE IN TRANSFORMER
LLMS typically consist of two main modules: the MLP and the self-attention. The attention module's weights are represented as matrix in Rdx3dd while the MLP uses weights R and T idxd where d denotes the hidden dimension. To investigate the impact of sparsity on these modules, we conducted two experiments during Phase-2 of LERT-Large-nnaase pretraining: (a) [MLP] - pruning both MLP and self-attention modules. Table 8 presents the SQuAD and GLUE results for these settings. As expected, we observe a consistent, albeit slight, decrease in model quality as more modules are sparsified. The marginal decrease in performance suggests that models relatively insensitive to the specific modules being pruned when using our SL.OPE pretraining method. This observation underscores the robustness of our approach and its ability to maintain competitive quality across diverse sparsity configurations.
G
PRETRAINING AND DOWNSTREAM EVALUATION
BERT- Largee-usee Uncased pretraining consists of two phases, illustrated in Figure 7 Phase I comprises 7.038 iterations with a global batch size of 65,536 and a sequence length of 128. Phase 2 includes 1,563 iterations with global batch size of 32,768 and sequence length of 512.
19

---

Published as a conference paper at ICLR 2025
Table 7: End-to-end speedup (x) before (left) and after (right) splitting the upsample matrix. both cases, the optimization discussed in 6 is used.
MODEL
TRAINING
INFERENCE No ADAPTER
INFERENCE 1.56% ADAPTER
INFERENCE 6.25% ADAPTER
OРT-66B
1.10-1.13
1.22-1.34
1.20-1.31
1.19-1.30
OPT-30B
1.09-1.14
1.23-1.32
1.18-1.28
1.16-1.27
OPT-13B
1.10-1.12
1.23-1.30
1.10-1.30
1.10-1.12
OPT-6.6B
1.08-1.08
1.21-1.19
1.12-1.13
1.11-1.12
OPT-2.6B
1.03-1.02
1.02-1.07
1.06-1.05
1.00-1.00
Table 8: SQuADv1.1 results on SeeT--Lrgg-Uncasee for different pruned modules.
PRUNED MODULES
SQUAD
GLUE
DENSE
90.44
80.22
MLP
90.28
79.03
MLP SELF-ATTENTION
89.35
77.72
Figure 7 shows the training loss for both phases under different sparsity settings. We observe that higher sparsity ratios generally lead to higher training loss in both phases. Interestingly, the loss/perplexity gap does not directly correlate with the observed accuracy drops in downstream tasks 4 32 50).
BERT Loatte-LLage-- CaaggeeUuccssee Uncased Pretraining Phase 1
BERT- Uncased Pretraining Phase: 2
5.5
Dense
2-4 Sparsity
10
Mixed 2:4 and 488 Spaniity
Spareity
4.5
Triining Lons
4.0
Training
3.5
3.0
2.5
2.0
1.5
1000
2000
3000
4000
5000
6000
7003
200
400
800
800
1000
1200
1.000
Iterations
Iterations
Figure 7: Training loss of BERT-Larg--nnasee on WikiCorpus dataset for phase I and 2.
We evaluated the pretrained LERT--LargeUnccsse models on the SQuAD v1.1 (48) and GLUE (77 benchmarks. SQuAD v1.1, comprehensive question-answering dataset based Wikipedia, is widely used for LLM training. We report the Fl score for SQuAD throughout the paper. GLUE, diverse benchmark for natural language understanding tasks, provides a single aggregated score across various challenges, facilitating model comparisons. The paper presents the average metric score for GLUE, while spec-si metrics are detailed in Appendix
H PERFORMANCE OVERHEAD OF BIDIRECTIONAL MASK
Table presents the runtime results Bidirectional Masks (60), a ofate-of-the-art N:M sparsity method. Our analysis demonstrates that the mask search and associated overheads of this approach result significant slowdowns compared to dense baselines. For these experiments, we utilized the repository provided in (60) and employed the same models used in their evaluation.
20

---

Published as a conference paper at ICLR 2025
Table 9: End-to-end slow-down of Bi-directional Mask (60) in comparison the dense baseline.
MODEL
DATASET
SLOW-DOWN (x)
MOBILENET v2
CIFARIO
5.08
RESNET-32
CIFARTO
5.07
VGGT9
CIFARTO IO
8.41
RENET-18
IMAGENET
3.66
RESNET-50
IMAGENET
3.01
I SPARSITY RATIO ANALYSIS DOUBLE-PRUNED BACKWARD PASS
As described in Section 2.1 our proposed sparse pretraining approach involves pruning weights in both the forward and backward passes. During the backward pass, we apply both row-wise and column-wise pruning. which introduces additional zero values the column-wise pruned weight matrices used in the forward pass. Theorem 2.1 demonstrates that the resulting sparsity ratio can be calculated using Equation [ Figure visualizes the imposed sparsity ratios for various N:M sparsity patterns. As expected, smaller N/M ratios lead to lower imposed sparsity ratios. Moreover, in most cases, the imposed sparsity ratio is significantly smaller than the original matrix's density ratio.
Imposed Sparsity in Row-wise and Column- wise Pruning
Imposed Sparsity Ratio
16
64
M
Figure 8: The imposed sparsity ratio when pruning the weight matrices in the backward pass.
J SENSITIVITY TO THE CHOICE OF PRUNING MATRIX
In linear layers, three matrices are involved in the forward and backward passes: the input, the output gradient, and the weights. Pruning each of these matrices can have distinct effects on model performance.
To identify the optimal pruning strategy, we conducted an experiment where we pretrained GPT2- Small for 100,000 iterations (a quarter of the full pretraining) while systematically applying both static and dynamic pruning to each the three matrices. Static pruning involves generating a random mask at initialization and applying throughout training. Dynamic pruning, the other hand, prunes matrices based on their magnitude at each iteration. For dynamic pruning, the dense matrix values computed and stored, and then pruned every step.
Figure presents the validation perplexity for these experiments. Notably, pruning the output gradient led to model divergence after few iterations and is not shown in the figure.
Analysis. As shown in Figure 9 static pruning consistently achieved lower perplexities. This behavior suggests that focusing computational resources elements that remain active throughout training
21

---

Published as a conference paper at ICLR 2025
GPT2-Small Test Perplexity
32
30
Tost Perplexity
28
26
24
22
Dases
18
Drnamit Praning
Upoame Wouse Presing
20
40
60
80
100
Iterations (x1000)
Figure 9: Validation perplexity on GPT2-Small pretraining for 100,000 iterations for different matrix pruning settings. Pruning the output gradients leads to divergence within the few iterations and hence reported.
can lead to improved performance. Furthermore, pruning weights resulted in lower perplexities compared to pruning inputs, indicating that weights generally better target for pruning.
Intuition. Pruning weights is analogous to removing connections between neurons. Pruning activation tensors is similar to introducing non-linear function (akin to max-pooling) before each linear layer. Pruning output gradients, however, lacks practical justification and introduces errors into the backward pass, leading to model divergence.
K IMPLEMENTATION DETAILS
This section details the implementation the custom functions and CUDA kernels used in Algo- rithm facilitate efficient sparse training.
Initialization, sparse matrix setup, and SpMM kernels. Before utilizing the cuSPARSELt APIs, a crucial initialization phase ensures proper configuration of essential variables for our computational task. Following initialization, we configure the sparse data formats tailored for sparse matrices. This involves initializing matrix descriptors, pruning the matrices, and compressing them into more compact representation. cuSPARSEL employs an automated search to determine the optimal kernel for executing SpMM. While setting up these sparse data formats incurs non-negligible computational cost, this overhead is mitigated by the repetitive nature of matrix multiplications during the training process.
Prune and compress. The gradient of the loss function with respect to the weights requires pruning using the same mask as the weight matrix. Consequently, contains 50% extra zero values in the dense format. To address this redundancy, we developed an optimized CUDA kernel, integrated into PyTrr, that masks the gradients accordingly, eliminating the storage of unnecessary data and reducing memory usage. The output of this operation is new matrix in Rdo.x Sia.
Sparse matrix addition. The cuSPARSEL sparse data format does not natively support addition operations. However, for matrices A and B sharing the same sparsity patterns, we developed an optimized CUDA kernel scamlessly integrated into the PyTorch training workflow. This kernel efficiently computes linear combinations of the form 3A + YB, where 3 and arbitrary user- defined constants. This functionality is particularly useful for adding sparse weights to gradients in optimizers that utilize weight decay.
Update Sparse Matrix. After the optimizer updates the weight tensor values based on its rules, need to update the sparse matrix format to reflect these changes. We implemented an optimized
22

---

Published as a conference paper at ICLR 2025
Table 10: GLUE results for each task in the experiments discussed in section
First
Last
Method
Phase
Rank
12
12
CoLA
SST-2
MRPC
STS B (com)
QQP
RTE
MNLI
QNLI
Blocks
Blocks
(moc)
(acc)
(f1)
(f1)
(a0c)
(acc)
(acc)
Dense
1,2
2:4
2:4
51.6
91.9
81.2
87.5
87.8
66.4
84.1
91.3
SLaPE
MLP Mixer
2:4
2:4
41.8
91.4
88.7
87.2
82.1
90.1
Only
SLoPE
MI Mixer
2
0
2:4
2:4
38.8
90.4
85.9
86.4
85.9
63.5
81.5
89.3
Self-Attention
SLOPE with Non-Lazy Adapters
SLOPE with Non-Lazy
2
40
2:8
2:4
29
89.7
83.7
85.6
85.2
66.8
79.9
87.4
Adapters
SLoPE with Non-Lazy
2:4
2:8
44.1
91.1
89.8
86.6
86.3
62.5
82.3
89.6
Adaptons
SL.Pe
1,2
2:4
2:4
37.9
91.4
85.4
86.6
85.8
62.5
80.7
88.6
SLoPe
1.2
2:4
2:4
38.5
91.4
85.8
86.8
85.8
63.9
80.8
88.4
SLOPE
1.2
2:4
2:4
39.2
91.3
86.4
86.6
63.5
80.8
88.2
SLoPE
1.2
64
2:4
2:4
42.7
90.3
85.1
86.8
85.7
66.4
80.3
88.5
WANDA
N/A
2:4
2:4
43.0
91.4
88.3
86.9
86.1
63.5
81.9
89.6
WANDA
N/A
2:8
2:4
4.6
0.88
81.3
81
83.3
53.8
76.7
83.9
WANDA
N/A
2:4
2:8
42.1
91.7
84.4
87.2
63.5
81.5
81.9
CUDA kernel that copies the weight tensors from the PyTorch format into the cuSPARSEL data type, enabling efficient storage and manipulation of sparse weights.
L TASK-SPECIFIC GLUE RESULTS
The GLUE benchmark 57 comprises eight distinct natural language understanding classification tasks. While Section pressnted the average GLUE score measure of overall model performance, this section provides a more detailed analysis by presenting the complete task-specific results for each training setting in Table 10
M INTEGRATION WITH FLASH ATTENTION
To show the compatibility of SLOPE with other optimization methods, test integrate SLOPE with FlashAttention-2 (8) and show that these approaches are orthogonal in practice and can boost the performance of the model separately. Table summarizes the speedup achieved with and without SLoPE or FlashAttention-2. As it can be observed, each of these methods can improve the speed of the model both in training and inference, and adding them together will increase the speedup even further.
N ADDITIONAL RELATED WORK
Model pruning. Pruning the models has been one the most effective methods to reduce the complexity of LLMs (24.. One can pretrain the LLMs sparsely (14) or the pruning can happen after dense pretraining 222 30), possibly followed by a fine-tuning stage to recover part the lost accuracy (16 20). Pruning the models after pretraining
be costly (1) and typically fails to maintain their accuracy (15) 5T). While the sparse pretraining methods improve the accuracy of the model, they either unstructured sparsity patterns that cannot be accelerated with the current hardware
23

---

Published as a conference paper at ICLR 2025
Table 11: Speedup of SL.PP and FlashAttention-2 (FA2)
OPT models.
MODEL SIZE
TRAINING
INFERENCE
INFERENCE SLOPE + FA2
INFERENCE SLOPE FA2
FA2
SLOPE
SLOPE FA2
FA
SLOPE
SLOPE FA2
FA2
PA2
1.13
1.53
1.36
1.95
1.30
1.91
30B
1.36
1.14
1.66
1.46
1.32
2.24
1.28
2.24
1.27
2.20
13B
1.47
1.12
1.84
1.61
1.30
2.48
1.30
2.24
1.12
2.19
6.7B
1.60
1.08
1.94
1.71
1.21
2.50
1.13
2.50
1.12
2.45
2.6B
2.26
1.05
2.56
2.47
3.09
2.92
155
or have significant overheads when searching for and applying their structured sparse masks
27 60 52).
Low-rank adapters. Low-rank adapters have emerged promising method to reduce the fine- tuning costs associated with pre-trained LLMs and enable more efficient task switching (25). Different quantization and initialization schemes have been proposed to reduce their overheads in LLM fine- tuning 12 19). Adding low-rank factors to sparse matrices a low-weight mechanism widely used to improve the accuracy of approximations of dense matrices (2). In machine learning, the sparse plus low-rank approximations are limited to attention heads (37 and pruning after pretraining (38) 31), and the sparse plus low-rank pretraining has been investigated. Additionally, the sparse plus low-rank fine-tuning work does not provide acceleration in both forward and backward pass of the fine-tuning process. Furthermore, the low-rank adapters these works are added at the beginning of the fine-tuning process, adding extra overheads to the fine-tuning process.
O EXPERIMENT SETUP, HYPERPARAMETERS COMPUTE RESOURCES
Our experiments were conducted on the Narval and Mist clusters at Compute Canada (7) and the Lonestar 6 cluster at the Texas Advanced Computing Center (54). Each Narval node is equipped with four Nvidia A 100 GPUs, each with 40GB of memory. Mist nodes feature four Nvidia V100 GPUs, each with 32GB of memory, while Lonestar 6 nodes have three Nvidia A100 GPUs, each with 40GB of memory. For our accuracy experiments, we emulated 2:4 and N:M sparsity using custom- destinddesiined, low-overhead CUDA kernels to prune weights both the forward and backward passes. We utilized mixture of available resources across the clusters, model accuracy is not hardware-dependent
Our speedup and memory saving experiments were conducted on single A100 GPU in the Narval cluster. We ran 1000 iterations of training or inference gather the necessary statistics. For speedup experiments, we reported the median of the 1000 samples mitigate the effects of outliers. Each memory reduction experiment the default hyperparameters found in the NVIDIA BERT codebase (42) and the FlashAttention GPT codebase (TO; 8). Further tuning of hyperparameters for sparse pretraining is left a future direction. Training -arage required approximately 32 hours on 64 A100-64GB GPUs. The pretraining of HPP2-Small/Large took 32 and 111 hours, respectively, on 64 V100-32GB GPUs.
run five times, and the median value reported. We employed
P COMPARISON WITH DENSE MODELS
To compare the performance of sparse models with dense models of the same size, have conducted an experiment with GPT2-Small, in which we have reduced the number of transformer blocks in the model to half of GPT2-Smai We call this configuration GPT2-Half. Tables 12 and T3 summarize the accuracy results for GPT2-Half on different zero-shot downstream tasks.
It can be observed that SLOPE outperforms GPT2-Half on average, while dynamic sparse training methods, such as SR-STE perform worse than it. Additionally, it is clear that adding low-rank adapters to the model improves the accuracy of all sparse pretraining methods.
24

---

Published as a conference paper at ICLR 2025
Table 12: Performance comparison across different GPT models, sparsity methods, and LoRA ranks
on various tasks. E-SR-STE stands for Extended SR-STE.
Model
Method
LoRA (r)
MMI.U
Are Challenge
Open Book QA
Average
GPT2-Small
Dense
r=0
22,9
20.7
16.2
19.94
GPT2-Small
SLOPE
r=0
23.0
19.3
16.0
19.43
GPT2-Small
SLOPE
r= 0.05%
23.0
19.4
16.2
19.53
GPT2-Small
SLOPE
2.1%
23.0
19.3
16.4
19.57
GPT2-Small
E-SR-STE
r=0
24.1
18.3
12.6
18.33
GPT2-Small
E-SR-STE
0.05%
24.1
18.4
14.2
18.90
GPT2-Small
E-SR-STE
2=1%
24.2
18.3
14.2
18,90
GPT2-Half
Dense
r=0
22.9
19.5
16.0
19.47
ZERO-SHOT GLUE RESULTS FOR GPT
We have tested the accuracy of the models on the zero-shot GLUE tasks in Language Model Evaluation Harness (18). Table 13 summarizes the achieved GLUE results by different models. can be seen that SLoPE outperforms other SR-STE and GPT2-Half on average. Additionally, SR-STE performs better than GPT2-Hap 2alf in GLUE task.
Table 13: Performance comparison of GPT models using different sparsity methods and LoRA ranks on GLUE tasks. E-SR-STE stands for Extended SR-STE.
Model
Method
LoRA (r)
CoLA
MNLI (m)
MNLI
MRPC
QNLI
QQP
RTE
SST2
Avg
(mm)
GPT2-Small
Dense
0
0
32.4
33.2
66.9
50.3
51.8
49.8
59.3
43.2
GPT2-Small
SLoP:
r=0
0
34.3
34,0
72.5
50.0
48.5
50.0
52.3
42.8
GPT2-Small
SLOPE
== 0.05%
0
34.3
34.1
72.6
49.8
48.8
50.9
52.3
42.9
GPT2-Small
SLOPE
== 2.1%
0
34.3
34.0
71.6
50.0
49.0
52.0
52.6
43.1
GPT2-Small
E-SR-STE
r=0
0
33.6
33.9
$7.1
50.7
50.4
55.2
54.7
42.5
GPT2-Small
E-SR-STE
r= 0.05%
0
33.1
33.6
$7.9
$1.0
50.5
55.4
55.0
42.6
GPT2-Small
E-SR-STE
r= 2.1%
0
33.3
33.5
58.2
51.0
50.5
55.2
55.2
42.6
GPT2-Half
Dense
r=0
0.0
33.9
33.8
53.6
51.1
47.7
56.7
50.6
41.1
R EXTENDED SR-STE AND FST IMPLEMENTATION DETAILS
Before we proceed with the details of Extended SR-STE and FST, we clarify the notations used in our manuscript and the FST paper (66 in table 14
Table 14: Description of Key Terms Description
Term
Sparse Pretraining
Common notation used in SLoPe and FST, indicating use of sparse weights during pretraining-
Dense Finetuning
Notation used in the FST paper, indicating an extended pre- training phase.
Downstream Finetuning
Performance after pretraining concludes, used to finetune the model for specific downstream tasks.
FST
Extended pretraining technique focused dense finetuning.
Extended SRT
Variation of sparse pretraining extended with additional fine- tuning.
We compare SLOPE with FST exclusively for training speedups and memory savings (Table-I-Page 7 and Table-2-Page-S Comparing pretraining quality between SLoPE and FST is less meaningful because the final models produced by these methods differ significantly in the number of parameters. Specifically, FST produces a dense model after sparse pretraining and dense finetuning (99% sparse pretraining 1% sparse + low-rank adaptation), while SLoPE produces sparse model augmented
25

---

Published as a conference paper at ICLR 2025
with lightweight low-rank adapters. The number of parameters in the FST model is approximately 2x larger than in SLoPE, which makes a direct quality comparison imbalance.
We compare SLoPe with Extended SR-STE in terms of model quality, focusing understanding the dynamics between static and dynamic masking under an equal number of parameters. This allows for fair, coople-to-aplie comparison between the methods (iso-pan (5o0-parms).. We refer this method as "Extended SR-STE" because, while the original SR-STE approach was designed for use with SGD, the FST paper extended it support other optimizers.
The FSTand Extended SR-STE code are available in Listing and Listing respectively.
def forward (ctx, x, weight, weight_ weight__ppars__f, bias):
ctx.save_tor_baee sor_backward (input, weight_sparse bias) ctx. for
shape = x. shape
x.view x. x.shape[-1]]
output torch.mm (x, weight_sparse.t0))
if bias is None:
return pntput.vies (*ctx. [: -1]
else:
return output. viex (*ctx. :: :-1], -1) bias
10
11 def backward (ctx, grad_output):
12
grad_output = grad_output half()
x, weight_I, bias save
grad_input grad_weight grad_bias None
15
if ctx, dtx.needd_innuu_gradd00
16
if grad_out.pu..strrdee -- (0, 0, 0):
grad grad_output = torch. like
device=grad
device, dtype=grad_outppe dttpe)
18
grad output grad output. view grad outputt shapel-11)
19
grad_input torch. (grad_output, weight_r.t()) view(
20
ctx shape)
if nex. 1xxxneed_iinut__radlll
x.view (-1, input. (hhp) input.shape(-11)
25
grad_output grad_outpute cinttput.veew11 grad shape
24
grad_weight = torch (to_sparse_ semi sestse__emmssstctueeeeeeeee (rrddooorrddrrrddototpttttttttt (), XVUE24=Truc), x)
if ctx. ctx.necch_inntt_rraaa2])
26
grad_bias grad_output.sumll
27
return grad_input, grad_weight None, None, grad_bias
Listing I: FST Algorithm
def forward (ctx, input, weight, mask, beight_factor]] sparsc_weight weight.clome () detachh sparse_weight [mask] = 0.
ctx. six.save__or_bacckee (Anputt (input, sparse_ weigot_factor mask weight)
output torch, matmul (input, sparse_weight.t())
output output..c..... clone clone
return output
10
@staticmethed
def backward (ctx, grad_output);
12
input, weight, weight_addition
saxee_
13
input_shape input. shape
14
if input. input.dim() = 3:
new w_t_b_atch_ieeee
[nput_shape[0] input_shapee1]
input input..... reshape hawp_batch_siieee -1)
17
grad_output grad_output. reshape (new_bateh_size, -1)
18
grac_output, grad_outpput_mas
prune_column wtw.oooooonsneeeeeeeeeeeeeeeeeee (grad_
19
grighteight = torch
grad_weight +- matmul (grad_output, (), input)
26

---

Published as a conference paper at ICLR 2025
grad, input torch. matmul (grad cutput, weight)
23
grac_input grad_input reshape (input_shape) return grad_input, grad_weight, None
24
Listing 2: Extended SR-STE Algorithm. The weights are stored as dense and are pruned on-the-fly.
COMPARISON OF DEPTH AND WIDTH PRUNING
Depth pruning refers to reducing the number of layers in a model, while width pruning means reducing the size of the weights inside each layer in the model. We have conducted an experiment with depth and width pruning on LLaMA-2-7B (56) and Gemma-2-28 and Gemma-2-96 (53) to compare the effects of depth and width pruning on the performance of the models. The configurations used for the models are summarized in Tables 15 16 Similar to (26), we reduced the aspect ratio of the Up-Sample and Down-Sample modules to half. Please note that this mechanism gives an advantage width pruning methods, as the number of parameters in the Self-A Aiff-Atttntion modules remain intact. Our experiments show that
Table 15: Model Configurations for LLaMA-2 7B Attributes base_emb_dingg 4096
Pruning Method
base aise_num__ lrc__num_quueee query heads: 32
base_num, kv heads: 32
Baseline
base_mi @eve_mlp_ae d__ 1 11008
base_num mum decoder layers: 32
head dim: 128
base juse_emb____ een__dimmbee__ee_eeeeeeeeeeeeeie 4096
base num -uerryguuueeeeeeee heads: 32
base_num_ky fetum_ky_heeet w_y_heads: 32
Depth Pruning
base_mlp_ding 11008
base_nume lrs__num_deccom lel_dddecee_ayaeeee 16 # half the number of layers
head head_dim: 128
base_emb_din dim: 4096
past_num__quer_heaal base num base __te__mlp_____ dim: 5504 # half the number of dimensions
32
@ase_num___ kv heads: 32
Width Pruning
base num decoder layers: 32
head dim: 128
Preliminary retraining loss curves, as shown in Figure 10 suggest no significance difference between depth-pruning and width-pruning during pretraining. Interestingly, in some cases, depth-pruning appears to outperform width-pruning.
T PROOFS
T. LEMMA
Proof. Considering matrix with N M column-wise pruned sparsity pattern, we want to prune the matrix using M sparsity pattern row-wise as well. Let's define random variable as the number of added non-zeros to M row-wise consecutive elements and Y as the number of non-zeros in M row-wise consecutive elements.
M-
Σ
Pr
ili
(12)
Replacing Pr[X = = Pr(y N + in Equation I2 we will get Equation 13 where we used a
change in dummy variable j = N +i.
27

---

Published as a conference paper at ICLR 2025
Table 16: Model Configurations for Gemma-9B
Pruning Method
Attributes
gase_emb__se d_dddddmm_-1 3584
base_num_quees _eads: 16
base_num_ks kv_heads:
Baseline
@ase_mlp_die 1i1p__dmmm/m_ 14336
20 # merged local and global attention
head_dimm 256
gase_emb_ding 3584
_ase_num_querr___e _eadss
Depth Pruning
base_mlp_dee 14336
base num decoder layers: 10 # half the merged layers head dim::ddim220 256
base_emb _mm__dim: 3584
base_num num query peery_eeads: 16
base base_num _um kv heads:
Width Pruning
base_mly aise_mlp_se dim: 7168 # half the number of dimensions
base __m_nuu____ decoder becoder_layers: 20 # merged local and global attention head_ dim: 256
Table 17: Model Configurations for Gemma-2B Attributes
Pruning Method
base_embbddin dim: 2304
base_num_quue et____uueree_e_eeee _eads: 8
base base_num __m kv ___dedd::
Baseline
base_mly dim: 9216
base_num_decoder___
12 # merged local and global attention
head_ding 256
base_embbddie 2304
base_num Baaimmmuerr_miie heads: 8
Dase_num_kv_heate
Depth Pruning
base_mlp_din 9216
base_nume num -eludddeeddrrreee layers: 6 half the merged layers
head dim: 256
dim: 2304
base_num svery_eeads:
base_num num__uum___ <v_hcads:
Width Pruning
base_ base_mlp lalp_mdmmmmm 4608 # half the number of dimensions
base num decoder poooddrrlayess: 12 # merged local and global attention _im: 256
M-N E| Σ P
Σ PriY 5 - N) J=N+1
(13)
Considering the definition of Y. be inferred that random variable Y has binomial distribution with success probability of N' M. As result Equation 14 shows the probability mass distribution of
Y.
(M)
M s (1
M) A N
Pr|Y = ] =
(14)
M
By replacing Equation 14 in Equation
13 will get Equation
15
28

---

Published as a conference paper at ICLR 2025
Cemma-2-2B Loss
2emmaa-2-9B Loss
Dense
Dense
12
Depth Pruning
Depth Pruning
Width Pruning
Width Proning
10
100
LOsE
5000 10000 15000 20000 25000 30000 35000
1000
2000
3000
4000
5000
6000
7000
Step
Step
LLaMA-2-7B Loss
Dense
10
Depth Pruning
Width Pruning
8
Loss
6
4
5000
10000
15000
20000
Step
Figure 10: Comparison of the loss of depth and width pruning methods.
( (1s((1)) 5) N) N=
M Σ
E
(15)
Let's define random variable Z as the added sparsity ratio to the matrix by the extra pruning. Since X was the number of added non-zeros in M consecutive elements, E| y E[X], and hence Equation
Σ (M)/ s (1 (1-1))1-533 M) N J=N+1
M
(M)
E|Z = D(A - D(AR()=
-D(RO)) =
(16)
M
T.2
THEOREM 2.2
Proof. In an optimization problem, we are aiming to find the optimal solution to Equation
17
min Ex [C(,, Wo)]
(17)
When using backpropagation, which is based on the chain rule in derivation, we compute the gradient in Equation 18
Fxx [v [Vx, Fix Vv; LW]
(18)
Let's define random variable M as a uniformly random mask of O's and I's. The mask will be 1 at each point with probability of N. 37- Let's define O A E[M]. O is a matrix of all N
37 ss As result
O @ W NN A for arbitrary matrix A.
29

---

Published as a conference paper at ICLR 2025
-
- Exly, e( T EM o W)]
M N
0 [ oo
(19)
By using the linearity of derivation and expectation operators, we can get the result in Equation 20 which proves the theorem.
M Eas AM W)ll
Vx C(X, W.)]
(20)
N
30


Reviews:
[{"review_id": "f2zY0XLoGq", "reviewer": "Reviewer_ywJx", "summary": "This paper introduces SLOPE, a novel pretraining method aimed at improving the efficiency of large language models (LLMs) by combining sparsity and low-rank approximations. SLOPE employs double-pruning techniques—applying N:M sparsity masks during both forward and backward passes—and introduces lazy low-rank adapters in the final training stages. The double-pruning approach reduces memory and computational overheads, while the lazy low-rank adapters help maintain model accuracy by mitigating the quality loss from sparsity.", "questions": "Please see the weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This is a technical solid article that designs optimized CUDA kernels that jointly optimize Nvidia 2:4 sparse kernels and low-rank calls through efficient tiling and scheduling.\n\n2. SLOPE yields up to 1.25x faster training and 1.54x faster inference on models with billions of parameters, with a memory reduction of up to 63% during training.", "weaknesses": "1. Structured and semi-structured sparsity with low-rank adapters, are not new concepts in the literature (e.g., see Losparse [1]).\n\n2. This article lacks an introduction to related work and baseline methods, such as FST. Readers unfamiliar with the field may find it challenging to follow the content.\n\n3. Could you provide results on additional zero-shot datasets in GLUE beyond MMLU, ARC-c, and OpenBookQA for GPT-2 when comparing SR-STE?\n\n\n[1] Li, Yixiao, et al. \"Losparse: Structured compression of large language models based on low-rank and sparse approximation.\" In International Conference on Machine Learning, pp. 20336-20350. PMLR, 2023.\n\nMinor: \n1. L 430 typo of \"accuracy\"\n\n2. The bold values in the Tables are misleading:\n\nTable 1: Why are some values of FST also bolded?\n\nTable 3: Why are the values of SR-STE not bolded on the MMLU dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lfuUA3d4g", "reviewer": "Reviewer_NKfR", "summary": "This paper introduces double-pruned sparsity, a technique that applies two rounds of N:M sparsity to the weight matrices of a neural network: one for accelerated forward pass and another on the transpose of the already N:M sparse matrix for an accelerated backward pass. The proposed method is combined with LoRA-tuning in the final 0.01 iterations of pre-training resulting in the method termed Double-Pruned Sparse Plus Lazy Low-rank Adapter Pre-training (SLoPE). The authors use efficient tiling of upsample tensors and kernel fusion to reduce overhead, resulting in further boost in training and inference speed. They show that their approach results in speedup and memory reduction of pre-training LLMs of various sizes. They also show their method improves zero-shot accuracy of GPT2-small compared to a sparse pre-training baseline.", "questions": "1-The authors show that their method is faster than FST during pre-training, which is expected since FST dynamically searches for a mask, whereas their method uses a static one, making the training column in Table 1 clear. However, the inference column of Table 1 suggests that their method is also faster than FST for inference. All sparse pre-training methods use the full model for inference. Does this imply that their method uses the sparse matrix for inference (e.g., during zero-shot evaluations)? Then what is the point of sparse pre-training? In the table caption, it’s mentioned that \"the lack of inference speedup in FST is due to the final dense pre-training during the final iterations, resulting in a dense model for inference.\" Could the authors clarify if you have included fine-tuning on a downstream task as inference? and how are the SLOPE values in the inference columns calculated?\n\n2-Is flash attention used for the baselines as well (line 312)?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1-The proposed double-pruned N:M sparsity for both forward and backward acceleration offers a novel perspective on N:M\nsparsity, improving flexibility compared to existing transposable N:M masks.\n\n2-The tiling and kernel fusion solution that integrates LoRAs with sparse weights for enhanced efficiency is interesting.\nThe authors effectively demonstrate the acceleration and memory reduction benefits of their approach across various models, outperforming FST [1].\n\n3- The method achieves performance surpassing SR-RTE[2] and approaches that of a densely pre-trained model in tasks like MMLU, Arch Challenge, and OpenQA.\n\n[1] Accelerating Transformer Pre-Training with 2:4 Sparsity, Hu et all, ICML 2024\n\n[2] Learning N:M Fine-grained Structured Sparse Neural Networks from Scratch, Zhou et. all, ICLR 2021", "weaknesses": "1- The most significant weakness is the limited evaluation of how the proposed sparsity impacts performance. The sparse pre-training method negatively affects performance, and the use of a static sparse matrix further limits flexibility. While the authors claim this does not hinder flexibility, more evidence across a broader range of downstream tasks is needed to support this claim. Given the inherent trade-off between speed and performance, the current experiments do not sufficiently demonstrate the method's effectiveness. Table 3 should include more tasks (tasks in FST[1]) to give the reader a clearer understanding of the impact.\n\n2- Even in the limited downstream task evaluation, some important baselines are missing. The authors put FST in parentheses after SR-STE in the text, although these are distinct methods: FST refers to [1] while ST-RTE is [2]. Zero-shot FST results in their own setting on these tasks are not reported. Additionally, while comparing their method to a larger dense model may be unfair, it would be informative to see how a smaller dense model performs without sparse pre-training.\n\n3- The impact of LoRA fine-tuning in the final iterations appears minimal. The benchmarks (MMLU, Arch Challenge, and OpenQA) are not reported for the LoRA ablations. Since a major contribution of this work is the tiling and kernel fusion solution designed to mitigate issues caused by adding LoRAs, it is concerning that adding LoRAs seems to have little effect, especially based on the current results.\n\n[1] Accelerating Transformer Pre-Training with 2:4 Sparsity, Hu et. all, ICML 2024\n\n[2] Learning N:M Fine-grained Structured Sparse Neural Networks from Scratch, Zhou et. all, ICLR 2021", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XBJWBmV8Ay", "reviewer": "Reviewer_B3ze", "summary": "This paper proposes SLoPE, a Double-Pruned Sparse Plus Lazy Low-rank Adapter pretraining method that improves the accuracy and accelerates training and inference of sparse LLMs. By incorporating a low-rank adapter during the final 1% of pretraining iterations, the model accuracy is well maintained. Experimental results show that the proposed method significantly speedups in both pretraining and inference phases and efficiently reduces memory footprint.", "questions": "Please see above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The authors propose a double-pruned backward pass to improve model quality and reduce mask search overhead. They introduce additional low-rank adapters in the final 1% iterations of pretraining and improving model capacity.\n\n2. The paper is well-written and easy to follow up. The finding of adding low-rank adapter in the last 1% iterations achieving better performance is interesting!", "weaknesses": "1. Why does introducing additional zeros in an N:M pattern during the computation of the input gradient in the backward pass of the first 99% of iterations not severely affect accuracy?\n\n2. The memory footprint for the pretraining should be higher than for inference, so why are the memory footprint reduction multiples similar? In the backward pass, calculating weight gradient involves a dense computation. While double-pruning reduces the computation of input gradients, the overall memory reduction should be more substantial.\n\n3. As discussed in section 2.2, the authors introduce a hyperparameter to balance the memory footprint, computational efficiency, and model quality. In the experiments on the effects of low-rank adapters, ranks are set to 4,16, and 64. Why weren’t larger ranks used? Since the low-rank adapter is only applied in the final 1% of iterations, the computational overhead should be minimal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)