SMT: Fine-Tuning Large Language Models with Sparse Matrices
Authors: Haoze He
Juncheng B Li
Xuan Jiang
Heather Miller
Venue: ICLR 2025 Poster (2025)
Keywords: Parameter-efficient Finetuning
Large Language Model
Large Language Model Systm
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a novel fine-tuning method (SMT) which achieves state-of-the-art performance in parameter-efficient fine-tuning, effectively closing the gap between SMT and full fine-tuning.

Abstract: Various parameter-efficient fine-tuning (PEFT) methods, including LoRA and its variants, have gained popularity for reducing computational costs. However, there is often an accuracy gap between PEFT approaches and full fine-tuning (FT), and this discrepancy has not yet been systematically explored. In this work, we introduce a method for selecting sparse sub-matrices that aims to minimize the performance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine-tuning computational costs and memory costs. We explored both gradient-based and activation-based parameter selection methods to identify the most significant sub-matrices for downstream tasks, updating only these blocks during fine-tuning. In our experiments, we demonstrated that SMT consistently surpasses other PEFT
baselines (e.g., LoRA and DoRA) in fine-tuning popular large language models such as LLaMA across a broad spectrum of tasks, while reducing the GPU memory footprint by 67% compared to FT. We also examine how the performance of LoRA and DoRA tends to plateau and decline as the number of trainable parameters increases, in contrast, our SMT method does not suffer from such issues.

Published as a conference paper at ICLR 2025
SMT: FINE-TUNING LARGE LANGUAGE MODELS WITH SPARSE MATRICES
Haoze He Juncheng Billy Li 1,, Xuan Jiang", Heather Miller!.2
CCamegie Mellon University, Twwo Sigma Investments, University of California, Berkeley (haozeh, junchenl, heather .mi millerr(cs... cmu. edu,
ABSTRACT
Various parameter-efficient fine-tuning (PEFT) methods, including LoRA and its variants, have gained popularity for reducing computational costs. However, there is often an accuracy gap between PEFT approaches and full fine-tuning (FT), and this discrepancy has not yet been systematically explored. In this work, we intro- duce method for selecting sparse sub-matrices that aims to minimize the perfor- mance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine- tuning computational costs and memory costs. We explored both gradient-basse based and activation-ba parameter selection methods to identify the most significant sub-matrices for downstream tasks, updating only these blocks during fine-tuning. our experiments, we demonstrated that SMT consistently surpasses other PEFT baselines (e.g., LoRA and DoRA) in fine-tuning popular large language models such as LLaMA across a broad spectrum of tasks, while reducing the GPU mem- ory footprint by 67% compared FT. We also examine how the performance of LoRA and DoRA A tends to plateau and decline as the number of trainable parame- ters increases, in contrast, our SMT method does not suffer from such issues.
INTRODUCTION
While the inherent generalization capability of Large Language Models (LLMs) is impressive, en- hancing performance on downstream tasks often still necessitates fine-tuning (Ding al., 2022] Chung et al.] 2022). However, as the size of these LLMs increases, there is pressing challenge to optimize the fine-tuning process for better computational efficiency and memory utilization. For ex- ample, fine-tuning pre-trained LLaMA 7B model without CPU offloading requuires at least 58 GB of GPU vRAM-113 13.6 GB for trainable parameters, 40 GB for Adam optimizer states and gradients, and 2 GB for activations. This requirement makes fine-tuning consumer-level GPUs such as the NVIDIA RTX 4090 with 24 GB of memory impractical (Zhao et al. 2024).
To address the prohibitive computational challenges of full parameter fine-tuning, many parameter- efficient fine- tuning (PEFT) methods have emerged over the past two years. LoRA and its vari-
Su et al.) 2023 Wang et al.) 2024b a reparameterize the full model weight into low-dimensional proxy parameter through low-rank adaptation method and successfully reduce both the optimizer memory and com- putational costs. However, even in state-of-the-a -hrt (SoTA) PEFT research, results show a notable performance gap between low-rank reparameterization methods and full parameter tuning across many datasets (Liu et al.) (024a)). Additionally, in this work, we report a less recognized phe- nomenon: low-rank adaptation methods tend experience a performance plateau as the parameter count (rank r) increases.
ants (Hu et al., 2021 Zhao et al. 2024 Dettmers et al. 2024 Liu et al. 2024b a
On the other hand, previous studies have attempted to select a small subset of the parameters fine tune through understanding the internal logic of LLMs.. Some knowledge editing methods, such as Constrained fine-tuning (Zhu et al. 2020), ROME (Meng et al. 2022a), and MEMIT (Meng al.
Equual Contribution.
Moor details about gradient-basel parameter selection methods are in Section 3 and Appendix
A Although some libraries such as Deepspeed can move the optimizer memory costs to CPU, it will also slow down the fine-tuning with extra I/O communication time (Rajbhandari al. 2020 Aminabadi al. 2022

---

Published as a conference paper at ICLR 2025
2022b), have shown that LLMs have memory sections located in distinct layers. These memories could be modified via fine-tuning (Zhu et al.) 2020). These works observed that domain -opeiin knowledge can be distributed separately and sparsely among layers. Motivated by these observa- tions, aiming to narrow the performance gap between PEFT and full fine-tuning, we proposed a Sparse Matrix luning(SMT) approach. We aim to fine-tune the most relevant sparse submatrices for optimal downstream performance. The gradient-based based activation selection method is inspired by Fisher Information (Sung et al.) 2021), and activation-based selection method is inspired by AWQ (Lin et al., 2024) which was previously applied for quantization. After comparing empirical performance of both GW-selection and AW- we opted for gradient- based selection despite AW-selection could have saved us the full backward propagation in warm-up steps. (see Section } for details). Our findings differ from those of (Geva et al. 2020; 2022), as our experiments show that fine-tuning attention mechanisms, particularly the value vectors, is more efficient than fine-tuning other weight matrices, including MLPs, where (Geva et al. 2020: 2022) suggest memory is stored.
In our experiments, our Sparse Matrix Tuning (SMT) approach achieves better performance com- pared to LoRA and DoRA using same amount of trainable parameters. Additionally, SMT narrows the accuracy gap between full fine-tuning overcomes the performance plateau of low-rank adapta- tion PEFT methods, and significantly outperforms LoRA and DoRA while utilizing less than 5% of trainable parameters. Our experimental results show that SMT consistently outperforms SoTA PEFT (including LoRA, DoRA, and SpIEL) methods by 2+ points when fine-tuning popular L.LMs (e.g. LLaMA series base mode( on commonsense reasoning and arithmetic reasoning benchmarks. For layers without selected sub-matrices, SMT freezes these layers, saving all their backward propaga- tion computational costs, parameters update computational costs, optimizer memory costs, and acti- vation memory costs. For layers with selected sub- matrices, SMT reduces the computational costs of backward propagation and parameter updates, as well as the optimizer and activation memory costs, to less than 1% of those incurred by standard full tuning. Below are our additional contributions:
Language Model Anatomy: We investigate the distinct impacts of attention mechanisms versus MLPs (Multi-Layer Perceptrons) in LLMs. Our findings indicate that lne-tuning the V vectors is the most effective among the Q K. and V vectors in attention layers, and that fine-tuning attention layers is more crucial for downstream performance than ML. layers.
Large Language Model System Efficiency: The SMT implementation significantly re- duces the computational costs of backward propagation, parameter updates, optimizer memory, and activation memory during fine-tuning. Our implementation is open source
2 BACKGROUND AND RELATED WORKS
Existing PEFT methods methods can be grouped into three main categories: addition, reparameter-
ization and specification. The Addition category, which involves adding extra adapters (He et al.)
2021] increases model size and slows down inference, so it falls outside the scope of this paper.
Reparameterization category: Many works on parameter-efficee
fine-tuning (PEFT) (Man-
grulkar et al., (022) have aimed to improve efficiency and performance by only fine-tuning lower-
dimensional adapters of model weights. Notable examples include LoRA (Hu et al., 20021) DoRA
(Liu et al., (024a)) QLoRA (Dettmers et al.] 2023), and several other variants (Liu et al. 2024b
Dettmers et al. 2023 Wang et al., 2024b]a). However, the results of these works still indicate a performance gap between PEFT methods and full fine-tuning (FT). Concurrent research (Biderman
et al. 2024) empirically demonstrated that such a gap is difficult if not impossible to eliminate, they also notice the performance saturation issue of LoRA, as we will discuss in Section $4.2
Specification category: Beside low-rank adaptation methods, sparsity-inspired approach is a nat- ural alternative to reduce computational costs and memory footprint by finetuning only subset of parameters. Previous works have also explored sparsity-inspired methods in transfer learning, model pruning, or fine-tuning, including Sparse Increment (ine-TuningSSIFTT (Song et al.) 2023). SHiRA (Bhardwaj et al. 2024), Fisher Mask (Sung et al. 2021), Random Masking (Xu & Zhang
2024]. Lottery-Ticket SFT (LT-SFT) (Ansell et al. (021). ). SpIEL (Ansell et al. 2024). and Diff- Pruning (Guo et al.] 2020). However, SIFT (Song et al. (023), Fisher Mask (Sung et al. 2021), random masking Xu & Zhang (2024). and LT-SFT (Ansell et al. 202I) still require full backward propagation to compute all gradients, offering no speed advantage over full fine-tuning (FT). While
"Base model is not yet instruction tuned
ht : // thub. com http////etooote//sssseeeeeee Batisaeeaaati___ee Xaturiix"uunnng
2

---

Published as a conference paper at ICLR 2025
Frosen
Trainable
m
Pretrained Weight
LoRA
R
Pretrained Weight
Instaated Weight
V- We C Re
V - Wo R
Vc Bes
Blocks
Figure 1: Differences between low-rank adaption method LoRA and SMT. The approach on the left illustrates the adaptation method used in LoRA, while the approach on the right represents the sub-matrix sparsity method utilized in SMT.
SpIEL (Ansell et al. 2024) partially reduces the need for backward propagation, its dynamic pa- rameter selection still necessitates full backward propagation for significant portion of training iterations. In LTT-FTT (Ansell et al. (021) and DiffPruning (Guo et al. 2020), the mask is applied the layer level, without the option selectively fine-tune individual parameters within the layer. Moreover, Sung et al. (2021); Ansell et al. (2021 2024); Guo et al. (2020) require full fine-tuning phases and allocate both the model and the optimizer GPUs, leading substantial GPU mem- ory consumption. The memory costs of Fisher Mask (Sung et al., 2021 are almost the same as the memory costs of the model parameters since the mask is stored as tensors with the same shape as the model parameters. (More details about Fisher Mask are in Appendix [.I]. Furthermore, SIFT (Song) et al. 2023) and SpIEL (Ansell et al.] (024) map discontinuous memory gradients to continuous memory addresses, creating significant time bottleneck. Our work builds existing strategies, but unlike previous methods, our matrix sparsity approach directly leverages task-specific gradient information to automatically adjust within-layer sparsity. This results in better speedup, reduced computational costs, and greater memory savings, as detailed in Section 3aand Section $4.4
Suppose we are given a pre-trained auto-regressive language model P*(2(z) parameterized by d. Each downstream task is represented by a training dataset of context-target pairs: Z (Ig, i........... I.....N where both x; and y are sequences of tokens. Equation (1) dedicates the LoRA(Hu ett al. 2021) fine-tuning process maximize the conditional language modeling objective, which uses a low-rank representation encode task-specific parameters. Specifically, LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture. This is formulated as A = 2 where é represents much smaller- sized set of parameters with ( << (bll. The resulting increment A can be as small 0.01% of the pre-trained weights parameter size dbll in gradient updates. This greatly reduces the number of trainable parameters and the GPU memory requirement while maintaining model performance.
lv| log( Peo-
max Σ
(1.))E
In our work, our proposed Sparse Matrix TuMing(SMT) uses matrix sparsity as the parameter- efficient approach. SMT's case, reusing Equation [ the θ represents the sub-matrices within the sparse weight matrices. SMT only fine-tunes sparse sub- matricet θ instead of fine-tuning all pre- trained weights. Fig. Fig--illlme Iillustrates the differences between weight low-rank adaption method LoRA and our proposed sparse matrices tuning approach SMT. In SMT, we slice the pre-trained weight into N sub-matrices and only fine-tune selected M sub-matrices. The dimension of sub- sub-matrices is I I, the total number of sub-matrices N in pre-trained weight is N update by representing the latter with sparse gradient matrix AWM. Wo AW Wo + AWM, where the number of fine-tuning sub-matrices m < N.
dx
SMT constrains its
Ixl
Since our proposed SMT method focuses on fine-tuning sub-matrices which are most relevant to downstream tasks' performance, identifying these sub-matrices is non-trivial. Our findings extend the observations from LoRA (Hu et al. (021)-Whin) low-rank adapters in LoRA are applied the Q. K. V, and O vectors, the paper does not address the relative importance of components between MLP and attention mechanisms, during fine-tuning. In Section $15], we explore these questions and provide experimental analysis to address them. Additionally, previous works (Zhu et al. 2020 MEMIT (Meng et al. (022b), and
does it determine which among Q K, and V are the most critical
Mooe details about LoRA in Appendix F
3

---

Published as a conference paper at ICLR 2025
(Gea et al. 2020: 2022) indicated that ferw-forward MLP layers of the LLMs most influential. However, through our experiment analysis 5.1 we found that tin-ttuiig attention layers is more efficient for improving downstream performance than MLP layers.
3
METHODOL.OGY
Table I: The experiments involved Full Fine-Tuning, SMT. LoRA, DoRA, and SpIEL on 4x A100 40GB GPUs using data parallel, with a batch size of 16. Communication between the GPU and CPU was facilitated via PCIe-G4.
(4)
Lisear Weight
(b)
(
01
D
X
0t
LLaMA-7B
Orelloe Nerth
PEFT method
#Params %
Time/s
Speedup
Figure 2: (a) A sparse weight matrix W. The green sub- aub-matrices with significant gradients can be updated. (b) Backward propagation calculation for partial gra- dient for weight matrix w. (c) Computation graph in auto-differential systems.
Full Fine-tuning
100
243
SMT
1.26
16.68
14.6%
LoRA
1.26
17,82
13.6x
DoRA
1.27
18.04
13.5x
SpIEI
1.26
25.45
9.6x
Selection phase (warm-up phase): The first step in our method is identifying the most salient sparse submatrices. We compare two strategies for this: Activation-awar selection (AW-selection) and Gradient-aware selection (GW-selection). Recent research (Sun et al. (2024) revealed the pres- ence of massive activations in LLMs. Additionally, AWQ (Lin et al. 2024) found that using ac- tivation distribution, rather than weight distribution, more effectively identifies key weights, sig- nificantly reducing quantization errors by protecting these critical weights. A strong alternative is selecting sparse matrices based Fisher Information (Sung al. 2021) of the weight matrices. Building on empirical Fisher derivation (see Appendix [B.1 for more details about Fisher information and how SMT differs with it), we propose GW-selection model's weight matrices that show the greatest gradient changes during a warm-up phase (Fig. 2]a) the beginning of fine-tuning. The GW-Sclection warm-up phase lasts fewer than 100 iterations, varying depending on the dataset. Appendix B.2 details AW-Selection and Table g provides empir- ical evidence that it significantly underperforms GW-selection. Therefore, for the remainder of this paper, our SMT method selects sparse submatrices using the GW-selection approach identifying top z% submatrices (z=0.5 with greatest gradient changes as critical submatrices.
identify specific submatrices within the
Efficient fine-tuning phase: After identifying the submatrices for fine-tuning, SMT reduces mem- ory requirements by freezing most of the weight matrices after the warm-up phase and storing the sparse weights in compressed format. By implementing optimizer offloading, the warm-up phase of SMT docsn't I cost extra GPU memory. After the warm-up stage, we disable offloading, move optimizers to GPUs, and use the FusedAdam optimizer from the deepspeed library (Aminabadi ct al. 2022 to speed up fine-tuning.
Ad Advantages of SMT: SMT significantly reduces backward computation costs z% (z=0.5) of those in Full Fine-tuning (FT) by calculating gradients for only a subset of the weights during back- propagation. For linear layers in LLMs, where Z = War, the gradients with respect weight matrix
W and input x can be calculated as Equation (
Øl
Vzf(x) az W
Vw
x
07
where 01/Oz is the gradient information from backward propagation in (Fig. b.c). (x) is the gradient matrix and and is the activation in the (Fig b)) (Fig.2b) also illustrates that only partial backward computations are necessary when we update selected sparse matrices. To calculate the sub-matrix gradient (highlighted in yellow), it is only necessary to multiply the yellow row in ål/O /0z with the yellow column in the activation xr Similarly, to calculate the green sub-matrix gradient, only need to multiply the green row in 0l/Oz with the green column in activation x. Note that in backward propagation, only reduce computation when derivative to gradient matrix among as illustrated by the green arrows in (Fig. 2 c). but not other necessary computation. (black arrows)
Forr I.L.MA-7B and LLaMA277B, when the number of trainable parameter is 33685504, the percent of trainable parameter in whole model is 33685504/6738411 1520 ez 0.5%.
4

---

Published as a conference paper at ICLR 2025
Besides. SMT reduces the memory costs of the optimizer gradients to 2% (z=0.5). Since SMT only updates selected sparse sub-matrices, partial gradients are stored to cut the memory costs of the Adam optimizer to 2% (z=0.5). This reduction is crucial because the memory cost of the Adam optimizer is typically twice the size of the model, which often consumes the majority of GPU RAM. SMT also reduces the gradient step computation costs to 2% since only partial gradient steps required.
Furthermore, SMT reduces the activation memory costs for the in forward pass 2% (z=0.5). Since SMT only computes the partial gradient, it saves the relevant portions of activations X neces- sary for the gradient calculation as represented in Equation. 2 In (Fig. 2].), to calculate the green and yellow gradients in the gradient matrix, we only need to save the yellow and green columns of the activation X. reduces the memory costs for the forward pass of the selected linear layer.
In SMT, all the layers except selected Q. K, and V vectors are frozen during fine-tuning. By do- ing this, SMT avoids all the weight backward propagation computational costs, parameters update computational costs, optimizer memory costs, and activation memory costs in frozen layers. The rationale for fine-tuning only the Q K, and V vector is detailed in Section $5.1
By applying sparse sub-matrix fine-tuning, SMT can reduce the fine-tuning memory costs of LLaMA-7B and LLLMMA-7B to less than 20GB and fit the fine-tuning into a 3090 24GB GPU. We also reduce the computation and achieve faster fine-tuning compared with FT and LoRA/DoRA, Section $ 4.4 provides more details.
Implementation: In SMT, we first sum up gradients from the attention linear layers in every single warm-up iteration. The summed-up gradient information is used to identify task-specific sparse blocks. After the warm-up steps, average the absolute values within the sub-matrices, select the sub-matrices with the largest value, and save the indices for the selected sub-matrices. In all of our experiments, we use 1 x 1 256 256 as sub-matrices block size. During the warm up steps, we can apply offload (Rajbhandari et al., 2020) on memory constraint GPU devices. Since SMT requires fewer than 100 warm-up steps in our experiments, it does not become a bottleneck during fine-tuning epochs. SMT updates the model during the warm-up phase to accelerate convergence. Additionally, SMT implements custom sparse linear layer to ensure that unselected gradients are not calculated, saved, and updated (Code Snipper(6). We replace the selected linear layers with these customized sparse linear layers.
The custom sparse linear layer applies a specialized sparse linear multiplication function, integrated into our customized sparse linear layers (Code Snippet This function calculates partial weight gradients based on the input, weight, and selected weight index. It significantly reduces the com- putational costs of backward propagation weight gradients to just 2% (z=0.5) and minimizes the memory usage of the returned partial gradients to only z% (z=0.5).
The specialized sparse linear multiplication function rewrites both forward and backward functions. In the forward pass (Code Snippet 7 of sparse linear multiplication function, only save selected activation & using ctx. save for backward rd (), and in the backward pass (Code Snippet R) customize matrix multiplication calculate the needed partial gradients given partial input and gradient index (shown in Fig. (1)). It is important to note that we do not use Sparse Matrix- Matrix KultiplicationSSPMMM because we concatenate the selected sparse sub-matrices and formed a m X l x / dense matrix as illustrated in right part of Fig. [ This would not costs additional time since memory allocations remain continuous within each sub-matrix Despite employing matrix sparsity, we still leverage the advantages of dense matrix multiplication.
Furthermore, SMT gathers sparse matrix but still leverages dense matrix. SMT customizes the function for gathering trainable parameters. This customized function selectively gathers weight sub-matrices in the Q, K, and vector layers and passes them to the Adam optimizer. By continuing to use the well well-designed F FusedAdam from the deepspeed library (Aminabadi et al. (022)) we maintain the computational speed of dense matrix weight updates. However, our approach reduces the gradient memory costs in the optimizer to just 2% (z=0.5).
"Sparse Matrix- Multiplication is significantly slower than General Matrix Multiply. More details in Appendix G
5

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS AND RESULTS
Model Architecture and Dataset: In our experimental setup, we use open-weight LLaMA-7BB LLaMA-13B, LLaMA2-7B, and LLaMA3-SB models (AT@ Meta 2024). In Subsection 4. 4.2 We perform fine-tuning on the Common Sense Reasoning tasks with 8 sub-tasks, each with a prede- fined training and testing set. We follow the setting of (Hu et al. 2023 Liu et al. 2024a) and amalga- mate the training datasets from all 8 tasks to create the final training dataset commonsensee 170k and conduct evaluations the individual testing dataset for each task. We calculate an average score to encapsulate the overall efficacy. In Subsection 4.3 we perform fine-tuning on Math10p h10K (Hu al. 2023) dataset which includes MultiArith, GSM_8K (Cobbe et al. 20211. AddSub, AQuA, Sing le5q, SVAMP datasets and evaluate the efficiency on their testsets.
Training Framework and SMT Hyper-parameters: We used the DeepSpeed (Aminabadi al., 2022) library for fine-tuning and accelerate (Gugger et al. 2022) library for inference eval- uation. Both training and fine-tuning are using dtype bfl6. All experiments are fine-tuned for epochs. In all our experiments in Section 4 sub-matrices are selected in blocks of size I = 256. We choose this specific sub-matrix dimension because it is the largest common factor of the column and row sizes of all linear layers in the LLaMA series models, using this dimension for slicing avoids remainder issues. We freeze all MLP layers and apply SMT only to the Q, K, and V vectors in the attention mechanism. In Section I. we explain the rationale why we only apply SMT to attention mechanism instead of MLP. At the end of the gradient warm-up iteration, SMT ranks the average absolute gradient values within each sub-matrix and selects those with the highest average values. We determined the sub-matrix selection metric and gradient calculation method through ablation studies, incorporating metrics used in previous research (Sung al., (021). Ablation studies and the rationale of such selection are explained more in detail in Appendix B We apply 100 warm-up
iterations all SMT experiments on Commonsense dataset and apply 25 warm-up iterations to all SMT experiments on Math1 1OK dataset. The number of warm-up up iterations is fine-tuned for each dataset. Detailed ablation studies on warm-up iterations are provided in Appendix H
PEFT Baselines: For state-of-the-arr (SOTA) baselines, we choose to include LoRA (Hu et al., 2021, DoRA (Liu et al. 2024a). and SpIEI. (Ansell et al., 2024). LoRA and DoRA fall un- der the reparameterization category, while SpIEI., like SMT,, belongs to the specification cate- gory. For hyper-parc ppper-parameter settings of LoRA, we follow the instructions suggested by (Bider- man et al.) 2024 Han & Michael Kalajdzievski 2023 Shhhh-ynng.. The LoRA adapters apply to Wg. g: WE,
W Watee W Wups and W Woown. a is determined by 2x rank.
Computational Resources: We conduct our experiments and implement SOTA baselines of LoRA (Microsoft) and DoRA (Shih-yang) to fine-tune LLaMA-7B and LLaMA2-7B model with 4 NVIDIA A100 40GB GPUs and fine-tune LLaMA-13B and LLaMA3-8B model with 4 NVIDIA A100 80GB GPUs Communication between the CPU and GPU is facilitated via PCIe-G4 and communication between GPUs is facilitated via Nvlink-3.
We evaluate the performance of SMT in terms of computational efficiency (wall-clock time speedup), memory usage (analysis for memory complexity) in Subsectiong 4.4 In this section, we mainly evaluate SMT in terms of popular NLP tasks to test its ability to generalize all downstream tasks. In Subsection 4.1 4.2 we evaluate the performance of SMT on 8 tasks in the Commonsense dataset, including Boo 1 Q, P PIOA. SICA, HellaSwag ARC-e. ARC- c, and OBQA, and we calculate an average score to encapsulate the overall efficacy. In Subsection 4.3 we perform fine-tuning on Math108 OK (Hu et al. 2023) dataset which includes Mu 1l i Airriiitt thh GSM_8 & AddSub, AQUA, SingleEq, SVAMP datasets and evaluate the efficiency of SMT on their testsets. Both datasets, Commonsense and Mathi0 OK, focus on the generalization ability of LLMs across different sub-tasks, ensuring that our results are robust. All of the experiments are evaluated using accuracy.
Evaluation Metrics:
4.1 COMMONSENSE REASONING
We evaluate SMT against the state-of-the-an (SoTA) weight low-rank adapter method includes LoRA and DoRA. To ensure fair comparison, fine-tuned model with SMT following the LoRA and DoRA configuration. We ensure all the syper-parameters including batch size, data type, learn- ing rate, and sequence length identical to what was reported in LoRA and DoRA (Hu et al.,

---

Published as a conference paper at ICLR 2025
Table 2: Accuracy comparison of LLaMA 7B, LLaMA 13B, LLaMA2 7B, and LLaMA3 8B with various PEFT methods eight commonsense reasoning datasets. Results of all the baseline meth- ods on LLaMA 7B, LLaMA 13B, LLaMA2 7B, LLaMA3 8B are taken from (Liu al. 2024a). Results of all SMT are obtained using the nyperr-paameters described in (Liu et al. 2024a) under same settings. Bold texts dedicate the performance SMT under the same numbers of param- eters where LoRA, DoRA, and SpIEL achieve the best performance. Blue texts dedicate the best performance of SMT. Please note that the performance of LoRA, DoRA, and SpIEL under larger numbers of trainable parameters can be found in Table 3
Model
PEFT method
AParams
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC=e
ARCac
OBQA
AVG
C1a5GPT((75B)
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
LoRA(Best)
0.83
67.5
80.8
78.2
83.4
80.4
78.0
62.6
79.1
76.3
DoRA(Best)
0.84
69.7
83.4
78.6
87.2
81.0
81.9
66.2
79.2
78.4
LLaMA-7B
(plEL/Best)
0.84
67.7
81.2
78.6
84.0
80.2
78.3
62.8
78.8
76.5
SMT
0.84
68.7
81.7
78.3
91.6
78.8
84.1
68.7
77.4
78.7
SMT(Best)
4,91
72.0
82.9
80.7
93.3
82.4
86.
70.6
83.0
$1.4
Full Fine-tuning
100
69.9
84.2
78.9
92.3
83.3
86.6
72.8
83.4
81.4
LoRA(Best)
0.67
72.1
83.5
80.5
90.5
83.7
82.8
68.3
82.4
80.5
DoRA(Best)
92.4
LLaMA-13B
SpEEL/Best)
0.68
73.2
84.3
81.4
91.2
84.1
83.1
68.8
82.8
81.1
SMT
81.7
93.7
86.7
73.7
85.2
SMT(Best)
4.91
72.6
86.
81.9
95.0
86.1
88.2
77.1
87.4
84.3
LoRA(Best)
0.83
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
LLaMA2-7B
SplEL/Best)
DoRA (Best)
0.83
0.42
70.5
72.0
83.1
80.6
79.9
80.8
85
89.1
83.4
83.0
84.5
81.2
65.8
71.0
81.8
81.2
80.5
78.3
SMT
80.8
74.0
81.0
SMT(Best)
4.91
72.6
85.2
82.0
94.4
85.7
87.8
74.5
85.0
83.4
Full Fine-tuning
83.4
78.7
74.7
83.4
LoRA(Best)
0.70
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
LLaMA3-8B
DoRA(Best)
79.9
95.5
90.5
80.4
(peEL(Best)
0.70
72.1
83.6
80.0
91.8
85.4
91.2
76.8
80.8
82.7
SMT
0.71
75.7
88.4
81,4
96.2
88.2
92.7
83.2
88.6
SMT(Best))
3.01
75.1
89.9
82.4
96.3
88.8
92.6
82.8
89.6
87.2
2021
Liu et al. 2024a). We re-implemented LoRA and DoRA and achieved their best performance
reported in (Liu al. 2024a).
Table 2 demonstrates that SMT consistently surpasses baseline methods across ILaMA-7B, LLaMAI3B, LLaMA2-7B, and LLaMA3-8B. Notably, by overcome plateau phenomenon, SMT further enhances accuracy of DoRA by 3.0%, 2.8%, 2.9%, and 2% on LLaMA-7B, LLaMA-13B, LLaMA2-7B, and LLaMA3-8B respectively. Notably, LoRA and DoRA will not achieve better per- formance with larger trainable parameters and exhibit the plateau phenomenon. In Subsection 4.2 we report and demonstrate the plateau issue in LoRA and DoRA and demonstrate S.MT overcomes this issue. Moreover, by fine-tuning less than 5% of all parameters, SMT achieves similar accuracy performance of full fine-tuning while speedup 14.6x (speedup details Table [ and save 99.5% of optimizer eeemory(memory bottleneck in fine-tuning, details discussed in Section 3)
SMT can also consistently surpass LoRA and DoRA under the same number of trainable parame- ters where LoRA and DoRA A achieve the best results, SMT can surpass their performance and also outstrip ChatGPT-3..5turbee LLaMA3-8B. LLaMA- 13B, and LLaMA-7B by 1.3%, 1.6%, 0.9%, and 0.3% respectively, under their best performance trainable parameter number.
For instance, SMT consistently surpasses DoRA on LLaMA2-7B
4.2 PLATEAU IN WEIGITT LOW RANK ADAPTION METHOOSS
In Table 3 we scale up the model size and presents how the performance of LoRA, DoRA, and SpIEL will be under larger number of trainable parameters. The corresponding visualization is provided Fig. in Appendix F We reimplement all the experiments of LoRA (Microsoft), DoRA (Shih-yang). and SpIEL (Ansell er al.) using their official repository and followed their recommendation of syper-parameters to achieve best performance under every single trainable pa- rameter size. We observe that for SpIEL, DoRA, and LoRA models, with some larger ranks, their performance slightly degrades. However, SMT continues improving its performance when we scale up the trainable parameter size. When we scale up the trainable parameter size to 4.91 %, SMT
Results of ChatGPP-3.S-turfee reported in DoRA (Shih-yang)

---

Published as a conference paper at ICLR 2025
significantly surpass DoRA by 3.8% and 4.9% on LLaMA-7B and LLaMA-2-7B fine-tuned mod- els. We postulate that such plateau phenomenon of LoRA or DoRA is due to their lossy low-rank approximation of the full weight information (includes lots of noise), whereas on most prominent submatrices (contains less noise) and remains full rank gradient updates for the selected portion, making SMT performs better. SpIEI selects fixed ratio of parameters from every layer. In contrast, SMT selects the most significant parameters, allows a different ratio of trainable parameters each layer, and allocates more trainable parameters more critical layers.
SMT focuses
Table 3: Accuracy comparison of LoRA, DoRA, and SMT under different scaling of trainable parameters on commonsense datasets. Given certain base model and PEFT method, we gradually increase the number of train- able parameters each line from left to right. On each line, the best performing model has
Table 4: Fine-tuned LLaMA-7B model performance on Common sense. AVG dedicates the average test score of eight subsets among Commonsense. MLP% and Attention% presents the percentage of trainable parameters apply to MLPs and attention mechanisms respectively.
Method
1.26
3.73
LoRA
76.4
75.0
75.3
LLaMA-7B
SpIEL
DoRA
77.5
72.6
78.4
77.4
78.2
76.0
77.3
76.8
77.2
77.5
77.6
76.4
Model
MLP
Attention
AVG
SMT
77.3
78.6
79.2
80.2
80.8
81.4"
0.84
76.7
LoRA
76.5
77.6
78.4"
77.6
77.3
77.0
SMT(0.4%%)
77.3
LLaMA2-7B
SpIEL
77.4
77.9
79.2
78.2
78.3
78.8
LLaMA-7B
0.21
0.63
77.8
DoRA
80.5
79.7
78.8
77.6
76.8
0.84
78.7
SMT
81.1
81.8
81.7
82.2
82.8
83.4"
4.3
OTHER DATASET
To ensure our findings above are generalizable, we further examine the performance of SMT un- der arithmetic reasoning dataset, Math10K (Hu et al. (023). Math10K dataset has six subsets including GSM8k, S SingleRg, SVAMP, Mu aelttirttti AddSub, and AQuA. More details about Math1 Ox dataset can be found in Appendix D To ensure a fair comparison, we follow the open source hyper-parameter instruction (Hu et al. 2023) to achieve best performance for LoRA and Dora, and apply the same typer-parameters to SMT while only fine-tune the learning rate. Table 5 reports the performance LoRA, DoRA, and SMT on the Mathl OK datase
9 We can observe that SMT surpasses the best achievable performance of LoRA and DoRA by 1.3% and 1.1% respectively using the same amount of trainable parameters. In addition, by scaling up the trainable model size to 1.26%, SMT achieves better performance and surpasses the best performance of LoRA and DoRA by 2.5% and 2.3% respectively.
Table 5: SMT, LoRA and DoRA reproduction, and experiment results on Math1 1 OK dataset.
Model
PEFT method
#Params%
GSM8k
SingleEq
SVAMP
MultiArith
AddSub
AQuA
AVG
LoRA(Best)
0.86
35.4
83.2
52.1
92.8
83.4
18.6
60.9
LLaMA-7B DoRA
0.86
35.2
83.7
51.8
92.8
82.8
20.2
61.1
SMT
0.86
34,2
84.6
53.6
91.5
85.8
23.6
62.2
SMT(Best)
1.26
35.6
85.3
54.8
93.4
86.8
24.2
63.4
4.4 MEMORY AND COMPUTATION SAVING: SMT vs. LOW-RANK ADAPTION METHODS
SMT is more computational efficient than weight low-rank adaption method when the number of trainable parameters are the same, weight low rank adaption methods need to maintain addi- tional adapters, which require additional forward computation. For instance, since LoRA maintains adapters A and B, and the forward propagation is:
= Wor + AW, = Wox + BAr
where the term BAr requires additional forward propagation calculation, which is cut off in SMT. Regarding memory costs, since SMT does not require additional low-rank adapters A and B, SMT achieve lower memory costs than LoRA and DoRA under the same amount of trainable param- eters setting. We illustrate this in Fig. Taking the popular LLaMA-13B model as an example,
'In accordance with the special announcement for Math1 OK (Hu et al.) we include training set evaluation results from the Mult. Arith th, AddSub, and $i elg datasets, as well as test set evaluation results from the GSM8k, SVAMP, and AQUA datasets. AVG denotes the average accuracy all evaluations.
8

---

Published as a conference paper at ICLR 2025
since the model size is approximately 25 GB, if fine-tune 1% of parameters, SMT can potentially save 250MB GPU memory compared to LoRA and DoRA. In Table we provide the fine-tuning time costs for SMT, Full Fine-tuning, LoRA, and DoRA. SMT achieves an 14.6x speedup compared to Full Fine-tuning and outperforms both LoRA and DoRA. Additionally, compared to SpIEL the specification category, SMT is 1.5x faster due its more efficient selection phase, more flexible parameter selection, and improved trainable parameter memory management. We conducted time profiling by averaging the fine-tuning time every 10 iterations over 1000 iterations, following a 500- iteration warm-up up period. Full fine-tuning utilized offload settings to accommodate the LLaMA model, which employs the Adam optimizer, within 40GB GPUs. SMT offers greater computational and memory efficiency, though these are secondary benefits compared to its primary focus.
FURTHER DISCUSSION 5
5.1 ATTENTION VERSUS MLP
In order study what components are more critical for LLM's downstream performance during fine-tuning, we conduct ablation studies that compares MLPs vs. attention layers by adjusting the ratio of their trainable parameters respectively. We apply SMT and fine-tune 0.86% of parameters on LLaMA-7B using Commonsense dataset. In Table we present four experiments. In the first row, all trainable parameters are allocated to MI.Ps. In the second row, both MI.ss and Q. V vectors from attention mechanisms receive 0.43% of trainable parameters. In the third row, 0.62% of trainable parameters are assigned to Q. K, V vectors from attention mechanisms and 0.21% to MLPs. In the fourth row, all trainable parameters dedicated to Q K, V vectors from attention mechanisms. To guarantee a fair comparison, all the other hyper- paremmarreetess and settings are the same among these experiments.
In these experiments, allocating Y% of trainable parameters to MLPs or attention mechanisms means ranking the average absolute gradient values of each sub-matrix within the MI.s atten- tion mechanisms and selecting those with the highest average values until the number of parameters reaches Y%. The results reveal significant performance gap between the first and fourth rows. The more trainable parameters we allocate to attention mechanisms, the better the fine-tuned model performs. When all SMT trainable parameters are applied attention mechanisms, the model out- performs the one where all parameters are allocated to MLPs by 2.0%. Our empirical findings challenge previous assumptions (Zhu al.) 2020; Meng et al. 2022al Geva et al., 2020 2022) that the memory sections of large language models are primarily located in feed-forward MLP layers.
5.2
VECTOR VERSUS Q, VECTOR
(a)
LLama-7B
(b)
LLama2-7B
PrPr
0.008
Prij
Magnitudes
Magnitudes
0.006
0.006
0.002
0.002
0.000
15
20
Layers
Layers
Figure 3: The magnitude of the gradient for the Q, K, and V vectors at each layer in LLMs.
Based on our observation that fine-tuning attention is more efficient during fine-tuning, in all of our SMT experiments in Section$ 4 we only allocate SMT trainable parameters to Q, K. V vectors from attention mechanisms. We rank the average absolute gradient values of every single sub-matrix within attention mechanisms and select those with the highest average values until the parameter tio limit is reached. Counter-intun intuitrrn we observed that the trainable parameters are predominantly assigned the vectors. As shown in Fig. 5 95.17% of the trainable parameters automatically

---

Published as a conference paper at ICLR 2025
assigned the V vectors by SMT. Fig. Aindicates that all V vectors have trainable parameters, while 22 out of 32 Q vectors and 21 out of K vectors are completely frozen.
experimented with assigning all trainable parameters to only K, or only Q. or only vectors, and fine-tuned 0.86% of the parameters on LLaMA-7B using the Commonsense dataset. Table presents four additional experiments where fine-tuned 0.86% of the parameters of LLaMA-7B using SMT on the Commonsense dataset. the first three rows, all trainable parameters allocated to the K vectors, Q vectors, and V vector, respectively. In the fourth row, the trainable parameters are assigned to Q, K, V vectors directly and allocated by SMT automatically. The trainable parameters are distributed among the K, Q, and vectors, as detailed in Fig. 5 with the trainable states of the QKV layers shown in Fig.
In ablation experiments,
Figure 4: A visualization of trainable Q K, V layers when fine-tuning 0.86% trainable parameters LLaMA-7B. LLaMA-7B has 32 layers of MLPs, each contains a Q vector, a K vector, and V vector. White layers are frozen and green layers contain trainable parameters.
Q Proje x, 2.8%% Proje 2.02%
Proje 95.77%
Figure 5:
Distribution
of trainable parameters among Q, K.
Table 6: K SMT, Q SMT, and V SMT assign all trainable parameters to only K, or only Q, or only V vectors respectively, and fine-tuned 0.84% of the parameters on LLaMA-7B using the Common- sense dataset. QKV SMT assign all trainable parameters to QKV vectors and select sub-matrices automatically.
Param location
#Params
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
K SMT
0.84
65.5
79.1
76.2
88.3
73.2
80.3
60.8
68.0
73.9
LLaMA-7B
SMT
0.84
65.7
79.3
75.5
88.2
72.5
80.1
59.6
72.5
75.3
V SMT
0.84
68.7
82.1
78.1
91.6
78.8
83.0
68.7
77.2
78.5
QKV SMT
0.84
68.7
81.7
78.3
91.6
78.8
84.1
68.7
77.4
78.7
The results show a significant performance gap when comparing the allocation of all trainable pa- rameters to the V vectors versus the Q and K vectors. Assigning all parameters to the V vectors outperforms the vectors by 4.6% and the Q vectors by 3.2%. These observations suggest that fine-tuning the V vectors is the most efficient compared to Q and K in this process; it also hints that SMT is able to effectively select sub-matrices containing crucial memory sections.
To provide further insight into the importance of V vectors, visualized the median magnitude of the absolute gradients for each vector across layers, as shown in Fig. 3 The results, averaged over 1000 iterations, are based on two models: LLaMA-7B and LLaMA2-7B. We observe that the gradients of the V vectors are significantly larger than those of the and K vectors, with the vector gradients being up to 10 times greater most layers. This larger gradient leads more substantial updates, making fine-tuning the V vectors more effective.
In Appendix C we show that the smaller magnitude of gradient for the Q and vectors caused by the design of scaling in the attention mechanism, where Vdz is not sufficiently large during pre-training.
CONCLUSION
Our empirical results suggest that fine-tuning attention layers are more critical than MLPs for down- stream performance; V vector the most influential vector for performance among Q, K. vectors. Overall, our SMT method is an appealing alternative to LoRA especially for practitioners with lim- ited compute resources, since SMT could achieve better accuracy than other SoTA PEFT methods (LoRA, A, DoRA and SpIEL) with the same amount of trainable parameter. SMT is also strong al- ternative vs. full fine-tuning, since we showed that the gap between SMT and full tuning very narrow.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
We sincerely thank Sida Wang for her valuable contributions to this project, including developing and finalizing the implementation methodology, generating critical experimental results that strengthened our conference presentation, and supporting the open-source release of our codebase. We also extend our gratitude Qianou (Christina) Ma, Chenyang Yang, Chen Liu, and Yu (Ivy) Yang for the suggestion in paper writing and their valuable feedback.
This research used the Bridges-2 at Pittsburgh Supercomputing Center(PSC) and Delta advanced computing. Pittsburgh Supercomputing Center is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. The Delta advanced computing is supported by the National Science Foundation (award OAC 2005572) and the State of Illinois. Delta joint effort of the University of Illinois Urbana-Champann and its National Center for Supercomputing Applications.
Though Heather Miller and Juncheng Billy Li are employees of Two Sigma Investments, this work was performed independently from Two Sigma Investments.
REFERENCES
Al@Meta. Llama 3 model card. 2024. URL http::://githbb..cmm/m/ 1l---lamaaa lama/
1lama3/blob http///ooommaia//MM..... CARD md
Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, al. rrepspeed-inference enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 1-15. IEFE, 2022.
Alan Ansell, Ivan Vulié, Hannah Sterz, Anna Korhonen, and Edoardo M. Ponti. Scaling sparse fine- tuning to large language models. https: com/ wwoom//dudaugge/eee 11m Accessed: May 20, 2024.
Alan Ansell, Edoardo Maria Ponti, Anna Korhonen, and Ivan Vulić, Composable sparse fine-tuning for lioss-lingual transfer. arXiv preprint (rXiv:2/00.0756/ 2021.
Alan Ansell, Ivan Vulié, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401. 16405, 2024.
Kartikeya Bhardwaj. Nilesh Pandey, Sweta Priyadarshi, Viswanath Ganapathy, Shreya Kadambi, Rafael Esteves, Shubhankar Borse, Paul Whatmough, Risheck Garrepalli, Mart van Baalen, et al. Sparse high rank adapters. Advances in Neural Information Processing Systems, 37:13685- 13715, 2024.
Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jen- nings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673 2024.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction finettuund language mod- els. arXiv preprint arXiv:22 10. 1416, 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2 10.1468,, 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: efficient finetuning of quantized Ilms (2023). arXiv preprint arXiv:2305.1433/ 2023.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.

---

Published as a conference paper at ICLR 2025
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin
Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06900 2022.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. arXiv preprint arXiv:2012.149133 2020.
Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. arXiv preprini arXiv:2203. 14680, 2022.
Nazli Goharian, Ankit Jain, and Qian Sun. Comparative analysis of sparse matrix algorithms for information retrieval. computer, 2:0-4, 2003.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan, Accelerate: Training and inference at scale made sim- ple, efficient and adaptable.
com/hgg wawaae/ aaaffaceaccccc acce 1 eraate 2022.
Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff prun- ing. arXiv preprint arXiv:2012.07463, 2020.
Daniel Han and Michael, Continued pretraining with unsloth. https:://mmsstth.aa//// contpretraining Accessed: May 20, 2024.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatricke and Graham Neubig. Towards a unified view of parameter-efficient transfer learning, arXiv preprint arXhv:211000336, 2021.
Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. Learning to solve arithmetic word problems with verb categorization. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 523-533, 2014.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685, 2021.
Zhiqiang Hu, Lei Wang. Yihuai Lan, Wanyu Xu, Ec-Peng Lim, Lidong Bing, Xing Xu, Soujanya Po- ria, and Roy Wei Lee. Math dataset. htttps://github com AGI -ee--Edgguuuunne
L.M ALMM-Adapters/tooo wtwpeeerrreeeeeemmmmmemmae wtw.innntaarrradeeeeeeeee
Accessed: May 20, 2024.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. Llm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. arXiv preprint arXiv:2304.0193. 2023.
Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiw:2312.03732, 2023.
Rik Koncel-Kedziorski. Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. Parsing algebraic word problems into equations. Transactions of the Association for Com- putational Linguistics, 3:585-597, 2015.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device Ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale gener- ation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv: 2017.
Shih- Yang Liu, Chien- Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. arXiv preprint arXiv:2402.094 09353, 2024a.
12

---

Published as a conference paper at ICLR 2025
Zequan Liu, Jiawen Lyn, Wei Zhu, Xing Tian, and Yvette Graham. Alora: Allocating low-rank adaptation for fine-tuning large language models. arXiv preprint arXiv:2403.16187, 2024b.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-effiieet fine-tuning methods. /ttpp::/ggtthe comm/nuggingfaceeppett 2022.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual asso- ciations in gpt. Advances in Neural Information Processing Systems, 35:1 15:17359-17772 2022a.
Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. arXiv preprint arXiv:2210000222): 2022b.
Microsoft. Lora. https://
Accessed: May 20, 2024.
Carlos Ordonez, Yiqun Zhang, and Wellington Cabrera. The gamma matrix to summarize dense and sparse data sets for big data analytics. IEEE Transactions on Knowledge and Data Engineering,
28(7):1905-19) 1918,, 2016.
Arkil Patel, Satwik Bhattamishra, and Navin Goyal. Are NLP models really able to solve sim- ple math word problems? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080-2094, Online, June 2021. Association for Computational Linguistics. doi: 10. 10.11865511/2222 naacl-main URL
http:a/w......o...o/e.eeaaaaaaaa nwaclnaalaaaaaaa 168.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Perfor- mance Computing, Networking, Storage and Analysis, pp. 1-16. IEEE, 2020.
Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint 108.01160..,1141,, 2016.
L. Shih-yang. Dora.
Accessed: May 20, 2024.
Weixi Song, Zuchao Li, Lefei Zhang, Hai Zhao, and Bo Du. Sparse is enough in fine-tuning pre- trained large language model. arXiv preprint arXiv:2312. /175, 2023.
Yusheng Su, Chi-Min Chan, Jiali Cheng, Yujia Qin, Yankai Lin, Shengding Hu, Zonghan Yang. Ning Ding, Xingzhi Sun, Guotong Xie, et al. Exploring the impact of model scaling on parameter- efficient tuning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pp.
2023.
Mingjie Sun, Xinlei Chen, J Zico Kolter, and Zhuang Liu. Massive activations in large language models. arXiv preprint arXiv:2402.17762, 2024.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. arXiv preprint arXi::2407.05000 2024a.
Zhengbo Wang, Jian Liang. Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint 2xXv:24477.18242 2024b.
Jing Xu and Jingzhao Zhang. Random masking finds winning tickets for parameter efficient fine- tuning. arXiv preprint vrXiv:2405.0256. 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- 11m training by gradient low-rank projection. arXiv preprint arXiv:2403.033 03340303307. 2024.
Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363, 2020.
13

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS WITH CODE SNIPPETS
LinearLayer_ iimerraaee..errissfpssttttttooo Module):
def (sel,, weight, bias-tione, index_list [I]: Super(LinearLayer_MMM Harrssssss self)._ weigh weight
weight
passing weight optinizer
self. weight. wwight.. requirereqi False
self.bias bias
salf, self.index, List Andex_List
rointain
'selected_wee veight"
self. DAff.selected_weige torch. torch.enpty empty(teniindex_Lu Block_dimension, Block_dinenssoo) weiht
True
project originat weight peraneters new traineble paraneters 'selected_xeighh"
for con(inexx_listte listl):
index index_List(i)
self.selected, weight Block dinension: Black dimension Block, dimension, :]
welf.weight. [index(0) Block dinension: index Block dirension Block_dirension
Block dimension.
index
dimension: index Block dinension
dinension]
self. self. welectee_ Parameterisell selifsseeeeeeeeeeeeeeeee
apply specialized sperse Linear cultiptication function
appyarZ.apply
Figure 6: Implementation of customized sparse linear layer.
tinerr/torcch.uutograd.
tstaticnethod
forwardictx, input, welghted,,eiggt,, natrix_index_List weight):: If maintain partial. input In 'input_list"
isput_list
for index in natrix, index_ list: Lnsut_List: iippnt i, index(1)-@lock_eine) eineesiiockii index(1)=Allcckdieennieee 3
inputt Input__ist') sub-matrics indext "ostrric_ideer__istt""" propagation
ctx, Listl input_List
betrix_index__liss
ctx.save__oo
outgut torch, natnul/ingut, teight.t(1)
return output
Figure 7: Implementation of customized forward in specialized sparse linear multiplication function.
class LinearZ1 terch, autogred, Function):
@steticwthedd
backvars
fetch purtial inputt (ist sub-eatrics Inder d'eatrox Mise"1)
colcalate partial gradient
"gradtyeiihh"
grighte tarch, empty Lnpot Lipptt__ttttt List) Black dimersion dinanniian lasgel List)::
Index = natrix Lnfex, listil)
weightight Block simessio::: 1 Block dirersion stock Block dinession, il torch mach wwt..ovtttttertteeeeee 2, 13%
(s, index(0) Sinersion index Bleck_dimmiinn Black dinemainn :l, Aipot_tis dimns) colestate di/i0 for backukod propagatice
grai_aapet sorch.attwut setpat, weight)
reture pradtient for activation solected cotrictr return Seput,
Figure 8: Implementation of customized backward in specialized sparse linear multiplication func- tion.
14

---

Published as a conference paper at ICLR 2025
B SUB-MATRICES SELECTION
SMT ranks the average absolute gradient values within each sub-matrix and selects those with the highest averages. The rationale behind this selection process is to enable SMT to automatically identify sub-matrices containing memory that is most relevant to downstream tasks. During fine- tuning, the absolute gradient values can indicate the relevance a block to these tasks hence requires more tuning. By averaging the absolute gradient values within each sub-matrix, we can determine the importance of the sub-matrix to specific downstream tasks. In Appendix [B.1] we further discuss why calculate the average absolute gradient values, and the differences between Fisher Information (Sung et al. 2021) and SMT.
B. FISHER INFORMATION SMT
What is True Fisher Mask and why is it crucial? In the context of neural networks, Fisher informa- tion (Sung et al. 2021) measures how much each parameter affects the model's predictions. is crucial concept used to determine the importance of parameters for the task at hand. The paper uses Fisher information to identify which parameters to update during training. The Fisher information matrix Fg is introduced as follows:
Ex DrL D (po(y)z) [l = 8 8 FGd + 0
(4)
where pe represents the output distribution of the model, given input z and parameter vector represents the probability of the model predicting class y given input æ; represents a small perturbation applied the parameter vector 0. It represents slight change in the parameter values; Fg is the Fisher information matrix, which quantifies how sensitive the model's predictions are to changes in each parameter 0; (po(poyy)a)) pe+s(y)<z)) is The Kullback-Leibler (KL) divergence, which measures how much the model's output distribution changes when the parameters are per- turbed by ò; ((83) represents higher-order terms, which become negligible as d - O. The Fisher information matrix Fg is further defined as:
Fg E Expp(a) [Ey-ps(tlz)) Vo log (y Vo log
(5)
where (xpp(()) represents The expectation over the input data distribution p(r), which captures how much variation exists across different inputs; Ey-ro (ul=) represents the expectation over the output distribution pa (y (x) of the model, given the current parameter values 0 and input æ;; Va log ps (ylz) (x) represents the gradient of the log-likelihood of the model's output with respect to the parame- ters This measures how sensitive the model's predictions are to changes in the parameters. Ve log po (y).x) represents the transpose of the gradient, creating matrix that measures the cor- relations between different parameters in terms of their effect on the model's output. This matrix captures how much each parameter influences the model's predictions, and the larger its value, the more important that parameter is for the task.
What is empirical Fisher Mask and how is it different with true fisher mask? (Sung et al. 2021 distinguishes between the true Fisher information and an approximation called the empirical Fisher. In the empirical Fisher approximation, instead of sampling from the model's output distribution pe ())æz) the known ground truth labels y are used directly. The empirical Fisher approximation is defined as:
N
F
N Σ
V o log po()
(6)
The empirical Fisher can be more computationally efficient because it avoids sampling from the model's 'ss predicted distribution. is used in (Sung et al.) 2021) as a heuristic for parameter impor- tance and is found to perform similarly to the true Fisher in practice. To construct the sparse mask based on the empirical Fisher, the top-k parameters with the highest Fisher information are selected. Specifically, the mask selects parameters 0 such that:
15

---

Published as a conference paper at ICLR 2025
0, {0, (0, sort(Fo)ps}
Where k is the number of parameters to be updated based on the desired sparsity level, and sort(Fa)js) represents the k-th largest Fisher information value. How does Empirical Fisher Mask Calculated? Fisher information (Sung et al. (021) uses the top-k accumulation squared gradi- ent values identify important parameters, which mathematically equivalent to our approach of using the average absolute gradient value SMT. Fisher information requires a fisher mask, my, defined as below:
{1
if Fo, 2 sort(Fe) otherwise
MFJ
0
This mask is applied as: 0masked mp O 0.
What is the differences between Fisher Mask and SMT? Fisher information (Sung et al. 2021) uses the top-k accumulation of squared gradient values identify important parameters, which mathematically equivalent to our GW-Selection approach of using the average absolute gradient value in SMT as following:
loggo(y1(z.)
where F. & is the SMT calculation of gradient information, using the mean absolute gradient value for each sub-matrix parameter 0; is The number of data samples; x; is The i-th input data sample; V: is The ground-truth label corresponding the i-th input; V Pa (a(w)a))) is The gradient of the log-likelihood of the model's output with respect to the selected sub-matrix parameter 0 for sample (1a, ys). Here, take the absolute value of this gradient.
Since [Sung et al. 2021 needs to store the fisher mask, my, for all parameters, the memory costs of Fisher Mask almost the same the memory costs of the model parameters because the mask is stored as tensors with the same shape as the model parameters. However, instead of storing mask with large memory costs, SMT deletes mask by projecting trainable parameters dense tensors. This is crucial since the large model size leads to memory bottleneck for GPUs. In addition, to calculate Fe. Fisher Mask requires full backward propagation and calculate gradient for all param- eters 6. while SMT only requires partial backward propagation. SMT only needs calculate the gradient/ smtt& á selected parameters Ô within sub-matrix. In every fine-tuning iteration of Fisher Mask (Sung et al. 2021), full backward propagation needs to be done. This will lead to signifi- cant computational costs and slow down the fine-tuning. However, by customizing linear layers and linear functions, SMT can do partial backward computation and speedup the whole fine-tuning.
Additionally, in Table 7 dient information. Our experimental results show that using the average absolute gradient value is the most effective approach for selective tuning within SMT framework.
further evaluate four different sub- sub-matrix selection methods using gra-
Table 7: SMT experiment results using different sub-matrices evaluation metrics Common- Sense dataset. This table aims to perform an ablation study to find which sub-matrices evalua- tion metrics is the best. All experiments fine-tune 0.84% overall parameters in Q, K, V vectors. SMT-1 setting: abs().mean() calculation. SMT-2 setting: mean().abs) calculation. SMT-3 setting: (bs)),,um(dim=(,, 3)) calculation. SMT-4 setting: sqrt(sum(abs() ** 2, dim=(1, 3))) calculation. mean().abs) is the best strategy.
Conclusion:
Model
BoolQ
PIQA
SIQA
HellaSwag WinoGrande ARC-e ARC-c OBQA
AVG
SMT-1
66.5
79.6
73.8
86.2
72.8
80.2
61.2
70.1
73.80
SMT-2
68.7
81.7
78.3
91.6
78.8
84.1
68.7
77.4
78.66
SMT-3
67.2
81.2
75.8
86.8
76.2
80.8
62.8
70.8
75.20
SMT-4
68.2
81.2
75.8
87.4
76.2
81.3
63.2
70.8
75.51
16

---

Published as a conference paper at ICLR 2025
B.2 AW-SELECTION vs. OW-SELECTION IN SMT
In large language models (LLMs), not all weights contribute equally to the model's performance. A small fraction of the weights, referred as salient weights, are significantly more important. We propose two salient weights selection methods in warm up phase, AW-Selection and GW-Selection, and discuss them in this section.
AW-Selection is inspired by AWQ (Lin et al. (024) which proposes a method to identify these weights by using the activation distribution of the neural network, instead of just considering the magnitude of the weights. How use AW-Selection to select trainable parameters? In the follow- ing, we explain how AWQ uses the activation information to select these salient weights.
Let us define the forward pass of a single layer in neural network. Given input activations x and weight matrix W, the output y is computed as:
y Wx
Each column of the weight matrix W corresponds to a specific channel neuron in the network, and each weight channel interacts with the input activations x to produce the output.
The key insight behind AW-Selection is that weight channels associated with larger activation magnitudes are more important for the model's overall performance. These channels process more important features of the input, and therefore, their corresponding weights need to be selected as trainable parameters.
The importance of a weight channel W; can be measured by the magnitude of its corresponding activations. Mathematically, the activation magnitude for channel can be represented as:
Activation Magnitude for channel = E [|xx||12
where x; is the activation corresponding to weight channel W;, and IE [||x]]2]] denotes the expected value of the L2 norm of the activations. Intuitively, channels with larger average activations are more important because they carry more information or contribute more heavily to the output.
To identify the salient weights, the AW-Selection involves the following Salient Weight Selection Strategy: (1) (1)Measure Activation Magnitude: For each weight channel W. the corresponding activation magnitude measured by calculating the average I.2 norm of the activations over a cali- bration dataset. (2) Rank Channels by Activation Magnitude: Once the activation magnitudes are computed for all channels, the channels are ranked based on their activation magnitudes. Channels with higher activation magnitudes are ranked higher. (3)Select Salient Channels: Based on the ranking, a small fraction of channels (typically 0.1% to 1%) are selected as salient channels. These the channels that correspond to the most important features in the model. Since AW-Selection only requires activations information, the forward propagation is sufficient to gather activations. Since AW-Selection doesn't 't require backward propagation and optimizer, it leads less computa- tional costs and memory costs compare to GW-Selcction.
Inspired by empirical Fisher derivation (see A.pendix[B.]jie more details about Fisher information and how SMT differs with it), we propose GW-selection to identify specific submatrices within the model's weight matrices that show the greatest gradient changes during a warm-up phase at the beginning of fine-tuning, Since GW-Selection requires full backward propagation, it updates the base model during the warm-up phase, and identify the salient sub- matrices according to the average of absolute value of gradients. Mathematical formulations for GW-selection can also be found in Appendix B.T
Further comparisons between fine-tuning results using Activation-awar parameter selection (AW- Selection) and Gradient-aware parameter selection (GW- Selection are presented in Table g A significant performance gap can be observed between AW-Selection and GW-Selection in sparsity fine-tuning. AW-Selection tends to overfit more easily than GW-Selection when fine-tuning the same number of trainable parameters under identical hyper-parameter settings. Even after ad- AW-Selection still performs substantially worse than GW-Selection. we present the best results for AW-Selection
justing the In Table &
17

---

Published as a conference paper at ICLR 2025
Table 8: Results on fine-tuning LLaMA-7B using gradient-aware parameters selection (GW-
Selection) and activation-aware parameter parameters selection (AW-Selection) on CommonSense dataset.
Model PEFT method
#Params %
BoolQ
PIQA
SIQA
HellaSwag
WinoGrande
ARC-e
ARC-c
OBQA
GW-Select
0.84
68.7
81.7
78.3
91.6
78.8
84.1
68.7
AW-Select
0.84
62.0
49.5
52.9
70.8
48.6
54.8
41.2
45.7
53.2
LLaMA-7B
GW-Select
4,91
72.0
AW-Select
4,91
62.5
50.4
48.7
68.8
50.2
56.1
40.8
44.8
52.8
More details about AWQ Lin et al. 2024): Once the salient channels are identified based on
activation magnitude, AWQ proposes a strategy protect these channels by scaling them. The following equation (Equation 5 in the AWQ paper) describes how the scaling is performed:
= a' = mig L(s
where sx is the average magnitude of the activation (per-channel), and a is hyperparameter that controls the balance between protecting the salient and non-salient channels. The objective function L(*x) minimizes the output difference after applying the scaling, ensuring that the performance degradation due to weight adjustment is minimized.
The hyperparameter controls how aggressively the salient channels protected. When is larger, the scaling factor xx is more aggressive, placing more emphasis on the channels with larger activations. Conversely, smaller results in more balanced protection between salient and salient channels. Similarly, in AW-Selection, a is used to control the percentage of trainable param- eters. When is larger, larger percentage of trainable parameters is selected. Conversely, smaller results in smaller number of trainable parameters.
B.3 ABLATION STUDIES ON SUB-MATRICES SELECTION
In table we present experimental results comparing the following selection methods: selects the top Y% of sub-matrices parameters across all Q, K, V layers; selects the top Y% of sub-matrices parameters within each Q, K, V layer; and random selection, which selects sub-matrices parameters randomly within all Q, K, V layers. Our findings show that automatic top Y% parameter selection across all K, Q V parameters consistently outperforms the top Y% selection within each Q, K, layer and the random selection method.
Table 9: Fine-tuning LLama-7B using SMT CommonSense dataset. This table aims to perform an ablation study to find which parameter selection approach the best. SMT-1: Select the top 0.84% of parameters from all Q V parameters. SMT-2: Select the top 0.84% of parameters in every layers of Q, K, V vectors. SMT-2: Randomly Select 0.84% of parameters in every layers of Q. K. vectors.
Model
BoolQ
PIQA
SIQA
HellaSwag WinoGrande ARC-e ARC-e OBQA
AVG
SMT-1
68.7
81.7
78.3
91.6
78.8
84.1
68.7
77.4
78,66
SMT-2
66.5
80.8
73.2
85.1
71.4
79.7
58.8
67.2
72.83
SMT-3
61.7
74.8
70.1
82.4
68.4
75.2
54.5
64.5
68.95
18

---

Published as a conference paper at ICLR 2025
C SMALL GRADIENT CAUSED BY SOFTMAX SATURATES
C.I WHAT CAUSE THE SMALL GRADIENT IN Q K VECTORS?
In the context of the scaled dot- dot-product attention mechanism, the softmax function is typically ap- plied to the scaled dot-produce between the query and key matrices, expressed as:
QK vdk.
softmax
where E RRnxd is the query matrix, K Rxd is the key matrix, and dy the dimension of the key vectors. The term QK E R represents the matrix of dot products between query and key vectors. The softmax function upplied along the rows of the scaled matrix:
S QKT ddk
where the elements of S are given by
Qi Kj vds
Sij
with Q, K2 denoting the dot product between the i-th query vector and the j-th key vector.
To understand why the gradient of the softmax function with respect to Q and K vectors becomes small when QK is large and /dg is not sufficiently large, begin considering the gradient of the softmax function itself. For a given row S; of the matrix, the softmax function is defined as:
e
(S:)
E SSa
The gradient of (S;)) with respect to S& is given by:
@o(S.) ((S))(1-((()) i oS 1 (S())
if k.
Now, to compute the gradient with respect to Q, we must apply the chain rule. The gradient of the scaled dot- dot-product S.j Qr-K, K, with respect to Q is:
Vd
aS K aQ,
dd
Thus, the gradient of the softmax output with respect Q; becomes:
(o(
do(S)) 0S.;
Σ
aQ.
OS,j
8Q,
Substituting the gradients, obtain:
@o = Σ (S.) (5ig o(S)) aQ,
K, Vdk
We now examine the behavior of the gradient when QK" becomes large and /dg is not large enough. If the values of QKT are large, the elements Sig will also be large. As Sij grows, the
19

---

Published as a conference paper at ICLR 2025
softmax function saturates, meaning that one of the softmax outputs will approach while the remaining outputs approach O. In such cases, the gradient of the softmax function becomes very small because for large St, the output ((,)) is close to for the largest element and close to 0 for all other elements, leading to the following observations:
1. For the largest S,, the softmax output approaches 1, and the gradient becomes:
(5
as: ((0)))1-))))) ~ 1 =0
2. For the smaller 83 (where j + i), the softmax output approaches 0, and the gradient also becomes very small:
0o(S,) as, - R0 O.
The small gradient arises because the softmax function compresses wide range of input val- ues into a narrow range of ouiput probabilities between O and I. As result, when one value dominates the others, the gradients with respect the input values shrink significantly.
Next, consider the impact of the scaling factor ddg. This scaling is introduced to mitigate the effect of large dot products by dividing the dot products Qi Kj by Vd,. However, if ddg is not large enough, the elements of S QK vdd. can still become large, leading to saturation in the softmax function. As a result, the gradients with respect to Q remain small.
In summary, the gradient of the softmax with respect to Q becomes small when QK is large and ddg is not large enough due to the saturation of the softmax function. When the scaled dot- product QK has large values, the softmax output tends to concentrate around for one element and vd.
near O for the others, resulting in near-zero gradients. The scaling factor /dz alleviates this issue to some extent, but if dg is small, the problem of small gradients persists, hindering the optimization process.
C.2 SCALING vdz
We now demonstrate how increasing da affects the variance of QK variance leads larger differences between the elements of QK
Vdz and how an increase in
Consider the dot-product between query and key vectors Q; K,. Let the elements of Q and K be independent and identically distributed (i.id.) random variables with mean 0 and variance o². The dot product Q; K2 is the sum of dg independent terms, each of variance o2. Therefore, by the properties variance, the variance of Q; Kg is:
Var (K)KKK)) = d2 o².
(9)
Now, when we apply the scaling factor
1 the variance of the scaled dot-product becomes: van"
Qi Kj
1 -VarQQ K2)
dy 02
Var
02.
v Vdb
dp
Thus, the variance Q remains constant and independent of d,, which stabilizes the variance VTc
of the scaled dot-products. However, Vdl is too small, the variance the dot-products QK" will increase. This increase in variance implies that the differences between the elements of QK' also become larger, as larger variance leads to more spread-out values, which accentuates the differences between the individual elements of QK Consequently, if the differences between the elements of QK increase, the softmax function is more likely to saturate, resulting in one element becoming significantly larger than the others, and thus leading smaller gradients, discussed earlier.
20

---

Published as a conference paper at ICLR 2025
D MATHIOK DATASET
Math1 OK dataset can evaluate the effectiveness of LLMs on the arithmetic reasoning task. Math10K incorporate six subsets including GSM8k, SingleEq, SVAMP, MultiArith, AddSub, and AQuA.(1) the GSM8K (Cobbe et al. 2021) dataset consists of high quality lin- guistically diverse grade school math word problems created by human problem writers, (2) the SVAMP (Patel et al. (021) benchmark consists of one-unknown arithmetic word problems for up- to-4 grade level students by making simple changes dataset, (3) the Mu t Arith (Roy & Roth 2016) dataset of math word problems requiring multi- ple reasoning steps and operations, (4) AddSub (Hosseini et al., 2014) dataset of addition and subtraction arithmetic word problems, (5) the AQuA (Ling al.J 2017) dataset of algebraic word problems with natural language rationales, and (6) the SingleBq (Koncel-Kodziorsk Kedziorski et al., (015) dataset of grade-school algebra word problems that map single equations with varying length;
set of problems from another existing
E PLATEAU ISSUE FOR LORA AND DORA
In Fig. visualize the performance of LoRA, DoRA, and SMT under different scaling of train- able parameters on CommonSense dataset.
SpIEL
LLaMA-7B
LLaMA2-7B
831
78
81
Performance
Performance
80
74
77
Params (%)
Params (%)
Figure 9:
Accuracy comparison of LoRA, DoRA, and SMT under different scaling of trainable
parameters on commonsense reasoning datasets.
F MORE DETAILS ABOUT LORA
In Equation. I for pre-trained weight matrix Wo. LoRA constrains its update by representing the latter with low-rank decomposition Wo AW Wo + BA, where B e Rdxr A Rdxr" and the rank & min(d,k) k).
21

---

Published as a conference paper at ICLR 2025
G DENSE MATRIX MULTIPLICATION IN SMT OUTPERFORMS SPARSE MATRIX MULTIPLICATION
In this section, we explain in detail why SMT's use of dense matrix multiplication with concate- nated sub-matrices outperforms sparse matrix multiplication, particularly when implemented with PyTorch's tensor. Censor.sparse format.
G.1 MEMORY OVERHEAD IN SPARSE MATRICES
Sparse matrices often incur significant memory overhead due to the storage of additional indices, which is not required for dense matrices, Let us consider a concrete example of the LLaMA2 model, where the projection matrices Q. K, and V have dimensions 4096 4096.
In the case of a dense matrix, all elements are stored directly, resulting in the need to store 4096 x 4096 16, 777, 216 values. However, in sparse matrix format, if only 5% of the elements 216 = 838, 861 values need to be stored. Despite this reduction
non-zero, then only 0.05 x 16, 216 in stored values, sparse matrices require storing the indices of non-zero elements, which introduces significant overhead. Specifically, for each non-zero element, two indices (row and column) must 677, 722 indices, effectively resulting in sparse matrices taking up to 5 times more memory than a concatenated dense matrix used in SMT. We illustrate this Fig. 10 and Fig. In Fig. I0 we present the code we use to test memory and Fig. 11 displays the memory costs for each generated tensor.
be stored, leading to an additional x 838, 861
The sparse matrix is Tensor3 Fig. [1 which contains 12.80M to store 1, 677, 722 indices and 3.20M to store 838, 861 elements, totaling 16.0 MB. Instead, the dense matrix Tensorl requires only 3.20 MB to store the same 838, 861 elements. This dense storage reduces memory usage by factor of 5 (16.0 MB / 3.2 MB). Tensor1 is the full dense matrix with size 4096 x 4096.
menoryTest():
randoa_ full, tensor the dense tensor aith size 4096x4096 sporse, tensor is the sporse tensor with size 4096X6096
dense_tensor is the dense tensor with size 39742113
Madon_full_tensed borch.rand((499. 1096).to(DEVIIC)
sparse_tensar torch. 4896)
dense_tensor (39, 2113)
(8:39, 8:2113]
sparse_tense tensor sparse_tensor ttt.stpareee..tooeeeii..
torch.cuda. empty_cachhe) reporter NenReporter() reporten.repont()
Figure 10: Implementation of memory costs test.
Element type
Size Used MEM
Tensor8
(4896, 4896)
64.00M
Tensor1
(397, 2113)
39M
Tensor3
(2, 838861)
12.80M
Tensor3
(838861,)
3.
Total Tensors: 20132660
Used Memory: 83.
Figure 11: Memory costs.
By concatenating the selected sparse sub-matricss into single dense matrix, SMT avoids the need for index storage, thereby achieving significantly lower memory consumption, especially in GPU- constrained environments.
22

---

Published as a conference paper at ICLR 2025
G.2 NONCONTINUOUS MEMORY IN SPARSE MATRICES
GPUs are optimized for dense matrix operations, benefiting from contiguous memory layout and better parallelization. In dense matrix operations, the data stored in continuous block of memory, ensuring that all elements are accessed in a regular and predictable manner. This leads to fewer cache misses and higher throughput during computation.
In contrast, sparse matrices inherently involve irregular memory access patterns because accessing each non-zero element requires retrieving its associated indices. Unlike SMT, which requires only single index for submatrix block and benefits from continuous memory within that block, sparse matrices must repeatedly resolve indices for individual elements and map them into contiguous memory. This leads to inefficient memory utilization, lower cache efficiency, and increased latency.
Additionally, sparse matrices often experience load imbalances during parallel computations, as dif- ferent rows or columns can contain vastly differing numbers of non-zero elements. This imbalance complicates workload distribution across GPU threads, reducing overall parallel efficiency. SMT overcomes these issues by leveraging its structured and dense submatrix memory representation, which minimizes irregular access and ensures higher computational performance. SMT's approach of concatenating sparse sub-matrics into dense matrices enables the use of highly optimized dense matrix multiplication libraries (such as cuBLAS on NVIDIA GPUs), ensuring maximum hardware utilization and significantly faster computations.
G.3 LACK HARDWARE SUPPORT
Modern hardware, especially GPUs, is optimized for dense linear algebra operations using libraries like cuBLAS (NVIDIA) or MKL (Intel). These libraries implement hardware-specific optimiza- tions, such as fused multiply-add (FMA) and matrix blocking, to ensure high performance for dense matrix multiplications.
In contrast, sparse matrix operations, though supported by libraries like cuSPARSE MKL SPARSE, suffer from hardware inefficiencies due to the irregular memory access patterns inher- ent in sparse matrices. These irregularities make it challenging to optimize sparse operations the same degree as dense matrix operations, resulting in lower overall performance.
G.4 ALGORITHMIC COMPLEXITY
Sparse matrix multiplication algorithms must handle conditional checks to skip zero elements during multiplication, adding overhead to the computation. In dense matrix multiplication, all elements are processed in a straightforward loop without such checks, simplifying the computation.
When sparsity is moderate, the overhead from checking and skipping zero elements in sparse matri- ces can outweigh the performance gains, making sparse matrix operations less efficient than dense matrix operations, where every element is multiplied and summed without conditions.
By avoiding the overheads of sparse matrix operations and leveraging the efficiency of dense matrix multiplication, SMT achieves faster fine-tuning with lower memory usage.
23

---

Published as a conference paper at ICLR 2025
H
WARM UP ITERATIONS
The number of warm-up iterations is determined by running SMT on the Commonsense dataset for 30, 70, 100. 130, and 160 warm-up iterations. We look the average of the performance actoss all tasks, and choose the number of warm-up iterations that has the best overall overage performance, as shown in the table We conducted similar experiments on the Math- 10K dataset, and found using the same procedure that 30 warm-up up iterations leads to the best performance on the Math- dataset tasks. Notably, although we conducted an ablation study select the number of iterations for different datasets, Table TO] shows that performance variance due to the number of warm-up iterations minimal. This trend is also observed in the Math- 10K dataset in Table [TT suggesting that 100 warm-up iterations can generally be applied across most datasets without additional tuning.
Table 10: SMT experiment results using different warm-up iterations CommonSense dataset. This table aims to find the best gradient warm-up steps. All experiments are using SMT to fine-tune 3.5% trainable parameters in Q K, V vectors of LLaMA-7B. SMT-30S: 30 gradient warm-up steps; SMT-70S: 70 gradient warm-up steps. SMT-100S: 100 gradient warm-up steps. SMT-130S setting: 130 gradient warm-up steps. SMT-160S: 160 gradient warm-up steps.
Model
BoolQ
PIQA
SIQA HellaSwag
WinoGrande ARC-e ARC-c OBQA
AVG
SMT-30S
69.6
79.2
92.7
81.5
84.9
69.0
78.6
SMT-70S
70.1
83.1
79.4
92.7
81.1
85.4
69.3
80.2
80.16
SMT-100S
71.0
82.3
80.0
92.9
81.8
86.2
70.4
80.2
80.60
SMT-130S
69.5
82.6
79.7
92.7
81.7
85.9
69.7
81.6
80.43
SMT-160S
68.9
82.4
79.7
93.0
82.4
85.4
70.2
82.0
80.50
Table 11: SMT experiment results using different warm-up iterations on Math-10K dataset. This table aims to find the best gradient warm-up steps. All experiments are using SMT to fine-tune 0.86% trainable parameters Q, V vectors of LLaMA-7B. SMT-20S: 20 gradient warm-up steps; SMT-30S: 30 gradient warm-up steps. SMT-50S: 50 gradient warm-up steps. SMT-70S setting: 70 gradient warm-up steps. SMT- 100S: 100 gradient warm-up steps.
Model
PEFT method
#Params%
GSM8k
SingleEq
SVAMP
MultiArith
AddSub
AQuA
AVG
SMT-20S
0.86
34.0
84.2
52.8
91.8
84.8
23.2
61.8
SMT-30S
0.86
34,2
84.6
53.6
91.5
85.8
23.6
62.2
LLaMA-7B
SMT-50S
0.86
34.4
83.7
52.8
91.0
85.2
22.6
61.6
SMT-70S
0.86
33.8
83.8
52.8
91.3
84.4
22.8
61.5
SMT-100S
0.86
33.6
83.9
53.1
91.3
85.4
22.8
61.7
24

---

Published as a conference paper at ICLR 2025
I SELECTION FOR SUBMATRIX SIZE
The rational behind 256 x 256 as sub-matrix block size is also under the consideration for general- ization and trade off between performance and time costs for non-continuous memory projection.
The choice of a 256 x 256 sub-matrix block size is based on generalization and a trade-off between performance and the time costs of non-continuous memory projection.
From a generalization perspective, we aim to select a block size that is applicable across all layers large language models (LLMs). Specifically, in the LLaMA family, the K, Q. V, and O vectors are designed with sizes such as 4096 x 4096 in LLaMA-7B, LLaMA-2-7B, 4096 x 1024 for K and in LLaMA-3-4 ---B-B., and 5120 5120 LLAMA-IBBLLLMMA 4096 x 1 11008 in LLaMA- 13BLLA-11BLLLM--22222
The MLPs have sizes like 4096 x 14336 in LLaMA-3-8B and 5120 x 13824 in Among these weight matrices, 256 256 the largest common block size, as 256 is the greatest common divisor of 1024, 4096, 5120, 11008, 14336, and 13824. This block size can not only be used LLaMA models but also in other LLM families such as Mistral and Phi.
While 256 256 is the largest common factor, alternative block sizes such as 128 x 128 and 64 64 also potential candidates. We conducted experiment, as shown in Tab. 12 to evaluate the performance of fine-tuned LLMs with different sub-matrix block sizes. The results indicate that smaller block sizes yield slightly better performance 0.5%), but they introduce additional time costs due to the memory projection required by SMT.
For non-continuous memory projection, SMT transfers sub-matrices dense memory, which in- curs time costs. Sparse matrices are often stored in compressed formats (e.g.. Compressed Sparse Row, Compressed Sparse Column), where only non-zero elements and their indices are stored Go- harian et al. )2003); Ordonez et al. (2016). Smaller block sizes lead to fewer elements per block, increasing the likelihood of data being scattered across different memory locations, which results in more frequent memory look-ups and higher overhead when assembling these blocks into dense representation. With block size of 256 x 256, the time costs memory projection are negligible, accounting for less than 3% of the total forward pass time. Specifically, for 256 x 256 block size, the total forward pass time seconds. In contrast, for 128 128 block size and 64 64 block size, the memory projection time increases to 5. 46-06 seconds (10%), and 1. 5e-500 05 seconds (24%), respectively. While smaller block sizes provide marginal performance improvements (+0.5 point), they significantly increase time costs. Therefore, we chose 256 x 256 as the optimal sub-matrix block size. The profiling results were recorded when fine-tuning 0.84% of LLaMA-7B parameters, in V vector where 2.1 % parameters were selected. All time profiling results are averaged over 100 iterations, after 100 warm-up iterations.
9e-05 05 seconds, and the memory projection time 1. 6e- 0
Table 12: Experiment results for SMT with different sub-matrix block sizes when fine-tuning 0.84% of LLaMA-7B parameters on the CommonSense dataset. SMT-256: 256 x 256 block size; SMT- 128: 128 x 128 block size; SMTT-64: x 64 block size. Other experimental settings are described in the experimental section of the main paper.
Model
BoolQ
PIQA
SIQA
HellaSwag WinoGrande ARC-e ARC-e OBQA
AVG
SMT-256
68.7
81.7
78.3
91.6
78.8
84.1
68.7
77.4
78.7
SMT-128
68.9
82.3
78.8
91.8
79.2
84.5
68.8
77.8
79.0
SMT-64
69.2
82.2
78.0
91.7
79.5
85.3
69.2
78.2
79.2
25


Reviews:
[{"review_id": "rwfb8grzdw", "reviewer": "Reviewer_TAx9", "summary": "This paper presents a novel PEFT method, Sparse Matrix Tuning. SMT uses a gradient based criteria to identify the most relevant submatrices of the model to update, and updates only those while freezing the remaining params during training. This allows for memory and speed gains during training which are similar or better to those made by other PEFT methods. SMT performs well compared to strong baseline PEFT methods LoRA DoRA and SpIEL.", "questions": "The implementation specifies (L242) \"we use 256 x 256 as sub-matri[x] block size\". L260 states \"Despite employing\nmatrix sparsity, we still leverage the advantages of dense matrix multiplication.\" What happens when it is per-parameter (ie 1x1)? Is there a tradeoff between method performance and efficiency being made in selecting sub-matrices instead of free parameters?  It seems the only reason to have blocks is computational simplicity. The motivation behind this choice should be made explicit, and the tradeoff investigated. What happens with a larger block size?\n\nNits:\nFigure 1 contrasts two techniques, but appears on first glance to be 4 boxes equally spaced. Consider modifying the figure to make more visually apparent the separateness of the techniques.\nL 054: submatrix -> submatrices, right?\nL 147: the whole pre-trained weight -> all the pretrained weights?\nL 163: for downstream -> for *improving* downstream\nL 238: sum up gradient -> sum up the gradient / sum up gradients\nL 239: iterations -> iteration, sum up gradient information are -> summed up gradient information is\nL 387:  \"commonsense reasoning datasets\" in Table 3 vs COMMONSENSE in Table 4... are these the same? If so, refer using same term\nTable 3 should be a plot.\nL426-429: this is highly repetetive, please revise\nThroughout: please stay within a single tense within a single paragraph (L302-317 especially, but also elsewhere)\nThroughout: missing articles, incorrect pluralization or lack thereof", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The problem addressed is important, the evaluation results are good. The approach could be significant to the field.", "weaknesses": "The method is poorly described in the section dedicated to its description. What exactly is happening during warm-up phase? It seems that the warm up phase involves taking 100 steps (on the target dataset?) *without updating the model params*, and taking the average of abs-value of gradients for each param of the model to identify sub-matrices. This is critical to specifying the methodology, and should be clearly stated, not deduced by the reader. \n\nNo experiments show ablations on the specification criteria, or the size of the 256x256 submatrix. The justification for restricting SMT to the KQV of the attention mechanism is relatively weak. The sole justification for restricting the SMT trainable params to the attention is a strange ablation comparing hardcoded percentages interpolating between only MLP and only attention. A more natural ablation would allow the selection criteria to place trainable submatrices wherever in the entire model as a baseline, and compare to a variety of restricted alternatives. \n\nThe writing throughout is relatively poor. Articles are frequently missing, pluralization is often wrong, tense is frequently switched. This can easily be fixed, and would greatly improve readability.\n\nGenerally, the method is sound, though poorly described. The experiments have good results, but the analysis leaves some to be desired.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cvPqFOS7CQ", "reviewer": "Reviewer_ovmN", "summary": "This paper introduces Sparse Matrix Tuning (SMT), a parameter-efficient fine-tuning (PEFT) method for large language models (LLMs), aiming to reduce computational and memory costs while maintaining performance close to full fine-tuning (FT). SMT selectively updates only the most significant submatrices in a model's weight matrix, chosen based on gradient changes, enabling efficient tuning without compromising on model accuracy. The authors compare SMT with established PEFT methods such as LoRA and DoRA, demonstrating that SMT outperforms these methods in both accuracy and computational efficiency, especially by overcoming the performance plateau that other low-rank adaptation methods encounter with increased parameters.", "questions": "None", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This work contributes to the growing field of parameter-efficient fine-tuning, a critical area for the advancement of LLMs. Efficient tuning techniques like SMT can play a major role in making large models more accessible and practical for diverse applications.\n\n2. The paper provides an insightful analysis of activation-awareness and gradient-awareness methods, ultimately selecting the gradient-based approach as more effective for identifying meaningful submatrices. This comparison enriches the understanding of PEFT techniques and optimizations.\n\n3. The SMT approach offers an improved strategy for identifying impactful sub-networks, which potentially boosts fine-tuning efficacy by focusing on parameters that most influence downstream performance.", "weaknesses": "1. While SMT improves on existing PEFT methods, the approach of selecting sub-networks for fine-tuning is not particularly novel, as similar sparse selection methods have been explored previously in transfer learning and model pruning.\n\n2. The paper does not address whether SMT offers better resistance to catastrophic forgetting compared to other PEFT methods. This omission leaves questions about SMT's stability and robustness in long-term, iterative fine-tuning scenarios.\n\n3. The paper lacks some necessary conceptual explanations and mathematical formulations, such as detailed descriptions of submatrix selection and significance metrics. This limited detail could hinder reproducibility and a deeper understanding of SMT's implementation and optimizations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hfw1RHJxcE", "reviewer": "Reviewer_2m9a", "summary": "This paper introduces a novel parameter-efficient fine-tuning (PEFT) method, termed Sparse Matrix Tuning (SMT). SMT builds on insights from prior research suggesting that domain-specific knowledge can be localized and sparsely distributed across layers. SMT identifies the most relevant sparse submatrices by analyzing gradients and then selectively fine-tunes only these submatrices. Experimental results demonstrate that SMT not only outperforms previous PEFT methods but also achieves performance levels close to full parameter fine-tuning. Further analysis reveals that SMT effectively mitigates the performance saturation commonly observed with other PEFT approaches.", "questions": "1. In Table 2, the authors provide full fine-tuning results only for LLaMA-7B and LLaMA2-7B. Adding full fine-tuning results for additional models would help to better contextualize SMT's performance advantage and strengthen the comparison with full fine-tuning methods.\n2. It is unclear how the sparsity hyperparameter $z$ is determined in the experiments. While the #Params% results seem largely influenced by LoRA ranks, SMT should, in principle, allow for a more flexible selection of sparsity levels. Providing details on the criteria or method used to select $z$ would clarify this aspect and help illustrate SMT's potential for adaptable parameter tuning.\n\n\nTypos and writing issues.\n\n1. In line 072, a colon is missing.\n2. In lines 146–148, there is an unnecessary space before the reference term.\n3. The use of bold and italic formatting is excessive in multiple sentences. This can make the text appear cluttered and distracting. Reducing the use of bold and italics to highlight only the most essential points will make the writing cleaner and enhance readability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. SMT inherits the performance benefits of prior sparse tuning methods while addressing their shortcomings in computational and memory efficiency.\n2. Analysis reveals that methods like LoRA, DoRA, and SpIEL suffer from performance saturation as tunable parameters scale, whereas SMT maintains consistent performance without this limitation.\n3. The gradient-based parameter selection in SMT enables deeper analysis of critical parameter distributions within LLMs. Results indicate that the Attention module, particularly the value (V) vector, requires more intensive fine-tuning in LLMs. This observation provides valuable insights and could potentially inform future studies in parameter-efficient tuning.", "weaknesses": "1. **Gap between Claimed and Actual Contribution:** While the authors claim that SMT aims to \"close the performance gap between PEFT and full fine-tuning,\" SMT’s primary focus appears to be on improving the computational and storage efficiency of existing sparse tuning methods rather than introducing fundamentally new fine-tuning strategies. This creates a disconnect between the paper’s stated goal of performance enhancement and its actual contributions, which seem more oriented toward efficiency. To strengthen the paper, it would be beneficial for the authors to align their claims with the actual contributions more closely.\n2. **Lack of Theoretical Support:** The claim that SMT can \"close the performance gap between PEFT and full fine-tuning\" lacks sufficient theoretical justification. The paper primarily discusses SMT’s implementation details and efficiency improvements without explaining why fine-tuning a sparse matrix should lead to performance gains comparable to full fine-tuning. Providing a theoretical explanation or framework to support this belief would significantly strengthen the paper. For instance, the authors could explore how sparse matrix fine-tuning might preserve essential parameter interactions or discuss relevant properties of sparse representations that could theoretically justify this performance potential.\n3. **Missing Baselines:** While the paper primarily focuses on enhancing the computational and storage efficiency of sparse tuning methods, the inclusion of additional sparse tuning baselines would provide a more comprehensive evaluation. Comparing SMT with other sparse tuning methods would highlight its specific advantages in efficiency. Additionally, several recent studies, such as LoRA-GA [1] and LoRA-Pro [2], also aim to bridge the gap between PEFT and full fine-tuning. Including these methods in both the discussion and experiments would allow for a more thorough assessment of SMT's performance and effectiveness relative to other state-of-the-art approaches.\n\n[1] Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.\n\n[2] Wang Z, Liang J. LoRA-Pro: Are Low-Rank Adapters Properly Optimized?[J]. arXiv preprint arXiv:2407.18242, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xvjxASlhEg", "reviewer": "Reviewer_5QaY", "summary": "This paper proposes a new parameter-efficient fine-tuning method which only fine-tunes the task-specific sub-matrix for large language models. The method first identifies the salient sub-matrix for each task with gradient-aware selection strategy. Then, only the salient sub-matrix will be optimized during the fine-tuning stage. The experiments demonstrate that the proposed method can achieve comparable or even better performance than full fine-tuning while requiring updating tiny part of parameters. This work also finds that fine-tuning attention layers are more critical than MLPs and the value matrix v is more influential than the query and key matrix Q, K.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The sparse matrix tuning method seems reasonable for improving downstream task performance.\n\n2. The results are quite positive and the analyses are extensive.\n\n3. The two interesting findings that attention layers and value matrix are more influential may inspire new insights in the field of parameter-efficient fine-tuning.", "weaknesses": "1. This SMT method requires warmup stage to identify the salient sub-matrix in LLMs and the warmup iterations rely on the specific downstream tasks and dataset size. It makes the method somewhat complicated to use. The generalization is limited.\n\n2. Compared to LoRA, this SMT method needs to update a subset of parameters of the original LLMs and it will change the original model, so that the original LLMs cannot be reused for downstream tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ykTPy5mMr4", "reviewer": "Reviewer_kjYr", "summary": "The PET method proposed in this paper divides parameter matrices into subblocks. Only a subset of these subblocks are changed in fine-tuning, everything else is frozen. The authors demonstrate convincing advantages over LoRA.", "questions": "- What is the intention behind using 256x256 sub-matrices? What happens in terms of quality and efficiency when you use smaller (down to 1x1) or larger sub-blocks?\n- iiuc the warm-up phase still needs the same memory as full fine-tuning. Can you avoid this phase by reusing sub-block selection from a different [multi-task] experiment for a new task?\n- The PET Speed-Ups in Table 1 seem fairly large to me.. I would expect 2-4x speed ups by LoRA, not ~14x. Could you cite work that reports LoRA speed-ups over full fine-tuning in a similar ballpark?\n- L223: is the gradient step computation cost really z% cheaper? I understand the argument for a single parameter matrix, but don't you need full gradients in layers closer to the output layer to correctly propagate the gradients to this point? Same for the activation costs in the forward pass (L223)\n- Why are some of the full fine-tuning results in Table 2 missing?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Good empirical evaluation across multiple model sizes and datasets\n- Insightful ablations in Sec. 5 and the appendix", "weaknesses": "- Some practical questions (e.g. block size and mask generalization) are not addressed\n- No comparison with other delta tuning / diff pruning approaches and random masking (https://arxiv.org/abs/2405.02596, https://aclanthology.org/2023.emnlp-main.931.pdf)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)