Sparse Matrix in Large Language Model Fine-tuning

## Abstract

Abstract LoRA and its variants have become popular parameter-efficient fine-tuning (PEFT) methods due to their ability to avoid excessive computational costs.
However, an accuracy gap often exists between PEFT methods and full fine-tuning (FT), and this gap has yet to be systematically studied.
In this work, we introduce a method for selecting sparse sub-matrices that aim to minimize the performance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine-tuning computational cost and memory cost.
Our Sparse Matrix Tuning (SMT) method begins by identifying the most significant sub-matrices in the gradient update, updating only these blocks during the fine-tuning process.
In our experiments, we demonstrate that SMT consistently surpasses other PEFT baseline (e.g. LoRA and DoRA) in fine-tuning popular large language models such as LLaMA across a broad spectrum of tasks, while reducing the GPU memory footprint by 67% compared to FT.
We also examine how the performance of LoRA and DoRA tends to plateau and decline as the number of trainable parameters increases, in contrast, our SMT method does not suffer from such issue.

## 1 Introduction

While the inherent generalization capability of Large Language Models (LLMs) is impressive, enhancing performance on downstream tasks often still necessitates fine-tuning Ding et al. (); Chung et al. ().
However, as the size of these LLMs increases, there is a pressing challenge to optimize the fine-tuning process for better computational efficiency and memory utilization.
For example, fine-tuning a pre-trained LLaMA 7B model without CPU offloading(^1^11Although some libraries such as Deepspeed can move the optimizer memory cost to CPU, it will also slow down the fine-tuning with extra I/O communication time Rajbhandari et al. (); Aminabadi et al. ().)
requires at least 58 GB of GPU vRAM—13.6 GB for trainable parameters, 40 GB for Adam optimizer states and gradients, and 2 GB for activation.
This requirement makes fine-tuning on consumer-level GPUs such as the NVIDIA RTX 4090 with 24 GB of memory impractical Zhao et al. ().

To address the prohibitive computational challenges of full parameter fine-tuning, many parameter-efficient fine-tuning (PEFT) methods have emerged over the past two years. LoRA and its variants Hu et al. (); Zhao et al. (); Dettmers et al. (); Liu et al. (, ) utilize the weight low-rank adaptation method and successfully reduce both the optimizer memory and computational cost.
However, even in state-of-the-art (SoTA) PEFT research, results show a notable performance gap between low-rank adaptation methods and full parameter tuning across many datasets Liu et al. ().
Additionally, in this work, we report a less recognized phenomenon: low-rank adaptation PEFT methods experience a performance plateau and subsequent decline as the parameter count (rank r) increases. Surprisingly, even with more trainable parameters, performance decreases.

On the other hand, previous studies have extensively analyzed the internal logic of LLMs. Some knowledge editing methods, such as Constrained fine-tuning Zhu et al. (), ROME Meng et al. (), and MEMIT Meng et al. (), have shown that LLMs have memory sections located in distinct layers. These memories can be modified via fine-tuning Zhu et al. ().
These works observed that domain-specific knowledge can be distributed separately and sparsely among layers.
Motivated by these observations, and to address the challenges of PEFT mentioned above, we proposed a *Sparse Matrix Tuning(SMT)* approach.
By applying matrix sparsity, we aim to identify and fine-tune the most relevant memory sections efficiently.
Contrary to Geva et al. (, )’s claim that transformer’s MLP layers primarily serve as key–value memories, we empirically demonstrate that attention mechanisms, particularly the value vector, store the largest number of memories and are the most influential during fine-tuning.

In our experiments, our Sparse Matrix Tuning (SMT) approach achieves better performance compared to LoRA and DoRA using same amount of trainable parameters.
Additionally, SMT closes the accuracy gap between full fine-tuning, overcomes the performance plateau of low-rank adaptation PEFT methods, and significantly outperforms LoRA and DoRA while utilizing less than 5% of trainable parameters.
Our experimental results show that SMT consistently outperforms SoTA PEFT (including LoRA and DoRA) methods by 2+ points when fine-tuning popular LLMs (e.g. LLaMA series base model(^2^22Base model is not yet instruction tuned)) on commonsense reasoning and arithmetic reasoning benchmarks.
Our experimental results show that SMT consistently outperforms DoRA, such as commonsense reasoning (+3.0/+2.8 on LLaMA-7B/13B,
+2.9 on LLaMA2-7B, and +2.0 on LLaMA3-8B) and arithmetic reasoning (+2.3 on LLaMA-7B).
In addition, SMT eliminates the accuracy gap between SMT and full fine-tuning, overcomes the plateau of low-rank adaption PEFT methods, and significantly exceeds performance of LoRA and DoRA with a small proportion of trainable parameters(5%<).
For layers without selected sub-matrices, SMT freezes these layers, saving all their backward propagation computational cost, parameters update computational cost, optimizer memory cost, and activation memory cost.
For layers with selected sub-matrices, SMT reduces the computational costs of backward propagation and parameter updates, as well as the optimizer and activation memory costs, to less than 1% of those incurred by standard full tuning (FT).

Below are our key contributions:

- •
PEFT SOTA Algorithm
We propose a novel fine-tuning method (SMT) which achieves state-of-the-art performance in parameter-efficient fine-tuning, effectively closing the gap between SMT and full fine-tuning.
In contrast, LoRA and DoRA’s performance saturation is faster than that of SMT as the number of trainable parameters grows.
- •
Language Model Anatomy:
We investigate the distinct impacts of attention mechanisms versus MLPs (Multi-Layer Perceptrons) in LLMs. Our findings indicate that attention layers are more critical than MLPs for downstream performance.
Among the Q,K,V vectors of attention layers, we found V to be the most influential for performance.
- •
Large Language Model System Efficiency:
The SMT implementation significantly reduces the computational cost of backward propagation, parameter updates, optimizer memory, and activation memory during fine-tuning.
Our implementation is open source.

## 2 Background and Related Works

Many works on parameter-efficient fine-tuning (PEFT) Mangrulkar et al. () have aimed to improve efficiency and performance by only fine-tuning lower-dimensional parameterization of model weights.
Notable examples include LoRA Hu et al. (), DoRA Liu et al. (), QLoRA Dettmers et al. (), and several other variants Liu et al. (); Dettmers et al. ().
However, the results of these works still indicate a performance gap between PEFT methods and full fine-tuning (FT).
Concurrent research Biderman et al. () empirically demonstrated that such a gap is difficult if not impossible to eliminate, they also notice the performance saturation issue of LoRA, as we will discuss in Section §4.3.

Figure: Figure 1: Differences between low-rank adaption method LoRA and SMT. Upper picture dedicates adaption approach in LoRA and lower picture represents the sub-matrices sparsity approach in SMT.
Refer to caption: extracted/5630734/SMT_vs_Adaptors.png

Beside low-rank adaptation methods, sparsity-inspired approach is a natural alternative to reduce computational costs and memory footprint. As they have been recently applied to accelerate LLM inference, H2O Zhang et al. () leveraged sparsity in KV cache eviction policy; DeepSparse Kurtic et al. () used L2-based distillation to promote sparsity.
In the case of sparsity-inspired fine-tuning, Song et al. () developed a Sparse Increment Fine-Tuning (SIFT) approach to reduce GPU memory cost. However, SIFT Song et al. () still requires full backward propagation to calculate all gradients, having no speed advantages compared to full fine-tuning(FT).
Moreover, SIFT maps discontinuous memory gradients to continuous memory addresses, creating a significant time bottleneck and resulting in longer fine-tuning time than FT.
Our work builds upon existing strategies, but unlike previous approaches, our matrix sparsity method directly incorporates task-specific gradient information to dynamically adjust the sparsity level for optimization, and we achieve speedup as we describe in Section §3.3.

Suppose we are given a pre-trained auto-regressive language model $P_{\Phi}(y|x)$ parameterized by $\Phi$. Each downstream task is represented by a training dataset of context-target pairs: $Z={(x_{i},y_{i})}_{i=1,\dots,N}$, where both $x_{i}$ and $y_{i}$ are sequences of tokens.
Equation (1) dedicates the LoRA Hu et al. () fine-tuning process to maximize the conditional language modeling objective, which uses a low-rank representation to encode task-specific parameters.
Specifically, LoRA freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture.
This is formulated as $\Delta\Phi=\Delta\Phi(\Theta)$, where $\Theta$ represents a much smaller-sized set of parameters with $|\Theta|\ll|\Phi_{0}|$. The resulting increment $\Delta\Phi$ can be as small as 0.01% of the pre-trained weights parameter size $|\Phi_{0}|$ in gradient updates. This greatly reduces the number of trainable parameters and the GPU memory requirement while maintaining or even enhancing model performance.

$$ $\max_{\Theta}\sum_{(x,y)\in\mathcal{Z}}\sum_{t=1}^{|y|}\log(P_{\Phi_{0}+\Delta \Phi(\Theta)}(y_{t}|x,y_{<t}))$ (1) $$

In our work, our proposed Sparse Matrix Tuning(SMT) uses matrix sparsity as the parameter-efficient approach. In SMT’s case, reusing Equation( 1), the $\Theta$ represents the sub-matrices within the sparse weight matrices. SMT only fine-tunes sparse sub-matrices $\Theta$ instead of fine-tuning the whole pre-trained weight. Figure 1 illustrates the differences between weight low-rank adaption method LoRA and our proposed sparse matrices tuning approach SMT. For a pre-trained weight matrix $W_{0}$, LoRA constrains its update by representing the latter with a low-rank decomposition $W_{0}+\Delta W=W_{0}+BA$, where $B\in R^{d\times r}$, $A\in R^{d\times r}$, and the rank $r\ll min(d,k)$. In SMT, we slice the pre-trained weight into $N$ sub-matrices and only fine-tune selected $M$ sub-matrices. The dimension of sub-matrices is $l\times l$, the total number of sub-matrices $N$ in a pre-trained weight is $N=\frac{d\times k}{l\times l}$.
SMT constrains its update by representing the latter with a sparse gradient matrix $\Delta W_{M}$, $W_{0}+\Delta W=W_{0}+\Delta W_{M}$, where the number of fine-tuning sub-matrices $m\ll N$.

Since our proposed SMT method focuses on fine-tuning sub-matrices which are most relevant to downstream tasks’ performance, identifying these sub-matrices is non-trivial.
Previous works Zhu et al. (), MEMIT Meng et al. (), and Geva et al. (, ) indicated that feed-forward MLP layers of the LLMs are most influential. However, through our experiment analysis§ 5.1, we found attention layers to be more relevant for performance than MLP layers.

## 3 Methodology

### 3.1 Selecting the most influential sparse sub-matrices

Figure: Figure 2: (a) A sparse weight matrix $W$. The green sub-matrices with significant gradients can be updated. (b) Backward propagation calculation for partial gradient for weight matrix $w$. (c) Computation graph in auto-differential systems.
Refer to caption: extracted/5630734/sparse_matrix_system_implement.png

Our methodology centers around the application of a *matrix sparsity* approach during fine-tuning LLMs.
Specifically, we select certain sub-matrices in weight matrices within the model’s weight matrices that exhibit maximal gradient changes during a 100 iterations warm-up phase (Fig. 2.a) at the beginning of fine-tuning.
Intuitively, our approach aims to select and modify the sub-matrices most relevant to the fine-tuning sub-task.
When fine-tuning LLMs, SMT improves computational efficiency and reduces memory needs during fine-tuning by not updating a portion of weight matrices after the warm-up phase and storing sparse weights in compressed form.

First and for most, SMT reduces the backward computation costs with respect to weight to 0.5% of those associated with Full Fine-tuning (FT).
SMT achieves this by decreasing the computation cost of gradients during backward propagation, as gradients are calculated for only a subset of the weights.
For linear layers in LLMs, where $Z=Wx$, the gradients with respect to weight matrix W and input x can be calculated as Equation (2):

$$ $\nabla_{x}f(x)=\frac{\partial l}{\partial Z}\cdot W;\hskip 19.91692pt\nabla_{W }f(x)=\frac{\partial l}{\partial Z}\cdot x$ (2) $$

where $\partial l/\partial z$ is the gradient information from backward propagation in (Fig. 2.b,c). $\nabla_{w}f(x)$ is the gradient matrix and and $x$ is the activation in the (Fig. 2.b). (Fig. 2.b) also illustrates that only partial backward computations are necessary when we update selected sparse matrices.
To calculate the sub-matrix gradient (highlighted in yellow), it is only necessary to multiply the yellow row in $\partial l/\partial z$ with the yellow column in the activation $x$.
Similarly, to calculate the green sub-matrix gradient, we only need to multiply the green row in $\partial l/\partial z$ with the green column in activation $x$.
Note that in backward propagation, we can only reduce computation when derivative to gradient matrix $w$ among as illustrated by the green arrows in (Fig. 2.c). but not other necessary computation. (black arrows)

Besides, SMT reduces the activation memory cost for the in forward pass to 0.5%. Since SMT only computes the partial gradient, it only saves the relevant portions of activation $X$ necessary for the gradient calculation as is represented in Equation. 2.
In (Fig. 2.b), to calculate the green and yellow gradients in the gradient matrix, we only need to save the yellow and green columns of the activation $X$.
This approach reduces the memory cost for the forward pass of the selected linear layer.

In addition, SMT reduces the memory cost of the optimizer gradients to 0.5%. Since SMT only updates selected sparse sub-matrices, only partial gradients are stored. This approach significantly cuts the memory cost of the Adam optimizer to 0.5%. This reduction is crucial because the memory cost of the Adam optimizer is typically twice the size of the model, which often consumes the majority of GPU RAM.

Furthermore, SMT reduces the gradient step computation cost to 0.5%. By updating selected sparse sub-matrices, SMT performs partial gradient steps and significantly reduce step computation cost.

In SMT, all the layers except selected Q, K, and V vectors are frozen during fine-tuning. By doing this, SMT avoids all the weight backward propagation computational cost, parameters update computational cost, optimizer memory cost, and activation memory cost in frozen layers. The rationale for fine-tuning only the Q, K, and V vector is detailed in Section §5.1.

By applying sparse sub-matrix fine-tuning, SMT can reduce the fine-tuning memory cost of LLaMA-7B and LLaMA2-7B to less than 20GB and fit the fine-tuning into a 3090 24GB GPU. We also reduce the computation and achieve faster fine-tuning compared with FT and LoRA/DoRA, Section § 3.3 provides more details.

### 3.2 Implementation

In SMT, we first sum up gradient from the attention linear layers in every single warm-up iterations. The sum up gradient information are used to identify task-specific sparse blocks. After the warm-up steps, we average the absolute values within the sub-matrices, select the sub-matrices with the largest value, and save the indices for the selected sub-matrices. In all of our experiments, we use $l\times l=256\times 256$ as sub-matrices block size. During the warm up steps, we can apply offload Rajbhandari et al. () on memory constraint GPU devices.
Since SMT requires fewer than 100 warm-up steps in our experiments, it does not become a bottleneck during fine-tuning epochs.
Additionally, SMT implements a custom sparse linear layer to ensure that unselected gradients are not calculated, saved, and updated (Code Snippet 6). We replace the selected linear layers with these customized sparse linear layers.

The custom sparse linear layer applies a specialized sparse linear multiplication function, integrated into our customized sparse linear layers (Code Snippet 7). This function calculates partial weight gradients based on the input, weight, and selected weight index. It significantly reduces the computational cost of backward propagation weight gradients to just 0.5% and minimizes the memory usage of the returned partial gradients to only 0.5%.

The specialized sparse linear multiplication function rewrites both forward and backward functions. In the forward pass (Code Snippet 7) of sparse linear multiplication function, we only save selected activation $x$ using ctx.save_for_backward(), and in the backward pass (Code Snippet 8), we customize matrix multiplication to calculate the needed partial gradients given partial input and gradient index(shown in Figure 2(b)).
It is important to note that we do not use Sparse Matrix-Matrix Multiplication(SPMM)(^3^33Sparse Matrix-Matrix Multiplication (SPMM) is significantly slower than General Matrix Multiply (GeMM))
because we concatenate the selected sparse sub-matrices and formed a $m\times l\times l$ dense matrix as illustrated in right part of figure 1. This would not cost additional time since memory allocations remain continuous within each sub-matrix. Despite employing matrix sparsity, we still leverage the advantages of dense matrix multiplication.

Furthermore, SMT gathers sparse matrix but still leverages dense matrix. SMT customizes the function for gathering trainable parameters. This customized function selectively gathers weight sub-matrices in the Q, K, and V vector layers and passes them to the Adam optimizer. By continuing to use the well-designed FusedAdam from the deepspeed library Aminabadi et al. (), we maintain the computational speed of dense matrix weight updates.
However, our approach reduces the gradient memory cost in the optimizer to just 0.5%.

### 3.3 Memory and Computation Saving: SMT vs. Low-rank Adaption Methods

SMT is more computational efficient than weight low-rank adaption method when the number of trainable parameters are the same, weight low rank adaption methods need to maintain additional adapters, which require additional forward computation. For instance, since LoRA maintains adaptors $A$ and $B$, and the forward propagation is:

$$ $h=W_{0}x+\Delta W_{x}=W_{0}x+BAx$ (3) $$

where the term $BAx$ requires additional forward propagation calculation, which is cut off in SMT. Regarding memory cost, since SMT does not require additional low-rank adapters $A$ and $B$, SMT can achieve lower memory cost than LoRA and DoRA under the same amount of trainable parameters setting. We illustrate this in Figure 1, by abandoning additional adaptor weight $A$ and $B$, SMT can achieve lower memory cost.
Taking the popular LLaMA-13B model as an example, since the model size is approximately 25 GB, if we fine-tune 1% of parameters, SMT can potentially save 250MB GPU meory.
In Table 1, we provide the fine-tuning time costs for SMT, Full Fine-tuning, LoRA, and DoRA. SMT achieves an 14.6× speedup compared to Full Fine-tuning and outperforms both LoRA and DoRA.
We conducted time profiling by averaging the fine-tuning time every 10 iterations over 1000 iterations, following a 500-iteration warm-up period. Full fine-tuning utilized offload settings to accommodate the LLaMA model, which employs the Adam optimizer, within the 40GB GPU.

## 4 Experiments and Results

### 4.1 Experimental Settings

Model Architecture and Dataset: In our experimental setup, we use open-weight LLaMA-7B, LLaMA-13B, LLaMA2-7B, and LLaMA3-8B models AI@Meta (). In Subsection§ 4.2§ 4.3, We perform fine-tuning on the Common Sense Reasoning tasks with 8 sub-tasks, each with a predefined training and testing set. We follow the setting of Hu et al. (); Liu et al. () and amalgamate the training datasets from all 8 tasks to create the final training dataset commonsense_170k and conduct evaluations on the individual testing dataset for each task. We calculate an average score to encapsulate the overall efficacy. In Subsection§ 4.4, we perform fine-tuning on Math10K Hu et al. () dataset which includes MultiArith, GSM_8K Cobbe et al. (), AddSub,
AQuA, SingleEq, SVAMP datasets and evaluate the efficiency on their testsets.

Training Framework and SMT Hyper-parameters: We used the DeepSpeed Aminabadi et al. () library for fine-tuning and accelerate Gugger et al. () library for inference evaluation. Both training and fine-tuning are using dtype bf16. All experiments are fine-tuned for 3 epochs.
In all our experiments in Section§ 4, sub-matrices are selected in blocks of size $l=256$.
We choose this specific sub-matrix dimension $l$ because it is the largest common factor of the column and row sizes of all linear layers in the LLaMA series models, using this dimension for slicing avoids remainder issues. We freeze all MLP layers and apply SMT only to the Q, K, and V vectors in the attention mechanism.
In Section§5.1, we explain the rationale why we only apply SMT to attention mechanism instead of MLP.
At the end of the gradient warm-up iteration, SMT ranks the average absolute gradient values within each sub-matrix and selects those with the highest average values. The rationale of such selection is explained more in detail in Appdendix B.
We apply 100 warm-up iterations to all SMT experiments on Commonsense dataset and apply 25 warm-up iterations to all SMT experiments on Math10K dataset.

PEFT Baselines: For state-of-the-art (SOTA) baselines, we choose to include LoRA Hu et al. () and DoRA Liu et al. (), both of them focus on fine-tuning using low-rank adaptation method.

Computational Resources: We conduct our experiments and SOTA baselines of LoRA Microsoft () and DoRA Shih-yang () to fine-tune LLaMA-7B and LLaMA2-7B model with 4 NVIDIA A100_40GB GPUs and fine-tune LLaMA-13B and LLaMA3-8B model with 4 NVIDIA A100_80GB GPUs. Communication between the CPU and GPU was facilitated via PCIe-G4 and communication between GPUs was facilitated via Nvlink-3.

Evaluation Metrics:
We have evaluated the performance of SMT in terms of computational efficiency (wall-clock time speedup), memory usage (analysis for memory complexity) in methodology Section§ 3. In this section, we will mainly evaluate SMT in terms of popular NLP tasks to test its ability to generalize to all downstream tasks. In Subsection§ 4.2§ 4.3, we evaluate the performance of SMT on 8 tasks in the Commonsense dataset, including BoolQ, PIQA, SIQA, HellaSwag, ARC-e, ARC-c, and OBQA, and we calculate an average score to encapsulate the overall efficacy.In Subsection§ 4.4, we perform fine-tuning on Math10K Hu et al. () dataset which includes MultiArith, GSM_8K, AddSub,
AQuA, SingleEq, SVAMP datasets and evaluate the efficiency of SMT on their testsets. All of the experiments are evaluated using accuracy.

### 4.2 Commonsense Reasoning

We evaluate SMT against the state-of-the-art(SoTA) weight low-rank adaptor method includes LoRA and DoRA. To ensure
a fair comparison, we fine-tuned model with SMT following the LoRA and DoRA configuration.
We ensure all the hyper-parameters including batch size, data type, learning rate, and sequence length are identical to what was reported in LoRA and DoRA Hu et al. (); Liu et al. ().
We re-implemented LoRA and DoRA and achieved their best performance reported in Liu et al. ().

Table 2 demonstrates that SMT consistently surpasses baseline methods across LLaMA-7B, LLaMA13B, LLaMA2-7B, and LLaMA3-8B. Notably, by overcome plateau phenomenon, SMT further enhances accuracy of DoRA by 3.0%, 2.8%, 2.9%, and 2% on LLaMA-7B, LLaMA-13B, LLaMA2-7B, and LLaMA3-8B respectively.
Notably, LoRA and DoRA will not achieve better performance with larger trainable parameters and exhibit the plateau phenomenon.
In Subsection§ 4.3, we report and demonstrate the plateau issue in LoRA and DoRA and demonstrate SMT overcomes this issue.
Moreover, by fine-tuning less than 5% of all parameters, SMT achieves similar accuracy performance of full fine-tuning while speedup 14.6$\times$ (speedup details in Table  1) and save 99.5% of optimizer memory(memory bottleneck in fine-tuning, details discussed in Section§ 3).

SMT can also consistently surpass LoRA and DoRA under the same number of trainable parameters where LoRA and DoRA achieve the best results, SMT can surpass their performance and also outstrip
ChatGPT-3.5-turbo(^4^44Results of ChatGPT-3.5-turbo are reported in DoRA Shih-yang ()).
For instance, SMT consistently
surpasses DoRA on LLaMA2-7B, LLaMA3-8B, LLaMA-13B, and LLaMA-7B by 1.3%, 1.6%, 0.9%, and 0.3% respectively, under their best performance trainable parameter number.

**Table 2: Accuracy comparison of LLaMA 7B, LLaMA 13B, LLaMA2 7B, and LLaMA3 8B with various PEFT methods on eight commonsense reasoning datasets. Results of all the baseline methods on LLaMA 7B, LLaMA 13B, LLaMA2 7B, LLaMA3 8B are taken from Liu et al. (). Results of all SMT are obtained using the hyper-parameters described in Liu et al. () under the same settings. Bold texts dedicate the performance of SMT under the same numbers of parameters where LoRA and DoRA achieve the best performance. Blue texts dedicate the best performance of SMT. Please note that the performance of LoRA and DoRA under larger numbers of trainable parameters can be found in Table 3.**
| Model | PEFT method | #Params% | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| ChatGPT(175B) | - | - | 73.1 | 85.4 | 68.5 | 78.5 | 66.1 | 89.8 | 79.9 | 74.8 | 77.0 |
| LLaMA-7B | LoRA(Best) | 0.83 | 67.5 | 80.8 | 78.2 | 83.4 | 80.4 | 78.0 | 62.6 | 79.1 | 76.3 |
| DoRA(Best) | 0.84 | 69.7 | 83.4 | 78.6 | 87.2 | 81.0 | 81.9 | 66.2 | 79.2 | 78.4 |  |
| SMT | 0.84 | 68.7 | 81.7 | 78.3 | 91.6 | 78.8 | 84.1 | 68.7 | 77.4 | 78.7 |  |
| SMT(Best) | 4.91 | 72.0 | 82.9 | 80.7 | 93.3 | 82.4 | 86.1 | 70.6 | 83.0 | 81.4 |  |
|  | Full Fine-tuning | 100 | 69.9 | 84.2 | 78.9 | 92.3 | 83.3 | 86.6 | 72.8 | 83.4 | 81.4 |
| LLaMA-13B | LoRA(Best) | 0.67 | 72.1 | 83.5 | 80.5 | 90.5 | 83.7 | 82.8 | 68.3 | 82.4 | 80.5 |
| DoRA(Best) | 0.68 | 72.4 | 84.9 | 81.5 | 92.4 | 84.2 | 84.2 | 69.6 | 82.8 | 81.5 |  |
| SMT | 0.68 | 71.1 | 84.4 | 81.7 | 93.7 | 83.2 | 86.7 | 73.7 | 85.2 | 82.4 |  |
| SMT(Best) | 4.91 | 72.6 | 86.1 | 81.9 | 95.0 | 86.1 | 88.2 | 77.1 | 87.4 | 84.3 |  |
| LLaMA2-7B | LoRA(Best) | 0.83 | 69.8 | 79.9 | 79.5 | 83.6 | 82.6 | 79.8 | 64.7 | 81.0 | 77.6 |
| DoRA(Best) | 0.42 | 72.0 | 83.1 | 79.9 | 89.1 | 83.0 | 84.5 | 71.0 | 81.2 | 80.5 |  |
| SMT | 0.84 | 72.0 | 83.8 | 80.8 | 93.3 | 82.8 | 86.7 | 74.0 | 81.0 | 81.8 |  |
| SMT(Best) | 4.91 | 72.6 | 85.2 | 82.0 | 94.4 | 85.7 | 87.8 | 74.5 | 85.0 | 83.4 |  |
|  | Full Fine-tuning | 100 | 72.8 | 83.4 | 78.7 | 92.7 | 85.5 | 86.2 | 74.7 | 83.4 | 82.2 |
| LLaMA3-8B | LoRA(Best) | 0.70 | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79.0 | 80.8 |
| DoRA(Best) | 0.71 | 74.6 | 89.3 | 79.9 | 95.5 | 85.6 | 90.5 | 80.4 | 85.8 | 85.2 |  |
| SMT | 0.71 | 75.7 | 88.4 | 81.4 | 96.2 | 88.2 | 92.7 | 83.2 | 88.6 | 86.8 |  |
| SMT(Best)) | 3.01 | 75.1 | 89.9 | 82.4 | 96.3 | 88.8 | 92.6 | 82.8 | 89.6 | 87.2 |  |

### 4.3 Plateau in Weight low rank adaption methods

In Table 3 and Figure 3, we scale up the model size and presents how the performance of LoRA and DoRA will be under larger number of trainable parameters. We reimplement all the experiments of LoRA Microsoft () and DoRA Shih-yang () using their official repository and followed their recommendation of hyper-parameters to achieve best performance under every single trainable parameter size.
We observe that both DoRA and LoRA models, with some larger ranks, their performance slightly degrades.
However, SMT continues improving its performance when we scale up the trainable parameter size. When we scale up the trainable parameter size to 4.91%, SMT significantly surpass DoRA by 3.8% and 4.9% on LLaMA-7B and LLaMA-2-7B fine-tuned models.
We postulate that such plateau phenomenon of LoRA or DoRA is due to their lossy low-rank approximation of the full weight information (includes lots of noise), whereas our SMT focuses on most prominent submatrices (contains less noise) and remains full rank gradient updates for the selected portion, making SMT performs better.

Figure: Figure 3: Accuracy comparison of LoRA, DoRA, and SMT under different scaling of trainable parameters on commonsense reasoning datasets.
Refer to caption: extracted/5630734/plateau_plot.png

Figure: Table 3: Accuracy comparison of LoRA, DoRA, and SMT under different scaling of trainable parameters on commonsense reasoning datasets. Given certain base model and PEFT method, we gradually increase the number of trainable parameters on each line from left to right. On each line, the best performing model has ^∗.

### 4.4 Other Dataset

To ensure our findings above are generalizable, we further examine the performance of SMT under arithmetic reasoning dataset, Math10K Hu et al. (). Math10K dataset has six subsets including GSM8k, SingleEq, SVAMP, MultiArith, AddSub, and AQuA.
More details about Math10K dataset can be found in Appendix C. To ensure a fair comparison, we follow the open source hyper-parameter instruction in Hu et al. () to achieve best performance for LoRA and Dora, and apply the same hyper-parameters to SMT while only fine-tune the learning rate. Table  5 reports the performance of LoRA, DoRA, and SMT on the Math10K dataset. We can observe that SMT surpasses the best achievable performance of LoRA and DoRA by 1.3% and 1.1% respectively using the same amount of trainable parameters. In addition, by scaling up the trainable model size to 1.26%, SMT achieves better performance and surpasses the best performance of LoRA and DoRA by 2.5% and 2.3% respectively.

**Table 5: SMT, LoRA and DoRA reproduction, and experiment results on Math10K dataset.**
| Model | PEFT method | #Params% | GSM8k | SingleEq | SVAMP | MultiArith | AddSub | AQuA | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-7B | LoRA(Best) | 0.86 | 35.4 | 83.2 | 52.1 | 92.8 | 83.4 | 18.6 | 60.9 |
| DoRA(Best) | 0.86 | 35.2 | 83.7 | 51.8 | 92.8 | 82.8 | 20.2 | 61.1 |  |
| SMT | 0.86 | 34.2 | 84.6 | 53.6 | 91.5 | 85.8 | 23.6 | 62.2 |  |
|  | SMT(Best) | 1.26 | 35.6 | 85.3 | 54.8 | 93.4 | 86.8 | 24.2 | 63.4 |

## 5 Further Dicussion

### 5.1 Attention versus MLP

In order to study what components are more critical for LLM’s downstream performance, we conduct ablation studies that compares MLPs vs. attention layers by adjusting the ratio of their trainable parameters respectively.
We apply SMT and fine-tune 0.86% of parameters on LLaMA-7B using Commonsense dataset. In Table 4, we present four experiments. In the first row, all trainable parameters are allocated to MLPs. In the second row, both MLPs and Q, K, V vectors from attention mechanisms receive 0.43% of trainable parameters. In the third row, 0.62% of trainable parameters are assigned to Q, K, V vectors from attention mechanisms and 0.21% to MLPs. In the fourth row, all trainable parameters are dedicated to Q, K, V vectors from attention mechanisms. To guarantee a fair comparison, all the other hyper-parameters and settings are the same among these experiments.

In these experiments, allocating X% of trainable parameters to MLPs or attention mechanisms means ranking the average absolute gradient values of each sub-matrix within the MLPs or attention mechanisms and selecting those with the highest average values until the number of parameters reaches X%. The results reveal a significant performance gap between the first and fourth rows. The more trainable parameters we allocate to attention mechanisms, the better the fine-tuned model performs. When all SMT trainable parameters are applied to attention mechanisms, the model outperforms the one where all parameters are allocated to MLPs by 2.0%.
Our empirical findings challenge previous assumptions Zhu et al. (); Meng et al. (); Geva et al. (, ) that the memory sections of large language models are primarily located in feed-forward MLP layers.

### 5.2 V Vector Versus Q, K Vector

**Table 6: K SMT, Q SMT, and V SMT assign all trainable parameters to only K, or only Q, or only V vectors respectively, and fine-tuned 0.86% of the parameters on LLaMA-7B using the Commonsense dataset. QKV SMT assign all trainable parameters to QKV vectors and select sub-matrices automatically.**
| Model | Param location | #Params% | BoolQ | PIQA | SIQA | HellaSwag | WinoGrande | ARC-e | ARC-c | OBQA | AVG |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-7B | K SMT | 0.84 | 65.5 | 79.1 | 76.2 | 88.3 | 73.2 | 80.3 | 60.8 | 68.0 | 73.9 |
| Q SMT | 0.84 | 65.7 | 79.3 | 75.5 | 88.2 | 72.5 | 80.1 | 59.6 | 72.5 | 75.3 |  |
| V SMT | 0.84 | 68.7 | 82.1 | 78.1 | 91.6 | 78.8 | 83.0 | 68.7 | 77.2 | 78.5 |  |
| QKV SMT | 0.84 | 68.7 | 81.7 | 78.3 | 91.6 | 78.8 | 84.1 | 68.7 | 77.4 | 78.7 |  |

Based on our observation that attention is more critical, in all of our SMT experiments in Section§ 4, we only allocate SMT trainable parameters to Q, K, V vectors from attention mechanisms. We rank the average absolute gradient values of every single sub-matrix within attention mechanisms and select those with the highest average values until the parameter ratio limit is reached.
Surprisingly, we observed that the trainable parameters are predominantly assigned to the V vectors.
In our ablation experiments, we experimented with assigning all trainable parameters to only K, or only Q, or only V vectors, and fine-tuned 0.86% of the parameters on LLaMA-7B using the Commonsense dataset.
As shown in Figure 5, 95.17% of the trainable parameters are automatically assigned to the V vectors by SMT. Figure 5 indicates that all V vectors have trainable parameters, while 22 Q vectors and 21 K vectors are completely frozen.
This suggest that  the V vectors contain most of the memory and are the most significant among the Q, K, and V.

Figure: Figure 4: A visualization of trainable Q, K, V layers when fine-tuning 0.86% trainable parameters on LLaMA-7B. LLaMA-7B has 32 layers of MLPs, each contains a Q vector, a K vector, and a V vector. White layers are frozen and green layers contain trainable parameters.
Refer to caption: extracted/5630734/QKV_diff.png

Table 6 presents four additional experiments where we fine-tuned 0.86% of the parameters of LLaMA-7B using SMT on the Commonsense dataset.
In the first three rows, all trainable parameters are allocated to the K vectors, Q vectors, and V vector , respectively. In the fourth row, the trainable parameters are assigned to Q, K, V vectors directly and allocated by SMT automatically. The trainable parameters are distributed among the K, Q, and V vectors, as detailed in Figure 5, with the trainable states of the QKV layers shown in Figure 5.

The results show a significant performance gap when comparing the allocation of all trainable parameters to the V vectors versus the Q and K vectors. Assigning all parameters to the V vectors outperforms the K vectors by 4.6% and the Q vectors by 3.2%.
These observations suggest that the V vectors are the most significant among the Q, K, and V vectors; it also hints that SMT is able to effectively select sub-matrices containing crucial memory sections.

## 6 Conclusion

Our proposed Sparse Matrix Tuning (SMT) achieves SoTA performance, narrowing the gap between SMT and full fine-tuning.
SMT can also reduce the computational cost of backward propagation, parameter updates, optimizer memory, and activation memory during fine-tuning to achieve 14.6$\times$ speedup.
The empirical evidence presented in our extensive experiments suggests that attention layers are more critical than MLPs for downstream performance; V vector is the most influential vector for performance among Q, K, V vectors.

## 7 Acknowledgements

We extend our gratitude to Beidi Chen and Xinyu Yang for the discussion and their valuable feedback. We thank Jixuan Leng and Geyang Guo for their full fine-tuning baseline implementation. We also thank Christina Ma for her suggestions.

## Appendix A Implementation Details with Code Snippets

Figure: Figure 6: Implementation of customized sparse linear layer.
Refer to caption: extracted/5630734/code_sparse_linear_layer.png

Figure: Figure 7: Implementation of customized forward in specialized sparse linear multiplication function.
Refer to caption: extracted/5630734/linearfunction_forward.png

Figure: Figure 8: Implementation of customized backward in specialized sparse linear multiplication function.
Refer to caption: extracted/5630734/linearfunction_backward.png

## Appendix B Sub-Matrices Selection

SMT ranks the average absolute gradient values within each sub-matrix and selects those with the highest averages. The rationale behind this selection process is to enable SMT to automatically identify sub-matrices containing memory that is most relevant to downstream tasks. During fine-tuning, the absolute gradient values can indicate the relevance of a block to these tasks hence it requires more tuning. By averaging the absolute gradient values within each sub-matrix, we can determine the importance of the sub-matrix to specific downstream tasks.

## Appendix C Math10K Dataset

Math10K dataset can evaluate the effectiveness of LLMs on the arithmetic reasoning task. Math10K incorporate six subsets including GSM8k, SingleEq, SVAMP, MultiArith, AddSub, and AQuA.(1) the GSM8K Cobbe et al. [] dataset consists of high quality linguistically diverse grade school math word problems created by human problem
writers, (2) the SVAMP Patel et al. [] benchmark consists of one-unknown arithmetic word
problems for up-to-4 grade level students by making simple changes to a set of problems from another existing dataset, (3) the MultiArith Roy and Roth [] dataset of math word problems requiring multiple reasoning steps and operations, (4) the AddSub Hosseini et al. [] dataset of addition and subtraction arithmetic word problems,
(5) the AQuA Ling et al. [] dataset of algebraic word problems with natural language rationales, and (6) the SingleEq Koncel-Kedziorski et al. [] dataset of grade-school algebra word problems that map to single equations with varying length;

## Peer Reviews

confidence: 2
contribution: 3
initial_rating: 6
presentation: 2
questions: The implementation specifies (L242) "we use 256 x 256 as sub-matri[x] block size". L260 states "Despite employing
matrix sparsity, we still leverage the advantages of dense matrix multiplication." What happens when it is per-parameter (ie 1x1)? Is there a tradeoff between method performance and efficiency being made in selecting sub-matrices instead of free parameters?  It seems the only reason to have blocks is computational simplicity. The motivation behind this choice should be made explicit, and the tradeoff investigated. What happens with a larger block size?

Nits:
Figure 1 contrasts two techniques, but appears on first glance to be 4 boxes equally spaced. Consider modifying the figure to make more visually apparent the separateness of the techniques.
L 054: submatrix -> submatrices, right?
L 147: the whole pre-trained weight -> all the pretrained weights?
L 163: for downstream -> for *improving* downstream
L 238: sum up gradient -> sum up the gradient / sum up gradients
L 239: iterations -> iteration, sum up gradient information are -> summed up gradient information is
L 387:  "commonsense reasoning datasets" in Table 3 vs COMMONSENSE in Table 4... are these the same? If so, refer using same term
Table 3 should be a plot.
L426-429: this is highly repetetive, please revise
Throughout: please stay within a single tense within a single paragraph (L302-317 especially, but also elsewhere)
Throughout: missing articles, incorrect pluralization or lack thereof
soundness: 3
strengths: The problem addressed is important, the evaluation results are good. The approach could be significant to the field.
summary: This paper presents a novel PEFT method, Sparse Matrix Tuning. SMT uses a gradient based criteria to identify the most relevant submatrices of the model to update, and updates only those while freezing the remaining params during training. This allows for memory and speed gains during training which are similar or better to those made by other PEFT methods. SMT performs well compared to strong baseline PEFT methods LoRA DoRA and SpIEL.
weaknesses: The method is poorly described in the section dedicated to its description. What exactly is happening during warm-up phase? It seems that the warm up phase involves taking 100 steps (on the target dataset?) *without updating the model params*, and taking the average of abs-value of gradients for each param of the model to identify sub-matrices. This is critical to specifying the methodology, and should be clearly stated, not deduced by the reader. 

No experiments show ablations on the specification criteria, or the size of the 256x256 submatrix. The justification for restricting SMT to the KQV of the attention mechanism is relatively weak. The sole justification for restricting the SMT trainable params to the attention is a strange ablation comparing hardcoded percentages interpolating between only MLP and only attention. A more natural ablation would allow the selection criteria to place trainable submatrices wherever in the entire model as a baseline, and compare to a variety of restricted alternatives. 

The writing throughout is relatively poor. Articles are frequently missing, pluralization is often wrong, tense is frequently switched. This can easily be fixed, and would greatly improve readability.

Generally, the method is sound, though poorly described. The experiments have good results, but the analysis leaves some to be desired.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: None
soundness: 3
strengths: 1. This work contributes to the growing field of parameter-efficient fine-tuning, a critical area for the advancement of LLMs. Efficient tuning techniques like SMT can play a major role in making large models more accessible and practical for diverse applications.

2. The paper provides an insightful analysis of activation-awareness and gradient-awareness methods, ultimately selecting the gradient-based approach as more effective for identifying meaningful submatrices. This comparison enriches the understanding of PEFT techniques and optimizations.

3. The SMT approach offers an improved strategy for identifying impactful sub-networks, which potentially boosts fine-tuning efficacy by focusing on parameters that most influence downstream performance.
summary: This paper introduces Sparse Matrix Tuning (SMT), a parameter-efficient fine-tuning (PEFT) method for large language models (LLMs), aiming to reduce computational and memory costs while maintaining performance close to full fine-tuning (FT). SMT selectively updates only the most significant submatrices in a model's weight matrix, chosen based on gradient changes, enabling efficient tuning without compromising on model accuracy. The authors compare SMT with established PEFT methods such as LoRA and DoRA, demonstrating that SMT outperforms these methods in both accuracy and computational efficiency, especially by overcoming the performance plateau that other low-rank adaptation methods encounter with increased parameters.
weaknesses: 1. While SMT improves on existing PEFT methods, the approach of selecting sub-networks for fine-tuning is not particularly novel, as similar sparse selection methods have been explored previously in transfer learning and model pruning.

2. The paper does not address whether SMT offers better resistance to catastrophic forgetting compared to other PEFT methods. This omission leaves questions about SMT's stability and robustness in long-term, iterative fine-tuning scenarios.

3. The paper lacks some necessary conceptual explanations and mathematical formulations, such as detailed descriptions of submatrix selection and significance metrics. This limited detail could hinder reproducibility and a deeper understanding of SMT's implementation and optimizations.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. In Table 2, the authors provide full fine-tuning results only for LLaMA-7B and LLaMA2-7B. Adding full fine-tuning results for additional models would help to better contextualize SMT's performance advantage and strengthen the comparison with full fine-tuning methods.
2. It is unclear how the sparsity hyperparameter $z$ is determined in the experiments. While the #Params% results seem largely influenced by LoRA ranks, SMT should, in principle, allow for a more flexible selection of sparsity levels. Providing details on the criteria or method used to select $z$ would clarify this aspect and help illustrate SMT's potential for adaptable parameter tuning.


Typos and writing issues.

1. In line 072, a colon is missing.
2. In lines 146–148, there is an unnecessary space before the reference term.
3. The use of bold and italic formatting is excessive in multiple sentences. This can make the text appear cluttered and distracting. Reducing the use of bold and italics to highlight only the most essential points will make the writing cleaner and enhance readability.
soundness: 3
strengths: 1. SMT inherits the performance benefits of prior sparse tuning methods while addressing their shortcomings in computational and memory efficiency.
2. Analysis reveals that methods like LoRA, DoRA, and SpIEL suffer from performance saturation as tunable parameters scale, whereas SMT maintains consistent performance without this limitation.
3. The gradient-based parameter selection in SMT enables deeper analysis of critical parameter distributions within LLMs. Results indicate that the Attention module, particularly the value (V) vector, requires more intensive fine-tuning in LLMs. This observation provides valuable insights and could potentially inform future studies in parameter-efficient tuning.
summary: This paper introduces a novel parameter-efficient fine-tuning (PEFT) method, termed Sparse Matrix Tuning (SMT). SMT builds on insights from prior research suggesting that domain-specific knowledge can be localized and sparsely distributed across layers. SMT identifies the most relevant sparse submatrices by analyzing gradients and then selectively fine-tunes only these submatrices. Experimental results demonstrate that SMT not only outperforms previous PEFT methods but also achieves performance levels close to full parameter fine-tuning. Further analysis reveals that SMT effectively mitigates the performance saturation commonly observed with other PEFT approaches.
weaknesses: 1. **Gap between Claimed and Actual Contribution:** While the authors claim that SMT aims to "close the performance gap between PEFT and full fine-tuning," SMT’s primary focus appears to be on improving the computational and storage efficiency of existing sparse tuning methods rather than introducing fundamentally new fine-tuning strategies. This creates a disconnect between the paper’s stated goal of performance enhancement and its actual contributions, which seem more oriented toward efficiency. To strengthen the paper, it would be beneficial for the authors to align their claims with the actual contributions more closely.
2. **Lack of Theoretical Support:** The claim that SMT can "close the performance gap between PEFT and full fine-tuning" lacks sufficient theoretical justification. The paper primarily discusses SMT’s implementation details and efficiency improvements without explaining why fine-tuning a sparse matrix should lead to performance gains comparable to full fine-tuning. Providing a theoretical explanation or framework to support this belief would significantly strengthen the paper. For instance, the authors could explore how sparse matrix fine-tuning might preserve essential parameter interactions or discuss relevant properties of sparse representations that could theoretically justify this performance potential.
3. **Missing Baselines:** While the paper primarily focuses on enhancing the computational and storage efficiency of sparse tuning methods, the inclusion of additional sparse tuning baselines would provide a more comprehensive evaluation. Comparing SMT with other sparse tuning methods would highlight its specific advantages in efficiency. Additionally, several recent studies, such as LoRA-GA [1] and LoRA-Pro [2], also aim to bridge the gap between PEFT and full fine-tuning. Including these methods in both the discussion and experiments would allow for a more thorough assessment of SMT's performance and effectiveness relative to other state-of-the-art approaches.

[1] Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.

[2] Wang Z, Liang J. LoRA-Pro: Are Low-Rank Adapters Properly Optimized?[J]. arXiv preprint arXiv:2407.18242, 2024.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the weaknesses.
soundness: 3
strengths: 1. The sparse matrix tuning method seems reasonable for improving downstream task performance.

2. The results are quite positive and the analyses are extensive.

3. The two interesting findings that attention layers and value matrix are more influential may inspire new insights in the field of parameter-efficient fine-tuning.
summary: This paper proposes a new parameter-efficient fine-tuning method which only fine-tunes the task-specific sub-matrix for large language models. The method first identifies the salient sub-matrix for each task with gradient-aware selection strategy. Then, only the salient sub-matrix will be optimized during the fine-tuning stage. The experiments demonstrate that the proposed method can achieve comparable or even better performance than full fine-tuning while requiring updating tiny part of parameters. This work also finds that fine-tuning attention layers are more critical than MLPs and the value matrix v is more influential than the query and key matrix Q, K.
weaknesses: 1. This SMT method requires warmup stage to identify the salient sub-matrix in LLMs and the warmup iterations rely on the specific downstream tasks and dataset size. It makes the method somewhat complicated to use. The generalization is limited.

2. Compared to LoRA, this SMT method needs to update a subset of parameters of the original LLMs and it will change the original model, so that the original LLMs cannot be reused for downstream tasks.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - What is the intention behind using 256x256 sub-matrices? What happens in terms of quality and efficiency when you use smaller (down to 1x1) or larger sub-blocks?
- iiuc the warm-up phase still needs the same memory as full fine-tuning. Can you avoid this phase by reusing sub-block selection from a different [multi-task] experiment for a new task?
- The PET Speed-Ups in Table 1 seem fairly large to me.. I would expect 2-4x speed ups by LoRA, not ~14x. Could you cite work that reports LoRA speed-ups over full fine-tuning in a similar ballpark?
- L223: is the gradient step computation cost really z% cheaper? I understand the argument for a single parameter matrix, but don't you need full gradients in layers closer to the output layer to correctly propagate the gradients to this point? Same for the activation costs in the forward pass (L223)
- Why are some of the full fine-tuning results in Table 2 missing?
soundness: 3
strengths: - Good empirical evaluation across multiple model sizes and datasets
- Insightful ablations in Sec. 5 and the appendix
summary: The PET method proposed in this paper divides parameter matrices into subblocks. Only a subset of these subblocks are changed in fine-tuning, everything else is frozen. The authors demonstrate convincing advantages over LoRA.
weaknesses: - Some practical questions (e.g. block size and mask generalization) are not addressed
- No comparison with other delta tuning / diff pruning approaches and random masking (https://arxiv.org/abs/2405.02596, https://aclanthology.org/2023.emnlp-main.931.pdf)

Meta-review: This paper introduces Sparse Matrix Tuning (SMT), a parameter-efficient fine-tuning method for large language models. By selectively updating the most significant submatrices based on gradient analysis, SMT reduces computational and memory costs while maintaining performance levels close to full fine-tuning. The method outperforms established PEFT approaches like LoRA and DoRA, effectively mitigating the performance plateau seen in other methods. SMT emphasizes the importance of fine-tuning attention layers, showing that the value matrix is particularly influential. Overall, SMT offers significant efficiency gains without sacrificing accuracy.

On the positive note, this paper advances the field of parameter-efficient fine-tuning (PEFT) for large language models (LLMs) through SMT. SMT uses a gradient-based method to identify and update the most impactful submatrices, enhancing computational and memory efficiency while maintaining performance. It overcomes limitations of previous methods, which suffer from performance saturation as tunable parameters increase. SMT provides new insights, particularly highlighting the importance of fine-tuning attention layers and the value (V) vector, which may inform future research. The method is supported by comprehensive empirical evaluations and extensive analyses, demonstrating its potential to improve downstream task performance effectively. Based on the aforementioned review, I would be leaning to accept the paper.

Decision: Accept (Poster)