SMT: Fine-Tuning Large Language Models with Sparse Matrices

Authors: Haoze He, Juncheng B Li, Xuan Jiang, Heather Miller
Primary area: foundation or frontier models, including LLMs
Keywords: Parameter-efficient Finetuning, Large Language Model, Large Language Model Systm

Abstract:
Various parameter-efficient fine-tuning (PEFT) methods, including LoRA and its variants, have gained popularity for reducing computational costs. However, there is often an accuracy gap between PEFT approaches and full fine-tuning (FT), and this discrepancy has not yet been systematically explored. In this work, we introduce a method for selecting sparse sub-matrices that aims to minimize the performance gap between PEFT vs. full fine-tuning (FT) while also reducing both fine-tuning computational costs and memory costs. We explored both gradient-based and activation-based parameter selection methods to identify the most significant sub-matrices for downstream tasks, updating only these blocks during fine-tuning. In our experiments, we demonstrated that SMT consistently surpasses other PEFT
baselines (e.g., LoRA and DoRA) in fine-tuning popular large language models such as LLaMA across a broad spectrum of tasks, while reducing the GPU memory footprint by 67% compared to FT. We also examine how the performance of LoRA and DoRA tends to plateau and decline as the number of trainable parameters increases, in contrast, our SMT method does not suffer from such issues.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730118990508

thread_type:
official_review

<thread_object>
creator: reviewer_kjyr

<content>
summary:
The PET method proposed in this paper divides parameter matrices into subblocks. Only a subset of these subblocks are changed in fine-tuning, everything else is frozen. The authors demonstrate convincing advantages over LoRA.

strengths:
- Good empirical evaluation across multiple model sizes and datasets
- Insightful ablations in Sec. 5 and the appendix

weaknesses:
- Some practical questions (e.g. block size and mask generalization) are not addressed
- No comparison with other delta tuning / diff pruning approaches and random masking (https://arxiv.org/abs/2405.02596, https://aclanthology.org/2023.emnlp-main.931.pdf)

questions:
- What is the intention behind using 256x256 sub-matrices? What happens in terms of quality and efficiency when you use smaller (down to 1x1) or larger sub-blocks?
- iiuc the warm-up phase still needs the same memory as full fine-tuning. Can you avoid this phase by reusing sub-block selection from a different [multi-task] experiment for a new task?
- The PET Speed-Ups in Table 1 seem fairly large to me.. I would expect 2-4x speed ups by LoRA, not ~14x. Could you cite work that reports LoRA speed-ups over full fine-tuning in a similar ballpark?
- L223: is the gradient step computation cost really z% cheaper? I understand the argument for a single parameter matrix, but don't you need full gradients in layers closer to the output layer to correctly propagate the gradients to this point? Same for the activation costs in the forward pass (L223)
- Why are some of the full fine-tuning results in Table 2 missing?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730295731445

thread_type:
official_review

<thread_object>
creator: reviewer_5qay

<content>
summary:
This paper proposes a new parameter-efficient fine-tuning method which only fine-tunes the task-specific sub-matrix for large language models. The method first identifies the salient sub-matrix for each task with gradient-aware selection strategy. Then, only the salient sub-matrix will be optimized during the fine-tuning stage. The experiments demonstrate that the proposed method can achieve comparable or even better performance than full fine-tuning while requiring updating tiny part of parameters. This work also finds that fine-tuning attention layers are more critical than MLPs and the value matrix v is more influential than the query and key matrix Q, K.

strengths:
1. The sparse matrix tuning method seems reasonable for improving downstream task performance.

2. The results are quite positive and the analyses are extensive.

3. The two interesting findings that attention layers and value matrix are more influential may inspire new insights in the field of parameter-efficient fine-tuning.

weaknesses:
1. This SMT method requires warmup stage to identify the salient sub-matrix in LLMs and the warmup iterations rely on the specific downstream tasks and dataset size. It makes the method somewhat complicated to use. The generalization is limited.

2. Compared to LoRA, this SMT method needs to update a subset of parameters of the original LLMs and it will change the original model, so that the original LLMs cannot be reused for downstream tasks.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730562628859

thread_type:
official_review

<thread_object>
creator: reviewer_2m9a

<content>
summary:
This paper introduces a novel parameter-efficient fine-tuning (PEFT) method, termed Sparse Matrix Tuning (SMT). SMT builds on insights from prior research suggesting that domain-specific knowledge can be localized and sparsely distributed across layers. SMT identifies the most relevant sparse submatrices by analyzing gradients and then selectively fine-tunes only these submatrices. Experimental results demonstrate that SMT not only outperforms previous PEFT methods but also achieves performance levels close to full parameter fine-tuning. Further analysis reveals that SMT effectively mitigates the performance saturation commonly observed with other PEFT approaches.

strengths:
1. SMT inherits the performance benefits of prior sparse tuning methods while addressing their shortcomings in computational and memory efficiency.
2. Analysis reveals that methods like LoRA, DoRA, and SpIEL suffer from performance saturation as tunable parameters scale, whereas SMT maintains consistent performance without this limitation.
3. The gradient-based parameter selection in SMT enables deeper analysis of critical parameter distributions within LLMs. Results indicate that the Attention module, particularly the value (V) vector, requires more intensive fine-tuning in LLMs. This observation provides valuable insights and could potentially inform future studies in parameter-efficient tuning.

weaknesses:
1. **Gap between Claimed and Actual Contribution:** While the authors claim that SMT aims to "close the performance gap between PEFT and full fine-tuning," SMT’s primary focus appears to be on improving the computational and storage efficiency of existing sparse tuning methods rather than introducing fundamentally new fine-tuning strategies. This creates a disconnect between the paper’s stated goal of performance enhancement and its actual contributions, which seem more oriented toward efficiency. To strengthen the paper, it would be beneficial for the authors to align their claims with the actual contributions more closely.
2. **Lack of Theoretical Support:** The claim that SMT can "close the performance gap between PEFT and full fine-tuning" lacks sufficient theoretical justification. The paper primarily discusses SMT’s implementation details and efficiency improvements without explaining why fine-tuning a sparse matrix should lead to performance gains comparable to full fine-tuning. Providing a theoretical explanation or framework to support this belief would significantly strengthen the paper. For instance, the authors could explore how sparse matrix fine-tuning might preserve essential parameter interactions or discuss relevant properties of sparse representations that could theoretically justify this performance potential.
3. **Missing Baselines:** While the paper primarily focuses on enhancing the computational and storage efficiency of sparse tuning methods, the inclusion of additional sparse tuning baselines would provide a more comprehensive evaluation. Comparing SMT with other sparse tuning methods would highlight its specific advantages in efficiency. Additionally, several recent studies, such as LoRA-GA [1] and LoRA-Pro [2], also aim to bridge the gap between PEFT and full fine-tuning. Including these methods in both the discussion and experiments would allow for a more thorough assessment of SMT's performance and effectiveness relative to other state-of-the-art approaches.

[1] Wang S, Yu L, Li J. LoRA-GA: Low-Rank Adaptation with Gradient Approximation[J]. arXiv preprint arXiv:2407.05000, 2024.

[2] Wang Z, Liang J. LoRA-Pro: Are Low-Rank Adapters Properly Optimized?[J]. arXiv preprint arXiv:2407.18242, 2024.

questions:
1. In Table 2, the authors provide full fine-tuning results only for LLaMA-7B and LLaMA2-7B. Adding full fine-tuning results for additional models would help to better contextualize SMT's performance advantage and strengthen the comparison with full fine-tuning methods.
2. It is unclear how the sparsity hyperparameter $z$ is determined in the experiments. While the #Params% results seem largely influenced by LoRA ranks, SMT should, in principle, allow for a more flexible selection of sparsity levels. Providing details on the criteria or method used to select $z$ would clarify this aspect and help illustrate SMT's potential for adaptable parameter tuning.


Typos and writing issues.

1. In line 072, a colon is missing.
2. In lines 146–148, there is an unnecessary space before the reference term.
3. The use of bold and italic formatting is excessive in multiple sentences. This can make the text appear cluttered and distracting. Reducing the use of bold and italics to highlight only the most essential points will make the writing cleaner and enhance readability.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730792972997

thread_type:
official_review

<thread_object>
creator: reviewer_ovmn

<content>
summary:
This paper introduces Sparse Matrix Tuning (SMT), a parameter-efficient fine-tuning (PEFT) method for large language models (LLMs), aiming to reduce computational and memory costs while maintaining performance close to full fine-tuning (FT). SMT selectively updates only the most significant submatrices in a model's weight matrix, chosen based on gradient changes, enabling efficient tuning without compromising on model accuracy. The authors compare SMT with established PEFT methods such as LoRA and DoRA, demonstrating that SMT outperforms these methods in both accuracy and computational efficiency, especially by overcoming the performance plateau that other low-rank adaptation methods encounter with increased parameters.

strengths:
1. This work contributes to the growing field of parameter-efficient fine-tuning, a critical area for the advancement of LLMs. Efficient tuning techniques like SMT can play a major role in making large models more accessible and practical for diverse applications.

2. The paper provides an insightful analysis of activation-awareness and gradient-awareness methods, ultimately selecting the gradient-based approach as more effective for identifying meaningful submatrices. This comparison enriches the understanding of PEFT techniques and optimizations.

3. The SMT approach offers an improved strategy for identifying impactful sub-networks, which potentially boosts fine-tuning efficacy by focusing on parameters that most influence downstream performance.

weaknesses:
1. While SMT improves on existing PEFT methods, the approach of selecting sub-networks for fine-tuning is not particularly novel, as similar sparse selection methods have been explored previously in transfer learning and model pruning.

2. The paper does not address whether SMT offers better resistance to catastrophic forgetting compared to other PEFT methods. This omission leaves questions about SMT's stability and robustness in long-term, iterative fine-tuning scenarios.

3. The paper lacks some necessary conceptual explanations and mathematical formulations, such as detailed descriptions of submatrix selection and significance metrics. This limited detail could hinder reproducibility and a deeper understanding of SMT's implementation and optimizations.

questions:
None
</content>
</thread_object>

</thread>
<thread>
creation date:
1730837968267

thread_type:
official_review

<thread_object>
creator: reviewer_tax9

<content>
summary:
This paper presents a novel PEFT method, Sparse Matrix Tuning. SMT uses a gradient based criteria to identify the most relevant submatrices of the model to update, and updates only those while freezing the remaining params during training. This allows for memory and speed gains during training which are similar or better to those made by other PEFT methods. SMT performs well compared to strong baseline PEFT methods LoRA DoRA and SpIEL.

strengths:
The problem addressed is important, the evaluation results are good. The approach could be significant to the field.

weaknesses:
The method is poorly described in the section dedicated to its description. What exactly is happening during warm-up phase? It seems that the warm up phase involves taking 100 steps (on the target dataset?) *without updating the model params*, and taking the average of abs-value of gradients for each param of the model to identify sub-matrices. This is critical to specifying the methodology, and should be clearly stated, not deduced by the reader. 

No experiments show ablations on the specification criteria, or the size of the 256x256 submatrix. The justification for restricting SMT to the KQV of the attention mechanism is relatively weak. The sole justification for restricting the SMT trainable params to the attention is a strange ablation comparing hardcoded percentages interpolating between only MLP and only attention. A more natural ablation would allow the selection criteria to place trainable submatrices wherever in the entire model as a baseline, and compare to a variety of restricted alternatives. 

The writing throughout is relatively poor. Articles are frequently missing, pluralization is often wrong, tense is frequently switched. This can easily be fixed, and would greatly improve readability.

Generally, the method is sound, though poorly described. The experiments have good results, but the analysis leaves some to be desired.

questions:
The implementation specifies (L242) "we use 256 x 256 as sub-matri[x] block size". L260 states "Despite employing
matrix sparsity, we still leverage the advantages of dense matrix multiplication." What happens when it is per-parameter (ie 1x1)? Is there a tradeoff between method performance and efficiency being made in selecting sub-matrices instead of free parameters?  It seems the only reason to have blocks is computational simplicity. The motivation behind this choice should be made explicit, and the tradeoff investigated. What happens with a larger block size?

Nits:
Figure 1 contrasts two techniques, but appears on first glance to be 4 boxes equally spaced. Consider modifying the figure to make more visually apparent the separateness of the techniques.
L 054: submatrix -> submatrices, right?
L 147: the whole pre-trained weight -> all the pretrained weights?
L 163: for downstream -> for *improving* downstream
L 238: sum up gradient -> sum up the gradient / sum up gradients
L 239: iterations -> iteration, sum up gradient information are -> summed up gradient information is
L 387:  "commonsense reasoning datasets" in Table 3 vs COMMONSENSE in Table 4... are these the same? If so, refer using same term
Table 3 should be a plot.
L426-429: this is highly repetetive, please revise
Throughout: please stay within a single tense within a single paragraph (L302-317 especially, but also elsewhere)
Throughout: missing articles, incorrect pluralization or lack thereof
</content>
</thread_object>

</thread>
<thread>
creation date:
1732162867089

thread_type:
official_comment

<thread_object>
creator: area_chair_mwne

<content>
title:
Reminder: Please respond and update the score if necessary

comment:
Dear Reviewers,

Kindly ensure that you respond proactively to the authors' replies so we can foster a productive discussion. If necessary, please update your score accordingly. We greatly appreciate the time and effort you’ve dedicated to the review process, and your contributions are key to making this process run smoothly.

Thank you,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Sparse Matrix Tuning (SMT), a parameter-efficient fine-tuning method for large language models. By selectively updating the most significant submatrices based on gradient analysis, SMT reduces computational and memory costs while maintaining performance levels close to full fine-tuning. The method outperforms established PEFT approaches like LoRA and DoRA, effectively mitigating the performance plateau seen in other methods. SMT emphasizes the importance of fine-tuning attention layers, showing that the value matrix is particularly influential. Overall, SMT offers significant efficiency gains without sacrificing accuracy.

On the positive note, this paper advances the field of parameter-efficient fine-tuning (PEFT) for large language models (LLMs) through SMT. SMT uses a gradient-based method to identify and update the most impactful submatrices, enhancing computational and memory efficiency while maintaining performance. It overcomes limitations of previous methods, which suffer from performance saturation as tunable parameters increase. SMT provides new insights, particularly highlighting the importance of fine-tuning attention layers and the value (V) vector, which may inform future research. The method is supported by comprehensive empirical evaluations and extensive analyses, demonstrating its potential to improve downstream task performance effectively. Based on the aforementioned review, I would be leaning to accept the paper.