RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates
Authors: Md Kowsher, Tara Esmaeilbeig, Chun-Nam Yu, Chen Chen, Mojtaba Soltanalian, Niloofar Yousefi
Accepted at: acl25

RoCoFT: Efficient Finetuning of Large Language Models with Row-Column Updates

Md Kowsher1,2*, Tara Esmaeilbeig3, Chun-Nam Yu1, Chen Chen2, Mojtaba Soltanalian3, Niloofer Yousef2
1Nokia Bell Labs, USA 2University of Central Florida, USA 3University of Illinois Chicago, USA
https://github.com/Kowsher/RoCoFT

Abstract
We propose Row-Column Fine-Tuning (RoCoFT), a parameter-efficient finetuning method for large language models based on updating only a few rows and columns of the weight matrices in transformers. Through extensive experiments with medium size language models like RoBERTa and DeBERTa, and large language models (LLMs) liken Bloom-7B, Llama2-7B and Llama2-13B, we show that our method gives comparable accuracies to the state-of-the-art Parameter-Efficient Finetuning methods while also being more memory and computation-efficient. We also study the reason behind the effectiveness of our method with tools from Neural Tangent Kernel (NTK) theory. We empirically demonstrate that our kernel, constructed using a restricted set of row and column parameters, is numerically close to the full-parameter kernel and gives comparable classification performance. Ablation studies are conducted to investigate the impact of different algorithmic choices, including the robustness of RoCoFT to any selection of rows and columns, as well as the optimal rank for the effective implementation of our method.

1 Introduction
Adapting Large Language Models (LLMs) to different downstream applications is the current prevailing paradigm for solving many Natural Language Processing (NLP) problems, such as sentiment analysis, machine translation, question answering, named entity recognition, and text summarization. LLMs like GPT-4 (Achiam et al., 2023) and Llama (Touvron et al., 2023) are trained on massive amounts of text data and contain billions of parameters. They give state-of-the-art performance on many NLP, mathematical reasoning (Hendrycks et al., 2021; Cobbe et al., 2021), and programming benchmarks (Jiang et al., 2024). Early works on transfer learning with pretrained LLMs, such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019), use full finetuning, which updates all the parameters of the LLMs when adapting to downstream tasks. This approach becomes impractical as language models continue to scale up (Hoffmann et al., 2022), since a separate copy of the model parameters needs to be stored for each downstream application. Updating all the parameters is also prone to overfitting and the loss of LLM capabilities due to catastrophic forgetting (Kirkpatrick et al., 2017). Adaptor methods (Houlsby et al., 2019; Kowsher et al., 2024) solve this problem of finetuning LLMs by introducing extra modules called adaptors with a small set of independent parameters. Only the parameters in the adaptors need to be optimized during finetuning, and their small size makes it efficient to adapt an LLM to many different tasks. Parameter-Efficient Finetuning (PEFT) is the study of adapting LLMs to downstream applications by finetuning only a very small set of parameters. Many PEFT methods have been proposed, including the popular LoRA (Hu et al., 2021) and its variants (Zhang et al., 2023c; Edalati et al., 2022; Hyeon-Woo et al., 2021), prefix and prompt tuning (Li and Liang, 2021; Lester et al., 2021), and many other more advanced and complex adaptor methods (He et al., 2021; Zeng et al., 2023). These PEFT methods are effective in reducing the number of parameters required to adapt to downstream tasks, while maintaining performance close to full finetuning.

Despite the numerous PEFT methods available, we pose a critical question: can we design even simpler PEFT methods capable of adapting LLMs to diverse downstream tasks in a more efficient way? A simpler method could not only enhance computational and storage efficiency but also offer deeper insights into why PEFT methods succeed as simpler methods are easier to analyze. We answer this question by presenting a new method called Ro-
CoFT, where the LLMs can be efficiently adapted by updating only a small subset of rows or columns in the transformer block weight matrices.

We evaluate the effectiveness of our approach across several benchmarks on different language models. Our experimental results demonstrate that RoCoFT outperforms current PEFT techniques in accuracies, requires fewer trainable parameters, and has faster training times. We further analyze our method using Neural Tangent Kernel (NTK) theory (Jacot et al., 2018; Malladi et al., 2023), demonstrating that, for a pretrained LLM, the NTKs derived from a restricted set of rows and columns closely resemble those computed from the full parameter set. This substantiates the effectiveness of our proposed method and further suggests that most of the critical features for fine-tuning are already acquired during the pretraining phase.

Our contributions are summarized as follows: (i) We introduce a new PEFT method called RoCoFT which gives comparable accuracies to state-of-the-art PEFT methods, while being more efficient in terms of memory and time complexity. These claims are validated through extensive experiments on language models of different sizes and many benchmark datasets. (ii) We analyze our method with empirical neural tangent kernels and show that these kernels are close to NTKs defined on the full parameter set, and they give comparable accuracies on many tasks when trained with kernel logistic regression. This explains why our method has performance close to full finetuning from the view of kernel methods. (iii) We perform extensive experiments and ablation studies on the design choices such as which and how many rows and columns to select to facilitate the implementation of our method.

2 Related Works

PEFT Methods: Parameter-Efficient Finetuning (PEFT) methods aim to finetune only a small number of existing or extra parameters of the LLM to achieve results comparable to finetuning all the parameters. Recently, numerous PEFT approaches have been proposed to advance this strategy. LoRA (Hu et al., 2021) and related methods (Zhang et al., 2023c; Kopiczko et al., 2023; Dettmers et al., 2024) modify existing weight matrices of the model by introducing trainable low-rank decomposition matrices, as adapters, into each layer of the Transformer (Vaswani et al., 2017) architecture. IA^3 (Liu et al., 2022) is another adaptor method that only trains scaling vectors for the key, value, and feed-forward weight matrices in the attention mechanism for task adaptation. Prefix-Tuning (Li and Liang, 2021) and Prompt-Tuning (Lester et al., 2021; Kowsher et al., 2023) work by adding task-specific continuous vectors as contexts for inputs and only updates those parameters while keeping the original LLM parameters frozen. MAM adaptors (He et al., 2021) generalize from both LoRA and prefix-tuning under a unified framework. Our method is closer to LoRA and IA^3 in that we modify the weight matrices in the transformer architecture. However, unlike these approaches, we introduce no extra parameters and modify the existing parameters in place. BitFit (Zaken et al., 2021) and LayerNorm Tuning (Zhao et al., 2023) finetune only the bias parameters and layernorm parameters respectively and are extremely parameter-efficient. However, unlike LoRA and our method they cannot increase the capacity of the finetuning model by increasing the rank, since the number of bias and layernorm parameters are fixed in a model.

Apart from low-rank adaptor methods Sparse Fine-Tuning is another group of PEFT methods that focuses on directly training only a very small subset of model parameters during finetuning. Sparsity can be achieved in two different ways, either by pruning after full finetuning or by selecting a sparse set of masks to train before finetuning. Diff pruning (Guo et al., 2021) encourages sparsity by using \( l_0 \) norm regularization during finetuning, while Ansell et al. (2022) makes use of the Lottery Ticket Hypothesis (Frankle and Carbin, 2018) to prune the weights after full finetuning. Unlike our proposed method they both require computational costs close to full finetuning. He et al. (2024) selects submatrix blocks as masks using maximal gradient change during warmup as criterion, while Sung et al. (2021) selects masks based on Fisher information. Both require some precomputation before a sparse mask can be selected for finetuning. Our method can be seen as belonging to both low-rank adaptor methods and sparse fine-tuning, as with few rows or columns chosen the updates are naturally both low-rank and sparse.

Neural Tangent Kernels: Jacot et al. (2018) and related studies (Lee et al., 2019) show that the training dynamics of an infinite-width multi-layer neural network with suitable Gaussian initialization can be completely described by a fixed kernel
called the Neural Tangent Kernel (NTK). This result is further expanded in Yang (2020) to any architecture for which forward and backpropagation can be expressed via nonlinearities and matrix multiplications. Although these results are asymptotic, this interesting connection between deep neural networks and kernel methods allows many questions about neural networks to be studied via kernels. For example, Wei et al. (2022) studied the generalization error of representations learned by deep neural networks through kernel regression with NTKs. Recently Malladi et al. (2023) proposed to study the effect of finetuning LLMs through their NTKs, and provided theoretical support to their approach. In this paper, we continue along this line of work to use NTKs to analyze PEFT methods.

3 RoCoFT

PEFT is a collection of methods for transferring pretrained LLMs to downstream tasks by optimizing only a very small set of (additional) parameters. Since most modern LLMs are based on the transformer architecture (Vaswani et al., 2017), there is a line of work in PEFT focusing on modifying the transformer block by freezing most of its parameters and training only a few additional parameters. The method proposed in Houlsby et al. (2019) adds adaptive layers to the transformer blocks, and only parameters in those adaptive layers need to be trained for effective transfer learning. LoRA (Hu et al., 2021) removes the need for adding adaptive layers by directly modifying the weight matrices used in the transformer blocks. There are multiple linear weight matrices in the transformer block taking up most of the parameters, including \( \mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v \) for the query, key and value matrices in the attention mechanism, and also the weights \( \mathbf{W}_{ff} \) for the MLP projection layers. LoRA makes use of a low-rank modification of these weight matrices

\[
\mathbf{W} = \mathbf{W}_0 + \mathbf{B}\mathbf{A},
\]

where \( \mathbf{W}_0 \) is the pretrained weight matrix, \( \mathbf{B} \) and \( \mathbf{A} \) are low rank matrices of rank \( r \), and \( \mathbf{W} \) is the weight matrix after finetuning. In this formulation only \( \mathbf{B} \) and \( \mathbf{A} \) are updated. If \( \mathbf{W} \) is of dimensions \( d \times k \), \( \mathbf{B} \) and \( \mathbf{A} \) will be of dimensions \( d \times r \) and \( r \times k \) respectively. If \( r \ll d, k \), this can lead to significant savings in terms of memory and runtime. IA\( ^3 \) in (Liu et al., 2022) also modifies the weight matrices in the transformer block, but instead of a low-rank modification they rescale the key and value matrices, and also the MLP layers using three learned vectors \( \mathbf{l}_k, \mathbf{l}_v \) and \( \mathbf{l}_{ff} \) dedicated to key, value and feedforward layers.

The success of these PEFT methods leads us to ask if there are even simpler methods for modifying the transformer block for effective finetuning. Here, we propose modifying only a few rows or columns in the weight matrices of the transformer block, for query, key, value weight matrices \( \mathbf{W}_q, \mathbf{W}_k, \mathbf{W}_v \) and also the weight matrices in the feedforward layer \( \mathbf{W}_{ff} \). These can be expressed as

\[
\mathbf{W} = \mathbf{W}_0 + \mathbf{R} \quad \text{and} \quad \mathbf{W} = \mathbf{W}_0 + \mathbf{C},
\]

where \( \mathbf{R} \) and \( \mathbf{C} \) are restricted weight matrices such that only at most \( r \) of the rows or columns are non-zero. In practice we don’t need to form these extra parameters \( \mathbf{R} \) and \( \mathbf{C} \) and can directly update the parameters in place. Our method is the same as LoRA in its flexibility with increasing the capacity of the finetuning model by increasing the rank \( r \), but it is simpler since there is no multiplication of low-rank matrices and all the parameters can be updated in place. There is also no need to worry about the initializations of \( \mathbf{A} \) and \( \mathbf{B} \), as studied in (Hayou et al., 2024). We call our method RoCoFT for Row and Column-based Fine-Tuning. See Figure 1 for an illustrative diagram.

4 Experiments

We evaluate the effectiveness of the proposed RoCoFT method across various NLP tasks, including the General Language Understanding Evaluation (GLUE) benchmark, question answering, text summarization, common sense reasoning, and mathematical reasoning. We select rows or columns from the beginning of the order. This reduces the complexity of the selection process, as different selection strategies do not significantly affect performance. In other words, any row or column yields similar results, contributing to robustness. Further details are provided in Appendix 6.

Baselines: For our baseline comparisons, we utilize prominent PEFT methods such as Adapter (Houlsby et al., 2019), Prompt Tuning (Lester et al., 2021), Prefix-Tuning (Li and Liang, 2021), (IA)\( ^3 \) (Liu et al., 2022), Bitfit (Zaken et al., 2021), LoRA (Hu et al., 2021), AdaLoRA (Zhang et al., 2023b), MAM Adapter (He et al., 2021), PROPELT (Zeng et al., 2023), LoKr (Edalati et al., 2022), (Wu et al., 2024), SFT (Ansell et al., 2024), Diff Pruning (Guo et al., 2020), LoRAFA (Zhang et al., 2023a), Vera
(Kopiczko et al., 2023), LoRA-XS (Balazy et al., 2024) and LoHa (Hyeon-Woo et al., 2021). The experimental setup for the GLUE benchmark follows Xu et al. (2023), while question answering and text summarization tasks are conducted according to Zhang et al. (2023b).

Datasets and Model Selection: For the GLUE benchmark, we evaluate our RoCoFT method on a diverse set of tasks, including CoLA, SST-2, MRPC, STS-B, QQP, MNLI, QNLI, and RTE from Wang et al. (2018), using both RoBERTa Base and Large models (Liu et al., 2019). For question answering, we utilize the SQuAD v1.1 (Rajpurkar et al., 2016) and SQuAD v2.0 (Rajpurkar et al., 2018) datasets with DeBERTa Base v3 (He et al., 2020). Text summarization is evaluated using the XSum (Narayan et al., 2018) and CNN/DailyMail (Hermann et al., 2015) datasets with the BART Large model (Lewis et al., 2019).

For LLM performance using RoCoFT, we conduct an extensive evaluation across thirteen benchmark datasets, covering both common sense reasoning and mathematical reasoning tasks, utilizing four LLMs: Bloom 7B (Le Scao et al., 2023), GPT-J 6B (Wang, 2021), LLaMA2-7B and LLaMA2-13B from Touvron et al. (2023). For common sense reasoning, we employ a wide range of datasets, including BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2020), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019), WinoGrande (Sakaguchi et al., 2021), ARC-easy and ARC-challenge (Clark et al., 2018), and OBQA (Mihaylov et al., 2018), ensuring comprehensive coverage of the model’s ability to handle diverse aspects of common sense reasoning. For mathematical reasoning, we use several specialized datasets, including MultiArith (Roy and Roth, 2016), GSM8K (Cobbe et al., 2021), AddSub (Hosseini et al., 2014), SingleEq (Koncel-Kedziorski et al., 2015), and SVAMP (Patel et al., 2021), to assess the model’s performance on arithmetic reasoning tasks. Detailed hyperparameter settings are provided in Appendix 10. The implementation, environment setup, and hardware details of the experiments are given in Appendix F.

Performance Analysis: Table 1 presents the performance of RoCoFT compared with baselines on the GLUE benchmark tasks (Wang et al., 2018). RoCoFT_{r-(Row/Column)} finetunes the model according to Equation (2), where in R and C the first r rows(columns) are nonzero, respectively. RoCoFT achieves competitive or superior results while updating significantly fewer parameters. For instance, RoCoFT_{3-Row}, with only 0.249 million trainable parameters on RoBERTa-base (Liu et al., 2019) outperforms methods like LoRA (Hu et al., 2021) and MAM Adapter (He et al., 2021), which utilize more parameters. Moreover, RoCoFT variants consistently rank among the top performers across multiple tasks such as the MRPC, QNLI, and RTE, demonstrating robustness and versatility.

Table 2 showcases the performance of our proposed RoCoFT across various LLMs and tasks. Notably, these methods consistently achieve superior or competitive results compared to existing PEFT techniques. For the BLOOMZ_{7B} model (Muenninghoff et al., 2022), the RoCoFT_{3-Row} method attains the highest accuracy on Social IQA (SIQA, 73.56%), AI2 Reasoning Challenge (ARC-C, 57.48%), OpenBookQA (OBQA, 72.92%), MultiArith (M.Ar., 79.76%), Arithmetic Sequence (A.S., 70.95%), and Single-Math Problems (S.MP, 54.42%). The RoCoFT_{3-Column} variant also performs exceptionally well, achieving top scores on WinoGrande (W.Gra., 72.50%) and Grade School Math 8K (GSM8K, 71.05%). Similarly, with the GPT-J_{6B} model (Wang, 2021), our methods maintain strong performance. The RoCoFT_{3-Row} method achieves the best results on Boolean Questions (BoolQ, 65.92%), MultiArith (89.45%), and S.MP (56.79%), while the RoCoFT_{3-Column} method excels on SIQA (69.96%) and SingleEq (S.eEq, 82.61%).

![A simplified overview of various PEFT methods and RoCoFT. Snowflake icon indicates frozen parameters while fire icon indicates trainable parameters.](page_120_120_1208_246.png)

Figure 1: A simplified overview of various PEFT methods and RoCoFT. Snowflake icon indicates frozen parameters while fire icon indicates trainable parameters.
<table>
  <tr>
    <th>LM</th>
    <th>PEFT Method</th>
    <th># TTPs</th>
    <th>CoLA</th>
    <th>SST2</th>
    <th>MRPC</th>
    <th>STS-B</th>
    <th>QQP</th>
    <th>MNLI</th>
    <th>QNLI</th>
    <th>RTE</th>
    <th>Avg.</th>
  </tr>
  <tr>
    <td rowspan="13">RobertaBase</td>
    <td>FT</td>
    <td>124.6M</td>
    <td>59.84</td>
    <td>92.89</td>
    <td>85.24/88.18</td>
    <td>90.48/90.16</td>
    <td>90.18/87.02</td>
    <td>86.27</td>
    <td>91.17</td>
    <td>72.43</td>
    <td>83.56/88.43</td>
  </tr>
  <tr>
    <td>Adapter<sup>b</sup></td>
    <td>7.41M</td>
    <td>61.53</td>
    <td>94.11</td>
    <td><b>89.81/90.85</b></td>
    <td>90.25/90.09</td>
    <td>89.81/<b>86.90</b></td>
    <td>86.27</td>
    <td>92.06</td>
    <td>73.56</td>
    <td>84.67/90.33</td>
  </tr>
  <tr>
    <td>Prompt tuning</td>
    <td>0.61M</td>
    <td>49.37</td>
    <td>92.09</td>
    <td>70.83/81.72</td>
    <td>82.44/83.11</td>
    <td>82.99/78.35</td>
    <td>80.57</td>
    <td>80.03</td>
    <td>58.12</td>
    <td>74.55/81.06</td>
  </tr>
  <tr>
    <td>Prefix-tuning</td>
    <td>0.96M</td>
    <td>59.31</td>
    <td>93.81</td>
    <td>84.25/85.03</td>
    <td>88.48/88.32</td>
    <td>87.75/84.09</td>
    <td>85.21</td>
    <td>90.77</td>
    <td>54.51</td>
    <td>80.51/85.31</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>0.66M</td>
    <td>58.58</td>
    <td>93.92</td>
    <td>83.00/85.52</td>
    <td>90.30/90.32</td>
    <td>87.99/84.10</td>
    <td>83.95</td>
    <td>90.88</td>
    <td>71.12</td>
    <td>82.46/86.64</td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>0.083M</td>
    <td>61.32</td>
    <td>93.12</td>
    <td>87.22/88.41</td>
    <td>90.34/90.27</td>
    <td>88.12/84.11</td>
    <td>84.64</td>
    <td>91.09</td>
    <td>77.98</td>
    <td>84.22/87.59</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>0.89M</td>
    <td>60.09</td>
    <td>93.31</td>
    <td>86.50/88.68</td>
    <td>90.66/90.47</td>
    <td>88.83/85.21</td>
    <td>86.54</td>
    <td>92.02</td>
    <td>74.92</td>
    <td>84.32/87.28</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>1.03M</td>
    <td>59.82</td>
    <td>93.92</td>
    <td>86.49/88.03</td>
    <td>90.83/90.73</td>
    <td>88.58/84.98</td>
    <td>86.26</td>
    <td>91.43</td>
    <td>70.04</td>
    <td>84.06/87.21</td>
  </tr>
  <tr>
    <td>MAM Adapter</td>
    <td>1.78M</td>
    <td>58.34</td>
    <td>94.24</td>
    <td>87.31/88.21</td>
    <td>90.74/90.42</td>
    <td>88.31/83.20</td>
    <td>86.63</td>
    <td>90.19</td>
    <td>72.62</td>
    <td>83.54/87.27</td>
  </tr>
  <tr>
    <td>PROPELT Adapter</td>
    <td>1.87M</td>
    <td>64.24</td>
    <td>93.85</td>
    <td>87.15/87.82</td>
    <td>90.33/<b>90.64</b></td>
    <td>89.22/85.79</td>
    <td>86.49</td>
    <td>91.56</td>
    <td>75.54</td>
    <td>84.79/88.08</td>
  </tr>
  <tr>
    <td>PROPELT prefix</td>
    <td>10.49M</td>
    <td>60.11</td>
    <td>93.63</td>
    <td>86.73/87.98</td>
    <td>90.30/90.19</td>
    <td>88.54/85.05</td>
    <td>86.22</td>
    <td>91.51</td>
    <td>63.31</td>
    <td>82.54/87.74</td>
  </tr>
  <tr>
    <td>PROPELT_LoRA</td>
    <td>1.77M</td>
    <td>57.94</td>
    <td>94.11</td>
    <td>87.42/88.87</td>
    <td>90.66/90.35</td>
    <td>88.90/85.55</td>
    <td>86.83</td>
    <td>92.04</td>
    <td>67.39</td>
    <td>83.16/88.25</td>
  </tr>
  <tr>
    <td>MoSLoRA</td>
    <td>1.67M</td>
    <td>60.57</td>
    <td>93.95</td>
    <td>86.74/87.98</td>
    <td>90.05/89.43</td>
    <td>88.76/85.62</td>
    <td><b>87.84</b></td>
    <td>90.60</td>
    <td>75.10</td>
    <td>84.20/88.70</td>
  </tr>
  <tr>
    <td>LoRA-XS</td>
    <td>0.26M</td>
    <td>58.49</td>
    <td>93.19</td>
    <td>86.65/87.49</td>
    <td>89.60/89.33</td>
    <td>87.13/84.31</td>
    <td>85.34</td>
    <td>90.42</td>
    <td>76.24</td>
    <td>83.26/87.04</td>
  </tr>
  <tr>
    <td>VeRA</td>
    <td>0.043M</td>
    <td>60.35</td>
    <td>93.89</td>
    <td>86.01/87.88</td>
    <td>89.27/89.41</td>
    <td>87.88/85.65</td>
    <td>85.64</td>
    <td>90.22</td>
    <td>75.32</td>
    <td>83.57/87.65</td>
  </tr>
  <tr>
    <td>LoRAFA</td>
    <td>0.44M</td>
    <td>60.49</td>
    <td>93.65</td>
    <td>88.18/89.98</td>
    <td>90.70/90.66</td>
    <td>88.90/85.46</td>
    <td>86.11</td>
    <td>91.42</td>
    <td>76.11</td>
    <td>84.45/88.70</td>
  </tr>
  <tr>
    <td>SFT</td>
    <td>0.90M</td>
    <td>64.45</td>
    <td>94.28</td>
    <td>87.74/88.64</td>
    <td>89.37/89.12</td>
    <td>87.24/85.11</td>
    <td>86.64</td>
    <td>92.11</td>
    <td>78.42</td>
    <td>85.03/87.62</td>
  </tr>
  <tr>
    <td>Diff Pruning</td>
    <td>1.24M</td>
    <td>62.45</td>
    <td>93.77</td>
    <td>88.00/89.21</td>
    <td>89.72/90.02</td>
    <td>88.62/85.54</td>
    <td>85.32</td>
    <td>92.14</td>
    <td>77.90</td>
    <td>84.74/88.26</td>
  </tr>
  <tr>
    <td rowspan="4">RoCoFT<sub>1-row</sub></td>
    <td>0.083M</td>
    <td>60.18</td>
    <td>94.06</td>
    <td>87.74/88.48</td>
    <td>90.70/90.47</td>
    <td>88.49/85.35</td>
    <td>85.23</td>
    <td>90.70</td>
    <td>76.61</td>
    <td>84.27/89.97</td>
  </tr>
  <tr>
    <td>0.249M</td>
    <td>63.53</td>
    <td><b>94.92</b></td>
    <td>89.71/90.74</td>
    <td><b>90.89/90.49</b></td>
    <td><b>89.97/86.80</b></td>
    <td>86.73</td>
    <td><b>92.12</b></td>
    <td><b>78.31</b></td>
    <td><b>85.65/90.61</b></td>
  </tr>
  <tr>
    <td>0.083M</td>
    <td>60.32</td>
    <td>93.88</td>
    <td>88.38/89.78</td>
    <td>90.23/90.14</td>
    <td>88.46/85.84</td>
    <td>85.35</td>
    <td>90.58</td>
    <td>76.74</td>
    <td>84.11/89.96</td>
  </tr>
  <tr>
    <td>0.249M</td>
    <td>62.95</td>
    <td>94.69</td>
    <td>89.18/90.94</td>
    <td>90.85/90.45</td>
    <td>89.86/86.38</td>
    <td>86.76</td>
    <td>91.89</td>
    <td><b>79.21</b></td>
    <td><b>85.55/90.69</b></td>
  </tr>
</table>

Table 1: Performance on GLUE tasks. Metrics used: MCC for CoLA, accuracy for SST-2, accuracy/F1 score for MRPC and QQP, Pearson/Spearman correlations for STS-B, and accuracy for MNLI, QNLI, and RTE. #TTPs denotes Total Trainable Parameters.

When scaled to larger models like LLaMA2<sub>7B</sub> and LLaMA2<sub>13B</sub> (Touvron et al., 2023), our methods continue to demonstrate their effectiveness. On LLaMA2<sub>7B</sub>, the RoCoFT<sub>3-row</sub> method secures the highest accuracy on BoolQ (69.36%), SIQA (78.09%), OBQA (76.96%), M.A.R. (90.55%), and GSM8K (77.37%). The RoCoFT<sub>3-column</sub> variant achieves top performance on HellaSwag (H.Sw., 89.46%) and S.eEq (82.48%). For LLaMA2<sub>13B</sub>, both RoCoFT<sub>3-row</sub> and RoCoFT<sub>3-column</sub> methods attain leading results on multiple tasks, with the RoCoFT<sub>3-row</sub> method achieving the highest accuracy on SIQA (79.54%), ARC-Easy (ARCe, 83.65%), A.S. (88.24%), and S.M.P. (66.60%).

These results highlight RoCoFT’s state-of-the-art performance and parameter efficiency, making it ideal for resource-constrained deployment. Additional results on question answering (SQuAD v1.1 & 2.0) and text summarization (Xsum & CNN/DailyMail) are discussed in Appendix A.

Efficiency Comparison: Our proposed method, RoCoFT, demonstrates significant parameter efficiency compared to existing PEFT techniques. Specifically, RoCoFT variants require substantially fewer trainable parameters while achieving competitive or superior performance.

For instance, as shown in Table 1, RoCoFT<sub>Row</sub> uses only 0.083 million trainable parameters for rank one and 0.249 million for rank three on the GLUE benchmark (Wang et al., 2018), outperforming methods like LoRA (Hu et al., 2021) and MAM Adapter (He et al., 2021), which use 0.89 million and 1.78 million parameters, respectively. Similarly, in question answering and summarization tasks (Table 7), our Row and Column methods utilize just 0.161 million trainable parameters, significantly less than LoRA and AdaLoRA (Zhang et al., 2023c), yet achieve higher or comparable performance. In terms of computational efficiency (Table 3), our method exhibits lower space and time complexity. Specifically, RoCoFT has a time/space complexity of \( O(d \times r) \), compared to LoRA’s \( O(2d \times r) \) and Prefix-Tuning’s \( O(L \times d \times l_p) \), where \( r \) is the rank, \( d \) is the model dimension, \( L \) is the number of layers, and \( l_p \) is the length of the prefix. Moreover, our method does not introduce any additional parameters into the model architecture, which also reduces the total number of parameters and requires less GPU memory and training time, as illustrated in Figure 2 (Left). RoCoFT variants have lower memory occupancy during training (approximately 2.85GB) compared to other methods like LoRA and AdaLoRA, and consistently require less training time across various datasets, as shown in Figure 2 (Right).

These results underscore the efficiency of our approach in terms of both parameter count and computational resources, highlighting its suitability for deployment in resource-constrained environments. In Appendix we present ablation studies on (i) the
<table>
  <tr>
    <th>LLM</th>
    <th>Method</th>
    <th># TTPs</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HSw.</th>
    <th>W.Gra.</th>
    <th>ARCE</th>
    <th>ARCC</th>
    <th>OBQA</th>
    <th>M.Ar.</th>
    <th>G.Sk.</th>
    <th>A.S.</th>
    <th>SeEq</th>
    <th>S.MF</th>
  </tr>
  <tr>
    <td rowspan="6">BLoOMZ<sub>15B</sub></td>
    <td>Prefix</td>
    <td>33.53M</td>
    <td>58.53</td>
    <td>62.24</td>
    <td>65.41</td>
    <td>48.32</td>
    <td>66.63</td>
    <td>68.13</td>
    <td>49.32</td>
    <td>63.51</td>
    <td>78.41</td>
    <td>66.45</td>
    <td>67.52</td>
    <td>66.94</td>
    <td>49.10</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>24.88M</td>
    <td>64.94</td>
    <td><b>74.68</b></td>
    <td>72.49</td>
    <td>55.89</td>
    <td>68.30</td>
    <td>73.21</td>
    <td>56.59</td>
    <td>72.85</td>
    <td>79.43</td>
    <td>70.25</td>
    <td>68.93</td>
    <td><b>70.93</b></td>
    <td>53.89</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>19.34M</td>
    <td>63.30</td>
    <td>73.33</td>
    <td>71.01</td>
    <td>52.50</td>
    <td>71.60</td>
    <td>69.45</td>
    <td>54.14</td>
    <td>68.60</td>
    <td>78.90</td>
    <td><b>71.17</b></td>
    <td>70.33</td>
    <td>70.84</td>
    <td>53.95</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>24.22M</td>
    <td>65.89</td>
    <td>73.92</td>
    <td>73.33</td>
    <td><b>56.65</b></td>
    <td>71.39</td>
    <td><b>73.46</b></td>
    <td>57.15</td>
    <td>72.31</td>
    <td>79.50</td>
    <td>70.93</td>
    <td>70.90</td>
    <td>70.59</td>
    <td>53.85</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>+Row</td>
    <td>13.37M</td>
    <td>66.33</td>
    <td>74.53</td>
    <td>73.56</td>
    <td>56.60</td>
    <td>72.14</td>
    <td>73.29</td>
    <td><b>57.48</b></td>
    <td><b>72.92</b></td>
    <td><b>79.76</b></td>
    <td>70.94</td>
    <td><b>70.95</b></td>
    <td>70.90</td>
    <td><b>54.42</b></td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>,Column</td>
    <td>13.37M</td>
    <td><b>66.34</b></td>
    <td>74.64</td>
    <td>73.12</td>
    <td>55.93</td>
    <td><b>72.50</b></td>
    <td>73.11</td>
    <td><b>57.19</b></td>
    <td><b>72.90</b></td>
    <td>79.72</td>
    <td>71.05</td>
    <td>70.88</td>
    <td>70.76</td>
    <td>54.38</td>
  </tr>
  <tr>
    <td rowspan="6">GPT-J<sub>6B</sub></td>
    <td>Prefix</td>
    <td>27.83M</td>
    <td>62.28</td>
    <td>65.04</td>
    <td>67.72</td>
    <td>44.15</td>
    <td>63.71</td>
    <td>63.59</td>
    <td>46.47</td>
    <td>58.31</td>
    <td>83.12</td>
    <td>67.44</td>
    <td>75.25</td>
    <td>78.46</td>
    <td>49.12</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>20.77M</td>
    <td>65.19</td>
    <td>67.58</td>
    <td><b>71.22</b></td>
    <td>45.16</td>
    <td>66.03</td>
    <td>64.10</td>
    <td><b>47.75</b></td>
    <td>63.92</td>
    <td>88.51</td>
    <td>72.45</td>
    <td>80.21</td>
    <td><b>82.03</b></td>
    <td>56.14</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>16.61M</td>
    <td>63.17</td>
    <td>68.51</td>
    <td>68.97</td>
    <td>45.79</td>
    <td>66.06</td>
    <td>62.42</td>
    <td>45.32</td>
    <td><b>65.42</b></td>
    <td>89.51</td>
    <td>72.04</td>
    <td>80.50</td>
    <td>81.50</td>
    <td>55.43</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>20.02M</td>
    <td>65.50</td>
    <td>67.63</td>
    <td>69.46</td>
    <td>45.60</td>
    <td>66.80</td>
    <td>63.56</td>
    <td>46.81</td>
    <td>63.82</td>
    <td>88.30</td>
    <td><b>72.82</b></td>
    <td><b>80.60</b></td>
    <td>81.24</td>
    <td>56.73</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>+Row</td>
    <td>11.62M</td>
    <td><b>65.92</b></td>
    <td><b>68.53</b></td>
    <td>69.90</td>
    <td><b>45.97</b></td>
    <td><b>66.87</b></td>
    <td><b>64.91</b></td>
    <td>45.12</td>
    <td>65.07</td>
    <td>89.45</td>
    <td>72.80</td>
    <td>80.45</td>
    <td>82.12</td>
    <td><b>56.79</b></td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>,Column</td>
    <td>11.62M</td>
    <td>65.12</td>
    <td>68.22</td>
    <td>69.96</td>
    <td><b>45.98</b></td>
    <td>66.78</td>
    <td>64.89</td>
    <td>45.70</td>
    <td>64.81</td>
    <td><b>89.74</b></td>
    <td>72.24</td>
    <td>80.23</td>
    <td><b>82.61</b></td>
    <td>56.70</td>
  </tr>
  <tr>
    <td rowspan="6">LLaMA-2<sub>7B</sub></td>
    <td>Prefix</td>
    <td>33.53M</td>
    <td>67.33</td>
    <td>79.46</td>
    <td>75.80</td>
    <td>76.04</td>
    <td>72.11</td>
    <td>71.67</td>
    <td>57.33</td>
    <td>69.98</td>
    <td>84.18</td>
    <td>68.47</td>
    <td>81.04</td>
    <td>80.00</td>
    <td>52.17</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>24.90M</td>
    <td>67.03</td>
    <td>78.69</td>
    <td>76.06</td>
    <td>88.85</td>
    <td>76.47</td>
    <td>76.50</td>
    <td>60.36</td>
    <td>74.22</td>
    <td>89.81</td>
    <td>77.07</td>
    <td><b>86.70</b></td>
    <td>83.01</td>
    <td>60.25</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>19.42M</td>
    <td>65.02</td>
    <td>78.10</td>
    <td>78.00</td>
    <td>87.57</td>
    <td>76.78</td>
    <td>75.48</td>
    <td>60.54</td>
    <td>74.02</td>
    <td>90.20</td>
    <td>76.13</td>
    <td>86.55</td>
    <td><b>83.70</b></td>
    <td>59.16</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>24.30M</td>
    <td>67.09</td>
    <td>79.37</td>
    <td>76.15</td>
    <td>88.86</td>
    <td><b>77.54</b></td>
    <td><b>76.54</b></td>
    <td>60.55</td>
    <td>74.63</td>
    <td>90.13</td>
    <td>75.68</td>
    <td>84.67</td>
    <td>82.14</td>
    <td>59.94</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>+Row</td>
    <td>13.47M</td>
    <td><b>69.36</b></td>
    <td><b>80.01</b></td>
    <td>78.09</td>
    <td>89.28</td>
    <td>76.73</td>
    <td>76.46</td>
    <td>60.55</td>
    <td><b>76.96</b></td>
    <td><b>90.55</b></td>
    <td><b>77.37</b></td>
    <td>86.12</td>
    <td>82.66</td>
    <td><b>60.75</b></td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>,Column</td>
    <td>13.47M</td>
    <td>69.32</td>
    <td><b>80.08</b></td>
    <td>77.99</td>
    <td><b>89.46</b></td>
    <td>76.41</td>
    <td>76.46</td>
    <td><b>60.59</b></td>
    <td>76.90</td>
    <td>90.42</td>
    <td>77.35</td>
    <td>86.16</td>
    <td>82.48</td>
    <td>60.35</td>
  </tr>
  <tr>
    <td rowspan="6">LLaMA-2<sub>13B</sub></td>
    <td>Prefix</td>
    <td>61.97M</td>
    <td>68.38</td>
    <td>80.99</td>
    <td>78.00</td>
    <td>80.00</td>
    <td>76.35</td>
    <td>77.62</td>
    <td>61.32</td>
    <td>72.94</td>
    <td>87.22</td>
    <td>71.09</td>
    <td>84.09</td>
    <td>81.28</td>
    <td>58.25</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>45.04M</td>
    <td><b>71.71</b></td>
    <td>82.55</td>
    <td>78.88</td>
    <td>91.60</td>
    <td>83.01</td>
    <td>83.04</td>
    <td><b>67.33</b></td>
    <td><b>81.76</b></td>
    <td>90.55</td>
    <td><b>80.19</b></td>
    <td>87.00</td>
    <td>87.10</td>
    <td>66.03</td>
  </tr>
  <tr>
    <td>(IA)<sup>3</sup></td>
    <td>36.02M</td>
    <td>71.39</td>
    <td>83.33</td>
    <td>78.32</td>
    <td><b>92.40</b></td>
    <td><b>83.24</b></td>
    <td>83.34</td>
    <td>66.43</td>
    <td>80.99</td>
    <td><b>91.88</b></td>
    <td>79.24</td>
    <td>88.16</td>
    <td>87.08</td>
    <td>65.63</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>44.94M</td>
    <td>71.19</td>
    <td><b>85.99</b></td>
    <td>79.15</td>
    <td>91.86</td>
    <td><b>83.24</b></td>
    <td>83.35</td>
    <td>67.05</td>
    <td>81.37</td>
    <td>91.27</td>
    <td>78.90</td>
    <td>86.89</td>
    <td>86.07</td>
    <td>65.85</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>+Row</td>
    <td>24.88M</td>
    <td>71.46</td>
    <td>83.32</td>
    <td><b>79.54</b></td>
    <td>91.86</td>
    <td>83.22</td>
    <td><b>83.65</b></td>
    <td>67.12</td>
    <td>81.54</td>
    <td>90.69</td>
    <td>79.70</td>
    <td><b>88.24</b></td>
    <td>87.28</td>
    <td>66.60</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>T</sub>,Column</td>
    <td>24.88M</td>
    <td>71.44</td>
    <td>83.52</td>
    <td>79.50</td>
    <td>91.84</td>
    <td>83.20</td>
    <td>83.39</td>
    <td>67.06</td>
    <td>81.73</td>
    <td>91.46</td>
    <td>79.63</td>
    <td>88.11</td>
    <td><b>87.58</b></td>
    <td><b>66.63</b></td>
  </tr>
</table>

Table 2: Accuracy comparison of commonsense and mathematical reasoning performance across different PEFT methods using LLMs.

![Bar chart showing memory cost and optimization memory for different methods](page_1012_1042_496_246.png)
![Bar chart showing training time per epoch for different datasets and methods](page_1512_1042_496_246.png)

Figure 2: The left figure shows the memory cost, where blue bars represent the original model weights and green bars represent optimization memory. The right figure shows the training time (in minutes) per epoch for each method.

robustness of row-column selection (6), (ii) the optimal rank \( r \) for RoCoFT (B.1), and (iii) RoCoFT with random weight selection (B.2).

5 Finetuning through the Lens of Neural Tangent Kernel Regression

In this section, we analyze the effectiveness of the RoCoFT method from the perspective of kernel methods. We examine how closely this fine-tuning approach resembles full fine-tuning by comparing their respective kernels. Kernel methods are classic machine learning algorithms that make use of kernel functions for learning nonlinear mapping of inputs, with SVMs (Cortes and Vapnik, 1995) and Gaussian Processes (Williams and Rasmussen, 2006) being the prime examples. A kernel function \( K : \mathcal{X} \times \mathcal{X} \to \mathbb{R} \) is a similarity function on the input space \( \mathcal{X} \) that satisfies certain symmetry and positive semi-definiteness conditions. Kernel methods differ from deep learning with neural networks in that the kernels (and hence the feature representations) are fixed during learning, while deep neural networks continuously update their feature representations during backpropagation. Jacot et al. (2018) made the important discovery that under certain conditions, in the infinite width limit, the training of deep neural networks can be described by a fixed kernel called the Neural Tangent Kernel (NTK). For a neural network function \( f_\theta : \mathcal{X} \to \mathbb{R} \) parameterized by \( \theta \), its NTK is defined by
Figure 3: Neural Tangent Kernels for SST-2 on 16-shot training data (plots on more tasks available in Appendix G)

<table>
  <tr>
    <th>Methods</th>
    <th>Space</th>
    <th>Time</th>
    <th>TTPs</th>
    <th>APs</th>
  </tr>
  <tr>
    <td>FT</td>
    <td>\( O(d \times d) \)</td>
    <td>\( O(d \times d) \)</td>
    <td>\( d^2 \)</td>
    <td>0</td>
  </tr>
  <tr>
    <td>(IA)^3</td>
    <td>\( O(l_k + l_v + l_{ff}) \)</td>
    <td>\( O(d_k + d_v + d_{ff}) \)</td>
    <td>\( 3d \)</td>
    <td>3d</td>
  </tr>
  <tr>
    <td>Prompt</td>
    <td>\( O(d \times l_p) \)</td>
    <td>\( O(d \times l_p) \)</td>
    <td>\( l_p^2 \)</td>
    <td>\( l_p^2 \)</td>
  </tr>
  <tr>
    <td>Prefix</td>
    <td>\( O(L \times d \times l_p) \)</td>
    <td>\( O(L \times d \times l_p) \)</td>
    <td>\( L \cdot l_p d \)</td>
    <td>\( L \cdot l_p d \)</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>\( O((d + d') \times r) \)</td>
    <td>\( O((d + d') \times r) \)</td>
    <td>\( 2dr \)</td>
    <td>\( 2dr \)</td>
  </tr>
  <tr>
    <td>LoRA-FA</td>
    <td>\( O((d + d') \times r) \)</td>
    <td>\( O((d + d') \times r) \)</td>
    <td>\( dr \)</td>
    <td>\( 2dr \)</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>\( O((d + d + r) \times r) \)</td>
    <td>\( O((d + d + r) \times r) \)</td>
    <td>\( 2dr + r^2 \)</td>
    <td>\( 2dr + r^2 \)</td>
  </tr>
  <tr>
    <td>LoHA</td>
    <td>\( O(2r \times (d + d)) \)</td>
    <td>\( O(2r \times (d + d)) \)</td>
    <td>\( 4dr \)</td>
    <td>\( 4dr \)</td>
  </tr>
  <tr>
    <td>BitFit</td>
    <td>\( O(d) \)</td>
    <td>\( O(d) \)</td>
    <td>\( d \)</td>
    <td>0</td>
  </tr>
  <tr>
    <td><i>RoCoFT<sub>Row</sub></i></td>
    <td>\( O(d \times r) \)</td>
    <td>\( O(d \times r) \)</td>
    <td>\( rd \)</td>
    <td>0</td>
  </tr>
  <tr>
    <td><i>RoCoFT<sub>Column</sub></i></td>
    <td>\( O(d \times r) \)</td>
    <td>\( O(d \times r) \)</td>
    <td>\( rd \)</td>
    <td>0</td>
  </tr>
</table>

Table 3: Space/Time Complexity; Total Trainable Parameters (TTPs) and Additional Parameters in the model (APs) for RoCoFT method and baseline methods for a single layer \( W \in \mathbb{R}^{d \times d} \). We define \( l_k, l_v, \) and \( l_{ff} \) as the dimensions of three learned vectors in IA^3; and \( l_p \) as the length of the prompt added to the input/layers in prompt tuning and prefix-tuning. For LoRA-type methods, we use \( r \) to represent the rank dimension.

\[
\mathbf{K}_\theta(\mathbf{x}, \mathbf{x}') = \langle \nabla f_\theta(\mathbf{x}), \nabla f_\theta(\mathbf{x}') \rangle = \sum_{\theta_i \in \theta} \frac{\partial f_\theta(\mathbf{x})}{\partial \theta_i} \frac{\partial f_\theta(\mathbf{x}')}{\partial \theta_i}
\]

where \( \nabla f_\theta(\mathbf{x}) \) is the Jacobian/gradient, and \( \theta_i \) are the individual parameters in \( \theta \). This asymptotic result defines the 'lazy'/linear learning regime, since the features are linear and fixed (defined by gradients \( \nabla f_\theta(\mathbf{x}) \)). Although this approximation is asymptotic, empirically researchers have found that the results of neural network training and kernel regression using NTKs can be close for many tasks in finite-width networks (Wei et al., 2022). Malлади et al. (2023) extends the NTK theory to model the finetuning of LLMs. As an alternative to LLM finetuning by SGD, given training data \( (\mathbf{x}_i, \mathbf{y}_i)_{i=1}^n \) for a classification task, we can instead solve the following kernel logistic regression problem

\[
\min_{f \in \mathcal{H}} \sum_{i=1}^n \mathcal{L}(f(\mathbf{x}_i), \mathbf{y}_i) + \frac{\lambda}{2} \| f \|_{\mathcal{H}}^2,
\]

where \( \mathcal{H} \) is the Reproducing Kernel Hilbert Space defined by the NTK \( \mathbf{K}_\theta \), and \( \mathcal{L}(\cdot, \cdot) \) is the logistic loss. For a two-class problem with \( \mathbf{y}_i \in \{0, 1\} \), this is equivalent to

\[
\min_{\alpha} \frac{\lambda}{2} \sum_{i,j=1}^n \alpha_i \alpha_j \mathbf{K}_\theta(\mathbf{x}_i, \mathbf{x}_j) - \sum_{i,j=1}^n y_i y_j \alpha_i \mathbf{K}_\theta(\mathbf{x}_i, \mathbf{x}_j) \\
+ \sum_{i=1}^n \log(1 + \exp(\sum_{j=1}^n \alpha_j \mathbf{K}_\theta(\mathbf{x}_i, \mathbf{x}_j))).
\]

This problem is convex in \( \alpha \) and therefore has no local minima. It is also clear that the solution is completely determined by the value of the NTK \( \mathbf{K}_\theta \) between all training samples \( \mathbf{x}_i \). Notice that \( \theta \) is fixed here (usually set to pretrained model weights), so the kernel \( \mathbf{K}_\theta \) is also fixed.

Below we want to compare the NTKs defined by the full parameter set \( \theta \) and the NTKs defined by some much smaller set of parameters \( \theta_S \) (can be subset of \( \theta \) or new adaptor variables) and show that they are close. If kernel regression using the full
<table>
  <tr>
    <th>16-shot (single)</th>
    <th>SST-2</th>
    <th>SST-5</th>
    <th>MR</th>
    <th>CR</th>
    <th>MPQA</th>
    <th>Subj</th>
    <th>TREC</th>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_R}, p=1 \)</td>
    <td>0.093(0.008)</td>
    <td>0.083(0.005)</td>
    <td>0.064(0.006)</td>
    <td>0.087(0.007)</td>
    <td>0.123(0.012)</td>
    <td>0.061(0.005)</td>
    <td>0.181(0.007)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_R}, p=2 \)</td>
    <td>0.130(0.014)</td>
    <td>0.113(0.012)</td>
    <td>0.092(0.011)</td>
    <td>0.126(0.021)</td>
    <td>0.182(0.017)</td>
    <td>0.073(0.008)</td>
    <td>0.197(0.008)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_C}, p=1 \)</td>
    <td>0.091(0.008)</td>
    <td>0.077(0.004)</td>
    <td>0.061(0.006)</td>
    <td>0.084(0.006)</td>
    <td>0.123(0.014)</td>
    <td>0.055(0.005)</td>
    <td>0.166(0.007)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_C}, p=2 \)</td>
    <td>0.127(0.016)</td>
    <td>0.108(0.012)</td>
    <td>0.089(0.011)</td>
    <td>0.122(0.018)</td>
    <td>0.184(0.018)</td>
    <td>0.069(0.009)</td>
    <td>0.185(0.008)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{LoRA}, p=1 \)</td>
    <td>0.090(0.022)</td>
    <td>0.092(0.019)</td>
    <td>0.086(0.021)</td>
    <td>0.108(0.024)</td>
    <td>0.081(0.030)</td>
    <td>0.069(0.023)</td>
    <td>0.140(0.071)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{LoRA}, p=2 \)</td>
    <td>0.107(0.028)</td>
    <td>0.110(0.024)</td>
    <td>0.102(0.027)</td>
    <td>0.132(0.036)</td>
    <td>0.098(0.034)</td>
    <td>0.087(0.028)</td>
    <td>0.151(0.078)</td>
  </tr>
  <tr>
    <th>16-shot (pair)</th>
    <th>MNLI</th>
    <th>SNLI</th>
    <th>QNLI</th>
    <th>RTE</th>
    <th>MRPC</th>
    <th>QQP</th>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_R}, p=1 \)</td>
    <td>0.177(0.011)</td>
    <td>0.198(0.039)</td>
    <td>0.076(0.028)</td>
    <td>0.140(0.019)</td>
    <td>0.073(0.009)</td>
    <td>0.046(0.008)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_R}, p=2 \)</td>
    <td>0.260(0.043)</td>
    <td>0.255(0.069)</td>
    <td>0.149(0.071)</td>
    <td>0.203(0.039)</td>
    <td>0.096(0.016)</td>
    <td>0.063(0.013)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_C}, p=1 \)</td>
    <td>0.176(0.013)</td>
    <td>0.194(0.040)</td>
    <td>0.073(0.028)</td>
    <td>0.142(0.023)</td>
    <td>0.073(0.010)</td>
    <td>0.044(0.006)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{\theta_C}, p=2 \)</td>
    <td>0.262(0.050)</td>
    <td>0.253(0.072)</td>
    <td>0.146(0.071)</td>
    <td>0.212(0.047)</td>
    <td>0.096(0.016)</td>
    <td>0.061(0.011)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{LoRA}, p=1 \)</td>
    <td>0.139(0.031)</td>
    <td>0.120(0.032)</td>
    <td>0.077(0.018)</td>
    <td>0.119(0.027)</td>
    <td>0.155(0.060)</td>
    <td>0.084(0.021)</td>
  </tr>
  <tr>
    <td>\( \mathbf{K}_{LoRA}, p=2 \)</td>
    <td>0.163(0.038)</td>
    <td>0.137(0.044)</td>
    <td>0.103(0.025)</td>
    <td>0.150(0.039)</td>
    <td>0.213(0.115)</td>
    <td>0.106(0.027)</td>
  </tr>
</table>

Table 5: Relative difference in kernels (compared to full parameter \( \mathbf{K}_\theta \)) on single-sentence and sentence-pair tasks.

NTK \( \mathbf{K}_\theta \) is close to the performance of full finetuning (i.e., the linear/lazy approximation is good, Condition 1), and if the NTK of a PEFT method using a smaller set of variables \( \theta_S \) is close to the full parameter NTK \( \mathbf{K}_\theta \) (Condition 2), then directly finetuning on those variables \( \theta_S \) is highly likely to achieve performance close to full finetuning. In the following we want to show Condition 1 is true for many (but not all) finetuning tasks, while Condition 2 is true for almost all the tasks we tested. Condition 1 also implies that good features sufficient for the downstream task are already contained in the pretrained model \( \theta \).

We compare the kernels of the 1-row and 1-column version of our RoCoFT method, and we denote the associated trainable parameters as \( \theta_{R}, \theta_{C} \subseteq \theta \). The corresponding kernels are defined as

\[
\mathbf{K}_{\theta_R}(x, x') = \sum_{\theta_i \in \theta_R} \frac{\partial f_\theta(x)}{\partial \theta_i} \frac{\partial f_\theta(x')}{\partial \theta_i}
\]
\[
\mathbf{K}_{\theta_C}(x, x') = \sum_{\theta_i \in \theta_C} \frac{\partial f_\theta(x)}{\partial \theta_i} \frac{\partial f_\theta(x')}{\partial \theta_i}.
\]

Notice *a priori* there is no reason why these kernels might be close to the full parameter kernel in Equation 5, since the gradient sums are over a much smaller non-random subset \( \theta_R, \theta_C \subseteq \theta \). We first compare few-shot learning performance of these kernels using kernel logistic regression with prompt-based finetuning, as done in Malladi et al. (2023). The kernels are computed with the pretrained RoBERTa-base model. From Table 4 we can see the performance of kernel logistic regression using \( \mathbf{K}_{\theta_R} \) and \( \mathbf{K}_{\theta_C} \) are surprisingly close to using the kernel for full parameters \( \mathbf{K}_\theta \), usually within the standard error of 5 runs using different random seeds. The performance of kernel logistic regression using \( \mathbf{K}_\theta \) is in turn close to full finetuning except for a few tasks including MNLI, SNLI, QNLI and MPQA, which are related to the prompt templates used (Gao et al., 2020). We also include the 16-shot NTK results for LoRA for comparison. Next we directly compare the kernel matrices \( \mathbf{K}_\theta, \mathbf{K}_{\theta_R} \) and \( \mathbf{K}_{\theta_C} \) for these few-shot learning problems directly. Figure 3 shows the empirical Neural Tangent Kernel values for the the task SST-2. More figures for the other tasks are available in Appendix G. As observed in Figure 3, the NTK for LoRA is not as close to full parameter kernel as the row and column parameter kernels. The SST-2 task is a two-class problem and hence their kernel matrices have 2x2 block structure. We can see that except for the magnitude of the entries in the kernel matrices, the patterns in the kernel matrices for the full parameter set \( \mathbf{K}_\theta \), 1-row set \( \mathbf{K}_{\theta_R} \) and 1-column set \( \mathbf{K}_{\theta_C} \) are extremely similar. More quantitatively, Table 5 shows the relative difference between the 1-row kernel \( \mathbf{K}_{\theta_R} \) and 1-column kernel \( \mathbf{K}_{\theta_C} \) with the full parameter kernel \( \mathbf{K}_\theta \) after normalization in \( \ell_1 \) and \( \ell_2 \) norms by flattening the kernel matrices. For example, the relative difference for \( \mathbf{K}_{\theta_R} \) is computed as
\[
\| (\mathbf{K}_{\theta_R}/\|\mathbf{K}_{\theta_R}\|_p) - (\mathbf{K}_\theta/\|\mathbf{K}_\theta\|_p) \|_p, \quad p = 1, 2.
\]
We can see that except for few tasks like MNLI, SNLI and TREC, the relative differences between kernels are between 5 to 15%, which are fairly small. These results across many tasks from NTK provide strong support for our proposal that finetuning only a few rows or columns can give performance comparable to full finetuning.

6 Ablation Study

Robustness of Row-Column Selection In this study, we demonstrate the robustness of our row
Figure 4: Accuracy comparison of Max, Min, Mixed, and random row and column selection methods across different datasets. The results show that the proposed selection techniques are robust across various tasks.

and column selection method through a detailed comparison of four selection strategies: Max, Min, Mixed, and random. These strategies are applied to both rows and columns of the weight matrices. For the Min, Max, and Mixed selection strategies, we employ a scoring criterion used in the Wanda method (Sun et al., 2023), a simple yet effective pruning technique that requires only the forward pass. Pruning a neural network involves scoring the weights by importance (e.g., by the absolute values of weights), and then remove the least important ones. We can adopt these strategies to rank rows and columns by importance and evaluate the effect of finetuning on them. Given a weight matrix \( \mathbf{W} \in \mathbb{R}^{d_{out} \times d_{in}} \) and input feature activations \( \mathbf{X} \in \mathbb{R}^{s \times d_{in}} \) from a length s sequence, Wanda calculates the importance score \( S_{ij} \) of the weight \( W_{ij} \) as

\[
S_{ij} = |W_{ij}| \cdot \|X_{\cdot j}\|_2,
\]

where \( \|X_{\cdot j}\|_2 \) is the 2-norm across the jth feature aggregated across all examples in batch. To determine the most important rows, we sum \( S_{ij} \) across the columns, yielding a row score vector \( S_{row} \in \mathbb{R}^{d_{in}} \). The rows are then sorted by this score, and we select the top r rows according to either the Max or Min scores. The same procedure is applied to columns by summing across the rows, producing a column score \( S_{column} \in \mathbb{R}^{d_{out}} \). The Mixed strategy takes half of the rows/columns from Min and half from Max, while the random strategy selects rows and columns uniformly at random.

Figure 4 presents the comparative results of these four strategies on the SST-2, RTE, QNLI, CoLA, and MNLI datasets for rank \( r = 4 \). Across all datasets, the results show consistent robustness, indicating that our method performs well regardless of the selection criteria—whether based on Max, Min, MinMax, or random selection of rows or columns.

7 Conclusions

We present a novel PEFT method, termed RoCoFT, which finetunes selected rows and columns of model weights. Through an extensive series of experiments, we demonstrate that our method achieves competitive performance relative to other PEFT techniques, while significantly improving both memory efficiency and training time. Furthermore, by employing kernel methods, we show that the restricted kernels generated by our approach achieve comparable accuracy to full fine-tuning kernels in kernel logistic regression tasks. This indicates that RoCoFT effectively captures the most salient features from the full parameter kernel space. Future works include combining our RoCoFT method with quantization to achieve more compressed models during finetuning. We would also like to extend the kernel approach to the study and comparison of more PEFT methods.

8 Limitations

While RoCoFT achieves strong empirical performance and computational efficiency, it has several limitations that we acknowledge.

Scope of NTK Analysis. Our theoretical insights rely on the NTK framework under the lazy training regime, which assumes minimal parameter updates during fine-tuning. While this helps explain RoCoFT’s learning behavior, NTK regression does not fully capture all training scenarios. For instance, on tasks such as QNLI, MNLI, and QQP (Tables 1 and 4), the NTK approximation diverges significantly from actual fine-tuning accuracy, suggesting that the fixed kernel representation struggles with task-specific adaptations. In contrast, tasks like
SST-2, MRPC, and RTE exhibit a stronger alignment between NTK predictions and empirical fine-tuning results, indicating that the effectiveness of the approximating fine-tuning with NTK regression depends on the dataset’s complexity and distributional properties.

Static Row-Column Selection. RoCoFT selects random rows and columns statically. While this reduces computational overhead, it lacks adaptability to evolving training dynamics, where parameter importance may shift over time. For instance, early-selected rows or columns may become less relevant as training progresses, potentially slowing convergence on tasks that require iterative feature refinement. A dynamic selection strategy could improve adaptability but would introduce additional computational costs.

Selective Parameter Coverage. Unlike LoRA-type methods that adjust all columns via low-rank updates, RoCoFT modifies only a sparse subset of rows and columns. While this improves efficiency, it may lead to under-adaptation in tasks where critical features are spread across non-selected parameters (e.g., highly compositional tasks like textual entailment). Expanding RoCoFT’s coverage through hybrid row-column-rank updates could address this limitation, though at the expense of increased parameter count.

Memory and Runtime Trade-offs. Although RoCoFT updates only half as many parameters as LoRA, the actual savings in runtime and memory usage are lower than 50%. This is due to the overhead from word embeddings and state parameters maintained by the optimizer. Future work could explore optimization strategies to further reduce computational overhead while preserving efficiency.

References

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.

Alan Ansell, Edoardo Ponti, Anna Korhonen, and Ivan Vulić. 2022. Composable sparse fine-tuning for cross-lingual transfer. In Annual Meeting of the Association for Computational Linguistics, pages 1778–1796.

Alan Ansell, Ivan Vulić, Hannah Sterz, Anna Korhonen, and Edoardo M Ponti. 2024. Scaling sparse fine-tuning to large language models. arXiv preprint arXiv:2401.16405.

Klaudia Balazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. 2024. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint arXiv:2405.17604.

Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439.

Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044.

Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try ARC, the AI2 reasoning challenge. arXiv preprint arXiv:1803.05457.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168.

Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks. Machine Learning.

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024. QLoRA: Efficient finetuning of quantized LLMs. Advances in Neural Information Processing Systems, 36.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. 2022. KronA: Parameter efficient tuning with kro-necker adapter. arXiv preprint arXiv:2212.10650.

Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. In International Conference on Learning Representations.

Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.

Demi Guo, Alexander Rush, and Yoon Kim. 2021. Parameter-efficient transfer learning with Diff Pruning. In Annual Meeting of the Association for Computational Linguistics.
Demi Guo, Alexander M Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning. arXiv preprint arXiv:2012.07463.

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. 2024. The impact of initialization on LoRA finetuning dynamics. arXiv preprint arXiv:2406.08447.

Haoze He, Juncheng Billy Li, Xuan Jiang, and Heather Miller. 2024. Sparse matrix in large language model fine-tuning. arXiv preprint arXiv:2405.15525.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.

Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. DeBERTa: Decoding-enhanced BERT with disentangled attention. arXiv preprint arXiv:2006.03654.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. NeurIPS.

Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.

Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014. Learning to solve arithmetic word problems with verb categorization. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pages 523–533.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In International conference on machine learning, pages 2790–2799. PMLR.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. 2021. FedPara: Low-Rank hadamard product for communication-efficient federated learning. arXiv preprint arXiv:2108.06098.

Arthur Jacot, Franck Gabriel, and Clément Hongler. 2018. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31.

Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):3521–3526.

Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015. Parsing algebraic word problems into equations. Transactions of the Association for Computational Linguistics, 3:585–597.

Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. 2023. VeRA: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.11454.

Md Kowsher, Nusrat Jahan Prottasha, and Prakash Bhat. 2024. Propulsion: Steering LLM with tiny fine-tuning. arXiv preprint arXiv:2409.10927.

Md Kowsher, Md Shohanur Islam Sobuj, Asif Mahmud, Nusrat Jahan Prottasha, and Prakash Bhat. 2023. L-TUNING: Synchronized label tuning for prompt and prefix in LLMs. arXiv preprint arXiv:2402.01643.

Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Lucioni, François Yvon, Matthias Gallé, et al. 2023. BIOOM: A 176B-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100.

Jaehoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. 2019. Wide neural networks of any depth evolve as linear models under gradient descent. Advances in neural information processing systems, 32.

Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461.

Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Motta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. 2023. A kernel-based view of language model fine-tuning. In International Conference on Machine Learning, pages 23610–23641. PMLR.

Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.

Niklas Muenninghoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. 2022. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01786.

Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745.

Arkil Patel, Satwik Bhattamishra, and Navin Goyal. 2021. Are NLP models really able to solve simple math word problems? arXiv preprint arXiv:2103.07191.

Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.

Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.

Subhro Roy and Dan Roth. 2016. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. WinoGrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106.

Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. 2019. SocialIQA: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728.

Mingjie Sun, Zhuang Liu, Anna Bair, and J Zico Kolter. 2023. A simple and effective pruning approach for large language models. In Workshop on Efficient Systems for Foundation Models ICML.

Yi-Lin Sung, Varun Nair, and Colin Raffel. 2021. Training neural networks with fixed sparse masks. In Advances in Neural Information Processing Systems.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.

Ben Wang. 2021. Mesh-Transformer-JAX: Model-Parallel Implementation of Transformer Language Model with JAX.

Alexander Wei, Wei Hu, and Jacob Steinhardt. 2022. More than a toy: Random matrix models predict how real-world neural representations generalize. In International Conference on Machine Learning, pages 23549–23588. PMLR.

Christopher KI Williams and Carl Edward Rasmussen. 2006. Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.

Taiqiang Wu, Jiahao Wang, Zhe Zhao, and Ngai Wong. 2024. Mixture-of-subspaces in low-rank adaptation. arXiv preprint arXiv:2406.11909.

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148.

Greg Yang. 2020. Tensor programs II: Neural tangent kernel for any architecture. arXiv preprint arXiv:2006.14548.

Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv preprint arXiv:2106.10199.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.
Guangtao Zeng, Peiyuan Zhang, and Wei Lu. 2023. One network, many masks: Towards more parameter-efficient transfer learning. arXiv preprint arXiv:2305.17682.

Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. 2023a. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023b. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023c. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303.10512.

Bingchen Zhao, Haoqin Tu, Chen Wei, Jieru Mei, and Cihang Xie. 2023. Tuning layernorm in attention: Towards efficient multi-modal LLM finetuning. arXiv preprint arXiv:2312.11420.
A Additional Experiments

The results in Table 6 demonstrate that RoCoFT achieves state-of-the-art performance while maintaining parameter efficiency. Notably, RoCoFT3-Row attains the highest accuracy on SST-2 (96.69%), MRPC (91.05/92.19), and RTE (87.83%), while RoCoFT3-Column leads in STS-B (92.52/92.31) and QQP (91.38/87.12). Despite using significantly fewer parameters (0.222M–0.666M), RoCoFT consistently outperforms or matches larger PEFT methods such as LoRA, PROPELT, and MoSLoRA. Compared to MAM Adapter, which excels in CoLA (67.39 MCC), RoCoFT3-Row achieves equivalent performance while surpassing it in other tasks. Moreover, RoCoFT1-Row and RoCoFT1-Column provide an optimal balance between efficiency and effectiveness, significantly outperforming BitFit with a similar parameter budget. These results highlight RoCoFT’s effectiveness in enhancing performance across diverse GLUE tasks while maintaining minimal computational overhead.

As shown in Table 7, our proposed methods demonstrate superior performance on both question answering and summarization tasks while utilizing significantly fewer trainable parameters. Specifically, on the SQuAD v1.1 dataset (Rajpurkar et al., 2016), the RoCoFT3-Row method achieves the highest Exact Match (EM)/F1 scores of 81.70/88.15, outperforming other PEFT methods such as LoRA and AdaLoRA (Zhang et al., 2023c), which require more parameters. Similarly, on SQuAD v2.0 (Rajpurkar et al., 2018), the RoCoFT3-Column attains the top ROUGE-2 score of 18.54 on XSum, showcasing its effectiveness in handling text summarization.

B More Ablation Study

B.1 Optimal Rank r for RoCoFT

We investigate the impact of varying the rank \( r \) on the performance of RoCoFT (Row and Column Fine-Tuning) and compare it with the widely used LoRA method within the RoBERTa-base attention block. We assess key metrics such as training time, accuracy, number of parameters, and memory consumption for each rank \( r \in \{1, 2, 4, 8, 64\} \) using the SST2 dataset. The results are summarized in Table 8.

From the table, we observe that as the rank \( r \) increases, both RoCoFT and LoRA exhibit improved accuracy. For lower ranks, such as \( r = 1 \) and \( r = 2 \), RoCoFT_row and RoCoFT_column consistently outperform LoRA in terms of both training time and parameter efficiency, while maintaining competitive accuracy. Specifically, for rank \( r = 1 \), RoCoFT_row achieves an accuracy of 0.913 while using only 0.022 million parameters, which is significantly fewer than LoRA’s 0.055 million parameters for the same rank, with a slight increase in accuracy. This demonstrates the parameter efficiency of RoCoFT at lower ranks.

As the rank increases to \( r = 8 \), both RoCoFT variants continue to show slight improvements in accuracy while maintaining a faster training time compared to LoRA. Notably, at higher ranks like \( r = 64 \), RoCoFT_row achieves the highest accuracy of 0.934 with a significantly lower memory footprint compared to LoRA (2.656 GB vs. 2.993 GB).

B.2 RoCoFT with random weight selection

To test our hypothesis that finetuning LLMs can work as long as there are sufficient number of free parameters spread throughout the LLM model for training, we implement a version RoCoFT where instead of rows and columns, we uniformly sample entries with probability \( p \) from the weight matrices for updates and freeze the rest. Note that this method is not computationally efficient compared to updating only rows and columns and is only meant for ablation studies. From Tables 11 and 12 we can see that updating random entries in the weight matrix is competitive with all other PEFT methods (we use \( p = 0.1 \) and \( p = 0.01 \) in these experiments). This gives further evidence that most good features are already acquired during pretraining and little learning is required during the finetuning stage.

C Implementation

In Algorithm 1, we present a simplified PyTorch implementation of RoCoFT (row version). The main idea is to replace the linear layers in transformer model with the shown module, where \( r \) rows are selected to be trainable weights and the remaining ones are frozen and converted to buffers (so that their gradients are not computed during backward pass). The forward function is simply the same as the original linear layer with the weights a concatenation of trainable and frozen weights. The version for columns are implemented similarly.
<table>
  <tr>
    <th>LM</th>
    <th>PEFT Method</th>
    <th>#TTPs</th>
    <th>CoLA</th>
    <th>SST2</th>
    <th>MRPC</th>
    <th>STS-B</th>
    <th>QQP</th>
    <th>MNLI</th>
    <th>QNLI</th>
    <th>RTE</th>
  </tr>
  <tr>
    <td rowspan="13">Roberta-large</td>
    <td>FT</td>
    <td>355.3M</td>
    <td>65.78</td>
    <td>95.50</td>
    <td>92.22/94.28</td>
    <td>91.74/91.96</td>
    <td>90.83/88.68</td>
    <td>89.21</td>
    <td>93.19</td>
    <td>81.40</td>
  </tr>
  <tr>
    <td>Adapter<sup>a</sup></td>
    <td>19.77M</td>
    <td>65.33</td>
    <td>96.37</td>
    <td>89.88/90.23</td>
    <td><b>92.58/92.42</b></td>
    <td>91.19/87.11</td>
    <td>91.00</td>
    <td>94.31</td>
    <td>85.25</td>
  </tr>
  <tr>
    <td>Prompt-tuning</td>
    <td>1.07M</td>
    <td>61.13</td>
    <td>94.61</td>
    <td>73.04/76.29</td>
    <td>78.51/78.99</td>
    <td>80.74/75.16</td>
    <td>68.15</td>
    <td>83.19</td>
    <td>60.29</td>
  </tr>
  <tr>
    <td>Prefix-tuning</td>
    <td>2.03M</td>
    <td>59.01</td>
    <td>95.76</td>
    <td>88.24/89.37</td>
    <td>90.92/91.07</td>
    <td>88.88/85.45</td>
    <td>89.30</td>
    <td>93.32</td>
    <td>74.01</td>
  </tr>
  <tr>
    <td>(IA)<sup>b</sup></td>
    <td>1.22M</td>
    <td>61.15</td>
    <td>94.61</td>
    <td>86.45/87.53</td>
    <td>92.22/86.25</td>
    <td>89.45/86.25</td>
    <td>88.63</td>
    <td>94.25</td>
    <td>81.23</td>
  </tr>
  <tr>
    <td>Bitfit</td>
    <td>0.222M</td>
    <td>67.01</td>
    <td>96.10</td>
    <td>90.93/92.13</td>
    <td>91.93/<b>93.38</b></td>
    <td>89.48/86.43</td>
    <td>89.98</td>
    <td>94.47</td>
    <td>87.73</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>1.84M</td>
    <td>64.47</td>
    <td>96.67</td>
    <td>87.50/88.19</td>
    <td>91.66/91.44</td>
    <td>90.15/86.91</td>
    <td>90.76</td>
    <td>95.00</td>
    <td>79.78</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>2.23M</td>
    <td>65.85</td>
    <td>94.95</td>
    <td>89.46/90.34</td>
    <td>92.05/91.80</td>
    <td>89.60/86.30</td>
    <td>90.36</td>
    <td>94.62</td>
    <td>77.98</td>
  </tr>
  <tr>
    <td>MAM Adapter</td>
    <td>4.20M</td>
    <td><b>67.39</b></td>
    <td>95.81</td>
    <td>90.12/92.07</td>
    <td>92.44/92.18</td>
    <td>90.87/86.65</td>
    <td>90.62</td>
    <td>94.31</td>
    <td>86.62</td>
  </tr>
  <tr>
    <td>PROPELT-Adapter</td>
    <td>5.40M</td>
    <td>65.55</td>
    <td>96.27</td>
    <td>89.71/91.15</td>
    <td>91.92/91.67</td>
    <td>90.67/87.74</td>
    <td><b>91.37</b></td>
    <td>94.80</td>
    <td>87.69</td>
  </tr>
  <tr>
    <td>PROPELT-Prefix</td>
    <td>26.85M</td>
    <td>62.24</td>
    <td>96.17</td>
    <td>90.04/91.92</td>
    <td>90.70/90.49</td>
    <td>89.30/86.30</td>
    <td>90.33</td>
    <td>94.73</td>
    <td>79.71</td>
  </tr>
  <tr>
    <td>PROPELT_LoRA</td>
    <td>4.19M</td>
    <td>61.90</td>
    <td>95.93</td>
    <td>87.31/89.87</td>
    <td>81.66/91.38</td>
    <td>90.93/88.05</td>
    <td>90.53</td>
    <td>94.93</td>
    <td>83.57</td>
  </tr>
  <tr>
    <td>MoSLoRA</td>
    <td>3.23M</td>
    <td>67.27</td>
    <td>96.17</td>
    <td>89.96/92.67</td>
    <td>90.97/91.72</td>
    <td>90.12/87.68</td>
    <td>90.29</td>
    <td>94.73</td>
    <td>82.41</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>0.222M</td>
    <td>65.70</td>
    <td>96.63</td>
    <td>89.97/90.79</td>
    <td>91.81/92.07</td>
    <td>90.17/86.15</td>
    <td>90.73</td>
    <td>94.20</td>
    <td>85.31</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>0.666M</td>
    <td><b>67.39</b></td>
    <td><b>96.69</b></td>
    <td><b>91.05/92.19</b></td>
    <td>91.09/92.10</td>
    <td>90.82/86.11</td>
    <td>90.98</td>
    <td>94.85</td>
    <td>87.83</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub>; Column</td>
    <td>0.222M</td>
    <td>64.89</td>
    <td>96.60</td>
    <td>89.12/90.24</td>
    <td>91.96/92.10</td>
    <td>90.17/85.83</td>
    <td>90.81</td>
    <td>94.17</td>
    <td>85.71</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub>; Column</td>
    <td>0.666M</td>
    <td>67.18</td>
    <td><b>96.67</b></td>
    <td>89.88/91.47</td>
    <td><b>92.52/92.31</b></td>
    <td><b>91.38/87.12</b></td>
    <td><b>91.13</b></td>
    <td>94.85</td>
    <td>87.82</td>
  </tr>
</table>

Table 6: RoBERTa-large models performance on GLUE tasks: Metrics used are MCC for CoLA, accuracy for SST-2, accuracy/F1 score for MRPC and QQP, Pearson/Spearman correlations for STS-B, and accuracy for MNLI, QNLI, and RTE.

<table>
  <tr>
    <th rowspan="2">PEFT Method</th>
    <th colspan="2">DeBERTaV3-base</th>
    <th colspan="4">BART-large</th>
  </tr>
  <tr>
    <th>#TTPs</th>
    <th>SQuADv1.1</th>
    <th>SQuADv2.0</th>
    <th>#TTPs</th>
    <th>XSum</th>
    <th>CNN/DailyMail</th>
  </tr>
  <tr>
    <td>FT</td>
    <td>184M</td>
    <td>82.83 / 88.14</td>
    <td>82.92 / 83.75</td>
    <td>460M</td>
    <td>40.73 / 16.19 / 30.13</td>
    <td>39.16 / 18.92 / 37.04</td>
  </tr>
  <tr>
    <td>Prompt tuning</td>
    <td>0.650M</td>
    <td>74.52 / 78.42</td>
    <td>73.59 / 76.72</td>
    <td>0.755M</td>
    <td>38.24 / 14.46 / 27.89</td>
    <td>37.42 / 17.43 / 34.92</td>
  </tr>
  <tr>
    <td>Prefix-tuning</td>
    <td>1.733M</td>
    <td>78.38 / 82.94</td>
    <td>74.94 / 79.04</td>
    <td>2.983M</td>
    <td>38.24 / 15.16 / 28.84</td>
    <td>38.32 / 17.72 / 35.76</td>
  </tr>
  <tr>
    <td>LoKr</td>
    <td>0.815M</td>
    <td>80.64 / 86.45</td>
    <td>80.14 / 81.96</td>
    <td>1.089M</td>
    <td>39.03 / 16.14 / 30.42</td>
    <td><b>40.83 / 19.10 / 38.75</b></td>
  </tr>
  <tr>
    <td>Bitfit</td>
    <td>0.172M</td>
    <td>80.53 / 86.25</td>
    <td>79.06 / 83.75</td>
    <td>0.672M</td>
    <td>39.10 / 16.87 / 30.43</td>
    <td>39.93 / 18.12 / 38.85</td>
  </tr>
  <tr>
    <td>LoHa</td>
    <td>0.765M</td>
    <td>81.43 / 88.02</td>
    <td>81.67 / 85.01</td>
    <td>1.285M</td>
    <td>40.12 / 18.08 / <b>32.39</b></td>
    <td>39.98 / 18.84 / 38.01</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>0.740M</td>
    <td>81.64 / 87.16</td>
    <td>82.56 / 85.75</td>
    <td>1.242M</td>
    <td><b>40.63 / 18.44 / 32.15</b></td>
    <td>40.74 / 19.10 / 39.24</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>0.810M</td>
    <td>81.16 / 87.75</td>
    <td>82.63 / <b>85.82</b></td>
    <td>1.663M</td>
    <td>40.95 / 18.28 / 31.84</td>
    <td>40.53 / 18.24 / <b>39.63</b></td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>0.161M</td>
    <td><b>81.70 / 88.15</b></td>
    <td>82.76 / 85.14</td>
    <td>0.597M</td>
    <td>40.12 / <b>18.48 / 31.93</b></td>
    <td><b>40.83 / 19.12 / 39.55</b></td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>0.161M</td>
    <td>81.63 / 88.11</td>
    <td>82.60 / 85.05</td>
    <td>0.597M</td>
    <td><b>40.62 / 18.54 / 32.17</b></td>
    <td>40.18 / 19.10 / 39.21</td>
  </tr>
</table>

Table 7: Results of DeBERTaV3-base on SQuAD v1.1, v2.0 benchmarks, reported using EM/F1 scores and BART-large on XSum and CNN/Daily Mail, reported using ROUGE metrics as ROUGE-1/ROUGE-2/ROUGE-L.

<table>
  <tr>
    <th>Rank</th>
    <th>Algorithm</th>
    <th>Time</th>
    <th>Accuracy</th>
    <th>Parameters</th>
    <th>Memory</th>
  </tr>
  <tr>
    <td rowspan="3">1</td>
    <td>LoRA</td>
    <td>3:12</td>
    <td>0.910</td>
    <td>0.055</td>
    <td>2762</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>3:00</td>
    <td>0.913</td>
    <td>0.022</td>
    <td>2372</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>2:59</td>
    <td>0.912</td>
    <td>0.022</td>
    <td>2373</td>
  </tr>
  <tr>
    <td rowspan="3">2</td>
    <td>LoRA</td>
    <td>3:25</td>
    <td>0.922</td>
    <td>0.110</td>
    <td>2768</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>3:00</td>
    <td>0.920</td>
    <td>0.055</td>
    <td>2410</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>3:00</td>
    <td>0.922</td>
    <td>0.055</td>
    <td>2414</td>
  </tr>
  <tr>
    <td rowspan="3">4</td>
    <td>LoRA</td>
    <td>3:27</td>
    <td>0.925</td>
    <td>0.221</td>
    <td>2771</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>3:01</td>
    <td>0.923</td>
    <td>0.110</td>
    <td>2450</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>3:01</td>
    <td>0.922</td>
    <td>0.110</td>
    <td>2451</td>
  </tr>
  <tr>
    <td rowspan="3">8</td>
    <td>LoRA</td>
    <td>3:29</td>
    <td>0.929</td>
    <td>0.442</td>
    <td>2783</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>3:03</td>
    <td>0.930</td>
    <td>0.221</td>
    <td>2336</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>3:02</td>
    <td>0.928</td>
    <td>0.221</td>
    <td>2335</td>
  </tr>
  <tr>
    <td rowspan="3">64</td>
    <td>LoRA</td>
    <td>3:33</td>
    <td>0.928</td>
    <td>3.538</td>
    <td>2993</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Row</sub></td>
    <td>3:06</td>
    <td>0.934</td>
    <td>1.769</td>
    <td>2656</td>
  </tr>
  <tr>
    <td>RoCoFT<sub>Column</sub></td>
    <td>3:05</td>
    <td>0.933</td>
    <td>1.769</td>
    <td>2653</td>
  </tr>
</table>

Table 8: Comparison with LoRA in terms of rank, training time (minutes), accuracy, number of parameters, and memory usage (MB).

D \( \Delta W \) Representation

A comparison of the \( \Delta W \) representations across different PEFT methods is provided in Table 9.

E Hyper-parameters for RoCoFT

The hyperparameters used in RoCoFT are provided in Table 10.

F Environmental Setup and Implementation Details

In order to implement RoCoFT, we have set up a comprehensive environment using key frameworks and tools to ensure efficient training and evaluation. We utilized PyTorch 2.4.1 as our primary deep learning framework, along with Hugging-face’s Transformers library version 4.44.1, which provides a wide array of pre-trained models and tokenizers, ensuring seamless integration with the RoCoFT method. To optimize the training process, we leveraged Accelerate 0.34.2, which is particu-
class RoCoFTRow(nn.Module):
    # inputs: F is the Linear Layer to be converted
    #         r is the rank (number of rows/columns to be selected)
    #         use_bias decides whether to train bias term
    def __init__(self, F, r, use_bias):
        # Set rows 1 to rank r as trainable weights
        self.trainable_W = nn.Parameter(F.weight[:r, :].clone())

        # Set rows r and above as non-trainable and move to buffer
        self.register_buffer('non_trainable_W', F.weight[r:, :].clone().detach())

        # Handle bias
        if F.bias is not None:
            self.bias = nn.Parameter(F.bias.clone().detach(), requires_grad=use_bias)
        else:
            self.bias = None

    def forward(self, x):
        full_weight = torch.cat([self.trainable_W, self.non_trainable_W], dim=1)
        out = torch.nn.functional.linear(x, full_weight, self.bias)
        return out

Algorithm 1: PyTorch pseudocode for replacing a linear layer in transformer model (row version)

<table>
  <tr>
    <th>Method</th>
    <th>\( \Delta W \) Reparameterization</th>
    <th>Notes</th>
  </tr>
  <tr>
    <td>Intrinsic SAID</td>
    <td>\( \Delta W = F(W^r) \)</td>
    <td>F : \( \mathbb{R}^r \to \mathbb{R}^d \), \( W^r \in \mathbb{R}^r \) are parameters to be optimized, and \( r \ll d \).</td>
  </tr>
  <tr>
    <td>LoRA</td>
    <td>\( \Delta W = W_{\text{down}} W_{\text{up}} \)</td>
    <td>\( W_{\text{down}} \in \mathbb{R}^{d \times r}, W_{\text{up}} \in \mathbb{R}^{r \times d}, \) and \( r \ll \{k, d\} \).</td>
  </tr>
  <tr>
    <td>KronA</td>
    <td>\( \Delta W = W_{\text{down}} \otimes W_{\text{up}} \)</td>
    <td>\( \text{rank}(W_{\text{down}} \otimes W_{\text{up}}) = \text{rank}(W_{\text{down}}) \times \text{rank}(W_{\text{up}}) \).</td>
  </tr>
  <tr>
    <td>DyLoRA</td>
    <td>\( \Delta W = W_{\text{down},b} W_{\text{up},b} \)</td>
    <td>\( W_{\text{down},b} = W_{\text{down}}[:, b, :], \quad W_{\text{up},b} = W_{\text{up}}[:, :, b], \quad b \in \{r_{\min}, \cdots, r_{\max}\} \).</td>
  </tr>
  <tr>
    <td>AdaLoRA</td>
    <td>\( \Delta W = PAQ \)</td>
    <td>\( P^T = P^T P \neq I = QQ^T = Q^T Q, A = \text{diag}(\sigma_1, \sigma_2, \ldots, \sigma_r) \).</td>
  </tr>
  <tr>
    <td>IncreLoRA</td>
    <td>\( \Delta W = W_{\text{down}} \Delta W_{\text{up}} \)</td>
    <td>\( \Lambda = [\lambda_1, \lambda_2, \ldots, \lambda_r] \) with \( \lambda_i \) being an arbitrary constant.</td>
  </tr>
  <tr>
    <td>DeltaLoRA</td>
    <td>\( \Delta W = W_{\text{down}} W_{\text{up}} \)</td>
    <td>\( W^{(t+1)} \leftarrow W^{(t)} + \left( W_{\text{down}}^{(t+1)} W_{\text{up}}^{(t+1)} - W_{\text{down}}^{(t)} W_{\text{up}}^{(t)} \right) \).</td>
  </tr>
  <tr>
    <td>LoRAPrune</td>
    <td>\( \Delta W = W_{\text{down}} W_{\text{up}} \odot M \)</td>
    <td>\( \delta = (W + W_{\text{down}} W_{\text{up}}) \odot M, \quad M \in \{0, 1\}_{n \times G}, \quad G \) is group number</td>
  </tr>
  <tr>
    <td>QLoRA</td>
    <td>\( \Delta W = W_{\text{down}}^{BF16} W_{\text{up}}^{BF16} \)</td>
    <td>\( Y^{BF16} = X^{BF16} \text{doubleDequant}(c_1^{P^{\text{out}}}, c_2^{P^{\text{in}}}, W^{NF4}) + X^{BF16} W_{\text{down}}^{BF16} W_{\text{down}}^{BF16} \)</td>
  </tr>
  <tr>
    <td>QA-LoRA</td>
    <td>\( \Delta W = W_{\text{down}} W_{\text{up}} \)</td>
    <td>\( W_{\text{down}} \in \mathbb{R}^{r \times L}, W_{\text{up}} \in \mathbb{R}^{L \times r}, L \) is the quantization group number of \( W \).</td>
  </tr>
  <tr>
    <td>LoFTQ</td>
    <td>\( \Delta W = \text{SVD}(W - Q_t) \)</td>
    <td>\( Q_t = q_N \left( W - W_{\text{down}}^{(t-1)} W_{\text{up}}^{(t-1)} \right), \quad q_N \) is \( N \)-bit quantization function</td>
  </tr>
  <tr>
    <td>Kernel-mix</td>
    <td>\( \Delta W^h = [B_{\text{LoRA}}^h B^h] \frac{A_{\text{LoRA}}^h}{A^h} \)</td>
    <td>\( B_{\text{LoRA}} \) is shared across all heads. \( B_h^A \) provides rank \( r \) update in each head.</td>
  </tr>
  <tr>
    <td>LoRA-FA</td>
    <td>\( \Delta W = W_{\text{down}} W_{\text{up}} = QRW_{\text{up}} \)</td>
    <td>\( W_{\text{down}} \) is frozen, and only \( W_{\text{up}} \) is updated.</td>
  </tr>
  <tr>
    <td>RoCoFT</td>
    <td>\( W = W_0 + R \\ W = W_0 + C \)</td>
    <td>R and C are restricted weight matrices such that only at most \( r \) of the rows or columns are non-zero.</td>
  </tr>
</table>

Table 9: Comparison of reparameterization of various PEFT methods.
<table>
  <tr>
    <th>Dataset</th>
    <th>Learning Rate</th>
    <th>Epochs</th>
    <th>Batch size</th>
    <th>Dropout Weight Decay</th>
    <th>Warmup Steps</th>
    <th>Learning Scheduler</th>
    <th>Bias</th>
    <th>Pruning Layer Rank</th>
    <th>Gradient Accumul.</th>
  </tr>
  <tr>
    <td>CgLA</td>
    <td>2e-3</td>
    <td>20</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>min</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SST2</td>
    <td>2e-4</td>
    <td>3</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>max</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>2e-3</td>
    <td>10</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>1e-3</td>
    <td>10</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>1e-4</td>
    <td>2</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>1e-3</td>
    <td>2</td>
    <td>16</td>
    <td>0.10 0.001</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>1e-3</td>
    <td>2</td>
    <td>16</td>
    <td>0.10 0.001</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>2e-3</td>
    <td>30</td>
    <td>32</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>False</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SQuADv1.1</td>
    <td>1e-4</td>
    <td>4</td>
    <td>16</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>SQuADv2.0</td>
    <td>1e-4</td>
    <td>4</td>
    <td>16</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>XSum</td>
    <td>1e-4</td>
    <td>4</td>
    <td>16</td>
    <td>0.10 0.01</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>DailyMail</td>
    <td>1e-4</td>
    <td>4</td>
    <td>16</td>
    <td>0.10 0.01</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>0</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>PiQA</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>SIQA</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>Hellaswag</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>W.Gra.</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>ARCc</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>ARCg</td>
    <td>2e-3</td>
    <td>2</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>OBQA</td>
    <td>2e-3</td>
    <td>1</td>
    <td>3</td>
    <td>0.10 0.00</td>
    <td>100</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>3</td>
  </tr>
  <tr>
    <td>MultiArith</td>
    <td>1e-3</td>
    <td>2</td>
    <td>8</td>
    <td>0.10 0.00</td>
    <td>500</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <td>Gsm8k</td>
    <td>1e-3</td>
    <td>2</td>
    <td>8</td>
    <td>0.10 0.00</td>
    <td>500</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <td>AddSub</td>
    <td>1e-3</td>
    <td>2</td>
    <td>8</td>
    <td>0.10 0.00</td>
    <td>500</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <td>SingleEq</td>
    <td>1e-3</td>
    <td>2</td>
    <td>8</td>
    <td>0.10 0.00</td>
    <td>500</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>2</td>
  </tr>
  <tr>
    <td>SVAMP</td>
    <td>1e-3</td>
    <td>2</td>
    <td>8</td>
    <td>0.10 0.00</td>
    <td>500</td>
    <td>cosine</td>
    <td>True</td>
    <td>random</td>
    <td>1e-05</td>
    <td>3</td>
    <td>2</td>
  </tr>
</table>

Table 10: Hyperparameters for RoCoFT (row and column)

larly helpful for distributed training across multiple GPUs and scaling large model deployments. This tool enabled us to efficiently manage computational resources and fine-tune the performance of large language models.

For our hardware setup, we utilized two distinct types of GPUs to optimize training based on the task requirements. For tasks like GLUE, question answering, and text summarization, we deployed NVIDIA A100 GPUs. These tasks, which are less computationally intensive compared to full LLM training, were efficiently handled by the A100s. For larger and more demanding tasks such as evaluating the performance of LLMs, we used NVIDIA H100 GPUs with 80 GB of VRAM. The H100 provided the necessary memory and computational power to handle the fine-tuning of LLMs, especially given the large model sizes and extensive data required for these tasks. This configuration allowed us to achieve significant speedups during both training and inference, while also managing memory-intensive processes with ease.

In addition to the hardware and software setup, special attention was given to the data pipeline to ensure smooth loading and processing of large datasets required for RoCoFT. Data preprocessing steps, such as tokenization and sequence padding, were handled by the Huggingface library, streamlining the preparation of input for the models. The combination of these tools and hardware resources ensured that we could efficiently implement RoCoFT across a variety of tasks while maintaining high performance and scalability.

G Additional Neural Tangent Kernel Results

Here we include additional results on our Neural Tangent Kernel experiments. Figure 5 shows the eigenvalue distribution of the full kernel \( \mathbf{K}_\theta \), 1-row kernel \( \mathbf{K}_{\theta_r} \) and 1-column kernel \( \mathbf{K}_{\theta_c} \) on different datasets. The eigenvalues are rescaled per dataset and we can see the eigenvalue distributions are very similar for the three NTK kernels. Table 13 shows the \( \ell_1 \) and \( \ell_2 \) norm difference between the kernel matrices of the 64-shot tasks, and the results are largely similar to the 16-shot results. The difference is mostly within 5-15%, but with smaller standard deviation than the 16-shot results over 5 random seeds. In Figure 6, we include a few more visualizations of the kernel matrices for the 16-shot tasks. We can see the three type of NTK matrices show very similar patterns across all tasks.

H Dataset Description

The datasets used in this study are listed in Table 14 and Table 15.

I Evaluation Metrics

We employ specific evaluation metrics tailored to each task within the GLUE benchmark suite (Wang et al., 2018) to assess the performance of our models comprehensively.

For the Corpus of Linguistic Acceptability (CoLA) task, we use the Matthews Correlation
<table>
  <tr>
    <th>LM</th>
    <th># TTPs</th>
    <th>ColA</th>
    <th>SST2</th>
    <th>MRPC</th>
    <th>STS-B</th>
    <th>QQP</th>
    <th>MNLI</th>
    <th>QNLI</th>
    <th>RTE</th>
  </tr>
  <tr>
    <td><b>RobertaBase</b></td>
    <td>12.4M</td>
    <td>63.15</td>
    <td>94.96</td>
    <td>88.03/89.18</td>
    <td>90.57/90.07</td>
    <td>89.29/86.94</td>
    <td>87.22</td>
    <td>92.60</td>
    <td>80.01</td>
  </tr>
  <tr>
    <td><b>RobertaLarge</b></td>
    <td>35.5M</td>
    <td>65.32</td>
    <td>96.59</td>
    <td>90.93/92.03</td>
    <td>92.10/92.05</td>
    <td>90.97/86.78</td>
    <td>90.89</td>
    <td>95.06</td>
    <td>87.91</td>
  </tr>
</table>

Table 11: RoBERTa models performance on GLUE tasks using 10% random sampling of trainable parameters from each weight matrix (\( p = 0.1 \)).

<table>
  <tr>
    <th>LLM</th>
    <th># TTPs</th>
    <th>BoolQ</th>
    <th>PIQA</th>
    <th>SIQA</th>
    <th>HSw.</th>
    <th>W.Gra.</th>
    <th>ARCc</th>
    <th>ARCc</th>
    <th>OBOA</th>
    <th>M.Ar.</th>
    <th>G.8K</th>
    <th>A.S.</th>
    <th>S.eEq</th>
    <th>S.MP</th>
  </tr>
  <tr>
    <td>BLOOMZ<sub>7B</sub></td>
    <td>70.4M</td>
    <td>65.76</td>
    <td>74.62</td>
    <td>73.50</td>
    <td>56.39</td>
    <td>72.11</td>
    <td>72.89</td>
    <td>56.88</td>
    <td>72.43</td>
    <td>79.78</td>
    <td>71.11</td>
    <td>70.76</td>
    <td>70.91</td>
    <td>54.37</td>
  </tr>
  <tr>
    <td>GPT-J<sub>6B</sub></td>
    <td>60.3M</td>
    <td>65.75</td>
    <td>68.63</td>
    <td>69.12</td>
    <td>45.50</td>
    <td>66.47</td>
    <td>64.99</td>
    <td>46.91</td>
    <td>65.37</td>
    <td>89.34</td>
    <td>72.62</td>
    <td>80.64</td>
    <td>82.14</td>
    <td>55.90</td>
  </tr>
  <tr>
    <td>LLaMA2<sub>7B</sub></td>
    <td>71.2M</td>
    <td>69.30</td>
    <td>80.12</td>
    <td>77.95</td>
    <td>89.40</td>
    <td>76.52</td>
    <td>76.57</td>
    <td>60.62</td>
    <td>76.92</td>
    <td>90.46</td>
    <td>77.32</td>
    <td>86.13</td>
    <td>82.49</td>
    <td>60.72</td>
  </tr>
  <tr>
    <td>LLaMA2<sub>13B</sub></td>
    <td>129.8M</td>
    <td>71.44</td>
    <td>83.37</td>
    <td>79.32</td>
    <td>91.95</td>
    <td>83.32</td>
    <td>83.99</td>
    <td>66.92</td>
    <td>81.32</td>
    <td>91.49</td>
    <td>80.04</td>
    <td>87.71</td>
    <td>87.64</td>
    <td>66.83</td>
  </tr>
</table>

Table 12: Accuracy comparison of commonsense and mathematical reasoning performance across different datasets using LLMs, using 1% random sampling of total trainable model parameters from each weight matrix (\( p = 0.01 \)).

![Eigenvalue spectrum of K_θ, K_{θ_c}, and K_{θ_c}^*](page_184_613_482_312.png)

Figure 5: Eigenvalue spectrum of \( K_{θ} \), \( K_{θ_c} \), and \( K_{θ_c}^* \). The eigenvalues with respect to each task are scaled with \( β_{task} \) for better representation.

Coefficient (MCC) as the evaluation metric. MCC is suitable for binary classification tasks, especially with imbalanced datasets, as it takes into account true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN):

\[
\text{MCC} = \frac{\text{TP} \times \text{TN} - \text{FP} \times \text{FN}}{\sqrt{(\text{TP} + \text{FP})(\text{TP} + \text{FN})(\text{TN} + \text{FP})(\text{TN} + \text{FN})}}.
\]

For the Microsoft Research Paraphrase Corpus (MRPC) and Quora Question Pairs (QQP) tasks, which evaluate the model’s ability to determine semantic equivalence between sentence pairs, we use both Accuracy and F1 Score as evaluation metrics. Accuracy measures the proportion of correctly identified paraphrase pairs, while the F1 score balances precision and recall:

\[
\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}},
\]

\[
\text{F1} = 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}},
\]

where precision and recall are defined as:

\[
\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}, \quad \text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}.
\]

For the Multi-Genre Natural Language Inference (MNLI) task, which involves classifying sentence pairs into entailment, contradiction, or neutral, we report the Average Matched Accuracy. This metric measures the model’s accuracy on the matched validation set (in-domain data), reflecting its ability to generalize across different genres.

For the Semantic Textual Similarity Benchmark (STS-B) task, which requires predicting the degree of semantic similarity between sentence pairs, we use both the Pearson and Spearman correlation coefficients. These metrics evaluate the linear and rank-order relationships between the predicted scores (\( x_i \)) and the ground-truth scores (\( y_i \)), respectively:

\[
\text{Pearson's } r = \frac{\sum_{i=1}^{n}(x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum_{i=1}^{n}(x_i - \bar{x})^2 \sum_{i=1}^{n}(y_i - \bar{y})^2}}
\]

\[
\text{Spearman's } \rho = 1 - \frac{6 \sum_{i=1}^{n} d_i^2}{n(n^2 - 1)},
\]

where \( \bar{x} \) and \( \bar{y} \) are the means of the predicted and ground-truth scores, \( d_i \) is the difference between the ranks of \( x_i \) and \( y_i \), and \( n \) is the number of data points.

These evaluation metrics provide a comprehensive assessment of our models across diverse linguistic tasks, enabling us to measure both classification accuracy and the ability to capture semantic nuances.
<table>
  <tr>
    <th>64-shot (single)</th>
    <th>SST-2</th>
    <th>SST-5</th>
    <th>MR</th>
    <th>CR</th>
    <th>MPQA</th>
    <th>Subj</th>
    <th>TREC</th>
  </tr>
  <tr>
    <td>\( K_{\theta_R}, p = 1 \)</td>
    <td>0.091(0.007)</td>
    <td>0.084(0.002)</td>
    <td>0.067(0.005)</td>
    <td>0.084(0.005)</td>
    <td>0.126(0.014)</td>
    <td>0.061(0.002)</td>
    <td>0.184(0.003)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_R}, p = 2 \)</td>
    <td>0.126(0.012)</td>
    <td>0.113(0.002)</td>
    <td>0.100(0.015)</td>
    <td>0.115(0.013)</td>
    <td>0.176(0.025)</td>
    <td>0.076(0.008)</td>
    <td>0.202(0.004)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_C}, p = 1 \)</td>
    <td>0.088(0.007)</td>
    <td>0.079(0.002)</td>
    <td>0.064(0.005)</td>
    <td>0.080(0.005)</td>
    <td>0.125(0.015)</td>
    <td>0.055(0.002)</td>
    <td>0.169(0.003)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_C}, p = 2 \)</td>
    <td>0.124(0.012)</td>
    <td>0.108(0.003)</td>
    <td>0.098(0.014)</td>
    <td>0.110(0.011)</td>
    <td>0.178(0.026)</td>
    <td>0.071(0.004)</td>
    <td>0.191(0.004)</td>
  </tr>
  <tr>
    <th colspan="2">64-shot (pair)</th>
    <th>MNLI</th>
    <th>SNLI</th>
    <th>QNLI</th>
    <th>RTE</th>
    <th>MRPC</th>
    <th>QQP</th>
  </tr>
  <tr>
    <td>\( K_{\theta_R}, p = 1 \)</td>
    <td>0.181(0.012)</td>
    <td>0.205(0.013)</td>
    <td>0.074(0.013)</td>
    <td>0.128(0.004)</td>
    <td>0.073(0.009)</td>
    <td>0.049(0.007)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_R}, p = 2 \)</td>
    <td>0.251(0.037)</td>
    <td>0.259(0.033)</td>
    <td>0.179(0.069)</td>
    <td>0.180(0.011)</td>
    <td>0.093(0.004)</td>
    <td>0.099(0.065)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_C}, p = 1 \)</td>
    <td>0.179(0.013)</td>
    <td>0.200(0.014)</td>
    <td>0.071(0.013)</td>
    <td>0.125(0.005)</td>
    <td>0.073(0.003)</td>
    <td>0.048(0.007)</td>
  </tr>
  <tr>
    <td>\( K_{\theta_C}, p = 2 \)</td>
    <td>0.254(0.040)</td>
    <td>0.257(0.034)</td>
    <td>0.172(0.065)</td>
    <td>0.186(0.013)</td>
    <td>0.093(0.004)</td>
    <td>0.099(0.067)</td>
  </tr>
</table>

Table 13: Relative difference in kernels (compared to full parameter \( K_\theta \)) on single-sentence and sentence-pair tasks for 64-shot tasks

![Neural Tangent Kernels on 16-shot training data for different tasks](page_104_1042_1272_1012.png)

Figure 6: Neural Tangent Kernels on 16-shot training data for different tasks
<table>
  <tr>
    <th>Dataset</th>
    <th>Domain</th>
    <th>Train</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>MultiArith</td>
    <td>Math</td>
    <td>–</td>
    <td>600</td>
  </tr>
  <tr>
    <td>AddSub</td>
    <td>Math</td>
    <td>–</td>
    <td>395</td>
  </tr>
  <tr>
    <td>GSM8K</td>
    <td>Math</td>
    <td>8.8K</td>
    <td>1,319</td>
  </tr>
  <tr>
    <td>AQuA</td>
    <td>Math</td>
    <td>100K</td>
    <td>254</td>
  </tr>
  <tr>
    <td>SingleEq</td>
    <td>Math</td>
    <td>–</td>
    <td>508</td>
  </tr>
  <tr>
    <td>SVAMP</td>
    <td>Math</td>
    <td>–</td>
    <td>1,000</td>
  </tr>
  <tr>
    <td>BoolQ</td>
    <td>CS</td>
    <td>9.4K</td>
    <td>3,270</td>
  </tr>
  <tr>
    <td>PIQA</td>
    <td>CS</td>
    <td>16.1K</td>
    <td>1,830</td>
  </tr>
  <tr>
    <td>SIQA</td>
    <td>CS</td>
    <td>33.4K</td>
    <td>1,954</td>
  </tr>
  <tr>
    <td>HellaSwag</td>
    <td>CS</td>
    <td>39.9K</td>
    <td>10,042</td>
  </tr>
  <tr>
    <td>WinoGrande</td>
    <td>CS</td>
    <td>63.2K</td>
    <td>1,267</td>
  </tr>
  <tr>
    <td>ARC-e</td>
    <td>CS</td>
    <td>1.1K</td>
    <td>2,376</td>
  </tr>
  <tr>
    <td>ARC-c</td>
    <td>CS</td>
    <td>2.3K</td>
    <td>1,172</td>
  </tr>
  <tr>
    <td>OBQA</td>
    <td>CS</td>
    <td>5.0K</td>
    <td>500</td>
  </tr>
</table>

Table 14: Overview of Datasets for Mathematical and Commonsense Reasoning

<table>
  <tr>
    <th>Dataset</th>
    <th>Train</th>
    <th>Validation</th>
    <th>Test</th>
  </tr>
  <tr>
    <td>SQuAD v1.1</td>
    <td>87.6k</td>
    <td>10.6k</td>
    <td>-</td>
  </tr>
  <tr>
    <td>SQuAD v2.0</td>
    <td>130k</td>
    <td>11.9k</td>
    <td>-</td>
  </tr>
  <tr>
    <td>XSum</td>
    <td>204k</td>
    <td>11.3k</td>
    <td>11.3k</td>
  </tr>
  <tr>
    <td>DailyMail</td>
    <td>287k</td>
    <td>13.4k</td>
    <td>11.5k</td>
  </tr>
  <tr>
    <td>CoLA</td>
    <td>8.55k</td>
    <td>1.04k</td>
    <td>1.06k</td>
  </tr>
  <tr>
    <td>SST2</td>
    <td>67.3k</td>
    <td>872</td>
    <td>1.82k</td>
  </tr>
  <tr>
    <td>MRPC</td>
    <td>3.67k</td>
    <td>408</td>
    <td>1.73k</td>
  </tr>
  <tr>
    <td>STS-B</td>
    <td>5.75k</td>
    <td>1.5k</td>
    <td>1.38k</td>
  </tr>
  <tr>
    <td>QQP</td>
    <td>364k</td>
    <td>40.4k</td>
    <td>391k</td>
  </tr>
  <tr>
    <td>MNLI</td>
    <td>393k</td>
    <td>9.8k</td>
    <td>9.8k</td>
  </tr>
  <tr>
    <td>QNLI</td>
    <td>105k</td>
    <td>5.46k</td>
    <td>5.46k</td>
  </tr>
  <tr>
    <td>RTE</td>
    <td>2.49k</td>
    <td>277</td>
    <td>3k</td>
  </tr>
</table>

Table 15: Summary of Datasets for GLUE, Question Answering, and Text Summarization

Reviewer paper summaries: This paper investigates a special type of PEFT method called RoCoFT, which only tunes a few rows and columns of matrices in large language models to ensure memory efficiency. Different row/column-selection strategies, such as maximum importance (defined in line 943), minimum importance, and random, have been shown to exhibit similar performance in subtasks of GLUE (Figure 4).

Extensive experimental results demonstrate its superior performance and memory efficiency in some subtasks compared to many existing PEFT methods on
  * Natural language understanding tasks (GLUE) with encoder-only models (RoBERTa base and large)
  * Commonsense and mathematical reasoning tasks with decoder-only models (Bloom 7B, GPT-J 6B, LLaMA2-7B, and LLaMA2-13B).

Additional experiments on its neural tangent kernel approximations show that it produce kernels more close to full-parameter kernels than LoRA.

This paper presents RoCoFT, a novel parameter-efficient finetuning (PEFT) method that only tunes the selected rows and columns in large language models (LLMs), which further simplifies and reduces the memory footprint of LLM training. Experimental results on medium-sized language models like RoBERTa and DeBERTa, plus large models like LLaMA show that RoCoFT maintains LLMs' end-task performance while reducing the models' training memory footprint and time. Additionally, the authors use Neural Tangent Kernel (NTK) to explain RoCoFT's effectiveness, empirically showing that the kernels derived from the selective updates in RoCoFT is more close to full parameter finetuning compared to PEFT baselines.
Summary of strengths: The idea of the paper is easy to follow. Extensive experiments have also been conducted to compare its performance and memory efficiency with baselines, along with multiple aspects of the proposed method being inspected. Although row-based PEFT methods have emerged in other literature [1] and are no longer novel, this work seems to be the first to propose this method.

- The paper presents a simple yet effective method to improve the training efficiency of PEFT methods. The implementation is easy to be adapted and integrate to existing applications, and the results show that RoCoFT successfully maintains language models' end-task performance while reducing their training time and memory consumption compared to baseline methods.
- The authors leverages the NTK method to empirically explain the effectiveness of RoCoFT over existing baseline methods, demonstrating insightful results to the audience and inspires future research directions. 
- The thorough ablation studies and analysis show the generalizability and robustness of RoCoFT on many tasks, regardless of the specific choice of row/column selection strategies, rank-size, etc.
Summary of weaknesses: The major weakness of the paper is its 1) presentation problems in experimental results and 2) overclaim issues.

Concerning the overclaim issue, state-of-the-art claim (line 10) normally requires more verification in recent large language models, such as Qwen-2.5, Llama-3.2, Gemma-2 (and Gemma-3 if available), also more baselines such as PiSSA [2], LISA [3], Dora [4]. This would require substantial revisions of the paper and incur an unnecessary burden for the authors. Considering the papers' strength in thoroughly investigating the row/column-based PEFT method, I would suggest removing the state-of-the-art claim and shifting the paper's focus to the analysis of row/column-based PEFT methods. This way only minor revisions are needed.

Regarding the presentation of the experimental results, especially in Tables 1 & 2, the bold fonts in various subtask metrics demonstrate inconsistent trends of RoCoFT's improvements, leading to ambiguous conclusions and failing to support the state-of-the-art claim. Besides removing the state-of-the-art claim, I would suggest using bold fonts only for overall average performance in GLUE, commonsense reasoning, and mathematical reasoning. This way the conclusions of the Tables are clearer.

Also, in Table 1, some baselines have better performance with more trainable parameters (e.g. PROPETL), while some baselines have worse performance with fewer trainable parameters (e.g. VeRA), which is not a fair comparison. Performance comparison under the same level of trainable parameters is expected.

I would consider raising my score if the aforementioned concerns are addressed.

### References

[1]: Li, Cen-Jhih, and Aditya Bhaskara. "An Efficient Row-Based Sparse Fine-Tuning." arXiv preprint arXiv:2502.11439 (2025).

[2]: Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. "Pissa: Principal singular values and singular vectors adaptation of large language models." NeurIPS 2024.

[3]: Pan, Rui, et al. "LISA: layerwise importance sampling for memory-efficient large language model fine-tuning." NeurIPS 2024.

[4]: Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." ICML 2024.

- The specific row/column selection strategy that is used in the experiments is unclear. The authors only mention that the RoCoFT is robust to different row/column selection strategies but fail to mention any of them in Section 3. Moreover, the methods mentioned in the appendix, i.e., using the scores proposed in existing work (Wanda) could be suboptimal. Even with forward pass only, one could still get more information, such as the outliers in the activations, which is well-studied and considered to have a significant effect on LLM finetuning. 
-  The training time comparison of RoCoFT method to existing baselines is confusing. It is unclear what the number represents at the y-axis of Figure 2. Is it training time per step or epoch? Moreover, to fairly compare the training time of different methods, the model's convergence speed should be considered as well. Instead of comparing the training time per step/epoch, a better experiment would be either 1) compare the performance of the model after being trained with different methods, given the same training wall time, or 2) compare the training time consumption of different methods to train the model to a target performance threshold on certain tasks.
Comments, suggestions and typos: * [Clarity] Table 1, 2: better explicitly specify the meaning of "TTPs" in table captions.
* Some missing citations: [1][2][3].
* I am curious if there is a semantic or behavioral difference between row-based and column-based turning.
* Section 5, line 498: NTK normally assumes the inspected point to be close to the optimum. I am wondering if the authors have made efforts to ensure that.

### References

[1]: Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. "Pissa: Principal singular values and singular vectors adaptation of large language models." NeurIPS 2024.

[2]: Pan, Rui, et al. "LISA: layerwise importance sampling for memory-efficient large language model fine-tuning." NeurIPS 2024.

[3]: Liu, Shih-Yang, et al. "Dora: Weight-decomposed low-rank adaptation." ICML 2024.

- The abbreviation TTPs is not explained when it first appears on Table 1. Please mention the full name of it in the table's caption.
- The Performance Analysis section is too overwhelming and difficult to read because there are too many parentheses there delivering each tasks's full name and corresponding model performance. To address this issue, I suggest the authors:
     1. mention all the abbreviations of the tasks in the Datasets and Model Selection section, and only refer to the tasks via abbreviations;
     2. avoid repeating the numbers that already appear on the tables. Repeating them in the content of the experiment analysis is useless. 
- Please add a column on Table 1 and Table 2 to demonstrate the average performance across all tasks on each method for better comparison. Furthermore, please explain what bolded and underlined numbers mean in the caption.
Overall assessment scores: 2.5

3
Metareview score: 3.5
Metareview: The paper introduces RoCoFT, a fine-tuning method that updates only selected rows and columns of transformer weight matrices, achieving comparable performance to full fine-tuning while reducing memory and computation requirements. The approach is evaluated on both medium and large language models and supported by an NTK analysis to explain its effectiveness.
Summary of reasons to publish: - The approach is simple to implement and reduces resource usage while maintaining task performance, as demonstrated in extensive experiments across different models and datasets.

- The paper provides both empirical evidence via ablation studies and theoretical insights through NTK analysis, enhancing understanding of the method’s behavior.