# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 5 Experiment
### Ablation Studies
#### 5.3.1 The Impact of Different \(\alpha\) and Different Datasets on Model Performance  
We employed a temperature parameter, \(\alpha\), to combine two normalized importance scores: the magnitude of the singular values and the contribution to the gradient. To explore how different values of \(\alpha\) (ranging from 0.1 to 10) affect model performance, we conducted tests across various datasets using the OPT-1.3B model in Table 7. Our findings reveal  
\begin{table}
\begin{tabular}{l l l l l l l} \hline \hline
**Model** & **Dataset** & **Method** & **Rank** & **Memory** & **Metric (Acc. \(\uparrow\))** & **Runtime** \\ \hline LLaMA-7B & OASST1 & QLoRA & 64 & 7836 MiB & 33.9\% & 4.33 h \\ LLaMA-7B & OASST1 & QLoGE & 256 & 9390 MiB & 34.0\% & 3.41 h \\ LLaMA-7B & Alpaca & QLoRA & 64 & 7836 MiB & 38.4\% & 22.72 h \\ LLaMA-7B & Alpaca & QLoGE & 256 & 9462 MiB & 38.6\% & 17.67 h \\ \hline LLaMA-13B & OASST1 & QLoRA & 64 & 13,258 MiB & 45.5\% & 7.06 h \\ LLaMA-13B & OASST1 & QLoGE & 320 & 14,542 MiB & 45.1\% & 5.87 h \\ LLaMA-13B & Alpaca & QLoRA & 64 & 13,986 MiB & 47.8\% & 37.65 h \\ LLaMA-13B & Alpaca & QLoGE & 320 & 14,604 MiB & 47.6\% & 31.34 h \\ \hline \hline \end{tabular}
\end{table}
Table 6: Five-shot MMLU benchmark accuracy across LLaMA models: comparing sizes and datasets. The symbol “\(\uparrow\)” indicates that upper values are better.  
\begin{table}
\begin{tabular}{l l c c c c} \hline \hline
**Dataset** & **Method** & **Rank** & **Memory** & **Acc. \(\uparrow\)** & **Runtime** \\ \hline \multirow{3}{*}{MNLI} & FT & - & 10,710 MiB & 90.2\% & 12.25 h \\  & LoRA & 8 & 4636 MiB & 90.6\% & 8.54 h \\  & LoGE & 256 & 5052 MiB & 89.9\% & 7.96 h \\ \hline \multirow{3}{*}{SST2} & FT & - & 9444 MiB & 96.4\% & 1.25 h \\  & LoRA & 8 & 4118 MiB & 96.2\% & 1.22 h \\  & LoGE & 256 & 4486 MiB & 95.7\% & 1.37 h \\ \hline \multirow{3}{*}{QNLI} & FT & - & 16,488 MiB & 94.7\% & 3.25 h \\  & LoRA & 8 & 7564 MiB & 94.9\% & 2.48 h \\  & LoGE & 256 & 8566 MiB & 94.3\% & 2.21 h \\ \hline \multirow{3}{*}{MRPC} & FT & - & 10,194 MiB & 90.9\% & 0.22 h \\  & LoRA & 8 & 5686 MiB & 90.1\% & 0.16 h \\ \cline{1-1}  & LoGE & 256 & 6032 MiB & 91.1\% & 0.17 h \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of fine-tuning RoBerta-large using Full-FT, LoRA, and LoGE on a subset of GLUE. Experiments were run with one RTX3090. The symbol “\(\uparrow\)” indicates that upper values are better.  
that models perform optimally when initialized with the top-K largest singular values. This supports findings from [PERSON] et al. [13], which noted a significant overlap with the top singular vectors in LoRA. Interestingly, relying solely on gradient contribution for subspace selection resulted in suboptimal performance, diverging from the results observed in Child Tuning [18] and SPT [19]. This could be due to our method's reliance on low-rank matrices to estimate input gradients, potentially omitting subspaces with the largest singular values and thus introducing significant estimation errors. Remarkably, we found \(\alpha=0.3\) to be the most effective setting for both datasets tested. This suggests that the fusion method and this particular hyperparameter setting are generally applicable, not just dataset-specific.