# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 5 Experiment
### Ablation Studies
#### 5.3.3 Different Selection for LoGE  
To further evaluate the optimal rank \(r\) for LoGE and determine the best insertion points, we conducted two different sets of experiments. The first set involved varying the rank \(r\) to observe its influence on model performance and acceleration ratio. The second set applied LoGE at different points within the model to assess the limitations of the low-rank estimator in maintaining performance.  
**Rank \(r\) choice for LoGE.** We adjusted the rank \(r\) from 16 to 512 for the OPT-1.3B models using the Self-Instruct datasets in Table 9. Unlike typical Parameter-Efficient Fine-Tuning (PEFT) methods, in our approach, the rank \(r\) directly influences the accuracy of gradient estimation. Our results show that even using the same low rank as LoRA, such as \(r=16\), can lead to significant gradient estimation errors, although the training runtime remains comparable. We applied the Low-Rank Gradient Estimator (LoGE) specifically to \(\mathbf{W}_{q_{i}}\) and \(\mathbf{W}_{v_{i}}\). However, as \(r\) increased, the computational demands of LoGE surpassed those of LoRA, especially when \(r\) reached 512, at which point the computational complexity approached almost \(\mathbf{O}(d^{2})\). To strike an optimal balance between trainable parameters, training time, and performance, we selected \(r=d/16\) for our studies.  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline
**PPL \(\downarrow\)** & \(\mathbf{\alpha=0}\) & \(\mathbf{\alpha=0.1}\) & \(\mathbf{\alpha=0.3}\) & \(\mathbf{\alpha=1}\) & \(\mathbf{\alpha=3}\) & \(\mathbf{\alpha=10}\) & \(\mathbf{\alpha=\infty}\) \\ \hline Self-instruct & 4.18 & 4.15 & 4.15 & 4.18 & 4.16 & 4.19 & 4.69 \\ Alpaca & 7.53 & 7.55 & 7.52 & 7.56 & 7.56 & 7.54 & 7.63 \\ \hline \end{tabular}
\end{table}
Table 7: Perplexity of different \(\alpha\) with different datasets for OPT-1.3B. A value of 0 means using only the magnitude of the singular value, and \(\infty\) means using only the contribution to the gradient. The symbol \({}^{\omega}\)\(\downarrow\)” indicates that lower values are better.  
\begin{table}
\begin{tabular}{l c c c c} \hline
**Model** & **SVD (s)** & **Fine-Tuning (s)** & **Total (s)** & **Ratios** \\ \hline LLaMA-7B & 612 & 39,420 & 40,032 & 1.53\% \\ LLaMA-13B & 1420 & 44,100 & 45,520 & 3.11\% \\ \hline \end{tabular}
\end{table}
Table 8: Time cost of SVD and the total fine-tuning process.  
**Linear layer choice for LoGE.** We varied the weights within the model to which LoGE was applied. Specifically, we avoided applying LoGE to weights in the Feed-Forward Network (FFN) due to dimensional scaling. However, the number of trainable parameters changed depending on which weights LoGE was applied to. Applying LoGE to more layers resulted in more compressed gradients, but we did not adjust the rank \(r\) with the increase in layers. We list the results in Table 10. Our findings align closely with existing PEFT methods [10,13]. Applying LoGE only to the layers \(\textbf{W}_{q_{i}}\) and \(\textbf{W}_{k_{i}}\) significantly reduced performance. While the performance of \(\textbf{W}_{v_{i}}\) and \(\textbf{W}_{\alpha_{i}}\) was closer to that of \(\{\textbf{W}_{q_{i}},\textbf{W}_{v_{i}}\}\), the increase in trainable parameters from using LoGE still resulted in a larger performance gap than reported in [13]. This may be due to the difference in the number of trainable parameters between our methods. Additionally, making \(\textbf{W}_{v_{i}}\) trainable increased the training time by approximately 1.1 times, likely due to the involvement of gradient calculations in the attention mechanism. Our results suggest that although increasing the number of trainable parameters can enhance model performance, excessive compression of gradients can be detrimental. Fine-tuning with just two trainable parameters, \(\textbf{W}_{q_{i}}\) and \(\textbf{W}_{v_{i}}\), appears sufficient, indicating that a balance is crucial for optimal fine-tuning efficiency.