# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 7 Conclusions  
In this paper, we propose the Low-Rank Gradient Estimator (LoGE), an efficient method for fine-tuning LLMs. LoGE incorporates model compression into low-rank weight update approximations, performing low-rank approximations of pre-trained weights during backward computation while preserving the full-rank pre-trained weights during forward computation. This approach reduces the computational cost of training a linear layer  
\begin{table}
\begin{tabular}{l c c c c c c c} \hline \hline
**Weight Type** & \(\textbf{W}_{q_{i}}\) & \(\textbf{W}_{k_{i}}\) & \(\textbf{W}_{v_{i}}\) & \(\textbf{W}_{\alpha_{i}}\) & \(\textbf{W}_{q_{i}r}\textbf{W}_{k_{i}}\) & \(\textbf{W}_{q_{i}r}\textbf{W}_{v_{i}}\) & \(\textbf{W}_{q_{i}r}\textbf{W}_{v_{i}r}\textbf{W}_{\alpha_{i}}\) \\ \hline PPL \(\downarrow\) & 5.44 & 5.54 & 4.36 & 4.44 & 4.83 & 4.15 & 4.21 \\ Runtime (h) & 1.42 & 1.42 & 1.52 & 1.44 & 1.52 & 1.60 & 1.95 \\ \hline \hline \end{tabular}
\end{table}
Table 10: PPL on Self-Instruct for OPT-1.3B models after applying LoGE to different linear layers. The symbol “\(\downarrow\)” indicates that lower values are better.  
\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline \(r\) & **16** & **32** & **64** & **128** & **256** & **512** \\ \hline PPL \(\downarrow\) & 6.09 & 5.07 & 4.57 & 4.15 & 3.82 & 3.71 \\ Runtime (h) & 1.59 & 1.57 & 1.60 & 1.60 & 1.62 & 1.72 \\ \hline \hline \end{tabular}
\end{table}
Table 9: PPL on Self-Instruct for OPT-1.3B models after applying different \(r\) values of LoGE. The symbol “\(\downarrow\)” indicates that lower values are better.  
by approximately 60%. Experiments across various models and tasks demonstrated that LoGE not only maintained performance but also significantly accelerated the fine-tuning process by about 20%. Additionally, LoGE can be combined with quantization methods, reducing the training time for quantization-based efficient fine-tuning by approximately 30% without any performance degradation. LoGE advances the field of fine-tuning on edge devices and holds significant practical implications for reducing the carbon footprint of deep learning.  
The following supporting information can be downloaded at: [[https://www.mdpi.com/article/10.3390/app15010082/s1](https://www.mdpi.com/article/10.3390/app15010082/s1)]([https://www.mdpi.com/article/10.3390/app15010082/s1](https://www.mdpi.com/article/10.3390/app15010082/s1)).  
Conceptualization, L.Z.; methodology, L.Z.; software, L.Z.; validation, L.Z. and Z.L.; formal analysis, L.Z.; investigation, L.Z.; resources, H.Z.; data curation, L.Z.; writing--original draft preparation, L.Z.; writing--review and editing, C.Y.; visualization, Y.Y.; supervision, Z.L.; project administration, H.Z.; funding acquisition, H.Z. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the National Key Research and Development Program of China (2022 YFC3602601). And this work was also supported by the Experimental Technology Reaseach Project of Zhejiang University (SYBJS202314).  
Not applicable.  
The raw data supporting the conclusions of this article will be made available by the authors on request. And a portotype project of LoGE is provided in Supplementary Materials.  
All authors were employed by the company Hangzhou Yihe Hui Sheng. The remaining authors declare that the re-search was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.  
The following abbreviations are used in this manuscript:  
LofELow-Rank Gradient Estimator  
LoRALow-Rank Adapter  
PegTParameter-Efficient Fine-Tuning  
LLMLarge Language Model