# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 2 Related Work  
**Large Language Models (LLMs)**[1, 2, 3, 4, 5, 6, 7] have become a leading paradigm in natural language processing, achieving top results in numerous tasks [20, 21] and forming the basis of chat systems [22]. To maximize the downstream performance, fine-tuning LLMs is essential. However, the size of LLMs has dramatically increased to enhance performance, exemplified by GPT-3, which boasts up to 175 billion parameters and requires approximately 350 GB of memory in FP16 format. This growth poses significant challenges for fine-tuning these models in a memory- and calculation-efficient manner.  
**Parameter-Efficient Fine-Tuning (PEFT)**[9, 10, 11, 12, 13] methods are meant to address this problem to reduce the number of trainable parameters by various methods while maintain  
Figure 1: Comparison between different fine-tuning strategies. (**a**) Full-Parameter Fine-Tuning, (**b**) LoRA, and (**c**) LoGE. By comparison, our architecture is more efficient by decomposing the weight during backpropagation.  
ing performance. PEFT typically involves adding adapter layers between the existing layers of a neural network. The pre-trained weights are frozen, and only the adapter layers are trained [9]. Prefix tuning [14] adds prefix parameters to the hidden states in every layer of the model. Prompt tuning [15] uses the template to reconstruct prompts and only updates the parameters related to prompt understanding. Low-Rank Adaptation (LoRA) [13] is a popular method that decomposes adapter weights into two low-rank matrices. These matrices are then fused with the pre-trained weights during inference. LoRA has demonstrated comparable performance to full-parameter fine-tuning while using much fewer parameters. AdaLoRA [16] utilizes SVD decomposition and prunes less significant singular values for more efficient updates. Another method uses orthogonal factorization to fine-tune diffusion models. LoRA-FA [17] keeps matrix A fixed with random initialization and only trains matrix B. Child Tuning [18] and SPT [19] only update a subset of parameters by strategically masking the gradient of the parameters not in the subset. Our method uses low-rank matrices to represent the subset of parameters in latent space, which allows for accelerating the calculation of backward computation without masking.  
**Quantization-Based PEFT.** Although PEFT reduces the trainable parameters in models, the pre-trained weights still require a large amount of memory. To address this challenge, QLoRA [23] compresses the pre-trained weights into 4-bit floats and integrates LoRA. Meanwhile, QA-LoRA [24] and EfficientDM [25] address the quantization challenges in LoRA-fused models. However, QLoRA and QALoRA introduce extra dequantization overhead in the calculation. In contrast, our approach eliminates the need for dequantization in the backward pass, thereby achieving significant speed improvements over these memory-efficient LoRA variants.  
**Low-Rank Structures in Deep Learning.** Low rank is widely used in machine learning, as many machine learning problems [26; 27; 28; 29] have certain intrinsic low-rank structures. Particularly in over-parameterized neural networks, it is observed that the networks tend to exhibit low-rank properties post-training [30]. Furthermore, several studies have explored low-rank updates to frozen models. For instance, ReLoRA [31] extends low-rank updates to encompass the entire training process of the original networks, and GaLore [32] employs low-rank decompositions of weight gradients to reduce memory demands during full training. To our knowledge, no existing work applies low-rank decomposition directly to the weights during training to reduce computational load. In our research, we decompose pre-trained weights to expedite gradient calculations.