# Efficient Fine-Tuning of Large Language Models via a Low-Rank Gradient Estimator
## 5 Experiment  
### Implementation Details  
We conducted experiments to evaluate the effectiveness of our proposed method and compared it with representative methods. **Datasets and Models:** We initially conducted experiments on small models including OPT-1.3B and OPT-2.7B [34], fine-tuning them on complete conversations. We used two distinct datasets: Self-Instruct [35] and Alpaca [36]. Following the benchmark in QLoRA [23], we also used the five-shot performance of the fine-tuned models on the Massively Multitask Language Understanding (MMLU) task [37]. To evaluate the efficiency and effectiveness of QLoGE, we scaled up our experiments to include larger models such as LLaMA2-7B and LLaMA2-13B [5]. We used two distinct datasets: OASST1 [38] and Alpaca [36]. The Self-Instruct dataset [35], generated by GPT-3, contains approximately 52K instructions. The OASST1 dataset [38] comprises human-created and annotated dialogues from ChatGPT-like assistants, featuring 161,443 messages in 35 languages and461,292 quality assessments. The Alpaca dataset [36] includes 52,000 instructions and demonstrations generated using OpenAI's text-davinci-003 engine. To evaluate LoGE with more complex tasks, we fine-tuned LLaMA-2-7B [5] on the MetaMathQA dataset [39] to assess their mathematical problem-solving capabilities on the GSM8K [40] validation sets. Additionally, the models were fine-tuned on the CodeFeedback dataset [41] and evaluated for coding proficiency using the Human-eval datasets [42]. Furthermore, training was extended to the WizardLM-Evol-Instruct dataset [43] with subsequent testing for conversational abilities on the MT-Bench dataset [44]. The GSM8K dataset [40] consists of 8.5K high-quality elementary school math problems, all authored by human writers. The MT-bench dataset [44] contains 3.3K expert-level pairwise human preferences for model responses generated by six models in response to 80 MT-bench questions. The Human-eval dataset [42] comprises 164 original programming problems, evaluating language comprehension, algorithms, and basic mathematics, some of which resemble simple software interview questions. These experiments were conducted using subsets containing 100K data points and were limited to one epoch of training to minimize overhead. To evaluate LoGE alongside other structured transformer-based large language models, we fine-tuned RoBERTa-large [2] on a subset of the GLUE benchmark [45], including MNLI, QNLI, SST-2, and MRPC. **Training Details:** For the OPT models trained on the Alpaca and Self-instruct datasets, we adhered mostly to the hyperparameters used in QLoRA. We employed the Adam optimizer with a learning rate of \(8\times 10^{-5}\). The moving average betas were set at [0.9, 0.999], and the weight decay was maintained at 0. And we used a constant learning rate scheduler. For LLaMA models trained on the Alpaca and Self-instruct datasets, we also adhered mostly to the hyperparameters used in QLoRA. We employed the Adam optimizer with a learning rate of \(2\times 10^{-4}\). The moving average betas were set at [0.9, 0.999], and the weight decay was maintained at 0. Additionally, we implemented a constant learning rate scheduler, incorporating a warmup ratio of 0.3. For fine-tuning LLMs with complex tasks, we used the Adam optimizer with a learning rate of \(2\times 10^{-5}\). The moving average betas were set at [0.9, 0.999], with no weight decay. Additionally, we utilized a cosine learning rate scheduler with a warmup ratio of 0.3. The hyperparameters for RoBerta-large were almost the same as the hyperparameters from LoRA [13], and is detailed in Table 2.  
For smaller models like RoBERTa-large [2] and OPT [34], we applied LoGE only to the q-proj and v-proj layers to preserve model performance. For larger models like LLaMA-7B, we extended the use of LoGE to all linear layers within the attention module, as larger models exhibit greater robustness to model compression. **Implement Details:** All experiments were conducted on RTX 3090 GPUs (NVIDIA, Santa Clara, CA, USA) using PyTorch (v2.2.0).