# Analyzing LLAMA3 Performance on Classification Task Using LoRA and QLoRA Techniques
## 2 Methodology
### Comparative Analysis of LoRA and QLoRA Techniques
#### 2.6.2 QLoRA  
QLoRA stands for quantized model weights with low-rank adaptation. Fine tuning using QLoRA is more memory efficient than LoRA. As shown in Table 3, the pretrained model in QLoRA is loaded into the GPU as 4-bit NormalFloat (NF4) data, which is a novel high-precision technique to quantize a pretrained model to 4-bit data.  
NF4 is optimal for normally distributed weights and yields better empirical results than 4-bit integers and 4-bit floats. QLoRA backpropagates gradients through the frozen 4-bit quantized pretrained language model to LoRA, where a small set of learnable low-rank adapter weights is then added. QLoRA takes LoRA a step further by quantizing the LoRA adapters' weights (smaller matrices) to a lower precision level (e.g., 4-bit instead of 8-bit). This further reduces the memory footprint and storage requirements.  
QLoRA is the better choice because of its even greater memory efficiency, but LoRA may be preferable because of its slightly faster training times. LoRA focuses on reducing the number of parameters that need to be fine tuned. At the same time, QLoRA takes it a step further by combining low-rank adaptation with quantization to lower memory requirements. Together, these methods make fine-tuning large models more accessible. QLoRA generally takes more time than LoRA during fine tuning because of the additional computational overhead involved in the quantization process, where model weights are converted to lower precision (like 4-bit) levels to reduce memory usage significantly.  
As shown in Table 3, the LLAMA3-8B model using QLoRA's NF4 data type requires only 3.857 GB of memory during fine tuning, where the memory requirement for loading the model itself took around 3.73 GB of space. Additionally, the space occupied by trainable parameters comes to 0.0097 GB, followed by 0.078 GB for gradient and 0.039 GB for optimizer states. As shown in Table 4, we used different values of \(r\) while conducting the experiments. We list the trainable parameters associated with each value of \(r\) for the LLaMA3 model, which has 8 billion parameters. The number of trainable parameters remains the same at LoRA and QLoRA settings.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**r** & **Linear-Layer Parameters** & **Attention-Layer Parameters** \\ \hline
8 & 21,024,808 (0.2794\%) & 6,836,224 (0.0910\%) \\ \hline
16 & 42,029,136 (0.5569\%) & 13,651,968 (0.1816\%) \\ \hline
32 & 84,037,792 (1.1073\%) & 27,283,456 (0.3622\%) \\ \hline
64 & 168,055,104 (2.1901\%) & 54,546,432 (0.7216\%) \\ \hline
128 & 336,089,728 (4.2860\%) & 109,072,384 (1.4325\%) \\ \hline
256 & 672,158,976 (8.2190\%) & 218,124,288 (2.8243\%) \\ \hline
512 & 1,344,297,472 (15.1875\%) & 436,228,096 (5.4932\%) \\ \hline \hline \end{tabular}
\end{table}
Table 4: LLaMA3-8B’s trainable parameters.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Type** & **Model Weight (Inferences)** & **Trainable Parameters** & **Gradients** & **AdamW** & **Memory for Fine** \\ \hline LoRA (FP32) & 29.80 GB (8 Billion \(\times\) 4) & 21,024,808 \(\times\) 4 bytes & 21,024,808 \(\times\) 4 bytes & 21,024,808 \(\times\) 8 bytes & 30.113 GB \\ \hline LoRA (FP16) & 14.9 GB (8 Billion \(\times\) 2) & 21,024,808 \(\times\) 2 bytes & 21,024,808 \(\times\) 4 bytes & 21,024,808 \(\times\) 4 bytes & 15.095 GB \\ \hline QLoRA (NF4) & 3.73 GB (8 Billion \(\times\) 0.5) & 21,024,808 \(\times\) 0.5 bytes & 21,024,808 \(\times\) 4 bytes & 21,024,808 \(\times\) 2 bytes & 3.857 GB \\ \hline \hline \end{tabular}
\end{table}
Table 3: Memory requirements for LoRA and QLoRA during fine tuning and inferences (\(\tau=8\)).