# Analyzing LLAMA3 Performance on Classification Task Using LoRA and QLoRA Techniques
## 3 Experiments and Results
### LoRA vs. QLoRA Performance  
We compare the performances of LoRA when it adapted to all the layers or the attention layers to the performances of QLoRA at similar configuration settings. The value of \(\alpha\) is set at \(2\times r\), where \(r\) starts at one and doubles in consecutive iterations. As shown in Figure 7, for \((r=1)\), the performances of LoRA and QLoRA are higher when they adapt to all the linear layers rather than only to the attention layers.  
Figure 6: LLaMA3—attention-layer adaptation, \((r=1,2,4,8,16,32,64,128,256,\) and \(512)\).  
Similarly, for \((r=2)\), the performances of LoRA and QLoRA are greater when they adapt to all the linear layers rather than only the attention-layer adaptation. This signifies that the adaptation of all-linear layers for LoRA and QLoRA performs better for lower values of \(r\) than their attention-layer adaptation counterparts.  
In Figure 8, it can be observed that the performance of the QLoRA, when all the linear layers are adapted, is consistent for both \((r=4)\) and \((r=8)\). Meanwhile, the performance of the LoRA (for both adaptations) increases as the value of \(r\) goes up from 4 to 8. Also, if we consider both Figures 7 and 8, the performance of the LoRA has increased with an increase in the value of \(r\), whereas the performance of the QLoRA has pretty much remained steady or is unaffected by the change in \(r\).  
As seen in Figure 9, the performance of the LoRA starts remaining steady with an increase in the value of \(\tau\), \((r>8)\). Also, QLoRA's performance for all-linear-layer adaptation starts degrading as r exceeds the value of 8. Also, QLoRA with attention-layer adaptation performs better than QLoRA with all-layer adaptation as the \(r\) value exceeds 8, \((r>8)\).  
As shown in Figure 10, the performance of the LoRA when all the linear layers are adapted keeps dropping significantly as the value of r is increased to 64 and 128, \((r>32)\). Compared to the results obtained for \((r=16\) and \(r=32)\), QLoRA's performance starts improving as the r value exceeds 32, \((r>32)\), for both linear and attention-layer adaptations.  
Figure 8: LoRA and QLoRA with linear and attention layers \((r=4\) and \(r=8)\).  
Figure 7: LoRA and QLoRA with linear and attention layers \((r=1\) and \(r=2)\).  
Furthermore, in Figure 11, the performances of the LoRA for both adaptations (linear and attention layers) keep declining with an increase in value of r. This also holds true for QLoRA, where the performance degrades for both adaptations as r exceeds 128, (\(r>128\)).  
Overall, based on the above experiments, the performances of LoRA and QLoRA with all-linear-layer adaptation remain steady at around 67%. Also, LoRA and QLoRA with all-layer adaptations perform slightly better than their attention-layer adaptation counterparts for a value of \((r=8)\). Furthermore, QLoRA with attention-layer adaptation performs comparatively better than its LoRA counterpart for an \((r=4)\) value. As shown  
Figure 11: LoRA and QLoRA with linear and attention layers (\(r=256\) and \(r=512\)).  
Figure 10: LoRA and QLoRA with linear and attention layers (\(r=64\) and \(r=128\)).  
Figure 9: LoRA and QLoRA with linear and attention layers (\(r=16\) and \(r=32\)).  
in Figures 7-11, the performances of LoRA and QLoRA with attention-layer adaptations remain consistent at around 67% as long as the value of \(r\) is \(\leq\)256.