# Analyzing LLAMA3 Performance on Classification Task Using LoRA and QLoRA Techniques
### Background
#### 1.2.1 LoRA  
The Low-Rank Adaptation (LoRA) technique [14] decomposes the weight-change matrix, \(\Delta W\), into a lower-rank representation. It injects several trainable matrices (parameters) into each model layer while freezing the original pretrained parameters. As a result, it can approximate the full-supervised fine tuning in a more parameter-efficient manner. It involves freezing a pretrained weight matrix, \(W_{pretrained}\in R^{d\times k}\), and training only low-rank perturbations (\(\Delta W\)) to selected weight matrices (A and B), as shown in Equations (1) and (2). \[W_{\textit{fintured}}=W_{\textit{pretrained}}+\Delta W \tag{1}\]  
where \(W_{\textit{pretrained}}\) represents the pretrained weight matrix. \(W_{\textit{pretrained}}\) is frozen during training and does not receive gradient updates, while A and B contain trainable parameters. The decomposition of the \(\Delta W\) matrix means representing the \(\Delta W\) matrix using two smaller matrices, A and B, such that A has the same number of rows as \(\Delta W\), and B has the same number of columns as \(\Delta W\). \(\Delta W\) accumulates gradient updates during adaptation. As shown in Equation (2), the weight-change matrix (\(\Delta W\)) for the newly adapted task can be decomposed into a lower-dimensional space without losing too much information. \[\Delta W=AB,\ \ A\in R^{d\times r}\textit{ and }B\in R^{r\times k} \tag{2}\]  
The user chooses which \(W_{\textit{pretrained}}\) to adapt ('target modules') and the rank, \(r<<d,k\). By doing so, only \((d\times r+r\times k)\) parameters are trained instead of (d x k), which reduces the memory and number of FLOPS required for computing the gradient. For example, if \(\Delta W\) has 20,000 rows and 30,000 columns, it stores 600,000,000 parameters. But, if we choose A and B with \(r=8\), then A has twenty thousand rows and eight columns, and B has eight rows and thirty thousand columns; that is, 20,000 \(\times\) 8 + 8 \(\times\) 30,000 = 400,000 parameters, which is about 1500\(\times\) fewer than 600,000,000. The hyperparameter \(r\) represents several matrix-A columns and matrix-B rows. It is used to specify the rank of low-rank matrices (A and B) used for adaptation. A lower value of \(r\) results in fewer parameters to learn during adaptation, leading to faster training and reduced computational cost. The value of \(r\), therefore, controls how many parameters will be trained during fine tuning. We analyze the effects of \(r\) on the model's performance in Section 5. \(W_{\textit{pretrained}}\) and \(\Delta W\) are multiplied by the same input (x), and their respective output vectors are summed coordinate-wise. Additionally, \(\alpha/r\) is used as a scaling factor for \(\Delta Wx\). The forward pass of the LoRA is represented by Equation (3). \[ \begin{split} h&=W_{pretrained}x+(\alpha/r)\Delta W\ x\\ h&=W_{pretrained}x+(\alpha/r)AB\ x\end{split}  \tag{3}\]  
We perform experiments where the target modules (layers to be adapted during fine tuning) are set at the attention layers comprising \(W_{q},W_{k},W_{v}\), and \(W_{o}\) matrices or all-linear layers. \(W_{q},W_{k},W_{v}\), and \(W_{o}\) represent the self-attention module's query, key, value, and output projection matrices. Another hyperparameter is \(\alpha\), a scaling factor that can adjust the magnitude of the combined result, consisting of the original model output plus the low-rank adaptation. Using \(\alpha\), it becomes possible to balance the pretrained model's knowledge and the new task-specific adaptation. A higher \(\alpha\) value would emphasize low-rank matrices (AB) more, whereas a lower \(\alpha\) value would reduce their influences, making the model rely more on the original parameters (\(W\)). In other words, adjusting \(\alpha\) helps to strike a balance and prevent underfitting and/or overfitting by regularizing the model. We analyze the effects of \(\alpha\) on the model's performance in Section 5.