# Enhanced Retrieval-Augmented Generation Using Low-Rank Adaptation
## 4 Experiments
### Model Comparison  
We first examine the impact of LoRA on model size, parameter requirements, and training efficiency. LoRA was applied to the query (\(W_{Q}\)) and value (\(W_{V}\)) projection matrices in the self-attention modules of the upper transformer layers. The following hyperparameters were used for fine-tuning: rank \(r=16\) and scaling factor \(\alpha=32\). This section presents an empirical evaluation of various LoRA configurations applied to a pre-trained BERT model on a sentence pair classification task. The objective of the experiments is to identify an optimal configuration that balances performance and parameter efficiency by varying the rank \(r\) and the scaling factor \(\alpha\) in the LoRA modules. The base model used in the experiments is BERT-based uncased, fine-tuned using LoRA to classify whether a question-answer pair is semantically coherent. The dataset consists of sentence pairs labeled as either valid or invalid, and performance is evaluated in terms of accuracy, loss, training time, and the number of trainable parameters relative to the full model size. The results, as shown in Table 1, indicate that the configuration with \(r=8\) and \(\alpha=16\) performed poorly, achieving only 49.56% accuracy while updating just 0.27% of the model parameters. This suggests that such a low-rank adaptation was insufficient to effectively capture task-relevant information. When the rank was increased to \(r=16\) and the scaling factor to \(\alpha=32\), the model performance improved dramatically, reaching 95.83% accuracy while training only 0.54% of \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \(r\) & \(\alpha\) & **Accuracy (\%)** & **Loss** & **1 Epoch Time** & **Trainable** & **Train** \\  & & & & **(min)** & **Parameter** & **Ratio** \\ \hline
8 & 16 & 49.56 & 0.505 & 1.68 & 294,912 & 0.27\% \\
16 & 32 & 95.83 & 0.230 & 1.69 & 589,824 & 0.54\% \\
32 & 64 & 98.14 & 0.301 & 1.72 & 1,179,648 & 1.07\% \\
64 & 128 & 98.23 & 0.235 & 1.74 & 2,359,296 & 2.11\% \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of LoRA configurations on model performance and efficiency. the total parameters. This configuration demonstrated a strong balance between parameter efficiency and predictive performance. Further increases in the rank and scaling factor, such as \(r=32\) with \(\alpha=64\) and \(r=64\) with \(\alpha=128\), led to only marginal accuracy improvements--98.14% and 98.23%, respectively. However, these improvements came at the cost of significantly higher parameter counts, up to 2.11% of the full model, thereby diminishing the benefits of parameter efficiency. Based on these results, we selected the configuration with \(r=16\) and \(\alpha=32\) as the optimal setting. It achieved a high level of accuracy with minimal additional training overhead, making it a practical and efficient choice for low-resource fine-tuning scenarios. This finding underscores the effectiveness of LoRA in enabling scalable adaptation while preserving model quality. To examine the effect of MMR on response generation quality, we conducted an ablation study by varying the MMR trade-off coefficient \(\lambda\) in the range \([0.1,0.9]\). This hyperparameter controls the balance between relevance and diversity during document selection. We evaluated the generated answers using BLEU-4, CIDEr, and SPICE metrics to measure lexical overlap, informativeness, and semantic structure quality, respectively, as summarized in Table 2.  
As shown in Table 2, a low \(\lambda=0.1\) leads to significantly degraded performance across all metrics, suggesting that insufficient attention to query relevance results in poor document selection. As \(\lambda\) increases, the BLEU-4 score rises consistently, peaking at \(\lambda=0.7\) with a score of 0.9456. CIDEr and SPICE also continue to improve with higher \(\lambda\), with the highest scores observed at \(\lambda=0.9\).