# Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
## 4 Experimentation
### Experimental Results  
This section presents the experimental results comparing the performance of fine-tuned models and RAG-enhanced models across multiple key metrics, including BLEU score, METEOR score, Query Performance, Multi-Turn Coherence, inference time, ROUGE-L score, Swahili Dialect Testing, and perplexity. The results highlight the improvements achieved by integrating retrieval mechanisms with generation techniques in low-resource Swahili NLP tasks. Table 4 provides a detailed comparison [48]. To assess the statistical significance of the observed improvements, paired \(t\)-tests were conducted to compare the performance metrics of fine-tuned models and RAG-enhanced models. The results confirm that the improvements in BLEU, METEOR, ROUGE-L, Query Performance, and Multi-Turn Coherence scores are statistically significant with \(p<0.01\). This statistical validation underscores the robustness of the enhancements achieved by integrating retrieval mechanisms. The results in Table 4 demonstrate that the RAG-enhanced models consistently outperform their fine-tuned counterparts across most metrics. For example, mT5 with RAG achieved a BLEU score of 56.88% and a METEOR score of 72.72%, reflecting superior translation accuracy and fluency. Similarly, the Query Performance metric shows significant improvements, with RAG-enhanced models achieving up to 84.34%, compared to 68.12% for the fine-tuned models. The ROUGE-L score and Swahili Dialect Testing results further highlight the advantages of integrating retrieval, as these metrics measure contextual alignment and handling of linguistic variations, respectively [49]. The perplexity values, which indicate the confidence of the model in generating responses, are notably lower for RAG-enhanced models, showcasing improved fluency and reduced uncertainty in responses. \begin{table}
\begin{tabular}{l c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**mT5**} & \multicolumn{2}{c}{**GPT-2**} & \multicolumn{2}{c}{**mBART**} & \multicolumn{2}{c}{**GPT-Neo**} \\ \cline{2-9}
**Metric** & **Fine-Tune** & **RAG** & **Fine-Tune** & **RAG** & **Fine-Tune** & **RAG** & **Fine-Tune** & **RAG** \\ \hline BLEU Score (\%) & 45.34 & 56.88 & 44.12 & 50.43 & 47.24 & 52.36 & 43.76 & 49.87 \\ METEOR Score (\%) & 62.12 & 72.72 & 59.48 & 66.13 & 63.87 & 68.54 & 60.32 & 65.98 \\ ROUGE-L Score (\%) & 66.21 & 81.54 & 70.67 & 76.43 & 51.42 & 63.89 & 53.34 & 61.28 \\ Query Performance (\%) & 68.12 & 84.34 & 65.62 & 76.51 & 70.14 & 78.93 & 67.15 & 75.98 \\ Multi-Turn Coherence (\%) & 75.43 & 88.25 & 72.38 & 81.23 & 78.12 & 82.46 & 73.24 & 80.98 \\ Swahili Dialect Testing (\%) & 64.31 & 78.24 & 61.54 & 71.42 & 66.12 & 74.38 & 62.87 & 70.45 \\ Perplexity (\%) & 24.32 & 18.12 & 26.45 & 20.34 & 22.87 & 19.54 & 25.12 & 21.45 \\ Inference Time (s) & 0.93 & 1.05 & 0.85 & 0.97 & 1.01 & 1.12 & 0.78 & 0.89 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison of metrics between fine-tuned and RAG models for Swahili NLP. \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Experiment Component** & **Parameter** & **Value** & **Description** & **Justification** \\ \hline \multirow{4}{*}{Retriever Mechanism} & \multirow{2}{*}{Backend} & \multirow{2}{*}{FAISS} & Vector similarity search & Enables efficient retrieval \\  & & & for retrieval & from large datasets \\ \cline{4-5}  & & & Metric for measuring & Suitable for high- \\ \cline{1-1} \cline{3-5}  & & & semantic relevance & dimensional embeddings \\ \hline \multirow{4}{*}{Optimization} & \multirow{2}{*}{Loss Function} & \multirow{2}{*}{Adam Optimizer} & Adaptive learning rate & Stabilizes \\  & & & algorithm & gradient updates \\ \cline{4-5}  & & & Measures error between & \\ \cline{4-5}  & & & predictions and & \\ \cline{4-5}  & & & ground-truth responses & \\ \hline \multirow{2}{*}{Frameworks} & \multirow{2}{*}{Libraries} & TensorFlow 2.x, Hugging & \multirow{2}{*}{\
\begin{tabular}{} \end{tabular} } & \multirow{2}{*}{
\begin{tabular}{} \end{tabular} } \\  & & Face Transformers & & evaluation & \\ \hline \hline \end{tabular}
\end{table}