# Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
## 4 Experimentation
### Experimental Results
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Dimension** & **FAISS (\%)** & **BM25 (\%)** & **TF-IDF (\%)** \\ \hline Semantic Matching (BLEU, ROUGE-L) & BLEU 56.88, ROUGE-L 71.54 & BLEU 53.72, ROUGE-L 69.34 & BLEU 50.43, ROUGE-L 67.23 \\ Swahili-Specific Handling (Query Performance, Dialect Testing) & Query Performance 84.34, Dialect Testing 78.24 & Query Performance 80.12, Dialect Testing 74.89 & Query Performance 77.34, Dialect Testing 72.12 \\ Multi-Turn Coherence & 88.25 & 84.23 & 82.12 \\ Perplexity (Lower is Better) & 18.12 & 19.78 & 20.34 \\ Inference Time & 1.05 & 0.97 & 0.92 \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance of retrieval models in RAG for Swahili NLP. where speed is critical. However, this efficiency comes at the cost of lower diversity and higher redundancy. On the other hand, Hybrid Retrieval took the longest retrieval time at 120 milliseconds due to its more comprehensive search and filtering mechanisms. Despite this, its superior performance across diversity, redundancy, and query overlap metrics made it the most effective retrieval mechanism overall. These findings demonstrate the trade-offs between speed and retrieval quality across different models. While simpler models like TF-IDF are faster, more sophisticated approaches such as Hybrid Retrieval offer significant advantages in terms of content diversity and relevance, particularly for low-resource languages like Swahili. These insights emphasize the need to balance retrieval accuracy and efficiency based on application requirements. The results presented in Table 8 show the performance of fine-tuned models and RAG-enhanced models across three key Swahili NLP tasks: translation, summarization, and question answering. These tasks are vital for building effective Swahili conversational systems, and the integration of retrieval mechanisms like RAG plays a significant role in improving model performance. For Translation Accuracy, which measures how well each model translates Swahili text while preserving meaning and context, mT5 outperformed the other models. The fine-tuned mT5 achieved a translation accuracy of 83.45%, but when enhanced with RAG, this increased to 88.34%, demonstrating how retrieval can improve the translation process by providing additional context. In contrast, GPT-Neo performed the weakest in both the fine-tuned (72.89%) and RAG-enhanced (76.45%) versions, indicating its limitations in handling Swahili translation tasks. In Summarization Coherence, which evaluates how logically and fluently the models generate summaries, mT5 again leads. The fine-tuned version of mT5 achieved 82.12% coherence, but this improved to 87.12% with RAG, showing the benefits of retrieval in producing contextually relevant and coherent \begin{table}
\begin{tabular}{l c c c c} \hline \hline
**Model + Retrieval** & **Retrieval Accuracy (\%)** & **Response Fluency (\%)** & **Relevance (\%)** & **Efficiency (s)** \\ \hline mT5 + FAISS & 88.00 & 87.34 & 86.45 & 1.05 \\ mT5 + BM25 & 85.67 & 84.12 & 83.45 & 1.10 \\ mT5 + TF-IDF & 82.45 & 80.34 & 81.78 & 0.95 \\ mT5 + Hybrid Retrieval & 88.00 & 88.00 & 88.00 & 1.20 \\ \hline GPT-2 + FAISS & 77.45 & 75.34 & 76.12 & 1.10 \\ GPT-2 + BM25 & 73.56 & 71.34 & 72.89 & 1.14 \\ GPT-2 + TF-IDF & 68.34 & 66.23 & 67.45 & 0.92 \\ GPT-2 + Hybrid Retrieval & 79.45 & 77.78 & 78.45 & 1.18 \\ \hline mBART + FAISS & 82.12 & 80.67 & 81.45 & 1.08 \\ mBART + BM25 & 78.45 & 76.34 & 77.12 & 1.14 \\ mBART + TF-IDF & 74.34 & 72.89 & 73.45 & 0.98 \\ mBART + Hybrid Retrieval & 85.67 & 84.45 & 83.78 & 1.25 \\ \hline GPT-Neo + FAISS & 79.45 & 77.34 & 78.12 & 1.10 \\ GPT-Neo + BM25 & 75.67 & 73.12 & 74.45 & 1.16 \\ GPT-Neo + TF-IDF & 72.34 & 70.45 & 71.89 & 0.96 \\ GPT-Neo + Hybrid Retrieval & 83.45 & 81.78 & 82.12 & 1.22 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparison of retrieval mechanisms across models for Swahili NLP.