# Leveraging Retrieval-Augmented Generation for Swahili Language Conversation Systems
## 4 Experimentation
### Experimental Results

Table 3: _Cont._The choice of metrics was driven by their relevance to evaluating text generation quality, contextual relevance, and linguistic diversity in Swahili NLP. BLEU, METEOR, and ROUGE-L were selected for their ability to measure lexical overlap, semantic alignment, and contextual coherence, respectively. BLEU evaluates precision by calculating n-gram overlap between generated and reference texts, making it useful for assessing fluency. METEOR incorporates synonym matching and paraphrasing, capturing linguistic variations beyond exact matches. ROUGE-L focuses on sequence alignment, measuring the overlap of the longest common subsequences between the generated and reference texts. These three metrics collectively address the generative quality of the models. Additionally, Query Performance and Multi-Turn Coherence metrics were introduced to evaluate conversational relevance and the model's ability to maintain context over multiple exchanges. Swahili Dialect Testing specifically measures the adaptability of the model across various dialects, ensuring linguistic inclusivity. Perplexity was included to quantify the model's uncertainty in generating responses, with lower values indicating higher confidence and fluency. Metrics like F1 score and accuracy were not used because they are primarily suited for classification or prediction tasks, where exact matches are critical. Similarly, semantic similarity measures, while relevant for retrieval tasks, do not directly evaluate the quality of generative outputs. The selected metrics collectively provide a comprehensive evaluation framework tailored to the needs of conversational Swahili NLP. Inference time, reported in seconds, shows a slight increase for RAG-enhanced models due to the additional retrieval step. However, this trade-off is justified by the substantial gains in accuracy and relevance, as seen in the BLEU and Multi-Turn Coherence scores. To address this, future optimization strategies include caching frequently accessed query embeddings, reducing retrieval latency through approximate nearest neighbor search, and leveraging more efficient embedding models to balance accuracy and speed. The inclusion of statistical tests adds further rigor to the analysis, validating that these improvements are not coincidental but rather a consistent outcome of the RAG framework. These results underscore the effectiveness of the RAG framework in addressing the challenges of low-resource languages like Swahili and highlight its potential for other underrepresented languages. The results in Table 5 highlight the performance differences among retrieval models when integrated into the RAG framework for Swahili NLP. Each dimension of the table provides insights into the strengths and weaknesses of the retrieval mechanisms. Semantic matching shows that FAISS achieved the highest BLEU score of 56.88% and ROUGE-L score of 71.54%, indicating its superior ability to understand and generate semantically aligned responses. BM25 performed moderately with BLEU and ROUGE-L scores of 53.72% and 69.34%, while TF-IDF achieved lower scores of 50.43% and 67.23%. For Swahili-specific handling, FAISS excelled with Query Performance at 84.34% and Dialect Testing at 78.24%, demonstrating its strength in addressing Swahili linguistic nuances. BM25 achieved 80.12% for Query Performance and 74.89% for Dialect Testing, while TF-IDF scored lower at 77.34% and 72.12%. Multi-Turn Coherence was highest with FAISS at 88.25%, reflecting its ability to maintain conversational context effectively. BM25 followed with 84.23%, and TF-IDF scored 82.12%. In terms of computational efficiency, FAISS had the lowest perplexity score of 18.12%, indicating confidence in generating coherent responses. However, its inference time was slightly higher at 1.05 s compared to BM25 at 0.97 s and TF-IDF at 0.92 s. These results demonstrate the trade-offs between semantic richness and computational speed.