## 4 Results
### Quantitative Analysis of Baseline and Fine-Tuned Model Performance  
The baseline model, based on the pretrained mT5-small, provided a reference for machine translation capabilities without fine-tuning the target language, Twi. The results (shown in Table 2) reveal that the pretrained mT5 model, while able to generate translations, exhibited suboptimal performance, especially for low-resource languages like Twi. The translation quality and fluency were noticeably weaker, highlighting the limitations of using a general multilingual pretrained model without adaptation to the target language.  
Following fine-tuning of the English-Twi model on our parallel corpus, significant improvements were observed across all evaluation metrics (Figure 3). The fine-tuned model outperformed the baseline, demonstrating the effectiveness of transfer learning in enhancing performance for low-resource languages. Fine-tuning allowed the model to better capture the linguistic nuances of Twi, resulting in more accurate and fluent translations. Specifically, there were notable improvements in both SpBLEU and ROUGE scores, reflecting enhanced n-gram precision and overall translation quality. Additionally, the WER score was substantially reduced, indicating better word-level alignment and fewer errors in the predicted translations.  
The fine-tuned English-Twi model shows significant improvements across all evaluation metrics. The SpBLEU score increased from 2.16% to 71.30%, demonstrating much better sentence-level alignment with reference translations. ROUGE-1 (unigram recall) improved by 50.01% (from 15.23% to 65.24%), and ROUGE-2 (bigram recall) saw a 53.04% improvement (from 7.18% to 60.22%), reflecting enhanced word and phrase matching. The ROUGE-L score, which evaluates sentence fluency, improved by 47.90%, from 14.22% to  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Metrics** & **Baseline (\%)** & **Fine-Tuned Model (\%)** & **Improvement (\%)** \\ \hline SpBLEU & 2.16 & 71.30 & 69.14 \\ ROUGE-1 & 15.23 & 65.24 & 50.01 \\ ROUGE-2 & 7.18 & 60.22 & 53.04 \\ ROUGE-L & 14.22 & 62.12 & 47.90 \\ WER (\%) & 183.16 & 68.32 & 114.84 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Performance comparison between pretrained mT5 and fine-tuned English-Twi model.  
Figure 3: Evaluation metrics heatmap.  
62.12%, indicating better sentence structure. Lastly, WER dropped significantly by 114.84% (from 183.16% to 68.32%), showing improved alignment and accuracy in the translation. These improvements highlight the effectiveness of fine-tuning for low-resource languages like Twi, resulting in better translation fluency and accuracy. A heat map illustrating these improvements is included in Figure 3.  
The reported 183.16% WER for the baseline model reflects unnormalized edit distance calculations for the full test set. This occurs when the system produces extremely poor translations requiring more edits than the reference length. We used the standard WER formula:  
\[WER=\frac{S+D+1}{N}\times 100\%\]  
where S = substitutions, \(D\) = deletions, \(I\) = insertions, and \(N\) = reference words. An example calculation from the test set is as follows:  
Reference: \"Me p\(\epsilon\) abobracase\" (4 words);  
Baseline output: \"\(\infty\)no \(\infty\)ko\(\beta\)fe kotenaa adwuma\" (6 words);  
Edit operations: 4 substitutions + 2 insertions.  
\[WER=\frac{4+0+2}{4}\times 100\%=150\%\]  
We normalized by reference length variability using the following formula:  
\[\text{nWER}=\frac{WER}{\sqrt{\frac{1}{n}\sum_{i=1}^{n}(y_{i}-\overline{y})^{2}}}\]  
where y\({}_{1}\) is the reference length for sample i, and \(\overline{y}\) is the average reference length across all samples. Table 3 provides both raw and length-normalized scores for further clarity:  
We also conducted paired permutation tests (10,000 iterations) comparing model variants as shown in Table 4.