## 4 Results
### Overall Performance of the Three Models  
Table 4 presents a summary of the performance of the models based on their accuracy scores for the three tasks. The performance of the three models across the three question sets shows that none of the models achieved up to 50% accuracy. As seen in Figure 7, the best performance stood at 35.3% overall. We equally evaluated the performance of the models based on their reasoning completeness, language understanding, and numerical representation accuracy, especially for the first question sets, where these metrics could be easily assessed. These metrics point to two types of error--the first involves the misrepresentation of Yoruba numbers as figures, and the second is the misrepresentation of the problem context. Since the former is the most prominent of the two types of errors, we will provide examples of how numerals in some questions are misrepresented in Section 4.4. Analysis of Numerical Representation Errors  
Here, we provide three examples that show the question asked and the translation. We then present the expected representation of the numerals for each question alongside the numeral representations used by the models in their calculations in the following table. **Question 21**: **Yoruba:**: _Evo lo po jin nituial otalletigba at arinuletotaleltigba?_  
**Translation:**: Which is greater between 260 and 265? **Question 31**: **Yoruba:**: _Ramo eldiabo ra goro eliubo ogbon ni egbosan naira. Ti o bt da goro eliubo kookan ni adota naira, are abt adanti olo nt Ramo eldiubo je lor oja yit?_ \begin{table}
\begin{tabular}{l c c c} \hline \hline  & \multicolumn{3}{c}{**Accuracy**} \\ \hline
**Model** & **Basic Yorubá Arithmetic Set (\%)** & **Yorubá Time, Days and Months (\%)** & **Yorubá Numerals (\%)** \\ \hline ChaGPT & 56 & 44 & 8 \\ \hline Gemini & 56 & 32 & 0 \\ \hline PaLM & 16 & 8 & 0 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Overall performance of the models based on their accuracy. Figure 7: Overview of the overall performance of the three models across all question sets. **Translation:** [PERSON] the cassava-flour seller bought 30 drums of cassava flour for N1800. If she sells each cassava drum at N60, how much profit or loss did she make? _Question 49_  
**Yorubq:**_[PERSON], lyuwo re ati avon omo von meta fe lo wo [PERSON] nti ti ti n ser. Owo iwoole fun awon omode je eedgebjeo naira, ti avon agblalogbi si je egbata naira. Elo ni owo ivole lupapo?_  
**Translation:** [PERSON], his wife, and three kids want to watch the live performance of [PERSON]. The gate fee for kids is N1500, and that for adults is N6000. How much is the gate fee altogether? Tables 5-7 show the semantic gap in common LLMs for an LRL like Yorubq, as reflected by the expected numerals in each question and their misrepresentations by the models. Overall, ChatGPT was able to accurately represent numerals completely correctly only in 21 out of 50 (42%) questions and answer the main gist of question 37 out of 50 (74%) times. Gemini, on the other hand, represented the numerals completely correctly in only 23 questions (46%) and was able to correctly interpret the questions 46 times (92%). The error analysis of the two models is presented in Figure 8. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Numerals** & **Expected Representation** & **ChatGPT** & **Gemini** \\ \hline ogbon & 30 & 40 & 30 \\ egbèsam & 1800 & 9 & 900 \\ addota & 50 & 15 & 50 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Numerical misrepresentation error analysis, example 2. \begin{table}
\begin{tabular}{l l l l} \hline \hline
**Numerals** & **Expected Representation** & **ChatGPT** & **Gemini** \\ \hline eedgebjeo & 1500 & 800 & 250 \\ egbäata & 6000 & 3000 & 600 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Numerical misrepresentation error analysis, example 3. Figure 8: Overview of the error analysis of ChatGPT and Gemini across the 50 questions in set one.