## 4 Results
### Question Set One: Basic Arithmetic Problems  
Figure 2 shows the heatmap that reflects the distribution of scores across the fifty basic arithmetic questions; the green color represents correct answers while the red color shows incorrect answers. The heatmap captures how well the models perform in the questions that relate to simple addition, subtraction, multiplication, and division. ChatGPT and Gemini scored 28 questions apiece (56% accuracy), with PaLM scoring 8 questions (16% accuracy).  
Figure 3 shows the distribution of how the responses from the three models align with the model answers for the first question set.  
Figure 1: How the models answered a sample question showing the logic.  
Figure 2: Performance heatmap of the three models on the first question set. The red colors represent the incorrect answers while the green colors represent the correct answers.  
From the response distribution, there were seven questions that all the models answered correctly, sixteen questions that two of the models answered correctly, twelve questions that only one model answered correctly, and fifteen questions that none of the models answered correctly. ChatGPT and Gemini showed different but comparable capabilities, while PaLM seemed to have the most difficulty in answering the questions. For instance, there were seven questions that only Gemini answered right, five questions that only ChatGPT answered right, and one question that only PaLM answered right. We tagged the fifteen questions (30% of all questions) that none of the models could answer correctly as \"very hard\". These questions all had seemingly complex Yoruba numerals, showing that the models struggled when dealing with questions involving \"complex\" Yoruba numerical values, most of which were wrongly represented.  
We equally assessed the performance of the models based on their reasoning completeness, language understanding, and numerical representation accuracy. We checked whether there was a logical breakdown in the step-by-step computations provided by the models, assigning a score of 1 or 0 for each complete or incomplete reasoning step to assess the reasoning completeness metric. We assessed the language understanding metric in the same manner, checking if the model \"gets the gist\" of the question or exhibited misinterpretation or misrepresentation of the question's context. The numerical representation accuracy checked if the figure used by the model in its calculation was the right equivalence of the Yoruba numeral given in the question. Instead of awarding 1 or 0 for right or wrong in this case, we considered the three possibilities that might have happened and awarded corresponding scores to each. Correct representation of all numbers = 2, one or two misrepresentations = 1, and completely misrepresented the figures = 0. Since PaLM did not give the logic used to arrive at its answers, we only assessed ChatGPT and Gemini.  
Figure 3: The distribution of how the answers of the three models map to the model answer on the first question set.  
Table 3 provides a clear perspective of the underlying factors that influenced the accuracy of both models in terms of the number of questions answered correctly. While both have the same number of correctly answered questions (56%), there is a great difference in their reasoning completeness, language understanding, and numerical representation accuracy. Gemini outperformed ChatGPT in these three metrics, accounting for the 56% disagreement between the answers generated by both models. The numerical representation accuracy level of both models is, however, low, affecting their final computation.