BEnQA: A Question Answering Benchmark for Bengali and English (2024)

Abstract: In this study, we introduce BEnQA1, a dataset comprising parallel Bengali and English exam questions for middle and high school levels in Bangladesh. Our dataset consists of approximately 5K questions covering several subjects in science with different types of questions, including factual, application, and reasoningbased questions. We benchmark several Large Language Models (LLMs) with our parallel dataset and observe a notable performance disparity between the models in Bengali and English. We also investigate some prompting methods, and find that Chain-of-Thought prompting is beneficial mostly on reasoning questions, but not so much on factual ones. We also find that appending English translation helps to answer questions in Bengali. Our findings point to promising future research directions for improving the performance of LLMs in Bengali and more generally in low-resource languages.

heading: 1 Introduction
text: Large language models (LLMs) like GPT-4 have shown impressive performance in many complex natural language processing tasks including reasoning and question answering, all of which have been subject to intense research in recent years (Bang et al., 2023; Liu et al., 2023). However, most of this research has focused on English and other highresource languages, often neglecting medium- to low-resource languages. To make matters more difficult, most of the benchmarks designed for measuring progress in LLM research are only available in English and a handful of other languages, making it very hard to compare different LLM skills in low-resource languages.
This issue is particularly critical in light of the growing use of LLMs like ChatGPT in the context
1We release the dataset at: https://github.com/ sheikhshafayat/BEnQA
of education (Khan Academy, 2023). The absence of LLMs that perform equally well in non-English languages would lead to further divide in access to education and technology, highlighting a critical need for more inclusive language model development.
Addressing this gap, our work specifically targets Bengali, a language spoken by 272 million speakers worldwide (Zeidan, 2023), yet considered a low-resource language. We make the following contributions:
• We introduce BEnQA, a science questionanswering dataset in English and Bengali, consisting of 5,161 questions taken from the Bangladeshi national curriculum for grade school exams. This dataset covers a wide range of subjects and question types, including those requiring multi-step reasoning.
• Our dataset is parallel in English and Bengali, allowing us to benchmark existing LLMs and measure the performance gap between them. Additionally, the parallel nature of our dataset ensures a fairer comparison between the two languages.
• Through our evaluation of various LLMs, we indeed observe a significant performance discrepancy between questions in English and Bengali. Additionally, we investigate the impact of different prompting techniques on LLMs’ performance in Bengali, such as Chain-of-Thought (CoT) prompting and appending English translation to the prompt.

heading: 2 Related Work
text: Multilingual Reasoning Benchmark
There have been many English benchmarks to evaluate the reasoning capabilities of LLMs, such as COPA (Roemmele et al., 2011), HellaSwag
1158
(Zellers et al., 2019), CosmosQA (Huang et al., 2019), and CommonsenseQA (Talmor et al., 2019). For non-English languages, one of the widely used benchmarks is X-COPA (Ponti et al., 2020). A recent work (Doddapaneni et al., 2023) provides a human translation of this dataset in several Indic languages, including Bengali. Besides these, BIG-Bench Hard (Suzgun et al., 2022) is a dataset consisting of a collection of 23 challenging BIGBench tasks (Srivastava et al., 2022) that measure a model’s reasoning ability across various tasks such as logical deduction, multi-step arithmetic, and more.
Recent works develop datasets using academic exam questions to benchmark the ability of current LLMs. The majority of academic exam question datasets are in English, with a few exceptions, such as IndoMMLU (Koto et al., 2023). Notable examples of English datasets include MMLU (Hendrycks et al., 2020), which covers multitask problems from grade school to college level; MATH (Hendrycks et al., 2021), consisting of competitive math problems; GSM8k (Cobbe et al., 2021), which consists of grade school mathematics problems; and the ARC dataset (Clark et al., 2018) that proposes grade school level multiple-choice science questions. Recent GPT-4 technical report (OpenAI, 2023) also included benchmark results of many exams such as the GRE, bar exam, and AP exam, as did other works such as Kung et al. (2023) and Choi et al. (2023). For Bengali, this lack of resources is especially evident. One existing data for Bengali is MGSM (Shi et al., 2022), a professionally translated grade school math problem from GSM8k in 10 languages, including Bengali. Our work aims to address this gap by introducing BEnQA, covering questions from various exam subjects collected from the official Bangladeshi school exams. BEnQA stands out as the only academic exam benchmark dataset available in Bengali.
Multilingual Prompting Chain-of-Thought (CoT) prompting (Wei et al., 2022, 2023) has ushered a new wave of work in eliciting reasoning behavior in large language models. Shi et al. (2022) demonstrated the effectiveness of CoT and reported an increase in performance in mathematical reasoning while doing step-by-step reasoning in non-English languages. The same work also reports that translating the questions using Google Translate and then doing step-by-
step reasoning in English often works even better. Ahuja et al. (2023) also used a similar approach. More recently, Huang et al. (2023) introduced cross-lingual thought prompting that first translates the original query into English and then does CoT reasoning in English.
However, one obvious issue with doing CoT reasoning in English is that it renders the LLM output less useful for non-English users. Especially, when LLMs are used in educational contexts, CoT often provides helpful hints for learners (Han et al., 2023). In our work, we demonstrate that it is possible to do CoT reasoning in the target language and improve performance, as long as we have access to the English translation of the questions.

heading: 3 BEnQA Dataset
text: Our dataset, BEnQA, is a parallel corpus of English-Bengali science questions in a multiplechoice format from 8th, 10th, and 12th grade exams. These questions are sourced from nationwide board exams in Bangladesh and are officially available in both English and Bengali, giving us a high-quality parallel corpus.

heading: 3.1 Dataset Curation
text: Examination questions in Bangladesh are predominantly available in print rather than digital format. Therefore, we collect examination papers and use readily accessible solution books to create the ground truth for the questions. We then employ four typists, proficient in both Bengali and English, to digitize the questions and their corresponding answers. This process enables the conversion of questions and their answers into a digital format with minimal errors.
All questions are of the multiple-choice format, each presenting four options. The typists are instructed to format mathematical equations and chemical formulas using LATEX. Questions involv-
ing figures are excluded. Additionally, we apply specific heuristics to filter out some questions that are not parallel, annotation errors, and poor-quality questions, including those with inconsistencies between the ground truths in English and Bengali, and non-parallel questions. The heuristics for filtering non-parallel questions are explained in detail in the next section.

heading: 3.2 Parallel Corpus Creation
text: Typically, questions in Bangladeshi national board exams are initially formulated in Bengali and then translated into English. This process allows us to create a parallel dataset. However, the question order in each version does not always align (i.e., the first question in the Bengali version can be the eleventh question in the English version). To address this, we develop a very simple yet effective algorithm for aligning the English and Bengali questions, illustrated in Figure 1.
First, we translate each Bengali question to English via Google Translate API. Within each subject, we then match the Google-translated version with the actual English translation of the question, using the cosine similarity of their embeddings. We use OpenAI text-embedding-ada-002 embedding to achieve this. After the initial matching, two native Bengali speakers who are also proficient in English, manually verify each question to ensure that each Bengali question corresponds to its English counterpart. We also filter out a few questions for which the ground truth is different in English and Bengali. Manual inspection reveals that such cases typically result from annotator errors or issues with the questions themselves.
Since these translations are often carried out by
school teachers with varying levels of English proficiency, some English questions may contain subtle grammatical errors or sound unnatural to native English speakers. We investigate the impact of these grammatical inconsistencies on model performance by creating a grammar-corrected version of the English questions using GPT-4 with human supervision in the loop. Our findings indicate that grammatical errors generally do not significantly affect model performance (see Appendix A).

heading: 3.3 Dataset Properties
text: In our proposed BEnQA corpus, there is a total of 5,161 questions available in both English and Bengali. This dataset comprises 55% (2,857) questions from the 12th grade, 36% (1,857) from the 10th grade, and 9% (447) from the 8th grade. The 12th-grade section encompasses various subjects like Mathematics, Physics, Chemistry, and Biology, further divided into part I and part II based on subtopics. For the 10th grade, it includes Mathematics I, Mathematics II, Physics, Chemistry, and Biology. In the case of the 8th grade, the dataset includes Mathematics and Science, with the latter being a comprehensive subject covering 8th-grade science. Figure 2b summarizes the subject and grade-wise question statistics.

heading: 3.4 Dataset Categorization
text: Our dataset encompasses various question types, each demanding specific skills, irrespective of the subject or grade. Based on the skills necessary for solving them, we categorize the questions into three distinct types:
• Factual Knowledge: These are questions that solely rely on knowledge of basic facts, events,
concepts, dates, etc. Answering such questions does not involve any form of analysis or reasoning.
• Procedural & Application: This category comprises questions that require the ability to apply a procedure, utilize a familiar concept, or employ a formula for solving.
• Reasoning: Questions falling into this category demand multiple steps of analysis or reasoning to arrive at a solution.
We utilize GPT-4 for the purpose of categorizing the questions into these groups. We use the prompt given in Table 3 to categorize them. The categorization results are depicted in Figure 2a for question-type distribution and Figure 2b for subject-wise breakdown. The majority of Biology and Chemistry questions lean towards factual knowledge. Conversely, Mathematics questions are primarily centered around procedural & application skills, with some instances demanding reasoning. In the case of Physics, the number of factual and procedural questions is nearly equal.

heading: 4 Experiment Setup
text: To assess the performance of existing LLMs on our newly created dataset, we evaluate them using the BEnQA dataset across several open-source and proprietary models. Following the recommendation from prior work on ChatGPT (Lai et al., 2023), we keep the system prompt in English for both Bengali and English datasets. Most of the benchmark results presented in the main paper are conducted in a zero-shot setting to save tokenization costs in proprietary models (Petrov et al., 2023). Additionally, experiments in 3-shot and 5-shot settings were performed as supplementary analyses, details of which are provided in Appendix C.

heading: 4.1 Models
text: To evaluate the current LLMs capability in our dataset, we use the following models:
English • Proprietary LLMs: GPT-4 (OpenAI, 2023)2,
GPT-3.53, Claude 2.14
2We utilized gpt-4-1106-preview API endpoint (https://www.openai.com/research/gpt-4)
3We utilized gpt-3.5-turbo-1106 API endpoint (https://platform.openai.com/docs/models/gpt-3-5-turbo)
4https://www.anthropic.com/index/claude-2
• Open-source LLMs: LLaMA-2 7B and 13B (Touvron et al., 2023)5, Mistral 7B6
Bengali • We utilize the proprietary models mentioned
above for our Bengali experiments.
It is worth noting that most open-source models do not perform well on Bengali. For this reason, we conduct all open-source model experiments on only the English version of the dataset.
Most of the other experiments (i.e., ablations, exploration of prompting techniques, and others) in this paper are conducted using GPT-3.5 Turbo in order to avoid the high cost associated with more advanced proprietary models, particularly for Bengali (Ahia et al., 2023; Petrov et al., 2023). Opting for benchmarking on GPT-3.5 ensures a balanced trade-off between value and cost.

heading: 4.2 Prompts
text: In our experiments, we explored whether certain prompting methods could enhance the performance of LLMs on our dataset. We utilized the Chain-ofThought (CoT) prompt, as recent work has noted its efficacy in improving reasoning tasks (Wei et al., 2023). Moreover, we also experimented without the CoT prompt for comparison. Detailed descriptions of the specific prompts used in our experiments are provided in Appendix F. Throughout all the Bengali experiments, the model was explicitly instructed to perform CoT reasoning in Bengali and avoid using English, aligning with our original goal of making the model’s output useful to Bengali users, especially if LLMs are used in an educational context. Another potential approach to get CoT steps in Bengali is by utilizing a translation method, by first generating CoT steps in English then translating them to Bengali. However, we chose not to rely on this method to avoid "translationese" problem, where the translated sentences are awkward or grammatically incorrect, which might result in a failure to capture the true intent or meaning of the original sentence.

heading: 4.3 Evaluation Metric
text: The evaluation process involves a manual assessment of model outputs to determine if the final answer aligns with the ground truth. It is important to note that our evaluation does not scrutinize the
5https://llama.meta.com/llama2/ 6https://mistral.ai/news/announcing-mistral-7b/
validity of intermediate reasoning steps.7 In other words, we consider the answer correct if the final result is accurate, without delving into the examination of the validity of each intermediate reasoning step.

heading: 5 Results

heading: 5.1 How do LLMs perform on BEnQA?
text: Performance across all subjects In this experiment, we investigate the performance of various existing LLMs across all subjects in the BEnQA dataset. The results of all models are presented in Figure 3. For English questions (Figure 3a), our result shows that proprietary LLMs are far ahead of open-source LLMs such as LLaMA-2 and Mistral. Among proprietary LLMs, GPT-4 is significantly ahead, followed by GPT-3.5. We also observe that the LLMs’ performance across subjects is relatively uniform, with dips in some subjects, such as 12thgrade Math II and 10th-grade Biology, indicating that these might be more challenging subjects for LLMs. The trends are also similar to Bengali questions (Figure 3b), where GPT-4 maintains a signif-
7We compared the intermediate CoT evaluation vs. final answer evaluation, and the result shows that the instances where the model gets the final answer right when the CoT steps are wrong are low. The details can be seen in Appendix B.
icant score gap. Interestingly, Claude2.1 shows a much closer performance to GPT-3.5 in Bengali, a contrast to its performance in English. The full benchmark table can be found in Appendix G.
English vs. Bengali performance To investigate the performance disparity between English and Bengali questions, we focus on two of the highest-performing models for comparison: GPT3.5 and GPT-4. As depicted in Figure 4a, a substantial performance gap exists between Bengali and English in GPT-3.5. However, in GPT-4 (Figure 4b), the gap is much smaller, and in fact, performance in both English and Bengali has improved by a large margin across all subjects.

heading: 5.2 Does Chain-of-Thought help?
text: Chain-of-Thought (CoT) prompting is typically used to enhance the performance of LLMs in tasks that require reasoning. In order to evaluate its effectiveness on our dataset, we conducted several experiments based on subject and question categories.
Performance by Subject As shown in Figure 5, Chain-of-Thought reasoning boosts GPT-3.5 performances across the dataset in English. We also observe a similar trend with Bengali, with more details available in Appendix C. Interestingly, when
we decompose the result by subjects, CoT does not help all subjects uniformly as shown in Figure 6. CoT seems to help Math the most and Biology the least; this is indeed correlated with the different portion of question categories in each subject (Figure 2b), where we see that biology has more factual questions while math has more reasoning and application-based.
Performance by Question Category To validate our subject-based performance findings, we further analyze the GPT-3.5 performance by question category. As depicted in Figure 7, we indeed see that reasoning and application questions particularly benefit from CoT prompting, with accuracy improvements ranging by 10–20%. In contrast, we observed a minimal improvement in factual questions, highlighting the need for alternative methods to enhance the factual question performance. It is also worth noting that in English, the gains in application and reasoning questions through CoT still cannot surpass the performance of factual questions. For Bengali, the improvement is mostly seen in application questions, but not so much in factual and reasoning, as elaborated in Appendix C.2.

heading: 6 Additional Experiments
text: In this section, we conduct additional experiments to explore whether we can improve the performance in Bengali using better prompting. All the experiments described in this section were conducted on GPT-3.5, which provides a good value of cost-efficiency vs. capability in Bengali.

heading: 6.1 Can the performance be improved by appending English translation?
text: Our hypothesis for the models’ subpar performance in Bengali might be attributed to two main factors: their unfamiliarity with Bengali scientific terminology and potential difficulty in understanding non-latin scripts (Lai et al., 2023). We expect that having access to a translation of the question would likely help the model to understand the context better. To confirm our hypothesis, we experiment with a randomly selected subset of 115 data points from each subject within the 10th-grade exam questions. Questions from 10th-grade exams were chosen because they provide a middle ground in terms of difficulty compared to 8th and 12th grade.
Prompt Details As illustrated in Figure 8, in addition to providing the model with the Bengali question, we append the English translation to the prompt. This results in a prompt that consists of both the original data and its translation.
Experiment Results As shown from the results in Figure 9, appending English translation appears to have a positive impact in all subjects. Throughout our work, all system prompts have been consistently in English. In this experiment, the questions were provided in Bengali along with the English translations. Hence the performance gain can be concluded to be coming from the appending of the English translations. The most improvement is shown in Biology, where scientific terminology can be found in the majority of questions. In contrast, for Math, the improvement is not as significant as in the other subjects. We also experimented with the case where we did not have access to gold English translations and replaced them with LLMgenerated translations. Our initial experiments suggest that appending LLM-generated translations might work just as well as appending human translation. More details can be found in Appendix C.3.

heading: 6.2 Does the translation appended prompting method generalize to other datasets?
text: We extend our experiments to other datasets, such as COPA and Big-Bench-Hard, to evaluate the applicability of our prompting strategy across various datasets. For COPA, we ask the model to answer Bengali COPA problems taken from IndicCOPA (Doddapaneni et al., 2023).
COPA As shown in Figure 10, a noticeable improvement is observed in Bengali COPA when English translations are appended alongside the Bengali question. The improvement is more pronounced when we use the actual English version of the COPA dataset (marked as Gold). However, even using GPT-generated translation, accuracy improves the performance by 7 to 8%.
We also try to see if translation appended prompting can be helpful for the other languages included in the original X-COPA (Ponti et al., 2020) languages. The results are shown in the Figure 11. Appending English translation increases accuracy in all but one language, with the effects most pro-
nounced in low-resource languages. Further research in utilizing this prompting technique is an interesting direction left for future work.
Big-Bench-Hard Bengali For this experiment, we curate Big-Bench Hard Bengali (BBH-BN) dataset using GPT-4. Since only a small proportion of tasks require reasoning skills in BEnQA (11% of the total questions), we chose Big-Bench Hard as an additional dataset alternative. We selected some tasks in Big-Bench Hard that require reasoning and also have equivalence to Bengali to add more questions in this category. When we say equivalence to Bengali, we mean ignoring some tasks, like word sorting, because the alphabetical order may not be the same upon translation to Bengali. In this experiment, two native Bengali speakers generated prompts for each task iteratively and then generated the prompt.
Our experiment shows that on average, there is a 5.8% increase in BBH-BN performance in Bengali when we append gold English translation, while appending GPT-4-generated English translation provides an average of 4.2% improvement. Details about Big-Bench-Hard task selections and results can be found in Appendix E.

heading: 7 Discussion
text: Unlike most other bilingual or multilingual NLP datasets, which are usually first curated in English and then translated into other languages, questions in our BEnQA datasets are curated in Bengali first and then translated into English by school teachers. This approach could also provide an interesting outlook on how L2 English speakers might use English and whether LLMs are robust to such linguistic variations.
Besides, our benchmark results highlight that LLMs, such as ChatGPT, still have some way to go in terms of catching up with performances in low-resource languages like Bengali. One interesting aspect we noticed is that, in Bengali, it is much harder to make the model output in a predefined format for automated evaluation. Future research should focus on improving the instructionfollowing ability of multilingual models.
Another key finding is the substantial performance gap between open-source and proprietary language models. The open-source models, which are more accessible in the context of developing countries, must stride ahead to catch up with the proprietary models to make sure the benefits of AI, particularly LLMs, are not limited to only some demographics. A recently published open multilingual LLM, Aya (Üstün et al., 2024), represents a promising step in this direction.
Furthermore, we demonstrated the feasibility of having LLMs respond in the target language, while leveraging the advantage of high-resource languages, such as English, in the inference pipeline by utilizing translation of the query. The benefit of this approach is that we do not need to have access to the manually-created gold translation (as we had in this case): translation using the same model or more powerful/specialized models would work too. However, the choice of translation method, whether it is another more proficient LLM or a domain-specific fine-tuned translation model, can influence the performance. This finding necessitates further research to optimize and refine the use of LLMs, especially in the context of low-resource languages.

heading: 8 Conclusion and Future Work
text: In this paper, we introduced BEnQA, a locally sourced dataset from Bangladesh, comprising middle-school and high-school level exam questions in both English and Bengali. Our dataset is provided in parallel format, allowing us to benchmark the performance discrepancy between both languages in a fairer way. Upon benchmarking several Large Language Models (LLMs) with this dataset, we found that performance in Bengali lags behind English, even for the best-performing models; and current open-source models significantly lag behind proprietary models. We also explored whether the performance in Bengali questions can be improved by appending English translations
to the prompts. This approach resulted in performance enhancements across most subjects in the BEnQA dataset on GPT-3.5 model. Notably, this improvement also extends to other datasets, such as COPA and Big-Bench-Hard. These findings pave a promising direction for future research in improving the LLMs performance in low-resource languages, specifically in Bengali.
Our findings open up several promising directions for future research. Our results have highlighted a possible gap in the current models’ understanding of Bengali terminology, leading us to develop a very simple prompting method that helps the performance in Bengali data significantly. We also show the efficacy of this prompting approach on Bengali COPA, X-COPA, and Bengali Big-Bench-Hard dataset. Future research should also focus on to see to what extent such prompting methods utilizing high-resource languages generalize to other datasets and languages.
Limitation
Our works have several limitations that should be acknowledged. Firstly, our dataset primarily focused on text-based questions, as we discarded figure-based questions during the dataset curation. This limitation might restrict the scope of our findings, as questions with visual elements often require more reasoning steps. Additionally, since the questions are multiple-choice, there might be a possibility that the models use a shortcut to answer the questions, especially for questions that do not require advanced reasoning skills like factual questions. Despite these limitations, our dataset serves as an important starting point for benchmarking LLMs in Bengali, which currently have limited resources available for question-answering and knowledge intensive tasks.
Ethical Considerations
The exam questions in BEnQA dataset are freely available and have been manually curated and reviewed to minimize the presence of harmful content. This dataset is publicly accessible and distributed under the CC BY-SA 4.0 license, and the evaluation code is available under the MIT License. Our work has been reviewed and received approval from the Institutional Review Board (IRB) at our institution. All annotators involved in this project were compensated above the minimum wage.

heading: Acknowledgements
text: This research was supported by the MSIT(Ministry of Science and ICT), Korea, under the National Program for Excellence in SW, supervised by the IITP(Institute of Information & Communications Technology Planning&Evaluation) in 2023 (2022- 0-01092).

heading: A Effect of Grammar Mistakes in BEnQA
text: To see the effect of subtle grammar mistakes and unnatural English translations, we selected 120 questions from each subject of the 10th grade. We
prompt GPT-4 to fix grammatical issues and unnatural translations if there are any. A native Bengali speaker proficient in English then went through the results and made necessary adjustments to the GPT4 corrections.8 The performance of GPT-3.5 on this subset of the dataset is presented in Figure 12. The difference between a grammar mistake-free version and the version with grammar mistakes is minimal, and a manual inspection revealed the slight discrepancy that is shown in Figure 12 emerged from the stochastic nature of GPT-3.5 rather than any grammar mistakes.
B Intermediate CoT Results
We did human evaluations on a subset of the responses, and the results can be seen in Table 1. The primary observations are: when the CoT is wrong, the model’s final answer is wrong, and when the CoT is right, the final answer is correct. The instances where the model accidentally gets the final answer right with the wrong CoT steps are pretty low.
Evaluating the CoT steps for all samples using human evaluation is impractical, and based on the analysis of this sample, we decided to evaluate the final answer to make the evaluation more scalable.

heading: C Additional BEnQA Results
text: C.1 Few-Shot Results
We adhere to zero-shot prompting throughout the paper as the preliminary experiments revealed that the difference between zero-shot and few-shot settings is not very significant (Figure 13) and conducting extensive few-shot experiments is costly
8Oftentimes, GPT-4 alters the original meaning of the question when doing grammar fixing, which is why we did not use this protocol to fix grammar for the whole dataset.
for proprietary models, and even more so in Bengali due to infertile tokenization.
Few-Shot Example Preparation: When preparing the few shot examples, it was made sure that at least the first three examples cover our categorization (factual, procedure and knowledge, reasoning) as described in Section 3.3. The five examples also covered all types of questions that (multiple choice and multi-selection type) usually appear in the exams. Finally, the examples in each topic were topic-wise diverse for each subject.
C.2 Effect of Chain-of-Thought Prompting
Chain-of-Thought (CoT) performance for English in 1 to 5-shot settings is illustrated in Figure 14. In all cases, employing CoT improves performance.
Similar observations can be made for GPT-3.5 in Bengali as well (Figure 15). We only chose k = 1, 3, and 5 to prevent high API costs associated
with low-resource languages.
CoT Subjectwise Breakdown: Similar to what we have observed in Section 5.2 We also note that doing CoT improves performance for Bengali. Figure 16 shows the subjectwise CoT performance breakdown for GPT-3.5.
We can also see the CoT performance breakdown by question type for Bengali in Figure 17. Unlike English, the performance in reasoning questions turns out to be on par with Bengali. Upon analysis, we conjecture that this discrepancy is due to the smaller size of data for the reasoning questions in some subjects, such as Physics. Moreover, the model by default also finds these questions harder to solve in Bengali compared to English as we can see that the base accuracy is much lower for Bengali than for English.
C.3 Does LLM-generated translation work as well for appending?
One obvious question raised by our translation appending prompt method is whether machine translation can work as a replacement in cases where we do not have access to human-generated translations.
Figure 18 shows that appending LLM-generated English translation works just as well, and in fact, outperformed the original translation for some of the subjects. We only used GPT-3.5 and GPT-4 translations in this experiment and did not incorporate Google Translate; because, in our observations, equations written in LATEX format tend to get broken by Google Translate.
The result of this experiment leaves open the possibility of incorporating such prompting techniques for a wide range of multilingual tasks where obtaining manual English translation would be quite difficult.

heading: D Evaluation of Translation Quality Using Backtranslation
text: We use the COPA dataset to validate the translation quality of different models. We first take the Bengali translation of COPA dataset from (Doddapaneni et al., 2023) and backtranslate the Bengali translation of COPA dataset from Bengali to English using GPT-4, GPT-3.5 and Google Translate and then compare it with the performance on original English COPA. The results are in the Figure 19.
As we can see, GPT-4 translation (BN to EN) provides the closest results to the original English translation, while GPT-3.5 and Google Translate are slightly lower.

heading: E Additional Experiments on COPA and BIG-Bench
text: Prompt Variation: We explore four possible variations of the prompting method:
• BN: Asking only in Bengali.
• BN + EN (Gold): Asking in Bengali and appending the English version from the original data.
• BN + EN (GPT-3.5): Asking in Bengali and appending the English translation done by GPT-3.5.
• BN + EN (GPT-4): Asking in Bengali and appending the English translation done by GPT4.
Big-Bench Hard Task Selection: From the 23 datasets officially provided by BIG-Bench Hard, we selected 14 tasks that can have equivalence in Bengali9. We chose Causal Judgement, Date Understanding, Disambiguation QA, Formal Fallacies, Logical Deductions Five, Logical Deductions Seven, Logical Deductions Three, Multistep Arithmetic, Navigate, Object Counting, Penguin In a Table, Reasoning About Colored Objects, Temporal Sequences, and Web of Lies tasks for our work.
These 14 tasks were machine-translated into Bengali using GPT-4 with three human-annotated examples as prompts for each task. The prompts were made iteratively by two Bengali speakers to reflect the nature of each task.
Experiment Result: Figure 20 shows the results for appending English translations in Big-BenchHard experiments. In 7 out of 14 cases, appending English translation helped, while in three cases, it slightly hurt the performance. For the remaining four cases, performance remained largely unaffected. The detailed numbers can be found in Table 11.

heading: F Prompts Used for Experiments
text: This section contains all the original prompts used for all the experiments.
Grammar-Corrected BEnQA Question Preparation: We use the prompt given in Table 2 to fix the grammatical issues of the original questions in English with GPT-4.
BEnQA Dataset Categorization: We mentioned in 3.3 that we classified our dataset questions into these three categories: Factual Knowledge, Procedural & Application, and Reasoning using GPT-4. We use the prompt given in Table 3 to categorize them.
For each question in the dataset, we take zeroshot approach with the prompt for categorization.
Table 7 represents the subject-wise question category in tabular visualization.
Chain-of-Thought Prompting for BEnQA Zeroshot benchmark: Table 4 shows the prompt for zero-shot benchmarking with the Proprietary models.
9Some BBH tasks like manipulating English alphabet letters, which do not have a direct ‘translation’ in Bengali.
Chain-of-Thought Prompting for BEnQA Fewshot benchmark: The prompt we use for fewshot benchmarking with the Proprietary models is shown in Table 5.
Prompt for Translation Appended Benchmark: As described in E, we use the prompt shown in Table 6 to restrict the model to only reason in English in the translation-append experiment.

heading: G Benchmark Statistics
text: This section contains benchmark results by types and datasets used for our experiments
BEnQA Zero-shot Benchmark Table 8 shows the zero-shot benchmark results on BEnQA.
BEnQA Few-shot Benchmark without Chain-ofThought Reasoning Table 9 shows the few-shot benchmark results on BEnQA without Chain-ofThought Reasoning.
BEnQA Few-shot Benchmark with Chain-ofThought Reasoning Table 10 shows the fewshot benchmark results on BEnQA with the use of Chain-of-Thought.
BIG-Bench-Hard Zero-shot Benchmark Table 11 shows the zero-shot benchmark results on selected reasoning-based BIG-Bench-Hard datasets.

heading: H BEnQA Questions Examples
text: We present some examples from the 10th-grade subjects by the categories consisting of both English and Bengali versions in Figures 21 & 22. We kept the questions as they are; that is, we did not correct the subtle grammatical awkwardness in the English version of some of the questions.

heading: Future Work
text: These findings pave a promising direction for future research in improving the LLMs performance in low-resource languages, specifically in Bengali. Our findings open up several promising directions for future research. Our results have highlighted a possible gap in the current models’ understanding of Bengali terminology, leading us to develop a very simple prompting method that helps the performance in Bengali data significantly. We also show the efficacy of this prompting approach on Bengali COPA, X-COPA, and Bengali Big-Bench-Hard dataset. Future research should also focus on to see to what extent such prompting methods utilizing high-resource languages generalize to other datasets and languages. Limitation
Our works have several limitations that should be acknowledged. Firstly, our dataset primarily focused on text-based questions, as we discarded figure-based questions during the dataset curation. This limitation might restrict the scope of our findings, as questions with visual elements often require more reasoning steps. Additionally, since the questions are multiple-choice, there might be a possibility that the models use a shortcut to answer the questions, especially for questions that do not require advanced reasoning skills like factual questions. Despite these limitations, our dataset serves as an important starting point for benchmarking LLMs in Bengali, which currently have limited resources available for question-answering and knowledge intensive tasks. Ethical Considerations
The exam questions in BEnQA dataset are freely available and have been manually curated and reviewed to minimize the presence of harmful content. This dataset is publicly accessible and distributed under the CC BY-SA 4. 0 license, and the evaluation code is available under the MIT License. Our work has been reviewed and received approval from the Institutional Review Board (IRB) at our institution. All annotators involved in this project were compensated above the minimum wage