Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
Authors: Omid Ghahroodi
Marzia Nouri
Mohammad Vali Sanian
Alireza Sahebi
Doratossadat Dastgheib
Ehsaneddin Asgari
Mahdieh Soleymani Baghshah
Mohammad Hossein Rohban
Venue: COLM (2024)
Keywords: LLM, Evaluation, Multitask, Persian Language Understanding
TL;DR: The Khayyam Challenge offers an evaluation framework for Persian-supporting LLMs, featuring 20,805 original, diverse questions accompanied by rich metadata.

Abstract: Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages.
In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,805 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs. We believe that the Khayyam Challenge will improve advancements in LLMs for the Persian language by highlighting the existing limitations of current models, while also enhancing the precision and depth of evaluations on LLMs, even within the English language context.

Published as a conference paper at COLM 2024
Khayyam Challenge (PersianMMMLU) Is Your LLM Truly Wise to The Persian Language?
Omid Ghahroodi o Marzia Nouri Mohammad Vali Sanian o*, Alireza Sahebi
Doratossadat Dastgheib Ehsaneddin Asgarit, Mahdieh Soleymani Baghshah Mohammad Hossein Rohban
Raia Center for Artificial Intelligence Research Computer Engineering Department, Sharif University of Technology, Iran *Qatar Computing Research Institute, Qatar
(oghahroodi98, warzia. 1999, mvs2667, arsahebi97 yegraal........ll.m...m segnail.com com d_ d_dastgheibesbu. ac. ir, easgar iehbku. edu. qa, (soleymani, rohban)@sharif..ed
Abstract
Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Addi- tionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages. In response to this ne- cessity, we introduce Khayyam Challenge (also known MMLL))) a meticulously curated collection comprising 20,805 four-choice questions sourced from diverse tasks extracted from Persian examinations, span- ning wide spectrum of subjects, complexities, and ages. The primary objective the Khayyam Challenge is to facilitate the rigorous evalua- tion of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various ed- ucational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use original, non-translated data tailored for Persian speakers, ensuring the framework free from translation challenges and errors while encompass- ing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked evaluation framework that combined all of these features into single comprehensive benchmark. Furthermore, we evaluate a wide range of ex- isting LLMs that support the Persian language, with statistical analyses and interpretations of their outputs. We believe that the Khayyam Challenge will improve advancements in LLMs for the Persian language by high- lighting the existing limitations of current models, while also enhancing the precision and depth of evaluations on LLMs, even within the English language context.
*These authors contributed equally this work and considered joint second authors. The order listed randomly reflect their equal contributions.
Thheee authors contributed equally to this work and considered joint corresponding authors.
The order of corresponding authors is listed randomly to reflect their equal contributions.
1

---

Published as a conference paper at COLM 2024
1 Introduction
Large Language Models (LLMs) have recently revolutionized applications of machine intelligence (Hong et al., 2024; Wu et al., 2023; Thirunavukarasu et al., 2023; Glukhov et al., 2023). The rapid deployment of these models within industrial and public sector solutions has made evaluating their capabilities an imperative task (Guo al., 2023). To address critical aspects of language understanding and bridge the gap between the knowledge that models observed during pretraining and the measures of success, the MMLU benchmark was introduced (Hendrycks et al., 2021). It assesses the ability of LLMs across diverse set containing 57 subjects.
Although automatic translation efforts like MMLU can bridge the gap in evaluation re- sources for languages other than English (Achiam et al., 2023), the cultural specificity of these questions limits their applicability across different languages and cultures. This is particularly true for Persian, a language with its own rich culture and literature, where direct translation of English MMLU content may not be suitable for certain culture-specific subjects. In addition, our investigation reveals significant complications in automatic translation, s
particularly in specialized domains like Physics, which can be attributed to the quality of state-of-the-art translations for specific subjects. Despite advancements in machine transla- tion technology, accurately conveying precise meaning in certain domains remains a hard task. For instance, the GPT-3.5 automatic translation often fails to include essential units such as "microjoules" in translations, leading to inaccuracies. Also, loss of information can occur in translation due to the existence of Persian words that lack exact equivalents in English. For example, in Persian, there are distinct terms for paternal aunt, maternal aunt, paternal uncle, and maternal uncle, while in English, only "aunt" and "uncle" are used (See Table 4). This underscores the inherent difficulty in translating domain-specific terminology accurately. For such reasons, some studies have evaluated LLMs on original non- non-Englishh datasets rather than the translated ones. For instance, Li et al. (2023) has introduced a Chinese dataset across 67 topics and indicated that current models struggle to achieve accuracies above a certain threshold. Also, Zhang et al. (2023) introduced a multilingual and multimodal dataset, and showed that multilingual text processing hardly achieves over 60% accuracy.
Motivated by the mentioned issues, we propose the Khayyam Challenge, also referred to as PersianMMLU, a benchmark designed to analyze the performance of LLMs in Persian and evaluate their knowledge and abilities comprehensively. Named in honor of the famed Persian polymath Omar Khayyam, whose contributions spanned various disciplines includ- ing Mathematics, Astronomy, Philosophy, and Poetry, the Khayyam Challenge embodies the multidimensional nature of Persian language understanding, This benchmark covers 38 subjects, including Mathematics and Physics, which require reasoning and computational ability, to Humanities and Social Sciences, demanding nuanced understanding and cultural sensitivity. Unlike the previous Persian datasets such as ParsiNLU (Khashabi et al., 2021), our benchmark includes more diverse topics in addition to different educational stages. Moreover, our dataset distinguishes itself from previous efforts like ParSQuAD (Abadani et al., 2021) by being originally constructed in Persian, naturally incorporating the nuanced semantics and intricacies inherent to the language itself, rather than solely focusing on literal translations. Unlike PersianQA (Ayoubi, 2021) and PQuAD (Darvishi et al., 2023), which are extractive datasets where models are tasked with extracting answers from given paragraphs and questions, our benchmark offers more comprehensive evaluation of LLMs. This is because the task of answer extraction alone may not sufficiently assess the models' overall language understanding and reasoning capabilíties. Our proposed dataset contains "Iran's national university entrance exam", and Kanoon Farhangi Amoozesh (Cultural Educational Center), wherein questions are accompanied by metadata for each question. This metadata includes the difficulty level, descriptive answer, the educational stage, the subject, and the specific topic of the question. Through the Khayyam Challenge, we aim to provide a holistic evaluation framework that reflects the diverse linguistic and cognitive challenges inherent in processing Persian text across various domains.
In our evaluations, we assessed several state-of-the-art language models, including GPT-3.5, GPT-4 (OpenAI, 2023), Aya (Üstün et al., 2024), PersianMind (Rostami et al., 2024), mGPT
2

---

Published as a conference paper at COLM 2024
(Shliazhko et al., 2022), mT0 (Muennighoff et al., 2022), Claude3-haiku (Anthropic, 2024), and XVERSE!, all purportedly equipped with some level of understanding of the Persian language. Our findings indicate that while most of these models struggle to grasp Persian nuances, particularly evident in domains such Calculus, Logic, and Geometry where accurate comprehension is essential, some exhibit comparatively better performance in contexts reliant on contextual understanding, such as Economics, Psychology, and Social studies. Notably, GPT-4 showcased relatively improved performance across multiple do- mains. However, there remains clear imperative need for further enhancements across all models, especially in technical disciplines like Discrete Mathematics, where precise language comprehension paramount for meaningful outcomes.
Furthermore, our evaluation unveiled new insights. We observed dependencies between the perceived difficulty of questions for humans versus LLMs. While both encounter similar challenges in tackling difficult questions, LLMs like GPT-4 demonstrated superior accuracy compared to humans, in questions intentionally designed to deceive, referred to as trapped questions. Moreover, our analysis identified biases in certain language models' responses, such as GPT-3.5 favoring particular choices, suggesting room for further improvement in these models.
Overall, the Khayyam Challenge (PersianMMLU) marks a significant step forward in evaluating the language understanding and abilities of LLMs that support the Persian language.
Our dataset and code are available on HuggingFace" and GitHub', respectively. Addition- ally, we set up leaderboard on HuggingFace to stay updated with the performance of other models.
2 Related work
2.1 Large Language Models
Over the past few years, there has been a significant improvement the performance of language models. This progress has been observed in line with the scaling law (Kaplan et al., 2020), thanks to the increasing size of training datasets, enhanced processing power, and new evolved model architectures. The continuing process of scaling the models resulted in LLMs like GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), Claude3, mTO (Muennighoff et al.2) 2022), XVERSE, Aya (Üstün et al., 2024), etc.
Even though Al models are highly capable of solving various tasks, they continue to encounter difficulties when it comes to real-world problems that, for example, require strong reasoning abilities complex mathematical calculations (Chang et al., 2024; Zhong et al., 2023). Therefore, we need to assess the effectiveness of these models in solving high- level tasks. This enables us to identify the weak points of the models and work towards improving them in the future.
Despite some of the recent LLMs being multilingual, studies indicate that their effectiveness is not as pronounced in non-Latin or low-resource languages as it is in English (Zhang et al., 2023). Consequently, it is essential assess multilingual LLMs on tasks that employ languages other than English.
2.2 Evaluation of LLMs
Several benchmarks have been developed to assess the performance of LLMs. One of the most significant benchmarks is MMLU (Hendrycks al., 2021), which evaluates language models for answering multiple-choice questions in 57 different tasks, but only in English.
hntps:///ittuub.com com httpt/tubb.oom/eeeeeaaaaaa xverse-ai
Pntps::/hhuggingface.co
hhtt:s://githuu.com//
"httss//hhugiingfac.coo
3

---

Published as a conference paper at COLM 2024
Meladata
Benchmark
Langnages
Type
Ans.
Diff. Lew.
Trap
MMLU
orig.
MCQA
15,90
X
57
AGIEval
Eng., Chi.
orig. Arans
MCQA,
8,82
inslanoes
Hot inslances
M33sam
(no Persian)
orig.
MCQA
12,317
X
eduu. stages
ParSQUAD
Pensan
70,500
X
177 titks
PersiarQQ
orig.
9,338
PQuAD
Pensian
orig
RC
80,000
X
19
ParsiNLU
Pensan
MCQA, RC, TE, QP MT
14,500
X
3 MCQA
Khayaam Challenge
orig.
MUDA
20,90
Table 1: Comparison various features Persian and English benchmarks. As descriptive answers and trapped questions were not defined for reading comprehension benchmarks, we marked those fields with a hyphen (-). Desc. Ans.: Descriptive Answer, Diff. Lev.: Difficulty Level, Eng: English, Chi: Chinese, orig.: original non-translated question, tran.: translated question, MCQA: Multiple-Choice Question Answering, FB; Fill in the Blank, RC: Reading Comprehension, SA: Sentiment Analysis, TE: Textual Entailment, QP: Question Paraphrasing, MT: Machine Translation., edu. stages: educational stages
M3Exam (Zhang et al., 2023) introduces multilingual, multimodal, and multilevel bench- mark for evaluating LLMs including more than 12K multiple-choice questions from languages (excluding Persian) at three educational stages. AGIEval (Zhong et al., 2023) is another benchmark that assesses the performance of LLMs on human-centric standardized exams in English and Chinese languages to measure their ability in human-level tasks.
There have been few benchmarks built to assess language models on the Persian lan- guage, including ParSQUAD (Abadani et al., 2021), PersianQA (Ayoubi, 2021), ParsiNLU (Khashabi et al., 2021), and PQuAD (Darvishi et al., 2023), in which some of their features are compared Table 1. ParSQUAD, PersianQA, and PQuAD present extractive datasets where models asked to extract answers from given paragraphs. While this task can gauge models' reading comprehension skills, it may not effectively evaluate general capabilities and inherent knowledge of models. PaarsiNLU evaluates language models based on 14500 questions from six language understanding tasks, including multiple-choice QA (MCQA), sentiment analysis, and more (Table 1). However, the questions in this benchmark fail to reach human-level complexity, thus inadequately assessing important skills of LLMs, includ- ing complex reasoning, needed for solving higher educational stages questions. Moreover, ParsiNLUU covers only three subject tasks in MCQA and lacks adequate metadata, such as question difficulty levels. This deficiency further restricts our capacity to evaluate the model's 's proficiency across specific tasks and different difficulty levels.
To address this gap, we introduce Khayyam Challenge, which features multiple-choice questions sourced from high-standard exams, and contains reach metadata to evaluate the innate knowledge and human-like skills of LLMs across different difficulty levels and educational stages.
3 Data
The Khayyam Challenge presents a robust dataset aimed at enhancing the evaluation of LLMs that support Persian, particularly in the context of multiple-choice question answering. This dataset encompasses diverse range of subjects, reflecting a comprehensive approach to assessing various cognitive abilities including language comprehension, reasoning, and knowledge recall across different educational stages.
The educational system Iran, from which this dataset draws, is structured into 12 years of schooling divided into segments: 6 years of primary education and years of secondary education. Primary education is split into lower primary school (LPS) for the first years, followed by upper primary school (UPS) for the next 3 years. Secondary education is similarly divided, with lower secondary school (LSS) encompassing the first 3 years, and upper secondary school (USS) comprising the final 3 years.
4

---

Published as a conference paper at COLM 2024
3.1 Data construction
The dataset is sourced from the "Pellekan Yadgiri (Learning Ladder)" website a part of the Kanoon Farhangi Amoozesh (Cultural Educational Center), educational institution in Iran. The platform provides resources for creating and administering tests, as well as standardized solutions for exercises. The dataset includes questions from various educa- tional stages and subjects, including items from the Iranian national university entrance examination. This diversity contributes the dataset's comprehensiveness and relevance.
While the dataset is primarily sourced from the Learning Ladder platform, we implemented additional processing steps. These included parsing HTML to extract attributes, filtering out questions with images or tables, removing explanatory answer questions, and deduplicating entries. We then reviewed the questions, corrected typos, and deleted those that required context that was not present in the question itself. These steps were crucial in refining the raw data into a structured, text-only dataset suitable for our research purposes.
We also note that the Learning Ladder platform serves a platform for questions con- tributed by a diverse group of educators from various backgrounds and disciplines in education. This diversity helps mitigate individual biases. Furthermore, the inclusion of questions from the university entrance exams, carefully selected from submissions by top educators, adds an additional layer of quality control. These features minimize the presence of potential biases in PersianMMLU. This dataset is distributed under a Creative Commons No Derivatives (CC ND) license, prohibiting the creation of derivative works.
We also tested the hypothesis that the data from this collection was included in the training data of the models based on Oren et al. (2024). We conducted this evaluation using the Aya model, the best publicly available model, which had access to the token probabilities. The hypothesis that the evaluation data was included in the training data was not proven, with a p-value of 0.3909.
3.2 Metadata characteristies
The Khayyam Challenge is enriched with valuable metadata that elevates its utility beyond simple aggregation questions. This metadata includes:
Educational Stage: Specifies the educational stage for which the question is in- tended (LPS, UPS, LSS, USS), allowing for the assessment of appropriateness and difficulty relative to the expected knowledge base at each stage of education.
Each question classified into one of five distinct difficulty lev- els: easy, relatively easy, medium, relatively difficult, and difficult. This nuanced categorization allows a detailed analysis of question difficulty and examinee per- formance. The difficulty level, like all metadata attributes, was extracted directly from the Learning Ladder platform. The platform uses an automated system to classify question difficulty based on student performance data. Questions answered incorrectly by a higher percentage of students are marked as more difficult.
Difficulty Level:
In addition to the correct answer, dataset provides de- tailed explanation for each question. This is crucial for understanding the reasoning behind the correct answer, facilitating deeper comprehension of the question.
Descriptive Answers:
Trap: Some questions contain a "trap" choice- an incorrect answer that might be easily mistaken for the correct one. These questions are referred to "trapped questions" and are generally more challenging, with the majority classified as difficult. This helps in understanding common misconceptions and the effectivenes veness of question design in truly testing knowledge and reasoning abilities. This attribute is also determined by the Learning Ladder platform, which analyzes frequently selected incorrect options to identify common misconceptions among students.
"https://peleyad.com// 9https://enkkaoom..r/
5

---

Published as a conference paper at COLM 2024
Candidate Choice Distribution: This metric provides the percentage of students selecting each answer choice for a given question.
Specific Topic: Questions are meticulously categorized into detailed subjects, such "Mathematics > Discrete Maths Combinatorics. This detailed classification enables targeted analysis of exam content and provides insights into the distribution and depth of questions across various subjects.
Year: Indicates the year when the question was designed, which can provide insights into the evolution of question complexity and educational standards over time.
The inclusion of this meladata not merely for augmentative purposes; it serves a critical role in enabling comprehensive analyses that can benefit educators, researchers, and devel- opers educational technologies. Specifically, allows for the comparison of performance between human examinees and LLMs on specific topics under varying difficulty levels. By assessing whether LLMs fall for the traps as humans or how they approach questions requiring complex thought processes, and limitations of current Al technologies in educational contexts.
can gain valuable insights into the capabilities
Moreover, the presence of descriptive answers supports the development of more sophis- ticated AT models by facilitating "chain of thought" processing, where the model learns to approach a problem step-by-step, mirroring human problem-solving methods. This not only enhances the model's problem-solving skills but also its ability to explain its reasoning in manner that is understandable to humans.
3.3 Data statistics
The dataset contains 20,805 multiple-choice questions across 38 tasks, spanning subject areas like humanities, mathematics, natural science, and social science, along with elements of intelligence testing. These questions necessitate blend of knowledge and reasoning. Additionally, the dataset includes 16,955 questions with human performance data, excluding Iran's ss national university entrance exam questions, and features 3,889 trapped questions. Figure 3-(a) depicts the allocation of questions among the main categories and their re- spective tasks. Figure 3-(b) outlines the distribution of questions based on their levels of difficulty. For more detailed information about the data, refer to Appendix A.
3.4 Key features
The Khayyam Challenge Dataset stands out for several reasons:
It spans a broad spectrum of subjects from literary comprehension to logic and intelligence testing, catering to different stages of education. This diversity makes it a versatile tool for assessing language models' capabilities across various domains.
Comprehensive Coverage:
The inclusion of detailed question metadata enhances the dataset's utility for nuanced analysis and model evaluation, providing valuable context for each question.
Rich Metadata:
New Data Utilization: By incorporating questions never before used in research, the dataset avoids common data contamination issues, offering a fresh challenge to language models.
Original, Non- -on-Transleted Content: Focused the Persian language, the dataset eliminates translation errors and incorporates cultural nuances, making it uniquely valuable for related linguistic and cultural studies.
Scalability: The dataset's design and sourcing methodology ensure its adaptability and expandability, allowing for straightforward updates and extensions without substantial human intervention.

---

Published as a conference paper at COLM 2024
Nathematics
GP7-3.5
Sdinnss
XVsitde 1.8
- Halku
t/i n.g 0.8
humanities
Avg on tooks
Figure 1: Comparison of accuracy across main categories for humans and various models
Experiments
4.1 Settings
We evaluated our benchmark using nine distinct models: GPT-4 (gpt-4-0613), GPT-3.5 (gpt- ..-tturbo-0125), Aya, XVERSE-13B, XVERSE-7B, PersianMind, mTOXL, mGPT, and Claude 3/ /aHiku. Detailed descriptions of each model can be found in Appendix B. Additionally, we benchmarked a standardized prompt-0 template (Figure 5) on the entire dataset and two other templates, prompt-1 (Figure 6) and prompt-2 (Figure 7), on subset of 1000 samples. We also conducted Chain-of-Thought (CoT) (Figure 8) on a subset of 1000 samples requiring CoT, such as mathematical questions. To ensure consistency and fairness throughout our experiments, we kept all model hyperparameters at their default values. Furthermore, we the temperature parameter zero for all models and did not impose any maximum limit on the number of tokens, allowing models the freedom to conduct any type of inference they desired.
4.2 Answer extraction methods
The input prompt did not restrict the model single-letter answers, might hinder the reasoning process. Therefore, we employed three distinct methods to extract the desired answer choice from the model output; "Regex", "Single Token Probability", and "Full Answer Probability".
In the "Regex" method, we developed detailed regex patterns tailored each model to accurately capture the desired choice from the model output (the patterns are available in our GitHub repository). In cases where regex patterns failed to identify the desired choice, we utilized pre-trained language model to generate embedding vectors for each of the answer choices and the model output. Finally, the choice with the most similar vector to the output's vector is selected the desired answer.
In the "Single Token Probability" method, applied to models providing token probabilities, used the probability distribution for the initial token in the model output to extract the probabilities associated with answer choice labels (e.g., 'a', "b, 'c', 'd' in prompt-1 template (Figure 6)). The label with the highest probability selected, leading to the choice of its corresponding option as the desired answer.
In the "Full Answer Probability" method, calculated the average of the logarithm of token probabilities for each answer choice by using the probability distribution of the initial token of the model output. We then chose the answer with the highest average as the desired option. This process is shown in equation 1.
1
Answer arg max
log(Pi)
(1)
:1
7

---

Published as a conference paper at COLM 2024
Our analysis shows that the "Regex" method has the highest accuracy, whereas the "Full Answer Probability" method demonstrated significant decrease in accuracy compared to the other two approaches.
4.3 Measure human performance
To measure human performance, we used the percentages of respondents who selected each answer choice for every question, as provided in the metadata. A question is labeled as correctly answered if the percentage selecting the correct answer exceeds the combined percentages of the other options. Otherwise, labeled as incorrectly answered. Human accuracy is then evaluated by counting the number of correctly and incorrectly answered questions.
4.4 Impact of translation quality
We aim to assess the impact of translation from state-of-the-art (SOTA) translation models on performance. To do so, we selected a set of 50 sample questions from three subjects our dataset and translated them to English with the assistance of field experts, well as using off-the-shelf translation models. Following this, we evaluated the GPT-4's performance both sets of samples: those translated with expert assistance and those translated by models alone. Our findings which are illustrated in Table 2a, revealed a notable decrease in performance when translated with the help of off-the-shelf models, highlighting the need for new dataset that does not depend solely on translated data. We also evaluated the impact of automatic translation on performance by translating a sample of English questions from the MMLU dataset (Hendrycks et al., 2021) into Persian and measuring GPT-3.5's performance on both sets. As shown Table 2b, automatic translation from English to Persian can reduce the performance of large language models.
PeL Topics
En acc.
En acc.
Fa acc.
MMLU Topics
En acc.
Fa acc.
Physics
58
76
50
Physics
42
26
Chemistry
74
86
Biology
76
74
Math
50
56
Machine Learning
46
40
(a)
(b)
Table 2: Accuracy rates of LLMs for (a): sample of questions from our dataset (Persian- MMLU) and their translations to English using an automatic translation model, and a field expert (denoted with an asterisk), and (b): sample of MMLU English questions and their translation to Persian using automatic translation model
4.5 Limitations of few-shot approach
We have developed a benchmark code for our dataset and used the zero-shot and CoT methods to calculate accuracy. Previous studies (Li et al., 2023; Zhong et al., 2023; Zhang et al., 2023) shows that using few shots on instruction-tuned models does not enhance accuracy and may even may decrease it. Therefore, we did not measure accuracy using few-shot techniques.
Main Categories
Human
GP144
Halku
GPT33
Aya
XVERSE-T3B
XVERSE-J
Porsian Mind
mTXXI
mCPPT
Random
Humanities
071
0.47
0.43
0.33
0.31
0.3
0.27
0.28
0.28
0.25
01.35
Mathematics
0:11
0.33
0.28
0.27
0.28
0.26
0.25
0.25
Natural Science
0.49
0.35
0.29
0.28
Arg an all tasks
0.77
0.5
0.18
0.37
0.31
0.31
0.29
0.29
0.29
0.25
0.25
Avg all questiions
0.76
0.69
0.11
0.36
0.32
0.3
0.29
0.28
0.28
0.25
0.25
Table 3: Human model accuracy for main categories: model answers extracted via "Regex"
8

---

Published as a conference paper at COLM 2024
5 Results and discussions
5.1 Results across all models
The evaluation results of the zero-shot method for all models across five main categories and three choice-extraction methods ("Regex", "Single Token Probability", and "Full Answer Probability"). as well as human performance, are presented in Table 3, and appendix Tables 9, 8.8.7 The more comprehensive results on all 38 tasks are reported in the appendix (Tables 10, 11, 12). Also, the results CoT and its comparison with zero-shot for GPT-3.5 using "Regex" method across three main categories on subset of dataset with 1000 questions are presented in appendix Table These results yield the following key findings:
Utilizing "Regex" method for answer extraction (Table 3) results in highest model performance compared to "Single Token Probability" (Table 9) and "Full Answer Probability" (Table 8) due to its more accurate and comprehensive choice extraction procedure. In the rest of the results section, we compare the accuracy models using the "Regex" method.
GPT-4 outperforms all other models in all five main categories, with an average accuracy of 6 percent higher than Claude3-haikuu the second-best performing model.
Aya, an open-source model, performs comparably or even better than GPT-3.5, a closed-source model, 4 tasks including Sociology USS and Economy USS. This demonstrates the convergence of open-source models' capabilities towards closed-source models.
Although MendianMind, 7B Persian- LLM, is trained and fine-tuned on billion Persian tokens (Rostami et al., 2024), its performance is equal to mTXXI. multilingual 3.7B LLM.
The performance gap between the best-performing model, GPT-4, and human aver- ages around 27%. In subjects like mathematics, this gap widens 31% accuracy for GPT-4. This exhibits a real challenge of current LLMs in solving human-level questions, especially in complex mathematical questions that need high-level math- ematical calculation and reasoning skills.
The models exhibit weaker performance in mathematics and natural science main categories compared to humanities and social sciences. This indicates their weaker performance on questions requiring high reasoning skills, compared to those mainly reliant on models' inherent knowledge. This underscores the necessity for enhancing the models' reasoning ability in the Persian language.
The CoT has improved the performance of GPT-3.5 in Mathematics questions by 10%, but decreased the performance in Humanities questions by 8%.
5.2 Accuracy trends
We demonstrated the accuracy of the models on all three difficulty levels of questions for different educational stages and question publication years in Figure and Figure 10 in the Appendix E, respectively. Key findings from these figures include:
Most model accuracies decline with the increasing publication year for questions with medium and easy difficulty levels, while human performance remains con- sistent. This suggests that humans may have adapted to the evolving questions' difficulty over time, whereas models have not.
There exists notable performance gap between difficult and medium questions in human performance, indicating that difficulty level has a stronger impact on human performance compared to LLMs. Since humans assigned difficulty levels to questions, this gap may stem from differing perceptions of difficulty between humans and I.LSs.
GPT-4 outperforms humans in difficult questions within four years, well as during the initial two educational stages (LPS and UPS). This result suggests that modern LLMss may excel at analyzing difficult questions compared to humans.

---

Published as a conference paper at COLM 2024
5.3 Selected choice distribution
Figures 12, 13, 14 in the Appendix E depicts the selected choice distributions of various mod- els. It reveals that despite surpasses it in accuracy, indicating more refined understanding despite its higher non- response rate. Comparatively, GPT-4 exhibits a more uniform distribution of choice selection, aligning closely with the Ground Truth and showing less bias than its predecessor, GPT-3.5, which tends towards selecting the second and third choices. High-performance models like GPT-4 demonstrate a closer alignment with Ground Truth distribution, indicating lower bias level and potentially higher utility in applications.
having the lowest count of unanswered questions, GPT-4
5.4 Trap analysis
Table in the Appendix F compares the performance of models and humans on trapped questions, using an x/y format where x represents overall accuracy and y the accuracy on trapped questions. The data shows that while traps often mislead students, leading to nearly random performance, models like GPTT44 exhibit only a slight drop in accuracy when faced with these traps. However, this drop in accuracy is most pronounced in the Social Sciences and Humanities. Notably, GPT-4 outperforms humans across all main categories in handling trapped questions, indicating its robustness against misleading choices and affirming the different perspectives of difficulty between humans and Al models.
5.5 Difficulty levels analysis
To explore how humans and LLMs perceive the difficulty of questions, we examined the accuracy of different models across three difficulty levels. Although our dataset included five labels ranging from easy to difficult to denote question difficulty, we combined the two relatively easy and relatively difficult labels into "easy" and "difficult", respectively, to ensure a more balanced distribution of question difficulty. Our experiment revealed a consistent trend: as question difficulty increased, both human and model answering accuracy decreased (See Tables 13, 14, 15 in the Appendix E).
Notably, in analytical and knowledge-bent baseed topics such as the humanities category, the GPT-4 model demonstrated superior performance compared to humans in tackling difficult questions. Conversely, models exhibiting more human-like performance, such as GPT-4, revealed that humans significantly outperformed them on easier questions. For additional statistical results, refer to Appendix G.
Conclusions
We introduced Khayyam Challenge, also known PersianMMLU, as the first frame- work for assessing LLMs in the Persian language across various tasks, difficulty levels, and educational stages. This framework includes comprehensive metadata such as human performance, difficulty levels, and traps. Our assessment encompassed examining the per- formance of current LLMs on these datasets, evaluating their ability to extract answers (probabilistic and rule-based paradigms), and considering various aspects highlighted in the metadata. Our findings revealed that while LLMs demonstrated relatively satisfac- tory performance in question-solving tasks (especially on GPT-4), they still significantly lagged behind human performance, particularly in tasks necessitating reasoning, Moreover, analysis of metadata concerning difficulty levels and trapped questions unveiled notable discrepancies between model and human behavior, suggesting fundamental differences in learning approaches. This underscores the necessity for adaptations in LLMs training methodologies to achieve human-like proficiency. For future works, we aim to develop an LLM that bridges the performance gap between existing open-source models and GPT-4.
10

---

Published as a conference paper at COLM 2024
7 Acknowledgement
The authors would like to express their sincere gratitude to Pellekan Yadgiri (Learning Ladder), a sub-organization of Kanoon Farhangi Amoozesh (Cultural Educational Center), for providing the valuable dataset and related metadata that was crucial for this research. We are particularly grateful to Masoud Tajfard for his kind assistance in facilitating the ac- quisition of this data, which was permitted for research purposes. The dataset and metadata made available by Pellekan Yadgiri were instrumental in enabling the analyses and findings presented in this work.
References
Negin Abadani, Jamshid Mozafari, Afsaneh Fatemi, Mohammd Ali Nematbakhsh, and Arefeh Kazemi. ParSQuAD: Machine translated SQuAD dataset for persian question answering. In 2021 7th International Conference on Web Research (ICWR). IEEE, May 2021.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. ar Xiv preprint arXiv:2303.08774 2023.
sonnet, Anthropic. The claude 3 model family: sonnet haiku. https://wwweer cdn. www.doo.nn.oi.....oo pdf, 2024.
Mohammad Yasin Ayoubi, Sajjad & Davoodeh. Persianqa: dataset for persian question answering. http://tttt...om../aaaa///////rr 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini
Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,, volume 33, pp. 1877-1901. Curran Associates, Inc, 2020. URL https::/prooeeednnggt newruinggn..nuuissseeeeeee.r.rps neurips..o httppssccppppeeaaaaeeeeepeeee//e files/ /at/le//pppe//222/ pdf..
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, Wei Ye, Yue Zhang, Yi Chang, Philip S. Yu, Qiang Yang, and Xing Xie. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol,, jan 2024. ISSN 2157-6904. doi: 04.1145/364120 URL https: //doi. www.oorg//..... 1141/114111111 Just Accepted.
Kasra Darvishi, Newsha Shahbodaghkhan, Zahra Abbasiantaeb, and Saeedeh Momtazi. Pquad: A persian question answering dataset. Computer Speech & Language, 80:101486, 2023. ISSN 0885-2308, doi: sciencedirect. com/ wtt.:///aaaeeeeetrte///////////a
101486. URL https://www
David Glukhov, Ilia Shumailov, Yarin Gal, Nicolas Papernot, and Vardan Papyan. Llm censorship: A machine learning challenge or computer security problem? arXio preprint vrXiv:2307.10710 2023.
Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, Deyi Xiong, et al. Evaluating large language models: A comprehensive survey. Xrv preprint ar arXiv:2310.19736, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
11

---

Published as a conference paper at COLM 2024
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-llm: Injecting the 3d world into large language models. Advances in Neural Information Processing Systems, 36, 2024.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec R Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020.
Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Malihe Alikhani, Moin Aminnaseri, Marzieh Bitaab, Faeze Brahman, Sarik Ghazarian, Mozhdeh Gheini, Arman Kabiri, Rabeeh Karimi Mahabadi, Omid Memarrast, Ahmadreza Mosallanezhad, Erfan Noury, Shahab Raji, Mohammad Sadegh Rasooli, Sepideh Sadeghi, Erfan Sadeqi Azer, Niloofar Safi Samghabadi, Mahsa Shafaei, Saber Sheybani, Ali Tazarv, and Yadollah Yaghoobzadeh. Parsinlu: A suite of language understanding challenges for persian, 2021.
Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. Xrv preprint ar Xix::306.6000 00912,,92222 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning, arXiv preprint arXiv:2211.01776, 2022.
OpenAl. Gpt-4 technical report, 2023.
Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024. URL htttps://operreieeee net/formm?i= KS8mIvetg2.
Pedram Rostami, Ali Salemi, and Mohammad Javad Dousti. PersianMind: Cross-Lingual Persian-English Large Language Model, 2024.
Oleh Shliazhko, Alena Fenogenova, Maria Tikhonova, Vladislav Mikhailov, Anastasia Kozlova, and Tatiana Shavrina. mgpt: Few-shot learners go multilingual, 2022. URL
https://arxiv http:://arxiv..ogabbb////ss www.//r/////....... 277820580.
Arun James Thirunavukarasu Darren Shu Jeng Ting, Kabilan Flangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine. Nature medicine, 1930-1990- 2023.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint XrXio22402.07827, 2024.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen 11m applications via multi-agent conversation framework. arXiv preprint XrXiv:2308.00 08150,855, 2023.
Wenxuan Zhang, Mahani Aljunied, Chang Gao, Yew Ken Chia, and Lidong Bing. M3exam: A multilingual, multimodal, multilevel benchmark for examining large language models. In shirth-seoenth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL //tpnreieee/eeeiie.e... wwwppeereviee.net/forrreeeeeeeee pet/forrmmiidhJPPATBB..
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models, 2023.
12

---

Published as a conference paper at COLM 2024
A
Data
Lege di taia بك حامل موg Sفbd ر1ا civ Byy jindua P.. كونسن sox. Sobim is فلوويطي Yowl Jرjو بنلد مور eنوننa del فون soy, 4 4,35 Lua oر ارووي مي احسسس
Question
listener with an eardrum area 60mm" perceives sound intenslly level from source as decibels. How much energy reaches the listener's car in 30 seconds in microjoules?
Correct Translation
listener with canal 60mmm perceives the sound intensity level from How much energy reaches the listener car 50 seconds?
decibels.
API Translation
Tawal bay Ae ماميي "عا ,/is 3) شاعر u9s Giee <Cuu s2) عون gud de al مرك0 مرم Gnd ممون 4oly Jokn ln uegfi
Question
In the following verse, which poetic devices has the poet employed? duck's feathers are white,
Translation
they've been washed The partridge's legs are so red, as if they 've been dipped in
soap
of blood"
In the following verse, which poetic options did the poet use? "It's white duck has put its clothes on soap
API Translation
partridge has put its leg in glass blood"
Question
+ري foylo هدي aiz has aie - jaj
Als G ااسسي 4/s = ز y كحن
Kian is the son of the sister of Zahra's Father, and he is the son of my mother's sister. My mother two sisters
Correct Translation
brothers. How many sisters
brothers
father
API Translatiin
Kian the son my aunt Zahra. have aunbs and 3 uncles. How many aunts and how many uncles does Zahra have respectively?
Table 4: Examples illustrating information loss in domain-specific translations from English to Persian Using the GPT-3.5 API.
2019
USS
2021
2023
LPS
2014
UPS
LSS
2016
2017
(a)
(b)
Figure Distribution of questions across publication year and educational stage
Main Category
Easy
Medium
Difficult
Sum
Social Science
678
1552
1194
3424
Mathematics
1461
3303
2300
7064
Natural Science
1299
2554
1999
5852
Humanities
724
1900
1841
4465
Sum
4162
9309
7334
20805
Table 5: Question distribution across main categories by difficulty level
13

---

Published as a conference paper at COLM 2024
Prysis
Medium
USG
Soleca
Bocka
Difficult
Easy
(a)
(b)
Figure 3: Question distribution with respect to categories, level of difficulty, and educational
stages. LPS: Lower Primary School, UPS: Upper Primary School, LSS: Lower Secondary
School, USS: Upper Secondary School
B
Models
GPT-4
OpenAl has introduced GPT-4 as a large multimodal model. However, the internal structure and operational specifics of GPT-4 are proprictary and not openly disclosed.
GPT-3.5
GPT-3.5, created by OpenAI, represents a significant advancement in natural language processing technology. boasts enhanced capabilities in understanding and generating human-like text, performing impressively across various language tasks like text completion, translation, and question answering, thanks to its expansive architecture and vast training data.
Claude3-haiku
The Claude Haiku model, developed recently by Anthropic, are proprictary models with undisclosed architecture and training specifics.
Aya
Aya, an open-source model by Cohere, is a massively multilingual generative language model capable of understanding instructions in 101 languages. Over half of these languages are classified as lower-resourced.
XVERSE-13B
XVERSE-13B, created by Shenzhen Yuanxiang Technology, stands as versatile multilingual large language model. Employing a prominent Decoder-only Transformer network structure, XVERSE-13B has a context length of 8k. It has been meticulously trained on vast and varied dataset containing 1.4 trillion tokens, covering more than 40 languages including Chinese, English, Russian, and Spanish.
14

---

Published as a conference paper at COLM 2024
Level
Easy
1600
Difficult
1400
1000
Count
800
400
200
Physics USS
Calculus USS
Socal Studies
Geometry UsS USS
Socal Studies UPS
Mathematics
Logic
Mathematica and Statistics USS
Persien Literatt literature LPS
Persian Lltarature UPS
Biology USS
Analytical Talent LSS
Mathomaticall and Logical Intelligencs UPS
Natural Sciences LSS
Mathematic USS
Persian Literature USS
Natural Sciences UPS
Thealegy LPS
Discrete Mathematics USS
Geography USS
Sociology USS
Economy
Persian Literature LSS
Socal Studies LSS
Mathematics UPS
Psychology USS
Mathemetica LPS
Philosophy USS
Chemistry USS
Speed and Accuracy UPS
History USS
Verbal and Lingulzsic Intelligence
Geology USS
Theology USS
Theolegy LSS
Probability and Statistics USS
Thealogy UPS
Sciences
Task
Figure 4: Question distribution across all tasks by difficulty level
XVERSE-7B
XVERSE-7B, much like its counterpart XVERSE-13B, is multilingual large language model developed by Shenzhen Yuanxiang Technology. It shares the same mainstream Decoder-only Transformer network structure but with slightly reduced parameter size of billion.
PersianMind
PersianMind is an openly available bilingual large language model. It enriches LLaMa2's vocabulary by integrating 10,000 Persian tokens and undergoes training on a dataset con- taining nearly 2 billion Persian tokens.
TOXL
TOXL originates from the fine-tuning of BLOOM models using xP3, a dataset that encom- passes multilingual content coupled with English prompts. The BLOOM models, which are extensive decoder-only language models, have been pre-trained on roughly 350 billion tokens. They share structural similarities with GPT-3.
15

---

Published as a conference paper at COLM 2024
mGPT
mGPT is an open-source adaptation of GPT-3, having multilingual capabilities. It undergoes pretraining across 61 languages, drawn from 25 diverse language families, leveraging Wikipedia and the C4 Corpus. Additionally, mGPT offers support for Persian language integration.
16

---

Published as a conference paper at COLM 2024
C Prompt template
1 متلس pgd دووم ambie مورد , باسعم 4ب) كزينماي جند سوالات زعر فر
كنبد مي o مناهدد
سولل
.uنن o داير از قطرر معادله .(m - r)x + (m + 1)y = معادل w!ll هر ازاي
اسسع كدام C داير محيط باتد C دايره روك A( A(-1,) 1)
تقله اكر
:زيوي كز
TVTx ()
Tı (T
r% or
TVTx (f
جوابد
Below you will find multiple-choice questions (with answers) regarding geometry for upper secondary school.
Question: For each m the equation (m - 2)x + (m + 1)y 6, is the equation of chord of a circle C. If the point A(-1,1) 1) lies on the circle C, what the circumference of circle C?
Choices:
1) 2/227
2) 2%t
3) 3%t
4) 2/3r
Answer:
Figure 5: Sample prompt-0 with English translation for enhanced readability
17

---

Published as a conference paper at COLM 2024
متوسساه pgd دووم aندسه مورد "ر (éWيly 4ا كزينما جند سوالات زبر خر
كنيد مي متاههد
سوال
اسسن C Co حااير از قطرك معادله .'m r)x + (m + 1)y & معادلة ه ازاا به اسسن كدام C داير محيط بانس C دابره روك A(- ), )) تقلم اك
a) 2/2
b) 2%
c) 3%t
d) 2//37
جواي
Below you will find multiple-choice questions (with answers) regarding geometry for upper secondary school.
Question: For each the equation (m 2)x + (m 1)y is the equation of chord of a circle C. If the point A(-1,1) 1) lies on the circle C, what is the circumference of circle C?
Choices:
a) 2/2m
b) 27
c) 37t
d) 2/3r
Answer:
Figure 6: Sample prompt-1 with English translation for enhanced readability
18

---

Published as a conference paper at COLM 2024
The following are multiple choice questions (with answer) about Upper Secondary School Geometry
سولل
.cum C هاار jز شطر معادله (m r)x + (m + 1)y & معادلة و اززاي as اسسن كدام C داير محيط باتس C دايرة رور ((-1,)) تقطل اكر
a) 2T
b) 2%
c) 3πt
d) 2//3
جواية
Below you will find multiple-choice questions (with answers) regarding geometry for upper secondary school.
Question: For each m the equation (m 2)x + (m 1)y is the equation of chord of a circle C. If the point A(-1,1) lies on the circle C, what the circumference of circle C?
Choices:
a) 2/2
b) 2%z
c) 37t
d) 2/3r
Answer:
Figure 7: Sample prompt-2 with English translation for enhanced readability
19

---

Published as a conference paper at COLM 2024
متوسساه pgà دووم aندسه مورد "ر (éwيly 4ا كزينما جند سوالات زبر فر كنيد مي متاههد
سوال
اسسن C Co حااي از قطرك معادله .'m r)x + (m + 1)y & معادلة ه اززا به اسسن كدام C داير محيط بانس C دابرة روك A(-1, ,, )) تقلم اكر
وزمماان كر
a) 2/2
b) 2%
c) 3%t
d) 2//37
كai فكر قدم as قدم باببد جواب
Below you will find multiple-choice questions (with answers) regarding geometry for upper secondary school.
Question: For each the equation (m 2)x + (m 1)y is the equation of chord of a circle C. If the point A(-1, 1) lies on the circle C, what is the circumference of circle C?
Choices:
a) 2/27
b) 27z
c) 37t
d) 2/3x
Answer: Let's think step by step
Figure 8: Sample CoT prompt with English translation for enhanced readability
20

---

Published as a conference paper at COLM 2024
D
Prompt & CoT
Main Categories
Prompt-0
Prompt-1
Prompt-2
Humanities
0.32
0.35
0.36
Mathematics
0.28
0.3
0.34
Natural Science
0.35
0.36
0.37
Social Science
0.43
0.42
0.42
Avg on all tasks
0.34
0.36
0.37
Avg on all questions
0.35
0.36
0.37
Table 6: Accuracy of GPT-3.5 across main categories for different prompts
Main Categories
Prompt-0
CoT
Humanities
0.36
0.28
Mathematics
0.31
0.41
Avg on all tasks
0.33
0.34
Avg on all questions
0.31
0.4
Table Accuracy of GPT-3.5 across main categories for main prompt and CoT
21

---

Published as a conference paper at COLM 2024
E Accuracy & candidate choice distribution
Categories
Aya
Humanities
Mathematics
Aug on tieks
0.77
05
044
0.37
0.26
0.3
0.29
0.6
0.26
0.24
0.25
Aug on all questiioss
0.76
0.89
0411
0.36
0.25
0.29
0.28
0.6
0.25
0.2
0.25
Table 8: Human vs model accuracy for main categories: model answers extracted via "Full Answer Probability" method
Main Categories
Human
GPT-4
Haiku
GPT-33
Aya
XVERSE-13B
XVERSE-7B
PersianMind
mTOXE
mGP]
Kandum
Humanities
0.71
0.47
0.43
0.33
0.32
0.29
0.27
0.6
0.27
0.23
0.25
Malhematics
0.72
0.41
0.33
0.52
0.27
0.26
0.27
025
0.27
0.23
0.25
Natural Science
0.79
0:49
042
0.37
0.33
0.29
0.2
0.26
0.21
0.25
Social Science
0.8
0.64
0.57
0.44
0.42
0.35
0.33
0.3
0.32
0.22
0.25
Avg. on
Avg on questiors
0.76
0.49
0.36
0.30
027
0.28
0.25
Table 9: Human model accuracy for main categories: model answers extracted via "Single Token Probability" method
22

---

Published as a conference paper at COLM 2024
Harka
Ava
PenianMind
Randam
Andlytica Thiant
0.99
0.52
0.35
0.35
0.22
033
0:31
0.36
0.25
6.2b
0225
Diiertte Vatherotios 186
0.23
0.25
0.36
0.56
0157
0.34
Fconumy 185
0.60
0.6
0.51
0.4
11:27
0.35
0326
1628
03
0.22
0.76
Geography
0.1
Coolagy L35
0.31
0.5
0.25
Hilex USS
0%
0.5g
0.47
0.36
0.32
0.55
0.29
0.31
0.27
0.25
0.26
Logk USS
0.88
0142
11.
0.31
11.2
0.35
0.29
0131
0.23
0.25
0.23
Logical Invelligance
LPS
Methematics 156
0.75
0.36
0.52
0.52
0.3
0.27
0.38
6.8
0.24
Mathumatizs LTS
0.88
0.49
0.37
0.36
0.27
0.26
0,07
0,55
0335
Matiamatios
Natural Sclences
0.61
0.5
0.76
Natural Sciences 138
0.7
0:48
0.53
0.3
03
0.76
0.37
0.77
Natural Sceecce
0.67
0.39
0.31
0-
0.76
Persia
Persian Literature L25
0.79
0.57
0.23
0.38
0.52
0.55
0.26
0.25
0.24
0225
Porsan Liceurae USS
0.59
0.25
0.32
0.28
1.2
0.25
0.26
0.36
0.26
0.25
0.25
Philosaphyy USB
0.32
025
0.27
0.25
Probability and Stamaice LS6
0.28
0142
0.37
0.32
0.25
0.25
0.25
0.22
0.66
Faychology 185
0.28
0.65
0.S
0.4
644
0.32
0335
0.35
3357
Stuties
0146
Socall
I17
0.36
01.8
0.22
Socooooyy US$
0.2
0.44
0:44
0.56
1.37
0.52
0.7
0.38
0.36
6.36
0.25
Speed and Accuracy UPS
0.31
0.38
0.29
0.7
0.25
0.28
0.27
0.2
Theology US
0.00
01:00
02
Thoslogy UDS
0.91
0.5
0.7
0.50
0.48
0.44
0.37
0.32
0.37
0.19
0.24
Thensogy LSS
0.54
0.44
0.30
0.30
0.26
0.26
(1.6
0.34
02
022
Linguisae Intelligence
025
ong
questions
0.99
012
0.55
Table 10: Accuracy of humans and different models across all subjects and educational stages: model answers extracted via "Regex" method. LPS: Lower Primary School, UPS: Upper Primary School, LSS: Lower Secondary School, USS: Upper Secondary School
23

---

Published as a conference paper at COLM 2024
Harka
Ava
PenianMind
mT0XL
Randam
Andlytica Thiant
0.99
0.52
0.35
0.35
0.22
03
0.2
0.27
0.22
0.25
0:25
Diiertte Vatherotios 186
0.23
0.25
0.36
0138
0.55
6.37
0.34
Fconumy 185
0.6
0.63
0.51
0.4
01:5
0.32
0.34
03
0.71
0.36
0.76
Geography
0.1
0.34
0.36
Coolagy L35
0.31
0.24
0.25
Hilex USS
0%
0.5g
0.47
0.36
0.53
0.3
0.29
0.38
0.21
0,27
0.26
Logk USS
0.42
11.4
0.31
0.3
0.25
0.20
0.29
0.26
0.25
Logical Invelligance
LPS
Methematics 156
0.75
0.36
0.52
0.51
0.25
0.25
0.25
0.21
0.36
0.24
Mathumatizs LTS
0.88
0.49
0.37
0.36
11.2
0.25
0.26
0.23
0.24
0335
Matiamatios
Natural Sclences
0.61
0.38
0.19
0.76
Natural Sctences 138
0.7
0:48
0.51
0.29
0.16
03
0.37
Natural Sceecce
0.67
0.36
0.11
0.79
0.18
0.3A
0.76
Persian Literature L25
0.79
0.57
0.23
0.38
0.35
0.32
0.26
0.25
0.2
0.31
0225
Porsan Liceuraee USS
0.59
0.25
0.32
0.28
0.25
0.26
0.25
0.23
0.25
0.25
Philosaphyy USB
0.32
025
0.36
0.22
0.25
Probability and Stamaice LS6
0.28
0142
0.35
0.32
0.25
0.25
0.2
0.23
02
6.36
0.56
Paychology 185
0.28
0.65
0.S
0.4
0.36
0.35
0.36
0.51
0.24
0.31
347
Stuties
0.3
0:17
Socall
I17
0.36
0.37
0134
0.22
Socooooyy US$
0.2
0.44
0:44
0.56
1:37
0.3
0.7
0.34
0.23
0.36
0.55
Speed and Accuracy UPS
0.31
0.38
0.24
0.36
0.25
0.22
0.36
024
Theology US
0.00
0.36
0.31
00.35
0.33
0,37
02
Thoslogy UDS
0.91
0.5
0.7
0.50
0.46
0.45
0.36
0.38
0.2
0.43
0.24
Thensogy LSS
0.8
0.54
0.44
0.30
0.38
0.25
0.25
(0.2
0.22
0.27
0226
Linguisae Intellignnce
ong
questions
0.27
0.25
Table 11: Accuracy of human and different models across all subjects and educational stages: model answers extracted via "Single Token Probability" method. LPS: Lower Primary School, UPS: Upper Primary School, LSS: Lower Secondary School, USS: Upper Secondary School
24

---

Published as a conference paper at COLM 2024
XVERSE-1
PenianMind
Randam
Analytical Talvent 185
0199
0.34
0.36
0.23
0.3
0.33
0159
0.35
0.35
Calculii 185
088
18:2
0.20
18.6
1822
0.34
0.26
1625
0.7
0.22
0.55
Chematry
0.31
0.31
0.76
Matherattos
Ecceumy 135
0.51
0.4
0.22
0.34
0.33
0.55
623
6.25
0.36
Congaaghy USS
085
0.5g
015
0.38
0.31
0.22
0.3
0.28
0.26
0.26
0.24
Cooligy
0.03
0-4
0.31
0.21
0.27
0.25
0.25
0.25
USB
0.47
0.29
11.0
0.26
USS
0.58
0.42
0.4
0.:3
0.22
0.31
0.29
0.23
625
0,33
Methumatical und Logical brollggmrr LPS
0.48
0.43
0.5
0.52
0.23
0.2b
0.29
1.25
0.25
0.24
0.25
Nathemansa
L55
0.36
0.24
Mathumatizs UPS
0.8
0.49
0,37
0.35
0.25
0.24
0.25
0.27
022
6.5
023
Mattemztics Las
0.8
0.33
1.33
0.:31
11.2
0.35
0.21
0.33
0.31
0.35
0.55
Mithematios and Statitiios USB
0:37
0.39
037
0.78
0.2
Natural Sctences
11.90
0.46
0.790
0.5
0176
0.17
Natural Sccence IPS.
0.81
0.77
0.67
0.46
0.37
0.31
0.29
6.15
0.7
Persan Liicrature LPS
0.84
0.33
0.42
0.22
0.37
0.3
0.25
0.24
0.22
0.22
0.25
0.25
Persin Litervase USS
0.59
0.35
0.32
0.234
0.25
0.25
0.26
0.25
0.26
0.22
0.25
Philosgphy USS
061
0.53
0.51
0.30
0.32
0.26
0.27
03
042
Physsis USS
Probability
Paychology USS
0.28
0.63
0.5
9.4
0.3
0.34
0.36
0.29
0223
6.25
327
Social Stuthes LFS
0.94
0.88
0.56
0.60
0.55
0.41
0.9
027
Sodil btudies
0.07
0.43
0.35
0:31
0.28
0.22
0.26
Socioloyy USB
0.44
0.56
0.26
0.37
0028
0.55
Speet and Accuracy UPS
0.8
0:41
0:31
0.38
1:27
0.55
0.2
0.76
0:24
034
Thanlogy LPS
0.81
0.82
0.23
0.62
0.47
0.36
0.35
0,27
0.42
0.24
0,27
Traslogs
0.30
0.24
Thesiogy USS
0.5
0.54
0.44
0.33
0.3
0.66
0.26
0.3h
0.32
0331
026
Verbal and Lingutica: Intellignnce UPS
0.2
0.54
0.85
0.42
0.35
0.39
0.25
0.29
0.22
0.25
Roology
0.26
0.31
0.2
0.26
0.77
Ang. on losks
0.7
0.86
0.39
0.31
0.29
0.36
0.27
0.24
0:25
questions
Table 12: Accuracy of human and different models across all subjects and educational stages: model answers extracted via "Full Answer Probability" method. LPS: Lower Primary School, UPS: Upper Primary School, LSS: Lower Secondary School, USS: Upper Secondary School
25

---

Published as a conference paper at COLM 2024
Human(Difficutt)
Human(Medium)
Human(Easy)
GPT-4(Difficuit)
GPT-4(Medium)
GPT-4(Easy)
GPT-3.5(Difficult
GPT-3.5(Medium)
0.9
GPT-3.5(Easy)
Aya(Difficult)
Aya(Medium)
Aya(Easy)
Haiku(Difficult)
0.8
Haiku(Medium)
Halku(Easy)
0.7
Accuracy
0.6
UPS
LSS
USS
Education Period
Figure 9: Accuracy of models and humans across three difficulty levels, segmented by question educational stage.
26

---

Published as a conference paper at COLM 2024
Human(Difficult)
- Human(Medium)
Human(Easy)
GPT-4(Difficuit)
GPT-4(Mediuu)
GPT-4(Easy)
GPT-3.5(Difficult
0.9
GPT-3.5(Medium)
GPT-3.5(Easy)
Aya(Difficult)
Aya(Medium)
Aya(Easy)
Haiku(Difficult)
Haiku(Medium)
halku(Easy)
0.7
Accuracy
0.6
0.5
0.4
2014
2016
2018
2020
2022
Years
Figure 10: Accuracy of models and humans across three difficulty levels, segmented by question publication year.
27

---

Published as a conference paper at COLM 2024
Main Categury
Difficatty
Homan
GPT4
Ava
mGrt
Bumunibia
Difficuh
634
0.4
0.36
0.31
0.29
0.28
0.56
0.6
0.2
0.25
0.24
Bemunitics
Mathematios
Panicolt
6336
0:33
0.27
0176
0.38
0.3
0.55
0.24
Nathematios
Fasy
6495
0.36
0.33
0.3
0.29
03
026
076
076
0.24
Mathematios
Mediim
6344
0:41
0.27
826
0.5
Natural Scacice
01:88
0.25
Natural Sciense
Mediun
692
6447
0.39
0.34
0.5
0.31
0,27
0.5
0.26
0.25
0.26
Social Sckence
Diticull
0:38
0:54
0.86
0.36
0.33
0.31
0.27
0.5
0.2
8.25
0.20
Aug all ttaks
Aug taalos
6.76
6.52
0.45
0.38
0.54
0.53
03
0.29
0.29
0.25
0.25
Arg un all queetives
Nog om ill queetions
0.76
0.49
041
0.36
0.30
03
0.29
0.25
0.28
0.25
0.25
Table 13: Human vs model accuracy for all tasks: model answers extracted via "Regex" method
Sibepry
Farmembre
Lamaaa
cot
1122
1120
Humunitiss
Medium
0.86
0.5
0.33
0.31
0.28
0.26
0.22
0.26
Mathematiss
Dinizult
0:36
0:36
0:31
013
0.26
0125
0.36
033
022
0.3
0.26
Mitthemakss
Easy
0.95
0.09
0.24
0.27
0.39
036
027
0.3
0.24
Scaence
Diffcult
Natural Scaenes
Easy
C67
0.48
0.41
0.37
0.31
0.2
0.2
0.99
0.25
Natural Beicnes
Modium
692
C47
0.39
0.34
0.31
0.3
0.26
0.5
0.2
0.1
0.36
6.4
0.1
Social Sekence
Medhum
697
6A6
0.46
0.40
0.58
0.34
0.3
Ang all Traks
Ang on Trnks
608
108
11:3
0.3
11:1
1070
1227
1070
1022
1025
Ang on all questions
Nig on questions
0.6
0.89
0.41
0.36
0.32
0.29
0.28
0.27
0.28
022
0.25
Table 14: Human vs model accuracy for tasks: model answers extracted via "Single Token Probability" method
Category
Dilically
Human
CPT4
Apa
XVERSE1E
Fimmmttics
Lintook
0331
0.4
0:36
0.33
01.3
0.30
11255
11:3
U1:11
Hummniities
Medium
0.86
0.5
0.36
0131
0.26
0124
0.26
Muthematies
Dinizut
0:36
0:36
0.31
01.
0.33
(136
0.36
0.3
014
013
0.24
Mathematts
Easy
0.99
C.99
0.38
0.35
0.25
0.26
0.28
0.34
0.5
0.3
0.24
Muthematiss
034
C:41
Scaznee
0.27
Natural Scaenes
Easy
C.7
0.68
0.52
0.37
0.32
0.25
0.2
0.23
0.25
Natoral Scaecce
Medium
699
0-47
0:39
0.54
0.20
033
0.36
0.34
6.7
015
0.76
Secial Sclenor
Social Sekenee
Medium
697
6.6
0.45
0.34
0.36
0.34
0.20
0.29
0.26
0124
Ang on all licks
Ang licks
678
002
633
0.33
0.5
0131
1229
0.35
0227
0024
1025
Arl on questions
Nog An questions
076
0.89
048
0.36
0.28
0.28
0.36
0.36
0.36
0.24
0.25
Table 15: Human vs model accuracy for all tasks: model answers extracted via "Full Answer Probability" method
28

---

Published as a conference paper at COLM 2024
No Aneover
SK
(RogEx)
Figure 11: Selected choice distribution of different models: model answers extracted via "Regex" method
29

---

Published as a conference paper at COLM 2024
No Aneover
SK
XVERSE-138
(RagEx+)
Figure 12: Selected choice distribution of different models: model answers extracted via "Regex" method
30

---

Published as a conference paper at COLM 2024
No Anewvr
SK
(Single probabilit))
Figure 13: Selected choice distribution of different models: model answers extracted via "Single Token Probability" method
31

---

Published as a conference paper at COLM 2024
No Anewvr
SK
(Full probability
Figure 14: Selected choice distribution of different models: model answers extracted via "Full Answer Probability" method
32

---

Published as a conference paper at COLM 2024
F Trap
Buuaniicc
0.1 2.22
0.47 0.53
843/00
0.33 0.22
0.81 0..24
03/00.2
0.27 7.22
0.28 2:
0.25 0.225
0.25/03. 0.23
0/35/0.0
Social Schenoe
088 08876335 6635
0.64 / 042
055
10.3
0.446.25
0.29
00.50.35
038 / 0.28
0.38 0.377
032/
0.33 7022
0..2
0.25 /022
trsks
117 7638
05/0.8
044.00 0.3
031/00
0,1
0:39/035
7335
Arg on all questions
0.2676366
0.86 /83
841
7631
036
0.3 /0.0
03/022
0222
0.820025
0.26 0.76
0035 0.36
0.25 0335
Table 16: Accuracy of humans and various models on main categories, including all ques- tions/ taraptt questions.
G Statistical tests
Model
T-Test response str
T-Test Question Body
T-Test
XVERSE-13B
4..2 0.0
-3.18 / 0.0
-7.52 / 0.0
mGPT
-0.13 0.9
0,06 0.95
2.35 / 0.02
PersianMind
-0.37 0.71
-2.4 / 0.02
-3.65 / 0.0
GPT-4
7.2 / 0.0
-11.54 / 0.0
-26.16 / 0.0
Aya
1.28 / 0.2
-4.36 / 0.0
-11.03 / 0.0
Haiku
-11.28 / 0.0
-13.08 / 0.0
-22.56 / 0.0
mTXXL
-0.3 / 0.77
-0.65 / 0.52
-4.67 0.0
XVERSE-7B
-0.19 0.85
-3.05 / 0.0
-6.51 / 0.0
GPT-3.5
-1.44 0.15
-7.03 / 0.0
-14.22 / 0.0
Human
NA
-11.87 2.17
-12992 5.08
Table 17: T-test comparison of question and response string lengths among the models
33

---

Published as a conference paper at COLM 2024 2024
Length rasponse ganemated by GPT-4
Langth deser plivo
descripive answer
Length questonss
Figure Comparing the lengths of GPT-4 generated strings, descriptive answers, and questions

---

Published as a conference paper at COLM 2024
H
Sample questions
تعر & (x) (-1)² (-1)² ( مورت as طبعي اعداد دiمنه 4ا و f توايع
(سسل) صحبح جزء زماد of 1) اسساع كدام (gof) (1)1) (g0 (11-))(880-11)) (20 حامل تدال
مفر .())
.(T)
r .(T)
Y .(F)
The functions f and are defined on the domain of natural numbers as follows: f(x) (-1)² and g. (x) (-1) What is the result of (101) (-f)(-))) (-f)))((102))- ?(102)? (() denotes the floor function).
1. zero
2.
3.
4.
Mathematics
USS
Trap:-
Difficult
1
f)
GPT-4
GPT-3.5
3 كزينه جواب توفيج
f) = (-1)* الب
2)g(z) (-1)
3) f)(x))(x
8(f(x) f(x) (- (-1)(x)
(g-((f))xx =
8(-f(x))=f(x)) (-1) -fxx
() 50
6) (g8
48
7) (gof) ((01)-
(- ((102)
50 48 = 2
,اml3 كزينه محيح كزينه بااراين
<pad> 1)//>>
Aya
Haiku
35

---

Published as a conference paper at COLM 2024
. افارد كششالت أصولل أز امل كدام or poimn كامل بمسوووتت |, تاقص انشاا داريم تمايل مما عبارت
RuiS ~ي
صشابيت .(1)
تكميل .(t)
استمرار .(t)
مجاورت .(F)
The phrase "We tend to perceive incomplete objects complete" refers to which principle of Gestalt principles?
1. Similarity
2. Closure
3. Continuity
4. Proximity
Psychology
USS
Trap:-
Easy
Jممل IT
Joمل (T
GPT-4
GPT-3355
اسس تكميل (2 كزينه محب باسج كفالت رواننااسي هي تكميل أمل داريم تجايل كم دارد افارد ابن به كامل صورت به را ناقس انشبا شكافهاي و كصسودها ببينيم أمل ابن كنبه بt فصنا I, موجود
الشبا العبب ج محدمد
توصبح
درك كامل صورت 47 | ااقص
كنومم حي
ديكر كغغالت
اصول تنمها كز سابر
دارند: افارم
مورت به ر مشايه الشياي منابية (1
كنيم مي درك واحد كروم
خطوط طاريم تمابل ما :n استمصار (3
بيوسة جة بكاار صورت به و انكال
بيينيم
مم به تزديك انشاي تتد مجاورت (4 درك واحد كروو صورتت كنيم مي
<pad> T>/s>
Aya
Haiku
36

---

Published as a conference paper at COLM 2024
بودع أبرننن جنن ما سلسا" كدام حكوست با زمان as ايييشمم as مسسوم جادد كنايش
ساساني - ابن جه .()) انكاايي ابن جة .(T)
ساماني حان .(t)
المكاي مان .(F
The opening of the Silk Road coincided with the rule of which dynasties in China and Iran?
1. Chen Sasanian
2. Chen Parthian
3. Han Sasanian
4. Han Parthian
History
USS
Trap:-
Medium
انشكاني حان (f
ساساني حان (t
GPT-4
GPT-3.55
الخكاني حان (4 كزينه محيج جواب
اسس
توفيح
حكومت دورر در ابريشم جادد
الخكاني سلسله -اك كشايش اعران در
در حان سلسله
206 سال أز جين حان سللااء حكومت ميلاي 220 عا ميلاد از بيش
دند. مرك مي
247 سال از ابران در امكااي سلسله حكومت ميلادي 224 ميلاد از بسق
داقتسد
الريشيم م جاده كفايش باايااين
فر مان سmسطه حكومت يا ممزمان
<pad> p)/s>
بود ابران در الشكان سلساه جمن
Aya
Haiku
37

---

Published as a conference paper at COLM 2024
قيمت (سوه) أكر حلء مر و موا فروختء مشتر به تهايت ور توليد حلع م au طي Yl بك شده برداتتت بو قبل حلل م ور كم افزودمهميي اارفف ماليات اعتعار بالد تومان ir, 4ا باار اا5 أنقام باند تومان dd با بابر
است تومان جند سوم جلل مر هر الزودد اارفس
ماللات با كاا فروش فيمت (افي)
اسسك تومان جند دوم رلم مر ور اقزوده ارزفس x ماليات يا كاS فروش قيمت (ج)
ارزفن ماليات إبرخ اسسو تومان جند شدم ورك جمم جمع و الزودد اازفس بf ماللات مجموع
(2)
(مسي درمد اقزوده
AYA ( . (4.1,, ITD (لفد) .(1)
AYD (c.) .1,000 (4.15,91) 9A1 (لفي) .(t)
PTO (,.)1,00- (4.1,, (لفي) .(5)
9TO (,.),, (4..1, 9A1 (لفل) .(F)
A product goes through three stages of production and is ultimately sold to the customer.
If the price of the product in the final (third) stage is 12, 200 Tomans and the credit for the value-added taxes paid in the previous stage is 550 Tomans, then:
(a) What is the selling price of the product with value-added tax in the third stage?
(b) What is the selling price of the product with value-added tax in the second stage?
(c) What is the total value-added tax collected?
1. (a)13, 125, (b)10, 500, (c)875
2. (a)13,983) 981, (b)11, 550, (c)875
3. (a)13, 125, (b)11, 550, (c)625
4. (a)13, 981, (b)10,50) 500, (c)625
Economy
USS
Trap:-
Difficult
.00,500 ( .13, 125 (لفي) كزببه
(c .1,,50 (ạ .1, 125 (لفد) (T
875 (z
625
GPT-4
GPT-3.5
T mnef محمع حوا
الووو Fid x مالت y15 فرون قينت
قمان 13, 125 سسم مرحل
الوودم إنف بالات 4 yıs فروف قينت (يع
قماا 15,550 دوم مرحل
مماردد الجودد ارفف باللات محسr l
زواز 625
وهيي L اسعم تمان 12,500
بال ys زروي فوست
بالاك
ووك تبايةة 625 إوار 5% بنو الزردد عوامد تواي 13,125 xn
براشنت الزودد الززق x اللle lego مرحل
إنن احتسب
ومات
باج
موم مرحل مر ys قوين بست مالتة by خوادد تماا 11,500 aد جواادي ازوددم انن x اااك مجموع ومان 625 4y واo شرطء س
<pad>.c. است محيج i كزين 

Aya
38 Haiku

---

Published as a conference paper at COLM 2024
قشت اولي كرنفد ورزش خبووة شفر جند برييي دااق از و شد o بتجوزو كلاس وارد معلمي نفر ميك كفت ججارمي ماعمه أز نفر gemh كفت سووي ماع تفر كهن دومي ماته أز نفر ججبرر
كساني راست كرطاند ورزش i5 كساني بااس معلم ماع أز كدام حهيج :هت بنجمي ماجج ز! كردوندك ورزش نفر جند تويدد دروج تكر ورزش -5
ققر بكك .()
نفر دو .(t) نفر سه .()
اسسن تكردم ورزش كسي .(F) A teacher entered a class of five students and asked how many of them had exercised yesterday. The first student said, "Four of us," " the second said, "Three of us," " the third said, "Two of us," the fourth said, "One of us," and the fifth said, "None of us." The teacher knew that those who had exercised were telling the truth, while those who hadn't t were lying. How many students had exercised?
1. One student
2. Two students
3. Three students
4. No student has exercised
Analytical Talent
LSS
Trap: 4
Difficult
نفر ( (
نفر awo
GPT-4
GPT-3.5
فهر gd كزينه است صحيح باسج توفيي
أموزنن دانش الطبارات توجه
i5 ماا تفر حجهرر فهت اوبي
اسسف دروع
is æا أز تفر منم كفن دومي
اسسن درست
is <اع iز نفر ددو اهف
سومي
استف رسست
is مما لز ششر ميك كفن جهاريي
اسس دروج
is ما محية كفي بنجمي
اسسن دروغ
ورزن داتسااوززان نقر دو بناانين كردط
<pad> />//>>
Aya
Haiku
39

---

Published as a conference paper at COLM 2024
مسنندع موارد كدام
مصاديق بمترتبب زير عبررات
ديكران تاطر كنف
مسارران ساير as توجه بمدو اتويوسس
ور أمنق دادن كوش
سالمندان به احترام
اججمامي اارشس فردي كنف اججماعي كنف .())
اججمااي منجار فردي كنف اججممعي كنف .(T)
اججماكي منجار دروني كنف فردي كنف .(t)
اجتماعي ارزشس رروني كنف فردي كنف .(F)
The following phrases correspond to which of the following scenarios?
Observing others' actions
Listening to music on the bus without regard to other passengers
Respecting the elderly
1. Social action Individual action - Social value
2. Social action Individual action - Social norm
3. Individual action Intrinsic action Social norm
4. Individual action Intrinsic action Social value
Sociology
USS
Trap:-
Easy
فردي كنف اجتمالي كنش (T اججمملي هنجار
فرد كنف
اجتماعي كنن (T اججمماي منجار
GPT-4
GPT--3.5
اججمالي كنش (2 كزينه جواب
اججماعي منجار فردي كسف
توصسي
كنف ديكر به ناططر كنف
اجتماعي
بهدون اتويوس
در أمنك دادن كوش
فردي كنف اان صسا سابر توجه
هنجار سالمناانن به أحترام
اجتماكي
ترتبب as ك5 2 كزبنه بنبراين
فردي كنف اجتماعي كنف مصادي
كندم مي بيان I اجتماعي منجار و
<pad> 1)//s>
اسست محيج
Aya
Haiku
40

---

Published as a conference paper at COLM 2024
داردع مبالةه بيت كدام
دارم خوف خود خاطر من كم اسست أن اعتز است قنر هر جبان شادي غم جون حاففا -() ففا كن تغبير بسييي نمي نو اي تدادن كنر 1, ما فيكنامي كوي فر .(T) باد باد كاران روز أن باد بااد باد باد دوسساارا وصل روز -(1) مكتيد يرامم جقنم إز اخبرر و إوسسان اصس سيلبب a خانه دمم كربه زز خوامم .(F)
Which verse has Hyperbole?
1. O Hafiz, since the world is passing through sorrow and joy, It is better that I keep my heart content.
2. They did not let us pass through the alley of good reputation If you don't like it, change your destiny.
3. May the day of reunion with beloved friends be remembered Remember the days of old.
4. Tonight, I cry so much that seems if flood has come into the house / Do not tell my friends that my eyes are full of tears.
Persian Language and Literature
UPS
Trap:-
Easy
بo خانه دهم كربه أز خواهم (f كزينه أز ايرر U اووسان امشب سيلاب
به خانه دهم كربة jز خواهم (F خير Iا دوستان امشب سيلاب مكتيد أمم جشم
مكتيد أبم ب جشم
GPT-4
GPT-3355
بيت ابن هر دارد ميالة كزبنه كريفه أز خواهد مي ك5 كويد مي شاع أبن ك5 كند تبدبل سيلاب به 1 خانه
<pad>.o دار مبالن f كزبنه 

اسسن اميز النهه مباا بازن
Aya
Haiku
41

---

Published as a conference paper at COLM 2024
داردع الشار فسيك ابماي ويزوي به مويد كدام د
مف جايكزين أب جركم طريق أز انسان ف مصر از بس كم است أبي -() أبرفني روحخامما رسواات فر موجدد أبدار للبهماي .(t)
تداانف فرار أب جرحة ف تداند محوس زياد اعماق فر .(T)
بستف فراوان املاح داراك عمووا كم ابحااي .(F)
Which case refers to the characteristics of fossil waters?
1. It is the water that is replaced through the water cycle after human consump- tion.
2. The water-rich layers found in river and groundwater sediments.
3. They are trapped deep and do not participate in the water cycle.
4. Waters that generally have abundant salts.
Geology
USS
Trap:-
Medium
محجوس زياد اعماق هر ot كرينه تدارند قرار أب جرحة ف شداند
و ماند شد محيو زباد اعماق هر (t تلارن: قرار أب جرخة فر
GPT-4
GPT-3355
اسسن It كزينه محبع جواب اللاق أبهابي as فسجعي أبهاي زمين زباد اعماق فر كs مووود قرار أب كة بر فر و فداند محبوس زمان طول در أبباا ابن نداننف للييماي زميففاسس فراندماا دايل به كرففاان جاي زمنن عميق
عنوان به أبب خح بر در كرف قرار شونم مي شناكعه فبييي ابهاي دبكري وو
دبكر كزينمماي 4ا موططط مر as دارند انشار أببا إ! نسسنندة فسمعي أسماي
طبيتي اي جركم توميف (1 رسويات در موجود أبماي توسي (2
ابرقتي و روداناما
<pad>(* 

املاع ج: و شور أبماي توصيف (4
Aya
Haiku
42

---

Published as a conference paper at COLM 2024
mmm با برخوود هر فرد دست كشينن عقب انعكاس فر
كتكس شر بلختمصاي حراادء مووف جند اسسع فرست داز
عسي دستكاه حري بخش به كند مي منتلل و خرد كيرنده بيام كم عمي باخته ه داردف التصام محيط
اررباط او جددصتماي باختمماي با دارند نعلق نخاعي عصب كه عميي بلتتمماي أز بخشي كنفد مي بررررر ويواا
بنانسيل هر تغيرري كني هي
(سينابس برفررر همايه بازو ناحية عخله با كه عممي باخته هر
اسن دادم غ أن الكتربكي
باكتمماي با دارد
قرار خااتسب ك
ماده فر انناا باأتماي جمم كه عصسي باحتدماي بحفي كنند مي برفرار (سبنابس همايه حسي عصمي
بك .1)
.()
.(T)
من .(F)
How many statements about the neurons involved in the withdrawal reflex of the hand when encountering a hot object are correct?
Each sensory neuron that transmits pain signals has a specific connection to the motor part of the peripheral nervous system.
Some of the nerve fibers belonging to the spinal nerve establish special connection with the multiple nerve fibers.
Any sensory neuron that synapses with the muscle the forearm changes its electrical potential.
Some of the nerve fibers whose cell bodies are located in the dorsal root gan- glion synapse with sensory nerve fibers.
1. One
2. Two
3. Three
4. Four
Biology
USS
Trap:-
Difficult
ججر (f
(سسl الشتاه F و r . ككزرنم am (
GPT-4
GPT-3.5
ترسح لنف 1/ aas محب جواب
سنتل و خوة كرندء باu uمي بااب و محيب عمي stiiws حسي júd ككب
نخايي تصي كي بحy دزد اكتممص اlowsاyy jl يخفي
و) ملمرواي بلككماي د تعاق كis Jbl sooje زنيطط عبمسنت
<pad>(r 

Aya
Haiku
43

---

Published as a conference paper at COLM 2024
L جابة مزارء صدنان صد با صدنان و مزار دمكان با دمكان مزار بكان بكاا ركم .1 190FAT sue
اسسن جند جديدد عدد در كان هزار و صدكان رقم فو Egaoco ملgدددد
9 .(1)
A .(T)
A .())
F .(F)
In the number 195483, the units digit swaps places with the thousands digit, the tens digit swaps places with the tens of thousands digit, and the hundreds digit swaps places with the hundreds of thousands digit. What is the sum of the two new digits in the hundreds and thousands places in the new number?
1.
2. 8
3. 5
4.
Mathematics
LPS
Trap:-
Difficult
1) 9
r) 8
GPT-4
GPT-3.55
ااm 8 (3 كزينه محيع جواب 4 بكان رقم اك 195483 عدد ور موار دماان 4ا ممقان مزاا بكان جابة مزار صدكان يا فان مب 951384 صورت جديد عدد شوند بودف خواهد
رقم د جديي جديد عدد بنن در 17 4 بربر (8) مزرراان (9) كان صبد اسسن
<pad>(t (r 

Aya
Haiku
44

---

Published as a conference paper at COLM 2024
اسسي) ft برابر امن اتهي أعدد اسسن فرست زبر مطالب از مورد جند
أرايش و كلييي كلر هيدرو با أحن زنك واكتش أز أمد بهددت نمك a سازندد كانيون (i)
داردف نصمير به زير بك خود الكتررني
خانوااد ععو تششمين الكبي خانوااد عبو سشمين مر كوالللسس بووندااي شمار تغاوت (4)
اسس r برابر ألكازوا
< SFF + rH+O SO₁ + FHF > واكش و HF توليد منوسط سر (4) بابر Oi .mol.min - برحسب بر اسسن mol.s برحسب ب SFf مصرف منوسط سرك (سسي) 181, بربر واكتش طرفف أحجم
دارفف فراواني كارررد الكترونيك منعت كر كه است مواد از دستممي as منعلق rr Ge عنصر (c)
.(1)
Y .(T)
Y .(²)
.(F)
How many of the following statements are correct? (The atomic number of iron is 26.)
(a) The constituent cation of the salt obtained from the reaction of iron with hy- drochloric acid has half-filled sublayer in its electron configuration.
(b) The difference in the number of covalent bonds in the sixth member of the alkynes family and the sixth member of the alkanes family is equal to 2.
(c) The average rate of production of HF in the reaction < SF4 2H20 - SO2 + 4HF >> in terms of mol.min- -- is 15 times the average rate of consumption of SF4 in terms of mol.s- (The volume of the reaction vessel is 16L.)
(d) Element 32Ge belongs category of substances that are widely used in the electronics industry.
1.
2. 2
3. 3
4.
Chemistry
USS
Trap: 1
Difficult
2
3) (4...))
GPT-4
GPT-3.5
(سسن ترست موود 3) كريي بايي
برصحا
Swssy نسك سلزنده كاتوون لسف فرست القنع (F=2 امن زنك واكس أسده يك خود الكتروني ارابش كلكك L
دارب ليمير زززز
كواانسس بوندماي شمار تااوت سسف ترست بيع (حتزي الكررا خلوالد عفر تشسس L' بار (ي) (مكو الكارحن غاووددء عقو ششسمين
<pad>(T 

اسن
Aya
Haiku
45

---

Published as a conference paper at COLM 2024
كsب است كسي تيكيت انسان أفلااون اخخلقي تطريه طيق
كند توجه اسس فوه عال كم عقل قوم به تنها خودء فراي ميان .() كردد ارااته عدالت شجاعت تواشة حكم ففيلت جعار به .(T)
ببردازن ففيلتها أنجام به بكسم و كرده دورك طببع افعال امبال از .(t) بكيدد ايمعددد و بنن اطارر و كرده اسساددي غصب و شجوت فواي مغابل در او عقل .(F)
According to Plato's 's ethical theory, a fortunate person is one who:
1. Focuses only on the power of reason, which is the highest faculty, among their faculties.
2. Embodies the four virtues of wisdom, self-discipline, courage, and justice.
3. Avoids natural desires and actions and solely engages in virtuous deeds.
4. Their reason stands firm against the forces of lust and anger, taking charge of managing the body.
Philosophy
USS
Trap:
Difficult
حكمت ففيلت جبار بs (t كرينه عدالت شجاعت تبشتري
حكمت ففيلت جبار به (8 عدالت شجامت ك تبشسدر
كر أراسنه
كرهف أرسسنه
GPT-4
GPT-3.5
اسسن 2 كزينه محبح باسع اسسان افلاطون الاااق تطلريه طبيق جعار به كم اسست كسي تيكيخت فار كرسشت حكمت اصلي ففبلت
كردده أرانه عدالت و شجاعت
جبا ابن كم بود معتقد افلاطون
أساا سعادت الخلقق اساسس ففيلت ابن تحقق فيالل به بايد انسان است
باشد خود زند فر ففابل
هستند درست نيز ديكر كزييمماي قود به تببا 1 كزينه تيستند كامل اما
كم حالي در كنده مي افارد عقل ممد بايد انسان بهد معتفد افلاطون ففايل تحفق جهت در را خود فواي بs تبباا كزينه كردد كار به اخلااقي افاراء طبيي افعال و امبال إز دورك م أفلاطون كs حالي در كددء مي ففايل تحقق دنالل ds بايد انسان بود به تبجاا نيز 4 كزيته باشس اخلاقي افارم بدن اداار در عقل تقشي معنفد افلاطون as حالي در كسد مي را خود قواي همد بابد انسان بود كار به الللاق ففايل تحقق جهت
<pad>(t 

كيدد
Aya
46 Haiku

---

Published as a conference paper at COLM 2024
أاعمنام الكتربيي بز داراك مفابه كوجك خيلي فلزي كم دو عم +ه و كر اك كنيد مي وارد /AN الكترركي ترووي -ع x و دارند قرار مم سات &. فاصله وارد عم بo نيوونن 1/9 الكتربكي تيروك كنيم دور فع أ! قنك فاصله ممان بي دويارة دهيم تماس
ف و مستتل 19p1 > 91999 91
(k 9 x ). C
"N.nd اسستو كولن ميكرو جند qf كنت مي
.()
r .(T)
.00
.(F)
Two very small and similar metallic spheres with unlike electric charges 91 > 0 and [92] 191l are placed 60 centimeters apart and exert an electric force of 0.9, N on each other. If we bring the spheres into contact and then separate them again to the same previous distance, they exert force of 1.6, N on each other. What the value of 91 in
microcoulombs? (k 9 x 10 Nim²
C
1.
2. 2
3. 10
4. 20
Physics
USS
Trap:-
Difficult
(Y كزينه
r) 10
GPT-4
GPT-3.5
دي شدم داده الللمات توجه يا الككريكي بار مقدار توانبم سوال
كشيم محاسيه 1, 91
بين الكتربكي تبروي برا كولن قانون اسسن زبر مورت الكتريكي بار
= k k91g2
بين الكتربيي تبروي إول سوال در
تيووون 0.9 يا برابر الكتربكي بار
ساتتوير 60 أنهاا بين فاصله و است
0.9 - k 9i6z اسس.
(0.6)2
بين الكتربكي شيروي دوو سوال در توووون 1.6 4ا برابر الكتريكي بار دو 60 همننا أنها بين قاصاه و اسس 1.6 = k 9192 اسسن ساتت (0.6)²
ممممان معالa دو ابن حح
أوردت دست ds 1 91 مفدار موووان
كونن ميكرو 91 10 I كزينه محبج كزينه بناراان
<pad>(( 

اسستف
Aya
Haiku
47

---

Published as a conference paper at COLM 2024
بك باشدع ديكز ررك a كلافي a5 ندييدايم تشنيه حال a ززا اسسا سياه كلافي مهر استدال اسسن تيد كبرك تتبجه ... أز أن هر a5 اسن اسدلال
كلي جزني موارد تعميمي استرراي .(1) حزني كلي موارد تعميمي استقراي .(T)
كيي جزف مقدمات فياسي اسندالل .(²)
جزي كيي مقدمات فياسي اسنلال .(F)
The argument "Every crow is black because we have never heard or seen crow of any other color" ... argument, where a ... conclusion is drawn from the
1. Inductive generalization particular instances general cases
2. Inductive generalization general instances particular cases
3. deductive reasoning specific premises general cases
4. deductive reasoning general premises particular cases
Logic
USS
Trap:
Difficult
موارد تعمبهي استقرري () كزينه
ككي موار0~ تعمممي استقراي (T
كمي جزفي
جزفي
GPT-4
GPT-3.5
استشراي () كزينه محبج جواب اسسن كلي جزني موارد تعميمي سوال شده ارراه اسندال توخيح اسسن تعميمي استرراعي استدلل بك موارد منامده از استدللل نوع ابن در به سياما كلا أمناهدد جزني سياه كلازها اههة كلي كيري تتتيجة
<pad>(f 

شوده مي رسيد هسستتد
Aya
Haiku
48

---

Published as a conference paper at COLM 2024
كررو فزرر كدام سوال علامت بمجاي زار الكوو at توجم 4ا
F9 F YA rg
T.
OF A ?
FF .0)
FA .())
FT .())
.(F)
According to the pattern below, which number would be placed instead of the question mark?
49
4
28
36
5
30
64
?
1.
2. 48
3. 42
4. 40
Mathematical and Logical Intelligence
UPS
Trap: 2,3
Difficult
f) 40
Y)8
GPT-4
GPT-3.5
تونن cos olce الكووي a توجم 4 4e الكو إبن o اعداد a5 درااتت
كند: و تفير زير موو
49 > 4- 28 > 36-
5- > 30 > 64 > 5- >?
استت صورت ابن 44 اعداد تفيير الكوي
تبدبل بعدي عدد عدد هر a5
:3 م شودد مي تبديل جعارم اول ععن هشتم و بيست به جعارم ششي و نسي به حشتت و بوسنت بنجم به ششم سسي
pم سيي به بنحم جبارم شصت به ام سيي بنجم بs ججارم شصت عدد سوال علامت بهجاي بنلراين كبرف حي فورر 42 اسنن 42 I كزينه محبج باسج بس
<pad>(F 

Aya
Haiku
49

---

Published as a conference paper at COLM 2024
اسسف بكسان وازء جفت هو هر ميان راطبيي بمجز كزييمها تمام در
بيامرووي أناا اتتمبيل و كمتي -(1) ننج و أهج / تحفيق علمي حستاورد .(t) أموزش أمدرسة كقشوو و جلسة .(t) زسساان و سررااا مفته و سهشنبة .(F)
In all options except .......... the relationship between both pairs of words is the same.
1. Boats and Cars / Swimming and Walking
2. Scientific achievement and Research / Treasure and Pain
3. Meeting and Discussion / School and Education
4. Tuesday and Week / Cold and Winter
Verbal and Linguistic Intelligence
UPS
Trap: 1
Difficult
تحفيق علمي دسساورد (T كزينه きهه و كنج
زسساان
اسرااا
مفته و سهشنيه (f
GPT-4
GPT-3.5
1 كزينه باسخة
وازة جفت دو ميان رابطه .1 كزينه خر رابطه كزمم ساير اسس. متغاوت اسسة بكسان وازء جفت دو مي ميان تخفيق علمي حسشاورد 2 كزينه در وازه جفت g اهر رنج كنج هستند معلولي و علت إاطهه مدرسه / كو لفتوو جلسه 3 وريةه رابطه در وازء جفت دو (هر اموزن هستند فعاليت مكان وي اسراا مفته سهفشيه A كزبنه رابطه فر وازد جفت دو اهر ي زسساان هستندم كل جزد
كنتي " ميان رابهه .1 كزبته در أما ببادررو فنا " يا " اتوسيي
<pad>/ وكو كهت جلسة (f كزينه ه ميان و رابطه أمموشس مدرسه اسست بكسان وازء جفت />

وسيلء إبله اووي اسس: متاارت ربطه دومي در اما اسسن فءاليت
اسسن متغاوت فهالتمما
Aya
Haiku
50

---

Published as a conference paper at COLM 2024
I
Overview
Pronge
Model
A
Besponse String
nash SeS Btw.LLLL_AMPM.../oooee...
Extraction Methods
Regex
Single Probability
Full Probability
Fre- trained
Quality Check
X
X
Analysis Performance Trap Questions
Analysls Choice
Difficulty Level Analysis
Distribution
Figure 16: Overview of Khayyam dataset process and analysis
51


Reviews:
[{"review_id": "M8PYFwWNP1", "reviewer": "Reviewer_WmnJ", "summary": "This paper introduces the Khayyam Challenge (PersianMMLU), a benchmark to evaluate LLMs for the Persian language. It comprises 20k+ multiple choice questions from 38 tasks / subjects, meta information such as difficulty levels and descriptive answers are also included. Several state-of-the-art models including both open-source and commercial models are tested against this newly introduced dataset. Detailed analysis is also conducted to provide new insights.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough reviews. We are grateful for highlighting the importance of having evaluation benchmarks for languages other than English, including Persian. The benchmark we introduced offers rich metadata and aims to address this need. We are also grateful that you found our current multilingual assessment and analysis useful.\nRegarding your questions:\n\n1. *1st paragraph: bias and potential licensing issues*\n\nR1.1 To address concerns of bias, we note that the website serves as a platform for questions contributed by a diverse group of educators from various backgrounds and disciplines in education. This diversity helps to mitigate individual biases. Furthermore, the inclusion of questions from the nationally standardized Konkur exam, which are carefully selected from submissions by top educators, adds an additional layer of quality control. These properties minimize the presence of potential biases in PersianMMLU.\n\nMoreover, the website features an authentication-required login interface that prevents access through common web crawling methods, thereby helping to prevent data contamination and ensuring the integrity of our dataset. Also, regarding the license, the PersianMMLU dataset will be distributed under the terms of the CC BY-ND license.\n\n2. *2nd paragraph: contamination detection method*\n\nR1.2 We tested the hypothesis that the data from this collection was included in the training data of the models based on paper [1]. We conducted this evaluation using the Aya101 model, the best publicly available model, which had access to the token probabilities. As most contamination detection methods require access to the model's probabilities or weights, which are not available for private/closed models, we used the best performing public model for this evaluation. The hypothesis that the evaluation data was included in the training data was not proven, with a p-value of 0.3909. A detailed explanation of this experiment is included in the supplement of the revised paper.\n\n3. *3rd paragraph: translation effect analysis*\n\nR1.3 Thank you for pointing this out. This issue has been addressed in our response to Reviewer 2, specifically in section R2.3.\n\n4. *4th paragraph: example*\n\nR1.4 We have provided a full example in the following anonymous link \n\nhttps://drive.google.com/file/d/1lgdgb09s4KNbGbfvZk486E6nuGSXEo2T/view?usp=sharing.\n\n[1] https://openreview.net/forum?id=KS8mIvetg2", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Evaluating language models in languages other than English is an important task, and this paper introduces a new benchmark dataset for a relatively less-studied language namely Persian, which would be valuable for future research.\n- The newly introduced dataset contains rich metadata, enriching the contribution.\n- Multiple multilingual models are employed in the experiments and analysis among different aspects such as trap analysis, selected choice distribution are considered.", "reasons_to_reject": "- Regarding the data collection process: Since the questions are sourced from a specific website, several potential issues arise, including bias in the questions, contamination of data from a single source, and potential licensing issues. The authors should address these concerns in their clarification.\n- The current discussion on data contamination is not convincing; the mere incorporation of previously unused questions does not guarantee the absence of contamination in the dataset. I would suggest employing data contamination detection methods for a more robust analysis.\n- While the authors mention the challenge of translating English questions into Persian as a motivation in the introduction, it might be better to explore this issue in the experiments as well. For example, assessing whether translated MMLU can effectively serve as a proxy for testing the model's proficiency in Persian, and identify any associated issues or weaknesses with such translated data.\n- A minor point: including a complete example would enhance clarity and provide a more vivid understanding.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LZgiquMO5", "reviewer": "Reviewer_tpuE", "summary": "This work introduces a benchmark akin to MMLU for evaluating and tracking the progress of LLMs for the Persian language. The benchmark is built by gathering the questions from the website of a private educational institution in Iran, which most notably designs and holds regular mock examination sessions for students at different education levels. The benchmark covers 38 subjects and comes with metadata (details in reasons to accept) that facilitates further analysis of the patterns in capabilities of LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the importance of language-specific benchmarks and their view on the value of metadata, including descriptive answers that foster community discussion.\n\nWe understand the concern about dataset privacy and publication. This issue has been already discussed with the data owner organization and necessary permission has been obtained for non-commercial use. The PersianMMLU dataset is distributed under the CC BY-ND license.\n\nRegarding the reviewer's questions: \n\n1. *What does 'We chose the answer selected by humans if the combined total of the remaining three options was lower' in 4.2 means? Does this mean that in the regex method, in some cases it was just assumed that the model would go with the same answer as the human?*\n\n  R2.1 In our dataset, we provide the percentage of respondents who chose each answer option for every question. We label a question as correctly answered by humans if the percentage selecting the correct answer is greater than the combined percentages of the other options; otherwise, we label it as incorrectly answered. Human accuracy is assessed by counting the number of correctly and incorrectly answered questions.\n\n2. *Why do the prompts provided in the appendix not use the same terminology for the grade levels as the main text?*\n\nR2.2 We thank the reviewer for their precise review. As noted in the figure captions, the English machine translation was provided only for readability purposes. In response to this comment, we have made the translations consistent and corrected them in the revised paper.\n\n3. *I'm not able to see the results corresponding to '4.3 Impact of translation quality' anywhere. Assessing the claims in that section without seeing qualitative results is not viable.*\n\n  R2.3 Thanks for pointing this out. We will include the relevant results in the camera-ready version. Our results show that model performance on automatic translations improves when high-quality translations (denoted with an asterisk in the following table) are used instead of automatic translations. Specifically, model performance on automatic translations improves for samples in three major subjects—Physics, Math, and Chemistry—by 18%, 2%, and 12% respectively when we use high-quality translations. The translation results are as follows:\n\n       En2Fa | Fa2En\n       En  Fa| En  En* Fa\n       Physics| 42  26|Physics| 58  76  50\n       Biology| 76  74|Chem| 74  82  86  \n       ML| 46  40|Math| 50  52  56", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I agree with the authors' argument on the importance of creating language-specific benchmarks rather than translating an English benchmark. So I consider this a huge strength. Additionally, I really like the metadata that's naturally included. The private educational institute that the questions are gathered from provides solution booklets after each session of the regularly held mock exams. The solution booklet includes metadata such as the grade for which the question was designed, its difficulty, descriptive solution, whether or not the question included a trap choice among the choices, students' correct answer rate (correlated with human performance), and the year that the question was designed. I think this metadata is very valuable and can lead to a discussion community on the benefits of including such metadata in creating benchmarks.", "reasons_to_reject": "Given that this data was collected from a private educational institute, I believe more details should be provided on permission to collect and licensing details. Additionally, I'm not fully sure I understand the answer extraction methods in 4.2. More clear details would be beneficial (also refer to the questions below).", "questions_to_authors": "Following the previous part of the review, I also have the questions below:\n- What does \"We chose the answer selected by humans if the combined total of the remaining three options was lower.\" in 4.2 mean? Does this mean that in the regex method in some cases it was just assumed that the model would go with the same answer as the human?\n- Why do the prompts provided in the appendix don't use the same terminology for the grade levels as the main text? For instance, using second \"grade of high school\" instead of \"upper secondary school\". Or is that an effect of using machine translation systems?\n- I'm not able to see the results corresponding to \"4.3 Impact of translation quality\" anywhere. Assessing the claims in that section without seeing qualitative results is not viable.", "extra_scores": {}}, {"review_id": "JTMNSsRiOM", "reviewer": "Reviewer_yLgN", "summary": "The paper introduces PersianMMLU which consists of a large number of multiple choice questions spanning a wide number of diverse tasks and subjects. The dataset notably does *not* use machine translation (as it is sourced from Persian examinations). The authors surface 3 main contributions: (i) comprehensive coverage of topics (ii) richness of associated metadata (iii) usage of new data to prevent contamination (though to be fair what was new at the time of experimentation is now no longer new in today's vastly connected & scrapable world) (iv) native Persian artifacts (v) inherent scalability.\n\nOverall, the paper is well-written, sets clear differentiation from existing Persian/Farsi datasets like ParSQuAD and ParsiNLU, and presents some metrics on existing LLMs on their new dataset.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough reviews. We are pleased that you appreciated the comprehensive coverage and richness of metadata in PersianMMLU, as well as our experiments benchmarking various settings - key strengths of our work.\n\nRegarding your questions:\n\n1. *Section 3.2 talks about the ways the dataset was augmented -- were these all done with human experts or were some extracted from original metadata from the learning ladder?*\n\nR3.1 We will clarify in Section 3.2 that all metadata attributes, including difficulty level, were extracted from Learning Ladder. Based on our investigation, Learning Ladder automatically classifies question difficulty using student performance data, marking questions answered incorrectly by more students as difficult. They also determine the \"trap\" attribute, identifying common wrong answers, by analyzing frequently selected incorrect options.\n\n2. *The dataset seems to be heavily based on data from the Learning Ladder website. Were there additional processing steps that the authors have implemented.*\n\nR3.2 You're correct that the data is primarily from Learning Ladder. It's worth noting that the Cultural Educational Center ensures the quality of the questions through student assessment, and a team of question designers and editors. However, we further refined the raw data by:\n- Parsing HTML to extract relevant attributes\n- Filtering out questions containing images or tables\n- Removing explanatory answer questions to maintain multiple-choice format consistency\n- Deduplicating questions\n\nWe will expand Section 3.1 with more details on these steps.\n\n3. *I did find that mT0XL outperforming a persian-english LLM to be interesting, especially given it's fairly strong performance on other MCQA datasets like Belebele (wondering if the authors dug deeper into this)?*\n\nR3.3 We agree this is an intriguing result worthy of further investigation. Although examining the specific factors was beyond our current scope, it's possible that the amount of Persian pretraining data for PersianMind (2 billion tokens) may not have been sufficient.\n\n4. *Were the authors able to do any fine tuning experiments?*\n\nR3.4 We did not fine-tune in this work, as our primary goal was to evaluate models' inherent zero-shot reasoning capabilities. However, fine-tuning, especially on other Persian MCQA datasets, is a logical next step. We will note this as future work.\n\nThank you again for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I appreciate the various experiments that the authors performed including benchmarking various settings like different answer extraction methods, the impact of translation quality, and how few-shot examples did not help on the given MCQA task. The results showing that regex answer extraction outperformed the other methods was not surprising.", "reasons_to_reject": "I have no reason to reject this paper but I do have a lot of questions.", "questions_to_authors": "Section 3.2 talks about the ways the dataset was augmented -- were these all done with human experts or were some extracted from original metadata from learning ladder? For example \"Difficulty Level\" specifies that each question was classified into one of five distinct difficulty levels but authors don't mention how or by whom.\n\nThe dataset seems to be heavily based on data from the Learning Ladder website. Were there additional processing steps that the authors have implemented? Overall, the dataset construction section (3.1) feels a tad sparse.\n\nI did find that mT0XL outperforming a persian-english LLM to be interesting, especially given it's fairly strong performance on other MCQA datasets like Belebele (wondering if the authors dug deeper into this)?\n\nWere the authors able to do any finetuning experiments? E.g. finetuning on MCQA (maybe even Persian MCQA) datasets and then running the model on this particular task?", "extra_scores": {}}, {"review_id": "rIT3KXiQ0j", "reviewer": "Reviewer_oJHg", "summary": "This paper describes a new QA dataset for the Persian language. Just like the MMLU dataset for English, this new 'PersianMMLU' dataset covers a wide range of subjects, ranging from the natural sciences to the humanities, and contains more than 20,000 questions. The questions are multiple-choice with 4 answer alternatives, of which one is correct. Each question has its associated metadata, which includes the educational stage for which the question is intended, the difficulty level (which I assume is relative to the intended user group), and a detailed explanation of why the indicated correct answer is correct. The latter is very useful, as it can be used for 'chain-of-thought' training. As a motivation for their work, the authors convincingly point out several shortcomings of using an automatically translated variant of English datasets like the MMLU. The paper also contains experiments of applying various models to answer the questions. An extensive appendix gives lots of details of various aspects of the dataset. \n\nUnless I missed it, it is not explicitly stated in the paper that the dataset will be made public upon acceptance of the paper. It is merely said that 'all associated code is publicly accessible in [anonymous link]'. I assume the dataset will also be published open to everyone, but the authors should state this explicitly.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of this dataset as a major contribution to Persian NLP, as well as the value of its metadata. Below, we provide clarifications and answers to the reviewer's questions in each section.  \n\n**Reasons To Reject: Publish the dataset**\n\nFor clarification, we commit to fully publish the dataset for free under the terms of the CC BY-ND license.\n\n**Questions To Authors**\n\n1. *1st and 2nd paragraphs: Remove promotional sentences*\n\nR4.1 We agree that the current wording may be interpreted as promotional. To address this, we have revised Section 3.1 to focus on factual details about Cultural Educational Center's operations relevant to the dataset.\n\n2. *3rd paragraph: Annual update of the dataset*\n\nR4.2 We plan to update our dataset annually with new samples and insights. This ensures that our dataset remains up-to-date and relevant.\n\n3. *4th and last paragraphs: Compute human performance*\n\nR4.3 We thank the reviewer. This point has been addressed in R2.1 (reviewer tpuE, answer to item 1).\n\n4. *5th and 6th paragraphs: Answer extraction methods*\n\nR4.4 The input prompt did not restrict the model to single-letter answers, as it might hinder the reasoning process. Therefore, we used three methods to extract the desired answer from the model output detailed as follows:\n\nThe \"Regex\" method employs regex patterns to extract answer choices (step 1). If this fails, a pre-trained language model generates embedding vectors for each answer choice and the model output. Then, the option with the most similar vector to the output is selected (step 2). Therefore, incorporating a pre-trained language model is part of the \"Regex\" method. For example, the regex patterns can extract answer choice from \"c) 3π\" at step 1, but it will fail with \"According to the given equation, the third option is correct,\" requiring step 2.\n\nThe \"Single Token Probability\" method uses the probability distribution for the initial token in the model's output to extract the probabilities associated with answer choice labels (e.g., 'a', 'b', 'c', 'd' in prompt-1 template). The label with the highest probability is selected, leading to the choice of its corresponding option.\n\nThe \"Full Answer Probability\" method utilizes the model's output initial token probability distribution to compute the average log of probabilities for each token of all answer choices. It selects the option with the highest average.\n\nA more detailed explanation is incorporated in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The dataset seems to be a major contribution to Persian NLP. The paper is well written. Everything related to the dataset is clear, but I have some questions about the experiment section.", "reasons_to_reject": "Unless the authors commit to publishing the dataset free to the world upon acceptance of the paper, the paper should be rejected. I believe the area chair should sort this out with the authors before deciding on publication.\n\nI didn't understand the experimentation section very well. See questions below.", "questions_to_authors": "Remove this sales talk, as it is completely unscientific: \n\"Since conducting its first examination in 1993, Kanoon has been at the forefront of educational innovation, serving a nationwide network of 450,000 students. The center is highly regarded for its comprehensive range of educational services, notably its facilitation of creating and administering smart, customized tests and providing standardized solutions for exercises through the Learning Ladder platform. This\ninitiative aims to enable educators to design and administer quality, customized tests effortlessly and to provide students with insightful feedback to enhance their learning process.\"\n\n \"The quality of the questions in the dataset is exceptionally high, a testament to the institution’s reputable standing in the educational sector\" How do you measure the quality of the questions? This just seems like more sales talk. Remove.\n\n\"The ongoing development of new questions for various educational stages and subjects allows for the continuous expansion and updating of the dataset, reducing the risks of data contamination and erosion.\" Are you committing to a regular update of the dataset, or what does this mean, really?\n\nHow was the human success rate in Figure 2 computed?\n\nExplain the regex method better. Give examples of answers where it is successful. The \"single probability probability\" and \"full answer probability\" were also a bit obscure to me. Shouldn't the model just output a single letter (\"a\", \"b\", \"c\", or \"d\") indicating the correct answer? \n\n\"In cases where Regex failed to identify the correct option, we utilized a pre-trained model to generate embeddings for each choice...\"  Is this still part of the Regex method, or is this an alternative method?\n\nI didn't understand this sentence, please explain: \"We chose the answer selected by humans if the combined total of the remaining three options was lower.\"", "extra_scores": {}}]

Author rebuttal:

Meta-review: The PersianMMLU dataset is a significant contribution to Persian NLP, mirroring the MMLU dataset for English by covering a wide range of subjects with over 20,000 high-quality multiple-choice questions. The dataset is enriched with valuable metadata, such as educational stage, difficulty level, and detailed explanations, which can aid in 'chain-of-thought' training and further analysis of LLM capabilities. This benchmark dataset addresses the inadequacy of merely translating English datasets and is poised to facilitate meaningful evaluations and advancements in Persian language models. The thorough documentation and inclusion of metadata from a reputable educational institution enhance its utility for the research community.

The paper has several issues that need addressing, primarily concerning the dataset's publication commitment and clarity in the experimental methodology. There is a lack of explicit confirmation that the dataset will be made publicly available, which is crucial for its acceptance. Additionally, the data collection process raises concerns about potential biases, licensing permissions, and data contamination, which the authors have not convincingly addressed. The explanation of the answer extraction methods is unclear and needs further elaboration. The paper also includes unscientific promotional content that detracts from its scholarly value and should be removed. Clarifications on certain experimental procedures and metrics used are necessary for a comprehensive understanding.

[comments from the PCs] Please revise your paper following the critical comments by the author, including improving the scholarly value of your work by removing promotional content, addressing licensing/bias/contamination concerns, and clarifying issues that relate to the data construction method.
Decision: Accept