Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?

Authors: omid  ghahroodi
marzia  nouri ♠⋄
mohammad vali sanian ♠⋄
alireza  sahebi ♠⋄
doratossadat  dastgheib
ehsaneddin  asgari
mahdieh  soleymani baghshah ♠⋄ †
mohammad hossein rohban
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 12:37:04

Abstract: Evaluating Large Language Models (LLMs) is challenging due to their generative nature, necessitating precise evaluation methodologies. Additionally, non-English LLM evaluation lags behind English, resulting in the absence or weakness of LLMs for many languages.
In response to this necessity, we introduce Khayyam Challenge (also known as PersianMMLU), a meticulously curated collection comprising 20,805 four-choice questions sourced from 38 diverse tasks extracted from Persian examinations, spanning a wide spectrum of subjects, complexities, and ages. The primary objective of the Khayyam Challenge is to facilitate the rigorous evaluation of LLMs that support the Persian language. Distinctive features of the Khayyam Challenge are (i) its comprehensive coverage of various topics, including literary comprehension, mathematics, sciences, logic, intelligence testing, etc aimed at assessing different facets of LLMs such as language comprehension, reasoning, and information retrieval across various educational stages, from lower primary school to upper secondary school (ii) its inclusion of rich metadata such as human response rates, difficulty levels, and descriptive answers (iii) its utilization of new data to avoid data contamination issues prevalent in existing frameworks (iv) its use of original, non-translated data tailored for Persian speakers, ensuring the framework is free from translation challenges and errors while encompassing cultural nuances (v) its inherent scalability for future data updates and evaluations without requiring special human effort. Previous works lacked an evaluation framework that combined all of these features into a single comprehensive benchmark. Furthermore, we evaluate a wide range of existing LLMs that support the Persian language, with statistical analyses and interpretations of their outputs. We believe that the Khayyam Challenge will improve advancements in LLMs for the Persian language by highlighting the existing limitations of current models, while also enhancing the precision and depth of evaluations on LLMs, even within the English language context.

Main paper: 
 introduction: 
 Large Language Models (LLMs) have recently revolutionized applications of machine intelligence (Hong et al., 2024; Wu et al., 2023; Thirunavukarasu et al., 2023; Glukhov et al., 2023) . The rapid deployment of these models within industrial and public sector solutions has made evaluating their capabilities an imperative task (Guo et al., 2023) . To address critical aspects of language understanding and bridge the gap between the knowledge that models observed during pretraining and the measures of success, the MMLU benchmark was introduced (Hendrycks et al., 2021) . It assesses the ability of LLMs across a diverse set containing 57 subjects.Although automatic translation efforts like MMLU can bridge the gap in evaluation resources for languages other than English (Achiam et al., 2023) , the cultural specificity of these questions limits their applicability across different languages and cultures. This is particularly true for Persian, a language with its own rich culture and literature, where direct translation of English MMLU content may not be suitable for certain culture-specific subjects. In addition, our investigation reveals significant complications in automatic translation, particularly in specialized domains like Physics, which can be attributed to the quality of state-of-the-art translations for specific subjects. Despite advancements in machine translation technology, accurately conveying precise meaning in certain domains remains a hard task. For instance, the GPT-3.5 automatic translation often fails to include essential units such as "microjoules" in translations, leading to inaccuracies. Also, loss of information can occur in translation due to the existence of Persian words that lack exact equivalents in English. For example, in Persian, there are distinct terms for paternal aunt, maternal aunt, paternal uncle, and maternal uncle, while in English, only "aunt" and "uncle" are used (See Table 4 ). This underscores the inherent difficulty in translating domain-specific terminology accurately. For such reasons, some studies have evaluated LLMs on original non-English datasets rather than the translated ones. For instance, Li et al. (2023) has introduced a Chinese dataset across 67 topics and indicated that current models struggle to achieve accuracies above a certain threshold. Also, Zhang et al. (2023) introduced a multilingual and multimodal dataset, and showed that multilingual text processing hardly achieves over 60% accuracy.Motivated by the mentioned issues, we propose the Khayyam Challenge, also referred to as PersianMMLU, a benchmark designed to analyze the performance of LLMs in Persian and evaluate their knowledge and abilities comprehensively. Named in honor of the famed Persian polymath Omar Khayyam, whose contributions spanned various disciplines including Mathematics, Astronomy, Philosophy, and Poetry, the Khayyam Challenge embodies the multidimensional nature of Persian language understanding. This benchmark covers 38 subjects, including Mathematics and Physics, which require reasoning and computational ability, to Humanities and Social Sciences, demanding nuanced understanding and cultural sensitivity. Unlike the previous Persian datasets such as ParsiNLU (Khashabi et al., 2021) , our benchmark includes more diverse topics in addition to different educational stages. Moreover, our dataset distinguishes itself from previous efforts like ParSQuAD (Abadani et al., 2021) by being originally constructed in Persian, naturally incorporating the nuanced semantics and intricacies inherent to the language itself, rather than solely focusing on literal translations. Unlike PersianQA (Ayoubi, 2021) and PQuAD (Darvishi et al., 2023) , which are extractive datasets where models are tasked with extracting answers from given paragraphs and questions, our benchmark offers a more comprehensive evaluation of LLMs. This is because the task of answer extraction alone may not sufficiently assess the models' overall language understanding and reasoning capabilities. Our proposed dataset contains "Iran's national university entrance exam", and Kanoon Farhangi Amoozesh (Cultural Educational Center), wherein questions are accompanied by metadata for each question. This metadata includes the difficulty level, a descriptive answer, the educational stage, the subject, and the specific topic of the question. Through the Khayyam Challenge, we aim to provide a holistic evaluation framework that reflects the diverse linguistic and cognitive challenges inherent in processing Persian text across various domains.In our evaluations, we assessed several state-of-the-art language models, including GPT-3.5, GPT-4 (OpenAI, 2023) , Aya (Üstün et al., 2024) , PersianMind (Rostami et al., 2024) , mGPT (Shliazhko et al., 2022) , mT0 (Muennighoff et al., 2022) , Claude3-haiku (Anthropic, 2024) , and XVERSE1 , all purportedly equipped with some level of understanding of the Persian language. Our findings indicate that while most of these models struggle to grasp Persian nuances, particularly evident in domains such as Calculus, Logic, and Geometry where accurate comprehension is essential, some exhibit comparatively better performance in contexts reliant on contextual understanding, such as Economics, Psychology, and Social studies. Notably, GPT-4 showcased relatively improved performance across multiple domains. However, there remains a clear imperative need for further enhancements across all models, especially in technical disciplines like Discrete Mathematics, where precise language comprehension is paramount for meaningful outcomes. Furthermore, our evaluation unveiled new insights. We observed dependencies between the perceived difficulty of questions for humans versus LLMs. While both encounter similar challenges in tackling difficult questions, LLMs like GPT-4 demonstrated superior accuracy compared to humans, in questions intentionally designed to deceive, referred to as trapped questions. Moreover, our analysis identified biases in certain language models' responses, such as GPT-3.5 favoring particular choices, suggesting room for further improvement in these models.Overall, the Khayyam Challenge (PersianMMLU) marks a significant step forward in evaluating the language understanding and abilities of LLMs that support the Persian language.Our dataset and code are available on HuggingFace2 and GitHub3 , respectively. Additionally, we set up a leaderboard4 on HuggingFace to stay updated with the performance of other models. 
large language models: 
 Over the past few years, there has been a significant improvement in the performance of language models. This progress has been observed in line with the scaling law (Kaplan et al., 2020) , thanks to the increasing size of training datasets, enhanced processing power, and new evolved model architectures. The continuing process of scaling the models resulted in LLMs like GPT-3 (Brown et al., 2020) , GPT-4 (OpenAI, 2023) , Claude3, mT0 (Muennighoff et al., 2022) , XVERSE, Aya (Üstün et al., 2024) , etc.Even though AI models are highly capable of solving various tasks, they continue to encounter difficulties when it comes to real-world problems that, for example, require strong reasoning abilities or complex mathematical calculations (Chang et al., 2024; Zhong et al., 2023) . Therefore, we need to assess the effectiveness of these models in solving highlevel tasks. This enables us to identify the weak points of the models and work towards improving them in the future.Despite some of the recent LLMs being multilingual, studies indicate that their effectiveness is not as pronounced in non-Latin or low-resource languages as it is in English (Zhang et al., 2023) . Consequently, it is essential to assess multilingual LLMs on tasks that employ languages other than English.Several benchmarks have been developed to assess the performance of LLMs. One of the most significant benchmarks is MMLU (Hendrycks et al., 2021) , which evaluates language models for answering multiple-choice questions in 57 different tasks, but only in English. (Zhang et al., 2023) introduces a multilingual, multimodal, and multilevel benchmark for evaluating LLMs including more than 12K multiple-choice questions from 9 languages (excluding Persian) at three educational stages. AGIEval (Zhong et al., 2023) is another benchmark that assesses the performance of LLMs on human-centric standardized exams in English and Chinese languages to measure their ability in human-level tasks.There have been a few benchmarks built to assess language models on the Persian language, including ParSQUAD (Abadani et al., 2021) , PersianQA (Ayoubi, 2021), ParsiNLU (Khashabi et al., 2021) , and PQuAD (Darvishi et al., 2023) , in which some of their features are compared in Table 1 . ParSQUAD, PersianQA, and PQuAD present extractive datasets where models are asked to extract answers from given paragraphs. While this task can gauge models' reading comprehension skills, it may not effectively evaluate general capabilities and inherent knowledge of models. ParsiNLU evaluates language models based on 14500 questions from six language understanding tasks, including multiple-choice QA (MCQA), sentiment analysis, and more (Table 1 ). However, the questions in this benchmark fail to reach human-level complexity, thus inadequately assessing important skills of LLMs, including complex reasoning, needed for solving higher educational stages questions. Moreover, ParsiNLU covers only three subject tasks in MCQA and lacks adequate metadata, such as question difficulty levels. This deficiency further restricts our capacity to evaluate the model's proficiency across specific tasks and different difficulty levels.To address this gap, we introduce Khayyam Challenge, which features multiple-choice questions sourced from high-standard exams, and contains reach metadata to evaluate the innate knowledge and human-like skills of LLMs across different difficulty levels and educational stages. 
data: 
 The Khayyam Challenge presents a robust dataset aimed at enhancing the evaluation of LLMs that support Persian, particularly in the context of multiple-choice question answering. This dataset encompasses a diverse range of subjects, reflecting a comprehensive approach to assessing various cognitive abilities including language comprehension, reasoning, and knowledge recall across different educational stages.The educational system in Iran, from which this dataset draws, is structured into 12 years of schooling divided into segments: 6 years of primary education and 6 years of secondary education. Primary education is split into lower primary school (LPS) for the first 3 years, followed by upper primary school (UPS) for the next 3 years. Secondary education is similarly divided, with lower secondary school (LSS) encompassing the first 3 years, and upper secondary school (USS) comprising the final 3 years.The dataset is sourced from the "Pellekan Yadgiri (Learning Ladder)" website5 , a part of the Kanoon Farhangi Amoozesh6 (Cultural Educational Center), an educational institution in Iran. The platform provides resources for creating and administering tests, as well as standardized solutions for exercises. The dataset includes questions from various educational stages and subjects, including items from the Iranian national university entrance examination. This diversity contributes to the dataset's comprehensiveness and relevance.While the dataset is primarily sourced from the Learning Ladder platform, we implemented additional processing steps. These included parsing HTML to extract attributes, filtering out questions with images or tables, removing explanatory answer questions, and deduplicating entries. We then reviewed the questions, corrected typos, and deleted those that required context that was not present in the question itself. These steps were crucial in refining the raw data into a structured, text-only dataset suitable for our research purposes.We also note that the Learning Ladder platform serves as a platform for questions contributed by a diverse group of educators from various backgrounds and disciplines in education. This diversity helps mitigate individual biases. Furthermore, the inclusion of questions from the university entrance exams, carefully selected from submissions by top educators, adds an additional layer of quality control. These features minimize the presence of potential biases in PersianMMLU. This dataset is distributed under a Creative Commons No Derivatives (CC ND) license, prohibiting the creation of derivative works.We also tested the hypothesis that the data from this collection was included in the training data of the models based on Oren et al. (2024) . We conducted this evaluation using the Aya model, the best publicly available model, which had access to the token probabilities. The hypothesis that the evaluation data was included in the training data was not proven, with a p-value of 0.3909.The Khayyam Challenge is enriched with valuable metadata that elevates its utility beyond a simple aggregation of questions. This metadata includes:• Educational Stage: Specifies the educational stage for which the question is intended (LPS, UPS, LSS, USS), allowing for the assessment of appropriateness and difficulty relative to the expected knowledge base at each stage of education.• Difficulty Level: Each question is classified into one of five distinct difficulty levels: easy, relatively easy, medium, relatively difficult, and difficult. This nuanced categorization allows a detailed analysis of question difficulty and examinee performance. The difficulty level, like all metadata attributes, was extracted directly from the Learning Ladder platform. The platform uses an automated system to classify question difficulty based on student performance data. Questions answered incorrectly by a higher percentage of students are marked as more difficult.• Descriptive Answers: In addition to the correct answer, our dataset provides a detailed explanation for each question. This is crucial for understanding the reasoning behind the correct answer, facilitating a deeper comprehension of the question.• Trap: Some questions contain a "trap" choice-an incorrect answer that might be easily mistaken for the correct one. These questions are referred to as "trapped questions" and are generally more challenging, with the majority classified as difficult. This helps in understanding common misconceptions and the effectiveness of question design in truly testing knowledge and reasoning abilities. This attribute is also determined by the Learning Ladder platform, which analyzes frequently selected incorrect options to identify common misconceptions among students.• Candidate Choice Distribution: This metric provides the percentage of students selecting each answer choice for a given question.• Specific Topic: Questions are meticulously categorized into detailed subjects, such as "Mathematics > Discrete Maths > Combinatorics." This detailed classification enables targeted analysis of exam content and provides insights into the distribution and depth of questions across various subjects.• Year: Indicates the year when the question was designed, which can provide insights into the evolution of question complexity and educational standards over time.The inclusion of this metadata is not merely for augmentative purposes; it serves a critical role in enabling comprehensive analyses that can benefit educators, researchers, and developers of educational technologies. Specifically, it allows for the comparison of performance between human examinees and LLMs on specific topics under varying difficulty levels. By assessing whether LLMs fall for the traps as humans or how they approach questions requiring complex thought processes, we can gain valuable insights into the capabilities and limitations of current AI technologies in educational contexts.Moreover, the presence of descriptive answers supports the development of more sophisticated AI models by facilitating "chain of thought" processing, where the model learns to approach a problem step-by-step, mirroring human problem-solving methods. This not only enhances the model's problem-solving skills but also its ability to explain its reasoning in a manner that is understandable to humans.The dataset contains 20,805 multiple-choice questions across 38 tasks, spanning subject areas like humanities, mathematics, natural science, and social science, along with elements of intelligence testing. These questions necessitate a blend of knowledge and reasoning. Additionally, the dataset includes 16,955 questions with human performance data, excluding Iran's national university entrance exam questions, and features 3,889 trapped questions.Figure 3 -(a) depicts the allocation of questions among the main categories and their respective tasks. Figure 3-(b ) outlines the distribution of questions based on their levels of difficulty. For more detailed information about the data, refer to Appendix A.The Khayyam Challenge Dataset stands out for several reasons:• Comprehensive Coverage: It spans a broad spectrum of subjects from literary comprehension to logic and intelligence testing, catering to different stages of education. This diversity makes it a versatile tool for assessing language models' capabilities across various domains.• Rich Metadata: The inclusion of detailed question metadata enhances the dataset's utility for nuanced analysis and model evaluation, providing valuable context for each question.• New Data Utilization: By incorporating questions never before used in research, the dataset avoids common data contamination issues, offering a fresh challenge to language models.• Original, Non-Translated Content: Focused on the Persian language, the dataset eliminates translation errors and incorporates cultural nuances, making it uniquely valuable for related linguistic and cultural studies.• Scalability: The dataset's design and sourcing methodology ensure its adaptability and expandability, allowing for straightforward updates and extensions without substantial human intervention. We evaluated our benchmark using nine distinct models: GPT-4 (gpt-4-0613), GPT-3.5 (gpt-3.5-turbo-0125), Aya, XVERSE-13B, XVERSE-7B, PersianMind, mT0XL, mGPT, and Claude 3/Haiku. Detailed descriptions of each model can be found in Appendix B. Additionally, we benchmarked a standardized prompt-0 template (Figure 5 ) on the entire dataset and two other templates, prompt-1 (Figure 6 ) and prompt-2 (Figure 7 ), on a subset of 1000 samples. We also conducted Chain-of-Thought (CoT) (Figure 8 ) on a subset of 1000 samples requiring CoT, such as mathematical questions. To ensure consistency and fairness throughout our experiments, we kept all model hyperparameters at their default values. Furthermore, we set the temperature parameter to zero for all models and did not impose any maximum limit on the number of tokens, allowing models the freedom to conduct any type of inference they desired. 
answer extraction methods: 
 The input prompt did not restrict the model to single-letter answers, as it might hinder the reasoning process. Therefore, we employed three distinct methods to extract the desired answer choice from the model output; "Regex", "Single Token Probability", and "Full Answer Probability".In the "Regex" method, we developed detailed regex patterns tailored to each model to accurately capture the desired choice from the model output (the patterns are available in our GitHub repository). In cases where regex patterns failed to identify the desired choice, we utilized a pre-trained language model to generate embedding vectors for each of the answer choices and the model output. Finally, the choice with the most similar vector to the output's vector is selected as the desired answer.In the "Single Token Probability" method, applied to models providing token probabilities, we used the probability distribution for the initial token in the model output to extract the probabilities associated with answer choice labels (e.g., 'a', 'b', 'c', 'd' in prompt-1 template (Figure 6 )). The label with the highest probability is selected, leading to the choice of its corresponding option as the desired answer.In the "Full Answer Probability" method, we calculated the average of the logarithm of token probabilities for each answer choice by using the probability distribution of the initial token of the model output. We then chose the answer with the highest average as the desired option. This process is shown in equation 1.EQUATIONOur analysis shows that the "Regex" method has the highest accuracy, whereas the "Full Answer Probability" method demonstrated a significant decrease in accuracy compared to the other two approaches.To measure human performance, we used the percentages of respondents who selected each answer choice for every question, as provided in the metadata. A question is labeled as correctly answered if the percentage selecting the correct answer exceeds the combined percentages of the other options. Otherwise, it is labeled as incorrectly answered. Human accuracy is then evaluated by counting the number of correctly and incorrectly answered questions.We aim to assess the impact of translation from state-of-the-art (SOTA) translation models on performance. To do so, we selected a set of 50 sample questions from three subjects of our dataset and translated them to English with the assistance of field experts, as well as using off-the-shelf translation models. Following this, we evaluated the GPT-4's performance on both sets of samples: those translated with expert assistance and those translated by models alone. Our findings which are illustrated in Table 2a , revealed a notable decrease in performance when translated with the help of off-the-shelf models, highlighting the need for a new dataset that does not depend solely on translated data. We also evaluated the impact of automatic translation on performance by translating a sample of English questions from the MMLU dataset (Hendrycks et al., 2021) into Persian and measuring GPT-3.5's performance on both sets. As shown in Table 2b , automatic translation from English to Persian can reduce the performance of large language models. Table 2 : Accuracy rates of LLMs for (a): a sample of questions from our dataset (Persian-MMLU) and their translations to English using an automatic translation model, and a field expert (denoted with an asterisk), and (b): a sample of MMLU English questions and their translation to Persian using automatic translation modelWe have developed a benchmark code for our dataset and used the zero-shot and CoT methods to calculate accuracy. Previous studies (Li et al., 2023; Zhong et al., 2023; Zhang et al., 2023) shows that using few shots on instruction-tuned models does not enhance accuracy and may even may decrease it. Therefore, we did not measure accuracy using few-shot techniques.Human GPT-4 Haiku GPT-3. 5 Results and discussions 
results across all models: 
 The evaluation results of the zero-shot method for all models across five main categories and three choice-extraction methods ("Regex", "Single Token Probability", and "Full Answer Probability"), as well as human performance, are presented in Table 3 , and appendix Tables 9, 8 . The more comprehensive results on all 38 tasks are reported in the appendix (Tables 10, 11, 12 ). Also, the results of CoT and its comparison with zero-shot for GPT-3.5 using "Regex" method across three main categories on a subset of dataset with 1000 questions are presented in appendix Table 7 . These results yield the following key findings:• Utilizing "Regex" method for answer extraction (Table 3 ) results in highest model performance compared to "Single Token Probability" (Table 9 ) and "Full Answer Probability" (Table 8 ) due to its more accurate and comprehensive choice extraction procedure. In the rest of the results section, we compare the accuracy of models using the "Regex" method. • GPT-4 outperforms all other models in all five main categories, with an average accuracy of 6 percent higher than Claude3-haiku, the second-best performing model. • Aya, an open-source model, performs comparably or even better than GPT-3.5, a closed-source model, in 4 tasks including Sociology USS and Economy USS. This demonstrates the convergence of open-source models' capabilities towards closed-source models. • Although PersianMind, a 7B Persian-English LLM, is trained and fine-tuned on 2 billion Persian tokens (Rostami et al., 2024) , its performance is equal to mT0XL, a multilingual 3.7B LLM. • The performance gap between the best-performing model, GPT-4, and human averages around 27%. In subjects like mathematics, this gap widens to 31% accuracy for GPT-4. This exhibits a real challenge of current LLMs in solving human-level questions, especially in complex mathematical questions that need high-level mathematical calculation and reasoning skills. • The models exhibit weaker performance in mathematics and natural science main categories compared to humanities and social sciences. This indicates their weaker performance on questions requiring high reasoning skills, compared to those mainly reliant on models' inherent knowledge. This underscores the necessity for enhancing the models' reasoning ability in the Persian language. • The CoT has improved the performance of GPT-3.5 in Mathematics questions by 10%, but decreased the performance in Humanities questions by 8%.We demonstrated the accuracy of the models on all three difficulty levels of questions for different educational stages and question publication years in Figure 9 and Figure 10 in the Appendix E, respectively. Key findings from these figures include:• Most model accuracies decline with the increasing publication year for questions with medium and easy difficulty levels, while human performance remains consistent. This suggests that humans may have adapted to the evolving questions' difficulty over time, whereas models have not. • There exists a notable performance gap between difficult and medium questions in human performance, indicating that difficulty level has a stronger impact on human performance compared to LLMs. Since humans assigned difficulty levels to questions, this gap may stem from differing perceptions of difficulty between humans and LLMs. • GPT-4 outperforms humans in difficult questions within four years, as well as during the initial two educational stages (LPS and UPS). This result suggests that modern LLMs may excel at analyzing difficult questions compared to humans.Figures 12, 13, 14 in the Appendix E depicts the selected choice distributions of various models. It reveals that despite GPT-3.5 having the lowest count of unanswered questions, GPT-4 surpasses it in accuracy, indicating a more refined understanding despite its higher nonresponse rate. Comparatively, GPT-4 exhibits a more uniform distribution of choice selection, aligning closely with the Ground Truth and showing less bias than its predecessor, GPT-3.5, which tends towards selecting the second and third choices. High-performance models like GPT-4 demonstrate a closer alignment with Ground Truth distribution, indicating a lower bias level and potentially higher utility in applications.Table 16 in the Appendix F compares the performance of models and humans on trapped questions, using an x/y format where x represents overall accuracy and y the accuracy on trapped questions. The data shows that while traps often mislead students, leading to nearly random performance, models like GPT-4 exhibit only a slight drop in accuracy when faced with these traps. However, this drop in accuracy is most pronounced in the Social Sciences and Humanities. Notably, GPT-4 outperforms humans across all main categories in handling trapped questions, indicating its robustness against misleading choices and affirming the different perspectives of difficulty between humans and AI models.To explore how humans and LLMs perceive the difficulty of questions, we examined the accuracy of different models across three difficulty levels. Although our dataset included five labels ranging from easy to difficult to denote question difficulty, we combined the two relatively easy and relatively difficult labels into "easy" and "difficult", respectively, to ensure a more balanced distribution of question difficulty. Our experiment revealed a consistent trend: as question difficulty increased, both human and model answering accuracy decreased (See Tables 13, 14 , 15 in the Appendix E).Notably, in analytical and knowledge-based topics such as the humanities category, the GPT-4 model demonstrated superior performance compared to humans in tackling difficult questions. Conversely, models exhibiting more human-like performance, such as GPT-4, revealed that humans significantly outperformed them on easier questions. For additional statistical results, refer to Appendix G. 
conclusions: 
 We introduced Khayyam Challenge, also known as PersianMMLU, as the first framework for assessing LLMs in the Persian language across various tasks, difficulty levels, and educational stages. This framework includes comprehensive metadata such as human performance, difficulty levels, and traps. Our assessment encompassed examining the performance of current LLMs on these datasets, evaluating their ability to extract answers (probabilistic and rule-based paradigms), and considering various aspects highlighted in the metadata. Our findings revealed that while LLMs demonstrated relatively satisfactory performance in question-solving tasks (especially on GPT-4), they still significantly lagged behind human performance, particularly in tasks necessitating reasoning. Moreover, analysis of metadata concerning difficulty levels and trapped questions unveiled notable discrepancies between model and human behavior, suggesting fundamental differences in learning approaches. This underscores the necessity for adaptations in LLMs training methodologies to achieve human-like proficiency. For future works, we aim to develop an LLM that bridges the performance gap between existing open-source models and GPT-4. 
 
 Appendix: Question ، ‫دسیبل‬ ۵۰ ‫را‬ ‫منبع‬ ‫یک‬ ‫از‬ ‫حاصل‬ ‫صوت‬ ‫شدت‬ ‫تراز‬ ‫است،‬ ‫مربع‬ ‫میلیمتر‬ ۶۰ ‫گوشش‬ ‫پرده‬ ‫مساحت‬ ‫که‬ ‫شنوندهای‬ ‫است؟‬ ‫میکروژول‬ ‫چند‬ ‫میرسد،‬ ‫شنونده‬ ‫این‬ ‫گوش‬ ‫پرده‬ ‫به‬ ‫ثانیه‬ ۵۰ ‫مدت‬ ‫در‬ ‫که‬ ‫انرژی‬ ‫میکند‬ ‫احساس‬A listener with an eardrum area of 60mm 2 perceives the sound intensity level from a source as 50 decibels.How much energy reaches the listener's ear in 50 seconds in microjoules?A listener with an ear canal area of 60mm 2 perceives the sound intensity level from a source as 50 decibels.How much energy reaches the listener's ear in 50 seconds?‫است؟‬ ‫برده‬ ‫بهره‬ ‫شاعری‬ ‫اختیارات‬ ‫کدام‬ ‫از‬ ‫شاعر‬ ‫زیر،‬ ‫بیت‬ ‫در‬ ‫است≪‬ ‫زده‬ ‫خون‬ ‫قدح‬ ‫در‬ ‫پای‬ ‫ساق‬ ‫دری‬ ‫کبک‬ / ‫است‬ ‫زده‬ ‫صابون‬ ‫به‬ ‫جامه‬ ‫سفید‬ ّ ‫بط‬ ‫≫گویی‬In the following verse, which poetic devices has the poet employed? "The duck's feathers are so white, as if they've been washed with soap / The partridge's legs are so red, as if they've been dipped in a cup of blood"In the following verse, which poetic options did the poet use? "It's as if a white duck has put its clothes on soap / the partridge has put its leg in a glass of blood" Question ‫دارم.‬ ‫دایی‬ ۳ ‫و‬ ‫خاله‬ ۲ ‫من‬ ‫است.‬ ‫من‬ ‫خاله‬ ‫پسر‬ ‫زهرا،‬ ‫عمه‬ ‫پسر‬ ‫کیان‬ ‫دارد؟‬ ‫عمو‬ ‫چند‬ ‫و‬ ‫عمه‬ ‫چند‬ ‫بهترتیب‬ ‫زهرا‬Kian is the son of the sister of Zahra's Father, and he is the son of my mother's sister. My mother has two sisters and three brothers. How many sisters and brothers does Zahra's father have?Kian is the son of my aunt Zahra. I have 2 aunts and 3 uncles. How many aunts and how many uncles does Zahra have respectively? OpenAI has introduced GPT-4 as a large multimodal model. However, the internal structure and operational specifics of GPT-4 are proprietary and not openly disclosed.GPT-3.5, created by OpenAI, represents a significant advancement in natural language processing technology. It boasts enhanced capabilities in understanding and generating human-like text, performing impressively across various language tasks like text completion, translation, and question answering, thanks to its expansive architecture and vast training data.The Claude 3 Haiku model, developed recently by Anthropic, are proprietary models with undisclosed architecture and training specifics.Aya, an open-source model by Cohere, is a massively multilingual generative language model capable of understanding instructions in 101 languages. Over half of these languages are classified as lower-resourced.XVERSE-13B, created by Shenzhen Yuanxiang Technology, stands as a versatile multilingual large language model. Employing a prominent Decoder-only Transformer network structure, XVERSE-13B has a context length of 8k. It has been meticulously trained on a vast and varied dataset containing 1.4 trillion tokens, covering more than 40 languages including Chinese, English, Russian, and Spanish. PersianMind is an openly available bilingual large language model. It enriches LLaMa2's vocabulary by integrating 10,000 Persian tokens and undergoes training on a dataset containing nearly 2 billion Persian tokens.mT0XL originates from the fine-tuning of BLOOM models using xP3, a dataset that encompasses multilingual content coupled with English prompts. The BLOOM models, which are extensive decoder-only language models, have been pre-trained on roughly 350 billion tokens. They share structural similarities with GPT-3.mGPT is an open-source adaptation of GPT-3, having multilingual capabilities. It undergoes pretraining across 61 languages, drawn from 25 diverse language families, leveraging Wikipedia and the C4 Corpus. Additionally, mGPT offers support for Persian language integration.‫را‬ ‫متوسطه‬ ‫دوم‬ ‫دوره‬ ‫هندسه‬ ‫مورد‬ ‫در‬ ‫پاسخ(‬ ‫)با‬ ‫گزینهای‬ ‫چند‬ ‫سواالت‬ ‫زیر‬ ‫در‬ ‫میکنید.‬ ‫مشاهده‬ ‫سوال:‬ ‫است.‬ C ‫دایره‬ ‫از‬ ‫قطری‬ ‫معادله‬ ،(m -۲)x + (m + ۱)y = ۶ ‫معادله‬ ،m ‫هر‬ ‫ازای‬ ‫به‬ ‫است؟‬ ‫کدام‬ C ‫دایره‬ ‫محیط‬ ‫باشد،‬ C ‫دایره‬ ‫روی‬ A(-۱, ۱) ‫نقطه‬ ‫اگر‬ ‫گزینهها:‬ ۲ √ ۲π (۱ ۲π (۲ ۳π (۳ ۲ √ ۳π (۴ ‫جواب:‬ Below youChoices: 1) 2 √ 2π 2) 2π 3) 3π 4) 2 √ 3πAnswer: Choices: a) 2 √ 2π b) 2π c) 3π d) 2 √ 3πAnswer:Figure 6 : Sample prompt-1 with English translation for enhanced readabilityThe following are multiple choice questions (with answer) about Upper Secondary School Geometry‫سوال:‬ ‫است.‬ C ‫دایره‬ ‫از‬ ‫قطری‬ ‫معادله‬ ،(m -۲)x + (m + ۱)y = ۶ ‫معادله‬ ،m ‫هر‬ ‫ازای‬ ‫به‬ ‫است؟‬ ‫کدام‬ C ‫دایره‬ ‫محیط‬ ‫باشد،‬ C ‫دایره‬ ‫روی‬ A(-۱, ۱) ‫نقطه‬ ‫اگر‬ ‫گزینهها:‬ a) 2 √ 2π b) 2π c) 3π d) 2 √ 3π‫جواب:‬ Choices: a) 2 √ 2π b) 2π c) 3π d) 2 √ 3πAnswer: ‫است.‬ C ‫دایره‬ ‫از‬ ‫قطری‬ ‫معادله‬ ،(m -۲)x + (m + ۱)y = ۶ ‫معادله‬ ،m ‫هر‬ ‫ازای‬ ‫به‬ ‫است؟‬ ‫کدام‬ C ‫دایره‬ ‫محیط‬ ‫باشد،‬ C ‫دایره‬ ‫روی‬ A(-۱, ۱) ‫نقطه‬ ‫اگر‬ ‫گزینهها:‬ a) 2 √ 2π b) 2π c) 3π d) 2 √ 3π‫کنیم‬ ‫فکر‬ ‫قدم‬ ‫به‬ ‫قدم‬ ‫بیایید‬ ‫جواب:‬ 2) g(x) = x -(-1) x 3) (g In the number 195483, the units digit swaps places with the thousands digit, the tens digit swaps places with the tens of thousands digit, and the hundreds digit swaps places with the hundreds of thousands digit. What is the sum of the two new digits in the hundreds and thousands places in the new number? EQUATIONhttps://github.com/xverse-aihttps://huggingface.co/datasets/raia-center/khayyam-challengehttps://github.com/raia-center/khayyam-challengehttps://huggingface.co/spaces/raia-center/PersianMMLUhttps://peleyad.com/https://en.kanoon.ir/ 


Decision: The PersianMMLU dataset is a significant contribution to Persian NLP, mirroring the MMLU dataset for English by covering a wide range of subjects with over 20,000 high-quality multiple-choice questions. The dataset is enriched with valuable metadata, such as educational stage, difficulty level, and detailed explanations, which can aid in 'chain-of-thought' training and further analysis of LLM capabilities. This benchmark dataset addresses the inadequacy of merely translating English datasets and is poised to facilitate meaningful evaluations and advancements in Persian language models. The thorough documentation and inclusion of metadata from a reputable educational institution enhance its utility for the research community.

The paper has several issues that need addressing, primarily concerning the dataset's publication commitment and clarity in the experimental methodology. There is a lack of explicit confirmation that the dataset will be made publicly available, which is crucial for its acceptance. Additionally, the data collection process raises concerns about potential biases, licensing permissions, and data contamination, which the authors have not convincingly addressed. The explanation of the answer extraction methods is unclear and needs further elaboration. The paper also includes unscientific promotional content that detracts from its scholarly value and should be removed. Clarifications on certain experimental procedures and metrics used are necessary for a comprehensive understanding.

[comments from the PCs] Please revise your paper following the critical comments by the author, including improving the scholarly value of your work by removing promotional content, addressing licensing/bias/contamination concerns, and clarifying issues that relate to the data construction method.

Reviews: review: paper_summary: paper_summary: This paper introduces the Khayyam Challenge (PersianMMLU), a benchmark to evaluate LLMs for the Persian language. It comprises 20k+ multiple choice questions from 38 tasks / subjects, meta information such as difficulty levels and descriptive answers are also included. Several state-of-the-art models including both open-source and commercial models are tested against this newly introduced dataset. Detailed analysis is also conducted to provide new insights.
strength_weakness: strength_weakness: - Regarding the data collection process: Since the questions are sourced from a specific website, several potential issues arise, including bias in the questions, contamination of data from a single source, and potential licensing issues. The authors should address these concerns in their clarification.
- The current discussion on data contamination is not convincing; the mere incorporation of previously unused questions does not guarantee the absence of contamination in the dataset. I would suggest employing data contamination detection methods for a more robust analysis.
- While the authors mention the challenge of translating English questions into Persian as a motivation in the introduction, it might be better to explore this issue in the experiments as well. For example, assessing whether translated MMLU can effectively serve as a proxy for testing the model's proficiency in Persian, and identify any associated issues or weaknesses with such translated data.
- A minor point: including a complete example would enhance clarity and provide a more vivid understanding.
paper_summary: paper_summary: This work introduces a benchmark akin to MMLU for evaluating and tracking the progress of LLMs for the Persian language. The benchmark is built by gathering the questions from the website of a private educational institution in Iran, which most notably designs and holds regular mock examination sessions for students at different education levels. The benchmark covers 38 subjects and comes with metadata (details in reasons to accept) that facilitates further analysis of the patterns in capabilities of LLMs.
strength_weakness: strength_weakness: Given that this data was collected from a private educational institute, I believe more details should be provided on permission to collect and licensing details. Additionally, I'm not fully sure I understand the answer extraction methods in 4.2. More clear details would be beneficial (also refer to the questions below).
questions: questions: Following the previous part of the review, I also have the questions below:
- What does "We chose the answer selected by humans if the combined total of the remaining three options was lower." in 4.2 mean? Does this mean that in the regex method in some cases it was just assumed that the model would go with the same answer as the human?
- Why do the prompts provided in the appendix don't use the same terminology for the grade levels as the main text? For instance, using second "grade of high school" instead of "upper secondary school". Or is that an effect of using machine translation systems?
- I'm not able to see the results corresponding to "4.3 Impact of translation quality" anywhere. Assessing the claims in that section without seeing qualitative results is not viable.
paper_summary: paper_summary: The paper introduces PersianMMLU which consists of a large number of multiple choice questions spanning a wide number of diverse tasks and subjects. The dataset notably does *not* use machine translation (as it is sourced from Persian examinations). The authors surface 3 main contributions: (i) comprehensive coverage of topics (ii) richness of associated metadata (iii) usage of new data to prevent contamination (though to be fair what was new at the time of experimentation is now no longer new in today's vastly connected & scrapable world) (iv) native Persian artifacts (v) inherent scalability.

Overall, the paper is well-written, sets clear differentiation from existing Persian/Farsi datasets like ParSQuAD and ParsiNLU, and presents some metrics on existing LLMs on their new dataset.
strength_weakness: strength_weakness: I have no reason to reject this paper but I do have a lot of questions.
questions: questions: Section 3.2 talks about the ways the dataset was augmented -- were these all done with human experts or were some extracted from original metadata from learning ladder? For example "Difficulty Level" specifies that each question was classified into one of five distinct difficulty levels but authors don't mention how or by whom.

The dataset seems to be heavily based on data from the Learning Ladder website. Were there additional processing steps that the authors have implemented? Overall, the dataset construction section (3.1) feels a tad sparse.

I did find that mT0XL outperforming a persian-english LLM to be interesting, especially given it's fairly strong performance on other MCQA datasets like Belebele (wondering if the authors dug deeper into this)?

Were the authors able to do any finetuning experiments? E.g. finetuning on MCQA (maybe even Persian MCQA) datasets and then running the model on this particular task?
paper_summary: paper_summary: This paper describes a new QA dataset for the Persian language. Just like the MMLU dataset for English, this new 'PersianMMLU' dataset covers a wide range of subjects, ranging from the natural sciences to the humanities, and contains more than 20,000 questions. The questions are multiple-choice with 4 answer alternatives, of which one is correct. Each question has its associated metadata, which includes the educational stage for which the question is intended, the difficulty level (which I assume is relative to the intended user group), and a detailed explanation of why the indicated correct answer is correct. The latter is very useful, as it can be used for 'chain-of-thought' training. As a motivation for their work, the authors convincingly point out several shortcomings of using an automatically translated variant of English datasets like the MMLU. The paper also contains experiments of applying various models to answer the questions. An extensive appendix gives lots of details of various aspects of the dataset. 

Unless I missed it, it is not explicitly stated in the paper that the dataset will be made public upon acceptance of the paper. It is merely said that 'all associated code is publicly accessible in [anonymous link]'. I assume the dataset will also be published open to everyone, but the authors should state this explicitly.
strength_weakness: strength_weakness: Unless the authors commit to publishing the dataset free to the world upon acceptance of the paper, the paper should be rejected. I believe the area chair should sort this out with the authors before deciding on publication.

I didn't understand the experimentation section very well. See questions below.
questions: questions: Remove this sales talk, as it is completely unscientific: 
"Since conducting its first examination in 1993, Kanoon has been at the forefront of educational innovation, serving a nationwide network of 450,000 students. The center is highly regarded for its comprehensive range of educational services, notably its facilitation of creating and administering smart, customized tests and providing standardized solutions for exercises through the Learning Ladder platform. This
initiative aims to enable educators to design and administer quality, customized tests effortlessly and to provide students with insightful feedback to enhance their learning process."

 "The quality of the questions in the dataset is exceptionally high, a testament to the institution’s reputable standing in the educational sector" How do you measure the quality of the questions? This just seems like more sales talk. Remove.

"The ongoing development of new questions for various educational stages and subjects allows for the continuous expansion and updating of the dataset, reducing the risks of data contamination and erosion." Are you committing to a regular update of the dataset, or what does this mean, really?

How was the human success rate in Figure 2 computed?

Explain the regex method better. Give examples of answers where it is successful. The "single probability probability" and "full answer probability" were also a bit obscure to me. Shouldn't the model just output a single letter ("a", "b", "c", or "d") indicating the correct answer? 

"In cases where Regex failed to identify the correct option, we utilized a pre-trained model to generate embeddings for each choice..."  Is this still part of the Regex method, or is this an alternative method?

I didn't understand this sentence, please explain: "We chose the answer selected by humans if the combined total of the remaining three options was lower."

Comments: title: Response to the Rebuttal
comment: I acknowledge reading the authors' response and thank them for addressing my questions. l believe my original scorer still correctly reflects my overall evaluation of the paper, and therefore, I'm keeping my score the same. I still believe that the "answer extraction methods" need to be described more clearly and recommend that as a major area of focus and perhaps rewriting for the camera-ready version to the authors.
Thanks for your response.

Regarding the first point about the "bias" of the dataset: While standard examinations can alleviate some bias, sourcing data from a single source can still introduce significant bias, such as in the style and difficulty of the questions. This is something I would like the authors to discuss in the paper.

On the data contamination issue, it's good to see that the authors conducted some numerical checks. However, the hypothesis that "the website features an authentication-required login interface that prevents access through common web crawling methods, thereby helping to prevent data contamination" is not very convincing. I assume there are also many copies outside this specific site if such an exam is a very important one. Therefore, one possible solution would be to check the n-gram overlap of the questions with common crawl dumps, which might be more convincing.

Overall, the authors have addressed some of my concerns, and I would like to increase my score to 6.
Thanks for the explanations. Great that the dataset will be accessible!
Thank you for your rebuttal. I acknowledge having read it.