SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Wenxuan Zhang $^{1}$ , Sharifah Mahani Aljunied $^{1}$ , Chang Gao $^{1,2*}$ , Yew Ken Chia $^{1,3*}$ , Lidong Bing $^{1}$ $^{1}$ DAMO Academy, Alibaba Group
<sup>2</sup>The Chinese University of Hong Kong <sup>3</sup>Singapore University of Technology and Design {saike.zwx, mahani.aljunied, gaochang.gao, yewken.chia, l.bing}@alibaba-inc.com
Despite the existence of various benchmarks for evaluating natural language processing models, we argue that human exams are a more suitable means of evaluating general intelligence for large language models (LLMs), as they inherently demand a much wider range of abilities such as language understanding, domain knowledge, and problem-solving skills. To this end, we introduce M3Exam, a novel benchmark sourced from real and official human exam questions for evaluating LLMs in a multilingual, multimodal, and multilevel context. M3Exam exhibits three unique characteristics: (1) multilingualism, encompassing questions from multiple countries that require strong multilingual proficiency and cultural knowledge; (2) multimodality, accounting for the multimodal nature of many exam questions to test the model's multimodal understanding capability; and (3) multilevel structure, featuring exams from three critical educational periods to comprehensively assess a model's proficiency at different levels. In total, M3Exam contains 12,317 questions in 9 diverse languages with three educational levels, where about  $23\%$  of the questions require processing images for successful solving. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text, particularly in low-resource and non-Latin script languages. Multimodal LLMs also perform poorly with complex multimodal questions. We believe that M3Exam can be a valuable resource for comprehensively evaluating LLMs by examining their multilingual and multimodal abilities and tracking their development. Data and evaluation code is available at https://github.com/DAMO-NLP-SG/M3Exam.
In recent years, large language models (LLMs) have demonstrated remarkable capabilities in various natural language processing (NLP) tasks [9, 31, 3, 4]. For instance, ChatGPT [30] shows an impressive ability to effectively respond to a wide range of questions and provide high-quality answers [7]. Their applications even extend beyond traditional NLP domains, as they have been integrated to address real-world challenges in diverse areas [22, 27, 40]. Given the increasing reliance on LLMs, the need for appropriate and comprehensive evaluations has become ever more critical. Such assessments should not only examine whether the models exhibit strong language understanding, but also evaluate their capabilities to handle complex problems requiring different kinds of skills [31].
Typically, NLP models are evaluated using well-designed benchmarks for specific tasks, such as SQuAD [34] for question-answering or WMT [8] for machine translation. Although useful, these task-specific benchmarks often emphasize on certain aspects, and thus do not adequately assess the breadth
*Chang Gao is a research intern at Alibaba. Yew Ken Chia is under the Joint Ph.D. Program between Alibaba and SUTD.
Figure 1: Example questions from M3Exam dataset. Correct answers are indicated in bold. Meta-information is provided with blue boxes attached to each question.
of abilities possessed by powerful modern LLMs. In comparison, tackling human exam questions often require diverse skills such as language understanding, complex reasoning, etc. Therefore, a new trend has emerged to utilize tests originally designed for humans to assess the performance of LLMs. For example, MMLU [17] contains exam questions covering 57 tasks across a diverse set of subjects for assessing models. AGIEval [43] collects questions from standardized exams such as law school admission tests and math competitions. GPT-4 [31] also uses a variety of human exams to test its ability in complex scenarios. These human-centric evaluations, approximating real-world applications and expectations, have been demonstrated to be valuable testbeds for gauging the artificial general intelligence (AGI) capabilities of LLMs.
Despite the advantages of evaluations based on human exams, current benchmarks exhibit several key limitations. Firstly, the majority of these benchmarks focus on questions in English [17, 43], neglecting the evaluation of a model's performance in a multilingual context. As many LLMs exhibit multilingual ability and are widely used across different countries and languages [30, 3, 4], it is important to evaluate their multilingual capabilities. Moreover, some existing multilingual benchmarks of traditional NLP tasks [14, 33, 31] that were created by translating original English datasets have been found to introduce an English-centric bias. This bias arises because the translation process, while making the benchmarks available in multiple languages, does not always capture culturally specific or unique concepts present in the target languages. This shows the importance of sourcing real data from various languages to represent their native cultural background [28]. Secondly, most benchmarks consider solely text-based questions, ignoring a significant portion of real-world exam questions include images. Considering this type of question is essential to test a model's multimodal understanding abilities in a wide range of practical applications [15]. Lastly, existing exam-type benchmarks usually draw from mixed exams such as college final exams or professional certificate tests [17, 43], the constructed resources are thus also comprised of questions from mixed levels. Gathering exam questions from varying educational levels is critical to assess and understand the level of intelligence that LLMs have developed. [10].
In this paper, we present M3Exam, a novel benchmark dataset designed for evaluating the artificial general intelligence of large language models. M3Exam has several unique characteristics: (1) Multilingualism - by gathering questions from official exams across multiple countries, the benchmark contains natural multilingual questions in different languages, which retain the social-cultural diversity of knowledge that may be essential for problem-solving; (2) Multimodality - we incorporate all types of questions including those that require images and carefully process these images to facilitate convenient model evaluation. We show that a significant proportion (approximately  $23\%$ ) of questions demand information from images for solving; (3) Multilevel structure - we adopt a top-down approach for data collection where we first select three critical educational periods (primary, middle, and high school) and source official exams from the culmination of each period, resulting in a benchmark with varying levels. In total, M3Exam comprises 12,317 questions in 9 diverse languages, with 2,816 questions involving one or more images. Each question includes the question text, candidate answer
options, ground-truth answer, and rich meta-information consisting of language, education level, subject, and whether images are involved. Some examples are shown in Figure 1.
We utilize a wide range of top-performing LLMs in both multilingual and multimodal settings to assess their performance on the newly introduced M3Exam dataset. Our findings indicate that the majority of existing models have difficulties in processing multilingual text, with GPT-4 [31] being the only model to achieve over  $60\%$  accuracy. Nevertheless, it still faces challenges with low-resource languages such as Javanese, and non-Latin script languages like Thai. Current multimodal models also underperform on M3Exam, with state-of-the-art models such as BLIP-2 [26] attaining less than  $50\%$  accuracy. A detailed examination further reveals that comprehending complex images and reasoning across images remain quite challenging for current models. Moreover, we surprisingly find that LLMs' performances do not show a monotonic decrease with the educational level increases which is quite different from human behavior, implying that the development of intelligence in LLMs may not necessarily align with that of human intelligence. Overall, we believe M3Exam can serve as a valuable resource for examining LLMs, both tracking their improvements in terms of multilingual and multimodal settings and providing insights into the development of model intelligence with different education levels.
2 M3Exam Benchmark Dataset
Exams are widely used to assess human intelligence at various educational stages, as they draw on the integration of diverse skills, including language understanding, world knowledge, cultural awareness, and logical reasoning, etc. Consequently, exam questions offer an ideal testbed for evaluating the general intelligence of LLMs. We propose three crucial design principles for constructing the M3Exam benchmark dataset with the exam questions:
- Multilingual Evaluation: While most existing datasets primarily focus on English, assessing LLMs' abilities in multiple languages with different cultural backgrounds, especially those low-resource languages, is crucial to apply LLMs in broad-range scenarios. To achieve this, collecting real-world natural data of different languages instead of translating from English data is of great importance, as culture and world knowledge are deeply rooted in authentic data [28].
- Multimodal Evaluation: In real-world scenarios, humans often encounter problems with different modalities such as images or audio. Multimodal evaluation is thus essential for testing an LLM's ability to jointly process information from multiple modalities, which reflects a key part of cognition capability. Therefore, to facilitate such evaluation, we include questions requiring images for successful solving.
- Multilevel Evaluation: Although education systems vary across countries, they typically organize learning into several stages (e.g., from primary school to middle school and then to high school), with examinations to assess students' readiness to advance to the next stage. The exams at the end of each period effectively reveal the general intelligence expectations within each country. Thus, evaluating LLMs with questions from these critical educational stages offers a comprehensive assessment of their capacity with different levels of intelligence requirements.
2.2 Language Selection and Data Collection
Following the design principle outlined above, we adopt a top-down approach to construct our dataset. To comprehensively evaluate the model, we select 9 languages including English (from the US), Chinese (from China), Italian (from Italy), Portuguese (from Brazil), Vietnamese (from Vietnam), Thai (from Thailand), Swahili (from Kenya), Afrikaans (from South Africa), and Javanese (from Indonesia). This selection is mainly driven by language and cultural diversity, with the aim of covering different language families, languages with varying levels of resources, written scripts, and their major spoken countries.
We then engage native speakers from each of the selected countries to collect official exam papers along with their answers at the end of each educational level, which are typically the graduation exams of primary school, middle school, and high school. We encourage them to 1) choose exams with the largest possible participation (e.g., if a period has two exams, one nationwide and one statewide, the
Table 1: Data statistics of M3Exam dataset. We rank languages by their ratio in the CommonCrawl corpus ("CC Size"), and report the detailed number of questions at each level: X/Y denotes there are X questions involving only pure text, and Y questions requiring images to solve.
Language	Code	Country	CC Size	Low	Mid	High	Total
English	en	US	46.175	306 / 106	505 / 204	1132 / 485	1943 / 795
Chinese	zh	China	4.632	83 / 14	347 / 335	281 / 104	711 / 453
Italian	it	Italy	2.726	220 / 107	291 / 140	318 / 160	829 / 407
Portuguese	pt	Brazil	1.131	86 / 96	182 / 98	645 / 278	913 / 472
Vietnamese	vi	Vietnam	1.056	170 / 16	361 / 12	1286 / 88	1817 / 116
Thai	th	Thailand	0.440	472 / 113	568 / 174	1154 / 114	2194 / 401
Swahili	sw	Kenya	0.008	186 / 4	248 / 0	-	434 / 4
Afrikaans	af	South Africa	0.007	91 / 36	138 / 63	54 / 64	283 / 163
Javanese	jv	Indonesia	0.004	205 / 4	172 / 1	-	377 / 5
Total	1819 / 496	2812 / 1027	4870 / 1293	9501 / 2816
nationwide exam should be collected); 2) collect all available subjects and up to five papers across different years for each subject to ensure a diverse range of questions. In the end, we collect in total 435 exam papers from nine countries. The details of those exam papers are in Appendix A.4.
2.3 Data Processing and Annotation
Given the diverse languages we consider, many collected exam papers are only available as images or in scanned versions. Therefore, we first conduct OCR to convert these papers into editable text versions. The original papers, editable text versions, and corresponding answer sets are then passed on to annotators of each specific language, who transform the data into a unified format. In terms of question scope, we focus on multiple-choice questions, as they allow for a standard automatic evaluation of the correctness of model outputs. We exclude subjective questions with free-response answers but include questions that can be easily adapted into the multiple-choice format, such as judging true-false statements.
Specifically, the annotators are asked to check the text content and fix potential errors due to OCR transformations. Then they need to separate the question text from a list of candidate answer options, and input the correct answer. We also address a limitation observed in previous benchmarks: inadequate or limited context information. Many questions require rich contextual information to answer, such as reading comprehension questions with passages or chemistry problems featuring brief introductions to new chemical phenomena. We specifically ask annotators to include such contextual background information. Furthermore, we also convert special formats into pure text, such as converting all equations into LaTeX format or using <br> to include a text span for representing a bold font. All these format adaptations aim to make the constructed benchmark mimic the real exam scenario. Multiple rounds of quality checks are conducted to ensure the data quality.
To accommodate questions containing both text and images, we instruct annotators to annotate images with placeholders, regardless of whether the images appear in the question text or option text. This ensures clarity on whether an image is needed and its original placement in the question. For example, (image) [image-x.jpg] will appear in the place of an image in the transformed text, and the corresponding image will be clipped and saved with the same name (i.e., image-x.jpg).
2.4 Data Statistics
At the end of the annotation and quality check, our newly introduced M3Exam dataset contains a total of 12,317 questions. Each question includes context information, the main question text, candidate options, the correct answer, and meta information, such as its language, level, subject, and whether images are needed to solve the question. Figure 1 shows examples of some questions.
Table 1 provides detailed statistics of M3Exam, broken down by language and level. The number of questions that involve only pure text, or require images are separately listed. We rank languages by their ratio in the CommonCrawl corpus, which is a widely-used data source for training LLMs. It can be observed that our selected languages span a wide range, from high-resource languages like English and Chinese, to extremely low-resource languages such as Javanese. Therefore, the diversity
of selected languages makes it well-suited for comprehensively assessing the multilingual capabilities of the model. The ratio of questions requiring images also varies across countries, from over  $60\%$  questions with images for Chinese, to languages with very few image-type questions. After obtaining the data, we group the questions for each language into four subject categories, namely language, math, social science, and natural science. We then randomly select three questions for each subject category of each level in each language and separate these as held-out development data, which can be used as in-context examples. The remaining questions are used as test data during the experiment.
3 Experiment Setups
To evaluate the performance of various LLMs on our newly introduced M3Exam dataset, we select a range of top-performing models in either multilingual or multimodal settings.
Text-only LLMs To process multilingual texts, we first take ChatGPT (gpt-3.5-turbo) [30] and GPT-4 (gpt-4) [31] from OpenAI, both of which have demonstrated strong multilingual abilities in preliminary studies [2, 7, 24, 31]. Additionally, we also adopt Claude (Claude-instant) from Anthropic, a model which is often considered to be comparable to ChatGPT [19]. We obtain the results of those close-source models via API call with the corresponding model type. Furthermore, we utilize two open-source models, namely BLOOM (176B) [36] and Vicuna (13B) [12]. BLOOM stands out as one of the largest open-source LLMs specializing in multilingual ability, having been trained with data encompassing 46 languages and 13 programming languages. Vicuna, on the other hand, was developed by fine-tuning the LLaMA model [37] on user-shared conversations. Although not specifically designed as a multilingual model, recent leaderboards have identified Vicuna as the top-performing open-source model on both English-only and non-English leaderboards [29].
Multimodal LLMs To evaluate LLMs on multimodal questions, we consider a range of state-of-the-art open-source models since closed-source models such as GPT-4 do not have official multimodal versions available currently. Specifically, we employ BLIP-2 [26] and InstructBLIP [15], which have demonstrated leading performance in various multimodal question-answering tasks. However, these models are limited to processing a single image per question. Since our M3Exam data may contain multiple images in the background description or as answer options, we additionally utilize Fromage [23] and OpenFlamingo [6], both of which are capable of handling multi-image inputs. We use their pre-trained model weights to directly conduct inference on our test data, and further impose a constraint decoding to generate only valid multiple-choice options for those models.
3.2 Settings
Zero-shot Evaluation We primarily evaluate various LLMs in zero-shot settings. There are three considerations for this setup decision: First, using a zero-shot approach to prompt the model mimics the natural process in real-world applications and the problem-solving process in exams. Secondly, most LLMs have limited context lengths (especially for multilingual models handling diverse languages) or cannot accept multiple images as input (for multimodal models), rendering them unsuitable for evaluation using multiple few-shot demonstrations. Third, since the majority of existing LLMs have undergone instruction tuning [32], they are readily capable of following instructions to output in the desired format. Nonetheless, we also compare zero-shot and few-shot settings from an empirical standpoint with ChatGPT in Section 4.1.
Prompt Following the convention of previous studies [17, 31], we clearly specify the subject type of each question by starting with "The following is a multiple choice question about {subject type}." Subsequently, we include an instruction
The following is a multiple choice question about Social Science. Please only give the correct option, without any other details or explanations.
What is a 100-year period of time called?
(A) decade (B) millennium
(C) century (D) light-year
以下是关于数学的单项选择题，请仅给出正确选项对应的选项序号而非其他细节。
将3克药放入100克水中，药与药水的比是
(A) 3:97 (B) 3:100 (C) 3:103
Figure 2: Illustrations of the prompt with two short questions. Model outputs are marked in green. Each option will take a new line in practice.
Table 2: Results on questions of different languages. Accuracy scores are reported.
en	zh	it	pt	vi	th	sw	af	jv	avg
random	25.01	25.93	33.77	21.41	25.21	22.89	25.00	25.05	25.00	25.47
passing	60.00	60.00	60.00	60.00	50.00	50.00	40.00	50.00	60.00	54.44
BLOOM	28.62	29.47	33.17	7.20	23.81	9.09	27.10	23.26	26.95	23.19
Vicuna	56.99	29.18	35.39	41.73	27.33	15.08	24.07	33.33	27.49	32.29
Claude	74.25	51.61	61.90	62.54	51.65	31.27	38.32	63.95	30.73	51.80
ChatGPT	75.98	61.00	67.94	62.43	57.18	34.09	53.04	68.99	37.47	57.57
GPT-4	87.55	79.47	83.23	74.24	70.49	56.04	65.89	84.11	55.26	72.92
"Please only give the correct option, without any other details or explanations." to constrain the model output for automatic evaluations. A question is then presented, along with its corresponding options each in a new line. Finally, the prompt ends with "Answer:" for the model to generate its output. It is important to note that all prompts are language-specific [31]. We translate the prompt for each language to ensure that the entire prompt presented to the model is monolingual. This prompt design is the same in both multilingual and multimodal settings, except we omit the format constraint for multimodal experiments as constraint decoding is applied. Two example prompts are shown in Figure 2. Detailed prompts as well as examples of different prompting strategies are provided in Appendix A.2.
**Evaluations** As all the questions are multiple-choice questions, we utilize accuracy as the evaluation metric. In most cases, the models can adhere to the instructions and produce only the option. Consequently, we take the first alphabetic letter of the model's output as the prediction and compare it with the ground truth answer to calculate the accuracy scores.
4.1 Multilingual Evaluation
Main multilingual results We present the results of various LLMs on different language data in Table 2. We also show the scores of random guesses ("random") and the conventional scores that are considered as passing the exam ("passing").<sup>2</sup> Overall, we observe that most models can only achieve less than  $60\%$  accuracy, with GPT-4 being a notable exception, achieving  $72.92\%$  and consistently outperforming all other models across different languages. BLOOM, although a multilingual model, gives unsatisfactory performance and is even worse than the random guess since it may generate invalid options. ChatGPT, Claude, and Vicuna show varying degrees of performance depending on the language. Vicuna, despite having a much smaller model size, gives reasonable performance for Latin-script languages. While ChatGPT and Claude have relatively similar performance in English (75.98% v.s. 74.25%), ChatGPT demonstrates better results in other languages, suggesting a more robust multilingual ability. When comparing performance across different languages, we observe that existing models generally perform worse for non-Latin languages, such as Chinese (despite being relatively high-resource), as well as low-resource languages like Javanese (even though it mostly uses the Latin script). In summary, the results on our newly introduced M3Exam dataset highlight the challenges and limitations faced by current LLMs in handling non-Latin and low-resource languages, suggesting that there is still a large room for improvement in their multilingual capabilities.
Handling non-English questions with different prompting strategies In multilingual settings, some pilot studies have discovered that using English task instructions [24] or employing a translate-test approach (i.e., translating target language data to English) [2] can lead to improved performance compared to using monolingual prompts in a specific language. To analyze the impact of different prompting strategies, we follow such two settings to create another two types of prompts for ChatGPT, denoted as "EN-Instruct" and "EN-Translation", respectively. Detailed examples of these two types
Note that the exact passing line depends on each specific exam. Here we provide the scores conventionally used in the corresponding countries, which can indicate the relative difficulty of the questions.