SlowGuess/NeurIPS_2023_OCR huggingface dataset:
<sup>3</sup>We use Google Translation API (https://translate.google.com/) for translating the data into English.
Table 3: Results on different prompting strategies based on ChatGPT.
Prompt	en	zh	it	pt	vi	th	sw	af	jv
Monolingual	75.98	61.00	67.94	62.43	57.18	34.09	53.04	68.99	37.47
EN-Instruct	-	60.56	69.30	61.42	57.57	32.70	49.30	70.16	38.27
EN-Translation	-	57.92	62.76	59.62	56.40	46.49	48.13	70.16	50.94
Few-shot	75.46	60.26	64.36	62.99	58.64	37.41	51.87	67.05	33.42
of prompts are given in Figure 6 in the Appendix, and the results are presented in Table 3, where we also show the performance of the original prompt ("Monolingual"). We can note that using English instructions does not consistently improve performance, potentially because our data originates from actual language data rather than merely translated English data. Consequently, using English prompts may not better elicit the knowledge required to solve the questions. The impact of using translated data ("EN-Translation") varies across different languages. On one hand, many questions are closely tied to each specific language, translated data may lose essential information in such cases, leading to poorer performance. On the other hand, translations could eliminate some barriers to understanding particular languages, especially those that the ChatGPT model struggles with, such as Thai and Javanese. Therefore, using the English translations of the questions greatly improves their performance.
Zero-shot v.s. few-shot setting To empirically investigate the impact of few-shot demonstrations, we run experiments on both zero-shot and few-shot settings with ChatGPT. Specifically, we use the held-out few-shot samples for each language, and append the few-shot samples after the instruction but before the final testing sample (see Figure 6 for detailed examples). The format for the few-shot samples is the same as the final test sample, except that the correct option is given after "Answer:" for those samples. We present the results of few-shot samples in Table 3, denoted as "Few-shot". It can be noticed that introducing few-shot examples does not necessarily lead to performance improvement on average. While for languages such as Portuguese and Vietnamese, prompting with few-shot examples result in an improvement, the model's performance in other languages such as Chinese and Swahili slightly decreases with few-shot demonstrations. The reason might be that existing LLMs are already familiar with the question format of human exams. Thus using in-context demonstrations does not provide any additional advantages. Moreover, the effectiveness of few-shot learning depends on many factors such as language complexity, the model's knowledge, the selection of few-shot examples, etc.
4.2 Multimodal Evaluation
Table 4: Results on questions with images. We report the performance on both questions with a single image ("Single"), multiple images ("Multi"), as well as the overall scores ("Overall").
Model	Size	# Input img	Single	Multi	Overall
random	-	-	25.00	25.00	25.00
Flan-T5	11B	0	49.70	40.34	48.30
ChatGPT	NA	0	60.36	28.57	55.60
Fromage	7B	many	21.45	30.25	22.77
OpenFlamingo	9B	many	30.18	27.73	29.81
BLIP-2	12B	1	51.18	36.97	49.06
InstructBLIP	12B	1	48.82	32.77	46.62
In Table 4, we present the performance of various models on English questions, as there are no existing LLMs handling both multilingual and multimodal settings. In addition to multimodal models, we provide random guess baselines, the performance of the Flan-T5 model (XXL version) [13], and the performance of ChatGPT. Although Flan-T5 is a text-only model, it has a similar parameter size to the selected multimodal models and serves as the text encoder for both BLIP-2 and InstructBLIP, making it a suitable comparison baseline. Similarly, we also use ChatGPT to understand the extent to which it can perform using only text-based inputs. We only feed the text part for each question to these two text-only models. For BLIP-2 and InstructBLIP models, we only take the first image as the input as they can only process a single image.
The following are multiple-choice questions about natural-science.
The diagram represents a light ray reflecting from a plane mirror. The angle of reflection for the light ray is ()
(A)  $25^{\circ}$
(B)  ${35}^{ \circ  }$
(C)  ${50}^{ \circ  }$
(D)  $65^{\circ}$
Figure 3: An example question with different model outputs (in green) on the left-hand side, as well as model outputs when asked to describe the image in detail on the right-hand side.
Model Outputs:
Fromage: D / Flamingo: B
BLIP-2: D /InstructBLIP:D
What does the image show? Describe all details including the figures, texts and numbers.
Fromage: Here's what it's supposed to look like: Lines under the mirrors are.
Flamingo: Draw a horizontal line so it will fall on the right side of the small object. Draw a line so that it will fall on the other side of the image. Together, draw a perpendicular line from the top of the image from the cross line between the two lines formed with the last two steps.
BLIP-2: a diagram showing the light ray path and path length through a plane mirror
Instruct-BLIP: plane mirror
We observe that most models do not yield satisfactory performance in general. When compared to Flan-T5, only the BLIP-2 model marginally surpasses its performance. This outcome is unexpected, as Flan-T5 can only process text as input and ignore the images, which intuitively suggests that it may lose crucial information. Upon closer examination, we discover that all existing multimodal models struggle to comprehend complex image details in exam questions (e.g., axis details in math questions, map details in geography questions), which are vital for various subjects. We present an example question and the corresponding outputs from different models in Figure 3. To further assess the extent to which models understand the image used in this question, we construct a new prompt: "What does the image show? Describe all details, including figures, texts, and numbers. Answer:" to gauge the models' behavior. As demonstrated in the right portion of the figure, only BLIP-2 captures relatively more accurate information about the image. However, none of the models can accurately discern details such as the marked angle  $65^{\circ}$ , making it impossible for them to solve this question.
For questions involving multiple images, the difficulty increases as cross-image reasoning becomes necessary. However, Fromage and OpenFlamingo, models specifically designed for handling multiple images, do not demonstrate clear improvements. Instead, they perform notably worse than BLIP-2 and InstructBLIP, which are only capable of handling single images. We find that they often struggle to comprehend even individual image details (as shown in the example in Figure 3). This finding suggests that pre-training on multiple images does not necessarily guarantee better multimodal understanding abilities. Overall, in comparison to existing multimodal datasets consisting of relatively simple visual question-answering tasks [5, 16], our M3Exam dataset presents a significant challenge to understanding image details and reasoning under cross-image and cross-modal settings. We provide more examples and discussions in Appendix A.3.
4.3 Multilevel Evaluation
Figure 4: Performance of different LLMs broken down along different levels.
One advantage of the M3Exam dataset is that it encompasses questions from three critical educational periods, namely low, mid, and high, which represent varying levels of difficulty. We here examine LLMs on questions from these three levels. The results are summarized in Figure 4. Comparing the performance on three different levels, the high level indeed generally has the lowest performance, showing its difficulty. Surprisingly, for almost all LLMs, whether text-only or multimodal models, there is no clear decreasing trend as the level increases. This observation contrasts with conventional human behaviors. For example, a high-school student who can achieve reasonable scores in graduation
exams should achieve much better results in exams of lower-level schools. Consequently, we expect that human performance will exhibit a monotonic decrease as the level increases.
This result suggests that although LLMs show impressive results on many tasks and are even said to spark artificial general intelligence [10], the emergence and development of intelligence in LLMs have significant differences from that of human intelligence and require further investigations. This is also reasonable since the "learning process" of LLMs is different from humans. They are typically trained on massive data first, making their knowledge heavily biased towards the data that are more common, while humans often learn from easy principles and knowledge to more complex reasoning and thinking skills. Moreover, this finding indicates that creating more challenging datasets might not be efficient for improving the models [42, 25, 21]. Instead, it might be more crucial to investigate the underlying reasons for LLM failures, even at primary school-level questions, and devise strategies to address these shortcomings.
Performance across various subjects In an effort to better understand the proficiency of models across different subject types, we evaluated ChatGPT's performance in four languages with diverse levels of resources, including English (en), Chinese (zh), Vietnamese (vi), and Thai (th). The results, as displayed in Figure 5, reveal some intriguing patterns. Notably, across all languages, the model tends to underperform in the math category. This suggests that the reasoning skills required in these questions present a great challenge for the model. Conversely, the model exhibits relatively stronger performance in the natural science and social science subjects across all languages, indicating a more effective handling of structured and factual information in these areas.
Figure 5: Performance of ChatGPT across different subject categories.
Large language models (LLMs) have witnessed remarkable advancements in recent years, enabling them to generate human-like text, answer complex questions, and perform a wide range of NLP tasks. These models, such as GPT-3 [9], Claude [4], GPT-4 [31], and PaLM2 [3] have demonstrated exceptional performance on various benchmarks and have been widely adopted in academia and industry. However, the evaluation of these models is a critical aspect that requires careful consideration to ensure reliable and comprehensive assessments.
For the evaluation of NLP models, traditional approaches primarily rely on established NLP benchmark datasets. Popular benchmarks such as GLUE [39], SuperGLUE [38], and SQuAD [34] focus on specific NLP tasks, such as question answering, sentiment analysis, and text classification. To facilitate multilingual evaluation, researchers have also developed multilingual benchmarks such as XTREME [18] and XTREME-R [35]. These benchmarks provide standardized evaluation settings, diverse language coverage, and task-specific evaluation metrics to assess models' performance in a multilingual setting [2, 7, 24]. In the multimodal context, the evaluation often involves assessing the model's ability to understand and generate content that combines multiple modalities, such as text, images, and videos. Some typical evaluation tasks include image captioning [11, 1], image question answering [5, 16], visual reasoning [20], video question answering [41] etc.
Although performance on typical benchmark datasets provides valuable insights into the capabilities of LLMs, it may not be sufficient to evaluate their general intelligence in real-world scenarios. To bridge this gap, there has been a growing trend of utilizing exams originally designed for humans to evaluate LLMs in recent times. An early work is the MMLU [17] dataset, which collects questions covering 57 tasks to test the model's world knowledge and multitask accuracy. More recently, similar benchmark datasets have been proposed following this direction, such as AGIEval [43] with various types of exams, C-Eval [19] and GAOKAO [42] benchmarks using exam questions in Chinese to evaluate Chinese LLMs, and IgakuQA [22] that evaluates ChatGPT on Japanese Medical Licensing Exams. However, these datasets suffer from several limitations, including limited language diversity,
the absence of multimodal evaluation, and the lack of multi-level evaluation. These limitations restrict the comprehensive assessment of LLMs in real-world scenarios.
We introduce M3Exam in this work, a novel benchmark dataset for evaluating LLMs by offering a multilingual, multimodal, and multi-level assessment. Our analysis of top-performing LLMs on M3Exam reveals that current models face challenges in processing multilingual text, especially in low-resource and non-Latin script languages. Additionally, state-of-the-art multimodal models struggle to achieve reasonable accuracy on M3Exam. Overall, it provides a valuable resource for tracking the progress of LLMs in multilingual and multimodal settings and offers insights into the development of model intelligence across various education levels. However, M3Exam only considers multiple-choice questions for now, making it unsuitable to evaluate LLMs for questions requiring creative writing. We will consider such questions in our future work.
We extend our gratitude to the data management and annotation team at Alibaba DAMO Academy, particularly Yanyan Zheng, Tantong Champaiboon, Nguyen Ngoc Yen Nhi, and Andila Putri Susanti. Their kind assistance in coordinating the annotators, processing the data, and conducting quality checks has been important to our work. We would also like to thank the annotators from various linguistic and cultural backgrounds who collaborated to contribute to this unique multilingual dataset.
[1] Harsh Agrawal, Peter Anderson, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, and Stefan Lee. nocaps: novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, pages 8947-8956. IEEE, 2019.
[2] Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. MEGA: multilingual evaluation of generative AI. CoRR, abs/2303.12528, 2023.
[3] Rohan Anil, Andrew M. Dai, Orhan First, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Diaz, Nan Du, Ethan Dyer, Vladimir Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, and et al. Palm 2 technical report. CoRR, abs/2305.10403, 2023.
[4] Anthropic. Introducing claude., 2023.
[5] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, pages 2425-2433, 2015.
[6] Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo, March 2023.
[7] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity. CoRR, abs/2302.04023, 2023.
[36] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilic, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Galle, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoit Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunj Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, and et al. BLOOM: A 176b-parameter open-access multilingual language model. CoRR, abs/2211.05100, 2022.
[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023.
[38] Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, pages 3261-3275, 2019.
[39] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupala, and Afra Alishahi, editors, Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, pages 353-355, 2018.
[40] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David S. Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. CoRR, abs/2303.17564, 2023.
[41] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, pages 1645-1653. ACM, 2017.
[42] Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, abs/2305.12474, 2023.
[43] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. CoRR, abs/2304.06364, 2023.
A.1 Dataset Documentation
We provide additional information on the introduced M3Exam dataset in this section.
A.1.1 Motivation
M3Exam is created to test models' multilingual and multimodal abilities through questions that are from real and official human exams. Current benchmarks on multilingual and/or multimodal evaluation still mainly focus on traditional NLP tasks, which have several limitations as described in the paper, we aim to bridge this gap through M3Exam. Moreover, we aim to provide insights into the development of machine intelligence through questions from different education levels.
A.1.2 Composition
- M3Exam contains textual questions, part of them need images to solve.
- There are 12,317 questions in total.
- Questions are from exam papers across multiple years for each language, thus they are representative of the expected knowledge of certain languages.
- M3Exam is self-contained. Part of the questions requiring images are released with the corresponding images and clearly identified.
- The dataset does not involve any specific person and does not contain any information that might be offensive, insulting, or threatening.
A.1.3 Usage and Distribution
- The dataset is released at https://github.com/DAMO-NLP-SG/M3Exam.
- The data is saved in JSON format, where an example is shown in the README.md file. An example code snippet is also provided showing how to read and process the data.
License: M3Exam is under CC BY-NC-SA License.
A.2 Examples of different prompting strategies
We show some detailed examples of different prompting strategies in Figure 6. It can be noticed from the example of the Thai question, that even when translating the data into English, the question might still be difficult to answer since it may involve background knowledge of each specific language.
A.3 More examples on multimodal questions
We present two additional examples of questions involving images in Figure 7. In the first question, even though an image is required to answer, the keyword "Gandhi" is already mentioned in the question text. As a result, a text-only model like Flan-T5 might be capable of providing the correct answer. Upon further examination of the models' descriptions, we observe that BLIP-2 and Instruct-BLIP offer relatively more accurate descriptions. BLIP-2 refers to nonviolent resistance or civil resistance, demonstrating its ability to capture the semantic or conceptual meaning of the images. Instruct-BLIP, on the other hand, provides a detailed description of the physical activity depicted in the figure. However, neither Fromage nor Flamingo delivers relevant descriptions of the photograph, or gives hallucination descriptions.
In the second example, we select a question with images present in the options. To obtain the descriptions, we separately feed each image to the model and show models' generations of two option images for simplicity. It is worth noting that for this relatively easy image understanding task, all models seem to provide more accurate descriptions. They can recognize the physical objects in the images and even identify the specific number of cubes in some cases. However, it is still difficult for them to consistently give accurate descriptions for all options. Overall, we can see that multimodal questions in M3Exam post a great challenge for existing multimodal models compared to previous multimodal tasks since they require a more accurate understanding of the involved images and may even need to reason across multiple images.
Zero-Shot Prompting	Few-Shot Prompting	English-Instruction
The following is a multiple choice question about Social Science. Please only give the correct option, without any other details or explanations.	The following are multiple choice questions about Social Science. Please only give the correct option, without any other details or explanations.	The following is a multiple choice question about Math. Please only give the correct option, without any other details or explanations.
What is a 100-year period of time called?
(A) decade
(B) millennium
(C) century
(D) light-year	Which term is used to describe money collected to pay for the services that a community provides?
(A) savings
(B) profit
(C) interest
(D) taxes	将3克药放入100克水中，药与药水的比是
(A) 3:97
(B) 3:100
(C) 3:103
Answer: English-Translation
Answer:
以下是关于数学的单项选择题，请仅给出正确选项对应的选项序号而非其他细节。	Answer: D	The following is a multiple choice question about Math. Please only give the correct option, without any other details or explanations.
将3克药放入100克水中，药与药水的比是
(A) 3:97
(B) 3:100
(C) 3:103	John's country has a prime minister and a Parliament that plan, organize, and make decisions. What does this statement describe about John's country?
(A) its interdependence
(B) its geography
(C) its population
(D) its government	Put 3 grams of medicine into 100 grams of water, the ratio of medicine to medicine water is
(A) 3:97
(B) 3:100
(C) 3:103
Answer:
答案:
The following is a multiple choice question about Social Science. Please only give the correct option, without any other details or explanations.
“A group of tourists travels by boat, admiring the beautiful landscape, such as Tham Lot Cave, Khao Phing Kan, Khao Ta-Pu, and also visiting the fossils at the Shell Cemetery.” In which area of Thailand do these tourists travel?
(A) Gulf of Thailand coast
(B) Andaman Sea coast
(C) Eastern Seaboard coast
(D) Eastern coast of Southern Thailand
Answer: