CMMLU: Measuring Massive Multitask Language Understanding in Chinese (2024)

Abstract: As the capabilities of large language models (LLMs) continue to advance, evaluating their performance is becoming more important and more challenging. This paper aims to address this issue for Mandarin Chinese in the form of CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural sciences, social sciences, engineering, and the humanities. We conduct a thorough evaluation of more than 20 contemporary multilingual and Chinese LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an accuracy of even 60%, which is the pass mark for Chinese exams. This highlights that there is substantial room for improvement in the capabilities of LLMs. Additionally, we conduct extensive experiments to identify factors impacting model performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models for Chinese.1

heading: 1 Introduction
text: Large language models (LLMs) have driven remarkable advancements in natural language processing and artificial intelligence, revolutionizing the field (Zhang et al., 2022; Scao et al., 2022; Zeng et al., 2023; Touvron et al., 2023a; OpenAI, 2023; Wu et al., 2023; Taori et al., 2023; Li et al., 2023a). However, assessing the knowledge and reasoning abilities of these models has become increasingly challenging, especially with the proliferation of LLMs that generate fluent and plausible responses.
To this end, researchers have created various benchmarks intended to evaluate different model capabilities (Wang et al., 2019b,a; Lin et al., 2022; Zellers et al., 2019; Hendrycks et al., 2021b; Chen et al., 2021). Specifically, Hendrycks et al. (2021a)
1The data and evaluation code are available at https: //github.com/haonan-li/CMMLU.
proposed MMLU, a benchmark that encompasses various tasks ranging from elementary mathematics and computer science to management and law, which can be used to comprehensively measure LLM capabilities in terms of the knowledge embedded in them. Due to its multiple-choice question format, which facilitates easy evaluation, and the breadth of subject areas it encompasses, it has become widely used as a fundamental assessment tool of the knowledge encoded by LLMs. However, this benchmark is in English, which limits its ability to assess LLMs in other languages. Although some researchers (OpenAI, 2023) have attempted to automatically translate it to evaluate LLMs in other languages, the inherent bias towards Western (and specifically US) culture in the dataset renders it unsuitable and even inappropriate for assessing LLMs across diverse cultures and languages.
In this paper, we propose CMMLU (Figure 1), a comprehensive Chinese assessment suite specifically designed to evaluate the advanced knowledge and reasoning abilities of LLMs in a Chinese linguistic and cultural context. CMMLU covers a wide range of subjects, comprising 67 topics from elementary to advanced professional levels. It includes subjects that require computational expertise, such as physics and mathematics, as well as disciplines within the humanities and social sciences. Many of these tasks are not easily translatable from other languages due to their specific contextual nuances and wording. Furthermore, numerous tasks within CMMLU have answers specific to China, which may not be universally applicable or considered correct in other regions or languages.
We assess GPT4, ChatGPT, and more than 20 advanced open-source multilingual and Chinese LLMs on CMMLU. The results reveal that the majority of these models struggle to achieve an accuracy score of 60%, relative to random accuracy of 25%. Notably, GPT4 achieves an average accuracy of 71%. These findings highlight the considerable
11260
room for improvement in LLMs in terms of Chinese knowledge and language understanding.
To gain a deeper understanding of the proficiency of the models in handling Chinese knowledge, we conduct a comprehensive analysis. We first focus on examining model performance across various subjects and find that all models exhibit uneven performance across different subjects, with comparatively higher scores in humanities and social sciences, but lower scores in China-specific and STEM subjects.
Furthermore, through extensive experiments, we find that: (1) most existing models do not benefit from chain-of-thought prompts in CMMLU; (2) few-shot examples help base models in the comprehension of tasks and enhance their reasoning abilities but do not help models that have undergone supervised fine-tuning (SFT) or reinforcement learning from human feedback (RLHF); (3) LLMs perform worse on questions with negation words compared to those without negation words, but recently-released models mitigate this disparity either through better pre-training data or fine-tuning; and (4) questions with sub-options (Section 4.2) are difficult for all existing LLMs, with even GPT4 dropping 20% in accuracy over such questions.

heading: 2 Related Work
text: Benchmarking plays a crucial role in measuring AI development, particularly in the domain of LLMs.
While benchmarks such as GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) have played an important role in tracking progress in natural language understanding (NLU) tasks, they primarily focus on specific language skills. With an increasing move to generative models which are highly adept at generating fluent outputs, the value of these benchmarks has diminished, and new datasets have been proposed to evaluate LLM abilities over more general tasks, such as reading comprehension (Rajpurkar et al., 2018; Kwiatkowski et al., 2019; Li et al., 2022), summarization (Hermann et al., 2015), commonsense reasoning (Clark et al., 2018; Talmor et al., 2019; Sakaguchi et al., 2020), mathematical reasoning (Hendrycks et al., 2021b; Cobbe et al., 2021), and code generation (Chen et al., 2021; Austin et al., 2021).
In order to comprehensively assess the capabilities of LLMs, some benchmarks have incorporated massive multi-task evaluations (Hendrycks et al., 2021a; Liang et al., 2022; Srivastava et al., 2023). An example is MMLU (Hendrycks et al., 2021a), which includes multiple domains and tasks based on real-world exams. It has become very popular for English LLM evaluation due to its standardized and simplified format, comprehensive nature, and real-world relevance. There have also been variants of MMLU in other languages (Koto et al., 2023, 2024).
Given that Mandarin Chinese has the largest
number of speakers worldwide, several benchmarks have been proposed for Chinese LLM evaluation. Following in the footsteps of GLUE and SuperGLUE, Xu et al. (2020) introduced CLUE, a benchmark for Chinese NLU that is widely used today. They also recently proposed SuperCLUE (Xu et al., 2023), which specifically focuses on LLMs. Recently, several Chinese benchmarks have emerged in the style of MMLU, which are concurrent work with this paper. In detail, Zhang and Li (2023) proposed ACLUE, focusing on ancient Chinese language understanding. Zeng (2023) presented MMCU, which covers four major domains (medicine, law, psychology, and education), with a particular focus on medicine and education. AGIEval (Zhong et al., 2023) includes problems from Chinese and English standardized exams. CEval (Huang et al., 2023) and M3KE (Liu et al., 2023) cover more than 50 tasks from standard exams in China, while C-Eval covers various professions, and M3KE focuses on education exams.
Compared to these benchmarks, CMMLU has several distinct features. Firstly, it includes more than 10 subjects that are not typically found in standard exams but are relevant to daily life, such as Chinese food culture, and Chinese driving rules. Secondly, it covers not only China-specific knowledge but also general world knowledge, such as world religion, world history, and global facts. Lastly, we have made our data completely public, enabling the community to evaluate their models freely and conveniently. A detailed comparison between CMMLU and other concurrent benchmarks is provided in Section 3.

heading: 3 CMMLU
text: Task Overview We created an extensive multitask test for Mandarin Chinese, which covers diverse areas of knowledge, including the humanities, social sciences, STEM (science, technology, engineering, and mathematics), and other areas that are important in daily life. It includes common test questions in subjects like mathematics, physics, and chemistry with answers that are not language or region specific, but also several tasks that are very region-specific, such as Chinese driving rules, Chinese food culture, and Chinese teacher qualifications. The questions in these tasks involve lots of China-related knowledge and can test a model’s understanding and adaptability to Chinese. In addition, CMMLU also contains tasks that can only
expressed in Chinese, such as ancient Chinese language and Chinese literature. The terms and concepts involved in these tasks rely heavily on specific Chinese expressions that cannot reasonably be attained through translation. The full list of subjects, the concepts tested in each subject, the number of questions, and the statistics of question and answer lengths are provided in Appendix A.
Data collection We hired four annotators with undergraduate or higher education levels to manually collect the questions and answers from freely available resources, at a rate of 50 CNY per hour. To prevent our questions from appearing in the training set of LLMs, we invested specific effort in identifying non-publicly available materials, mock exam questions, and questions from quiz shows. More than 80% of our data was crawled from PDFs (after OCR), which further reduces the possibility of it occurring in LLM training data. The entire collection process took around 250 hours.
Format Each question in the dataset is a multiplechoice question with 4 choices, only one of which is correct; see Figure 2 for an example. The questions are expressed as fill–in–the-blank (by choosing the correct option), or direct-answer questions. For chemical formulae and mathematical expressions, we use a 50:50 mixture of LATEX and plain text, where plain text was only allowed if an expression is commonly used and not prone to ambiguity (as judged by the annotators). For instance, the chemical expression for water can be written in plain text as H2O, or in LATEX format as $H_{2}O$.
Quality Check To further check data quality, we randomly sampled 5% questions with answers for each subject, and conduct detailed verification through online resources. We estimate that there is around 2% of noise in the data, in terms of the correct answer not being present or being incorrectly labeled. Based on the results in Section 4 that most models struggle to achieve an average accuracy of 60%, we believe such an error rate does not compromise the overall results.
Statistics CMMLU contains 11,528 questions across 67 subjects. Each subject has at least 105 questions, which we split into a few-shot development set with 5 questions, and a test set with more than 100 questions. In terms of task types, CMMLU comprises 17 STEM tasks, 13 humanities tasks, 22 social science tasks, and 15 other tasks.
Among these, 16 tasks are China-specific, which means they either do not exist in other countries or regions, or their answers may be different in other places. We provide an example for each subject type in Appendix C.
Table 1 presents a breakdown of the composition of the CMMLU test set for each supercategory, including the number of tasks, number of questions, average question counts for each subject, maximum and minimum counts of questions, and average token length for question and choices. Meanwhile, Figure 7 provides a visualization of the token lengths of questions and answers for each subject.
Comparison to concurrent benchmarks CEval (Huang et al., 2023) and M3KE (Liu et al., 2023) are two similar benchmarks concurrent with our work. We compare the task distribution of these benchmarks in Table 2, and demonstrate that CMMLU contains more culture-related and regionrelated tasks. While there are differences in task distribution, we acknowledge that these datasets exhibit similarities in the task types and can, therefore, be jointly used as assessment criteria for evaluating
the Chinese language capabilities of large models. We further assess the overlap between CMMLU and both of these benchmarks. For this purpose, we first sort the four choices for each question to eliminate the influence of choice order. Subsequently, we concatenate the question string with the sorted choice strings. Then, we remove all punctuation marks, including underscores and brackets, from the resulting strings. The final overlap, computed using exact string matching, yields a total of 74 for CEval and 158 for M3KE. This overlap accounts for approximately 1% of our dataset.

heading: 4 Experiments
text: To provide an overview of existing LLMs on language understanding for Chinese, we evaluate two commercial LLMs and more than 20 opensource LLMs of different sizes, language combinations, and training strategies (i.e. either base or SFT/RLHF models). We analyse their performance and investigate several factors that could affect LLM performance.
Setup Our goal is to assess LLM performance on CMMLU, which contains multiple-choice ques-
tions with one correct answer per question. There are several strategies to perform multiple-choice question answering. In this paper, for closed models where we cannot access the model weights (e.g. ChatGPT), we input the question with all candidate choices, have the model to generate the output, and use a series of regular expressions (regex) to extract the prediction. We call this the free generation strategy. For open-source models, we follow Hendrycks et al. (2021a) in inputing the question and choices, and prompt the model by asking for the answer key. Then we obtain the logits of the next predicted token, and select the token with the highest probability from ‘A’, ‘B’, ‘C’, and ‘D’ . We name this the next token prediction strategy. A third strategy is to select the answer with the lowest perplexity when concatenated with the question.
We compared the different strategies in Appendix G, and found that next token prediction to be the most effective method. Therefore, for the remainder of the paper, we report results for next token prediction, other than in Section 4.2 where we use free generation for part of our analysis, using a regex customized to ChatGPT and ChatGLM responses (see Appendix H for details).
Prompt We introduce each question with the phrase以下是关于[主题]的单项选择题，请直 接给出正确答案的选项 “Here are some multiplechoice questions about [subject], please provide the correct answer directly”, and evaluate models in both zero-shot and few-shot settings. For zeroshot evaluation, we present a question with choices directly after the prompt. For few-shot evaluation, we provide up to 5 demonstration examples with answers before the question. The prompt concludes with the phrase答案是：“Answer:”, as shown in the example in Figure 2. If the context exceeds the model’s maximum length with few-shot examples, we dynamically remove the longest examples based on sub-token count.
Models We assessed more than 20 models of different sizes from 12 model families. For closed models, we evaluated ChatGPT and GPT4.2 For open-source models, we selected: (1) English and multilingual models: BLOOM-7.1B (Scao et al., 2022), BLOOMZ-7.1B (Muennighoff et al., 2022), LLaMA-7B/13B/30B/65B (Touvron et al., 2023a), Bactrian-X-LLaMA (BXLLaMA)-7B/13B (Li et al., 2023a), Falcon-7B/40B (Almazrouei et al., 2023), LLaMA2-7B/13B/70B (Touvron et al., 2023b), and Chinese-LLaMA (ZHLLaMA)7B/13B (Cui et al., 2023); and (2) Chinese models: Baichuan-7B/13B and Baichuan2-7B/13B (Yang et al., 2023), ChatGLM-6B and ChatGLM26B (Zeng et al., 2023), Xverse-13B,3 InternLM7B/20B (Team, 2023), MOSS-SFT-16B (OpenLMLab, 2023), Chinese-GLM-10B (Du et al., 2022), and BatGPT-15B (Li et al., 2023b). The details of these models are provided in Appendix F.

heading: 4.1 Main Results
text: Table 3 shows the performance of all models under the five-shot setting. Since the zero-shot results are similar to the five-shot results, we provide them in Appendix J.1.
Results by model From the first block of the table, we can observe the following: (1) LLaMA270B is the best open-source multilingual model, achieving an average accuracy of 53.21%, close to the ChatGPT performance of 55.51%. However, there is still a substantial gap between LLaMA270B and GPT4 (70.95%); (2) 7B pre-trained multilingual models (except LLaMA2-7B) achieve nearrandom results of 25% (since they are lower than 30%, they are not displayed in the table); and (3) For the multilingual models, having been explicitly fine-tuned in Chinese consistently improves
2The evaluation was conducted in May, 2023 for ChatGPT and July, 2023 for GPT4.
3https://github.com/xverse-ai/ XVERSE-13B
performance (BXLLaMA and ZHLLaMA vs. LLaMA, BLOOMZ vs. BLOOM).
From the second block, we see that: (1) Among the Chinese LLMs, Baichuan2-13B achieves the best overall performance (beats ChatGPT) with only 13B parameters, which we attribute to the high quality of the training data; and (2) Several Chinese LLMs achieve competitive results compared to LLaMA2-70B, with less than 20B parameters. This demonstrates that when focusing on a single language, high-quality monolingual (or bilingual) training data can enable small models (7B or 13B) with strong capabilities compared to multilingual training data. An overall observation is that models from the same family always improve with increasing model size.
Results by subject From the perspective of subject type, all models exhibit relatively high performance in the humanities, social sciences, and other subjects; medium performance on China-specific subjects; and low performance on STEM subjects. We attribute this to the nature of each subject type, and the capability of LLMs: (a) the humanities and social sciences assess recall of factoids, which is relatively easy for LLMs; (b) China-specific topics encompass information that is either absent from
the training data or inconsistent in multilingual training data; and (c) STEM topics usually require complex reasoning, which has been proven to be difficult for existing LLMs. As expected, Chinese LLMs exhibit smaller gaps between China-specific subjects and other categories.
We compare the performance of the bestperforming Chinese model, Baichuan2-13B, with the best-performing multilingual model, GPT4, for each subject. We categorize the subjects and present the results in Figure 3. The full results can be found in Appendix J.2.
From the figure, we note that model performance appears to be unbalanced, excelling in certain subjects but struggling in others. Specifically, ancient Chinese and college actuarial science are the most challenging subjects for both Baichuan2 and GPT4, yielding slightly better results than random, while the legal and moral basis is one of the easiest subjects for both models. When comparing the two models, we find that for most subjects, GPT4 outperforms Baichuan2 by a large margin, while Baichuan2 surpasses GPT4 in 8 subjects, 6 of which are China-specific and the other 2 (arts and philosophy) contain a strong Chinese element.4
4Due to these subjects contain a mixture of Chinese el-
0 25 50 75 Virology
Machine Learning High School Physics
High School Mathematics High School Chemistry
High School Biology Genetics
Elementary Mathematics Electrical Engineering
Conceptual Physics Computer Science
College Medical Statistics College Mathematics
College Engineering Hydrology College Actuarial Science
Astronomy Anatomy
World Religions World History
Professional Law Philosophy
Marxist Theory Logical
Jurisprudence International Law
Global Facts College Law
Arts Sociology
Security Study Public Relations
Professional Psychology Professional Accounting
Marketing Management
Journalism High School Geography
Economics College Education
Business Ethics Sports Science
Professional Medicine Nutrition
Legal And Moral Basis Human Sexuality
Food Science Elementary IT
Computer Security College Medicine
Clinical Knowledge Agronomy
Traditional Chinese Medicine Modern Chinese
High School Politics Ethnology
Elementary Commonsense Elementary Chinese
Education Construction Project Management
Chinese Teacher Qualification Chinese Literature
Chinese History Chinese Foreign Policy Chinese Food Culture Chinese Driving Rule
Chinese Civil Service Exam Ancient Chinese
Baichuan2-13B GPT4
China specific
Other
Social Sciences
Humanities
STEM
Figure 3: GPT4 vs. Baichuan2-13B-Chat on each subject (zero-shot). For fair comparison, we use free generation for both models.
These findings suggest that including region- and culture-specific data in training is essential to accommodate users with different language backgrounds.

heading: 4.2 Analysis
text: In order to gain a more comprehensive understanding of each LLM’s performance on CMMLU, we explored three factors that may enhance model performance and two factors that could potentially diminish its performance. Specifically, we investigated whether the following factors can improve model performance: (1) utilizing chain-of-thought
ements and global elements, we did not categorize them as China-specific.
Model STEM Overall
DA COT DA COT
ChatGPT 45.22 46.58 53.14 52.73 ChatGLM2-6B 42.42 42.56 49.61 49.34 Baichuan2-13B-Chat 45.18 42.70 58.77 52.82 BatGPT-15B-sirius 38.13 34.66 45.26 42.87 InternLM-Chat-20B 42.09 32.31 53.52 43.29 Xverse-13B-Chat 40.13 30.53 52.96 39.27
opposite (Liu et al., 2023; Zeng, 2023). In this context, we use CMMLU as a case study to investigate in-context learning (ICL) in LLM evaluation on multiple-choice questions.
As illustrated in Figure 4, we present the overall accuracy of models using varying numbers of in-context examples. There is a clear discrepancy that, when provided with only one example, base models exhibit an overall boost, whereas fine-tuned models experience a decline in performance. We conjecture this is because base models are primarily optimized for natural text and may struggle to follow instructions. Providing examples helps these models better understand the task. In contrast, SFT/RLHF models are optimized to follow instructions, and the introduction of examples introduces a certain degree of mismatch with the data
Impact of model size on performance We explored how the model’s performance improves with an increase in the number of parameters. To this end, we examine several model families and present their five-shot accuracy in relation to model size in Figure 5.
From the figure, we can see that both LLaMA and LLaMA2 achieve 5-point increases as the model size grows from 7B to 13B, while Baichuan shows a remarkable 10-point improvement despite Baichuan-13B having 0.2T more training tokens than Baichuan-7B. We believe that having 7 billion
parameters limits the model’s capability on numerous tasks, while doubling the parameters to about 13 billion substantially enhances certain capabilities and improves memorization. As the model size continues to increase (as seen with LLaMA and LLaMA2), the efficiency of performance improvement decreases, with a 5x increase in model size resulting in a 7% improvement for LLaMA and a 15% improvement for LLaMA2. Comparing LLaMA2 and Baichuan, it becomes evident that a smaller model equipped with higher-quality monolingual training data can surpass the performance of a larger model with insufficient monolingual training data in terms of monolingual performance.
Are questions with negation more challenging? Previous research has pointed out that language models may encounter challenges with negation expression (Kassner and Schütze, 2020; Hosseini et al., 2021; Truong et al., 2023). To investigate whether this issue persists in the context of Mandarin Chinese and LLMs, we firstly employ string matching to classify the test set into questions with and without negation words. We then compare the performance of different models on these two subsets, noting that questions involving negation compromise approximately 10.7% of the data.
In Table 5, we present 4 model families, from which we see that most models (with the exception of GPT4 and ChatGLM2) perform less effectively on questions containing negation compared to those without, aligned with the findings of previous studies, and highlighting this common limitation of large language models.
Interestingly, developers have successfully mitigated this problem in different stages of development. For example, LLaMA2 appears to have enhanced the abilities of the model to process negation using SFT/RLHF: the accuracy gap between questions with and without negation decreases by about 5% after applying SFT/RLHF. Baichuan shows that better pre-training can also effectively alleviate this issue. Specifically, Baichuan2 reduces the gap to 1–2% compared to Baichuan’s 8–10% using improved pre-training data. The trend is also similar with ChatGLM2. This is an encouraging finding in terms of recently-developed LLMs having improved core language processing abilities.
Are questions with sub-options more challenging? There is a typical question type in Chinese exams called sub-option questions. These ques-
tions include a main statement along with multiple sub-options, and a question about the count, order, or selection of the sub-options, requiring the model to have deeper reasoning and inference skills (see the example in Figure 6). The sub-options in CMMLU can appear in different formats, such as “a, b, c...;①,②,③...”, and account for about 10.8% of the dataset. We classified the data into two subsets based on sub-option presence, and present the results in Table 6. We observed that all models have weaker performance on sub-option questions, with the decline ranging from 10% to 20%. Intuitively, the COT prompt should alleviate such a problem by guiding the model to analyze the suboptions one by one. However, the observation is that ChatGLM2 and BatGPT benefit from COT prompt while Baichuan doesn’t.

heading: 5 Conclusion
text: We introduce CMMLU, a groundbreaking benchmark designed to assess multi-task language understanding capabilities in Chinese. Our experimental findings reveal substantial opportunities for improvement within existing large language models. Through extensive analysis, we identify several factors that impact model performance and propose actionable directions for enhancing LLMs. We are confident that our benchmark dataset and analytical insights will empower researchers to effectively evaluate and design Chinese LLMs.
Limitations
Despite CMMLU having extensive coverage of tasks, and there being extensive analyses across different LLMs and aspects of the dataset in the paper, there remain limitations. First, CMMLU is exclusively in multiple-choice format, which constrains the assessment of generative LLMs in aspects such
as linguistic fluency and reasoning. Second, inherent to all open-source benchmarks, it is impossible to completely avoid problems like LLMs having been trained on the test set, and other data contamination issues.
Ethics Statements
All data in CMMLU has been sourced from public resources that permit redistribution. Additionally, all test instances in CMMLU have undergone thorough review to ensure the exclusion of any examples that raise ethical concerns.

heading: A CMMLU Subjects
text: Table 7 lists all subjects of CMMLU. The table also provides details for each subject test, including the concepts covered, the supercategory to which each subject belongs, and the total number of questions.

heading: B CMMLU Question and Answer Lengths

heading: C CMMLU Examples
text: Table 8 provides examples from CMMLU in each category.

heading: D CMMLU Difficulty Distribution
text: We analyze the difficulty distribution of CMMLU from two perspectives. Firstly, the CMMLU benchmark encompasses a diverse range of difficulty levels: 5 subjects at primary school level, 10 at middle/high school level, 23 at college level, and 29 at professional level, ensuring a comprehensive difficulty spectrum.
Secondly, to estimate the difficulty distribution within each subject, we evaluated the top 20 models from our main results table. Each question was treated as a data point, and we recorded the number of models correctly answering each question. This approach allowed us to map out the difficulty distribution across subjects.
Figure 8 reveals that the majority of subjects exhibit a single peak in their difficulty distribution. This single-peak pattern indicates a uniform level of difficulty within these subjects, suggesting a consistent challenge for models across the range of questions. However, certain subjects, such as machine learning
(located at position [9, 1]) and professional law (at position [10, 3]), display dual peaks. This dualpeak pattern signifies a notable presence of both relatively easy and challenging questions, with fewer intermediate-level questions. Despite the presence of two peaks, the transition between these peaks is gradual rather than abrupt, indicating a smooth progression in difficulty levels within these subjects.
E Emergent Ability shown in CMMLU subjects
We assessed the concept of emergent ability using the LLaMA-2 model family. Figure 9 illustrates the performance of the LLaMA-2 pre-trained models (7B, 13B, and 70B) across various subjects. The figure indicates that, for most subjects, there is a correlation between increased model size and enhanced performance. Notably, in subjects like college education (position [2, 4]), elementary commonsense (position [3, 6]), human sexuality (position [4, 7]), and public relations (position [5, 12]), the performance of the 7B and 13B models is comparable, while the 70B model shows a significant improvement.
However, since LLaMA-2-70B model has been trained on a more extensive dataset compared to its 7B and 13B counterparts, which likely includes more comprehensive coverage in these specific domains. We cannot simply attribute it to emergent ability. In addition, these tasks are mostly belongs to social science rather than STEM (which might need intensive reasoning). Given these complexities, we leave the exploration of emergent ability in our future research endeavors.

heading: F Models Evaluated in this Paper
text: ChatGPT/GPT4 are GPT models developed by OpenAI and fine-tuned using reinforcement learning from human feedback (RLHF). As commercial products, specific details about the model size, training data, and training process remain undisclosed.
Falcon is a decoder-only model created by TII and trained on 1,000B tokens of RefinedWeb (Penedo et al., 2023) data. Due to the high quality of its training data, Falcon-40B performs competitively with LLaMA-65B on various benchmarks.
LLaMA is an auto-regressive language model proposed by Meta. It incorporates several structural improvements over the vanilla transformer and is trained on a mixture of publicly available data sources. LLaMA has demonstrated performance that is comparable to or even superior to models that are ten times its size.
LLaMA2 is an upgraded version of LLaMA developed by Meta. The preprocessing stage involves more robust data cleaning and updating data mixes, and the model employs a 40% increase in the total token count during training. Additionally, it up-samples the most factual sources to enhance knowledge and reduce hallucinations. Grouped-query attention (GQA) has been employed to reduce GPU memory usage.
BLOOM is a multi-lingual targeted LLM developed by BigScience. It is trained on 46 natural languages and 13 programming languages. The largest BLOOM model consists of 176B parameters, but deploying such a large model can be challenging. In this paper, we evaluate the performance of the 7B BLOOM model.
BLOOMZ is derived from BLOOM through fine-tuning on a cross-lingual task mixture (xP3), which is an instruction-following dataset. BLOOMZ exhibits competitive performance with models that have a larger number of parameters across various non-generation tasks.
Bactrian-X is a series of LLMs (LLaMA, BLOOM, mT5) proposed by MBZUAI. These models are fine-tuned on a multilingual instruction-following dataset that encompasses 52 languages. All the fine-tuned Bactrian-X models demonstrate performance improvements compared to their corresponding base models in multilingual generation settings.
ChatGLM and ChatGLM2 are bidirectional dense models pre-trained using the General Language Model (GLM) algorithm developed by Tsinghua University. They support bilingual (Chinese and English) language processing. ChatGLM is a version of GLM that is enhanced with supervised fine-tuning, feedback bootstrap, and reinforcement learning with human feedback, specifically optimized for Chinese question answering (QA) and dialogue tasks. In this paper, we evaluate the performance of 10B and 6B models of GLM.
BatGPT jointly developed by Wuhan University and Shanghai Jiaotong University, is a bilingual (Chinese and English) and bidirectional language model. BatGPT is initialized with a novel parameter expansion method, which enables it to absorb knowledge from the pre-training of other LLMs. With a bidirectional autoregressive architecture and further enhancement through Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human and AI Feedback (RLHAF), BatGPT is able to handle long-range, multi-turn question-answering tasks effectively and alleviate concerns regarding memory limitations. The evaluation of the 15B version is presented in this work.
MOSS-SFT is an open-source Chinese language model proposed by Fudan University. It is comparable to ChatGPT in terms of training scale and alignment techniques. MOSS-SFT is initialized with CodeGen and further pre-trained on 100B Chinese tokens and 20B English tokens. The Supervised Fine-Tuned (SFT) version of MOSS-SFT enables the model to follow instructions in multi-turn dialogues.
Chinese-LLaMA is part of the Chinese-LLaMA-Alpaca project, an open-source initiative that extends the vocabulary of LLaMA and Alpaca to include more Chinese tokens. The models are then further trained on a larger Chinese corpus to enhance their performance.
Baichuan and Baichuan2 are large language model families publicly released by Baichuan Intelligent Technology. Both include versions with 7B and 13B parameters, as well as base and chat variants. Baichuan models are trained on high-quality corpora totaling 1.4 trillion tokens, which surpasses LLaMA13B by 40%. The models offer support for both Chinese and English languages, and have an extensive context window of 4096. Baichuan2 series is trained on nearly twice the amount of high-quality data, resulting in additional performance enhancements.
Xverse is a 13B multilingual large language model developed by Shenzhen Yuanxiang Technology. It is trained on 1.4 trillion tokens from diverse sources and supports an extensive 8k context length, efficient tokenization, and advanced training technologies, making it both versatile and efficient.
InternLM is an open-source, lightweight training framework developed collaboratively by Shanghai AI Laboratory in partnership with researchers from various universities and companies. Its primary objective is to facilitate model pre-training without the need for extensive dependencies. Utilizing a unified codebase, it supports both large-scale cluster pre-training on thousands of GPUs and fine-tuning on a single GPU, achieving remarkable performance enhancements. Notably, InternLM achieves nearly 90% acceleration efficiency when training on 1024 GPUs. Based on the InternLM framework, a model family including 7B and 20B versions as well as base and chat variants was released.

heading: G Strategies for Estimating Model Choices
text: Effective evaluation of language models remains an open challenge in NLP, even for simple tasks such as multiple-choice question answering (Biderman et al., 2024). In this section, we compare three strategies for multiple-choice question evaluation. We introduce the mechanism of each strategy, explain its rationale, and compare their efficiency, strengths, and weaknesses. For convenience, we assume the question is “textQ”, and the four choices are: “textA”, “textB”, “textC”, “textD”.
Strategy 1 – Next Token Prediction The idea is to input the question along with all candidate choices and prompt the model with a direct answer text, such as “The answer is: ”. We then retrieve the probabilities of the next predicted token and compare these probabilities over the four choice indicator tokens, typically [A,B,C,D]. The token with the highest probability is treated as the model’s choice.
• Example input:
– Question: textQ A. textA B. textB C. textC D. textD Answer:
• Efficiency: High
• Pro: Most efficient method.
• Con: The model may not tend to generate a token from these choice letters.
• How to mitigate the cons: Provide few-shot examples with their expected answers.
• Benchmarks or frameworks using this strategy: MMLU (Hendrycks et al., 2021a), HELM (Liang et al., 2022).
Strategy 2 – Perplexity Comparison After combining question with all candidate choices. We concatenate each candidate answer with the full question and candidates text. These concatenated texts are then input to the model for a forward pass, and we compute the perplexity for each. The sequence with the lowest perplexity is treated as the model’s choice.
• Example input (4 inputs):
– Question: textQ A. textA B. textB C. textC D. textD Answer: A. textA
– Question: textQ A. textA B. textB C. textC D. textD Answer: B. textB
– Question: textQ A. textA B. textB C. textC D. textD Answer: C. textC
– Question: textQ A. textA B. textB C. textC D. textD Answer: D. textD
• Efficiency: Low
• Pro: Aligns with the objective of language model optimization as perplexity reflects the true probability of a model generating the given text.
• Con: Low efficiency. Usually take 4x time (for a 4-choice question) compared to Next Token Prediction.
• How to mitigate the cons: Efficient implementation that only computes the same prefix once.
• Benchmarks or frameworks use this strategy: LM-Evaluation-Harness (Gao et al., 2023), OpenCompass (Contributors, 2023).
Strategy 3 – Free Generation We input the question and candidate choices to the model and prompt it by asking for the correct choices. We allow the model to continue generating text, and then use the auxiliary method to match the patterns and extract the model’s choices.
• Example input:
– Question: textQ A:textA B:textB C:textC D:textD Answer:
• Efficiency: Medium/Low
• Pro: Allow various prompting,
• Con: Need answer extraction via human/model/regular expression. This process can be costly and error-prone. The generation can be very long, resulting in significant time consumption.
• How to mitigate the cons: Train a robust answer extraction model, or design robust regular expressions. Use a small temperature when doing generation.
• Benchmarks or frameworks use this strategy: OpenCompass, C-Eval (Huang et al., 2023).
Table 9 compares models performance using strategy 1 and strategy 3. Since strategy 2 is timeconsuming, we didn’t conduct results on it. From the table, we find that using next token prediction achieves a higher score than using the free generation strategy for all models, but the gap is less than 3% for most of the models under the zero-shot setting (with the exception of BatGPT which is about 5%). For both zero-shot and five-shot settings, the gap between strategy 1 and 2 is positively correlated to the proportion of the instances that cannot match any choice using regex. Hence, we believe using the next token prediction to force the model to make a choice among the given choices can effectively reflect its knowledge capacity.

heading: H Regular expressions matching algorithm
text: The pseudocode in Algorithm 1 outlines the ExtractChoice function for extracting choices from an LLM output string.
Initially, the function examines whether the first character of the string corresponds to a valid choice and returns that choice if true. To accommodate the complex responses of different LL.M.s, we adopt a four-step matching mechanism.
First: Identify and extract choices by seeking patterns of some choice statements, such as the term "answer" (answer) followed by valid options. Second: Employ a pattern to recursively identify and extract the choices mentioned in the string, iterating until they finally appear. Third: Use weak single matching patterns. Fourth: Check for responses that mention a single choice.
If there is no matching pattern or unique selection, "E" is returned by default, indicating that no selection was confidently extracted.
Algorithm 1 Algorithm for Extracting Choices from Response Strings 1: procedure EXTRACTCHOICE(response) 2: response← convert to string(response) 3: choices← [′A′,′ B′,′ C′,′ D′] 4: if first character of response ∈ choices then 5: return first character of response 6: end if 7: patterns1← [ 8: (r’答案(选项)?(是|为)：? ?([ABCD])’, 3), 9: (r’答案(是|为)选项 ?([ABCD])’, 2), 10: (r’故?选择?：? ?([ABCD])’, 1), 11: (r’([ABCD]) ?选?项(是|为)?正确’, 1), 12: (r’正确的?选项(是|为) ?([ABCD])’, 2), 13: (r’答案(应该)?(是|为)([ABCD])’, 3), 14: (r’选项 ?([ABCD]) ?(是|为)?正确’, 1), 15: (r’选择答案 ?([ABCD])’, 1), 16: (r’答案?：?([ABCD])’, 1), 17: (r’([ABCD])(选?项)?是?符合题意’, 1), 18: (r’答案选项：? ?([ABCD])’, 1), 19: (r’答案(选项)?应?该?为(.*?)([ABCD])’, 3), 20: (r’textbf{\(([ABCD])’, 1) 21: ] 22: patterns2← [ 23: (r’([ABCD])(.*?)当选’, 1), 24: (r’([ABCD])(.*?)正确’, 1) 25: ] 26: 27: patterns3← [ 28: (r’[^不]是：? ?([ABCD])’, 1), 29: (r’^选项([ABCD])’, 1) 30: ] 31: 32: for each patterns in [patterns1, patterns2, patterns3] do 33: for each (pattern, idx) in patterns do 34: if pattern is found in response then 35: answer ← matched group(idx) 36: if answer ∈ choices then 37: return answer 38: end if 39: end if 40: end for 41: end for 42: pattern4← r’^[^ABCD]*([ABCD])[^ABCD]*$’ 43: if pattern4 is matched in response then 44: answer ← matched group(1) 45: if answer ∈ choices then 46: return answer 47: end if 48: end if 49: return “E” ▷ Return E as default if no match is found 50: end procedure

heading: I Correlation with other Benchmarks
text: To investigate the correlation between model performance on CMMLU and other benchmarks, we choose 6 popular English LLMs and 5 benchmarks to conduct correlation analysis.
From Figure 10 we find that CMMLU demonstrates a strong correlation with four of these benchmarks, which span areas such as mathematics, commonsense reasoning, and coding. The exception is the PIQA task, where the relevance is somewhat diminished due to most models achieving high scores (>80%) on this task. However, we still obtain a high correlation of r = 0.88.

heading: J Breakdown of Model Performance
text: J.1 Results of Zero-shot Table 11 presents zero-shot results of the LLMs on CMMLU across the 5 sub-categories.
J.2 The results for each subject We compare the 0-shot and 5-shot results of selected LLMs that achieved higher performance on each subject in Table 10. We further analyze the performance distribution of multiple LLMs across all subjects in Figure 11. It is evident from the figure that LLMs with higher performance exhibit diverse abilities across various tasks, while those with lower performance face challenges in most subjects. Furthermore, the scatter plot distribution indicates comparable performance levels among LLMs across different subjects.
J.3 The effect of chain-of-thought prompting Table 12 shows the breakdown of the models performance using chain-of-thought prompting.

heading: Future Work
text: Given these complexities, we leave the exploration of emergent ability in our future research endeavors