SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C-EVAL: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Yuzhen Huang\*1 Yuzhuo Bai\*2 Zhihao Zhu $^{1}$  Junlei Zhang $^{1}$  Jinghan Zhang $^{1}$  Tangjun Su $^{1}$  Junteng Liu $^{1}$  Chuancheng Lv $^{2}$  Yikai Zhang $^{1}$  Jiayi Lei $^{1}$  Yao Fu $^{3}$  Maosong Sun $^{2}$  Junxian He\*4
$^{1}$ Shanghai Jiao Tong University  $^{2}$ Tsinghua University  $^{3}$ University of Edinburgh
<sup>4</sup>The Hong Kong University of Science and Technology
ceval.benchmark@gmail.com
https://cevalbenchmark.com
New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-EVAL, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-EVAL comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-EVAL is accompanied by C-EVAL HARD, a subset of very challenging subjects in C-EVAL that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-EVAL, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over  $60\%$ , suggesting that there is still significant room for improvement for current LLMs. We anticipate C-EVAL will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users. $^{1}$
Evaluation benchmarks are at the core role for AI development. While traditional NLP benchmarks were mostly designed to measure specific and relatively simple abilities, large language models (LLMs), or foundation models, have demonstrated various new capabilities and shifted the evaluation focus to more general and intricate skills, such as broad world knowledge and complex reasoning. To align with the new era of LLMs, new benchmarks are proposed recently to probe a diverse set of LLM abilities. For example, MMLU (Hendrycks et al., 2021a), BIG-bench (Srivastava et al., 2022), and HELM (Liang et al., 2022) benchmarks attempt to aggregate a wide range of NLP tasks for holistic evaluation. Some other benchmarks specifically focus on advanced LLM abilities that emerge with scale, such as reasoning (Cobbe et al., 2021), hard math problem-solving (Hendrycks et al., 2021b), and coding (Chen et al., 2021). While traditional NLP benchmarks are becoming obsolete, these new ones are extensively used in recent research to drive development of the latest LLMs (Taylor et al., 2022; Chowdhery et al., 2022; Hoffmann et al., 2022; Touvron et al., 2023; OpenAI, 2023).
However, these modern benchmarks primarily target English language, resulting in limited understanding of LLMs' capabilities in other languages. In this work, we focus on evaluating the advanced abilities of foundation models in a Chinese context, one of the most widely spoken language in the
*Equal Contribution. Full list of individual contributions is detailed in Appendix A.
$\dagger$  Correspondence to Junxian He <junxianh@cse.ust.hk>. Work done while affiliated with SJTU.
<sup>1</sup>The C-EVAL data and evaluation code are available at https://github.com/hkust-nlp/ceval.
Figure 1: Overview diagram of C-EVAL. Different colors of the subjects indicate four difficulty levels: middle school, high school, college, and professional.
world. Although there has been a recent surge in powerful Chinese LLMs, such as GLM-130B (Zeng et al., 2023), Wenxin Yiyan (Baidu, 2023), and MOSS (OpenLMLab, 2023), the corresponding evaluation significantly lags behind, with the CLUE benchmark (Xu et al., 2020), the Chinese counterpart of GLUE (Wang et al., 2019), serving as the best available standard. We emphasize that simply translating English benchmarks as in OpenAI (2023), even with flawless translations, does not fulfill the goal – LLMs intended for use in a Chinese environment should be evaluated on their knowledge of Chinese users' primary interests, such as Chinese culture, history, and laws, as well as other competencies unique in Chinese society. In contrast, English benchmarks tend to exhibit geographical biases towards the domestic knowledge of the regions that produce them.
To narrow the gap between Chinese LLM development and evaluation, we present C-EVAL, the first comprehensive Chinese evaluation suite to thoroughly assess LLMs' advanced knowledge and reasoning abilities in a Chinese context. C-EVAL consists of 13948 multiple-choice exam questions spanning 52 diverse disciplines, ranging from humanities to science and engineering, as depicted in Figure 1. The questions are collected from four difficulty levels: middle school, high school, college, and professional tests. Along with C-EVAL, we introduce C-EVAL HARD as an accompanied benchmark, a subset of particularly challenging subjects in C-EVAL that demands highly advanced reasoning abilities to solve, such as advanced mathematics and college physics. Notably, C-EVAL HARD is among the few benchmarks for advanced reasoning where GPT-4 still struggles, achieving an accuracy of  $53.3\%$ , making it the first Chinese benchmark at this level.
We conduct experiments to evaluate the most advanced LLMs on C-EVAL in both answer-only and chain-of-thought settings. Results show that GPT-4 is the only model that surpasses  $60\%$  average accuracy. However, its  $66.4\%$  accuracy indicates that there is still large room for improvement in current LLMs. Despite not specially tailored for Chinese data, GPT-4, ChatGPT, and Claude emerge as the top three performers on C-EVAL. Upon examining the results of LLMs focused on Chinese, we find that while some models managed to narrow the gap on Chinese knowledge test with ChatGPT, acquiring reasoning abilities seems more challenging. On C-EVAL HARD, in particular, most models could only retain near-random accuracy. In addition to its use as a whole, we envision C-EVAL as a suite of benchmarks, subsets of which could be separately utilized to assess certain model abilities of interest and analyze important strengths and limitations of foundation models. We hope C-EVAL could guide the developers to understand the abilities of their models from multiple dimensions and facilitate the growth of foundation models for Chinese users.
2 The C-EVAL Evaluation Suite
2.1 Design Principle
Overview: The motivation of C-EVAL is to help developers quickly understand the abilities of their models from multiple dimensions, so that they could target the shortcomings of the models and
Category	# Subjects	# Questions
In terms of topic
STEM	20	4495
Humanities	11	2676
Social Science	10	2845
Other	11	3932
In terms of difficulty level
Middle School	7	1409
High School	8	1594
College	25	6249
Professional	12	4696
In terms of split
Dev	52	260
Valid	52	1346
Test	52	12342
Total	52	13948
Table 1: Statistics of C-EVAL.
洛伦兹曲线越是向横轴凸出_ The more the Lorenz curve is convex to the horizontal axis, _.
A. 基尼系数就越大，收入就越不平等 the larger the Gini coefficient, the more unequal the income.
B. 基尼系数就越大，收入就越平等 the larger the Gini coefficient, the more equal the income.
C. 基尼系数就越小，收入就越不平等 the smaller the Gini coefficient, the more unequal the income.
D. 基尼系数就越小，收入就越平等 the smaller the Gini coefficient, the more equal the income.
答案：A Answer: A
Figure 2: Example from college economics. English translations are shown for better readability.
improve them accordingly. To this end, we focus on the relatively advanced abilities of LLMs such as world knowledge and reasoning, which are arguably the most critical skills for LLMs nowadays. While different LLMs may perform similarly in simple scenarios like casual conversations, complex tasks are often the key differentiators between LLMs (OpenAI, 2023). Therefore, we construct C-EVAL from real-world, challenging human exams in China that are used to assess humans' abilities from multiple dimensions. We only select questions of a multi-choice format, similar to Hendrycks et al. (2021a), because: (1) metrics are clearly defined (i.e. accuracy), and (2) multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of foundation models, which we consider could be easily exploited and reflected in various downstream applications through specialized instruction tuning (Chung et al., 2022; Wang et al., 2022). Each question has four choices and only one choice is the correct answer. LLMs are intended to be used to solve these questions through prompting. The questions in C-EVAL span 52 diverse disciplines that we later cluster them into broader categories as STEM, humanities, social science, and other areas. Summarized statistics of C-EVAL is shown in Table 1, and more detailed statistics per subject are in Appendix B.
Attempt to mitigate data contamination: Exam questions from national tests, such as China's national college entrance exams (commonly known as Gaokao) and national professional exams, are often widely distributed and accessible online. Consequently, these questions may inadvertently be crawled and incorporated into LLM pretraining, leading to potential data leakage issues. To mitigate this risk, we collect our data either from mock exams or from small-scale local exams, such as those available online from specific high schools. This deviates from previous work that built benchmarks using the exact questions from official national exams (Zhong et al., 2023). Moreover, most samples in C-EVAL are sourced from PDF or Microsoft Word documents on the Internet, rather than directly from plain text or structured questions. These documents are subsequently parsed and carefully annotated by the authors to obtain the final structured format, a process that often involves complex  $\mathrm{LATEX}$  equation conversion for certain subjects. This further minimizes the risk of data contamination.
2.2 Data Collection
Subject selection: C-EVAL covers four difficulty levels: middle school, high school, college, and professional. We include the standard subjects for middle and high school levels in China, except for the English subject. For the college level, we select 25 representative subjects from all the 13 official categories of undergraduate majors listed by the Ministry of Education in China, at least one subject from each category is included in C-EVAL to assure comprehensiveness. For the professional level, we refer to the official national vocational qualification directory in China and choose 12
2We also exclude the middle school Chinese subject since the corresponding exam mostly focuses on writing responses to questions with few multi-choice questions.
<sup>3</sup>The Undergraduate Major Catalogue of Higher Institutions in China.
$^{4}$ National Vocational Qualifications Directory in China.
25 °C时, 将pH=2的强酸溶液与pH=13的强碱溶液混合, 所得混合液的pH=11, 则强酸溶液与强碱溶液的体积比是(忽略混合后溶液的体积变化)
At  $25^{\circ}\mathrm{C}$ , when a strong acid solution with  $\mathrm{pH} = 2$  is mixed with a strong alkali solution with  $\mathrm{pH} = 13$ , the resulting mixture has a  $\mathrm{pH}$  of 11. The volume ratio of the strong acid solution to the strong alkali solution is (ignoring any volume changes upon mixing)
A. 11:1 B. 9:1 C. 1:11 D. 1:9
答案：B
Answer: B
答案解析：
1. pH=13的强碱溶液中c(OH-) = 0.1mol/L，pH=2的强酸溶液中c(H+) = 0.01mol/L，酸碱混合后pH=11，即 c(OH-) = 0.001mol/L。
2. 设强酸和强碱溶液的体积分别为x和y，则  $\mathrm{c(OH^{-}) = (0.1y - 0.01x) / (x + y) = 0.001}$ ，解得  $x:y = 9:1$ 。
1. In the strong alkali solution with  $\mathrm{pH} = 13$ ,  $\mathrm{c}(OH^{-}) = 0.1\mathrm{mol / L}$ , and in the strong acid solution with  $\mathrm{pH} = 2$ ,  $\mathrm{c}(H^{+}) = 0.01\mathrm{mol / L}$ . After mixing, the  $\mathrm{pH}$  is 11, which means that  $\mathrm{c}(OH^{-}) = 0.001\mathrm{mol / L}$ .
2. Assuming the volumes of the strong acid and strong alkali solutions are  $x$  and  $y$ , respectively, then:  $c(OH^{-}) = (0.1y - 0.01x) / (x + y) = 0.001$ . Solving for  $x:y = 9:1$ .
Figure 3: An development example with explanations from high school chemistry. English translations are shown below the corresponding Chinese text for better readability.
representative ones, such as physician, legal professional, and civil servant qualification exams. We also cluster these subjects into four categories in terms of their topic: STEM (Science, Technology, Engineering, and Mathematics), Social Science, Humanities, and Other areas. All the 52 subjects and their assigned categories are illustrated in Figure 1.
Data sources: The primary source of our data is mock exams freely available on the internet. In addition, a portion of the college-level questions are past exam questions from top universities in China, publicly shared by the students. A minor fraction of college questions are mock questions for the national graduate entrance exam, sourced from the Weipu website $^{5}$  – these questions are not freely available to the public, and we have obtained their authorization to include around 2000 such questions into C-EVAL.
Data Processing: The collected data come in various formats, primarily as PDF or Microsoft Word documents and a minor fraction as web pages. PDF documents are initially processed into text. All questions are subsequently parsed – automatically when possible, and otherwise manually by the authors – into a structured format, as exemplified in Figure 2. For subjects with complex mathematical notations such as many subjects in the STEM category, we manually convert them into standard Latex formats, similar to Hendrycks et al. (2021b); Taylor et al. (2022). All the questions in C-EVAL are processed to include exactly four choices. Most of the original questions were accompanied by four choices already, and we eliminate questions with fewer than four options and randomly drop incorrect choices for questions with more than four options. All questions also go through the standard data preprocessing pipeline, such as dedduplication and cleaning. Following this, the questions undergo several rounds of human validation by the authors, and all the Latex notations are ensured to be complied without syntax errors. We process at least around 200 questions for each subject, and randomly split the questions into a development set, a validation set, and a test set within each subject. The development split per subject consists of five exemplars to facilitate few-shot evaluation. These dev exemplars are also annotated with explanations to enable few-shot chain-of-thought settings (Wei et al., 2022), as we detail next. The validation and test set are created with a 1:9 ratio, where the validation set is intended to be used for hyperparameter tuning.
Explanation data generation: Chain-of-thought (COT) reasoning (Kojima et al., 2022; Wei et al., 2022) – that prompts LLMs to generate a text sequence of reasoning process along with the final answer – has shown great success on reasoning-heavy tasks. Compared to zero-shot COT, the few-shot version is more commonly used and achieves the state-of-the-art performance on various tasks (Gao et al., 2022; Wang et al., 2023; Zhou et al., 2023; Xie et al., 2023). To facilitate the potential usage of C-EVAL in a few-shot COT setting, we combine automatic generation and human annotation to produce high-quality explanation data for the development split. Specifically, we first prompt GPT-4 to generate step-by-step explanation to explain the ground-truth answer, then we
$^{5}$ https://kaoyan.cqvip.com/view/postgraduate/index.aspx
设有界区域  $\Omega$  由曲面  $2x + y + 2z = 2$  与三个坐标平面围成， $\Sigma$  为整个表面的外侧；计算曲面积分：  
 $I = \iint_{\Sigma} (x^2 + 1) \, d\mathcal{Y} \, d\mathcal{Z} - 2y \, d\mathcal{Z} \, d\mathcal{X} + 3z \, d\mathcal{X} \, d\mathcal{Y} = \_$   
Let  $\Omega$  be a bounded region enclosed by the surface  $2x + y + 2z = 2$  and three coordinate planes, and let  $\Sigma$  be the surface of the region  $\Omega$ . Calculate the surface integral:  $I = \iint_{\Sigma} (x^2 + 1) \, d\mathcal{Y} \, d\mathcal{Z} - 2y \, d\mathcal{Z} \, d\mathcal{X} + 3z \, d\mathcal{X} \, d\mathcal{Y} = \_$   
A.  $1/2$  B.  $1$  C.  $3/2$  D.  $5/2$   
答案：A  
Answer: A
Figure 4: Example from advanced mathematics, a subject in C-EVAL HARD. English translations are shown below the corresponding Chinese text for better readability.
manually revise the generated explanations to obtain the final explanations. Details on prompting GPT-4 are in Appendix C. A dev example with explanations is illustrated in Figure 3.
2.3 C-EVAL HARD
We select 8 challenging math, physics, and chemistry subjects from C-EVAL to form a separate benchmark, C-EVAL HARD, which includes advanced mathematics, discrete mathematics, probability and statistics, college chemistry, college physics, high school mathematics, high school chemistry, and high school physics. These subjects often involve with complex IATEX equations and require non-trivial reasoning abilities to solve. An example from advanced mathematics is shown in Figure 4. C-EVAL HARD aligns with recent efforts to create difficult benchmarks to assess advanced reasoning abilities (Hendrycks et al., 2021b; Suzgun et al., 2022), which are the key differentiators among various LLMs and could reflect LLMs' potential in general and complex scenarios. We emphasize that C-EVAL HARD is the first Chinese benchmark to provide highly complicated reasoning questions.
2.4 Evaluation
We use accuracy as the metric. While ground-truth labels of the development and validation splits are released, we keep the labels of the test split private. This is to ensure the fair use of the C-EVAL, as the C-EVAL data may unconsciously be included in pretraining data due to web crawling. Instead, users are required to submit their model predictions to https://cevalbenchmark.com to automatically obtain the test accuracy, where a public leaderboard is maintained. Users have the option to include their submission results in the live leaderboard, depending on their own preference.
We evaluate LLMs in both zero- and five-shot settings on C-EVAL, where the five exemplars are from the development split. We adopt regular expressions to extract answer choices from the model responses, ensuring that we can successfully extract answers for nearly all cases. We report answer-only (AO) results on both zero- and five-shot settings and chain-of-thought (COT) results on the five-shot setting only, as we found that it was often difficult to extract the answer choices from zero-shot COT predictions where the generation does not follow specific patterns. Prompts of AO and COT are shown in Appendix D. We note that in the COT setting, the five-shot exemplars could exceed the maximum context length of some LLMs for certain subjects. In such cases, we dynamically reduce the number of exemplars to fit within the context window.
To give a comprehensive view of the status of LLM in a Chinese language context, we evaluate 11 accessible top-performing LLMs that are able to process Chinese input, covering diverse organizations and varying in size, as shown in Table 2. ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023) are the strongest GPT model variants from OpenAI. Claude (Anthropic, 2022), developed by Anthropic, is often considered comparable to ChatGPT. We evaluate both the Claude-v1.3 and Claude-instant-v1.0 variants, with Claude-instant being a lighter version. Bloomz-mt (Muennighoff et al., 2022) is based on the pretrained multilingual BLOOM model (Scao et al., 2022) with multitask prompted finetuning, thus is suitable for non-English languages. LLaMA (Touvron et al., 2023) is probably
Model	Creator	#Parameters	Access
GPT-4	OpenAI	undisclosed	API
ChatGPT	OpenAI	undisclosed	API
Claude-v1.3	Anthropic	undisclosed	API
Claude-instant-v1.0	Anthropic	undisclosed	API
Bloomz-mt	BigScience	176B	Weights
LLaMA-65B	Meta	65B	Weights
GLM-130B	Tsinghua	130B	Weights
ChatGLM-6B	Tsinghua	6B	Weights
Chinese-LLaMA-13B	Cui et al.	13B	Weights
Chinese-Alpaca-13B	Cui et al.	13B	Weights
MOSS	Fudan	16B	Weights
Table 2: Models evaluated in this paper.
Model	STEM	Social Science	Humanities	Other	Average
Random	25.0	25.0	25.0	25.0	25.0
GPT-4	65.2	74.7	62.5	64.7	66.4
ChatGPT	49.0	58.0	48.8	50.4	51.0
Claude-v1.3	48.5	58.6	47.3	50.1	50.5
Bloomz-mt	39.1	53.0	47.7	42.7	44.3
GLM-130B	36.7	55.8	47.7	43.0	44.0
Claude-instant-v1.0	38.6	47.6	39.5	39.0	40.6
ChatGLM-6B	33.3	48.3	41.3	38.0	38.9
LLaMA-65B	32.6	41.2	34.1	33.0	34.7
MOSS	31.6	37.0	33.4	32.1	33.1
Chinese-Alpaca-13B	27.4	39.2	32.5	28.0	30.9
Chinese-LLaMA-13B	28.8	32.9	29.7	28.0	29.6
Table 3: Zero-shot average accuracy  $(\%)$  in answer-only setting. We report the average accuracy over the subjects within each category. "Average" column indicates the average accuracy over all the subjects.
the best open-weight foundation model so far that achieves the highest accuracy on the English MMLU benchmark within open-weight models. The aforementioned models except Bloomz-mt are English-oriented during development, while they are able to process Chinese input because a minor fraction of Chinese text is present in the pretraining data.
We further include recent LLMs developed by Chinese institutions or individuals that is Chinese-oriented. GLM-130B (Zeng et al., 2023) and ChatGLM-6B (THUDM, 2023a) are based on the General Language Model architecture (GLM, Du et al. (2022)) trained on English and Chinese data. ChatGLM-6B is further adapted on conversational data. Chinese-LLaMA (Cui et al., 2023) is an adaptation of LLaMA, which is further pretrained on Chinese data. Chinese-Alpaca (Cui et al., 2023) performs instruction tuning based on Chinese-LLaMA. MOSS (OpenLMLab, 2023) is the first publicly available Chinese LLM, and it follows a training procedure similar to ChatGPT. We note that there are some other commercial Chinese-oriented LLMs whose weights and APIs are not directly open to the public at the time of writing this paper, such as Wenxin Yiyan (Baidu, 2023), Tongyi Qianwen (Alibaba, 2023), and Xunfei Xinghuo (iFLYTEK, 2023), these models may have strong performance, yet we are not authorized to evaluate and publicize their results. Therefore, we only report results from models with open APIs or weights in this work, while we anticipate the developers of other models to submit and optionally publicize their models' results in our website. A detailed description of the evaluated models can be found in Appendix E.
General comparison: Zero- and five-shot answer-only results are shown in Table 3 and Table 4 respectively. We report the average accuracy, while a detailed breakdown of accuracy per subject is provided in Appendix F. GPT-4 is the only model that exceeds  $60\%$  average accuracy, highlighting the challenge presented by C-EVAL. GPT-4 significantly outperforms all other models, with the second-best model, ChatGPT, trailing over 14 percentage points behind in both zero- and five-shot settings. Claude-v1.3 achieves similar performance to ChatGPT, in terms of both the category-wise
Model	STEM	Social Science	Humanities	Other	Average
Random	25.0	25.0	25.0	25.0	25.0
GPT-4	67.1	77.6	64.5	67.8	68.7
ChatGPT	52.9	61.8	50.9	53.6	54.4
Claude-v1.3	51.9	61.7	52.1	53.7	54.2
Claude-instant-v1.0	43.1	53.8	44.2	45.4	45.9
GLM-130B	34.8	48.7	43.3	39.8	40.3
Bloomz-mt	35.3	45.1	40.5	38.5	39.0
LLaMA-65B	37.8	45.6	36.1	37.1	38.8
ChatGLM-6B	30.4	39.6	37.4	34.5	34.5
Chinese-LLaMA-13B	31.6	37.2	33.6	32.8	33.3
MOSS	28.6	36.8	31.0	30.3	31.1
Chinese-Alpaca-13B	26.0	27.2	27.8	26.4	26.7
Table 4: Five-shot average accuracy (%) in answer-only setting. We report the average accuracy over the subjects within each category. "Average" column indicates the average accuracy over all the subjects.
Model	STEM	Social Science	Humanities	Other	Average
Random	25.0	25.0	25.0	25.0	25.0
GPT-4	67.3	76.5	64.4	66.6	68.3
Claude-v1.3	51.9	63.2	50.9	53.6	54.2
ChatGPT	47.8	58.3	47.7	48.5	50.0
Claude-instant-v1.0	43.3	52.7	41.3	42.4	44.5
ChatGLM-6B	29.9	40.0	37.9	34.5	34.5
MOSS	27.3	38.1	33.6	29.4	31.2
LLaMA-65B	28.0	36.3	29.3	30.0	30.3
GLM-130B	24.8	33.1	30.8	30.0	28.8
Chinese-LLaMA-13B	20.5	30.5	28.2	27.1	25.4