C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Authors: Yuzhen Huang
Yuzhuo Bai
Zhihao Zhu
Junlei Zhang
Jinghan Zhang
Tangjun Su
Junteng Liu
Chuancheng Lv
Yikai Zhang
jiayi lei
Yao Fu
Maosong Sun
Junxian He
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Chinese evaluation
foundation models
TL;DR: We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context.

Abstract: New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

C-EVAL: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Yuzhen Huang")
Yuzhuo Bai*2
Zhihao Zhu'
Juuleei Zhang'
Jinghan Zhang' Jiayi Lei*
Tangjun Su
Junteng Liu
Chuancheng Lv²
Yikai Zhang'
Yao Fu
Maosong Sun?
Junxian He
Shanggaai Jiao Tong University *Tsinghua University
*University Edinburgh
"The Hong Kong University of Science and Technology
ceval. www.bbnnmmkkkkgmii..... https://eealleenhhnarr.comm
Abstract
New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-EVAL the first comprehensive Chi- nese evaluation suite designed to assess advanced knowledge and reasoning abilities foundation models in a Chinese context. C-EVAL comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-EVAL is accompanied by C-EvAL HARD, subset very challenging subjects in C-EVAL that requires advanced reasoming abilitics to solve. We conduct comprchensive evaluation of the most advanced LLMs on C-EVAL, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there still significant room for improvement for current LLMs. We anticipate C-EvAL will help analyze important strengths and shorteomings of foundation models, and foster their development and growth for Chinese users]
1 Introduction
Evaluation benchmarks are at the core role for Al development. While traditional NLP benchmarks were mostly designed to measure specific and relatively simple abilities, large language models (LLMs), or foundation models, have demonstrated various new capabilities and shifted the evaluation focus more general and intricate skills, such as broad world knowledge and complex reasoning. To align with the new era of L.LMs, new benchmarks are proposed recently to probe a diverse set of LLM abilities. For example, MML.U (Hendryeks al., 2021a), BIG-bench (Srivastava al. 2022). and HELM (Liang et al., (202)) benchmarks attempt to aggregate wide range of NI.P tasks for holistic evaluation. Some other benchmarks specifically focus on advanced L.M abilities that emerge with scale, such reasoning (Cobbe al., 2222), hard math problem-solving (Hendrycks et al.) (021b), and coding (Chen et al.
2021). While traditional NLP benchmarks are becoming obsolete, these new ones are extensively used in recent research to drive development of the latest LLMs (Taylor et al.
2022 Chowdhery et al. 2022; Hoffmann et al. 2022 Touvron et al. 2023 OpenAl 2023).
However, these modern benchmarks primarily target English language, resulting in limited under- standing of LLMs' capabilities in other languages. this work, we focus on evaluating the advanced abilities of foundation models in a Chinese context, one of the most widely spoken language in the
"qqual Contribution. Full list individual contributions detailed Appendix Correspondence Junxian He <junxianh @esc..st.hkk, Work done while affiliated with SJTU. 'The C-EVAL data and evaluation code are available at https///gthub..oo//kusstt
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

LEONLLARGAii Srg INBU B MrMerdlggg Irgins nea 20 pagaanin
ITEARW/CoOOa betwoit
i6n @a/ppraiing Sprten
22.9 ARLLI
anwor-uicctt Hattenaica
Other
H6- Accoustons
SRRTAACerr Nathenatia
STEM C-EVAL Humaniky
XRIcR/Collee Phouisty
2/E/ Servans
XYPRch/Colage Phosica
CXRT/CInca Medidiee
Seeteiigg Sctcol Chemittt
deimmingg Sctcel Physia
144 School Dology
BRAPPPa Protection
5p Science
64 Modiciee
Nitee Scncel Cueriety
Cueetonteee Scrcel Phyica
s=ie/Wolle Swregy
RRBg/resche Oselitcatiee
Soclal Sclanee
(Chers Tenguge and Heeuter
Eniling Terught
EORE Stieee RIT migh AUT Riliii takvol Peiites
Eowllee Ectnomic
Edoologica Moral Culivattinn
Crree modem Chisste History
swEiiiigh schsel Hatery
ES triig//wete School neeyrashy DoTii Sehrol Paiiies
swils/iigh School Chinees
R=ni//oddi School lilstary
Figure I: Overview diagram C-EVAL. Different colors of the subjects indicate four difficulty levels; middle
school, high school, college, and professional.
world. Although there has been a recent surge in powerful Chinese LLMs, such as GLM-130B (Zeng
eet al., 2023), Wenxin Yiyan (Baidu 2023), , and MOSS (OpenLMLab) 2023), the corresponding evaluation significantly lags behind, with the CLUE benchmark (Xu et al. (020), the Chinese coun- terpart of GLUE (Wang al., 2019), serving as the best available standard. We emphasize that simply translating English benchmarks as in OpeenA (202)), even with flawless translations, does not fulfill the goal goal-I.IM L.LMs intended for use in a Chinese environment should be evaluated on their knowledge of Chinese users' primary interests, such as Chinese culture, history, and laws, as well as other competencies unique Chinese society. In contrast, English benchmarks tend to exhibit geographical biases towards the domestic knowledge of the regions that produce them.
To narrow the gap between Chinese LLM development and evaluation, we present C-EVAL, the first comprehensive Chinese evaluation suite to thoroughly assess LLMs' advanced knowledge and reasoning abilities in Chinese context. C-EvAL consists of 13948 multiple-choiee exam questions spanning 52 diverse disciplines, ranging from humanities to science and engineering, as depicted in Figure [ The questions collected from four difficulty levels: middle school, high school, college, and professional tests. Along with C-EVAL, we introduce C-EVAL. HARD as an accompanied benchmark, a subset of particularly challenging subjects in C-EvAL that demands highly advanced reasoning abilities to solve, such as advanced mathematics and college physics. Notably, C-EVAL HARD among the few benchmarks for advanced reasoning where GPT-4 still struggles, achieving an accuracy of 53.3%, making it the first Chinese benchmark at this level.
We conduct experiments to evaluate the most advanced LLMs on C-EVAL in both answer-only and chain-of thothht settings. Results show that GPT-4 is the only model that surpasses 60% average accuracy. However, its 66.4% accuracy indicates that there is still large room for improvement in current I.MMs. Despite not specially tailored for Chinese data, GPT-4, CharGPTT, and Claude emerge as the top three performers C-EVAL.. Upon examining the results of LLMs focused on Chinese, we find that while some models managed to narrow the gap on Chinese knowledge test with ChatGPT, acquiring reasoning abilities seems more challenging. On C-EVAL HARD, particular, most models could only retain near-random accuracy. In addition to its use as a whole, we envision C-EVAL as a suite of benchmarks, subsets of which could be separately utilized to assess certain model abilities of interest and analyze important strengths and limitations of foundation models. We hope C-EVAL could guide the developers to understand the abilities of their models from multiple dimensions and facilitate the growth of foundation models for Chinese users.
2
The C-EVAL Evaluation Suite
2.1 Design Principle
Overview: The motivation of C-EVAL is to help developers quickly understand the abilities of their models from multiple dimensions, so that they could target the shortcomings of the models and

---

Category
Subjects
Questions
3 The more the Lurenz curve is convex to the horizontal axis,
In terms of topic
STEM
20
4495
Humanitics
11
2676
Social Science
10
2845
A. * tt #
Other
11
3932
the larger the Gini coefficient, the more unequal the income.
In terms of difficulty level
B. 3 X
Middle School
7
1409
the larger the Gini coefficient, the more equal the income.
High School
8
1594
C. t the smaller the Gini coefficient, the mure unequal the income.
A A
College
25
6249
Professional
12
4696
D. A #0
terms of split
the smaller the Gini coefficient, the more equal the income.
Dev
52
260
2 : A Answer: A
Valid
52
1346
Test
52
12342
Total
52
13948
Figure 2: Example from college economics. English trans- lations are shown for better readability.
Table I: Statistics of C-EVAL.
improve them accordingly. To this end, we focus on the relatively advanced abilities of LLMs such as world knowledge and reasoning, which are arguably the most critical skills for LLMs nowadays. While different LLMs may perform similarly in simple scenarios like casual conversations, complex tasks are often the key differentiators between L.LMs (OpenAl 2023). Therefore, we construct C-EVAL from real-world, challenging human exams in China that are used to assess humans' abilities from multiple dimensions. We only select questions of multi-choice format, similar to (Hendrycks) al. 202 because: (I) metrics clearly defined (i.e. accuracy), and (2) multi-choice questions are a simple but good proxy to evaluate the potential of advanced abilities of foundation models, which we consider could be easily exploited and reflected in various downstream applications through specialized instruction tuning (Chung et al. 2022 Wang et al. 2022). Each question has four choices and only one choice is the correct answer. LLMs are intended to be used solve these questions through prompting. The questions in C-EVAL span 52 diverse disciplines that we later cluster them into broader categories as STEM, humanities, social science, and other areas. Summarized statistics of C-EVAL is shown in Table [ and more detailed statistics per subject are in Appendix
Attempt to mitigate data contamination: Exam questions from national tests, such as China's s national college entrance exams (commonly known as Gaokao) and national professional exams, are often widely distributed and accessible online. Consequently, these questions may inadvertently be crawled and incorporated into LLM pretraining, leading to potential data leakage issues. To mitigate this risk, we collect our data either from mock exams from small-scale local exams, such those available online from specific high schools. This deviates from previous work that built benchmarks using the exact questions from official national exams (Zhong al.. 2023). Moreover, most samples in C-EVAL are sourced from PDF or Microsoft Word documents on the Internet, rather than directly from plain text structured questions. These documents annotated by the authors to obtain the final structured format, a process that often involves complex ETEX equation conversion for certain subjects. This further minimizes the risk of data contamination.
subsequently parsed and carefully
2.2 Data Collection
Subject selection: C-EvAL covers four difficulty levels: middle school, high school, college, and professional. We include the standard subjects for middle and high school levels in China, except for the English subject[] For the college level, we select 25 representative subjects from all the 13 official categories of undergraduate majors listed by the Ministry of Education in China [ at least one subject from each category included in C-EVAL to assure comprehensiveness. For the professional level, we refer to the official national vocational qualification directory in China and choose 12
We also exclude the middle school Chinese subject since the corresponding exam mostly focuses on writing responses to questions with few multi-choice questions.
The Undergraduate Major Catalogue of Higher Institutions in China
National Vocational Qualifications Directory in China.
3

---

25 CC 2
25 'C, when strong acid solution with pH-2 2 is mixed with strong alkali solution with pH-13, the resulting mixture has pH of 11. The volume ratio of the strong acid solution the strong alkali solution is (ignoring any volume changes upon mixing)
A. 11:1
9.91
C. 1:11
D. 1:9
: B
Answer:
*
13 #c(OH )-0.mo0/.,
pH #o FF pt
-0000m0......
2. 02 FR Alc((H))=(001)) 0,010/+-))000001 #xx-=11
Explanation:
In the strong alkali solution with pH-13, c(ONT)- 0.1 molll. and in the strong acid solution with plt-2, c(I1*)
0.01mol/L. After mixing. the pH is 11, which means that c(OH - 0.001 mol/L.
2. Assuming the volumes of the strong acid and strong alkali solutions are x and y. respoctively, then: c(OH)=
(0.ly (x01x)(x+y)) - 0.001. Solving for x:y -911.
Figure 3: An development example with explanations from high school chemistry. English translations are shown below the corresponding Chinese text for better readability.
representative ones, such as physician, legal professional, and civil servant qualification exams. We also cluster these subjects into four categories in terms of their topic: STEM (Science, Technology, Engineering, and Mathematics), Social Science, Humanities, and Other areas. All the 52 subjects and their assigned categories are illustrated in Figure [
The primary source of our data is mock exams freely available on the internet. In addition, a portion of the college-level questions are past exam questions from top universities in China, publicly shared by the students. A minor fraction of college questions are mock questions for the national graduate entrance exam, sourced from the Weipu website these questions are not freely available to the public, and we have obtained their authorization to include around 2000 such questions into C-EVAL.
Data sources:
The collected data come various formats, primarily as PDF or Microsoft Word documents and a minor fraction as web pages. PDF documents are initially processed into text. All questions are subsequently parsed automatically when possible, and otherwise manually by the authors into a structured format, as exemplified in Figure 2 For subjects with complex mathematical notations such as many subjects in the STEM category, we manually convert them into standard ВБЕЕ formats, similar to Hendrycks et al. (2021b): Taylor et al. (2022). All the questions in C-EVAL are processed to include exactly four choices. Most the original questions were accompanied by four choices already, and we eliminate questions with fewer than four options and randomly drop incorrect choices for questions with more than four options. All questions also go through the standard data preprocessing pipeline, such as deduplication and cleaning. Following this, the questions undergo several rounds of human validation by the authors, and all the LSTEX notations ensured to be complied without syntax errors. We process at least around 200 questions for each subject, and randomly split the questions into a development set, a validation set, and a test set within each subject. The development split per subject consists of five exemplars to facilitate few-shot evaluation. These dev exemplars are also annotated with explanations to enable few-shot saain-of-though settings (Wei al.) 2022).
Data Processing:
detail next. The validation and test set created with a 1:9 ratio, where the validation set is intended to be used for hyperparameter tuning.
Explanation data generation: Chain-of-thought (COT) reasoning (Kojima et al.) 2022 Wei et al. 2022) - that prompts LLMs to generate a text sequence of reasoning process along with the final answer has shown great success reasoning- heavy tasks. Compared to zero-shot COT, the few-shot version is more commonly used and achieves the statc-of-the- performance on various tasks (Gao et al.) 2022 Wang al. 2023 Zhou al.) 2023 Xie et al. 2023). To facilitate the potential usage of C-EVAL in a few-shot COT setting, we combine automatic generation and human annotation to produce high-quality explanation data for the development split. Specifically, first
prompt GPT-4 to generate step-by-sted explanation to explain the ground-trum -trnh answer, then we

---

ILg (x*+ 1))+dddd2222))) 2ydzdx 32dxdy=
Let be bounded region enclosed the surface 2x y + and three coordinate planes, and I the surface of the region ft Calculate the surface integral: 1= flg (x* 1) Zydzdx + 3aatryy
1/2
B. C. 3/2
5/2
#
Answer: A
Figure 4: Example from advanced mathematies, subject in C-EvAL. HARD. English translations are shown below the corresponding Chinese text for better readability.
manually revise the generated explanations to obtain the final explanations. Details on prompting GPT- are in Appendix C
A dev example with explanations is illustrated in Figure 3
2.3 C-EVAL HARD
We select 8 challenging math, physics, and chemistry subjects from C-EVAL form separate benchmark, C-EVAL. HARD, which includes advanced mathematics, discrete mathematics, probability and statistics, college chemistry, college physics, high school mathematics, high school chemistry, and high school physics. These subjects often involve with complex 18liX equations and require non-trivial reasoning abilities solve, An example from advanced mathematics shown Figure 4 C-EVAL HARD aligns with recent efforts to create difficult benchmarks to assess advanced reasoning abilities (Hendrycks
al. 2021b Suzgun al. 2022), which are the key differentiators among various LLMs and could reflect LLMs' potential in general and complex scenarios. We emphasize that C-EVAL HARD the first Chinese benchmark provide highly complicated reasoning questions.
2.4 Evaluation
We use accuracy as the metric. While ground-truth labels of the development and validation splits are released, keep the labels of the test split private. This is ensure the fair use of the C-EVAL, as the C-EVAL data may unconsciously be included in pretraining data due to web crawling. Instead, users are required submit their model predictions to https://cevalleecchartt comm to automatically obtain the test accuracy, where public leaderboard is maintained. Users have the option to include their submission results in the live leaderboard, depending on their own preference.
3 Experiment
3.1 Setup
We evaluate LLMs in both zero- and five-shot settings on C-EVAL, where the five exemplars are from the development split. We adopt regular expressions extract answer choices from the model responses, ensuring that we can successfully extract answers for nearly all cases. We report answer- only (AO) results on both zero- and five-shot settings and tfain-of-thought (COT) results on the five-shot setting only, as we found that it was often difficult to extract the answer choices from zero-shot COT predictions where the generation does not follow specific patterns. Prompts of AO and COT are shown in Appendix D We note that in the COT setting, the five-shot exemplars could exceed the maximum context length of some LLMs for certain subjects. In such cases, we dynamically reduce the number of exemplars to fit within the context window.
3.2 Models
To give comprehensive view of the status of LLM in Chinese language context, we evaluate 11 accessible top- ppp-effrrmnng LLMs that are able to process Chinese input, covering diverse organizations and varying in size, as shown in Table ChatGPT (OpenAl (022) and GPT-4 (OpenAl 2023) are the strongest GPT model variants from OpenAL. Claude (Anthropic 2022). developed by Anthropic, often considered comparable to ChatGPT. We evaluate both the Claude-v1.3 and Claude-instant- v1.0 variants, with Claude-instane being lighter version. Bloomz-mt (Muennighoff et al.) 2022) is based on the pretrained multilingual BLOOM model (Scao al. (002)) with multitask prompted finetuning, thus is suitable for non-Englishh languages. LLaMA Touvron et al. 2023) is probably
5

---

Model
Creator
#Parameters
Access
GPT-4
OpenAI
undisclosed undisclosed
API
ChatGPT
OpenAl
API
Claude-v1.3
Anthropic
undisclosed
API
Maude-instant-vle
Anthropic
undisclosed
API
Bloomz-mi mt
BigScience
176B
Weights
LLaMA-65B
Meta
65B
Weights
GLM-1 30B
Tsinghua
130B
Weights
ChatGLM-6B
Tsinghua
6B
Weights
Chinese-LLaMA- LLaMA-13B
Cui
Weights
Chinese-Alpaca- 13B
Cui
Weights
MOSS
Fudan
168
Weights
Table Models evaluated this paper.
Model
STEM
Social Science
Humanities
Other
Average
Random
25.0
25.0
25.0
25.0
25.0
GPT-4
65.2
74.7
62.5
64.7
66.4
ChatGPT
49.0
58.0
48.8
50.4
51.0
Claude-v1.3
48.5
58.6
47.3
50.1
50.5
Bloomz-mt
39.1
53.0
47.7
42.7
44.3
GLM-130B
36.7
55.8
47.7
43.0
44.0
Claude-instant-vL.0
38.6
47.6
39.5
39.0
40.6
ChatGLM-6B
33.3
48.3
41.3
38.0
38.9
LLaMA-65B
32.6
41.2
34.1
33.0
34.7
MOSS
31.6
37.0
33.4
32.1
33.1
Chinese-Alppe Aaaesse-Appca-1113
27.4
39.2
32.5
28.0
30.9
Chinese-LIaMAA-Ie.
28.8
32.9
29.7
28.0
29.6
Table Zero-shot average accuracy (%) in answer-only setting. report the average accuracy over the subjects within each category. "Average" column indicates the average accuracy over all the subjects.
the best ween-weight foundation model so far that achieves the highest accuracy on the English MMLU benchmark within open-weight models. The aforementioned models except Bloomz-mt are English-oriente during development, while they are able to process Chinese input because a minor fraction of Chinese text present in the pretraining data.
We further include recent LLMs developed by Chinese institutions or individuals that is Chinese- oriented. GLM-130D (Zeng et al. 2023) and ChatGLM-6B (THUDM 2023a) based on the General Language Model architecture (GLM, Du et al. )022)) trained on English and Chinese data. ChatGLM-6B is further adapted conversational data. Chinese-LLaMA (Cui al.
2023) is
adaptation of LLaMA, which is further pretrained on Chinese data. Mlinese-Alpace (Cui et al.
2023) performs instruction tuning based on Chinese-LLaMA. MOSS (OpenLMLab 2023) is the
first publicly available Chinese LLM. and it follows a training procedure similar to ChatGPT. We note that there are some other commercial Chinese-oriented LLMs whose weights and APIs are not directly open to the public at the time of writing this paper, such Wenxin Yiyan (Baidu 2023). Tongyi Qianwen (Alibaba 2023), and Xunfei Xinghuo (iFLYTEK) (023), these models may have strong performance, yet we not authorized to evaluate and publicize their results. Therefore, we only report results from models with open APIS or weights in this work, while we anticipate the developers of other models to submit and optionally publicize their models' results in our website. A detailed description of the evaluated models can found Appendix
3.3 Results
General comparison: Zero- and five-shot answer-only results are shown in Table and Table respectively. We report the average accuracy, while a detailed breakdown of accuracy per subject is provided in Appendix F. GPT-4 the only model that exceeds 60% average accuracy, highlighting the challenge presented by C-EVAL. GPT-4 significantly outperforms all other models, with the second-best model, ChatGPT, trailing over 14 percentage points behind both zero- and five-shot settings. Claude-v1.3 achieves similar performance to ChatGPT, in terms of both the category-

---

Model
STEM
Social Science
Humanities
Other
Average
Random
25.0
25.0
25.0
25.0
25.0
GPT-4
67.1
77.6
64.5
67.8
68.7
ChatGPT
52.9
61.8
50.9
53.6
54.4
Claude-v1.3
61.7
52.1
53.7
54.2
Claude-instant-VI..
43.1
53.8
44.2
45.4
45.9
GLM-130B
34.8
48.7
43.3
39.8
40.3
Bloomz-mt
35.3
45.1
40.5
38.5
39.0
LLaMA-65B
37.8
45.6
36.1
37.1
38.8
ChatGLM-6B
30.4
39.6
37.4
34.5
34.5
31.6
37.2
33.6
32.8
33.3
MOSS
28.6
36.8
31.0
30.3
31.1
Chinese- Laaesse-lpcca--11 13B
26.0
27.2
27.8
26.4
26.7
Table 4: Five-shot average accuracy (%) in answer only setting. We report the average accuracy over the subjects within each category. "Average" column indicates the average accuracy over all the subjects.
Model
STEM
Social Science
Humanities
Other
Average
Random
25.0
25.0
25.0
25.0
25.0
GPT-4
67.3
76.5
64.4
66.6
68.3
Claude-v1.3
51.9
63.2
50.9
53.6
54.2
ChatGPT
47.8
58.3
47.7
48.5
50.0
Claude-instant-vl.
43.3
52.7
41.3
42.4
44.5
ChaiGL.M-B
29.9
40.0
37.9
34.5
34.5
MOSS
27.3
38.1
33.6
29.4
31.2
LLaMA-65B
28.0
36.3
29.3
30.0
30.3
GLM-130B
24.8
33.1
30.8
30.0
28.8
Chinese-ILaMAA-13B
20.5
30.5
28.2
27.1
25.4
Table Five-shot average accuracy (%) chain-of-though setting. report average accuracy over subjects within each category. "Average" column indicates the average accuracy over all the subjects. Bloomz-mt and Chinese-Al Alpaca- 13B are excluded as they could not generate valid reasoning and thus fail to answer for many questions.
average and the overall average. In addition average accuracy, Table in Appendix reveals that GPT-4 surpasses ChatGPT in almost every subject, indicating a comprehensive advantage. Among Chinese-oriented models, GLM-130B exhibits the best performance. ranking the fifth terms of both zero- and few-shot performance, 7.0 and 14.1 points behind ChatGPT in zero- and five-shot settings respectively. We observe that smaller models, despite undergoing instruction tuning, still struggle to achieve 40% accuracy. This contradicts recent assertions that a 10B-scale instruction-tunee model can achieve comparable performance to ChatGPT (Taori al. 2023 Chiang al. (023) we argue that while these models may perform well on simpler tasks, their inherent advanced abilities significantly lag behind when faced with more complex scenarios.
Does few-shot prompting help? Comparing Table to Table 3 we find that while few-shot prompting helps many models achieve better results, hurts performance of GLM-130B, Bloomz-mt, ChatGLM-6B, MOSS, and Aateesse-llaaaa-l- 13B. All of these models have undergone instruction tun- ing and we hypothesize that the accuracy drop is because that these models have not (appropriately) incorporated few-shot demonstrations into the instruction tuning stage, as emphasized in Chung al. (2022), thus sacrificing few-shot in-context learning performance to obtain enhanced zero-shot instruction-follooingg abilities.
Does chain-of-thoughh prompting help? The average accuracy in the COT setting reported in T5 while Table TO in Appendix FrcenddxxFprriiiiii provides a detailed breakdown of the accuracy per subject. We exclude Bloomz-mt and Chhnese-Alpaca- 13B since these two models are unable to generate valid COT reasoning for a large portion of questions, failing to produce final answers. All models achieve comparable lower average accuracy than in the answer-only setting. This suggests that
GGLM- 130B incorporates instruction tuning the pretraining stage.

---

Model
Zero-shot AO
Five-shot AO
Five-shot COT
GPT-4
53.3
54.9
56.8
Claude-v1.3
37.6
39.0
39.2
ChatGPT
36.7
41.4
35.0
Claude-instant-vl.
32.1
35.5
33.4
Bloomz-mt
30.8
30.4
GLM-130B
30.7
30.3
22.6
LLaMA-65B
29.8
31.7
21.4
ChatGLM--B
29.2
23.1
26.1
MOSS
28.4
24.0
21.6
Chinese-LLaMA-133
27.5
27.3
15.4
Chinese-Alpaca-113
24.4
27.1
Table 6: Average accuracy on C-EVAL HARD in both answer-only (AO) and thain-of-though (COT) settings.
COT prompting does not necessarily improve results for many subjects in C-EVAL. The primary reasons for this twofold: (1) many subjects in C-EVAL not reasoning-intensivve and additional reasoning steps may impair performance. This observation is supported by Chung et al. (2022), who noted performance degradation MMLU with COT prompting. (2) Some models fail to leverage the benefits of COT prompting, particularly those that did not undergo COT-inclusive instruction tuning. Chung al. (2022) reported this, noting 8-point accuracy drop when using COT MMLU with the 540B PaLMM model. This finding partly elucidates the significant decrease in performance still observe
of the GLM that COT prompting leads to considerable improvements for some models in certain subjects - for example, detailed results in Table show that COT improves GPT-4's performance in college physies from 50.6% to 60.2%, in probability and statistics from 53.6% to 62.0%, ChatGLM's performance in middle school physics from 20.2% 41.0%, and in high school geography from 29.2% 38.2%.
and ILaMA-65B models in the COT setting. Encouragingly,
Difference between English- and Chinese-oriented models: GLM-130B is the best- persfrreronn Chinese-oriented model in our assessment, thus we focus comparing it to the represented English- oriented model, ChatGPT, in zero-shot answer-only settings. We do not analyze GPT-4 here since it is not the same level as all other models, and comparing GL.M-130B it is very helpful and informative. As illustrated in Table 3 while GLM-130B underperforms ChatGPT by 7.0 points on overall average. the gap significantly narrows on the social science and humanities category, lagging only 2.2 and 1.1 points behind respectively. This reflects that by leveraging more Chinese data, the model might achieve performance on par with or even superior ChatGPT in areas pertaining to oriented
Chinese knowledge, such as history, politics, and law, highlighting situations where models may have the upper hand. However, concurrently, we note a significant difference of 12.3 points between GLM- 130B and ChatGPT in the STEM category, which implies substantial gap in more complex tasks that necessitate advanced reasoning skills.
Results on C-EVAL HARD: T the average accuracy on C-EVAL HARD. GPT-4 can only achieve 53.3%, 54.9%, 56.8% accuracy on zero-shot AO, five-shot AO, and five-shot COT settings respectively, implying the difficulty of C-EVAL HARD. Interestingly, chain-of-thought prompting improves GPT-4 slightly on these extremely challenging subjects. Indeed, only GPT-4, ChatGPT, and Claude manage to make meaningful progress improving by at least 10 points over a random baseline. Our results further confirm that some critical distinction among LLMs comes out when the tasks become complex enough. We underscore the importance of evaluating LLMs in such challenging settings, as current LLM development goes beyond creating a casual chatbot - involves the development of complex systems or agents capable of interacting with various data types, receiving feedback, reasoning and using tools, and even performing actions (Mialon et al. 2023).
Results on the validation split: Since we do not publicly release the labels for our test split, we provide the average accuracy on the validation split as a reference for developers. The validation split comprises a total of 1346 questions, with each subject contributing fewer than 30 validation questions on average. Therefore, tracking accuracy on specific subject may not yield significant insights. Instead, we report the average answer-only accuracy across all subjects in Table 7 The average validation accuracy closely mirrors the average test accuracy as presented in Table 3 and Table Additionally, the model ranking on the validation split broadly corresponds to that on the test split. These observations suggest that developers may use the average validation accuracy as a good indicator for expedited development processes.
8

---

Model
Zero-shot
Five-shot
GPT-4
66.7
69.9
Claude-v1.3
52.1
55.5
ChatGPT
50.8
53.5
Bloomz-mt
45.9
38,0
GLM-130B
44.2
40.8
43.2
47.4
ChatGLMM6B
39.7
37.1
LLLMA-65B
38.6
39.8
MOSS
35.1
28.9
Chinese- Alpacas--
32.0
27.2
Chinese-LLaMA-1BB
29.4
33.1
Table 7: Average accuracy the validation split in the answer-only setting.
4 Related Work
English benchmarks: Traditional English benchmarks mainly focus on assessing certain abilities of models a single task single type of tasks, such natural language understanding (NLU, Wang et al. (2019)), reading comprehension (Rajpurkar et al.] 2018), machine translation (Bojar et al. 2014), and summarization (Hermann et 2015 Narayan al. 2018). As r
representative example, the GLUE benchmark (Wang et al. (2019) combines a collection of NLU tasks, and has witnessed superhuman model performance due to the burst of pretraining models such as BERT (Kenton & Toutanova 2019) and GPT (Radford et al. 2019). In order to assess the capabilities of LLMs more comprehensively. recent benchmarks have cast light on the broader knowledge and advanced abilities. The MMLU benchmark (Hendrycks et al.) 2021a) provides multi-domaie and multi-task evaluation collected from real-world examinations and books. LLMs' performance on MMLU fluctuates around random-chance accuracy until they reach the scale of GPT-3. The BIG-bench benchmark (Srivastaval al. 2022) consists of 204 diverse tasks, some of which are considered be beyond the capabilities of current LLMs. The HELM benchmark (Liang et al., (022) aggregates 42 different tasks and evaluates LLMs with 7 metrics ranging from accuracy to robustness.
Despite the flourishing of English benchmark, language abilities in Chinese language environment remain under-developed. The CLUE benchmark (Xu al. 2020 the first uuse used and best available Chinese benchmark. Recently, the AGIEval benchmark (Zhong al., (023) contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination. The MMCU benchmark (Zeng) 2023) consists of tests from four major domains including medicine, law, psychology and education, which are also collected from Chinese College Entrance Exam, qualification test as well as university examinations. Compared to AGIEval and MMCU, C-EVAL (1) has broader coverage of domains (2. (2) features four different levels of difficulty - particularly, the C-EVAL HARD benchmark is the first Chinese benchmark to provide sophisticated reasoning problems, and (3) makes effort to mitigate data leakage our questions mostly come from mock exams as PDF or Microsoft Word documents that are further processed by us, while AGIEval and MMCU collects the exact questions from past national exams in China.
Chinese benchmarks:
large-scale Chinese NLU benchmark, and still serves as the most widely
Discussion
We believe that the evaluation of LLMs should transcend the scope of casual conversational bots, guiding developers in preparing LLMs to function in more complex scenarios. This was the pri- mary motivation behind the creation of C-EVAL, challenging evaluation suite. We hope that C-EVAL along with C-EVAL HARD have made important progress on this direction particularly in Chinese context. We also note that, C-EVAL, along with all the English-languuae benchmarks, is far from perfect for L.M evaluation. There are many other abilities such as reasoning over and calling APIs, as well as multiple aspects that extend beyond accuracy, including safety, bias, and robustness. We leave further exploration their evaluation for future work.

---

Acknowledgement
We thank the anonymous reviewers for their comments. Yuzhuo Bai is supported by the National Key R&D Program of China (No. 2020AAA0106502) and Institute Guo Qiang Tsinghua University.
References
Alibaba. Tongyi qianwen. Alibaba Blog, 2023. URL hwwph:tttt//oootooo aliyun..aaiiu....mm com
Anthropic. Introducing claude. Anthropic Blog, 2022. URL https://wwwt http:::/www..nthropicccco com/
matee//introuccing--oo claude
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:22 2022.
Baidu. Wenxin yiyan. Baidu Blog, 2023. URL https:///yyan baidu. com.
Ondfej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Level- ing, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Aleš Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceed- ings of the Ninth Workshop Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 1..1115/v1/1143300 URL https://aclanthologg org 1/////1433330 3302
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An sper-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URLhttps
//Imsys. , wtgg/bllg//223333 03- vicuna/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction caatuction-finetuneed language models. arXiv preprint arXiv:2210.11416 2022.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchir Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:21 / 10. 14168, 2021.
Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for chinese llama and alpaca. arXiv preprint arXiv:2304.08177, 2023. URL htttttt///axxiii org/ abs/ 2304 08177
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang. and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp- 320-335, 2022.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamic Callan, and Graham Neubig. Pal: Program- aided language models. arXiv preprint SXXi:2221111044 2022.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021a. http:ppps///peeeeeeeeeeeeee//res
10

---

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang. Dawn Song. and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In J. Vanschoren and S. Yeung (eds.), Proceedings of the Neural Infor- mation Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021b. URL httppe//feeaatts--oooonrrrrpppee
neuripp.cc/paper. neurips cc/ccccpppaooc..c fateerfffllsppperrreeeeee
pdf
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 220Ki:::203.155556 2022.
iFLYTEK. Xunfei xinghuo. iFLYTEK Blog, 2023. URL https:// yfy.n...oo.uuu..oommm.mmm...
Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. Bert:
Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pp. 4171- 4186, 2019.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Alice Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openrevierr net formm?dd- e2TBb5y0yFf
Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models. arXiv preprint arXiv:2211.091 2022.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasumuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302. 078422 2023.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual generalization through multitask finetuning. arXiv preprint arXiv:2211.01788 2022.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 18653vv//v1/ D18-1206. URL httptsss///c..aathoooogg/ wwwpggg......og/// /rg/ 1206
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv: 13474, 2022.
OpenAl. Chatgpt:
Optimizing language models for dialogue. OpenAl Blog, 2022. URL https
//openai.com/bllg//bai chatgpt/
OpenAl. GPT-4 technical report. arXiv preprint crX1::333330000..777..4, 2023.
OpenLMLab. Moss.
2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9 2019.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784 789, 2018.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXhv:2211.05100 2022.
11

---

Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro. Aditya Gupta, Adrià Garriga-Alosoo et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.046155 2022.
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of ofthun- can solve them. arXiv preprint arXiv:2210.02261, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An inttructionfolloviiii llama model. https://githul: cww coom/tttsu-alle www.aaaaaaaodda.......ao allaca 2023.
Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arX1v:2211.09088. 2022.
THUDM. ChatGLM. httpps//gthhuu..coo///b//////a/p SAaam/HHHDCCCChCCLL2-00000 6B 2023b.
httpp//aw..ccooooooo///aaaaaaa.. 2023a.
THUDM. Cha https://
com
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1391 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019, URL https:://operreviie net/ Forun/id==
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL httpp:://penreeves peep/fooummiddIPIIINNM
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruet: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 10560, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htt.httts///ppeeeiieeee www.:nruffffmmiiaa VjQQMeSB__
Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, Min- Yen Kan, Junxian He, and Qizhe Xie. Decom- position enhances reasoning via self-evaluation guided decoding. arXiv preprint arXiv:2305.0063: 2023.
Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng. Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. CLUE: A Chinese language understanding evaluation benchmark. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 4762- 4772, Barcclona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10. 1/200:3/1/22200.oooinnoo m1i..419. URL httpsttts:://// sattps://aclantholloe org 2020 coling-man main. 419
Aohan Zeng, Xiao Liu, Zhengxiao Du Zihan Wang. Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Zhiyuan Liu, Peng Zhang, Yuxiao Dong, and Jie Tang. GLM-130b: An open bilingual pre-trained model. In The Eleventh International Conference on Learning Representations, 2023. URL htttp::///oot //tbenre//opeerviee.e www.:s/onntoooooooodooe-e AworrrPU
12

---

Hui Zeng. Measuring massive multitask chinese understanding, arXiv preprint arXiv22304.19886 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint 230.::233000003344, 2023.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoe V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL http://pppaeaeeeeeeeee/ooooetaaa
13

---

A Author Contributions
Data collection, annotation, and initial validation: Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Yikai Zhang, and Jiayi Lei collected the first version of the data, including some necessary manual annotation from PDF or Word documents. They then cross-validated the collected data, Junlei Zhang prompted GPT-4 to generate the explanation data, and then Yuzhen Huang, Yuzhuo Bai, and Junxian He manually revised the explanations.
Data processing: Yuzhen Huang did most of the data processing job including a lot manual revisions, such fixing typos/incorrect formats of the questions from PDF documents and ensuring all the XTTх notations can be correctly compiled. Jinghan Zhang did the data deduplication.
Evaluation: Yuzhen Huang tested ChatGPT and MOSS; Yuzhuo Bai tested ChatGLM; Junlei Zhang tested GLM-130B; Chuancheng Lv tested Bloomz-mt, LLaMA, Chinese-LLaMA, and Chinese- Alpaca; Yao Fu tested GPT-4 and the Claude model variants.
Website and submission system: Zhihao Zhu built the website and the online submission system.
Paper Writing: Yuzhen Huang, Yuzhuo Bai, and Junxian He wrote the main content this paper, while other authors helped proofread.
Advising: Yao Fu, Maosong Sun, and Junxian He take advisor roles this project. Junxian He the main advisor, initializing and organizing this project.
B Detailed Stats of C-EVAL
Table 8 lists all the C-EVAL tasks and their broader categories, well as the number of questions included each task.
C Explanation Data Generation
We show an example of the automatic COT explanation generation via GPT-4 in Figure 5 We use five human-written question-explanation pairs prompt GPT-4 to generate explanation giving question and its correct answer. The generated explanations are further revised manually to obtain the final explanations.
D Evaluation Prompts
show the evaluation prompts of answer-only and chain-of-thought test Figure (nd Figure respectively.
E Details of the models being evaluated
ChatGPT and GPT-4 are GPT-series models that enhanced to follow human instructions and be more helpful, harmless and honest using Reinforcement Learning from Human Feedback. GPT-4 additionally enables image inputs and goes through well-designed post-training alignment process, as well as having larger scale than most of the existing model. GPT-4 achieves human-level performance various benchmark, and even scored be the top 10% in some simulated exams.
Claude is the latest Anthropic-series LLM that also focuses on human intention alignment. Ap- plying the constitutional AI approach(Bai al. 2022 Claude manages to be both helpful and trustworthy, Claude-instant the lighter vesrion with less cost and faster inference of Claude.
BLOOMZ-mt (Muennighoff et al. 2022) is created by combining multitask prompted finetuning to the pretrained multilingual BLOOM model al. (2022), using not only English prompts but also machine-translate prompts to match the language of multilingual tasks, and are found to capable at task- and lansutge-agnostic generalization. We evaluate the 176B version in our experiment.
14

---

Subject
Category
# Questions
Advanced Mathematics ())
STEM
197
College Chemistry (* 3 (2-))
STEM
253
College Physies (6)))))))))))
200
College Programming (7+ tE
STEM
384
Computer Architecture (
STEM
219
Computer Network (if 12
STEM
195
Discrete Mathematics ( #X
STEM
174
Electrical Engineer (ft: L.
STEM
381
High School Biology (78g 950)
STEM
199
High School Chemistry
(r
STEM
196
High School Mathematics
(P)
STEM
189
High School Physies 46
STEM
199
Metrology Engineer (1E if
248
Middle School Biology (B71)) (0:
STEM
218
Middle School Chemustry (87 (244)
STEM
210
Middle School Mathematics (3J ()))))
STEM
201
Middle School Physics ( (1))))))
STEM
202
Operating System (fft (6))))
STEM
203
Probability and Statistics (HE (()))))
STEM
189
Veterinary Medicine (P)
STEM
238
Business Administration [T 1
Social Science
339
College Economics 20
Social Science
557
Education Science (#X
Social Science
304
High School Geography (1911))
Social Science
202
High School Politics ( (FU((1))
Social Science
200
Mao (aaee
Social Science
248
Marxism (3) 3...EF *
Social Science
203
Middle School Geography (67
Social Science
125
Middle School Politics (5)c) (2)h)))
Social Science
219
Teacher Qualification (36( 07
Social Science
448
Art Studies (Z*
Humanities
336
Chinese Language and Literature (#
Humanities
237
High School Chinese ( (#5 x2)
Humanities
202
High School History (0) 0j
Humanities
207
Ideological and Moral Cultivation (B
Humanities
196
Law (12--)
Humanities
250
Legal Professional (1 (1
Humanities
243
Logic ()
Humanities
231
Middle School History ( ))
Humanities
234
Modern Chinese History (i du
Humanities
240
Professional Guide (5 B81
Humanities
300
Accountant ( 2: if
Other
497
Basic Medicine 411
Other
199
Civil Servant (2: M)
Other
481
Clinical Medicine (1i (1
Other
227
Environmental Impact Assessment Engineer (174 IE
Other
317
Fire Engineer (iE: 11 II
Other
318
Physician (BE We (8)
Other
497
Plant Protection (RIP)
Other
226
Sports Science (%)
Other
204
Tax Accountant ( (T]
Other
497
Urban and Rural Planner (1: (Lb())
Other 469
Table 8: Summary of all subjects.
LLaMA Touvrrn al. 2023) is a Transformer-und arrhhierreerr LLM that is trained on mixture of several open sources. Applying several improvements on vanilla Transformers used by previous LLMs, and optimized to improve the training efficiency, LLaMA shows strong language abilities and can surpass models having 10x larger parameters than LLaMA. We evaluate the LLaMA-65B version in our experiment.
GLM-130B and ChatGL--6B are based on the General language model (GLM) structure that gain benefits from its bidirectional attention advantage. By using alterable number and length of
15

---

[5-shot examples]
T -
The following are multiple-choice questions about middle school physics China. Please generate correct reasonable explanation based on the following questions and answers.
+ 6E
Which the following renewable energy source?
A. petroleum
B. coal C. nuclear fuel D. solar energy
;
Answer:
What is your explanation generated?
(A) - T
(B) カ
(C) 14
f # EXTNMMREEREER B
* 7
4. (D) A
- B TA
3
AT X ED
Explanation: Let's think step by step,
(A) Petrolcum fossil extracted from deep underground and not a renewable energy source.
2. (B) Coal is also fossil fuel extracted from deep underground and is not a renewable energy source.
3. (C) Nuclear fuel refers the fuel used in nuclear reactors, including uranium, plutonium, etc. Although energy
be generated through nuclear reactions, nuclear fuel is limited and is renewable energy source.
(D) Solar energy refers to the energy obtained from solar radiation and renewable energy source, hecause the sun rises every day and will not be exhausted due to use. Therefore, the answer is C.
Figure An example of generating explanations via GPT-4 The red text is the autocompleted response from model, while the preceding text is the user-inputted prompt. We indicate English translation below the corresponding Chinese text for each paragraph.
#H II #
The following are multiple-choice questions about clinical medicine in China. Please select the correct answer.
3E 65% 10/ #F: pH 7.26, PaCO, 82mmHg, PaO, 45mmHg. T 1E
A
A 65-ycar-old male patient has been in coma for 10 hours. Blood gas test shows pH of 7.26, PaCO, 82mmHg, and PaO, 45mmHlg. Which of the following treatments is not appropriate?
A. # B. *
C. ボ A. mechanical ventilation B. respiratory stimulants C. supportive therapy D. high concentration oxygen therapy
D. $
X ::
Answer: D
[5-shot examples]
* #
Silicon dust can cause silicosis, and most pathogenic silicon dust particles have diameter
A. > 5pm
5pm
C.1 2um
2-2 pm
Answer: C
Figure 6: An example of few-shot evaluation in answer-only scenarios, while zero-shot evaluation is similar by removing the exemplars. The red text is the autocompleted response from model, while the preceding text is the inputted prompt. We indicate English translation below the corresponding Chinese text.
blanks, GLM can adapt to various tasks. GLM-130B a bilingual pre-trained GLM that ultilizes self- supervised learning and multitask instruction pretraining. GLM-130B also realized INT4 quantization
16

---

The following are multiple-choice questions about computer architecture in China. Please select the correct answer.
[5-shot examples]... ...
( Σ क H ii 25 44500H, If the content of register R is sct to 1500H, the content of memory address 1500H 2500H, the content of memory address 2500H is 3500H, and the value of the program counter (PC) is 4500H, using the relative addressing mode, the operand the effective address of 2500H is
15 2500H, # 3500H,, PCB5
1500H
B. 2500H - - - B
3500H
D. 4500H
: 1R
1. #H
के AJ
# 55500--5500-1000000
#!. 3U3500H.
#
Answer: Let's think step by step,
1. According to the information given in the question, the content of memory unit 1500H is 2500H, and the content of memory unit 2500H 3500H. This indicates that the current instruction address is 2500H-
1500H-1000HL
2. According to the relative addressing mode, in order to obtain the actual operand address of the effective address with the value of 2500H, the current instruction address of 1000H needs to be added to it, resulting in the actual operand address of 3500H. Therefore, the answer
Figure 7: example of few-shot evaluation in chain-of-thougee scenarios, while zero-shot evaluation is similar by removing exemplars. The red text is the autocompleted response from model, while the preceding text the inputted prompt. We indicate English translation below the corresponding Chinese text.
with little no quality degradation that significantly accelerate its inference efficiency. ChatGLM-6B is lightweight conversational version of the GLM family that has been specially optimized for Chinese contexts. CharGLM-6B also applies quantization so it can be deployed with consumer- grade graphic memory requirement as little as 6GB. We evaluate on the fp16 settings for both two models in our experiment.
Chinese-LLaMa. is an adaptation of original LLaMA into Chinese language environments. ChiMese-LLaMM expands the original LLaMA by adding 20K Chinese tokens into its vocabulary, and secondarily pre-trained and instruction fine-tuned on Chinese data. We evaluate Chinese-I. 13B our experiment, the largest Chinese-LLaMA variant.
Chinese-Alpace is based on the Chinese-LLaMA checkpoint that is further tuned on Chinese instruction tuning data. We evaluate Chinese-Alpaca-1 13B in experiment, the largest Chinese- Alpaca variant.
MOSS is the first open-source Chinese LLM that matchs ChatGPT on both the training scale and alignment techniques. MOSS is initialized with CodeGen(Nijkamg) et al. 2022), being pretrained on 100B Chinese tokens and 20B English tokens, and has further integrated supervised fine-tuning and preference model, as well as plugin augmentation, but not all the version are publicly available. We evaluate the mosss-moon-003--55 version in our experiment.
F Breakdown of Model Performance
Table 9 and Table 10 show the detailed breakdown of accuracy per subject of four representative models s in zero- and five-shot settings respectively, while we refer the readers to our website leader- board hwarr///s//sseeloooooooooooooone models.
for a detailed breakdown results for all
17

---

Subject
GPT-4
ChatGPT
GLM-130B
Claude-instant-VLe
Advanced Mathematics
48.6
38.2
28.9
36.4
College Chemistry
54.0
36.2
29.0
25.9
College Physics
50.6
34.1
29.5
33.5
College Programming
72.2
56.1
33.3
40.6
Computer Architecture
73.1
62.7
38.3
42.5
Computer Network
74.9
60.8
36.3
44.4
Discrete Mathematics
37,9
34.6
28.8
Electrical Engincer
48.7
38.9
31.6
32.7
High School Biology
69.1
49.1
37.7
40.0
High School Chemistry
55.2
45.3
36,0
39.5
High School Mathematics
41.0
28.3
34,9
24.7
High School Physics
65.1
36,6
26,9
40.0
Metrology Engineer
68.0
58.4
49.8
48.4
Middle School Biology
88.0
68.2
51.6
46.4
Middle School Chemistry
82.7
63.2
45.9
47.0
Middle School Mathematics
40.1
32.2
34.5
Middle School Physies
61.2
41.6
48.9
Operating System
79.3
70.4
42.5
46.4
Probability and Statistics
50.0
36.7
25,9
27.7
Veterinary Medicine
75.7
56.7
47.6
43.8
Business Administration
63.1
48.2
43.2
40.2
College Economics
67.6 6
47.3
38.6
40.8
Education Science
67.4
$4.8
$2.6
43.0
High Schnol Geography
73.6
55.6
51.1
42.7
High School Politics
74.4
50.0
45.5
37.5
Mao Zedong Thought
$6.6
67.6
49.3
Marxism
70.9
69.3
62.0
Middle School Goography
78.7
57.4
58.3
49.1
Middle School Politics
85.5
69.4
61.1
$7.0
Teacher Qualification
84.5
69.9
70.7
54.9
Art Studies
57.4
47.7
49.3
33.2
Chinese Language and Literature
61.2
$1.7
46.4
35.9
High School Chinese
38.8
37.6
25.8
31.5
High School History
48.4
48.4
41.2
Ideological and Moral Cultivation
59.3
58.1
Law
37.6
34.8
Legal Professional
55.3
40.5
35.8
36.7
Logic
62.3
38.7
34.3
36.3
Middle School History
83.6
65.2
65.2
48.8
Modern Chinese History
63.7
46.2
59.9
42.5
Professional Tour Guide
68.8
53.4
63,2
35.0
Accountant
63.4
46.3
39.7
38.1
Basic Medicine
78.9
61.1
46.9
39.4
Civil Servant
45.5
45.2
36.1
Clinical Medicine
$1.0
43.5
Environmental Impact Assessment Fagineer
59.4
52.0
45.6
41.3
Fire Engineer
45.0
41.5
35.8
29.4
Physician
75.6
59.4
44.7
41.5
Plant Protection
71.4
56.3
49.7
48.2
Sports Science
70.0
$3.9
44.4
40.6
Tax Accountant
56.0
38.4
33.9
38.1
Urban and Rural Planner
59.1
49.0
43.5
38.0
Table 9; Zero-shot answer only accuracy per subject.
G
Option Bias
The distribution of the correct answer is shown in the table III We admit that there are fluctuations in the proportion of different options. However, these fluctuations are relatively small around a random 25% level and similar MMLU.
To verify the option bias exists on models, we permuted order of the choices the questions and randomly selected distinct permutations of choices for each question. Then we evaluate ChatGPT, ChaiGLMM66 and ChatGL.M2-RR (THUDM) (023b) on the different permutations of C-EVAL.
The shot results in the answer-only setting shown in table 12 We report the average accuracy
18

---

Subject
GPT-4
ChatGPT
ChatGLM
Claude-instant-vle
Advanced Mathematics
59.7
48.0/ 29.5
3.5/23.1
39.9 41.6
College Chemistry
55.4
39.7/33.9
17.07
33.5 13.5/32.11
College Physics
50.6/
35.2/
14.8/30.1
31.8/31.3
College Programming
78.1
57.9/57.9
18.7/24.0
58.5
Computer Architecture
74.67
68.9/63.7
30.1
52.3/
Computer Network
77.8/77.2
/9.5
36.3/28.1
46.8/
Discrete Mathematics
66.7
43.1/30.7
16.0/26.1
32.7
Electrical Engineer
49.9/
39.5/44.0
23.9/31.3
31.7
ligh School Biology
72.6/70.3
53.7/48.0
28.0/26.9
44.6 38.9
High School Chemistry
52.3/
$2.9/6.00
19.5
39.0
High School Mathematics
38.0/42.8
34.3/37.3
4.2/24.1
/3.5
High School Physics
69.1/61.1
43.4/34.9
10.9/2177
44.0/ 26.9
Metrology Engineer
70.3/71.2
63.0/5988
41.1/48.
50.7 55.3
Middle School Biology
91.2/86.5
67.7/63.5
30.7/38.0
57.3
Middle School Chemistry
81.1/71.4
71.4/60.0
36.8/42.
58.4
Middle School Mathematics
66.7/66.7
44.6/42.4
3.4/20.3
33.3
Middle School Physics
81.5/7877
66.9/57.9
20.2/410
/
Operating System
82.1/804
60.5
40.8/27.4
57.0/
Probability and Statistics
53.6/62.0 62.0
33.7/42.8
27.7/26.5
34,9/31.9
Veterinary Medicine
80.0/ 80.0
64.3/58.1
39.5/35.2
53.0/5.33
Business Administration
67.1
52.8/46.8
32.6/34.9
44.2/42.9
College Economics
71.4/7.44
52.5/51.3
22.1/30.0
/
Education Science
68.5
59.6/54.4
37.8/33.0
51.1/519
High School Geography
70.2
58.4/55.1
29.2/38.2
52.8/49.
High School Politics
77.8/69.3
52.3/51.1
29.0/
/
Mao Zedong Thought
79.9
60.7/63.5
46.6/
47.0/
Marxism Middle School Geography
81.6/82.7
71.5/
40.2
63.5
83.3/83.3
60,2/56.5
33.3/44.4
52.0/
Middle School Politics
87.1/8555
74.1/
41.5/ 40.5
60.6/
Teacher Qualification
85.0
76.9
48.9/49.1
62.2
Art Studies
66.1/66.1
52.3
34.6/ 36.9
38.6
Chinese Language and Literature
67.2/6700
50.2/517
32.5/37.3
41.2 44.0
High School Chinese
37.6/39.3
36.0/27.5
/ 19.1
30.9/27.0
High School History
68.1/68. 68.1
$4.4/45.1
35.2/40.1
52.7 $2.7/40.77
Ideological and Moral Cultivation
77.3/77.3
68.9
52.3
6288633 63.4
Law
60.6/ 54.8
43.9/34.8
23.5/
38.0 33.0
Legal Professional
54.4/48.4
44.7/32.6
27.0/26.5
30.5
Logic
63.3/63.2
37.7
32.8/314
/
Middle School History
84.5/
62.8/63.8
42.7
5
Modern Chinese History
68.9/67.0
52.8/514
36,3/39.2
44.8 44.8
Professional Tour Guide
70.
61.3/620
44.0/519
43.6/44.0
Accountant
64.6/61.9
51.7/42.0
23.9/32.7
47.2/41.3
Basic Medicine
80.6/78.3
61.1/60.6
31.4/3.77
49.7/45.1
Civil Servant
/2.5
46.9/43.1
24.5/
42.0/41.3
Clinical Medicine
76.0/73.5
55.0/53.0
34.5/32.0
36.5
Environmental Impact Assessment Engineer
63.7/59.
50.9/49.1
35.7
47.3/47.3
Fire Engincer
50.0/49. 49.6
42.6/37.9
28.4/32.3
36.5/36.2
Physician
76.8
63.7/54.6
34.8/37.7
46.5
Plant Protection
78.9/80.
65.8/57.8
43.2/36.2
50.3/50.3
Sports Science
/
55.3
42.2/41.1
45.6 46.7
Tax Accountant
58.0 28.0/58.22 58.2
42.0/35.7
16.3/305
42.7 /317
Urban and Rural Planner
63.2 65.6
$2.2/49.3
36.8 /37.3
4 42.3
Table 10: Five-shot accuracy per subject. We report both the answer only (AO) and chain-of-thought (COT)
accuracy an AO COT format.
Option
C-Eval
MMLU
22.9%
23.1% 1%
B
26.0%
24.7%
C
26.4%
25.5%
D
24.7%
26.7%
Table The distribution correct answer.
across permutations and the variance of the overall accuracy. The results imply that the variance across different permutations is relatively small.
19

---

Model
STEM
Social Science
Humanities
Other
C-Eval Hard
Average
Var
ChatGPT
48.1
58.1
48.6
49.9
36.3
50.5
0.6
ChatGLM-66
33.4
47.5
41.9
37.5
28.7
38.8
0.7
ChatGl M22-6
38.1
58.3
49.6
45.2
29.3
45.9
0.2
Table 12: Zero-shot average accuracy (%) in answer-only setting. We report the average accuracy over 5
permutation within each category and overall accuracy. "Average" column indicates the average accuracy over permutation. "Var" column indicates the variance the overall accuracy.
H Compute and Resources Used for Evaluation
During experiments to evaluate different LLMs C-EVAL, utilize cluster with 8 A100- 80GB GPUs to run the inference for models with released weights, such resources are required due deploying three large models Bloomz-mt (176B). LLaMA-65B, and GLM-130B. In most cases the inference on C-EVAL is finished within one day. For models with API access, we just run the inference with CPUs which finishes mostly within one day as well.
20


Reviews:
[{"review_id": "1jz4XPWAdj", "reviewer": "Reviewer_UDRo", "summary": "", "questions": "", "limitations": "while this paper introduces a valuable dataset, this work itself is lack of new insights.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The datasets are comprehensive with different difficulties and disciplines. The datasets also supports various test setups.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "2xv3l9wK4T", "reviewer": "Reviewer_jtRh", "summary": "", "questions": "", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1-Comprehensive Evaluation Suite: The paper introduces C-EVAL, a comprehensive evaluation suite specifically designed to assess the advanced knowledge and reasoning abilities of large language models in a Chinese context. The suite covers multiple-choice questions across various difficulty levels and a wide range of disciplines, making it a valuable resource for evaluating the capabilities of different models.\n2-Real-World Challenging Exams: C-EVAL is constructed using real-world, challenging human exams in China, which are used to assess human abilities across multiple dimensions. This ensures that the evaluation is relevant to real-world scenarios and provides a meaningful assessment of model performance in complex tasks.\n3-Focus on Advanced Abilities: The paper emphasizes the evaluation of advanced abilities in large language models, such as world knowledge and reasoning, which are crucial for their practical applications. By focusing on these complex tasks, the authors provide a more in-depth analysis of the strengths and weaknesses of various models.\n4-Mitigation of Data Contamination: The authors take precautions to mitigate potential data contamination issues by carefully sourcing the data from mock exams and small-scale local exams, rather than using exact questions from official national exams. This ensures the fairness and integrity of the evaluation process.\nOverall, the paper's strengths lie in its comprehensive evaluation suite, focus on advanced abilities, careful data handling, evaluation of Chinese-oriented models, and assessment of challenging scenarios, all of which contribute to a valuable and rigorous evaluation of large language models.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "zGfQzvX9pW", "reviewer": "Reviewer_KZk1", "summary": "", "questions": "", "limitations": "They address some but not very much about the limitations. I think whether a model knows prerequisite knowledge about one question matters. I asked some easy professional questions but GPT returns the wrong answer. As a result, I think if we only use accuracy as the metric, it is vital to consider the impact of some prerequisite knowledge on a question. Besides, the submission lacks some comparison between this work and previous work. And this work contains a wide range of domains whose subsets can be used separately for a specific domain. However, there may be cases that for one domain, a benchmark that focuses on that domain works better than this work does.\n", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The submission proposes a suite of benchmarks in a Chinese context, covering a wide range of domains. The benchmark fully considers the difficulty of problems by selecting questions from different difficulty levels, making its evaluation results more reliable. I think it has high quality, considering the lack of a Chinese context benchmark and its wide range of domains, which is quite helpful for later researchers. Moreover, all questions are carefully collected and preprocessed, trying their best to minimize possible ethical and social implications. The models evaluated are comprehensive, including both general-purpose models and specially trained models for the Chinese context. This work also compares different settings, such as zero-shot and few-shot settings, answer-only, and COT settings.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "8Q1GBeUnNQ", "reviewer": "Reviewer_WJEA", "summary": "", "questions": "", "limitations": "Yes.", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "The benchmark is useful for the current situation that there's a lack of inclusive dataset for the evaluation of increasing releases of LLMs. The scores on the benchmark can reflect how a LLM performs on the wide range of topics in Chinese. The data sources used for the construction of the dataset is of high quality as the questions are mostly exam questions in real use cases.  The authors conducted careful preprocessings and manual annotations to secure the quality of the data.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "ECINoST5A7", "reviewer": "Reviewer_ZHas", "summary": "", "questions": "", "limitations": "The limitations and and potential negative societal impact of their work are well addressed.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The proposed benchmark is comprehensive for evaluating LLMs’ knowledge and reasoning abilities in a Chinese context. This benchmark has been widely used by many recent LLMs (according to their website).\n- The benchmark setting is reasonable. All questions and answers are manually checked, which guarantee the accuracy of the benchmark.\n- The paper is very clear. The experiments adequately validate the LLMs’ ability of ICL and CoT.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper presents a benchmark for analysing Chinese LLMs.  Generally, reviewers agree that the benchmark is comprehensive it seems likely this benchmark will push forward work in Chinese language modelling.  This should be an interesting addition to the Benchmarks track.
Decision: Accept (Poster)