SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 5: Five-shot average accuracy (\%) in chain-of-thought setting. We report the average accuracy over the subjects within each category. "Average" column indicates the average accuracy over all the subjects. Bloomz-mt and Chinese-Alpaca-13B are excluded as they could not generate valid reasoning and thus fail to answer for many questions.
average and the overall average. In addition to average accuracy, Table 9 in Appendix F reveals that GPT-4 surpasses ChatGPT in almost every subject, indicating a comprehensive advantage. Among Chinese-oriented models, GLM-130B exhibits the best performance, ranking the fifth in terms of both zero- and few-shot performance, 7.0 and 14.1 points behind ChatGPT in zero- and five-shot settings respectively. We observe that smaller models, despite undergoing instruction tuning, still struggle to achieve a  $40\%$  accuracy. This contradicts recent assertions that a 10B-scale instruction-tuned model can achieve comparable performance to ChatGPT (Taori et al., 2023; Chiang et al., 2023) – we argue that while these models may perform well on simpler tasks, their inherent advanced abilities significantly lag behind when faced with more complex scenarios.
Does few-shot prompting help? Comparing Table 4 to Table 3, we find that while few-shot prompting helps many models achieve better results, it hurts performance of GLM-130B, Bloomz-mt, ChatGLM-6B, MOSS, and Chinese-Alpaca-13B. All of these models have undergone instruction tuning, and we hypothesize that the accuracy drop is because that these models have not (appropriately) incorporated few-shot demonstrations into the instruction tuning stage, as emphasized in Chung et al. (2022), thus sacrificing few-shot in-context learning performance to obtain enhanced zero-shot instruction-following abilities.
Does chain-of-thought prompting help? The average accuracy in the COT setting is reported in Table 5, while Table 10 in Appendix F provides a detailed breakdown of the accuracy per subject. We exclude Bloomz-mt and Chinese-Alpaca-13B since these two models are unable to generate valid COT reasoning for a large portion of questions, failing to produce final answers. All models achieve comparable or lower average accuracy than in the answer-only setting. This suggests that
$^{6}$ GLM-130B incorporates instruction tuning in the pretraining stage.
Model	Zero-shot AO	Five-shot AO	Five-shot COT
GPT-4	53.3	54.9	56.8
Claude-v1.3	37.6	39.0	39.2
ChatGPT	36.7	41.4	35.0
Claude-instant-v1.0	32.1	35.5	33.4
Bloomz-mt	30.8	30.4	-
GLM-130B	30.7	30.3	22.6
LLaMA-65B	29.8	31.7	21.4
ChatGLM-6B	29.2	23.1	26.1
MOSS	28.4	24.0	21.6
Chinese-LLaMA-13B	27.5	27.3	15.4
Chinese-Alpaca-13B	24.4	27.1	-
Table 6: Average accuracy on C-EVAL HARD in both answer-only (AO) and chain-of-thought (COT) settings.
COT prompting does not necessarily improve results for many subjects in C-EVAL. The primary reasons for this are twofold: (1) many subjects in C-EVAL are not reasoning-intensive, and additional reasoning steps may impair performance. This observation is supported by Chung et al. (2022), who noted performance degradation on MMLU with COT prompting. (2) Some models fail to leverage the benefits of COT prompting, particularly those that did not undergo COT-inclusive instruction tuning. Chung et al. (2022) reported this, noting an 8-point accuracy drop when using COT on MMLU with the 540B PaLM model. This finding partly elucidates the significant decrease in performance of the GLM-130B and LLaMA-65B models in the COT setting. Encouragingly, we still observe that COT prompting leads to considerable improvements for some models in certain subjects – for example, detailed results in Table 10 show that COT improves GPT-4's performance in college physics from  $50.6\%$  to  $60.2\%$ , in probability and statistics from  $53.6\%$  to  $62.0\%$ , ChatGLM's performance in middle school physics from  $20.2\%$  to  $41.0\%$ , and in high school geography from  $29.2\%$  to  $38.2\%$ .
Difference between English- and Chinese-oriented models: GLM-130B is the best-performing Chinese-oriented model in our assessment, thus we focus on comparing it to the represented English-oriented model, ChatGPT, in zero-shot answer-only settings. We do not analyze GPT-4 here since it is not at the same level as all other models, and comparing GLM-130B to it is not very helpful and informative. As illustrated in Table 3, while GLM-130B underperforms ChatGPT by 7.0 points on overall average, the gap significantly narrows on the social science and humanities category, lagging only 2.2 and 1.1 points behind respectively. This reflects that by leveraging more Chinese data, the model might achieve performance on par with or even superior to ChatGPT in areas pertaining to Chinese knowledge, such as history, politics, and law, highlighting situations where Chinese-oriented models may have the upper hand. However, concurrently, we note a significant difference of 12.3 points between GLM-130B and ChatGPT in the STEM category, which implies a substantial gap in more complex tasks that necessitate advanced reasoning skills.
Results on C-EVAL HARD: Table 6 shows the average accuracy on C-EVAL HARD. GPT-4 can only achieve  $53.3\%$ ,  $54.9\%$ ,  $56.8\%$  accuracy on zero-shot AO, five-shot AO, and five-shot COT settings respectively, implying the difficulty of C-EVAL HARD. Interestingly, chain-of-thought prompting improves GPT-4 slightly on these extremely challenging subjects. Indeed, only GPT-4, ChatGPT, and Claude manage to make meaningful progress – improving by at least 10 points – over a random baseline. Our results further confirm that some critical distinction among LLMs comes out when the tasks become complex enough. We underscore the importance of evaluating LLMs in such challenging settings, as current LLM development goes beyond creating a casual chatbot – it involves the development of complex systems or agents capable of interacting with various data types, receiving feedback, reasoning and using tools, and even performing actions (Mialon et al., 2023).
Results on the validation split: Since we do not publicly release the labels for our test split, we provide the average accuracy on the validation split as a reference for developers. The validation split comprises a total of 1346 questions, with each subject contributing fewer than 30 validation questions on average. Therefore, tracking accuracy on a specific subject may not yield significant insights. Instead, we report the average answer-only accuracy across all subjects in Table 7. The average validation accuracy closely mirrors the average test accuracy as presented in Table 3 and Table 4. Additionally, the model ranking on the validation split broadly corresponds to that on the test split. These observations suggest that developers may use the average validation accuracy as a good indicator for expedited development processes.
Model	Zero-shot	Five-shot
GPT-4	66.7	69.9
Claude-v1.3	52.1	55.5
ChatGPT	50.8	53.5
Bloomz-mt	45.9	38.0
GLM-130B	44.2	40.8
Claude-instant-v1.0	43.2	47.4
ChatGLM-6B	39.7	37.1
LLaMA-65B	38.6	39.8
MOSS	35.1	28.9
Chinese-Alpaca-13B	32.0	27.2
Chinese-LLaMA-13B	29.4	33.1
Table 7: Average accuracy on the validation split in the answer-only setting.
English benchmarks: Traditional English benchmarks mainly focus on assessing certain abilities of models on a single task or a single type of tasks, such as natural language understanding (NLU, Wang et al. (2019)), reading comprehension (Rajpurkar et al., 2018), machine translation (Bojar et al., 2014), and summarization (Hermann et al., 2015; Narayan et al., 2018). As a representative example, the GLUE benchmark (Wang et al., 2019) combines a collection of NLU tasks, and has witnessed superhuman model performance due to the burst of pretraining models such as BERT (Kenton & Toutanova, 2019) and GPT (Radford et al., 2019). In order to assess the capabilities of LLMs more comprehensively, recent benchmarks have cast light on the broader knowledge and advanced abilities. The MMLU benchmark (Hendrycks et al., 2021a) provides multi-domain and multi-task evaluation collected from real-world examinations and books. LLMs' performance on MMLU fluctuates around random-chance accuracy until they reach the scale of GPT-3. The BIG-bench benchmark (Srivastava et al., 2022) consists of 204 diverse tasks, some of which are considered to be beyond the capabilities of current LLMs. The HELM benchmark (Liang et al., 2022) aggregates 42 different tasks and evaluates LLMs with 7 metrics ranging from accuracy to robustness.
Chinese benchmarks: Despite the flourishing of English benchmark, language abilities in Chinese language environment remain under-developed. The CLUE benchmark (Xu et al., 2020) is the first large-scale Chinese NLU benchmark, and still serves as the most widely-used and best available Chinese benchmark. Recently, the AGIEval benchmark (Zhong et al., 2023) contains data from the Chinese College Entrance Exam, Chinese lawyer qualification test and Chinese civil service examination. The MMCU benchmark (Zeng, 2023) consists of tests from four major domains including medicine, law, psychology and education, which are also collected from Chinese College Entrance Exam, qualification test as well as university examinations. Compared to AGIEval and MMCU, C-EVAL (1) has a broader coverage of domains (§2.2), (2) features four different levels of difficulty – particularly, the C-EVAL HARD benchmark is the first Chinese benchmark to provide sophisticated reasoning problems, and (3) makes an effort to mitigate data leakage – our questions mostly come from mock exams as PDF or Microsoft Word documents that are further processed by us, while AGIEval and MMCU collects the exact questions from past national exams in China.
We believe that the evaluation of LLMs should transcend the scope of casual conversational bots, guiding developers in preparing LLMs to function in more complex scenarios. This was the primary motivation behind the creation of C-EVAL, a challenging evaluation suite. We hope that C-EVAL along with C-EVAL HARD have made important progress on this direction particularly in a Chinese context. We also note that, C-EVAL, along with all the English-language benchmarks, is far from perfect for LLM evaluation. There are many other abilities such as reasoning over and calling APIs, as well as multiple aspects that extend beyond accuracy, including safety, bias, and robustness. We leave further exploration on their evaluation for future work.
We thank the anonymous reviewers for their comments. Yuzhuo Bai is supported by the National Key R&D Program of China (No. 2020AAA0106502) and Institute Guo Qiang at Tsinghua University.
Hui Zeng. Measuring massive multitask chinese understanding. arXiv preprint arXiv:2304.12986, 2023.
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. Agieval: A human-centric benchmark for evaluating foundation models. arXiv preprint arXiv:2304.06364, 2023.
Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=WZH7099tgfM.
A Author Contributions
Data collection, annotation, and initial validation: Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Yikai Zhang, and Jiayi Lei collected the first version of the data, including some necessary manual annotation from PDF or Word documents. They then cross-validated the collected data. Junlei Zhang prompted GPT-4 to generate the explanation data, and then Yuzhen Huang, Yuzhuo Bai, and Junxian He manually revised the explanations.
Data processing: Yuzhen Huang did most of the data processing job including a lot of manual revisions, such as fixing typos/incorrect formats of the questions from PDF documents and ensuring all the  $\mathrm{LATEX}$  notations can be correctly compiled. Jinghan Zhang did the data dedduplication.
Evaluation: Yuzhen Huang tested ChatGPT and MOSS; Yuzhuo Bai tested ChatGLM; Junlei Zhang tested GLM-130B; Chuancheng Lv tested Bloomz-mt, LLaMA, Chinese-LLaMA, and Chinese-Alpaca; Yao Fu tested GPT-4 and the Claude model variants.
Website and submission system: Zhihao Zhu built the website and the online submission system.
Paper Writing: Yuzhen Huang, Yuzhuo Bai, and Junxian He wrote the main content of this paper, while other authors helped proofread.
Advising: Yao Fu, Maosong Sun, and Junxian He take advisor roles in this project. Junxian He is the main advisor, initializing and organizing this project.
B Detailed Stats of C-EVAL
Table 8 lists all the C-EVAL tasks and their broader categories, as well as the number of questions included in each task.
C Explanation Data Generation
We show an example of the automatic COT explanation generation via GPT-4 in Figure 5. We use five human-written question-explanation pairs to prompt GPT-4 to generate explanation giving question and its correct answer. The generated explanations are further revised manually to obtain the final explanations.
D Evaluation Prompts
We show the evaluation prompts of answer-only and chain-of-thought test in Figure 6 and Figure 7 respectively.
E Details of the models being evaluated
ChatGPT and GPT-4 are GPT-series models that are enhanced to follow human instructions and be more helpful, harmless and honest using Reinforcement Learning from Human Feedback. GPT-4 additionally enables image inputs and goes through well-designed post-training alignment process, as well as having a larger scale than most of the existing model. GPT-4 achieves human-level performance on various benchmark, and even scored to be the top  $10\%$  in some simulated exams.
Claude is the latest Anthropic-series LLM that also focuses on human intention alignment. Applying the constitutional AI approach(Bai et al., 2022), Claude manages to be both helpful and trustworthy. Claude-instant is the lighter vessel with less cost and faster inference of Claude.
BLOOMZ-mt (Muennighoff et al., 2022) is created by combining multitask prompted finetuning to the pretrained multilingual BLOOM model(Scao et al., 2022), using not only English prompts but also machine-translated prompts to match the language of multilingual tasks, and are found to be capable at task- and language-agnostic generalization. We evaluate the 176B version in our experiment.
Subject	Category	# Questions
Advanced Mathematics (高等数学)	STEM	197
College Chemistry (大学化学)	STEM	253
College Physics (大学物理)	STEM	200
College Programming (大学编程)	STEM	384
Computer Architecture (计算机组成)	STEM	219
Computer Network (计算机网络)	STEM	195
Discrete Mathematics (离散数学)	STEM	174
Electrical Engineer (注册电气工程师)	STEM	381
High School Biology (高中生物)	STEM	199
High School Chemistry (高中化学)	STEM	196
High School Mathematics (高中数学)	STEM	189
High School Physics (高中物理)	STEM	199
Metrology Engineer (注册计量师)	STEM	248
Middle School Biology (初中生物)	STEM	218
Middle School Chemistry (初中化学)	STEM	210
Middle School Mathematics (初中数学)	STEM	201
Middle School Physics (初中物理)	STEM	202
Operating System (操作系统)	STEM	203
Probability and Statistics (概率统计)	STEM	189
Veterinary Medicine (兽医学)	STEM	238
Business Administration (工商管理)	Social Science	339
College Economics (大学经济学)	Social Science	557
Education Science (教育学)	Social Science	304
High School Geography (高中地理)	Social Science	202
High School Politics (高中政治)	Social Science	200
Mao Zedong Thought (毛泽东思想和中国特色社会主义理论体系概论)	Social Science	248
Marxism (马克思主义基本原理)	Social Science	203
Middle School Geography (初中地理)	Social Science	125
Middle School Politics (初中政治)	Social Science	219
Teacher Qualification (教师资格)	Social Science	448
Art Studies (艺术学)	Humanities	336
Chinese Language and Literature (中国语言文学)	Humanities	237
High School Chinese (高中语文)	Humanities	202
High School History (高中历史)	Humanities	207
Ideological and Moral Cultivation (思想道德修养与法律基础)	Humanities	196
Law (法学)	Humanities	250
Legal Professional (法律职业资格)	Humanities	243
Logic (逻辑学)	Humanities	231
Middle School History (初中历史)	Humanities	234
Modern Chinese History (近代史纲要)	Humanities	240
Professional Tour Guide (导游资格)	Humanities	300
Accountant (注册会计师)	Other	497
Basic Medicine (基础医学)	Other	199
Civil Servant (公务员)	Other	481
Clinical Medicine (临床医学)	Other	227
Environmental Impact Assessment Engineer (环境影响评价工程师)	Other	317
Fire Engineer (注册消防工程师)	Other	318
Physician (医师资格)	Other	497
Plant Protection (植物保护)	Other	226
Sports Science (体育学)	Other	204
Tax Accountant (税务师)	Other	497
Urban and Rural Planner (注册城乡规划师)	Other	469
Table 8: Summary of all 52 subjects.
LLaMA (Touvron et al., 2023) is a Transformer-architecture LLM that is trained on a mixture of several open sources. Applying several improvements on vanilla Transformers used by previous LLMs, and optimized to improve the training efficiency, LLaMA shows strong language abilities and can surpass models having  $10\mathrm{x}$  larger parameters than LLaMA. We evaluate the LLaMA-65B version in our experiment.
GLM-130B and ChatGLM-6B are based on the General language model (GLM) structure that gain benefits from its bidirectional attention advantage. By using alterable number and length of
... [5-shot examples] ...
以下是中国关于初中物理的单项选择题，请根据以下题目和答案，生成一段正确，合理的答案解析。
The following are multiple-choice questions about middle school physics in China. Please generate a correct and reasonable explanation based on the following questions and answers.
下列属于可再生能源的是
Which of the following is a renewable energy source?
A. 石油
B.煤炭
C. 核燃料
D. 太阳能
A. petro
eum B. coal
C. nuclear fuel
D. solar energy
答案：D
Answer: D
你生成的答案解析是什么？
What is your explanation generated?
答案解析: 让我们一步一步思考,
1. (A) 石油是一种化石能源，是从地下深处开采出来的，不属于可再生能源。
2. (B) 煤炭也是一种化石能源，是从地下深处开采出来的，不属于可再生能源。
3. (C) 核燃料是指用于核反应堆中的燃料，包括铀、钚等，虽然可以通过核反应产生能量，但是核燃料是有限的，不属于可再生能源。
4. (D) 太阳能是指从太阳辐射中获取的能量，是一种可再生能源，因为太阳每天都会升起，不会因为使用而枯竭。
所以答案是D。
Explanation: Let's think step by step,
1. (A) Petroleum is a fossil fuel extracted from deep underground and is not a renewable energy source.
2. (B) Coal is also a fossil fuel extracted from deep underground and is not a renewable energy source.
3. (C) Nuclear fuel refers to the fuel used in nuclear reactors, including uranium, plutonium, etc. Although energy can be generated through nuclear reactions, nuclear fuel is limited and is not a renewable energy source.
4. (D) Solar energy refers to the energy obtained from solar radiation and is a renewable energy source, because the sun rises every day and will not be exhausted due to use.
Therefore, the answer is C.
Figure 5: An example of generating explanations via GPT-4. The red text is the autocompleted response from model, while the preceding text is the user-inputted prompt. We indicate English translation below the corresponding Chinese text for each paragraph.
以下是中国关于临床医学的单项选择题，请选出其中的正确答案。
The following are multiple-choice questions about clinical medicine in China. Please select the correct answer.
男性，65岁，昏迷10小时。血气分析：pH7.26， $\mathrm{PaCO}_2$ 82mmHg， $\mathrm{PaO}_2$ 45mmHg。下列哪项处理不适宜
A 65-year-old male patient has been in a coma for 10 hours. Blood gas test shows a pH of 7.26,  $\mathrm{PaCO}_2$ $82\mathrm{mmHg}$  and  $\mathrm{PaO}_2$ $45\mathrm{mmHg}$ . Which of the following treatments is not appropriate?
A. 机械通气
B. 呼吸兴奋剂
C. 支持疗法
D. 高浓度给氧
A. mechanical ventilation B. respiratory stimulants C. supportive therapy D. high concentration oxygen therapy
硅尘可导致硅沉着病，其中致病力最强的硅尘颗粒直径为
Silicon dust can cause silicosis, and the most pathogenic silicon dust particles have a diameter of
A.  $>5 \mu \mathrm{m}$
B.  $4 \sim 5 \mu \mathrm{m}$
C.  $1 \sim 2 \mu \mathrm{m}$
D.  $2 \sim 3 / \mu \mathrm{m}$
答案：C
Answer: C
Figure 6: An example of few-shot evaluation in answer-only scenarios, while zero-shot evaluation is similar by removing the exemplars. The red text is the autocompleted response from model, while the preceding text is the inputted prompt. We indicate English translation below the corresponding Chinese text.
blanks, GLM can adapt to various tasks. GLM-130B is a bilingual pre-trained GLM that utilizes self-supervised learning and multitask instruction pretraining. GLM-130B also realized INT4 quantization
以下是中国关于计算机组成考试的单项选择题，请选出其中的正确答案。
The following are multiple-choice questions about computer architecture in China. Please select the correct answer.
设寄存器R的内容  $(\mathrm{R}) = 1500\mathrm{H}$  ，内存单元1500H的内容为2500H，内存单元2500H的内容为3500H，PC的值为4500H，采用相对寻址方式，有效地址为2500H的操作数是
If the content of register R is set to 1500H, the content of memory address 1500H is 2500H, the content of memory address 2500H is 3500H, and the value of the program counter (PC) is 4500H, using the relative addressing mode, the operand at the effective address of 2500H is
A. 1500H
B.  ${2500}\mathrm{H}$
C. 3500H
D.  ${4500}\mathrm{H}$
答案：让我们一步一步思考，
1. 根据题目中的信息，内存单元1500H的内容为2500H，内存单元2500H的内容为3500H，表示当前指令的地址为2500H- 1500H=1000H。
2. 根据相对寻址方式，有效地址为  $2500 \mathrm{H}$  的操作数需要加上当前指令地址  $1000 \mathrm{H}$  才能得到实际的操作数地址，即  $3500 \mathrm{H}$ 。
所以答案是C。
Answer: Let's think step by step,
1. According to the information given in the question, the content of memory unit  $1500\mathrm{H}$  is  $2500\mathrm{H}$ , and the content of memory unit  $2500\mathrm{H}$  is  $3500\mathrm{H}$ . This indicates that the current instruction address is  $2500\mathrm{H}-1500\mathrm{H}=1000\mathrm{H}$ .
2. According to the relative addressing mode, in order to obtain the actual operand address of the effective address with the value of  $2500\mathrm{H}$ , the current instruction address of  $1000\mathrm{H}$  needs to be added to it, resulting in the actual operand address of  $3500\mathrm{H}$ .
Figure 7: An example of few-shot evaluation in chain-of-thought scenarios, while zero-shot evaluation is similar by removing the exemplars. The red text is the autocompleted response from model, while the preceding text is the inputted prompt. We indicate English translation below the corresponding Chinese text.
with little to no quality degradation that significantly accelerate its inference efficiency. ChatGLM-6B is a lightweight conversational version of the GLM family that has been specially optimized for Chinese contexts. ChatGLM-6B also applies quantization so it can be deployed with a consumer-grade graphic memory requirement as little as 6GB. We evaluate on the fp16 settings for both two models in our experiment.
Chinese-LLaMA is an adaptation of original LLaMA into Chinese language environments. Chinese-LLaMA expands the original LLaMA by adding 20K Chinese tokens into its vocabulary, and is secondarily pre-trained and instruction fine-tuned on Chinese data. We evaluate Chinese-LLaMA-13B in our experiment, the largest Chinese-LLaMA variant.
Chinese-Alpaca is based on the Chinese-LLaMA checkpoint that is further tuned on Chinese instruction tuning data. We evaluate Chinese-Alpaca-13B in our experiment, the largest Chinese-Alpaca variant.
MOSS is the first open-source Chinese LLM that matches ChatGPT on both the training scale and alignment techniques. MOSS is initialized with CodeGen(Nijkamp et al., 2022), being pretrained on 100B Chinese tokens and 20B English tokens, and has further integrated supervised fine-tuning and preference model, as well as plugin augmentation, but not all the version are publicly available. We evaluate the moss-moon-003-sft version in our experiment.
F Breakdown of Model Performance
Table 9 and Table 10 show the detailed breakdown of accuracy per subject of four representative models in zero- and five-shot settings respectively, while we refer the readers to our website leaderboard https://cevalbenchmark.com/static/leaderboard.html for a detailed breakdown of results for all models.