SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Subject	GPT-4	ChatGPT	GLM-130B	Claude-instant-v1.0
Advanced Mathematics	48.6	38.2	28.9	36.4
College Chemistry	54.0	36.2	29.0	25.9
College Physics	50.6	34.1	29.5	33.5
College Programming	72.2	56.1	33.3	40.6
Computer Architecture	73.1	62.7	38.3	42.5
Computer Network	74.9	60.8	36.3	44.4
Discrete Mathematics	62.1	37.9	34.6	28.8
Electrical Engineer	48.7	38.9	31.6	32.7
High School Biology	69.1	49.1	37.7	40.0
High School Chemistry	55.2	45.3	36.0	39.5
High School Mathematics	41.0	28.3	34.9	24.7
High School Physics	65.1	36.6	26.9	40.0
Metrology Engineer	68.0	58.4	49.8	48.4
Middle School Biology	88.0	68.2	51.6	46.4
Middle School Chemistry	82.7	63.2	45.9	47.0
Middle School Mathematics	65.5	40.1	32.2	34.5
Middle School Physics	80.9	61.2	41.6	48.9
Operating System	79.3	70.4	42.5	46.4
Probability and Statistics	50.0	36.7	25.9	27.7
Veterinary Medicine	75.7	56.7	47.6	43.8
Business Administration	63.1	48.2	43.2	40.2
College Economics	67.6	47.3	38.6	40.8
Education Science	67.4	54.8	52.6	43.0
High School Geography	73.6	55.6	51.1	42.7
High School Politics	74.4	50.0	45.5	37.5
Mao Zedong Thought	74.9	56.6	67.6	49.3
Marxism	77.7	70.9	69.3	62.0
Middle School Geography	78.7	57.4	58.3	49.1
Middle School Politics	85.5	69.4	61.1	57.0
Teacher Qualification	84.5	69.9	70.7	54.9
Art Studies	57.4	47.7	49.3	33.2
Chinese Language and Literature	61.2	51.7	46.4	35.9
High School Chinese	38.8	37.6	25.8	31.5
High School History	66.5	48.4	48.4	41.2
Ideological and Moral Cultivation	73.8	65.1	59.3	58.1
Law	56.1	42.1	37.6	34.8
Legal Professional	55.3	40.5	35.8	36.7
Logic	62.3	38.7	34.3	36.3
Middle School History	83.6	65.2	65.2	48.8
Modern Chinese History	63.7	46.2	59.9	42.5
Professional Tour Guide	68.8	53.4	63.2	35.0
Accountant	63.4	46.3	39.7	38.1
Basic Medicine	78.9	61.1	46.9	39.4
Civil Servant	62.2	45.5	45.2	36.1
Clinical Medicine	70.5	51.0	43.5	38.0
Environmental Impact Assessment Engineer	59.4	52.0	45.6	41.3
Fire Engineer	45.0	41.5	35.8	29.4
Physician	75.6	59.4	44.7	41.5
Plant Protection	71.4	56.3	49.7	48.2
Sports Science	70.0	53.9	44.4	40.6
Tax Accountant	56.0	38.4	33.9	38.1
Urban and Rural Planner	59.1	49.0	43.5	38.0
Table 9: Zero-shot answer only accuracy per subject.
G Option Bias
The distribution of the correct answer is shown in the table 11. We admit that there are fluctuations in the proportion of different options. However, these fluctuations are relatively small around a random  $25\%$  level and similar to MMLU.
To verify if the option bias exists on models, we permuted the order of the choices in the questions and randomly selected 5 distinct permutations of choices for each question. Then we evaluate ChatGPT, ChatGLM-6B and ChatGLM2-6B (THUDM, 2023b) on the 5 different permutations of C-EVAL. The zero-shot results in the answer-only setting are shown in table 12. We report the average accuracy
Table 10: Five-shot accuracy per subject. We report both the answer only (AO) and chain-of-thought (COT) accuracy in an AO / COT format.
Option	C-Eval	MMLU
A	22.9%	23.1%
B	26.0%	24.7%
C	26.4%	25.5%
D	24.7%	26.7%
Table 11: The distribution of the correct answer.
across 5 permutations and the variance of the overall accuracy. The results imply that the variance across different permutations is relatively small.
Model	STEM	Social Science	Humanities	Other	C-Eval Hard	Average	Var
ChatGPT	48.1	58.1	48.6	49.9	36.3	50.5	0.6
ChatGLM-6B	33.4	47.5	41.9	37.5	28.7	38.8	0.7
ChatGLM2-6B	38.1	58.3	49.6	45.2	29.3	45.9	0.2
Table 12: Zero-shot average accuracy  $(\%)$  in answer-only setting. We report the average accuracy over 5 permutation within each category and overall accuracy. "Average" column indicates the average accuracy over 5 permutation. "Var" column indicates the variance of the overall accuracy.
H Compute and Resources Used for Evaluation
During our experiments to evaluate different LLMs on C-EVAL, we utilize a cluster with 8 A100-80GB GPUs to run the inference for models with released weights, such resources are required due to deploying three large models - Bloomz-mt (176B), LLaMA-65B, and GLM-130B. In most cases the inference on C-EVAL is finished within one day. For models with API access, we just run the inference with CPUs which finishes mostly within one day as well.