Title: C-EVAL: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models
Benchmark: C-EVAL
Summary: The paper introduces C-EVAL evaluation suite for assessing advanced knowledge and reasoning abilities of foundation models in Chinese, It spans four difficulty levels and 52 disciplines. It also introduces C-EVAL HARD a subset of challenging subjects that require advanced reasoning.
Phenomenon type: General Capability (A broadly useful ability, which could be relevant to multiple applications)
Target phenomenon: Knowledge and reasoning in Mandarin Chinese and on questions situated in the Chinese context
Phenomenon defined by authors: No
Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Multiple choice questions from real-world human exams in China at different difficultly levels (e.g., high school, college) and different disciplines (e.g., STEM, humanities). 
Task item definition: An MCQ question with four possible answers.
Task source: Human exam questions (e.g. GRE questions)
Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 12342
Dataset size notes: Dev: 260, Valid: 1346
Dataset metadata available: Yes
Dataset metadata detail: topic area (e.g., STEM, humanities) and difficultly level (e.g., middle school)
Sampling method: Convenience sample (creators found a set of tasks that was readily accessible)
Split reviewed: Test, Train, Validation
Response format: Multiple choice
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric aggregation: Simple Mean
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: Subject/exam (and by extension difficulty) 
Metric access: Outputs alone
Authorship: Academia
Benchmark availability: Yes
Task face validity: They follow the lead of popular knowledge and reasoning benchmarks, so it's hard to say here. 
Metric face validity: Not sure about this. Compared to other similar benchmarks, yes. In general, probably not. 
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Knowledge / Cultural
New taxonomy: General Application / Cultural Knowledge