Title: Benchmarking Large Language Models on CMExam: A Comprehensive Chinese Medical Exam Dataset

Benchmark: CMExam
Summary: CMExam is a benchmark based on Chinese medical licensing exam questions, to evaluate LLMs on domain-specific knowledge and reasoning. It benchmarks several general and medical domain LLMs on multiple-choice and explanation tasks.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Medical knowledge, clinical reasoning

Phenomenon defined by authors: Yes
Phenomenon definition: "Ability to understand and reason with medical knowledge to correctly answer Chinese medical licensing exam questions"

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Tasks are Chinese medical exam questions in multiple choice formats with open-ended explanations asked for, but not evaluated systematically

Task item definition: A medical question with multiple-choice answers 

Task definition detail: While open-ended reasoning is used and analyzed, all questions are actually MCQs & all scoring only takes MCQ performance into account

Task source: Human exam questions (e.g. GRE questions)
Task source detail: From exam repositories; explanations extracted from public prep materials/sites

Task ecology: Representative task (e.g. answering medical licensing exam questions)
Dataset size: 68119
Dataset metadata available: Yes
Dataset metadata detail: Disease category, department, discipline, competency, difficulty

Sampling method: Convenience sample (creators found a set of tasks that was readily accessible), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Disease group, department, discipline, competency, difficulty

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Compared to human performance

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine