Title: CliMedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models in Clinical Scenarios

Benchmark: CliMedBench
Summary: CliMedBench is a Chinese clinical medical benchmark with 33.7k QA items across 14 core scenarios derived from real-world medical records and exams, measuring LLMs’ clinical reasoning and language abilities. It includes evaluations of 11 LLMs.

Contribution: Proposes a novel adaptive testing method (agent-based CAT) grounded in Item Response Theory to alleviate cost concerns, but stresses the "real" benchmark is still just evaluating all models across all questions

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: medical reasoning, diagnostic accuracy, clinical QA

Phenomenon defined by authors: Yes
Phenomenon definition: Defines clinical reasoning and related capabilities such as hallucination resistance, information retrieval, and instruction via the "personas" of real medical practitioners eg. radiographer; pharmacist

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Purpose notes: The selected "personas" feel a bit random: doctor, med student, patient, radiologist, pharmacist. Surely there are other people in a hospital.
Task definition: Multiple-choice, sequencing, and open-ended questions based on real-world Chinese clinical scenarios.

Task item definition: One task item can be a MCQ from an EHR, a sequencing task like reordering surgical steps, or open-ended responses like writing discharge summaries

Task source: Human exam questions (e.g. GRE questions), Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Combination of electronic health records, exam data, expert rephrasing, and LLM generation with filtering

Task ecology: Partial real task (e.g. answering medical questions collected from real people), Representative task (e.g. answering medical licensing exam questions)
Task ecology detail: Evaluation of actual clinical decisions/tasks, but in a segmented way

Dataset size: 33735
Dataset metadata available: Yes
Dataset metadata detail: clinical role, task type, scenario ID, source (e.g., EHR, exam), evaluation axes if relevant

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code), Sequencing
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Mix of automatic metrics and expert review
Metric aggregation: Simple Mean
Metric statistics: Simple mean
Metric subscores: Yes
Metric subscores detail: 14 clinical scenarios and 7 evaluation axes

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: No
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Medical professionals rated benchmark scenarios and Spearman correlation to another benchmark (MedBench) was computed

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine