Title: Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark

Benchmark: ClinicBench
Summary: ClinicBench is a benchmark for evaluating large language models (LLMs) on complex clinical decision-making tasks. It includes 17 datasets, 11 tasks, and evaluates 22 LLMs under both zero- and few-shot settings using automatic and human evaluations.

Contribution: Novel datasets focus on open-ended decision-making, long document processing, and new drug analysis, as opposed to MCQ Q&A

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: clinical decision-making, clinical reasoning

Phenomenon defined by authors: Yes
Phenomenon definition: "comprehensive evaluation of LLMs in the clinic", defined as “reasoning, generation, and understanding” in clinical scenarios like treatment recommendation, patient education, and drug interaction prediction

Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: 11 tasks in clinical reasoning, generation, and understanding, using 17 datasets with multiple input/output types, including MCQs and open-ended responses.

Task item definition: Varies by task; e.g. one item may be a multiple-choice question about patient treatment based on a referral letter (Referral QA), or generating plain language discharge instructions from clinical notes (Patient Education).

Task definition detail: Six datasets are newly created to reflect real-world clinical challenges

Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: More than half just other benchmarks - other half adapted from various sources with LLM-aided question crafting

Task ecology: Partial real task (e.g. answering medical questions collected from real people), Constructed task (e.g. predicting medical diagnoses from clinicians' notes)
Task ecology detail: Some tasks simulate real-world decisions (e.g., treatment recommendations) but outputs aren't used in practice.

Dataset size: 20000
Dataset size detail: Only listed as >20000
Dataset metadata available: Yes
Dataset metadata detail: Dataset sources, sizes, task types, and evaluation metrics

Sampling method: Random sample (creators defined a task space and sampled from it), Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Accuracy for QA, ROUGE-L for generation, F1 for NER, and human ratings for factuality, safety, completeness, preference

Metric aggregation: Simple Mean
Metric statistics: Averages and win/tie percentages in human eval; no advanced statistics reported

Metric subscores: Yes
Metric subscores detail: Separate scores for each task and metric (e.g., factuality vs completeness)

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Discusses face validity, few-shot effects, human expert comparison, and fine-tuning types

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine