Title: MEDCALC-BENCH: Evaluating Large Language Models for Medical Calculations

Benchmark: MEDCALC-BENCH
Summary: MedCalc-Bench tests LLMs on 55 tasks emulating real-world "medical calculators" using patient notes; tasks require knowledge recall, attribute extraction, and computation.

Contribution: Task format is quite unique; requires quantitative reasoning for emulation of existant "medical calculators" (simple programs where formulae are filled by user with numbers), extraction of numbers to use from text and relevant medical knowledge

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Quantitative medical reasoning, arithmetic computation, medical domain knowledge

Phenomenon defined by authors: Yes
Phenomenon definition: Medical calculators are online tools used by clinicians to perform various calculations. The tested ones are all listed as "popular" on MDCalc, the most commonly used online medical calculator website by clinicians

Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Widely-agreed
Task definition: Compute a clinical value (e.g. risk score, lab value) from a patient note by emulating the appropriate calculator

Task item definition: In: patient note + question (e.g., “What is the patient’s LDL?”), out: ground truth answer + explanation

Task definition detail: 55 calculators, input can include 1–31 parameters

Task source: Real task examples (e.g. GitHub issues), Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Curated from public patient notes (PMC etc.) + GPT-4 extraction + manual synthesis

Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Medical calculators are used in practice, input via free-text simulates unstructured notes from real clinicians
Dataset size: 1047
Dataset metadata available: Yes
Dataset metadata detail: Calculator type, attribute types, ground truth, explanations

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric definition detail: ±5% for numeric, exact match for rule/date; GPT-4 error categorization

Metric aggregation: Simple Mean
Metric statistics: Binomial standard error reported

Metric subscores: Yes
Metric subscores detail: By type of calculation: Lab, Physical, Date, Dosage, Risk, Severity, Diagnosis

Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: Yes
Human baseline: No
Authors discuss validity: Yes
Author validity detail: GPT-4 error classification (validated at 89% on sample)

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine