Title: EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries

Benchmark: EHRNoteQA
Summary: EHRNoteQA is a benchmark designed to evaluate LLM performance in answering clinical questions grounded in electronic health records (EHRs). The benchmark includes questions hand-reviewed by physicians across four clinical specialties. It evaluates LLMs’ capabilities under different reasoning requirements, providing both answer annotations and evidence spans.

Contribution: Both the 962 questions and LLM performance on the benchmark are exhaustively checked by human physicians (three humans per question). Noteworthily deep analysis of suitability as proxy for real physician performance, both theoretically and via additional human checking of LLM benchmark answers

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: clinical reasoning, question answering

Phenomenon defined by authors: Yes
Phenomenon definition: "benchmark to evaluate LLMs in real-world clinical scenarios for answering clinicians’ questions regarding patient discharge summaries"

Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Given a discharge summary, the model must answer a physician-written question based on its content, optionally providing an evidence span.

Task item definition: One task includes a discharge note, a clinical question, and expected answer type (e.g., yes/no, short span, or long answer with evidence) 

Task definition detail: Questions vary in complexity and type. Each item has an annotated answer and evidence span(s), and task types include retrieval-augmented and vanilla QA.

Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language), LLM-generated task examples (e.g. Filtered from responses to a prompt)
Task source detail: Discharge summaries come from MIMIC-III; all questions LLM-generated and adapted/validated by three physicians.

Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Although EHRs are real, questions are constructed to simulate realistic physician questions.

Dataset size: 962
Dataset size detail: Small because all questions are hand-verified
Dataset metadata available: Yes
Dataset metadata detail: note ID, specialty, question type, answer type, reasoning type

Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number), Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Metric definition: Exact Match (accuracy, F1, precision, recall), Human ratings (text quality, preference, NOT manual scoring of other metrics), LLM-as-a-Judge (text quality, preferences, NOT extracting answers for other metrics)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: By question type (yes/no, factoid, reasoning), specialty, and augmentation setting

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: Yes
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Particularly, notably detailed analysis of suitability as proxy for real clinician evaluation, using another human evaluation of LLM answers.

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine
Other notes: The best paper in my batch for ecological validity/comparison to human baseline