Title: MEDEVAL: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation
Benchmark: MEDEVAL
Summary: MEDEVAL is a multi-level, multi-task, and multi-domain medical benchmark. The paper collects data from several healthcare systems and annotations from experts. It evaluates generic and domain-specific language models under zero-shot and fine-tuned settings.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: performance on medical examination healthcare tasks
Phenomenon defined by authors: No
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Not defined
Task definition: They identify two types of tasks (NLU task and NLG task) at two levels (sentence-level and document-level). Sentence level, NLU: identifying diagnostic properties , Sentence level, NLG: sentence disambiguation; Document level, NLU: categorizing reports into specific
diagnostic codes, Document level, NLG: medical summarization.
Task item definition: I couldn't find the dataset; the link they included is broken. From their description in the appendix, a single item varies based on the task, but primarily consists of one of the four tasks, level name (sentence or report), and task content.
Task source detail: They adapt existing medical datasets (that haven't been used as benchmarks) and add new expert annotations
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Dataset size: 8,801 (I did the math here, they provide the ratios)
Dataset size notes: train/validate/test of 7:2:1
Dataset metadata available: Yes
Dataset metadata detail: body parts
Sampling method: Random sample (creators defined a task space and sampled from it)
Response format: Multiple choice, Free response (e.g. summary paragraph, executable code)
Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), They use an ambiguity classifier as well from previous work
Metric aggregation: Simple Mean
Metric statistics: simple mean/sum
Metric subscores: Yes
Metric subscores detail: Body parts (chest, foot, ankle)
Metric access: Outputs alone
Few-shot evaluation: Yes
Authorship: Academia
Benchmark availability: No, link is broken
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: No
Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine