Title: RaLEs: a Benchmark for Radiology Language Evaluations

Benchmark: RaLEs
Summary: RaLEs is a benchmark for evaluating language understanding and generation on radiology reports, which it defines as a distinct domain. It covers six datasets across four tasks. It compares general, biomedical, clinical, and radiology-specific models and includes new datasets and metrics.

Contribution: Includes a new procedure selection dataset and a de-identified NER dataset; evaluates calibration, uncertainty, and performance under label scarcity.

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Radiology-specific language understanding and generation

Phenomenon defined by authors: Yes
Phenomenon definition: Detailed, extensive definition of the "radiology report as a domain", wrt content, context, and availability.

Definition scope: Comprehensive
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: NER, relation extraction, summarization, and procedure selection tasks, using real radiology reports.

Task item definition: Each task item is a snippet of a radiology report used for classification, summarization, relation extraction or NER, with the corresponding label.

Task source: Real task examples (e.g. GitHub issues), Modified from another benchmark (e.g. translation into another language)
Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Tasks simulate actual clinical decisions like choosing procedures or summarizing findings.

Dataset size: 13657
Dataset size notes: 150864, 11413
Dataset metadata available: Yes
Dataset metadata detail: Institution origin, patient ID, anatomy/modality info

Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragraph, executable code), Structured response (e.g. valid JSON, API call alone)
Response format detail: NER is token-wise labeling, summarization and classification are full-sequence outputs

Metric definition: Exact Match (accuracy, F1, precision, recall), n-gram (BLEU, ROUGE, chrF), Distribution (perplexity, calibration, correlation)
Metric aggregation: Simple Mean
Metric subscores: Yes
Metric subscores detail: Task-specific and in/out-of-domain subscores

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Academia
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No
Results realism: The benchmark is itself realistic
Human baseline: No
Authors discuss validity: Yes
Author validity detail: Discusses domain generalization, label scarcity, calibration, uncertainty

Taxonomy: Medicine / 
New taxonomy: Specific Application / Medicine