Title: FACTPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidence

Benchmark: FACTPICO
Summary: FACTPICO evaluates the factuality of LLM-generated plain language summaries of medical randomized controlled trials (RCTs). It features fine-grained expert annotations across five key dimensions and includes both human and LLM-generated rationales.

Contribution: Rich analysis, but incredibly unscalable - all evaluation was done by expert humans, which explains why they only tested 3 models

Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Factuality in plain language summarization

Phenomenon defined by authors: Yes
Phenomenon definition: Factuality is defined as accurate representation of critical trial elements (Population, Intervention, Comparator, Outcome - that's PICO) and their results, with particular focus on correctness of added explanatory content

Definition scope: Subset
Definition integrity: Single cohesive phenomenon
Definition integrity detail: Not applicable
Phenomenon contested: Contested
Task definition: Generate plain-language summaries of abstracts of randomized controlled trials.

Task item definition: Each item is an RCT abstract, the output always its summary.

Task source: Author-crafted task examples (e.g. hand-written examples, manual transformation of existing data into questions), Modified from another benchmark (e.g. translation into another language)
Task source detail: Based on RCT descriptions from Evidence Inference 2.0 dataset, sampled for exclusion of those which already have human summaries

Task ecology: Partial real task (e.g. answering medical questions collected from real people)
Task ecology detail: Summarizing technical trial outcomes into accessible language is a real-world medical communication need.

Dataset size: 115
Dataset metadata available: Yes
Dataset metadata detail: PICO annotations, human rationales.

Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically), Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Free response (e.g. summary paragraph, executable code)
Metric definition: Human ratings (text quality, preference, NOT manual scoring of other metrics)
Metric definition detail: Time-intensive human scoring of free-text responses across the PICO dimensions
Metric aggregation: Simple Mean
Metric statistics: Flesch Kincaid, Rouge-L, Kendalls, Spearmans

Metric subscores: Yes
Metric subscores detail: Subscores for each PICO element and Evidence Inference

Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: Authors analyze inter-annotator agreement, correlation with expert judgments, and rationale similarity.

Taxonomy: NLP / Summarization
New taxonomy: General Application / Summarization