CheXGenBench: A Unified Benchmark For Fidelity, Privacy and Utility of
  Synthetic Chest Radiographs
Authors: name: Raman Dutt; name: Pedro Sanchez; name: Yongchen Yao; name: Steven McDonagh; name: Sotirios A. Tsaftaris; name: Timothy Hospedales
Published: 2025-05-15 16:59:17

Abstract: We introduce CheXGenBench, a rigorous and multifaceted evaluation framework
for synthetic chest radiograph generation that simultaneously assesses
fidelity, privacy risks, and clinical utility across state-of-the-art
text-to-image generative models. Despite rapid advancements in generative AI
for real-world imagery, medical domain evaluations have been hindered by
methodological inconsistencies, outdated architectural comparisons, and
disconnected assessment criteria that rarely address the practical clinical
value of synthetic samples. CheXGenBench overcomes these limitations through
standardised data partitioning and a unified evaluation protocol comprising
over 20 quantitative metrics that systematically analyse generation quality,
potential privacy vulnerabilities, and downstream clinical applicability across
11 leading text-to-image architectures. Our results reveal critical
inefficiencies in the existing evaluation protocols, particularly in assessing
generative fidelity, leading to inconsistent and uninformative comparisons. Our
framework establishes a standardised benchmark for the medical AI community,
enabling objective and reproducible comparisons while facilitating seamless
integration of both existing and future generative models. Additionally, we
release a high-quality, synthetic dataset, SynthCheX-75K, comprising 75K
radiographs generated by the top-performing model (Sana 0.6B) in our benchmark
to support further research in this critical domain. Through CheXGenBench, we
establish a new state-of-the-art and release our framework, models, and
SynthCheX-75K dataset at https://raman1121.github.io/CheXGenBench/