# Trust but verify: Image-aware evaluation of radiology report generators

Sayeh Gholipour Picha, Dawood Al Chanti, Alice Caplier

► To cite this version:

Sayeh Gholipour Picha, Dawood Al Chanti, Alice Caplier. Trust but verify: Image-aware evaluation of radiology report generators. Machine Learning with Applications, 2026, 23, pp.100851. 10.1016/j.mlwa.2026.100851. hal-05476777

HAL Id: hal-05476777

https://hal.science/hal-05476777v1

Submitted on 26 Jan 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Trust but verify: Image-aware evaluation of radiology report generators

Sayeh Gholipour Picha ID*, Dawood Al Chanti, Alice Caplier

Université Grenoble Alpes, Grenoble, 38000, France

## ARTICLE INFO

**Keywords:**

Chest X-ray
Radiology report generation
Image-aware evaluation
Interpretability
Multimodal AI

## ABSTRACT

Large language and vision-language models have greatly advanced automated chest X-ray report generation (RRG), yet current evaluation practices remain largely text-based and detached from image evidence. Traditional machine translation metrics fail to determine whether generated findings are clinically correct or visually grounded, limiting their suitability for medical applications.

This study introduces a comprehensive, image-aware evaluation framework that integrates the VICCA (Visual Interpretation and Comprehension of Chest X-ray Anomalies) protocol with the domain-specific semantic metric MCSE (Medical Corpus Similarity Evaluation). VICCA combines visual grounding and text-guided image generation to assess visual-textual consistency, while MCSE measures semantic and factual fidelity through clinically meaningful entities, negations, and modifiers. Together, they provide a unified, semi-reference-free assessment of pathology-level accuracy, semantic coherence, and visual consistency.

Five representative RRG models, R2Gen, M2Trans, CXR-RePaiR, RGRG, and MedGemma, are benchmarked on 2461 MIMIC-CXR studies using a standardized pipeline. Results reveal systematic trade-offs: models with high pathology agreement often generate semantically weak or visually inconsistent reports, whereas textually fluent models may lack proper image grounding. By integrating clinical semantics and visual reliability within a single multimodal framework, VICCA establishes a robust paradigm for evaluating the trustworthiness and interpretability of AI-generated radiology reports.

## 1. Introduction

Large language models (LLMs) and vision-language models (VLMs) have catalyzed rapid advances in automated chest X-ray report generation (RRG), pushing AI deeper into healthcare. However, evaluation has not kept pace, assessment still relies primarily on machine-translation metrics such as BLEU (Papineni et al., 2002), BERTScore (Zhang et al., 2019), and COMET (Rei et al., 2020), which inadequately capture clinical correctness and offer no guarantee of image grounding. Text-only evaluation scores cannot determine whether a report's statements are truly supported by the underlying image evidence or whether they maintain structural consistency across anatomical regions, both of which are essential for safe clinical use. Moreover, AI-generated reports are prone to hallucinations, where clinically incorrect or speculative findings are introduced. Such errors can be particularly misleading in radiology: for instance, when two out of three predicted pathologies are correct, the third, erroneous one may still appear plausible to the reader and thus be mistakenly accepted as valid.

In clinical practice, omissions (False Negatives) and hallucinations (False Positives) have asymmetric risks. If a radiologist or an AI-generated reports misses a focal opacity (a white patch that should not

be there), the patient may leave the hospital with undiagnosed pneumonia. By the time they return, the infection could be septic. Conversely, hallucinating cardiomegaly or pleural effusion can trigger unnecessary downstream testing (e.g., echocardiography, CT) and increase patient burden and cost. These failure modes motivate evaluation that verifies whether generated statements are supported by image evidence rather than fluent text alone. A report that sounds professional but is not supported by image evidence is more dangerous than a poorly written report that is factually correct.

Our prior work, Visual interpretation and comprehension of chest X-ray anomalies in generated report (VICCA) (Gholipour Picha et al., 2025), illustrated in Fig. 1, introduced a reference-free and expert-independent framework to assess the alignment between visual and textual information in radiology reports. VICCA integrates two complementary modules: (1) a visual grounding component that localizes image regions corresponding to textual phrases, and (2) A diffusion model that synthesizes anatomically plausible CXR images conditioned on the report text, ensuring coherence with the original image. Together, these components enable an objective and image-aware assessment of

<footnote>* Corresponding author.</footnote>

<footnote>E-mail addresses: sayeh.gholipour-picha@grenoble-inp.fr (S. Gholipour Picha), dawood.al-chanti@grenoble-inp.fr (D. Al Chanti),
alice.caplier@grenoble-inp.fr (A. Caplier).</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Overview of the VICCA framework (Visual Interpretation and Comprehension of Chest X-ray Anomalies). VICCA integrates visual grounding and text-guided image generation to evaluate whether the content of an AI-generated radiology report is consistent with the underlying image.</figcaption></figure>

whether each textual statement is visually supported, while detecting potential hallucinations or omissions in AI-generated reports.

Unlike evaluation methods that rely on human experts, VICCA quantifies reliability through feature-level validation and medical plausibility checks. It leverages anatomical alignment and pathological detection metrics benchmarked against established, clinically verified models. As demonstrated in our previous work, VICCA can automatically determine whether a generated report is *ambiguous, incomplete, or inconsistent* with the underlying image. Using its conditional diffusion model, VICCA can also visualize textual ambiguity by generating CXRs that either closely match the original anatomy or diverge when the report content is incoherent or overly complex.

In the present study, we extend this framework by coupling VICCA with our previously proposed domain-aware medical corpus similarity evaluation (MCSE) metric (Picha et al., 2024). Unlike conventional lexical metrics, MCSE evaluates textual similarity by identifying clinically meaningful entities, negations, and modifiers, capturing the semantic and factual fidelity of medical narratives. Together with pathology-level agreement measures, this integration enables a comprehensive, cross-model evaluation of radiology report generators, revealing where models, and even their evaluation metrics, diverge or contradict one another.

We benchmark four state-of-the-art RRG models, R2Gen (Chen et al., 2020), M2Trans (Miura et al., 2021), CXR-RePaiR (Endo et al., 2021), and RGRG (Tanida et al., 2023), alongside one large-scale vision-language model, MedGemma (Sellergren et al., 2025), on 2461 studies from the MIMIC-CXR dataset (Johnson, Pollard, Berkowitz et al., 2019; Johnson, Pollard, Greenbaum et al., 2019). Reports are generated directly from CXR images without additional postprocessing, except for MedGemma, whose outputs required cleaning to isolate the clinically relevant “Findings” section due to the presence of unrelated text information.

Each model is evaluated across complementary dimensions of clinical validity, pathology label agreement using CheXbert (Smit et al., 2020), derived micro- and macro-F1 scores as well as Jaccard similarity; semantic fidelity using our MCSE metric; and visual-textual consistency through VICCA, which measures both visual grounding coverage and region-level reliability via Multi-Scale Structural Similarity Index (MS-SSIM).

Extensive prior research has demonstrated that traditional MT metrics such as BLEU, ROUGE, and even learned embedding-based measures are insufficient for evaluating AI-generated clinical reports such as Freitag et al. (2024) and Pauli et al. (2025). Our results reinforce this finding by showing that these metrics can misrepresent model performance when clinical accuracy and visual grounding are not explicitly measured. We argue that reliable evaluation of medical AI systems requires multimodal, reference-free approaches capable of assessing semantic fidelity, visual alignment, and structural consistency simultaneously.

Our analysis reveals several systematic trade-offs across the evaluated models. For instance, models that achieve high pathology-level

agreement often produce semantically weak or fragmented narratives that fail to convey a coherent clinical interpretation. Conversely, models generating textually fluent and semantically consistent reports sometimes lack strong image grounding, suggesting that linguistic quality does not necessarily imply visual correctness. Furthermore, region-level reliability, as measured by structural similarity between localized regions in real and generated images, remains deceptively stable across models with widely varying textual metrics. This observation highlights that surface-level or single-modality evaluations can obscure deeper inconsistencies between textual content and visual evidence.

This study makes the following key contributions:

* We present a comprehensive multimodal evaluation protocol for automated chest X-ray report generation, combining textual, semantic, and visual assessments into a unified framework.

* We benchmark five representative models, R2Gen, M2Trans, CXR-RePaiR, RGRG, and MedGemma, on a large subset of the MIMIC-CXR dataset, standardizing outputs and evaluation procedures for reproducibility.

* We integrate the domain-aware semantic similarity metric (MCSE) with VICCA's visual grounding and region-level reliability modules, enabling the first large-scale analysis of how text-based and image-based evaluations interact.

* We uncover systematic discrepancies between pathology agreement, semantic coherence, and image grounding, showing that strong performance in one dimension does not guarantee clinical reliability in another.

* We release a practical evaluation checklist and a set of guidelines for applying image-aware, reference-free evaluation in future RRG development and deployment.

The remainder of this paper is organized as follows: Section 2 reviews related work on text-based and multimodal evaluation for report generation. Section 3 summarizes the core principles of the VICCA and MCSE frameworks that reinforce our analysis of RRG models. Section 4 introduces the proposed evaluation protocol and metrics. Section 5 describes the dataset, model configurations, and experimental setup. Section 6 presents the comparative results and detailed analysis. Section 7 discusses key implications, limitations, and ethical considerations. Finally, Section 8 concludes the paper with perspectives for future research on clinically trustworthy AI evaluation.

## 2. Related work

### 2.1. Text-based evaluation metrics for report generation

Automated radiology report generation (RRG) has traditionally been evaluated using natural language processing (NLP) metrics that originated from machine translation (MT), such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), METEOR (Banerjee & Lavie, 2005), and CIDEr (Vedantam et al., 2015). While these measures quantify lexical
overlap between generated and reference texts, they fail to capture
factual accuracy, clinical reasoning, or the correspondence of textual
statements to visual evidence. Subsequent embedding-based metrics,
such as BERTScore (Zhang et al., 2019) and COMET (Rei et al., 2020),
introduced contextualized semantic similarity, yet fail to determine
whether the described findings are clinically correct or grounded in
the medical image. As highlighted in several recent studies (Freitag
et al., 2024; Pauli et al., 2025), MT-style metrics remain inadequate
for evaluating the reliability and safety of AI-generated clinical reports.
In this work, we therefore move beyond these text-only measures
and focus on domain- and image-aware evaluation criteria that better
reflect clinical trustworthiness.

## 2.2. Clinical and domain-aware evaluation approaches

Beyond radiology, entity alignment has also been studied in other
domains using reasoning-based approaches, such as abductive align-
ment across multiple knowledge graphs (Akhtar et al., 2025). Although
these methods target structured multilingual resources rather than
generated clinical narratives, they address a similar challenge of seman-
tic consistency across heterogeneous representations, complementing
entity-based evaluation strategies in medical report generation.

To address the shortcomings of generic NLP metrics, a number of
domain-specific methods have been introduced to evaluate medical
report generation more meaningfully. In one study (Endo et al., 2021)
proposed the CXR-RePaiR framework, which incorporated the CheXbert
vector similarity metric based on the CheXbert labeler (Smit et al.,
2020). By extracting thoracic pathology labels from both generated
and reference reports, this approach assesses clinical agreement in
a structured manner. However, its generalization is limited because
CheXbert was trained on a fixed set of chest X-ray pathologies from
CheXpert (Irvin et al., 2019), making it dataset and modality-specific.

Yu et al. (2023) proposed an entity-based evaluation method using
RadGraph (Jain et al., 2021), which measures the overlap of med-
ical entities and relations between generated and reference reports.
Although this approach improves interpretability by incorporating clin-
ical entities, it still relies on exact entity matches and is restricted to the
chest X-ray domain.

To overcome these limitations, our previous work introduced the
Medical Corpus Similarity Evaluation (MCSE) metric (Picha et al.,
2024). MCSE evaluates textual similarity through clinically meaningful
entities, modifiers, and negations, rather than raw lexical overlap. This
allows for a semantically and clinically informed comparison of reports,
though it still depends on the availability of a reference report.

## 2.3. Multimodal and image-aware evaluation frameworks

Recent work has extended report evaluation beyond text to account
for the alignment between the generated description and the underlying
image. Our previous framework, Visual Interpretation and Compre-
hension of Chest X-ray Anomalies (VICCA) (Gholipour Picha et al.,
2025), introduced a reference-free multimodal evaluation strategy that
jointly considers textual and visual consistency. VICCA employs visual
grounding to localize image regions corresponding to textual phrases
and a text-guided diffusion model to generate anatomically consistent
synthetic chest X-rays, allowing assessment of alignment and detection
of potential hallucinations.

In this work, we extend these ideas by integrating VICCA's visual-
textual reliability assessment with MCSE's semantic evaluation and
standard pathology agreement measures. This combination enables a
comprehensive, multimodal analysis of RRG models, bridging textual,
semantic, and visual dimensions of clinical validity. Unlike prior stud-
ies, our objective is not to compare against traditional NLP metrics,
but rather to establish a clinically meaningful evaluation paradigm
that directly reflects the reliability and interpretability of AI-generated
radiology reports.

Table 1 presents pros and cons of common evaluation approaches
for radiology report generation.

## 2.4. Representative report generation models

To ensure a representative and methodologically diverse evalua-
tion, we selected five radiology report generation models spanning
distinct paradigms in medical vision-language modeling, from earlier
task-specific architectures to more recent foundation-scale systems.
Specifically, the selected models cover: (i) classical encoder–decoder
generation (R2Gen), (ii) multimodal transformer fusion (M2Trans),
(iii) retrieval-augmented refinement (CXR-RePaiR), (iv) explicit visual
grounding during decoding (RGRG), and (v) a foundation-scale vision-
language model (MedGemma). All models were chosen based on the
availability of publicly released checkpoints to ensure reproducibil-
ity. This diversity allows us to examine whether observed evaluation
failures are tied to specific architectural choices or reflect systematic
limitations across report generation paradigms.

R2Gen (Chen et al., 2020) is an encoder–decoder transformer de-
signed for radiology report generation. It employs a memory-driven
recurrent attention mechanism to capture long-range dependencies
between visual and textual tokens, producing coherent narrative de-
scriptions of chest X-rays. R2Gen serves as a strong baseline for image-
to-text generation.

M2Trans (Miura et al., 2021) extends a multimodal transformer
architecture to better align visual and textual modalities through cross-
attention fusion. It explicitly integrates image and report embeddings
during decoding, enhancing pathology-level prediction accuracy and
mitigating exposure bias common in sequential text generation.

CXR-RePaiR (Endo et al., 2021) combines retrieval and generation
by matching the input image with semantically similar training cases
and refining the retrieved reports using a transformer-based genera-
tor. This retrieval-augmented design improves factual correctness and
provides a clinically interpretable mechanism for report composition.

RGRG (Tanida et al., 2023) (Radiology-Grounded Report Genera-
tion) introduces an interactive generation paradigm where the model
grounds intermediate textual tokens to spatial regions within the X-ray
during decoding. This explicit grounding enables tighter alignment be-
tween linguistic entities and visual features, improving interpretability.

MedGemma (Sellergren et al., 2025) represents a foundation-scale
visual-language model trained on diverse biomedical data, extending
the Gemini family to medical imaging. Unlike task-specific RRG archi-
tectures, MedGemma performs zero-shot or few-shot report generation
by leveraging large-scale pretraining, but often produces verbose or
mixed-domain outputs that require clinical filtering.

Together, these five systems encompass a wide methodological spec-
trum, from classical encoder–decoder designs to retrieval-augmented
and foundation-level models, making them suitable for evaluating the
robustness and clinical validity of different RRG paradigms under a
unified multimodal assessment framework.

## 3. Overview of VICCA and MCSE

Before presenting the methods and results, we briefly summarize
the two core components of our evaluation framework, previously
introduced as VICCA and MCSE, and outline their complementary roles
in the present analysis.

Unlike reference-based metrics such as F1, Jaccard, or even MCSE,
which depend on the completeness and accuracy of the ground-truth
report, VICCA is a reference-free and expert-independent framework
designed to assess the alignment between AI-generated radiology re-
ports and their corresponding chest X-ray images. This distinction is
particularly relevant in radiology, where reference reports may omit
secondary findings or describe them implicitly, potentially penalizing
models that produce clinically valid but unannotated observations.

VICCA addresses this limitation by grounding textual phrases di-
rectly in the image and employing a text-guided conditional diffusion
model to generate anatomically consistent synthetic CXRs. This enables
quantification of visual–textual consistency, detection of hallucina-
tions, and evaluation of region-level reliability, independent of report
completeness or labeling bias.
<table><caption>**Table 1**
Comparison of common evaluation approaches for radiology report generation. The table highlights the strengths and limitations of existing text-based, semantic, and image-aware metrics.</caption><thead><tr><td>Evaluation approach</td><td>Strengths</td><td>Limitations</td></tr></thead><tbody><tr><td>Lexical overlap (BLEU, ROUGE, METEOR)</td><td>Simple, fast, reproducible; widely adopted in NLP</td><td>Insensitive to paraphrasing and clinical equivalence; poorly correlated with clinical correctness</td></tr><tr><td>Embedding-based similarity (BERTScore, SciBERT)</td><td>Captures semantic similarity beyond exact wording</td><td>Domain mismatch risk; does not explicitly model medical entities, negation, or factual correctness</td></tr><tr><td>Entity/label-based metrics (CheXbert, RadGraph)</td><td>Clinically interpretable; evaluates pathology presence and relations</td><td>Limited label space; ignores narrative coherence and descriptive context; prone to saturation effects</td></tr><tr><td>Semantic entity similarity (MCSE, proposed)</td><td>Accounts for clinical entities, modifiers, negation, and paraphrasing; robust to surface variation</td><td>Text-only; does not verify whether statements are visually supported by the image</td></tr><tr><td>Image-aware evaluation (VICCA, proposed)</td><td>Links textual findings to image regions; enables visual plausibility and hallucination detection</td><td>Computationally heavier; relies on grounding and generative model quality</td></tr></tbody></table>

**Grounding model and localization output.** For visual grounding, VICCA employs a fine-tuned Grounding DINO model (Liu et al., 2024) adapted to chest X-ray data. The model takes as input a CXR image and a textual phrase and produces a set of candidate bounding boxes with associated confidence scores. Only bounding boxes exceeding a fixed confidence threshold are retained for further analysis. Each retained box represents a hypothesized visual correspondence to the queried phrase.

**Diffusion-based anatomical conditioning.** To assess the reliability of grounded regions, VICCA uses a text-to-image diffusion model conditioned on both the report text and a binary anatomical mask derived from the original CXR. The anatomical mask, obtained via automated lung segmentation, constrains the generation process to preserve global anatomical structure while allowing pathology-specific variations. This conditioning prevents anatomically implausible generations and ensures that differences observed between original and generated images primarily reflect semantic discrepancies in the report rather than structural distortions.

**Reliability assessment.** Reliability is evaluated by registering the grounded bounding boxes onto the generated image and computing region-level similarity using MS-SSIM. High similarity indicates that the textual description induces consistent visual features in anatomically corresponding regions, while low similarity suggests potential semantic mismatch or hallucination. Reliability scores are therefore reported descriptively, since not all reports yield valid grounded regions across models, violating the assumptions required for paired statistical testing.

In contrast, MCSE is a domain-aware semantic metric that measures textual similarity through clinically meaningful entities, modifiers, and negations. It captures semantic fidelity and factual correctness beyond surface-level lexical overlap. MCSE operates by first extracting primary medical entities from each report, along with their associated modifiers and negations, and then computing semantic similarity between these entities using a pretrained domain-specific BERT embedding model.

Given a reference report with extracted entities $T = (t_1, ..., t_N)$ and a generated report with entities $\hat{T} = (\hat{t}_1, ..., \hat{t}_M)$, MCSE combines exact entity matches with soft semantic matches between non-identical entities. For unmatched entities, cosine similarity is computed in a clinical embedding space, and normalized similarity scores are aggregated as defined in Eqs. (1)–(2). Here, $|C|$ accounts for the number of entities that match exactly between reference and candidate texts, while the sum of $S_j$ captures partial or paraphrastic matches for the remaining entities. The final MCSE score is normalized to the range $[0, 1]$, with higher values indicating stronger semantic alignment between the generated and reference reports.

$$y_{i,j} = \text{Similarity}(t_i, \hat{t}_j),$$

$$S_j = \frac{\max_i y_{i,j}}{\max_i y_{i,j} + \bar{y}_{i,j}}, \qquad (1)$$

$$\text{MCSE} := \frac{|C| + \sum_{j=1}^{M'} S_j}{M}, \qquad (2)$$

Unlike label-level metrics, MCSE captures nuanced clinical consistency, including paraphrasing, descriptive variation, and explicit negations, making it particularly suitable for evaluating narrative quality in radiology reports.

Importantly, VICCA and MCSE were developed independently and trained on distinct data, ensuring that their assessments are complementary rather than correlated. MCSE evaluates the textual dimension, how well RRG models preserve clinical semantics and factual accuracy, while VICCA focuses on the visual dimension, determining whether those textual statements are supported by image evidence. Together, the combined VICCA+MCSE framework provides a holistic, multimodal evaluation of radiology report generation systems and forms the foundation of the protocol described in the following section.

**Illustrative Examples.** To further clarify the advantages of reference-free evaluation, we present two representative cases from our analysis that highlight common limitations of reference-based metrics and how VICCA compensates for them.

### Case 1: Model identifies a true finding missing in the reference report

**Predicted pathologies:** Cardiomegaly, Atelectasis, Lung Opacity

**Reference pathologies:** Edema

**Generated report:** “The heart appears enlarged, which could be due to cardiomegaly, and there are areas of increased density in the lower lung fields, which could represent consolidation, pneumonia, or atelectasis”.

**Reference report:** “Mild prominence of the pulmonary vasculature is consistent with mild pulmonary edema”.

**Interpretation:** The RRG model correctly identified opacity and atelectasis, both visible in the radiograph but absent from the reference report as illustrated in Fig. 2. Traditional reference-based metrics would penalize these as false positives, whereas VICCA’s grounding confirmed accurate localization over the lower lung zones and additionally revealed an unreported opacity within the mediastinal region, verifying that the model’s prediction was consistent with the underlying image evidence.
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Example where the generated report correctly identifies additional findings visible in the image but omitted from the reference report.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Example where the generated report omits true pathological findings visible in the image.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Comparison of CheXbert-based micro-F1 and macro-F1 scores for each RRG model.</figcaption></figure>

**Case 2: Model hallucinates unsupported findings**

**Predicted pathologies:** No findings

**Reference pathologies:** Lung opacity

**Generated report:** "The lungs appear clear, with **no obvious** consolidation, **opacities**, or pneumothorax".

**Reference report:** "**Opacities** within the left upper lobe and left lung base".

**Interpretation:** In this case illustrated in Fig. 3, the RRG model failed to describe the existing lung opacity, generating a report inconsistent with the image content. Reference-based metrics would not detect this error since there is an opacity entity in the generated report

appears linguistically coherent. VICCA, however, identified the absence of grounded regions corresponding to opacity and measured a low region-level reliability score, effectively flagging this hallucination.

**4. Evaluation protocol**

Building on the reviewed literature and the representative set of RRG models presented in Section 2.4, our evaluation protocol aims to provide a unified and clinically grounded framework for assessing the reliability of AI-generated chest X-ray reports. Unlike prior approaches that focus solely on lexical or semantic similarity, our goal is to integrate multiple complementary perspectives, textual, semantic, and visual, within a single standardized pipeline that reflects how radiologists assess report trustworthiness in practice.

Specifically, the proposed protocol combines:
<caption>**Table 2**</caption>

Statistics of the filtered MIMIC-CXR test set used for evaluation. Studies overlapping with the MS-CXR training set were excluded to prevent data leakage.

<table><thead><tr><th>Statistic</th><th>Value</th></tr></thead><tbody><tr><td>Original MIMIC-CXR test studies</td><td>3269</td></tr><tr><td>Excluded due to MS-CXR overlap</td><td>808</td></tr><tr><td><strong>Final evaluation studies</strong></td><td><strong>2461</strong></td></tr><tr><td>Studies labeled as <em>No Finding</em></td><td>533</td></tr><tr><td>Studies with ≥1 pathology</td><td>1563</td></tr><tr><td>Number of CheXbert pathology classes</td><td>14</td></tr><tr><td>Most frequent pathologies</td><td>Lung Opacity, Pleural Effusion, Cardiomegaly</td></tr><tr><td>Least frequent pathologies</td><td>Fracture, Lung Lesion, Pleural Other</td></tr></tbody></table>

1. **Pathology-level agreement**, which quantifies how well the generated reports capture medically relevant findings using the CheXbert classifier (Smit et al., 2020) to compute micro- and macro-F1 as well as Jaccard similarity.

2. **Semantic fidelity**, evaluated through the Medical Corpus Similarity Evaluation (MCSE) metric (Picha et al., 2024), which measures conceptual alignment between generated and reference reports by detecting clinical entities, modifiers, and negations rather than relying on lexical overlap.

3. **Visual-textual consistency**, assessed through the VICCA framework (Gholipour Picha et al., 2025), which employs visual grounding to link report phrases to image regions and diffusion-based generation to test visual plausibility and detect potential hallucinations.

This multimodal evaluation strategy allows us to examine not only what each model says but also whether those statements are visually justified and clinically coherent. By analyzing these three dimensions jointly, we can uncover discrepancies that remain hidden in traditional text-only evaluations and gain a more holistic understanding of model behavior in realistic clinical scenarios.

## 5. Experimental setup

To approximate a realistic clinical deployment scenario, we apply the proposed evaluation framework to the outputs of the five RRG models described in Section 2.4. All evaluations are conducted on a subset of 2461 studies from the MIMIC-CXR test set (Johnson, Pollard, Berkowitz et al., 2019; Johnson, Pollard, Greenbaum et al., 2019), ensuring that each case includes both a frontal chest X-ray and a corresponding reference report.

**Test set filtering and leakage prevention.** The original MIMIC-CXR test split contains 3269 studies. However, a subset of these studies is also included in the MS-CXR (Boecking et al., 2022) annotations, which are used during training for visual grounding and visual alignment in VICCA. To prevent any form of data leakage between training and evaluation, we explicitly excluded all test studies that appear in the MS-CXR training set. After removing these overlapping cases, the final evaluation set consists of 2461 studies. All reported results on MIMIC-CXR are computed exclusively on this filtered test set. No additional filtering or subsampling was applied beyond this exclusion. Table 2 summarizes the composition of the filtered MIMIC-CXR test set used throughout our experiments.

### 5.1. Text processing step

To generate the radiology reports for the test set, we used each model's publicly available pretrained weights. Reports were produced

<caption>**Table 3**</caption>

Example of a MedGemma report before and after cleaning. The preprocessing step focuses on the clinically relevant Findings section to align outputs with radiology reporting conventions.

<table><thead><tr><th>Original MedGemma Output</th><th>Cleaned report (Findings only)</th></tr></thead><tbody><tr><td>“Okay, here’s a description of the chest X-ray... **Overall Impression:**** The heart appears enlarged, and there are areas of increased density... **Specific Findings:**** Bones intact, mediastinum normal, diaphragms unremarkable... **Possible Considerations:**** Cardiomegaly, infiltrates, need for CT... **Disclaimer:**** Preliminary interpretation only...”</td><td>“The heart appears enlarged, with increased density in the lower lung fields suggesting possible consolidation or infiltrates. Bones are intact; mediastinum and diaphragm appear normal. Consider cardiomegaly and infiltrates; correlation with clinical context recommended”.</td></tr></tbody></table>

directly from the CXR images without any additional prompt engineering or text postprocessing, except in the case of the foundation model **MedGemma**. Consequently, the text inputs to the VICCA, MCSE, and pathology detection modules correspond to the raw outputs of the RRG models, with MedGemma being the only exception.

For MedGemma, we employed the 4B multimodal instruction-tuned variant (MedGemma-4B-IT) to produce radiology-style outputs, using a maximum of 1000 new tokens per generation. As a foundation model derived from Gemma and adapted for medical imaging, MedGemma often produces verbose, mixed-domain responses that extend beyond the typical radiology report format. To align its outputs with clinical reporting conventions, an additional postprocessing step was applied to extract the most relevant Findings section, as illustrated in Table 3.

Unlike the RRG models trained specifically for radiology report generation, MedGemma is primarily designed for general medical reasoning and non-expert interpretation of CXR images. As a result, its responses resemble those of conversational AI systems, frequently organized into sections such as “Overall Impression”, “Specific Findings”, “Differential Diagnosis”, “Recommendations”, or “Disclaimer”, with varying structures across samples. This variability complicates the direct extraction of clinically focused content, particularly since the “Findings” or “Impression” sections are not consistently defined.

Moreover, due to the large number of generated tokens, MedGemma outputs occasionally contain hallucinated entities or redundant content. Although reducing the generation length could mitigate verbosity, this would compromise the accuracy and anatomical-pathological alignment of the output. Instead, we implemented a targeted summarization procedure that identifies and extracts the paragraph most relevant to clinical interpretation. This process relies on detecting the “Findings” keyword and analyzing the density of medical entities within each section to isolate the portion most suitable for evaluation. This post-processing step affects only formatting and section selection and does not alter the semantic content of the generated reports.

After generation, we evaluate each model's output along the three complementary dimensions defined in our protocol 4. This end-to-end setup allows an integrated evaluation of each RRG model, simultaneously examining its textual coherence, semantic fidelity, and visual trustworthiness under clinically plausible conditions.

### 5.2. Evaluation metrics

In this section, we describe the evaluation metrics used to assess the performance of radiology report generation (RRG) models that were referenced earlier but not yet formally defined. Table 4 summarizes the notation used throughout the evaluation framework.
<table><caption>**Table 4**
Summary of notation used in the evaluation framework.</caption><thead><tr><th>Symbol</th><th>Description</th></tr></thead><tbody><tr><td>y</td><td>Reference radiology report associated with a chest X-ray image</td></tr><tr><td>ŷ</td><td>Report generated by a report generation model</td></tr><tr><td>T = (t<sub>1</sub>, ..., t<sub>N</sub>)</td><td>Sequence of clinical entities extracted from reference report y</td></tr><tr><td>T̂ = (t̂<sub>1</sub>, ..., t̂<sub>M</sub>)</td><td>Sequence of clinical entities extracted from generated report ŷ</td></tr><tr><td>C</td><td>Set of exact entity matches between T and T̂</td></tr><tr><td>r<sub>i</sub>, ř<sub>j</sub></td><td>Unmatched reference and generated entities after removing exact matches</td></tr><tr><td>y<sub>i,j</sub></td><td>Cosine similarity between entities r<sub>i</sub> and ř<sub>j</sub> in clinical embedding space</td></tr><tr><td>S<sub>j</sub></td><td>Normalized semantic similarity score for generated entity ř<sub>j</sub></td></tr><tr><td>MCSE</td><td>Medical Corpus Similarity Evaluation score between y and ŷ</td></tr><tr><td>A, B</td><td>Sets of pathology labels extracted from generated and reference reports (CheXbert)</td></tr><tr><td>J(A, B)</td><td>Jaccard similarity between pathology sets A and B</td></tr><tr><td>τ</td><td>Confidence threshold used to accept a grounded bounding box</td></tr><tr><td>Coverage</td><td>Proportion of report phrases that yield at least one grounded region</td></tr><tr><td>Reliability</td><td>Mean MS-SSIM score between localized regions in original and generated images</td></tr></tbody></table>

**Textual pathology label accuracy (CheXbert).**

* **Micro-F1** aggregates true positives, false positives, and false negatives across all pathology classes, and is therefore dominated by frequent conditions such as *No Finding* and *Pleural Effusion*.

* **Macro-F1** computes the F1-score independently for each pathology and then averages across classes, assigning equal weight to common and rare findings. This is particularly important in clinical settings, where infrequent but critical conditions (e.g., *Fracture*, *Lung Lesion*) should not be overshadowed by prevalent labels.

**Pathology-set overlap (jaccard similarity).** To quantify overlap between pathology labels extracted from generated and reference reports, we compute the Jaccard similarity. Given the sets of pathologies A (generated) and B (reference), the Jaccard index is defined as:

$$J(A, B) = \frac{|A \cap B|}{|A \cup B|} \qquad (3)$$

This metric penalizes both missing findings and spurious mentions, providing a strict set-level agreement score.

**Localization coverage.** Localization coverage measures the proportion of textual phrases in a report that can be visually grounded in the corresponding chest X-ray. For each extracted phrase, visual grounding is considered successful if the grounding model returns at least one bounding box with confidence greater than a fixed threshold $\tau$.

Formally, coverage is defined as:

$$Coverage = \frac{\# \text{ grounded phrases}}{\# \text{ extracted phrases}} \qquad (4)$$

In all experiments, the confidence threshold was set to $\tau = 0.25$. This definition captures the presence of visually groundable content independently of localization precision, which is evaluated separately through reliability.

**Region-level reliability (MS-SSIM).** Reliability evaluates the consistency between localized regions in the original CXR and the corresponding regions in the text-conditioned generated image. For each successfully grounded phrase, MS-SSIM is computed between the cropped bounding

box in the original image and the corresponding region in the generated image.

Reliability is first computed at the bounding-box level, then averaged across all grounded phrases within a study to obtain a per-study reliability score. Finally, model-level reliability is reported as the mean reliability across all studies with valid grounded regions.

Studies without any valid grounded regions are excluded from reliability aggregation, which may result in unequal sample sizes across models.

Together, these metrics allow us to disentangle pathology correctness, semantic fidelity, and visual grounding reliability in a unified evaluation framework.

The following sections detail the results of this multi-level evaluation, illustrating how VICCA can provide deeper insight into the strengths and limitations of state-of-the-art RRG models beyond traditional metrics.

# 6. Results and analysis

All evaluation metrics used in the following sections are formally defined in Sections 3 and 5.2.

## 6.1. Textual pathology label accuracy

To evaluate whether the generated reports reflect clinically relevant findings, we first assess the accuracy of predicted pathologies using the CheXbert model. Trained on CheXpert classes, CheXbert classifies radiology reports into 14 thoracic disease categories. This comparison between generated and reference labels offers insight into each model's ability to capture medically meaningful content.

Given that pathology extraction is a multi-label classification task, we report both **micro-F1** and **macro-F1** scores. These scores range from 0 (worst) to 1 (best). Among the evaluated models, **R2Gen** shows the weakest alignment with reference findings (micro-F1 = 0.306, macro-F1 = 0.182), frequently producing vague descriptions and omitting key pathologies. In contrast, **MedGemma** (micro-F1 = 0.480, macro-F1 = 0.324) proves highly competitive, achieving the strongest balance across both common and rare classes, as reflected in its highest macro-F1 score. **M2Trans** reaches the second-highest pathology-level agreement (micro-F1 = 0.474), followed closely by **RGRG** (0.470) and **CXR-RePaiR** (0.452). These results are summarized in Fig. 4. When no explicit pathologies are detected, models default to the “No Finding” label. This behavior affects performance on the dominant “No Finding” class.

In addition to global F1-scores, we also report performance for each pathology along with its support, i.e., the number of cases for that pathology in the test set. This breakdown is particularly relevant in medical evaluation, since certain findings (e.g., “Pleural Effusion”, “Cardiomegaly”) are much more prevalent than others (e.g., “Fracture”, “Lung Lesion”). High-support classes tend to dominate micro-F1 scores, while low-support but clinically critical classes may be overlooked unless reported separately. By presenting pathology-by-support metrics in Fig. 5, we highlight whether models generalize beyond frequent findings and remain clinically reliable for rare but important conditions.

* **High-support classes** (e.g., Pleural Effusion, Cardiomegaly, Lung Opacity): All models achieve relatively higher precision and recall since these findings dominate the dataset. M2Trans and RGRG capture them more consistently, while CXR-RePaiR shows balanced detection. R2Gen under-predicts them, often defaulting to “No Finding”.

* **Low-support but clinically critical classes** (e.g., Fracture, Lung Lesion, Pleural Other): Performance is weak across all models, particularly in recall. This drags down macro-F1, which weights each pathology equally. MedGemma performs relatively better on these rare classes, as reflected in its higher macro-F1, though overall performance remains limited.
<figure><img src="image_6.png" /><figcaption>Fig. 5. Per-class F1 score distributions for the top 15 most prevalent pathologies in the test set.</figcaption></figure>

* **No Finding:** Predicted too frequently by R2Gen, inflating apparent accuracy in normal cases but masking pathology detection. CXR-RePaiR provides a better balance between “No Finding” and actual disease mentions, while MedGemma reduces this bias by predicting more diverse pathologies.

Key takeaways.

* **R2Gen:** Consistently underpredicts pathology classes and defaults heavily to “No Finding”.

* **MedGemma:** Strongest overall balance, with the highest macro-F1; performs better than others on rare classes, though still limited in absolute terms.

* **M2Trans:** Achieves the second-best pathology alignment, especially strong on high-support classes, but still produces some empty predictions.

* **RGRG:** Balanced performance, capturing common pathologies well, though occasionally overpredicts compared to references.

* **CXR-RePaiR:** More diverse in its pathology predictions than R2Gen and maintains a better balance between “No Finding” and true disease mentions, but introduces some false positives.

Pathology-set overlap (jaccard similarity). In addition to F1 scores, we report Jaccard similarity between the sets of pathologies extracted from generated and reference reports. Across all models, mean Jaccard values range from 0.25 to 0.38 (Table 5), with large interquartile ranges. Despite these numerical differences, Jaccard scores exhibit limited discriminative power, as many reports collapse to identical pathology sets dominated by the “No Finding” label. As a result, Jaccard similarity shows low variance and weak association with semantic fidelity, motivating the use of MCSE for narrative-level evaluation.

<caption>Table 5</caption>

Jaccard similarity between pathology sets extracted from generated and reference reports. Values are reported as mean ± standard deviation, along with median and interquartile range (IQR).

<table><thead><tr><td>Model</td><td>Mean</td><td>Std</td><td>Median</td><td>IQR</td></tr></thead><tbody><tr><td>RGRG</td><td>0.335</td><td>0.310</td><td>0.333</td><td>0.50</td></tr><tr><td>R2Gen</td><td>0.249</td><td>0.347</td><td>0.000</td><td>0.40</td></tr><tr><td>CXR-RePaiR</td><td>0.307</td><td>0.277</td><td>0.250</td><td>0.50</td></tr><tr><td>MedGemma</td><td>0.370</td><td>0.309</td><td>0.333</td><td>0.33</td></tr><tr><td>M2Trans</td><td>0.380</td><td>0.377</td><td>0.333</td><td>0.67</td></tr></tbody></table>

## 6.2. Semantic similarity evaluation using MCSE

To assess the semantic alignment between the generated and reference reports, we employ our proposed metric, **MCSE**, to evaluate conceptual consistency in clinical narratives illustrated in Fig. 6.

As the reports are generated directly from true CXR images, we expect a meaningful degree of semantic similarity with their corresponding references. Among the models, **RGRG** achieves the highest average MCSE score (0.696), reflecting strong alignment with reference narratives. **R2Gen** follows closely (0.676), though with high variability across cases: some reports align well semantically, while others remain overly generic. **CXR-RePaiR** (0.657) and **MedGemma** (0.645) also performs competitively, maintaining reasonably coherent narratives. In contrast, **M2Trans** obtains the lowest MCSE (0.587), indicating lower semantic alignment between generated and reference narratives.

Table 6 summarizes the distribution of MCSE scores across models, reporting mean, standard deviation, median, and interquartile range (IQR) over the 2460 aligned studies.
<figure><img src="image_7.png" /><figcaption>Fig. 6. Distribution of MCSE scores aggregated across all models.</figcaption></figure>

<caption>**Table 6**</caption>

Statistical summary of MCSE scores across RRG models. Higher mean MCSE indicates better semantic alignment with reference reports.

<table><thead><tr><th>Model</th><th>Mean MCSE</th><th>Std</th><th>Median</th><th>IQR</th></tr></thead><tbody><tr><td>RGRG</td><td><strong>0.696</strong></td><td>0.132</td><td>0.696</td><td>0.079</td></tr><tr><td>R2Gen</td><td>0.676</td><td><strong>0.468</strong></td><td>0.687</td><td>0.086</td></tr><tr><td>CXR-RePaiR</td><td>0.657</td><td>0.286</td><td>0.652</td><td>0.055</td></tr><tr><td>MedGemma</td><td>0.645</td><td><strong>0.070</strong></td><td>0.645</td><td><strong>0.042</strong></td></tr><tr><td>M2Trans</td><td>0.587</td><td>0.278</td><td>0.609</td><td>0.108</td></tr></tbody></table>

<figure><img src="image_8.png" /><figcaption>Fig. 7. Comparative plot of MCSE vs. cheXbert Jaccard scores across all models.</figcaption></figure>

*Statistical significance of MCSE differences.* To evaluate whether observed differences in MCSE scores are statistically meaningful, we performed non-parametric testing on strictly paired samples using a Friedman test, followed by post-hoc paired Wilcoxon signed-rank tests with Holm correction. The Friedman test revealed a significant effect of model choice on semantic similarity ($\chi^2 = 2135.47, p < 10^{-16}$), indicating that MCSE scores differ substantially across models.

Post-hoc analysis confirmed that these differences are robust: **RGRG** significantly outperforms all other models, while **M2Trans** consistently underperforms. Notably, **R2Gen** also achieves significantly higher MCSE scores than M2Trans despite exhibiting greater variance in its outputs. Together, these results demonstrate that the observed differences in semantic similarity are not attributable to random variation.

*Relation between MCSE and jaccard similarity.* While Jaccard similarity measures set-level overlap of pathology labels, MCSE evaluates semantic fidelity of the full clinical narrative. As shown in Fig. 7, these two metrics exhibit no meaningful association. In practice, CheXbert

<figure><img src="image_9.png" /><figcaption>Fig. 8. Comparative analysis of localization using coverage for each models.</figcaption></figure>

Jaccard similarity saturates at 1.0 for all models, yielding zero variance and rendering statistical testing and correlation analysis undefined. This saturation arises from coarse label extraction and the dominance of generic labels such as *No Finding*. Consequently, pathology-set overlap metrics fail to discriminate between report generators under this evaluation setup, indicating that pathology-set overlap alone does not discriminate between models under this evaluation setup.

## 6.3. Visual grounding and localization

Beyond textual similarity, VICCA enables validation through visual grounding, associating textual phrases with corresponding regions on the chest X-ray. We evaluate each model's localization coverage, defined as the proportion of identified phrases successfully grounded with bounding boxes.

Localization coverage is defined as the proportion of report phrases that yield at least one valid grounded region (see Section 5.2), and should not be interpreted as a measure of localization accuracy or reliability. Fig. 8 shows the coverage across all models. **RGRG** achieves the highest coverage (0.844), closely followed by **M2Trans** (0.823). **R2Gen** (0.686) and **CXR-RePaiR** (0.677) fall into the mid-range. By contrast, **MedGemma** records the lowest coverage (0.297), indicating that although it performs well textually, its grounding of findings to image regions remains limited.

Importantly, coverage alone does not guarantee clinical precision as grounding quantity does not directly reflect anatomical correctness. Models like M2Trans and RGRG produce more boxes overall, but the quality of alignment with meaningful anatomical structures varies.

* **RGRG:** Strongest coverage and good phrase-to-region alignment.

* **M2Trans:** High coverage, though some boxes are noisy or weakly grounded.

* **R2Gen/CXR-RePaiR:** Moderate coverage; grounding consistency is weaker than RGRG.

* **MedGemma:** Struggles to produce bounding boxes, highlighting limited grounding capability.

This highlights the need to evaluate both the quantity and quality of grounding for clinically interpretable outputs.

## 6.4. Reliability of localized regions

We further assess the reliability of grounded regions by evaluating the consistency between localized regions in the original and generated images using MS-SSIM. This metric captures structural similarity, providing insight into how well a given report preserves visual characteristics in the generated image.
<figure><img src="image_10.png" /><figcaption>Fig. 9. Comparative distribution of mean reliability scores across models.</figcaption></figure>

Unlike localization coverage, reliability scores are very close across models, ranging from 0.666 to 0.690. RGRG (0.690) and MedGemma (0.687) slightly outperform others, followed by M2Trans (0.680) and CXR-RePaiR (0.675). R2Gen lags marginally (0.666), though the difference is small.

These results in Fig. 9 suggest that region-level structural consistency is relatively stable across models, even when textual semantics and pathology predictions differ significantly.

Region-level reliability scores are reported descriptively, as insufficient paired observations were available across all models to support non-parametric statistical testing.

* **RGRG:** Most reliable overall; balances semantic quality with structural consistency.

* **MedGemma:** Strong reliability despite weak localization, suggesting coherent structural mapping when boxes exist.

* **M2Trans:** Reliable on average, though variability across samples is higher.

* **CXR-RePaiR:** Consistently reliable, though slightly lower than RGRG.

* **R2Gen:** Lowest reliability, consistent with weaker text-pathology alignment.

MS-SSIM thus captures a complementary dimension of model performance, structural plausibility, that is not apparent through text-based metrics alone.

## 6.5. Failure mode analysis (Qualitative Micro-Examples)

To complement aggregate quantitative metrics, we present representative qualitative failure cases that expose model-specific limitations. These micro-examples illustrate distinct failure modes that are not fully captured by label-based or reference-driven metrics, and highlight the diagnostic value of jointly evaluating semantic consistency and visual grounding through MCSE and VICCA.

### 6.5.1. Case 1: CXR-RePaiR – Textually similar but clinically misleading retrieval

Table 7 summarizes a representative failure case produced by the CXR-RePaiR model, comparing the reference report, generated report, and the corresponding pathology labels. Although the generated report remains semantically close to the reference narrative, it introduces additional findings not supported by the ground truth.

In this case, MCSE assigns a high semantic similarity score (0.85), reflecting strong overlap in disease progression, anatomical context, and descriptive structure. However, the pathology-set overlap measured by Jaccard similarity is low (0.20), as the generated report

<caption>Table 7</caption>

Qualitative failure example for the CXR-RePaiR model. Despite high semantic similarity, the generated report introduces clinically unsupported findings.

<table><thead><tr><th colspan="2">Model: CXR-RePaiR</th></tr><tr><th>Reference report</th><th>Generated report</th></tr></thead><tbody><tr><td>There is unchanged evidence of the known massive irritated right lung process. Slightly progressive opacifications in the periphery of the left lung. Unchanged size of the cardiac silhouette.</td><td>Complete opacification of the right hemithorax is due to collapse of the small residual of aerated right lung compromised by the chronically elevated right hemidiaphragm and probable small right pleural effusion. Heterogeneous opacification in the left lung, particularly the suprathilar region, could be aspiration or early pneumonia. Previous pulmonary vascular congestion has improved, and moderate cardiomegaly is minimally smaller.</td></tr><tr><th>Reference labels</th><th>CheXbert labels</th></tr><tr><td>Lung Opacity, Cardiomegaly</td><td>Lung Opacity, Pleural effusion, Cardiomegaly</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>Fig. 10. Example of a CXR-RePaiR failure case where high semantic similarity masks clinically unsupported findings.</figcaption></figure>

introduces spurious pathologies such as pleural effusion. A purely label-based evaluation would therefore suggest poor agreement, without revealing the underlying semantic alignment.

Fig. 10 further illustrates this discrepancy through visual grounding. Although several phrases from the generated report can be localized, the predicted finding Cardiomegaly shows no corresponding visual support, as it is neither supported by the reference report nor visually evident in the image. This exposes a clinically misleading over-interpretation introduced by the model.

This example highlights a characteristic failure mode of retrieval-augmented generation: reports that are fluent and semantically plausible yet introduce unsupported clinical hypotheses. Such behavior is difficult to identify using pathology-label metrics such as CheXbert or Jaccard similarity alone, but becomes apparent when semantic similarity and visual grounding are evaluated jointly.

### 6.5.2. Case 2: M2Trans – High pathology overlap but low visual reliability

Fig. 11 and Table 8 illustrate a case where pathology-level overlap appears deceptively strong, while visual reliability is poor. CheXbert extraction yields identical number of pathology sets for the reference and generated reports, resulting in a Jaccard similarity of 1.0. However, the generated report explicitly negates pulmonary findings and omits pneumonia, leading to a marked divergence in visual grounding.

As a result, the grounded regions derived from the generated report focus solely on the cardiac silhouette, whereas the reference grounding highlights basilar lung involvement for Pneumonia. This discrepancy yields a low region-level reliability score (34.6%), despite label overlap.
<figure><img src="image_12.png" /><figcaption>**Fig. 11.** Failure case for M2Trans. The reference grounding highlights basilar pneumonia with partial overlap near the cardiac silhouette, whereas the generated report focuses exclusively on cardiomegaly, omitting the infectious component.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>**Fig. 12.** Failure case for MedGemma. The reference grounding highlights diffuse interstitial edema, while the generated report induces grounding attempts for a fabricated humeral fracture and subcutaneous emphysema, resulting in low reliability.</figcaption></figure>

**Table 8**

Qualitative failure example for the **M2Trans** model. Although pathology-set overlap is high, the generated report fails to preserve the visual semantics of the reference findings.

<table><thead><tr><th colspan="2">Model: M2Trans</th></tr><tr><th>Reference report</th><th>Generated report</th></tr></thead><tbody><tr><td>The hand of the patient obscures the lower half of the left chest. There is enlargement of the cardiac silhouette with indistinctness of engorged pulmonary vessels, consistent with elevated pulmonary venous pressure. In the appropriate clinical setting, superimposed basilar pneumonia could be considered.</td><td>The cardiomediasitastal and hilar contours are stable with moderate cardiomegaly. The heart is moderately enlarged. The lung fields are clear. There is no focal consolidation, pulmonary edema, pleural effusion, or pneumothorax.</td></tr><tr><td>Reference labels</td><td>CheXbert labels (Generated)</td></tr><tr><td>Pneumonia, Cardiomegaly</td><td>Cardiomegaly, Cardiomegaly</td></tr></tbody></table>

This example exposes a characteristic failure mode of transformer-based report generators: accurate repetition of dominant or frequent labels without preserving the spatial or clinical context in which those findings appear. Such failures are invisible to pathology-set overlap metrics but are revealed by joint evaluation of semantic consistency and visual reliability.

### 6.5.3. Case 3: MedGemma – Plausible narrative with fabricated anatomical detail

In this case (Fig. 12 and Table 9), MedGemma produces a fluent and clinically plausible narrative but introduces additional anatomical findings that are not supported by the reference report or visual evidence. Although the reference mentions a possible positional subluxation of the “left” humeral head, the generated report instead describes a “right-sided” fracture, reflecting a semantic extrapolation beyond the available information.

**Table 9**

Qualitative failure example for the **MedGemma** model. The generated report introduces anatomically plausible but unsupported findings that are absent from the reference.

<table><thead><tr><th colspan="2">Model: MedGemma</th></tr><tr><th>Reference report</th><th>Generated report</th></tr></thead><tbody><tr><td>A frontal upright view of the chest was obtained portably. Since the prior study, mild interstitial edema persists but has improved. There is no focal consolidation, pleural effusion, or pneumothorax. Heart size is stable. Aortic tortuosity is unchanged. The left humeral head appears inferiorly subluxed with respect to the glenoid; however, it is not adequately visualized on this film and may be positional.</td><td>There are increased interstitial markings, which could be due to age-related changes, mild pulmonary vascular congestion, or underlying lung disease. The mediastinum appears within normal limits. <strong>Bones: There is a fracture of the proximal humerus on the right side.</strong> <strong>Other: There is subcutaneous emphysema in the right upper chest.</strong> Due to the portable nature of the image and patient positioning, further evaluation is recommended.</td></tr><tr><td>Reference labels</td><td>CheXbert labels (Generated)</td></tr><tr><td>Edema, no Pneumonia</td><td>Lung Opacity, Fracture</td></tr></tbody></table>

During visual grounding, these fabricated entities trigger bounding-box predictions in anatomically inconsistent regions, yielding very low region-level reliability. At the same time, the mild interstitial edema described in the reference is displaced spatially in the generated grounding, further reducing alignment between text and image.

This example highlights a characteristic behavior of foundation-scale instruction-tuned models: the generation of fluent, well-structured clinical language that may extrapolate beyond the available visual or textual evidence. Such hallucinations may remain undetected by surface-level semantic metrics or pathology extraction alone, but are
<figure><img src="image_14.png" /><figcaption>**Fig. 13.** Scatter plot of mean reliability versus MCSE scores. Semantically similar reports do not always guarantee reliable localized grounding.</figcaption></figure>

effectively exposed by VICCA through joint evaluation of semantic consistency, anatomical localization, and visual reliability.

Taken together, these qualitative examples contextualize the aggregate results presented earlier. They demonstrate that strong pathology overlap, fluent language, or even high semantic similarity in isolation can mask clinically meaningful failure modes. By exposing discrepancies between textual claims and visual evidence, VICCA enables a more fine-grained diagnosis of model behavior that complements quantitative comparisons.

## 6.6. Integrated comparison

Fig. 13 illustrates the relationship between semantic similarity (MCSE) and region-level reliability. The observed positive yet shallow trend indicates that semantic alignment and visual reliability do not always evolve in parallel across models.

* **R2Gen:** Weak pathology alignment, limited semantics, and localization low coverage.

* **M2Trans:** Strongest in pathology-level prediction with high coverage, but narrative variability reduces semantic and reliability consistency.

* **RGRG:** Most balanced overall, combining strong semantic alignment with the highest localization coverage and reliable grounding.

* **CXR-RePaiR:** Clinically stable and consistent, providing moderate coverage with solid reliability across cases.

* **MedGemma:** Competitive in pathology accuracy and semantic similarity, with notably stable semantic outputs, but exhibits limited localization coverage, which constrains visual interpretability.

Correlation analysis between MCSE, CheXbert Jaccard similarity, and region-level reliability yielded undefined coefficients due to constant or missing inputs. This confirms that semantic similarity, pathology overlap, and visual reliability capture largely independent dimensions of report quality.

These findings also raise important ethical considerations regarding over-trust, accountability, and bias, which we discuss in Section 7.3.

## 7. Discussion

While the Results section reports quantitative performance and statistical significance, this section focuses on interpreting the observed trade-offs and their implications for clinical deployment.

**Table 10** summarizes the comparative evaluation. **RGRG** demonstrates the most balanced performance, excelling in semantic alignment, localization, and reliability. **M2Trans** provides the best pathology-level prediction but suffers from weaker semantic narratives. **MedGemma** performs strongly on rare pathologies and text metrics but underperforms in grounding. **CXR-RePaiR** offers stable and balanced performance across metrics. **R2Gen** remains the weakest overall, with limited pathology accuracy and grounding, though occasionally achieving competitive semantic similarity.

An important observation concerns the limited discriminative power of pathology-set overlap metrics. Although CheXbert-based Jaccard similarity reports non-zero mean values across models, these scores exhibit high variance, coarse medians, and large interquartile ranges, reflecting frequent collapse to identical or near-identical pathology sets dominated by generic labels such as “No Finding”. As a result, Jaccard similarity provides weak separation between report generators and shows little association with semantic or visual fidelity. This limitation motivates the use of semantically informed metrics such as MCSE and image-aware evaluation through VICCA for meaningful comparison.

The inability to apply paired statistical testing to reliability scores reflects meaningful model behavior rather than missing methodology. Several models fail to produce grounded regions consistently, violating the assumptions required for paired analysis. This reinforces the role of grounding coverage itself as a discriminative signal and highlights the complementary nature of semantic and visual reliability metrics.

These findings suggest that future evaluation frameworks should explicitly combine semantic, visual, and uncertainty-aware signals rather than relying on any single metric.

### 7.1. Implications for deployment and clinical use

The proposed evaluation highlights that different report generation models may be preferable depending on the intended clinical context and risk tolerance. Importantly, these observations reflect relative behavior under the present evaluation framework rather than clinical endorsement of any system.

Models such as **RGRG**, which combine strong semantic fidelity (highest MCSE) with the highest localization coverage and reliable visual grounding, appear better suited for scenarios where interpretability and traceability of findings are critical, such as clinical triage, decision support, or auditing assistance. In such settings, the ability to verify whether textual statements are supported by image evidence is essential to mitigate hallucinations and omission errors.

Foundation-scale models such as **MedGemma** demonstrate stable semantic behavior and comparatively stronger performance on rare pathology classes, as reflected in macro-F1 and low MCSE variance. However, their limited grounding coverage suggests that human oversight remains necessary when anatomical localization or spatial verification is required. These models may therefore be more appropriate for exploratory analysis, educational support, or as assistive tools rather than autonomous reporting systems.

In contrast, **M2Trans** achieves strong pathology-level agreement but exhibits significantly weaker semantic coherence and higher variability. This indicates that accurate label prediction alone does not guarantee clinically meaningful narratives, limiting its suitability for settings where report readability and contextual reasoning are important.

Overall, these findings suggest that deployment decisions should consider not only pathology accuracy but also semantic robustness and visual grounding. Multidimensional evaluation frameworks such as VICCA can help practitioners and system designers identify model-specific strengths and failure modes before integration into clinical workflows.
<caption>**Table 10**</caption>

Comparison of RRG models evaluated with VICCA. RRG provides the strongest balance across dimensions, while MedGemma is textually strong but visually weak.

<table><caption>**Table 11**</caption><thead><tr><th>Model</th><th>CheXbert Micro-F1</th><th>MCSE (mean)</th><th>Localization coverage</th><th>Reliability (mean)</th></tr></thead><tbody><tr><td>RGRG</td><td>0.470</td><td><strong>0.696</strong></td><td><strong>0.844</strong></td><td><strong>0.690</strong></td></tr><tr><td>R2Gen</td><td>0.306</td><td>0.676</td><td>0.686</td><td>0.666</td></tr><tr><td>CXR-RePaiR</td><td>0.452</td><td>0.657</td><td>0.677</td><td>0.675</td></tr><tr><td>M2Trans</td><td>0.474</td><td>0.587</td><td>0.823</td><td>0.680</td></tr><tr><td>MedGemma</td><td><strong>0.480</strong></td><td>0.653</td><td>0.297</td><td>0.687</td></tr></tbody></table>

Cross-dataset evaluation on IU-Xray test set using VICCA and MCSE.

<table><thead><tr><th>Model</th><th>CheXbert Micro-F1</th><th>CheXbert Macro-F1</th><th>MCSE (Mean)</th><th>Reliability (Mean)</th></tr></thead><tbody><tr><td>R2Gen</td><td>—</td><td>0.000</td><td><strong>0.708</strong></td><td><strong>0.518</strong></td></tr><tr><td>RGRG</td><td>0.245</td><td>0.134</td><td>0.681</td><td>0.516</td></tr><tr><td>CXR-RePaiR</td><td>0.233</td><td>0.148</td><td>0.612</td><td>0.507</td></tr><tr><td>MedGemma</td><td>0.218</td><td>0.094</td><td>0.654</td><td>0.499</td></tr></tbody></table>

## 7.2. Generalization under dataset shift: IU-Xray evaluation

To assess the robustness of the proposed evaluation framework under dataset shift, we additionally evaluated VICCA and MCSE on the IU-Xray test set (Liu et al., 2025) and results are presented in Table 11. Unlike MIMIC-CXR, IU-Xray contains shorter, more tem- plated reports and follows different reporting conventions, providing a complementary evaluation setting.

Overall, pathology-level agreement measured by CheXbert F1 scores decreased substantially across all models compared to MIMIC-CXR, reflecting both domain shift and sensitivity of label extraction to re- porting style. Despite this degradation, semantic similarity measured by MCSE remained relatively high, and visual grounding metrics re- mained stable across models. This indicates that semantic coherence and visual-textual alignment persist even when explicit pathology ex- traction becomes unreliable.

Notably, R2Gen achieved the highest MCSE score on IU-Xray despite near-zero macro-F1, suggesting that its generated narratives remained semantically close to reference reports even when explicit pathology mentions diverged. RGRG maintained balanced performance across semantic similarity and visual grounding, while MedGemma produced fluent but occasionally less precise reports. CXR-RePaiR remained con- servative, with lower semantic overlap and grounding coverage.

Localization coverage remained high for most models, confirming that VICCA successfully grounds the majority of generated phrases even under distribution shift. Region-level reliability scores were also consistent across models, suggesting that the diffusion-based visual reliability assessment captures stable structural alignment independent of dataset-specific reporting styles.

M2Trans was excluded from this evaluation, as its publicly avail- able checkpoint failed to generate valid reports on IU-Xray. Adapting or retraining the model specifically for this dataset would introduce additional confounding factors and was therefore outside the scope of this analysis. Among the evaluated models, only R2Gen was trained on IU-Xray, while the remaining models operated in a zero-shot setting.

Taken together, these results demonstrate that while pathology- level metrics are highly sensitive to dataset-specific conventions, the proposed VICCA and MCSE framework provides a more robust and transferable assessment of report quality across datasets.

## 7.3. Ethical and regulatory considerations

The results of this study highlight several ethical risks associated with the deployment of radiology report generation systems. In partic- ular, models that produce fluent and confident narratives may induce over-trust, even when their reports contain unsupported or weakly grounded findings. Our analysis demonstrates that strong textual co- herence or pathology-level accuracy does not guarantee semantic fi- delity or reliable alignment with image evidence, underscoring the

risk of clinicians implicitly trusting well-formed text without sufficient verification.

Accountability and responsibility remain challenging in such set- tings. When errors arise from AI-generated reports, it is often unclear whether responsibility lies with the model developer, the deploying institution, or the clinician who relied on the output. By explicitly linking textual statements to visual evidence and quantifying semantic and grounding reliability, VICCA provides an audit-oriented frame- work that supports transparency and post-hoc analysis rather than autonomous decision-making. This distinction is particularly important for regulatory and clinical governance contexts, where explainability and traceability are prerequisites for responsible use.

Finally, performance disparities across datasets and pathology cate- gories reveal potential sources of bias. Models may underperform on rare conditions, specific reporting styles, or patient subgroups, even when aggregate metrics appear favorable. Reference-free evaluation frameworks such as VICCA should therefore be viewed as tools for sys- tematic auditing, model comparison, and risk assessment, rather than as substitutes for expert judgment. Used appropriately, such frameworks can help identify failure modes, guide model selection, and inform regulatory evaluation without displacing clinical responsibility.

## 8. Conclusion

The comparative evaluation of five representative radiology report generation (RRG) models demonstrates that no single system performs consistently well across all clinically relevant dimensions. Each model exhibits distinct strengths and weaknesses, with some excelling in pathology-level accuracy, others in semantic coherence or visual align- ment, yet none providing a uniformly reliable clinical interpretation. These findings underscore the limitations of single-metric evaluation and motivate the need for multidimensional frameworks that reflect the complexity of clinical reporting.

In this work, we employed the VICCA and MCSE frameworks as complementary tools to analyze RRG performance from both visual and semantic perspectives. This integrated evaluation reveals systematic trade-offs between pathology accuracy, semantic fidelity, and visual grounding, offering a more nuanced understanding of how current models succeed or fail in representing medical reality. Our results show that improving clinical trustworthiness requires coordinated progress across text generation, visual reasoning, and grounding, rather than isolated optimization of individual metrics.

Beyond benchmarking, this study establishes VICCA as a founda- tion for future evaluation and development of trustworthy medical reporting systems. Several directions naturally follow from this work. First, incorporating radiologists into the evaluation loop would enable structured human feedback on grounding quality, semantic adequacy, and failure modes, allowing VICCA to support hybrid human–AI au- diting rather than purely automated assessment. Second, combining
VICCA with uncertainty quantification techniques could help distinguish confident, well-supported findings from ambiguous or weakly grounded statements, providing an additional safeguard against over-trust in fluent model outputs. Third, extending the framework to other imaging modalities such as CT and MRI would allow investigation of whether the observed trade-offs between semantics, grounding, and reliability persist in volumetric and multi-sequence settings. Finally, VICCA may serve as a diagnostic tool for model-specific fine-tuning, enabling targeted improvements in grounding, narrative coherence, or pathology sensitivity based on identified weaknesses.

Looking forward, we believe that evaluation frameworks such as VICCA should evolve from passive benchmarking tools into active components of the model development and validation cycle. By explicitly linking textual claims to visual evidence and semantic consistency, VICCA provides a pathway toward more transparent, accountable, and clinically aligned AI systems. Ultimately, such evaluation-driven development is essential for the safe integration of report generation models into real-world radiology workflows.

## CRediT authorship contribution statement

**Sayeh Gholipour Picha:** Conceptualization, Methodology, Validation, Formal Analysis, Resources, Data curation, Writing – original draft, Visualization. **Dawood Al Chanti:** Conceptualization, Methodology, Resources, Writing – review & editing, Supervision, Project Administration, Funding Acquisition. **Alice Caplier:** Conceptualization, Methodology, Resources, Writing – review & editing, Supervision, Project Administration, Funding acquisition.

## Code repository

All experiments reported in this paper are reproducible using the publicly available implementation on GitHub at https://github.com/sayeh1994/vicca.git. In addition, we provide an online Hugging Face API that allows direct evaluation using the MCSE and VICCA pipelines at https://huggingface.co/spaces/sayehghp/vicca.

## Declaration of Generative AI and AI-assisted technologies in the writing process

During the preparation of this work, the author used the ChatGPT service in order to improve the readability and language of the work. After using this tool/service, the author reviewed and edited the content as needed and takes full responsibility for the content of the publication.

## Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Sayeh GHOLIPOUR PICHA reports financial support was provided by University Grenoble Alpes Grenoble Institute of Technology. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

## Acknowledgments

We would like to acknowledge that all the computations presented in this paper are performed using the infrastructure (https://gricad.univ-grenoble-alpes.fr), which is supported by Grenoble research communities. All authors have contributed significantly to this work and take responsibility for its content.

## Data availability

The authors do not have permission to share data.

## References

Akhtar, M. U., Liu, J., Xie, Z., Cui, X., Liu, X., & Huang, B. (2025). Multilingual entity alignment by abductive knowledge reasoning on multiple knowledge graphs. *Engineering Applications of Artificial Intelligence*, 139, Article 109660. http://dx.doi.org/10.1016/j.engappai.2024.109660.

Banerjee, S., & Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In *Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization* (pp. 65–72). Ann Arbor, Michigan: Association for Computational Linguistics.

Boecking, B., Usuyama, N., Bannur, S., Castro, D. C., Schwaighofer, A., Hyland, S., Wetscherek, M., Naumann, T., Nori, A., Alvarez-Valle, J., Poon, H., & Oktay, O. (2022). Making the most of text semantics to improve biomedical vision–language processing. In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, & T. Hassner (Eds.), *Computer vision – ECCV 2022* (pp. 1–21). Cham: Springer Nature Switzerland.

Chen, Z., Song, Y., Chang, T.-H., & Wan, X. (2020). Generating radiology reports via memory-driven transformer. In *Proceedings of the 2020 conference on empirical methods in natural language processing* (pp. 1439–1449). Online: Association for Computational Linguistics, http://dx.doi.org/10.18653/v1/2020.emlnp-main.112.

Endo, M., Krishnan, R., Krishna, V., Ng, A. Y., & Rajpurkar, P. (2021). Retrieval-based chest X-Ray report generation using a pre-trained contrastive language-image model. In *Proceedings of machine learning research: vol. 158, Proceedings of machine learning for health* (pp. 209–219).

Freitag, M., Mathur, N., Deutsch, D., Lo, C.-K., Avramidis, E., Rei, R., Thompson, B., Blain, F., Kocmi, T., Wang, J., Adelani, D. I., Buchicchio, M., Zerva, C., & Lavie, A. (2024). Are LLMs breaking MT metrics? Results of the WMT24 metrics shared task. In B. Haddow, T. Kocmi, P. Koehn, & C. Monz (Eds.), *Proceedings of the ninth conference on machine translation* (pp. 47–81). Miami, Florida, USA: Association for Computational Linguistics, http://dx.doi.org/10.18653/v1/2024.wmt-1.2.

Gholipour Picha, S., Al Chanti, D., & Caplier, A. (2025). VICCA: Visual interpretation and comprehension of chest X-ray anomalies in generated report without human feedback. *Machine Learning with Applications, 21*, Article 100684. http://dx.doi.org/10.1016/j.mlwa.2025.100684.

Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghoo, B., Ball, R., Shpanskaya, K., Seekins, J., Mong, D. A., Halabi, S. S., Sandberg, J. K., Jones, R., Larson, D. B., Langlotz, C. P., Patel, B. N., Lungren, M. P., & Ng, A. Y. (2019). CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. *Proceedings of the AAAI Conference on Artificial Intelligence, 33*(01), 590–597. http://dx.doi.org/10.1609/aaai.v33i01.3301590.

Jain, S., Agrawal, A., Saporta, A., Truong, S., Duong, D. N., Bui, T., Chambon, P., Zhang, Y., Lungren, M. P., Ng, A. Y., Langlotz, C., & Rajpurkar, P. (2021). RadGraph: Extracting clinical entities and relations from radiology reports. In *Thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 1)*.

Johnson, A. E. W., Pollard, T. J., Berkowitz, S. J., Greenbaum, N. R., Lungren, M. P., Deng, C.-y., Mark, R. G., & Horng, S. (2019). MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports. *Scientific Data*, 6(1), 317. http://dx.doi.org/10.1038/s41597-019-0322-0.

Johnson, A. E. W., Pollard, T. J., Greenbaum, N. R., Lungren, M. P., ying Deng, C., Peng, Y., Lu, Z., Mark, R. G., Berkowitz, S. J., & Horng, S. (2019). MIMIC-CXR-jpg, a large publicly available database of labeled chest radiographs.

Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. In *Text summarization branches out* (pp. 74–81). Barcelona, Spain: Association for Computational Linguistics.

Liu, W., Xue, Y., Lin, C., & Boumaraf, S. (2025). IU X-ray dataset.

Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., Zhu, J., & Zhang, L. (2024). Grounding DINO: Marrying DINO with grounded pre-training for open-set object detection.

Miura, Y., Zhang, Y., Tsai, E., Langlotz, C., & Jurafsky, D. (2021). Improving factual completeness and consistency of image-to-text radiology report generation. In K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur, I. Beltagy, S. Bethard, R. Cotterell, T. Chakraborty, & Y. Zhou (Eds.), *Proceedings of the 2021 conference of the North American chapter of the association for computational linguistics: human language technologies* (pp. 5288–5304). Online: Association for Computational Linguistics, http://dx.doi.org/10.18653/v1/2021.naacl-main.416.

Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: A method for automatic evaluation of machine translation. In *Proceedings of the 40th annual meeting of the association for computational linguistics* (pp. 311–318). Philadelphia, Pennsylvania, USA: Association for Computational Linguistics, http://dx.doi.org/10.3115/1073083.1073135.

Pauli, A. B., Augenstein, I., & Assent, I. (2025). Mind the style gap: Meta-evaluation of style and attribute transfer metrics. In C. Christodoulopoulos, T. Chakraborty, C. Rose, & V. Peng (Eds.), *Findings of the association for computational linguistics: EMNLP 2025* (pp. 21550–21564). Suzhou, China: Association for Computational Linguistics, http://dx.doi.org/10.18653/v1/2025.findings-emnlp.1175.
Picha, S. G., Chanti, D. A., & Caplier, A. (2024). Semantic textual similarity assessment in chest X-ray reports using a domain-specific cosine-based metric. In *Proceedings of the 17th international joint conference on biomedical engineering systems and technologies - volume 1: bIOINFORMATICS* (pp. 487–494). INSTICC, SciTePress, http://dx.doi.org/10.5220/0012429600003657.

Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A neural framework for MT evaluation. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), *Proceedings of the 2020 conference on empirical methods in natural language processing* (pp. 2685–2702). Online: Association for Computational Linguistics, http://dx.doi.org/10.18653/v1/2020.emnlp-main.213.

Sellergren, A., Kazemzadeh, S., Jaroensri, T., Kiraly, A., Traverse, M., Kohlberger, T., Xu, S., Jamil, F., Hughes, C., Lau, C., et al. (2025). MedGemma technical report. arXiv preprint arXiv:2507.05201.

Smit, A., Jain, S., Rajpurkar, P., Pareek, A., Ng, A., & Lungren, M. P. (2020). CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In *Conference on empirical methods in natural language processing*.

Tanida, T., Müller, P., Kaissis, G., & Rueckert, D. (2023). Interactive and explainable region-guided radiology report generation. In *2023 IEEE/CVF conference on computer vision and pattern recognition* (pp. 7433–7442). http://dx.doi.org/10.1109/CVPR52729.2023.00718.

Vedantam, R., Zitnick, C. L., & Parikh, D. (2015). CIDER: Consensus-based image description evaluation. In *2015 IEEE conference on computer vision and pattern recognition* (pp. 4566–4575). http://dx.doi.org/10.1109/CVPR.2015.7299087.

Yu, F., Endo, M., Krishnan, R., Pan, I., Tsai, A., Reis, E. P., Fonseca, E. K. U. N., Lee, H. M. H., Abad, Z. S. H., Ng, A. Y., Langlotz, C. P., Venugopal, V. K., & Rajpurkar, P. (2023). Evaluating progress in automatic chest X-ray radiology report generation. *Patterns*, 4(9), Article 100802. http://dx.doi.org/10.1016/j.patter.2023.100802.

Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). BERTScore: Evaluating text generation with BERT. ArXiv, abs/1904.09675.