Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

## Abstract

Abstract We propose a two-stage multimodal framework that enhances disease classification and region-aware radiology report generation from chest X-rays, leveraging the MIMIC-Eye dataset. In the first stage, we introduce a gaze-guided contrastive learning architecture for disease classification. It integrates visual features, clinical labels, bounding boxes, and radiologist eye-tracking signals and is equipped with a novel multi-term gaze-attention loss combining MSE, KL divergence, correlation, and center-of-mass alignment. Incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%), while also improving precision and recall, highlighting the effectiveness of gaze-informed attention supervision. In the second stage, we present a modular report generation pipeline that extracts confidence-weighted diagnostic keywords, maps them to anatomical regions using a curated dictionary constructed from domain-specific priors, and generates region-aligned sentences via structured prompts. This pipeline improves report quality as measured by clinical keyword recall and ROUGE overlap. Our results demonstrate that integrating gaze data improves both classification performance and the interpretability of generated medical reports.

## 1 Introduction

Radiology reports anchor clinical decision making: treatment plans, follow-up imaging, and even surgical interventions often depend on the language a radiologist chooses to record (Casey et al., 2021; Liu et al., 2019). Accordingly, report-generation systems must be precise as well as capturing subtle pathologies and be explainable, so that every statement can be traced back to verifiable image evidence (Tanida et al., 2023). Producing such reports automatically from chest X-rays is therefore both a high-impact goal and a stringent test of multi-modal reasoning (Yang et al., 2023).

Yet the underlying data are stubbornly heterogeneous. Pixel-level visual cues, sentence-level textual descriptions, and time-stamped attentional traces collected via eye-tracking each operate on different scales and carry different noise profiles (Karargyris et al., 2021; Lanfredi et al., 2022). Aligning these modalities is complicated by (i) reader-specific gaze patterns, (ii) limited bounding-box coverage, and (iii) the need to express findings in radiologist-approved terminology. A successful solution must fuse all three signals without diluting any one of them (Ma et al., 2024).

this end, we present four contributions. An overview of the pipeline is shown in Figure 1.

- •
Fixation-Guided Contrastive Learning for Disease Prediction.
We pair dual-view contrastive learning with a novel composite gaze-supervision loss that unifies pixel-wise fidelity (Mean Squared Error (MSE), Kullback-Leibler (KL) Divergence ) with pattern-aware similarity (Pearson correlation) and geometric alignment via a normalized center-of-mass term, under quality-adaptive weighting by fixation density/quality. To our knowledge, this is the first unified objective in radiology gaze supervision to jointly optimize these four facets, closing spatial-precision and center-misalignment gaps of prior pixel-wise losses and yielding consistent gains.
- •
Quantitative Gaze-Attention Validation. Through controlled ablations, we show that the model consistently attends to clinically relevant regions: Jensen-Shannon divergence between model and human fixation maps falls below 0.5, and correlation metrics improve systematically under gaze supervision.
- •
Region-Grounded, Keyword-Based Report Generation. The report-generation stage converts classifier predictions into coherent, region-aware radiology reports by (1) extracting confidence-weighted diagnostic keywords, (2) aligning them to canonical anatomical regions, and (3) emitting region-conditioned sentences via a prompt and Large Language Model (LLM) based approach.
- •
A Modular, Interpretable Pipeline Built on MIMIC-Eye. All components are modular and ready to use, built entirely on publicly available data. Intermediate outputs are explicitly retained, ensuring transparency, reproducibility, and clinical inspectability.

Figure: Figure 1: Overview of the proposed multi-modal pipeline. Images, bounding-box masks, free-text reports, and eye-tracking fixations are aligned through a contrastive objective; downstream modules generate region-grounded radiology reports.
Refer to caption: x1.png

## 2 Related Work

Recent medical vision-language models couple large-scale chest-X-ray corpora with transformer backbones to align image embeddings and report tokens (Lu and Wang, 2025; You et al., 2023). Approaches such as MedCLIP (Wang et al., 2022), BioViL (Bannur et al., 2023), and Llama-Med pre-train with paired (image, sentence) contrastive objectives and subsequently fine-tune for tagging or report generation, demonstrating strong zero-shot transfer to unseen pathologies (Zhang et al., 2023). Although effective at global alignment, these methods operate on whole-image/whole-sentence pairs and provide limited guidance on where in the image a predicted phrase originates (Li et al., 2025). Parallel efforts leverage gaze traces as an auxiliary supervisory signal: fixation maps are injected either as soft attention masks or as auxiliary channels, encouraging the encoder to focus on diagnostically salient regions without requiring extra pixel-level labels (Ma et al., 2024; Wang et al., 2024).

Complementary to vision-language alignment, keyword and region-
aware generators explicitly ground narrative statements in anatomic sub-regions (Tanida et al., 2023; Chen et al., 2024). A complementary direction prompts LLMs with gaze and region cues to steer generation without retraining (Kim et al., 2025). Pipelines such as MS-CXR (Boecking et al., 2024) and REFLACX (Lanfredi et al., 2022) first predict diagnostic keywords, then slot them into structured prompt conditioned on pre-computed bounding boxes, yielding reports with higher factual correctness. These frameworks, however, depend on accurate region detectors and omit attentional cues. Finally, contrastive and multimodal fusion techniques combine heterogeneous inputs; images, clinical labels, bounding boxes, and gaze sequences within a unified representation space; InfoNCE (Information Noise-Contrastive Estimation) style losses balance the modalities while late-fusion transformers aggregate their features (Liu et al., 2021; Hayat et al., 2022). Our work intersects these strands by integrating gaze-guided contrastive learning with a region-grounded, keyword-driven generator, thereby coupling attentional supervision with spatially explicit report generation.

## 3 Methodology

### 3.1 Overview and Motivation

Our goal is to build a multimodal fusion architecture that integrates four complementary information sources encountered during chest X-ray interpretation: (1) the chest X-ray; (2) binary masks that mark anatomically defined bounding boxes; (3) the radiologist’s transcription; and (4) eye-tracking fixation sequences recorded during the radiologist’s reading. We promote cross-modal alignment by applying an InfoNCE loss to selected modality pairs; specifically image-text and image -fixation embeddings; while the primary supervision remains a multi-label disease-classification objective. This contrastive coupling encourages each modality-specific encoder to produce features that are mutually informative, yet robust to missing or noisy channels.

Unlike prior pipelines, we incorporate eye-tracking data as an auxiliary supervisory signal, offering fine-grained attentional guidance without explicit localization labels. Pre-rendered bounding box masks, introduce broad spatial priors without requiring dense supervision; bridging the gap between image space and semantic concepts. Together, these components yield features that transfer well across datasets and support interpretable downstream generation tasks.

### 3.2 Input Modalities and Contrastive Learning

Figure 2 gives an overview of the four encoders. Unless noted otherwise, all projected feature vectors have dimensionality $d=768$.

##### Radiograph ( img ).

Each $224\times 224$ radiograph is processed by a ViT-base (Dosovitskiy et al., 2020) backbone that has been transfer-learned on CheXpert (Irvin et al., 2019). From the class token embedding (CLS) token $\mathbf{z}_{\text{img}}\in{R}^{768}$ we derive the image embedding with a *three-layer* projection head:

$$ $\mathbf{h}_{\text{img}}=\mathrm{LN}\left(W_{2}\,\mathrm{GELU}\left(\mathrm{LN}(W_{1}\mathbf{z}_{\text{img}})\right)\right),$ (1) $$

where $W_{1},W_{2}$ are learned linear maps, $\mathrm{LN}$ denotes Layer Normalization and $\mathrm{GELU}$ denotes Gaussian Error Linear Unit. A Dropout (0.15) layer follows each linear transformation in the high-capacity (“enhanced”) configuration.

##### Bounding-box mask ( bbox ).

For each study, we pre-render a binary $224\times 224$ mask whose ones mark the union of all reader-supplied bounding boxes. The mask is first flattened and $\ell_{2}$-normalized, then passed through a single linear projection (512 $\rightarrow$ $d$), followed by $\mathrm{LN}$ and GELU, to obtain the embedding $\mathbf{h}_{\text{bbox}}$.

Figure: Figure 2: Multimodal chest-X-ray classifier: image, bounding-box, text, and gaze features are independently encoded, concatenated, refined by a cross-modal attention block, and passed through an MLP fusion network whose global and condition-specific heads are ensembled to predict eight disease labels.
Refer to caption: x2.png

##### Report text ( text ).

Transcripts are encoded by Bio-ClinicalBERT (Alsentzer et al., 2019). We freeze the backbone and project the [CLS] representation with

$$ $\mathbf{h}_{\text{text}}=W_{4}\,\mathrm{GELU}\left(\mathrm{LN}(W_{3}\mathbf{z}_{\text{text}})\right),$ (2) $$

followed by $\mathrm{LN}$ and Dropout, matching the implementation.

##### Fixation sequence ( gaze ).

A variable-length sequence of $(x,y,\Delta t,\text{pupil})$ tuple is first embedded to 64 dimensions and then encoded by a bi-directional gated recurrent unit (bi-GRU). The enhanced model employs two layers with hidden size 384, giving a 1,536-dimensional concatenated state that is linearly projected to $\mathbf{h}_{\text{gaze}}$. All embeddings are $\ell_{2}$-normalised. During training, we randomly drop an entire modality with probability $p=0.1$ to promote robustness.

#### 3.2.1 Gaze -Model Variants: Baseline vs. Enhanced

We instantiate two configurations that differ solely in how they exploit fixation data.

##### Baseline-gaze.

Fixation tuples are embedded and summarised by the Bi-GRU; the resulting
vector $h_{\text{gaze}}$ is concatenated with the other modality features (no spatial
attention is produced). To enable fair comparison, we still allocate an
“attention-map” head, but force it to output a $224{\times}224$ zero tensor, so the
baseline never receives gradient signal from the gaze-specific losses.

##### Enhanced-gaze.

The enhanced variant adds four mechanisms: (1) Explicit attention maps. A lightweight decoder transforms fusion features into a spatial map $A_{\text{model}}\in\mathbb{R}^{224\times 224}$, indicating where the network “looks”; (2) Human supervision. Raw fixations are converted into a smoothed heat-map $A_{\text{gaze}}$, which supervises $A_{\text{model}}$ via the
multi-term loss in Eq. (5); (3) Gaze -text alignment. An InfoNCE loss couples $h_{\text{gaze}}$ and $h_{\text{text}}$ whenever both modalities are present, encouraging semantic consistency between viewed regions and mentioned findings; and (4) Multi-scale attention. The decoder predicts attention at $224^{2}$, $112^{2}$, and $56^{2}$ resolutions, and the loss in Eq. (5) is computed at each scale with shared weights. Together, these additions train the network to emulate radiologists’ visual search patterns rather than merely consume fixation statistics as auxiliary features.

### 3.3 Fusion and Prediction

Embeddings are concatenated to a 3,072-dimensional token and passed to a two-layer MLP fusion block ($3{,}072\rightarrow 1{,}536\rightarrow 768$, GELU, Dropout 0.15). When GPU memory allows, we prepend a single multi-head cross-modal attention layer (8 heads) that refines each modality with information from the others. We set $\alpha=0.7$ to balance dominant global cues with fine-grained class heads, boosting minority-class recall while mitigating overfitting. The fused vector feeds a global classifier and eight one-vs-rest heads; the final logit for condition $c$ is

$$ $\ell_{c}=\alpha\cdot\ell^{\text{global}}_{c}+(1-\alpha)\cdot\ell^{\text{spec}}_{c},$ (3) $$

### 3.4 Training Objectives

##### (1) Classification Loss.

We use focal loss with class-balanced positive weights.

##### (2) InfoNCE contrastive loss.

For image -gaze and image -text pairs we minimize the InfoNCE objective (van den Oord et al., 2019), derived from noise-contrastive estimation (Gutmann and Hyvärinen, 2012):

$$ $\mathcal{L}_{\text{NCE}}=-\frac{1}{|P|}\sum_{(i,j)\in P}\log\frac{\exp(\operatorname{sim}(\mathbf{h}_{i},\mathbf{h}_{j})/\tau)}{\sum_{k}\exp(\operatorname{sim}(\mathbf{h}_{i},\mathbf{h}_{k})/\tau)},$ (4) $$

where $\operatorname{sim}(\mathbf{u},\mathbf{v})=\frac{\mathbf{u}^{\mathsf{T}}\mathbf{v}}{\lVert\mathbf{u}\rVert\lVert\mathbf{v}\rVert}$ and $\tau=0.07$.

##### (3) Proposed Enhanced gaze-attention loss.

Let $A^{\text{model}}$ be the $224\times 224$ spatial attention predicted from the fused token and $A^{\text{gaze}}$ the fixation-derived heat map. The multi-component loss is:

$$ $\displaystyle\mathcal{L}_{\text{gaze}}$ $\displaystyle=w_{q}\Big{(}\underbrace{\lVert A^{\text{model}}-A^{\text{gaze}}\rVert_{2}^{2}}_{\mathcal{L}_{\text{MSE}}}+\underbrace{\operatorname{KL}\big{(}\sigma(A^{\text{gaze}})\parallel\sigma(A^{\text{model}})\big{)}}_{\mathcal{L}_{\text{KL}}}+\underbrace{1-\rho(A^{\text{model}},A^{\text{gaze}})}_{\mathcal{L}_{\text{corr}}}+\underbrace{\frac{\lVert\operatorname{CoM}(A^{\text{model}})-\operatorname{CoM}(A^{\text{gaze}})\rVert_{2}}{\sqrt{224^{2}+224^{2}}}}_{\mathcal{L}_{\text{CoM}}}\Big{)},$ $$

where $w_{q}=\sqrt{N_{\text{fix}}}\cdot q_{\text{score}}$ weights each sample by fixation quality.

##### (4) Gaze -text contrastive loss.

We apply the same InfoNCE formulation to the projected gaze and text embeddings when both are present.

##### Total loss.

$$ $\mathcal{L}=\mathcal{L}_{\text{cls}}+\lambda_{1}\mathcal{L}_{\text{NCE}}+\lambda_{2}\mathcal{L}_{\text{gaze}}+\lambda_{3}\mathcal{L}_{\text{gaze--text}},$ (5) $$

with $(\lambda_{1},\lambda_{2},\lambda_{3})=(0.1,0.3,0.15)$ to prioritize gaze-based spatial grounding (0.3), refine cross-modal semantics via gaze -text alignment (0.15), and keep NCE as a stabilizing regularizer (0.1).
All experiments: AdamW (LR $6{\times}10^{-6}$), batch $32$ ($8$ low-mem), $40$ epochs, cosine; “Fine Tune” folds validation into training for a final pass (test unchanged; hyperparameters fixed). For baselines: $5{\times}10^{-6}$, $32$, $35$; ViT-only: $5{\times}10^{-5}$, $128$, $20$. Experiments ran on an Intel Core i9-14900K CPU and a single NVIDIA RTX 4090 GPU (24 GB VRAM).

### 3.5 Two-Stage Keyword Extraction Pipeline

We extract keywords per condition in two steps. Stage 1: Gemini 2.5 Pro reads the full report and the eight target pathologies, then proposes a ranked list for each condition (temperature=$0.1$, $top-k=1$). Requests run in mini-batches of 30 with exponential backoff (base 3 s; timeout 120 s) and progress logging. Stage 2: A second Gemini pass filters the candidates by removing lexical variants, boilerplate (e.g., “no evidence of”), duplicates, and cross-condition leakage. It outputs a simple YES/NO per keyword; confidence is suppressed so decisions rely on semantic meaning, reducing bias from overconfident errors. On the development set (7,322 keywords), 49.5% are kept and 50.5% dropped. The final vocabulary is compact and precise (about $390\pm 230$ unique keywords per condition; e.g., Atelectasis 164, Lung Opacity 782, Support Devices 683) and is used for anatomical-region matching and structured report generation.

### 3.6 Anatomical Region Mapping and Report Generation

We maintain a dictionary of 17 thoracic regions, each represented by a bounding-box tuple $(x_{\min},y_{\min},x_{\max},y_{\max})$ and a list of lexical aliases. During training, EyeGaze/REFLACX boxes are normalized to $[0,1]$ using the recorded image dimensions $(W,H)$, validated, and robustly aggregated per region; at inference, Gemini-cleaned keywords are matched (case-insensitive, fuzzy similarity) to the alias lists. Successful matches activate binary region flags, yielding a sparse 17-dimensional anatomical mask shared across image, gaze, and text streams. The normalized region bounds are later scaled to a $512{\times}512$ dimension for mask rendering and visualization.

##### Normalization, validation, and robust averaging.

For each annotation table we (i) normalize coordinates, (ii) apply a validation gate requiring $0\leq x_{1}<x_{2}\leq 1$, $0\leq y_{1}<y_{2}\leq 1$, known $(W,H)$, a valid region identifier, and a confidence score (for REFLACX); and (iii) aggregate all valid boxes per region by an element-wise median. If the median is degenerate (non-positive width/height or area $<\tau_{\text{box}}$), we fall back to a confidence-weighted average, with $b_{i}\!\in\![0,1]^{4}$ and confidences $c_{i}$ (default $c_{i}{=}1$ if absent), returning a mapping $M[r]\!\in\![0,1]^{4}$ for each region $r$. The complete procedure is summarized in Algorithm [1](https://arxiv.org/html/2508.13068v1#alg1).

Figure: Algorithm 1 Anatomical Region Bounds Aggregation

##### Report generation.

The classifier outputs posterior probabilities for eight target pathologies. Conditions with $p(c)>0.60$ and their activated regions are passed to a Gemini 2.5 Pro prompt that (temperature=$0.3$, top-$k{=}1$) injects regional context, enforces radiology style, and produces distinct findings and impression . The prompt includes strict instructions to avoid unsupported statements. API calls use up to five retries with exponential backoff; on final failure, a concise local fallback paragraph is emitted. We serialize per-condition probabilities, matched keyword sources, and contributing region indices to support interpretability dashboards. This replaces the earlier (unused) phrase-level provenance scheme and preserves the high-recall region mapping while leveraging LLM’s fluency to produce coherent, anatomically faithful reports without rigid prompts.

## 4 Evaluations

### 4.1 Dataset curation and alignment process

##### Dataset.

We curate a task-specific subset of MIMIC-Eye v1.0.0 to obtain a fully aligned, multimodal corpus that supports both gaze-aware detection and region-grounded report generation. The source archive couples 3,689 posterior–anterior chest radiographs with two heterogeneous eye-tracking streams:

- •
EyeGaze   High-frequency binocular gaze, automatically generated bounding boxes for seventeen thoracic regions, and single-reader audio transcripts.
- •
REFLACX   Radiologist fixations, spoken descriptions, and free-hand lesion ellipses but no anatomical region masks.

Coverage across modalities is uneven. Modality Coverage Analysis Of the 3,689 source studies, 3,502 (94.9%) contain valid radiographs, 3,445 (93.4%) provide usable gaze sequences, 3,398 (92.1%) include transcripts, and 1,847 (50.1%) provide complete bounding-box annotations from EyeGaze; every REFLACX study lacks region masks entirely, necessitating computational completion.

A small number of radiographs are unreadable owing to truncated JPEGs; several EyeGaze sessions contain malformed gaze tables or mismatched identifiers; and every REFLACX study lacks region masks altogether. Our curation procedure therefore proceeds in three stages.

Integrity filtering. We discard studies with corrupt images or invalid gaze logs, retaining only cases that provide a valid radiograph and at least one usable fixation sequence. Quality Assurance Metrics. Specifically, we exclude 184 studies (4.99%): 67 corrupted images, 89 malformed gaze tables, and 28 missing identifiers. Post-filtering, the retained corpus achieves 99.2% image validity, 97.8% gaze-sequence completeness, and 100% transcript availability.

Fixation normalisation. EyeGaze coordinates are already screen-normalised. REFLACX pixel coordinates are mapped to the unit square by dividing by the recorded image crop, yielding a common $(x,y)\in[0,1]^{2}$ reference frame. Pupil area is harmonised by converting left- and right-eye diameters to area and scaling by the subject-specific mean of the first two valid seconds, matching the relative scale used in REFLACX.

Normalization Validation. Cross-dataset alignment achieves a spatial correlation of $r=0.94$ between EyeGaze and REFLACX normalised fixations, while pupil-area scaling reduces inter-subject variance by 73.2%, with standardised areas spanning $[0.1,2.8]$ relative units.

Bounding-box completion. EyeGaze region boxes are kept as-is. To compensate for their absence in REFLACX, we train a lightweight YOLO on EyeGaze annotations and infer one highest-confidence box per region for every REFLACX image.

The resulting corpus supplies, for every retained study:

(1) a radiograph; (2) a normalised fixation sequence with per-sample pupil area and duration; (3) a complete set of seventeen thoracic region masks; (4) the original radiology-report text; and (5) CheXpert-style condition labels.

Quantitative Results. Our pipeline processes the 3,689 initial radiographs and produces 2,877 fully-aligned multimodal samples, yielding a 67.1% retention rate. We partition the dataset patient-wise into 1,984 training samples (80.1%), 493 validation samples (19.9%), and 400 test samples (16.2%). The curated dataset exhibits moderate class imbalance: No Finding (38.2%), Lung Opacity (23.1%), Support Devices (18.7%), Atelectasis (14.2%), Cardiomegaly (13.6%), Pleural Effusion (12.9%), Edema (11.8%), and Pneumonia (9.4%). Multi-label cases constitute 42.7% of the corpus, with a mean label density of $1.67\pm 0.92$ conditions per study. All preprocessing scripts and the manifest that link gaze, images, region masks, and clinical labels will be released to facilitate reproducibility.

### 4.2 Disease Classification and Gaze-Attention Evaluation

##### Modality Ablation Results.

**Table 1: Signal classification performance comparison across input modalities. Each row denotes a metric, and each column corresponds to a modality combination.**
|  | Modalities |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Metrics | Images,<br>Labels, BBox | +<br>Transcription | +<br>Fixations | + Fixation<br>Enhanced | + Fine Tune<br>(Val$\rightarrow$Train, $n=493$) |
| AUC | 0.821 | 0.822 | 0.834 | 0.823 | 0.849 |
| F1 | 0.579 | 0.597 | 0.622 | 0.597 | 0.631 |
| Recall | 0.673 | 0.703 | 0.702 | 0.703 | 0.726 |
| Precision | 0.509 | 0.519 | 0.559 | 0.519 | 0.557 |
| Loss | 0.491 | 2.126 | 0.537 | 2.126 | 2.034 |
| Pearson Correlation | 0.198 $\pm$ 0.198 | 0.229 $\pm$ 0.194 | 0.225 $\pm$ 0.173 | 0.237 $\pm$ 0.191 | 0.253 $\pm$ 0.174 |
| MSE | 0.045 $\pm$ 0.016 | 0.044 $\pm$ 0.015 | 0.044 $\pm$ 0.015 | 0.043 $\pm$ 0.015 | 0.043 $\pm$ 0.014 |
| P Value | 0.005 $\pm$ 0.042 | 0.004 $\pm$ 0.037 | 0.004 $\pm$ 0.039 | 0.004 $\pm$ 0.034 | 0.006 $\pm$ 0.053 |
| Jensen -Shannon Divergence | 0.464 $\pm$ 0.082 | 0.455 $\pm$ 0.080 | 0.456 $\pm$ 0.080 | 0.451 $\pm$ 0.079 | 0.446 $\pm$ 0.077 |
| Normalized Scanpath Saliency | 0.109 $\pm$ 0.048 | 0.118 $\pm$ 0.047 | 0.044 $\pm$ 0.015 | 0.123 $\pm$ 0.046 | 0.134 $\pm$ 0.045 |
| Human Attention Entropy | 9.873 $\pm$ 0.323 | 9.884 $\pm$ 0.319 | 9.898 $\pm$ 0.313 | 9.898 $\pm$ 0.313 | 9.898 $\pm$ 0.313 |
| Model Attention Entropy | 10.561 $\pm$ 0.095 | 10.577 $\pm$ 0.093 | 10.549 $\pm$ 0.089 | 10.562 $\pm$ 0.091 | 10.584 $\pm$ 0.067 |

Table 1 reveals four key findings. First, the baseline modality (images, labels, and bounding boxes) yields strong results (AUC = 0.821, F1 = 0.579), validating the utility of spatial priors. Second, adding transcriptions slightly increases performance (AUC = 0.822, F1 = 0.597). Third, incorporating raw fixations improves both AUC (0.834) and F1 (0.622), confirming their value as weak supervision. When fixations are used as explicit spatial supervision (*Fixation-Enhanced*), performance remains comparable (AUC = 0.823, F1 = 0.597), but yields interpretable attention maps. Finally, LLM hallucinations remain possible; mitigation beyond prompt grounding and thresholding is left to future work. Fine-tuning on 493 validation samples restores peak performance (AUC = 0.849, F1=0.631), indicating that even limited human annotations substantially enhance model utility. As illustrated in Figure 3, the predicted saliency closely mirrors expert fixations, visually corroborating the quantitative alignment metrics reported above. All reported test results use image-only inference; other modalities are training-time signals only.

For *Fixation-Enhanced*, six complementary metrics quantify model -gaze alignment. Pearson correlation is $0.237\pm 0.191$, which meets Cohen’s moderate threshold ($r\geq 0.30$) (Cohen, 1988). MSE is $0.043\pm 0.015$, and Jensen -Shannon divergence is $0.451\pm 0.079$, close to the inter-reader upper bound of 0.45 (Bylinskii et al., 2019). Normalized Scanpath Saliency (NSS) is $0.123\pm 0.046$; below the human-alignment threshold of 1.0 (Peters and Itti, 2005; Bylinskii et al., 2019), but still indicative of saliency correlation. Human attention entropy is $9.898\pm 0.313$, while model entropy is $10.562\pm 0.091$, both consistent with the typical 9 -11 bit range in clinical gaze studies (Zhang and et al., 2024). These results, reported as $\mu\pm\sigma$, align closely with inter-reader statistics from MIMIC-Eye (Hsieh et al., 2023), confirming that gaze-supervised contrastive learning yields interpretable attention without sacrificing diagnostic utility. For percentile-based P-scores, values $>0.50$ imply fixation -saliency concentration (Riche et al., 2013).

Figure: Figure 3: Human -model attention alignment. Radiologist fixation map (left) vs. *Fixation-Enhanced* saliency (right) on the same case; matching hotspots (lower right lung, paracardiac) show gaze-guided supervision targets clinically relevant regions.
Refer to caption: images/gaze_attention_comparison.png

##### Per-Condition Performance.

Table 2 reports precision, recall, F1, and AUC per class. The model is strongest on No Finding (F1 = 0.74, AUC = 0.85; 168 cases), as expected given the larger support and more homogeneous appearance. Among pathologies, Support Devices and Pleural Effusion are the most balanced (F1 = 0.62/0.60; AUC $\approx$ 0.90), benefitting from distinctive, high-contrast cues (e.g., lines/tubes, costophrenic blunting). Edema achieves the highest recall (0.81) but modest precision (0.38; F1 = 0.51), indicating sensitivity to diffuse opacities with confusion against Atelectasis/Lung Opacity. Atelectasis and Pneumonia show lower precision (0.29/0.26) and the lowest AUC for Pneumonia (0.69), reflecting small supports (56/43) and substantial visual overlap in a weakly supervised setting. In safety-critical radiology, high *recall* is preferred to minimize missed pathology; our macro recall of 0.73 satisfies this priority while maintaining competitive macro AUC 0.85 and macro F1 0.63. The per-class precision gaps are attributable primarily to (i) class imbalance and limited examples, (ii) ambiguity among consolidation-like findings, and (iii) coarse label granularity, rather than overfitting. Practical remedies include per-class threshold calibration at fixed sensitivity, cost-sensitive tuning with harder negative mining, additional region-level supervision, and expanding minority-class data. The macro averages (precision 0.56, recall 0.73, F1 0.63, AUC 0.85) match the ’Fine Tune’ ablation denotes a final pass in which the 493-case validation split is merged into the training set; the held-out test set remains unchanged, and all hyperparameters are fixed. It indicates stable global behaviour while delivering clinically meaningful discrimination across most findings. External state-of-the-art comparisons are omitted because, to our knowledge, no prior work reports radiology report generation results on the integrated MIMIC-Eye dataset; existing gaze-aware or report-generation systems evaluate on other datasets/splits with different label spaces and protocols.

**Table 2: Test-set metrics per condition on MIMIC, using CheXpert-based transfer learning.**
| Condition | Precision | Recall | F1 | Accuracy | Support | AUC |
| --- | --- | --- | --- | --- | --- | --- |
| Atelectasis | 0.29 | 0.66 | 0.40 | 0.73 | 56 | 0.79 |
| Cardiomegaly | 0.36 | 0.62 | 0.45 | 0.73 | 71 | 0.78 |
| Edema | 0.38 | 0.81 | 0.51 | 0.82 | 48 | 0.89 |
| Lung Opacity | 0.36 | 0.65 | 0.46 | 0.67 | 88 | 0.72 |
| No Finding | 0.71 | 0.77 | 0.74 | 0.77 | 168 | 0.85 |
| Pleural Effusion | 0.48 | 0.78 | 0.60 | 0.81 | 73 | 0.90 |
| Pneumonia | 0.26 | 0.42 | 0.32 | 0.81 | 43 | 0.69 |
| Support Devices | 0.51 | 0.80 | 0.62 | 0.84 | 65 | 0.90 |
| Macro Avg | 0.56 | 0.73 | 0.63 | 0.81 | 400 | 0.85 |

### 4.3 Evaluation of Report Generation Quality

**Table 3: Report generation performance comparison across LLMs, reported as mean $\pm$ standard deviation.**
|  | LLM Models |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Metric | Gemini<br>2.5 Pro | LLaMA 4<br>Scout-17B | MedGemma<br>27B-IT | BioMistral<br>7B-DARE | Qwen 3<br>32B |
| BLEU | 0.092 $\pm$ 0.089 | 0.049 $\pm$ 0.049 | 0.078 $\pm$ 0.078 | 0.061 $\pm$ 0.049 | 0.047 $\pm$ 0.042 |
| ROUGE | 0.245 $\pm$ 0.118 | 0.172 $\pm$ 0.081 | 0.221 $\pm$ 0.109 | 0.203 $\pm$ 0.081 | 0.176 $\pm$ 0.071 |
| METEOR | 0.316 $\pm$ 0.131 | 0.324 $\pm$ 0.116 | 0.321 $\pm$ 0.136 | 0.291 $\pm$ 0.118 | 0.325 $\pm$ 0.111 |
| BERT Score | 0.732 $\pm$ 0.073 | 0.683 $\pm$ 0.068 | 0.693 $\pm$ 0.072 | 0.707 $\pm$ 0.062 | 0.680 $\pm$ 0.058 |
| CHEXPERT F1 | 0.546 $\pm$ 0.237 | 0.526 $\pm$ 0.212 | 0.538 $\pm$ 0.221 | 0.537 $\pm$ 0.205 | 0.528 $\pm$ 0.208 |
| RADGRAPH F1 | 0.129 $\pm$ 0.134 | 0.118 $\pm$ 0.109 | 0.132 $\pm$ 0.136 | 0.090 $\pm$ 0.093 | 0.091 $\pm$ 0.088 |
| MEDICAL Score | 0.353 $\pm$ 0.118 | 0.278 $\pm$ 0.082 | 0.334 $\pm$ 0.115 | 0.309 $\pm$ 0.112 | 0.274 $\pm$ 0.082 |

Table 3 reports scores on 400 test reports across five LLMs. Surface-overlap metrics remain low across models, e.g., for Gemini 2.5 Pro: BLEU $0.092\pm 0.089$, ROUGE $0.245\pm 0.118$, METEOR $0.316\pm 0.131$; reflecting paraphrasing and omission of low-frequency terms (Papineni et al., 2002; Lin, 2004; Banerjee and Lavie, 2005). Gemini achieves the strongest semantic similarity (BERT Score $0.732\pm 0.073$) (Zhang* et al., 2020) and the best discourse coherence (MEDICAL Score $0.353\pm 0.118$) (Deutsch et al., 2023); METEOR is marginally higher for Qwen 3 32B ($0.325\pm 0.111$) and LLaMA 4 Scout-17B ($0.324\pm 0.116$). Clinically, Gemini leads on CheXpert F1 ($0.546\pm 0.237$); a mid-tier “Fair” label agreement (Irvin et al., 2019); while MedGemma 27B-IT attains the highest RadGraph-F1 ($0.132\pm 0.136$) with Gemini close behind ($0.129\pm 0.134$), indicating lingering gaps in fine-grained entity/relation grounding (Jain et al., 2021). Overall, the results point to good semantic fidelity but limited phrase-level factual alignment, motivating structured prompts with RadGraph entities, relation-aware decoding, and/or RL fine-tuning to raise RadGraph-F1 and MEDICAL without sacrificing diversity.

## 5 Conclusion

We presented a two-stage multimodal framework that unifies visual, textual, spatial, and attentional cues for chest-X-ray interpretation. Stage 1 employs gaze-guided contrastive learning with a composite attention loss, yielding macro AUC 0.85, macro F1 0.63, and macro recall 0.73, while improving human -model attention alignment (Pearson $\approx$ 0.25). Stage 2 converts classifier outputs into region-grounded reports via a keyword -anatomy dictionary; the LLM attains BERTScore 0.73, CheXpert F1 0.546, RadGraph-F1 0.129, and MEDICAL 0.353, indicating good semantic fidelity with remaining gaps in phrase-level factual grounding.

Limitations and outlook.
Our aligned split contains 2,877 studies, motivating evaluation on larger, multi-center datasets for broader generalizability. REFLACX lacks region masks, so synthetic YOLO boxes may bias spatial supervision. Eye-tracking data is uncommon; although the baseline functions without gaze, the enhanced model’s full benefits depend on this signal. We evaluated five LLMs for report generation: Gemini 2.5 Pro, LLaMA 4 Scout-17B, MedGemma 27B-IT, BioMistral 7B-DARE, and Qwen 3 32B, including open-weight models to mitigate reproducibility concerns. Gemini led on semantic/coherence metrics (BERT Score $0.732\pm 0.073$, MEDICAL $0.353\pm 0.118$), while MedGemma achieved the top RadGraph-F1 ($0.132\pm 0.136$); despite this, fine-grained factuality remains limited (RadGraph-F1 $\approx$ 0.13). Future work will scale data and add region-level/weak labels; calibrate per-class decision thresholds for high-recall clinical use; explore self-supervised gaze prediction to reduce hardware needs; and extend to CT and ultrasound. By coupling human attentional priors with region-aware generation, the framework moves toward transparent, clinically trustworthy AI report generating.

## References

- Casey et al. (2021)
Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, Víctor Suárez-Paniagua, Richard Tobin, William Whiteley, Honghan Wu, and Beatrice Alex.
A systematic review of natural language processing applied to radiology reports.
*BMC Medical Informatics and Decision Making*, 21(1):179, 2021.
ISSN 1472-6947.
doi:[10.1186/s12911-021-01533-7](https://doi.org/10.1186/s12911-021-01533-7).
URL [https://doi.org/10.1186/s12911-021-01533-7](https://doi.org/10.1186/s12911-021-01533-7).
- Liu et al. (2019)
Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi.
Clinically accurate chest x-ray report generation, 2019.
URL [https://arxiv.org/abs/1904.02633](https://arxiv.org/abs/1904.02633).
- Tanida et al. (2023)
Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert.
Interactive and explainable region-guided radiology report generation.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 7433–7442, June 2023.
- Yang et al. (2023)
Shuxin Yang, Xian Wu, Shen Ge, Zhuozhao Zheng, S. Kevin Zhou, and Li Xiao.
Radiology report generation with a learned knowledge base and multi-modal alignment.
*Medical Image Analysis*, 86:102798, 2023.
ISSN 1361-8415.
doi:[https://doi.org/10.1016/j.media.2023.102798](https://doi.org/https://doi.org/10.1016/j.media.2023.102798).
URL [https://www.sciencedirect.com/science/article/pii/S1361841523000592](https://www.sciencedirect.com/science/article/pii/S1361841523000592).
- Karargyris et al. (2021)
Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy T. Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A. Krupinski, and Mehdi Moradi.
Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai development.
*Scientific Data*, 8(1):92, 2021.
ISSN 2052-4463.
doi:[10.1038/s41597-021-00863-5](https://doi.org/10.1038/s41597-021-00863-5).
URL [https://doi.org/10.1038/s41597-021-00863-5](https://doi.org/10.1038/s41597-021-00863-5).
- Lanfredi et al. (2022)
Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F. Auffermann, Jessica Chan, Phuong-Anh T. Duong, Vivek Srikumar, Trafton Drew, Joyce D. Schroeder, and Tolga Tasdizen.
REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays.
*Scientific Data*, 9(1):350, 2022.
ISSN 2052-4463.
doi:[10.1038/s41597-022-01441-z](https://doi.org/10.1038/s41597-022-01441-z).
URL [https://doi.org/10.1038/s41597-022-01441-z](https://doi.org/10.1038/s41597-022-01441-z).
- Ma et al. (2024)
Chong Ma, Hanqi Jiang, Wenting Chen, Yiwei Li, Zihao Wu, Xiaowei Yu, Zhengliang Liu, Lei Guo, Dajiang Zhu, Tuo Zhang, Dinggang Shen, Tianming Liu, and Xiang Li.
Eye-gaze guided multi-modal alignment for medical representation learning.
In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, *Advances in Neural Information Processing Systems*, volume 37, pages 6126–6153. Curran Associates, Inc., 2024.
URL [https://proceedings.neurips.cc/paper_files/paper/2024/file/0b9536e186a77feff516893a5f393f7a-Paper-Conference.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/0b9536e186a77feff516893a5f393f7a-Paper-Conference.pdf).
- Lu and Wang (2025)
Yinbin Lu and Alan Wang.
Integrating language into medical visual recognition and reasoning: A survey.
*Medical Image Analysis*, 102:103514, 2025.
ISSN 1361-8415.
doi:[https://doi.org/10.1016/j.media.2025.103514](https://doi.org/https://doi.org/10.1016/j.media.2025.103514).
URL [https://www.sciencedirect.com/science/article/pii/S1361841525000623](https://www.sciencedirect.com/science/article/pii/S1361841525000623).
- You et al. (2023)
Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K. Hong, Woonhyuk Baek, and Byungseok Roh.
Cxr-clip: Toward large scale chest x-ray language-image pre-training.
In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda-Mahmood, and Russell Taylor, editors, *Medical Image Computing and Computer Assisted Intervention – MICCAI 2023*, pages 101–111, Cham, 2023. Springer Nature Switzerland.
ISBN 978-3-031-43895-0.
doi:[10.1007/978-3-031-43895-0_10](https://doi.org/10.1007/978-3-031-43895-0_10).
URL [https://github.com/kakaobrain/cxr-clip](https://github.com/kakaobrain/cxr-clip).
- Wang et al. (2022)
Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun.
Medclip: Contrastive learning from unpaired medical images and text.
In *Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)*, pages 3876–3887. Association for Computational Linguistics, 2022.
doi:[10.18653/v1/2022.emnlp-main.256](https://doi.org/10.18653/v1/2022.emnlp-main.256).
URL [https://aclanthology.org/2022.emnlp-main.256](https://aclanthology.org/2022.emnlp-main.256).
- Bannur et al. (2023)
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando Pérez-García, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, and Ozan Oktay.
Learning to exploit temporal structure for biomedical vision-language processing.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, pages 15016–15027, June 2023.
- Zhang et al. (2023)
Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang.
Knowledge-enhanced visual-language pre-training on chest radiology images.
*Nature Communications*, 14(1):4542, 2023.
ISSN 2041-1723.
doi:[10.1038/s41467-023-40260-7](https://doi.org/10.1038/s41467-023-40260-7).
URL [https://doi.org/10.1038/s41467-023-40260-7](https://doi.org/10.1038/s41467-023-40260-7).
- Li et al. (2025)
Mingjian Li, Mingyuan Meng, Michael Fulham, David Dagan Feng, Lei Bi, and Jinman Kim.
Enhancing medical vision-language contrastive learning via inter-matching relation modeling.
*IEEE Transactions on Medical Imaging*, 44(6):2463–2476, 2025.
doi:[10.1109/TMI.2025.3534436](https://doi.org/10.1109/TMI.2025.3534436).
- Wang et al. (2024)
Bin Wang, Hongyi Pan, Armstrong Aboah, Zheyuan Zhang, Elif Keles, Drew Torigian, Baris Turkbey, Elizabeth Krupinski, Jayaram Udupa, and Ulas Bagci.
Gazegnn: A gaze-guided graph neural network for chest x-ray classification.
In *Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*, pages 2194–2203, January 2024.
- Chen et al. (2024)
Qi Chen, Yutong Xie, Biao Wu, Xiaomin Chen, James Ang, Minh-Son To, Xiaojun Chang, and Qi Wu.
Act like a radiologist: Radiology report generation across anatomical regions.
In *Proceedings of the Asian Conference on Computer Vision (ACCV)*, pages 1–17, December 2024.
- Kim et al. (2025)
Yunsoo Kim, Jinge Wu, Su Hwan Kim, Pardeep Vasudev, Jiashu Shen, and Honghan Wu.
Look & mark: Leveraging radiologist eye fixations and bounding boxes in multimodal large language models for chest X-ray report generation.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, *Findings of the Association for Computational Linguistics: ACL 2025*, pages 17680–17694, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
URL [https://aclanthology.org/2025.findings-acl.909/](https://aclanthology.org/2025.findings-acl.909/).
- Boecking et al. (2024)
Benedikt Boecking, Naoto Usuyama, Sruthi Bannur, Daniel Coelho de Castro, Anton Schwaighofer, Stephanie Hyland, Himanshu Sharma, Moritz T. Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez Valle, Hoifung Poon, and Ozan Oktay.
Ms-cxr: Making the most of text semantics to improve biomedical vision-language processing.
PhysioNet, 2024.
URL [https://doi.org/10.13026/9g2z-jg61](https://doi.org/10.13026/9g2z-jg61).
RRID:SCR_007345.
- Liu et al. (2021)
Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi.
Contrastive multimodal fusion with tupleinfonce.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pages 754–763, October 2021.
- Hayat et al. (2022)
Nasir Hayat, Krzysztof J. Geras, and Farah E. Shamout.
Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images.
In Zachary Lipton, Rajesh Ranganath, Mark Sendak, Michael Sjoding, and Serena Yeung, editors, *Proceedings of the 7th Machine Learning for Healthcare Conference*, volume 182 of *Proceedings of Machine Learning Research*, pages 479–503. PMLR, 05–06 Aug 2022.
URL [https://proceedings.mlr.press/v182/hayat22a.html](https://proceedings.mlr.press/v182/hayat22a.html).
- Dosovitskiy et al. (2020)
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.
An image is worth 16x16 words: Transformers for image recognition at scale.
*CoRR*, abs/2010.11929, 2020.
URL [https://arxiv.org/abs/2010.11929](https://arxiv.org/abs/2010.11929).
- Irvin et al. (2019)
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng.
Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison.
In *Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence*, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.
ISBN 978-1-57735-809-1.
doi:[10.1609/aaai.v33i01.3301590](https://doi.org/10.1609/aaai.v33i01.3301590).
URL [https://doi.org/10.1609/aaai.v33i01.3301590](https://doi.org/10.1609/aaai.v33i01.3301590).
- Alsentzer et al. (2019)
Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott.
Publicly available clinical BERT embeddings.
In Anna Rumshisky, Kirk Roberts, Steven Bethard, and Tristan Naumann, editors, *Proceedings of the 2nd Clinical Natural Language Processing Workshop*, pages 72–78, Minneapolis, Minnesota, USA, June 2019. Association for Computational Linguistics.
doi:[10.18653/v1/W19-1909](https://doi.org/10.18653/v1/W19-1909).
URL [https://aclanthology.org/W19-1909/](https://aclanthology.org/W19-1909/).
- van den Oord et al. (2019)
Aaron van den Oord, Yazhe Li, and Oriol Vinyals.
Representation learning with contrastive predictive coding, 2019.
URL [https://arxiv.org/abs/1807.03748](https://arxiv.org/abs/1807.03748).
- Gutmann and Hyvärinen (2012)
Michael U. Gutmann and Aapo Hyvärinen.
Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.
*J. Mach. Learn. Res.*, 13(1):307–361, February 2012.
ISSN 1532-4435.
- Cohen (1988)
Jacob Cohen.
*Statistical Power Analysis for the Behavioral Sciences*.
Routledge, 2nd edition, 1988.
- Bylinskii et al. (2019)
Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand.
What do different evaluation metrics tell us about saliency models?
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 41(3):740–757, 2019.
- Peters and Itti (2005)
R.J. Peters and L. Itti.
Components of bottom-up gaze allocation in natural images.
In *IEEE CVPR*, pages 1–7, 2005.
- Zhang and et al. (2024)
Wei Zhang and et al.
Evaluating human gaze and model attention on glaucoma oct interpretation.
*Medical Image Analysis*, 2024.
In Press.
- Hsieh et al. (2023)
Chihcheng Hsieh, Chun Ouyang, Jacinto C. Nascimento, Joao Pereira, Joaquim Jorge, and Catarina Moreira.
MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications.
[https://doi.org/10.13026/pc72-as03](https://doi.org/10.13026/pc72-as03), 2023.
PhysioNet. RRID:SCR_007345.
- Riche et al. (2013)
Nicolas Riche, Matei Mancas, Thierry Dutoit, and Bernard Gosselin.
Saliency and human fixations: State-of-the-art and study of comparison metrics.
In *IEEE ICCV Workshops*, pages 1153–1160, 2013.
- Papineni et al. (2002)
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation.
In *Proceedings of the 40th Annual Meeting on Association for Computational Linguistics*, ACL ’02, page 311–318, USA, 2002. Association for Computational Linguistics.
doi:[10.3115/1073083.1073135](https://doi.org/10.3115/1073083.1073135).
URL [https://doi.org/10.3115/1073083.1073135](https://doi.org/10.3115/1073083.1073135).
- Lin (2004)
Chin-Yew Lin.
ROUGE: A package for automatic evaluation of summaries.
In *Text Summarization Branches Out*, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
URL [https://aclanthology.org/W04-1013/](https://aclanthology.org/W04-1013/).
- Banerjee and Lavie (2005)
Satanjeev Banerjee and Alon Lavie.
METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.
In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss, editors, *Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization*, pages 65–72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics.
URL [https://aclanthology.org/W05-0909/](https://aclanthology.org/W05-0909/).
- Zhang* et al. (2020)
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi.
Bertscore: Evaluating text generation with bert.
In *International Conference on Learning Representations*, 2020.
URL [https://openreview.net/forum?id=SkeHuCVFDr](https://openreview.net/forum?id=SkeHuCVFDr).
- Deutsch et al. (2023)
Daniel Deutsch, George Foster, and Markus Freitag.
Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration.
In Houda Bouamor, Juan Pino, and Kalika Bali, editors, *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing*, pages 12914–12929, Singapore, December 2023. Association for Computational Linguistics.
doi:[10.18653/v1/2023.emnlp-main.798](https://doi.org/10.18653/v1/2023.emnlp-main.798).
URL [https://aclanthology.org/2023.emnlp-main.798/](https://aclanthology.org/2023.emnlp-main.798/).
- Jain et al. (2021)
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar.
Radgraph: Extracting clinical entities and relations from radiology reports, 2021.
URL [https://arxiv.org/abs/2106.14463](https://arxiv.org/abs/2106.14463).
- Pan et al. (2016)
Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel E. O’Connor, and Xavier Giró-i-Nieto.
Shallow and deep convolutional networks for saliency prediction.
*CoRR*, abs/1603.00845, 2016.
URL [http://arxiv.org/abs/1603.00845](http://arxiv.org/abs/1603.00845).
- Fisher (1992)
R. A. Fisher.
*Statistical Methods for Research Workers*, pages 66–70.
Springer New York, New York, NY, 1992.
ISBN 978-1-4612-4380-9.
doi:[10.1007/978-1-4612-4380-9_6](https://doi.org/10.1007/978-1-4612-4380-9_6).
URL [https://doi.org/10.1007/978-1-4612-4380-9_6](https://doi.org/10.1007/978-1-4612-4380-9_6).
- Menéndez et al. (1997)
M.L. Menéndez, J.A. Pardo, L. Pardo, and M.C. Pardo.
The jensen-shannon divergence.
*Journal of the Franklin Institute*, 334(2):307–318, 1997.
ISSN 0016-0032.
doi:[https://doi.org/10.1016/S0016-0032(96)00063-4](https://doi.org/https://doi.org/10.1016/S0016-0032(96)00063-4).
URL [https://www.sciencedirect.com/science/article/pii/S0016003296000634](https://www.sciencedirect.com/science/article/pii/S0016003296000634).
- Le Meur and Baccino (2013)
Olivier Le Meur and Thierry Baccino.
Methods for comparing scanpaths and saliency maps: strengths and weaknesses.
*Behavior Research Methods*, 45(1):251–266, 2013.
ISSN 1554-3528.
doi:[10.3758/s13428-012-0226-9](https://doi.org/10.3758/s13428-012-0226-9).
URL [https://doi.org/10.3758/s13428-012-0226-9](https://doi.org/10.3758/s13428-012-0226-9).

## Appendix A.1  MIMIC-Eye Dataset Specifications and Processing Details

### A.1.1 Dataset Composition

**Table 4: Core dataset metrics.**
| Metric | Count | Description |
| --- | --- | --- |
| Total Patients | 3,192 | Unique individuals in the dataset |
| REFLACX Records | 2,617 | Records annotated with REFLACX labels |
| Eye Gaze Records | 1,100 | Records with gaze-tracking information |
| Full Multimodal Records | 63 | Records containing image, REFLACX & gaze data |

**Table 5: File distribution across modalities.**
| File Type | Count | Description |
| --- | --- | --- |
| JPG Images | 6,292 | Chest X-ray images in JPEG |
| CSV Files | 113,043 | Eye gaze and associated metadata |
| JSON Files | 4,112 | Structured reports and annotations |

### A.1.2 Detailed Modality Analysis

**Table 6: Distribution of modality combinations.**
| Modality Combination | Count | % of Records |
| --- | --- | --- |
| Image + REFLACX | 2,653 | 69.9% |
| Image + Eye Gaze | 1,037 | 28.4% |
| Full Multimodal (Image + REFLACX + Gaze) | 63 | 1.7% |
| Image Only | 0 | 0% |

##### REFLACX Annotations.

- •
Total REFLACX Records: 2,617
- •
Unique REFLACX Patients: 2,199
- •
REFLACX Records with Eye Gaze: 63 (2.4%)
- •
Dataset Coverage: 71.6% of total records

##### Eye-Gaze Tracking.

- •
Total Eye-Gaze Records: 1,100
- •
Unique Eye-Gaze Patients: 1,038
- •
Eye-Gaze Records with REFLACX: 63 (5.7%)
- •
Dataset Coverage: 30.1% of total records

**Table 7: Condition distribution in the MIMIC-Eye dataset. Eight high-prevalence conditions were retained for analysis, while six tail classes with low frequency were removed to ensure statistical power, clinical relevance, and balanced multimodal coverage (see Section A.1.3).**
| Condition | Total | % of Dataset |
| --- | --- | --- |
| No Finding | 1,130 | 33.9% |
| Lung Opacity | 831 | 24.9% |
| Pleural Effusion | 750 | 22.5% |
| Support Devices | 686 | 20.6% |
| Cardiomegaly | 624 | 18.7% |
| Atelectasis | 593 | 17.8% |
| Edema | 445 | 13.4% |
| Pneumonia | 370 | 11.1% |
| Consolidation (Removed) | 149 | 4.5% |
| Pneumothorax (Removed) | 124 | 3.7% |
| Lung Lesion (Removed) | 95 | 2.9% |
| Enlarged Cardiomediastinum (Removed) | 68 | 2.0% |
| Fracture (Removed) | 46 | 1.3% |
| Pleural Other (Removed) | 31 | 0.9% |

### A.1.3 Condition Filtering Strategy

We retained eight primary conditions (top section of Table A.1.3) and excluded the six tail classes for three complementary reasons:

- 1.
Statistical Power and Model Stability - Each retained condition exceeds 10% prevalence, delivering at least 250 training samples and adequate positive cases for validation/testing. Tail classes fall below 5%, inflating variance and hindering robust multi-label optimisation.
- 2.
Clinical Relevance and Non-Redundancy - The eight selected phenotypes represent the most common findings on portable CXRs in critical-care settings and are routinely used for triage. Several discarded labels (e.g., Consolidation, Fracture) are radiographically subsumed by broader retained categories such as Lung Opacity or Pleural Effusion, introducing label redundancy without tangible clinical benefit.
- 3.
Balanced Multi-Modal Coverage - Full-multimodal studies ($n=63$) overwhelmingly feature the eight kept conditions ($\geq 90\%$ coverage), whereas the six tail labels occur in only four fully multimodal cases. Retaining them would preclude meaningful gaze-condition alignment experiments.

This pruning preserves $>$86% of the original label information while yielding a balanced, interpretable, and computationally tractable dataset.

### A.1.4 Inter-condition Correlation Matrix

Figure: Figure 4: Pearson correlation coefficients between binary condition labels prior to pruning. Strong negative associations are observed between No Finding and all pathological classes (mean $\rho\approx-0.29$). Positive couplings are most pronounced for fluid-related findings such as Pleural Effusion -Edema ($\rho=0.24$) and Atelectasis -Support Devices ($\rho=0.24$).
Refer to caption: images/condition_correlation_matrix.png

### A.1.5 Latent-space Structure via t-SNE

To visualise non-linear relationships in the CheXpert-initialised image feature space,
we projected 3,654 study-level vectors into two dimensions using t-SNE
(perplexity = 40, $\theta=0.5$). The composite plot highlights global structure
and class imbalance, while condition-specific overlays reveal pathology-dependent manifolds.

Figure: Figure 5: t-SNE class-specific overlays. Each subplot highlights the distribution of a given condition: (A.3) Atelectasis, (A.4) Cardiomegaly, (A.5) Consolidation, (A.6) Edema, (A.7) Lung Opacity, (A.8) No Finding, (A.9) Pleural Effusion, (A.10) Support Devices. Observed manifolds align with expected radiographic co-occurrences and variations.
Refer to caption: images/tsne_atelectasis.png

### A.1.6 Gaze Normalization Procedure

The REFLACX fixation dataset provides gaze coordinates in image pixel space
(‘x_position‘, ‘y_position‘), whereas the EyeGaze dataset records gaze as
normalized screen coordinates (‘FPOGX‘, ‘FPOGY‘) in the range $[0,1]$.
To create a common dataset, we normalized REFLACX gaze values into the
same $[0,1]$ coordinate system.

### Step 1: Extract Image Bounds

REFLACX includes bounding boxes of the displayed image within the DICOM
viewer:

- •
Image bounds: xmin_shown_from_image, ymin_shown_from_image,
xmax_shown_from_image, ymax_shown_from_image.
- •
Screen bounds: xmin_in_screen_coordinates,
ymin_in_screen_coordinates,
xmax_in_screen_coordinates,
ymax_in_screen_coordinates.

### Step 2: Normalize REFLACX Gaze

We compute normalized gaze coordinates as:

$$ $x_{\text{norm}}=\frac{x_position-xmin_shown_from_image}{xmax_shown_from_image-xmin_shown_from_image}$ $$

$$ $y_{\text{norm}}=\frac{y_position-ymin_shown_from_image}{ymax_shown_from_image-ymin_shown_from_image}$ $$

Values outside $[0,1]$ due to calibration noise are clipped.

### Step 3: Alignment with EyeGaze

EyeGaze coordinates (‘FPOGX‘, ‘FPOGY‘) are already normalized in
screen space. After normalization, both datasets represent gaze positions
in the same $[0,1]$ range, enabling direct comparison and fusion.

### Step 4: Pupil Normalization

REFLACX provides a precomputed pupil_area_normalized. For
EyeGaze, we estimate the pupil area from left and right pupil diameters:

$$ $A=\frac{\pi}{2}\left(\frac{LPD}{2}^{2}+\frac{RPD}{2}^{2}\right)$ $$

We then normalize pupil area relative to a baseline average computed
over the first 1–2 seconds of valid gaze data.

### Step 5: Fixation Duration

Fixation duration is derived as:

$$ $d=timestamp_end_fixation-timestamp_start_fixation$ $$

for REFLACX, and directly from FPOGD for EyeGaze.

### Outcome

After these steps, both REFLACX and EyeGaze datasets share the following
common fields:

- •
Normalized gaze position: $(x_{\text{norm}},y_{\text{norm}})$
- •
Pupil area (normalized)
- •
Fixation duration
- •
Timestamp

This harmonization ensures comparability of gaze features across datasets.

### A.2 Hyperparameter Settings

##### Enhanced gaze models.

Learning rate $6\times 10^{-6}$; epochs $40$; cosine scheduler with warmup; batch size $32$ (or $8$ on low-memory systems).

**Table 8: Training schedules used across model variants.**
| Model | Learning rate | Batch size | Epochs | Scheduler |
| --- | --- | --- | --- | --- |
| Enhanced-gaze (ours) | $6\times 10^{-6}$ | 32 | 40 | Cosine + warmup |
| MIMIC baseline | $5\times 10^{-6}$ | 32 | 35 | Cosine |
| ViT-only | $5\times 10^{-5}$ | 128 | 20 | Cosine |

**Table 9: Optimizer and regularization configuration.**
| Optimizer (AdamW) | Value | Notes |
| --- | --- | --- |
| $\beta_{1}$ | $0.9$ |  |
| $\beta_{2}$ | $0.98$ |  |
| $\epsilon$ | $1\times 10^{-8}$ |  |
| Weight decay | *per-run arg* | As set in training arguments |
| Gradient clipping | $0.4$ | $\texttt{max_grad_norm}=0.4$ |

### Bounding Box Completion using YOLOv8n for REFLACX Data

We developed an enhanced bounding box completion framework that integrates radiologist-provided REFLACX annotations with predictions from YOLOv8n to achieve comprehensive spatial coverage in chest X-rays. The framework ensures both clinical fidelity and computational efficiency, while providing standardized representations suitable for multimodal medical AI models. A total of 17 anatomical regions were defined to consistently capture the thoracic cavity. This standardization addresses variability in annotation styles and facilitates uniform downstream processing.
To preserve expert knowledge, REFLACX annotations are prioritized. YOLOv8n predictions are only used to fill missing or incomplete regions. This strategy ensures maximum coverage without overriding radiologist expertise. All completed bounding boxes are transformed into spatial attention masks. Gaussian smoothing is applied to generate soft anatomical boundaries, enabling more effective integration with multimodal models.
A low confidence threshold of $0.05$ was adopted to maximize medical sensitivity, while an IoU threshold of $0.5$ was applied to manage overlapping regions. These thresholds were selected to balance recall of subtle findings with control over redundant detections. The framework employs memory-optimized inference for large-scale processing. Comprehensive quality control measures are implemented, including:

- •
Spatial coverage metrics: percentage of image area covered by annotations,
- •
Anatomical completeness: assessment of essential region coverage,
- •
Confidence distribution analysis: evaluation of detection reliability, and
- •
Source attribution: breakdown of contributions from REFLACX versus YOLOv8n.

This design provides richer and more standardized spatial context, enabling downstream multimodal models to benefit from improved spatial fidelity and clinical robustness.

### A.3 Attention Metrices Thresholds and Report Generation Evaluation

**Table 10: Interpretation and practical thresholds for attention alignment metrics. Thresholds are drawn from cognitive psychology, saliency benchmarking, and clinical gaze -AI alignment studies.**
| Metric | Interpretation | Thresholds / Benchmarks |
| --- | --- | --- |
| Pearson Correlation ($r$) | Measures linear alignment between human and AI attention maps. | $r\geq 0.30$ = moderate alignment (Cohen, 1988).<br>$r\geq 0.50$ = strong alignment.<br>Typical radiology gaze studies: $0.20$ -$0.40$ acceptable (Cohen, 1988). |
| Mean Squared Error (MSE) | Pixel-wise distance between normalized human and AI attention maps. | No universal cutoff; lower is better.<br>MSE $\leq 0.05$ generally indicates good alignment in saliency benchmarking (Pan et al., 2016). |
| P-value (statistical test) | Significance of AI -human correlation above chance. | $p<0.05$ = statistically significant alignment.<br>$p<0.01$ = strong evidence against null hypothesis (Fisher, 1992). |
| Jensen -Shannon Divergence (JSD) | Distribution similarity of attention maps (bounded [0,1]). | JSD $<0.20$ = strong similarity;<br>$0.20$ -$0.40$ = moderate similarity.<br>Inter-radiologist JSD $\approx 0.45$ = human upper bound (MIMIC-Eye) (Menéndez et al., 1997). |
| Normalized Scanpath Saliency (NSS) | Measures how well model saliency coincides with fixation locations. | NSS $\geq 1.0$ = good human-level alignment (Le Meur and Baccino, 2013).<br>Values $<0.2$ = weak, but still indicate non-random overlap. |
| Human Attention Entropy | Entropy of human fixation maps; reflects variability in gaze. | Typical radiology range: 9 -11 bits (clinical gaze studies).<br>Values outside this may indicate atypical fixation patterns. |
| Model Attention Entropy | Entropy of AI saliency maps; reflects diversity of model focus. | Desirable range similar to human entropy (9 -11 bits).<br>Large deviations suggest over- or under-concentration of attention. |

**Table 11: Evaluation metrics and threshold interpretations for report generation**
| Metric | Purpose | Threshold Interpretation |
| --- | --- | --- |
| BLEU-1 to BLEU-4 | Measures n-gram precision; evaluates lexical overlap with reference reports | >0.20 indicates acceptable word-level match; >0.30 suggests good domain alignment |
| ROUGE-1 / ROUGE-2 / ROUGE-L | Recall-oriented metric capturing clinical phrase and sequence overlap | ROUGE-L F1 >0.25 considered reasonable for medical reports; higher recall (>0.35) desirable in impression section |
| Clinical Keyword Overlap | Alignment of disease-specific and anatomical terminology between generated and reference reports | Coverage >70% ensures core clinical terms preserved; lower overlap risks omission of key conditions |
| Sentence-BERT Similarity | Embedding-based contextual coherence across sections | Cosine similarity >0.80 indicates strong semantic alignment; 0.65–0.80 suggests partial but acceptable agreement |

### A.6 Clinical Knowledge Integration

#### A.6.1 Anatomical Region Mapping

To support anatomically grounded modeling, we defined 17 standardized chest X-ray regions covering the cardiac, pulmonary, pleural, and mediastinal compartments. Each region is encoded with normalized bounding box coordinates and annotated with its clinical significance. This design provides a consistent spatial reference system for integrating human knowledge with AI attention mechanisms.

**Table 12: Standardized anatomical regions with definitions and clinical significance.**
| Region | Definition | Clinical Significance |
| --- | --- | --- |
| Cardiac silhouette | Heart border and mediastinal contour | Cardiomegaly, heart failure assessment |
| Left lung | Complete left pulmonary field | Primary site for pathology detection |
| Right lung | Complete right pulmonary field | Primary site for pathology detection |
| Left upper lung zone | Left lung above hilum level | Upper lobe pathology, TB predilection |
| Left mid lung zone | Left lung at hilum level | Middle lobe syndrome, lingular pathology |
| Left lower lung zone | Left lung below hilum level | Aspiration pneumonia, effusion |
| Right hilar structures | Right pulmonary vessels and bronchi | Lymphadenopathy, vascular congestion |
| Left hilar structures | Left pulmonary vessels and bronchi | Lymphadenopathy, vascular congestion |
| Right costophrenic angle | Right diaphragm -chest wall junction | Pleural effusion detection |
| Left costophrenic angle | Left diaphragm -chest wall junction | Pleural effusion detection |
| Upper mediastinum | Superior mediastinal compartment | Support devices, central lines |
| Trachea | Central airway structure | Endotracheal tube placement |

#### A.6.2 Condition-to-Region Clinical Knowledge Matrix

To integrate domain expertise, we constructed a condition-to-region mapping matrix. Each medical condition is linked to its *primary* and *secondary* anatomical regions, alongside an *attention weight* reflecting clinical importance. A textual rationale provides medical justification for the associations.

**Table 13: Clinical condition -to -region mapping with weights and rationale.**
| Condition | Primary Regions | Secondary Regions | Weight | Clinical Rationale |
| --- | --- | --- | --- | --- |
| Atelectasis | Left lung, Right lung | Lower lung zones | 0.8 | Gravity-dependent collapse, post-operative complications |
| Cardiomegaly | Cardiac silhouette | Upper mediastinum | 0.95 | Heart size $>$50% thoracic width, CHF indicator |
| Edema | Left lung, Right lung | Hilar structures | 0.7 | Bilateral perihilar distribution, Kerley B lines |
| Lung opacity | Lung zones (upper/mid/lower) | Entire lungs | 0.85 | Consolidation, diffuse patterns |
| Pleural effusion | Costophrenic angles | Lower lung zones | 0.9 | Gravity-dependent fluid collection |
| Pneumonia | Upper and lower lung zones | Mid lung zones | 0.8 | Lobar or bronchopneumonia patterns |
| Support devices | Upper mediastinum, Cardiac silhouette, Trachea | Hilar structures | 0.9 | Central lines, ET tubes, pacemakers |
| No finding | Cardiac silhouette, Left lung, Right lung | Upper lung zones | 0.6 | Normal baseline anatomical assessment |

This structured mapping enables the multimodal model to align disease-specific reasoning with anatomically localized evidence, improving explainability and clinical coherence.

### A.6.3 Report Template System and Model Outputs

This section documents the report generation framework, including the prompt template used for LLM-driven radiology reporting and representative outputs from different models for a given DICOM study.

#### A.6.3.1 Prompt Template

The following template was employed for all LLM-based report generation experiments. It integrates structured system instructions, clinical analysis data, and task-specific instructions to ensure consistent reporting style and lexical alignment with expert references.

#### A.6.3.2 Ground Truth Report (Reference)

For DICOM ID 6c2b39fa-2c251fcf-addd31da-83faee60-044fa8f9, the expert-provided reference report is:

##### Given Report:

##### Cleaned Report:

Findings:
Enteric tube passes below the inferior field of view. The lungs are clear without consolidation, effusion or edema. The cardiomediastinal silhouette is within normal limits. No acute osseous abnormalities.
Impression:
Enteric tube in place. No focal consolidation.

#### A.6.3.3 Model-Generated Reports

##### Gemini 2.5 Pro

Findings: A gastric tube is seen extending into the upper abdomen with its tip likely within the stomach. The lungs are well-aerated. There is no focal consolidation, pleural effusion, or pneumothorax. The cardiac silhouette and mediastinal contours are within normal limits. The visualized osseous structures are unremarkable.
Impression: No acute cardiopulmonary process. A gastric tube is in place.

##### Qwen3

Findings: The chest radiograph demonstrates aerated lung fields without evidence of focal consolidation or pleural effusion in any zone. The cardiac silhouette is within normal limits. A gastric tube is visualized extending into the upper abdomen; however, the tip is not fully imaged. An additional lead is noted in the upper mediastinum. No acute osseous abnormalities are identified.
Impression: No acute cardiopulmonary process. A gastric tube and an additional lead are visualized.

##### Llama 4

Findings: The cardiac silhouette appears normal in size. Lung fields are aerated without focal consolidation or pleural effusion. A gastric tube is visible, extending into the upper abdomen, but its tip is not imaged. No acute infiltrates or opacities are seen.
Impression: No acute cardiopulmonary process identified. Clinical correlation is recommended regarding support devices.

##### MedGemma

Findings: The lungs are well-expanded bilaterally. No evidence of focal consolidation, pleural effusion, or pneumothorax. The cardiac silhouette and mediastinal contours are within normal limits. A gastric tube is visualized extending into the upper abdomen; the tip is not fully imaged. A laparoscopic gastric band is also noted.
Impression: No acute cardiopulmonary process. Visible support devices include a gastric tube and a laparoscopic gastric band.

##### BioMistral

Findings: A gastric tube is present in the upper abdomen, with the tip not imaged. The lungs are well-expanded bilaterally with no pleural effusion. Cardiomegaly is present. An acute infiltrate is seen in the left upper lung zone and a chronic infiltrate in the right lower lung zone.
Impression: Gastric tube present. Cardiomegaly and pulmonary infiltrates. No pleural effusion or pneumothorax.

#### A.6.3.4 Comparative Summary

Table [14](https://arxiv.org/html/2508.13068v1#Ax1.T14) summarizes the alignment of model outputs with the ground truth report.

**Table 14: Comparison of model-generated reports against ground truth reference.**
| Model | Tube Detection | Lung Findings | Cardiac Findings | Extra/Hallucinated |
| --- | --- | --- | --- | --- |
| Ground Truth | Yes | Clear | Normal silhouette | None |
| Gemini 2.5 Pro | Yes | Clear | Normal | None |
| Qwen3 | Yes | Clear | Normal | Lead (hallucinated) |
| Llama 4 | Yes | Clear | Normal | Suggests correlation |
| MedGemma | Yes | Clear | Normal | Laparoscopic band |
| BioMistral | Yes | Infiltrates (false) | Cardiomegaly (false) | Multiple findings |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 1
questions: See weakness section.
soundness: 2
strengths: The main strengths of the paper are that it presents a novel multimodal framework that combines eye-tracking data, radiology text, anatomical masks, and chest X-ray images. The two-stage design, which combines region-grounded text generation and gaze-guided classification, is both conceptually and clinically sound. The methodological rigor of the proposed method is further reinforced by thorough ablations and meticulously documented dataset curation.
summary: This work proposes a new gaze-guided, region-grounded framework for interpreting CXRs that combines visual, textual, and attentional modalities is presented in this paper. The proposed work aims to improve interpretability and report accuracy by using human fixation data for spatial supervision. The method advances medical AI reporting and has potential for clinical applicability.
weaknesses: Major:
1. The main flaw is that I fail to see how this approach advances clinical practice. Annotations are even more challenging than finding medical data to train AI models.  It might not be feasible to design an eye gaze experiment to regulate the annotations in the suggested method.
2. One of the major concerns is the experimental validation of the proposed method. A few points that are lacking: a) an experiment that uses shuffled/mismatched transcripts or eliminates text from Stage 1 (and from any pretraining that touches the same split); b) a label-only text control (text→labels logistic baseline) to measure the amount of signal derived from text alone during training; and c) leakage checks to ensure that the text used in contrastive learning does not contain templated CheXpert phrases that mirror targets.
3. The Introduction section of the paper, which primarily discusses the problem statement, clinical motivation, and hypothesis of the proposed work, is poorly written. As a result, the paper becomes challenging to understand.
4. The authors should show some examples of generated reports and ground truth reports as qualitative results.
5. “Region grounding” is asserted but not convincingly validated. Region signals come from (i) EyeGaze boxes, (ii) YOLO-predicted boxes for REFLACX, (iii) keyword→region alias mapping at inference. Specially in case of REFLACX, if keyword→region mapping and report prompts depend on YOLO-generated regions, the text may be steered by potentially incorrect regions, confounding the interpretation of generator scores and any “grounding” visualization.
6. Also in experimental results, surface metrics like BLEU, ROUGE, METEOR, etc. are low across LLMs. The RadGraph-F1 score is ~0.13 (Table 3), which is modest and directly contradicts the claim of “entity-level” grounding.
7. The experiments lack statistical validation. Without that, it is insufficient to validate the key results. It would also help if authors add confidence intervals to the results.

Minor:
1. Hyperparameter selection is not discussed. For example, in Eq. 6, how $\lambda_{NCE}$ and $\lambda_{gaze}$ are selected.
2. The attention map is referred to as the attention rollout. Rollout can blur at deeper layers, so it is important to compare alternative techniques (LRP, Grad-CAM, etc).
3. Several mathematical notations are not consistent. Such as loss functions are represented as $\mathcal{L}$ and $L$ multiple times.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: see Weaknesses
soundness: 2
strengths: 1. The use of radiologist gaze as explicit spatial supervision is innovative and well-motivated. The trust-calibrated composite loss (MSE, KL, Pearson, CoM) is a thoughtful approach to aligning model attention with human expertise.
2. The detailed alignment procedure for MIMIC-Eye demonstrates careful handling of heterogeneous annotations. 
3. It provides clear evidence of incremental benefits from each modality, and the extended ablations examine backbone choices systematically.
4. Comparing six different LLMs provides valuable insights into generation quality trade-offs.
summary: This paper presents a two-stage multimodal framework for chest X-ray interpretation that integrates radiologist eye-tracking data with visual and textual information from the MIMIC-Eye dataset. Stage 1 introduces a gaze-token classifier that fuses image patches, bounding-box masks, transcription embeddings, and radiologist fixations using a trust-calibrated composite loss function. Stage 2 converts classifier predictions into region-grounded diagnostic reports by extracting confidence-weighted keywords, mapping them to 17 thoracic regions, and using an LLM to generate structured reports. The authors report improvements in both classification performance and attention alignment when incorporating gaze supervision.
weaknesses: 1. The entire evaluation is on a single dataset (MIMIC-Eye, n=2,877 aligned samples). It lacks comparison with state-of-the-art on standard benchmarks.
2. When comparing report generation performance, the author does not compare their methods with current report generation models.
3. This method introduces many modules for intermediate fixation prediction but they do not compare the computational efficiency of their method and current report generation methods.
4. I think this method is somehow cascaded several modules, while these modules cannot show their novelty.

---

confidence: 3
contribution: 3
initial_rating: 4
presentation: 2
questions: Addressed in weaknesses.
soundness: 2
strengths: Important problem 
- Producing radiology reports that ground findings in correct spatial regions in the image is a high impact problem. 

Multimodal approach 
- This paper proposes a method to integrate many streams of data with complementary information (the image, the text report, gaze data, and bounding box data) to improve generation performance, which is an interesting approach.
- The particular approach of using gaze data does seem to improve performance compared to not using gaze data and is an interesting way of using gaze to guide image supervision.
summary: This paper presents an approach to radiology report generation that is trained using multiple modalities of information: the image, gaze data, text report, and bounding boxes. The approach proceeds in two steps, first generating classification labels using a ViT that has features and attention generated from the multimodal input. In the second step, a LLM is used to construct a text report based on the outputs from step 1 as well as a disease/location-specific vocabulary constructed offline.
weaknesses: Doubts about the report generation step
- This approach to report generation seems to be more pattern memorizing and pattern matching than true spatial grounding. The authors rely on condition-specific vocabularies (built up offline via processing the MIMIC database with an LLM) to result in phrases that “sound like” radiologists, as well as “ground” those findings using 17 general thoracic regions identified via normalized bounding boxes. While I believe this approach would appear to work well on average (by repeating condition- and region-specific phrases found in the reference MIMIC database), I am not convinced it will generalize or work on edge cases. For example, what if conditions veer from the “typical” vocabulary constructed offline? Is this procedure expected to be repeated at every healthcare institution to match the institute-specific vocabulary? What if conditions appear in abnormal locations or anatomy is abnormal?

Difficult to follow and missing technical details
- I find the organization of the paper confusing, making the method difficult to follow. The main issue I have with the organization is the lack of “big picture” introductions to key parts of the method. The authors present parts of their method using distinct paragraphs with little-to-no explanation for how these distinct parts fit together or what the motivation for them is.  For example:
     - Prior to Section 3.5, there is no motivation given for the offline vocabulary construction, so when this vocabulary construction procedure is described the reader does not yet know how/why this vocabulary will be used. This sort of bottom-up description makes it difficult to follow the method.
     - There are many components to the supervision: contrastive learning, a supervised loss, a gaze-mediated loss. However, these multiple supervision streams are not introduced and contextualized in the Method “Overview and Motivation” section. As a result, when they appear as standalone paragraphs later in the Methods, it is difficult to follow all the different sources of supervision and why they are being used.  
     - The authors refer to details about the dataset (e.g., bounding boxes) many times in the Methods section without first describing what those bounding boxes indicate (nor any other details of the dataset), so I don’t know what the bounding boxes are localizing. 
     - The section describing the “Enhanced gaze” configuration in 3.2.1 is difficult to parse, particularly because many components rely on equations presented further on in the Methods. For example, the authors refer to using a “composite loss…[to] align A_model with A_gaze”, but it’s not clear what this “alignment” involves until the equation is presented in Section 3.4.
- In addition to difficult organization, there are also many missing technical details, making the method likely very difficult to replicate. If code is not released, I would be unable to replicate this method. Some examples of missing detail are listed below. 
     - Curriculum scheduler used for incorporating gaze
     - In line 191, starting “each term…”, how is modulation performed/computed?
     - Precise configurations for each modality encoder
     - In line 172, how is the variable length gaze data first embedded prior to the biGRU?
     - Prompts for vocabulary generation and filtering?
     - In line 209, “otherwise the block is skipped.” When is the block skipped? 
     - Are the original images 224 x 224 or are these downsized? If downsized, why? Downsizing is known to reduce accuracy with small abnormalities in medical image interpretation.
     - How are the thoracic regions defined and localized per image?
     - Line 260, “Gemini-cleaned keywords are matched (case-insensitive, fuzzy similarity) to the alias lists.” How is this matching performed, what is the criteria for a successful match?
     - The train/val/test split has percentages that sum to 116.2%, which is not typical for distinct dataset splits.

No baseline methods
- The authors present ablations (comparing their method with and without certain components), but do not compare their method against any of the many other report generation pipelines out there. Without baselines, I cannot assess the strength of the system as a whole nor its ability to meaningfully boost performance/spatial grounding compared to existing solutions.

Typos/grammar/writing
- I think the authors have used \cite{}, which includes citations in-text, where they should have used \citep{}, which includes citations in parentheses. The authors should switch the citations to the correct format as appropriate to improve readability. 
- Paragraph 2 assumes a successful report generation system must use gaze data in training, but this is not true.
- Line 236, “Thus 60% of the optimization signal targets label accuracy…” This percentage is not a standard interpretation of loss weights. The scale of each of these losses may be different from one another, so a loss even with a small weight can account for a larger “percentage” of the total loss.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: 1. What are the specific forms/hyperparameters of course scheduling and "bounded gating"? Has it been adapted for different departments/readership groups? 

2. What is the motivation and anti-leakage measure for "Fine Tune incorporates validation into training"? Is it only used for finalization and does it not change the hyperparameters? 

3. What are the quality assessments and failure rates of YOLO boxes in REFLACX? How do the metrics change if the automatic boxes are removed? 

4. Are the prompt words, blacklist/whitelist, and regional alias dictionary on the generation side open source? Are the parameters and temperatures of different LLMs consistent? 

5. Why not include gaze-text comparisons (the author only uses image-text/image-gaze)? Has this been tried, and what are the results?
soundness: 2
strengths: Highly practical: Gaze alignment is introduced during training, but only images are used during inference. The paper uses a "learnable gaze token + composite alignment loss (MSE/KL/correlation/centroid) + curriculum-based weighting" to constrain attention to human gaze during training. The final test does not rely on any gaze or text input, while still improving discrimination metrics (AUC/F1) and obtaining more interpretable attention maps. This "gaze-for-training, image-only-inference" design is more user-friendly for clinical deployment.
summary: The paper's direction is significant: coupling human attention with regionalized text generation to pursue interpretable diagnostics. However, further work is needed in areas such as evidence for alignment strength, reliability of regional supervision, factual reporting/reproducibility, and statistical completeness. The paper's persuasiveness would be significantly enhanced if it were supplemented with multi-seed statistics, external validation, open-source backend reproducibility, RadGraph constraints, and YOLO box quality assessment.
weaknesses: 1. Insufficient distinction between novelty and prior work.
The paper introduces gaze supervision into multimodal chest radiograph classification and report generation, using a core approach of gaze tokens + composite alignment loss + regionalized report generation. However, compared to existing work using gaze for medical representation alignment/attention guidance (e.g., using gaze as weak supervision, as channel/mask input, or as a constraint in multimodal contrastive learning), the methodological incrementality and necessity are insufficiently justified. The related work section primarily provides a side-by-side review, lacking a point-by-point difference table. It is recommended to provide a structural difference diagram/table and a comparison under the same training budget to clarify the need for a learnable gaze token and this specific combination of losses. 

2. The interpretation of the "gaze coherence" metric is overly optimistic.
The authors emphasize that the Pearson correlation r≈0.306 achieves "moderate correlation," and the JSD≈0.437 is close to the upper bound for readers. However, the NSS is only 0.138, far below the commonly used human alignment threshold of 1.0. The model's attention entropy also falls short of the upper bound for humans. These observations suggest that "quantitative alignment is weak—visual similarity is insufficient to support strong alignment conclusions." Recommendations include reporting cross-subject/cross-task stratified correlation distributions, introducing Spearman rank correlations/piecewise linear correlations, and statistical significance/confidence intervals to avoid selectively optimistic interpretations of single correlations. 

3. Dataset curation and supervision sources may introduce bias. To complete the missing region boxes in REFLACX, the authors trained YOLO using EyeGaze annotation and inferred a maximum confidence box for each region on REFLACX. This was then used for region mapping and report generation. This "self-trained completion" method is prone to mistaking noisy boxes for ground truth, impacting the reliability of region-level supervision and reported geographic attribution. Furthermore, only 2,877 of 3,689 cases (67.1%) were ultimately retained, and this was based on single-center data, raising concerns about generalization. Recommendations include quality assessment of YOLO boxes (intersection of union (IoU)/localization error, sensitivity analysis of subsequent metrics), external data validation, and sensitivity analysis of different thresholds/numbers of candidates (K). 

4. Report generation is weak and relies on a proprietary LLM, posing high risks for reproducibility and fairness. Keyword extraction and generation rely on Gemini 2.5 Pro (two-stage filtering and text generation), and the mapping of lesion probabilities to 17 regions is fed to the LLM using a heuristic threshold. However, most text metrics (such as BLEU/ROUGE/METEOR) are low, with RadGraph-F1 at the entity-relationship level being approximately 0.13. The authors omit external horizontal comparisons with existing data/protocols. Uncontrollable updates to the proprietary API also weaken reproducibility and fair comparisons. Recommendations: Provide fully open-source backend reproducible experiments, a list of fixed versions/prompt templates, and comparisons with existing public benchmarks. 

5. Potential Impact of Class Selection and Imbalance Treatment
The authors retain eight overrepresented classes and discard minority classes with a combined representation of <1.5%. While this simplification is practical, it may inflate the overall AUC/F1 ratio and obscure the value of gaze for rare disease types. Recommendation: Provide results/recall-cost curves for the minority class scenario and conduct threshold calibration experiments. 

6. Figure 2/The "bounded gating layer" in the method description lacks a specific functional form and hyperparameters; the details of the attention rollout resampling/normalization are somewhat vague.