Rethinking Radiology Report Generation: From Narrative Flow to Topic-Guided Findings
Authors: Sheng Cheng
Devika Subramanian
Venue: ICLR 2026 Poster (2026)
Keywords: Radiology report generation
large-language models
chest X-rays
multi-modal alignment
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: Radiology reports aren't stories. We show that forcing models to generate them sequentially is flawed. Our topic-driven paradigm generates distinct clinical findings independently, achieving SOTA performance and higher factual accuracy.

Abstract: Vision-Language Models (VLMs) for radiology report generation are typically trained to mimic the narrative flow of human experts. However, we identify a potential limitation in this conventional paradigm. We hypothesize that optimizing for narrative coherence encourages models to rely on linguistic priors and inter-sentence correlations, which can weaken their grounding in direct visual evidence and lead to factual inaccuracies. To investigate this, we design a controlled experiment demonstrating that as textual context increases, a model's reliance on the input image systematically decays. We propose LLaVA-TA (Topic-guided and Anatomy-aware), a new fine-tuning framework that directly addresses this challenge by re-engineering the generation process. Instead of producing a linear narrative, LLaVA-TA decomposes the report into a set of independent, clinically-relevant topics. By training the model to generate a discrete finding for each topic conditioned on both the full image and its corresponding anatomical region, we reduce the model's reliance on narrative flow and enforce stricter visual grounding. Our experiments show that LLaVA-TA sets a new state of the art on the MIMIC-CXR dataset, significantly improving clinical accuracy on metrics like RadGraph F1 (from 29.4 to 44.0) and CheXpert F1-14 (from 39.5 to 71.5) over strong baselines. Our work demonstrates that dismantling a report's narrative structure to enforce independent, visually-grounded observations is a crucial and effective step toward building more accurate and reliable medical VLMs.

Published as a conference paper at ICLR 2026
RETHINKING RADIOLOGY REPORT GENERATION: FROM NARRATIVE FLOW TO TOPIC-GUIDED FIND- INGS
Sheng Cheng Devika Subramanian Department of Computer Science Rice University
6100 Main St., Houston, Texas, 77005, USA {sc159, devi sewviaaeee @r .e.edu
ABSTRACT
Vision-Languag! Models (VLMs) for radiology report generation are typically trained to mimic the narrative flow of human experts. However, we identify a po- tential limitation in this conventional paradigm. We hypothesize that optimizing for narrative coherence encourages models to rely on linguistic priors and inter- sentence correlations, which weaken their grounding in direct visual evidence and lead to factual inaccuracies. To investigate this, we design a controlled ex- periment demonstrating that textual context increases, model's reliance the input image systematically decays. We propose LLaVA-TA (Topic-guided and Anatomy-aware), new fine-tuning framework that directly addresses this challenge by re-engineering the generation process. Instead of producing a linear narrative, LLaVA- VA decomposes the report into a of independent, clinically- relevant topics. By training the model to generate a discrete finding for each topic conditioned both the full image and its corresponding anatomical region, we re- duce the model's reliance narrative flow and enforce stricter visual grounding Our experiments show that LLaVA-Te sets new state the art on the MIMIC- CXR dataset, significantly improving clinical accuracy on metrics like RadGraph FI (from 29.4 4 to 44.0) and CheXpert F1-14 (from 39.5 to 71.5) over strong base- lines. Our work demonstrates that dismantling report's ''s narrative structure to force independent, disually-grounder observations is a crucial and effective step toward building accurate and reliable medical VLMMs.
INTRODUCTION
The automated generation of radiology reports from medical images, such chest X-rays (CXRs), is a pivotal task in medical AI with the potential to streamline clinical workflows and reduce radiologist burnout. Unlike general-purpose vision-language tasks, radiology report generation (RRG) (Chen) et al. 2022; Tanida et al., 2023 Tanno et al. 2025) operates with minimal tolerance for error, de- manding exceptional factual precision, spatial grounding of findings to specific anatomy, and adher- ence to structured medical language-whert where ambiguity can have severe clinical consequences.
We propose that potential pitfall in adapting large Vision-Language Models (VLMs) (Wang al. 2024 Liu et al. 2023 Grattafiori et al. 2024 OpenAl et al. 2024) to RRG lies in their implicit optimization for narrative flow. Our central hypothesis is that conventional training, by treating the report as monolithic text sequence, inadvertently teaches models to prioritize linguistic coherence over strict visual fidelity. This learned reliance on the textual context of previously generated sen- tences could become actively detrimental, allowing it to bias subsequent findings and override direct evidence from the image itself-a process that we suspect weakens visual grounding and fosters clinical hallucination. To investigate this vulnerability, pre-trained VLM, LLaVA-Rad (Zambrano Chaves et al.) 2025). We prompted the model to com- plete the last K sentences of ground-truth report given the preceding sentences, comparing its
designed controlled experiment with
"Corresponding author.

---

Published as a conference paper at ICLR 2026
performance when conditioned on the true CXR versus blank image. Our findings provide strong empirical validation for our initial hypothesis, as shown in Fig. I Crucially, the performance gap between using the real versus the blank image diminished sigmifcantly the textual prefix grew longer (i.e., as decreased). This demonstrates that with sufficient textual context, the model in- creasingly disregards the visual input, relying instead on learned language patterns complete the report-a critical vulnerability for a diagnostic task.
This core issue of narrative-induced bias persists across existing methodologies. For instance, recent works like MAIRA A-2 (Bannur et al., 2024) and DART (Park et al. (025) have introduced explicit grounding objectives and self-correction modules better align text with disease regions. Similarly, approaches such as Poati- Supervision (Chen et al.) (025) and Context-Enhancee RRG (Del-) brouck et al. 2025) attempt to enforce clinical accuracy through hierarchical training or rich context integration. However, despite these advancements in alignment, these models largely retain the autoregressive architecture that allows earlier generated tokens to dictate the probability of future findings, leaving the narrative bias intact. Even specialized medical VLMs have not fully escaped this trap (Li et al.] 2023] Chen et al. 2024; Hyland et al. 2024 Tu et al. 2023). LLa for instance, improves performance by filtering out sentences describing historical comparisons but still generates the core findings as a sequential narrative. Other notable approaches, such as COMG (Gu
al. 2024), which incorporates anatomical masks, the Multi-Graine framework (Liu et al.j 2024 with its sentence-level contrastive learning, have advanced the field. Yet, they do not fun- dependency that allows contextual priors to override direct visual evidence, failing to achieve topic-level disentanglement.
damentally dismantle the centence-to-sentee
To address this fundamental limitation, we propose LLAVA-TA (Topic-guided and Anatomy-aware), a novel fine-tuning framework that re-envisions report generation. Instead of producing a linear narrative, LLaVA-TA decomposes the task by distinet medical topics corresponding to anatomical regions (e.g., lungs, heart, mediastinum). The model is trained to generate short, independent para- graphs for each relevant topic, conditioned on the entire image. This topie-guided approach confers two critical advantages. First, it enhances visual grounding. By breaking the sequential chain of dependency, we force the model to ground its statements for each topic (c.g., "lungs") directly and solely on the visual evidence present in the corresponding anatomical area, preventing knowl- edge from a previously generated sentence (e.g., about the "heart") from biasing the text. The visual grounding provided by CXR segmentation model that provides masks for each topic, as shown in Fig. 2 Second, it reduces hallucination via tighter supervision. Training on short, topic-isolated ground-truth sentences provides a much stricter and more focused learning signal. This structure minimizes the model's ability to make "lucky guesses" based on spurious sentence correlations, a common failure mode during training that reinforces hallucinatory behavior.
Our key contributions are as follows:
Diagnosing the "Narrative Flow" Problem. We empirically validate that narrative- floww- based training is fundamental limitation in RRG, demonstrating how it induces a harmful trade-off between textual coherence and visual grounding.
LLaVA-TA: A Topic-Guided Generation Framework. We propose LLaVA-TA, novel framework that re-engineers the generation process by decomposing it into independent, clinically relevant topies enforce stricter visual grounding.
State-of-the-Ar Clinical Accuracy and Grounding. We demonstrate through extensive experiments that LLaVA-TA sets new state of the art, producing reports that are signifi- cantly more factually accurate, spatially grounded, and less prone to clinical hallucination than prior methods.
2 METHODS
The LLaVA-TA framework enhances RRG by replacing unstructured, narrative-based training with structured supervision that explicitly aligns clinical topics with their corresponding anatomical re- gions in the chest X-ray. We first define "Narrative Bias" as follows: Narrative Bias is the tendency of autoregressive model marily on the linguistic probability P(yely<)) rather than the visual conditional probability P(y. As illustrated in Fig. LLaVA-TA comprises three core stages to address this issue: (1) topic-guided
I) to maximize the likelihood of the next token ye based pri-
2

---

Published as a conference paper at ICLR 2026
Metrica ROUGE4
Stow Differnnn Coeling biiee mg)
RedGraph
Stere Deffrnnce tertiep Mlaak (pp
Chelpprt-14
scme Codteenne corling baddeep
Chelpart
Diference deeilee Moled
40
Nicro
Macre F1-14
Hacro-11-5
Micro-1-lde
Nicro-Fl-s4
Matrefl:341
Nucro-f1:51
RoPder Baatk -1
Natrlor laskk-2
lank
Notrier
lasti-2
lastk
NaPrier
ladk-a
lackk-2 -2
Figure 1: Empirical validation of the model's over-reliance textual priors The plots show the score difference between generating report findings with the original image ver- a blank image across four metric categories. The y-axis represents this performance gap (Orilmg -BlcckIng;; larger value means the visual information provides a greater benefit. The x-axis rep- resents the amount of textual context provided to the model, where "NoPrior" is full generation and "lastk -K" is the task of generating only the final K sentences. Across all metrics, the performance gap consistently shrinks as more textual context is given (i.e., as K decreases from to 1). This demonstrates that the model increasingly ignores the visual input the textual prefix grows, relying instead on learned linguistic patterns to complete the report. This result provides strong evidence that training with narrative flow weakens visual grounding. Detailed results are shown in Appendix A
visual evidence.
(a) Anatomy Segmentation
[b) Radiology Report Topic Split (DeepSeekV3)
Oricinal
The No focal consooidaaiion is seen.
coleifedd
There pleuns effusion proumothorax.
The heart la normal In size with posi cost-surg cal changes, Indluding intact mediastinal wires.
CXAS model
king tahin
organilungs:
The lungs are volume.
Discase level:
Complete list defined topica
A right lower lung calofiee granuloma is present. disveoeeecioo
atslectasia cardlomegaly, consolidation edema.
Focal consolisation absent.
entarged cardoomed ediastnum,
Tople SORS
diseaseepliunll Pieural effusion
fracture. lung lesion.Jun
opacily: pleural ertusion, pleural
abdomio
Poart
direeseegnneummtheeee Pheumathorax absent
other, oneumonis, pneumothorax
organ level:
organ heart
heart, kinge: pleura.
diaphragm
medadiinur
vartecras
support modiastinal present.
support devices, patent
Special categories:
(c) LLaVA-TA architecture
Topic split reports quality evaluation for MiMiC-CXR._I
training set
2T1
The lungs are law in volume. right lower lung caloleed gronuloma present
BLEU4
Rouge. L
60.8
ROJER
viv the image <images
Toplo-epeclle
and the zog notttted part nimages, Frovide a desoription of the find rgs tn the rodiolgyy Image lowing organo disasses: unigis
ground truth centences
Macro-F1-5
Macm-F1-114
Figure 2: LLaVA-TA framework. Our pipeline transforms unstructured data into aligned, topic- specific training instances.
(a) First, a pretrained segmentation model (CXAS) generates anatomy- as spatially-aware visual inputs. (b) Concurrently, an LLM
specific masks for each CXR to (DeepSeek-V3) V33 dismantles the narrative structure of each report, decomposing it into discrete, cleaned sentences aligned with clinical topic ontology. (c) During fine-tuning, the VA-TA model is conditioned on both the original image and the relevant anatomical mask, and is prompted to generate a finding for single topic. (d) A quantitative evaluation confirms that our report decom- position process preserves high lexical and clinical fidelity compared to the original reports.
3

---

Published as a conference paper at ICLR 2026
decomposition of ground-trume -ruth reports into atomic, mentation to create spatially apatiall--aware aware visual inputs; and (3) a vision-language fine-tuning process that learss to generate topic-specific findings from these aligned pairs.
statements; (2) anatomical seg-
2.1 TOPIC-GUIDES REPORT DECOMPOSITION
Our first step is to disentangle the complex, multi-topic narratives of clinical reports into clean, image-grounded training targets. To achieve this, we employ DeepSeek-V3 (DeepSeek-Al et al.) 2025), an foollootionnfoloowingg LLM, preprocess and decompose each report into set of discrete, lopic-specific sentences. The topics are shown in Fig. b).
This process consists of key steps:
Cleaning: Comparative phrases (e.g.. "compared to prior exam") and speculative language (e.g., "possibly atelectasis") are removed to retain only evidence-based findings visible in the current image.
Each sentence is mapped clinical topic using hierarchical ontology. Topics are prioritized as follows: (1) pathology-Ievel (c.g.. "disease- connosscoonoiiiiiioon) (2) anatomy-level (c.g.. "organ-lungs"), and (3) auxiliary categories (c.g.. "support devices").
Topic Splitting:
To ensure high-quality sentence splits, we apply additional rules: each sentence must refer sin- gle topic, comparative or ambiguous language is rewritten for clarity, and negations are standardized (e.g., "Pneumothorax is absent"). This yields structured, direct, and image-grounded sentence tar- gets for each report. The complete list of topics are provided in the Appendix B The full prompt for applying the topic-split is shown in the Appendix C
Each radiology report is transformed into set of clean, topic-specific findings. This process en- forces one sentence per topic and provides clear supervision targets that can be aligned with local- ized visual inputs.
We further evaluate the quality of the topic-split reports using both lexical metrics and radiology- specific metrics over the entire SAMIC-CX--JPG training set (Johnson et al.) (2119)) by concatenat- ing them as a whole paragraph first; the results are shown Fig.2(d), The evaluation demonstrates strong alignment between preprocessed and original medical reports, with high BLEU-1 (58.7), BLEU-4 (38.0) and ROUGE-L (60.8) scores confirming lexical and semantic fidelity. The supe- rior RadGraph-FI (67.2) highlights robust retention of structured clinical entities and relations, as this metric extracts clinical entities (e.g.. "lung", "opacity") and the relations connecting them (e.g.. "opacity" "located in" - "lung"), computing the Fl overlap of these semantic triplets, while Micro-F1-14 (0.64) and Macro-F1-14 (0.61) indicate effective fine-grained label preservation.
2.2 ANATOMICAL REGION SEGMENTATION
To provide spatial grounding for each textual topic, we generate anatomical segmentation masks that isolate the relevant visual regions within the CXR. We employ CXAS (Seibold al., 2022
2023.. specialized UNet-based model, to segment each image into seven key anatomical regions: vertebrae, ribs, diaphragm, mediastinum, abdomen, heart, and lung halves.
Each clinical topic from our ontology is mapped to one more of these anatomical regions via predefined lookup table (Appendix [B). a topic pertains to multiple regions (e.g., tdisease-pleural effusion" may involve the lungs and diaphragm). their corresponding masks are merged. This map- ping yields two aligned visual inputs for each topic-sentence pair: (1) the original global image I, and (2) the topic-specific segmented image S, which provides a localized view. This dual-input strategy focuses the model's s attention, improving interpretability and reducing noise from irrelevant regions.
2.3 VISION-LANGUAGE FINE- TUNING
Our fine-tuning process, adapted from LLaVA-RO Rad,, trains model to generate topic-specific findings conditioned on both global and localized visual evidence. The architecture consists of three core components: (1) Vision Encoder: BiomedCL.IP-XX (Zhang al. 2025 Zambrano Chaves al.
4

---

Published as a conference paper at ICLR 2026
putvo
diosse-caddioeeeey
TSmall Bilatoral pressnk!
CCmprssiioo
vertateed body preaent.
Sentinates
tile present, atomash
MBG
Figure 3: Attention Visualization of LLaVA-TA. We adopt aggregation using max across all attention layers and heads of the attention scores for the currently generated tokens for the features corresponding to the two input images. The examples above illustrate how LLaVA-TA grounds clinically meaningful tokens in the appropriate image regions, demonstrating spatial interpretability for specific findings in the generated report (bottom row).
2025
pretrained on 697k paired radiology image-report data, processes both the original image
and the segmented image S. producing two distinct visual embeddings, Z, and Zs. (2) Multimodal Aligument Layer: A learnable MLP projects the visual embeddings Z₁ and Zs from the vision encoder's feature space into the language model's word embedding space. (3) Language Model: Vicuna-7B-vl. (Zheng et al. 2023), an instruction-tung -eunndd LLM, is used to generate topic-specifie findings ruto-regrsssively.
Each training instance is constructed from structured prompt containing two special <image> tokens: "Given the image <image> and the segmented part <image>, describe the findings for the topic: topic.". During the forward pass, the token embeddings corresponding to the two <image> placeholders are replaced by the projected visual embeddings from the MLP. This creates a single, unified input sequence of embeddings that seamlessly integrates textual and visual information for the language model.
The model is trained to maximize the likelihood of the ground- truth topic sentence Y (y/1 .... VL) by minimizing the standard auto-regressive cross coos-entropy loss:
L
(()) = Σ log P(
(yllyee Bet. Xprompt: Z1, Zs;
where 0 represents the trainable model parameters, Yt the i-th token in the target sentence, Yece are the preceding ground-truth tokens, and Xpompp is the structured text prompt.
We follow the established two-phase training protocol: (1)Phase 1: Cross-Modal Alignment. The vision encoder and LLM are kept frozen. Only the MLP is trained to align the visual features with the LLM's token space. (2)Phase 2: Instruction Fine-Tuning. The vision encoder remains frozen while the MLP and the full LIMM are fine-tuned end-to-end using the auto-regressive loss objective defined above. This phase teaches the model generate coherent, topic-specific sentences that are conditioned on both the global and localized image features.
2.4 INFERENCE PROCESS
During inference, we evaluate our model under two distinct settings to provide a comprehensive assessment. The first setting measures fine-grad fine-grained, topic-level accuracy, while the second evaluates the model's ability to generate a complete and clinically coherent report.
2.4.1 TOPIC-LEVEL EVALUATION
This setting provides controlled evaluation the model's core ability to generate a factually correct sentence for given clinical topic. During evaluation, each test instance consists of the image pair SI and one ground-truth topic (c.g., diisease-consoliaatin generated by the model is then compared directly against the corresponding ground-truth sentence
as the prompt. The single sentence
5

---

Published as a conference paper at ICLR 2026
for that specific topic. This process isolates the model's conditional text generation quality from the separate challenge of topic selection.
2.4.2 REPORT-LEVEL EVALUATION
This setting assesses the model's s practical ability to synthesize a complete and comprehensive clin- ical report. To achieve this, the model iterates through a set of topics and generates a sentence for each; the final report is the concatenation
these individual outputs. The topic set is compiled as follows: (1) Organ- and special- level topics. To ensure systematic and comprehensive anatomical coverage, the model is prompted with the complete, predefined set of all organ-level and auxiliary topics for every report. (2) Disease-level topics. We use the ground-truth disease labels from the test set as prompts. This is a deliberate experimental design choice to isolate the generative model's performance from the variance and potential errors of an upstream disease classification model, abling a fair and direct comparison of text generation quality against other methods. The complete, multi-sentence generated report is then evaluated against the full ground truth report.
IMPLEMENTATION DETAILS
3.1 DATASET
We conduct training and evaluation on the MIMIC-CXR-IGG dataset (Johnson et al., 2019a b), following the official train/validation/test splits. We retain only frontal-view images and exclude studies that lack "Findings" section, consistent with the filtering protocol of LLaVA-Rad. After preprocessing, we obtain 212,379 training, 1,721 validation, and 3,029 test image-report pairs.
Following our topic-splitting procedure using DeepSeek-V3. each report is decomposed into mul- tiple topic-level sentences. This results in 1,161,75 1,51 topic-aligned training pairs, 9,388 validation pairs, and 17,19 17.193 test pairs each consisting of an image, localized anatomical region, and a list of topic-specific sentences.
To evaluate cross-domain generalization, we apply the same preprocessing pipeline the IU-Xray FDemmer-Fushman dataset (Demner-I Fushman et al. 2013), yielding 2,950 images and 16,13! image-topic pairs after topic decomposition and filtering.
Al eeeiciaccan was passet
Support codiom
Aliiagaric dede praend.
purer My p1001s
Rtens-ling
Alg wittertt Inatodly Caurgge Prebtiie
pnlleree Praiing
Figure 4: Qualitative example of LLaVA-TA's report generation and attention visualization. (a) shows the report generated by LLaVA-TA; (b) shows the output from the baseline model LLaVA- Rad. Green highlights denote correct predictions, orange indicates partially correct sentences (in- cluding hallucinations), and red marks omissions. (c) and (d) visualize VA-TA'sss attention for the partially correct and omitted findings, respectively. Notably, even when .L-VA-TA fails fully verbalize finding, its attention maps correctly highlight the lesion area in the image. This spatial grounding provides interpretable feedback, helping radiologists identify potential errors, thereby in- creasing trust in model outputs.
3.2 TRAINING DETAILS
We adopt a two-stage training procedure and the same hyper-parameters LLaVA-Rad. In stage 1, we first train the multimodal alignment MLP while keeping the vision encoder and language model frozen. This stage rums for epoch the MIMIC-CXR-L 256 and a learning rate of le -3 with a cosine scheduler and warm-up up ratio of 0.03. In stage 2, fine-tune both the language model and MI. using LoRA (Hu et al. 2021) with rank of 128.
training set. We use batch size of

---

Published as a conference paper at ICLR 2026
The vision encoder remains frozen. Training is performed for epochs the CXMIC-CXR-- training set, using a batch size of 64 and a cosine learning rate of le with a 0.03 warm-up ratio. We report results using the final model.
All experiments are conducted on 4 A100 GPUs (40GB), and our implementation builds on the LLd codebase with modifications to support topic-level supervision and dual-image input.
4 RESULTS
We evaluate LLaVA-TA to answer three key questions: (1) Does topic-guides generation outperform state-of-the-an narrative-based methods clinical accuracy? (2) What are the specific contributions of topic disentanglement versus anatomy-aware grounding? (3) Does our approach generalize to new datasets and provide interpretable, visually-grounded outputs?
Table Main performance on the MIMIC-CXR-JPG test set. LLaVA-TA establishes new state of the art, significantly outperforming all baselines. The strong performance of ablation, LLaVA- T, confirms that topic disentanglement the primary driver of improvement. While the additional anatomical masks in LLaVA-TA offer comparable performance here, they are crucial for enabling parameter-officien fine-tuning (Tables 2 and
Model
Size
CheXpert
RadGraph F1
BLEU
ROUGE-1
Miero-ag
Macro-avg
Micr-aag
Macro-avg FI-14+ FI-5+
(4)
FI-14
FI-S
FI-14 FI-5
F1-14+
F1-5+
22.8
35.3
RGRG
<IB
54.7
37.3
12.6
26.4
Flamingo-CXR
<BB
51.9
56.5
20.5
10.1
29.7
Qwen2-VL
7B
20.6
17.0
16.5
25.0
22.4
19.3
20.3
11.0
12.0
0.8
LLaVA
7B
22.9
23.4
15.4
17.5
23.7
26.9
17.0
20.3
4.5
21.0
1.3
13.8
22.2
CheXagent
7B
39.3
41.2
24.7
34.5
39.4
42.1
27.3
35.8
20.5
16.9
4.7
21.5
MATRA-I
56.0
38.6
55.3
42.3
51.7
28.9
GPT-VV
35.5
25.8
20.4
19.6
35.6
33.3
25.3
29.6
13.2
16.4
1.9
13.2
Llama 3.2
11B
33.3
31.8
21.3
26.3
34.4
37.4
25.0
32.6
9.0
10.2
U.7
11.1
Mcd-PaLM
84B
57.9
39.8
32.3
11.3
27.3
LLaVA-Rad
7B
58.1
57.7
40.1
50.3
57.7
59.8
42.9
$3.0
30.8
40.0
16.1
30,8
LLaVA-T (fopic)
90.0
71.2
83.7
90.3
78.2
88.7
46.5
50.9
32.1
60.9
LLaVA-T deport)
7B
80.2
88.5
63.0
87.0
81.6
88.7
71.4
87.7
34.2
43.9
25.7
41.8
LLaVA-TA TA (topic)
7B
83.5
89.6
71.5
87.3
83.6
90.1
77.9
88.3
44.0
50.9
31.8
6006
LLaVA-TA (roport)*
89.0
62.4
88.7
70.7
87.4
24.8
Table 2: In-distribution performance on MIMIC-CRR with MLP-only fine-tuning, In this parameter- efficient setting. LLAVA-TA dramatically outperforms both the L.L. VA-RRa baseline and our topic- only LLaVA-T ablation. The results demonstrate that when the L.M frozen, the explicit spatial guidance from anatomical masks is critical for learning an effective mapping from vision to lan- guage.
Model
Size
Micro-avg
Macro-avg
CheXpert
Micro-avg
Macro-avg
RadGraph Fl
BLEU (l)
ROUGE-L
F1-14
F1-5
F1-14 F1-3
F1-14+
F1-5+
F1-14+
F1-5+
LLaVA-Rad
42.1
40.6
39.0
16.5
LLaVA-T (topicy*
27.7
27.3
22.3
27.7
27.3
21.9
23.0
58
2.9
LLaVA-T (repart)*
78
11.1
7.4
7.0
6.6
11.0
7.4
7.3
6.6
1.6
0.8
0.1
1.6
LLaVA-IA (topie]*
82.7
89.6
41.4
57.3
LLaVA-TA treport)*
7B
79.6
89.2
61.9
87.2
79.5
89.1
69.6
87.7
31.1
41.1
22.5
38.6
4.1 TOPIC-GUIDED GENERATION IMPROVES CLINICAL ACCURACY
We first compare LLaVA-TA against state-of-the-ard models on the MIMIC-CXR-JPG test set. As
shown in Table. D our method establishes new state of the art, significantly outperforming all
baselines acToss both clinical accuracy (Smit et al. 2020 Irvin et al. 2019: Jain et al. 2021 and lexical similarity metrics (Papineni et al. 2002 Lin 2004).
Most critically, LLaVA-TA demonstrates substantial gains in metrics that measure factual accuracy. Compared to the strong LLaVA- -Ad baseline, our model improves the RadGraph Fl score from 29.4 to 34.3 and the CheXpert Macro-F1-14 from 39.5 to 62.4 at the report level. This significant improvement confirms that by decomposing the generation task into independent topics, our model produces more clinically precise and semantically correct findings. These results directly support

---

Published as a conference paper at ICLR 2026
central hypothesis: by preventing the model from relying inter-sentence linguistie priors, force it to ground its findings more strietly in visual evidence, leading to higher factual accuracy.
The superior performance on lexical metrics (c.g.. BLEU-4 of 24.8 and ROUGE-L of 42.6) indicates that this enhanced accuracy does not come at the cost of linguistic quality. On the contrary, by focusing on topic time, LLaVA-TA generates more precise and relevant language. Notably, our 7B parameter model also outperforms much larger, general-purpose models like Med-PaLM M (84B) and GPT-4V, highlighting that our structured, topic-guided supervision is more efficient and effective strategy than simply scaling language model size.
The performance gap between the topic-level and report-level evaluations is expected. Metries like RadGraph Fl score an entity as false positive if the ground truth contains no mention of it, even if the model correctly generates negative finding (e.g.. "Heart is normal"). The topic-level evaluation avoids this by only assessing topics present in the ground truth, providing cleaner measure of conditional generation quality. The results of LLaVA-Raal LLaVA-T, and LLaVA-TA are the median of 500 bootstrap iterations. More detailed results can be found in Appendix. D and E
4.2 TOPIC DISENTANGLEMENT UNLOCKS PERFORMANCE, WHILE ANATOMY ENHANCES EFFICIENCY
To isolate the contributions of our core design choices, we conduct series of ablation studies. We compare our full LLaVA-TA model against LLaVA-T (which uses topic-guided decomposition but only the original image, without anatomical masks) and the LLaVA-Rad baseline.
Topic Disentanglement is the Primary Driver of Performance. As shown in Table 1 simply introducing sopic-guided training (LLaVA-T) without anatomy masks already yiclds massive per- formance gains over the narrative-based LLaVA-Rad. RadGraph FI improves from 29.4 to 34.2, and CheXpert Macro-Fl-14 jumps from 39.5 to 63.0, This demonstrates that breaking the narrative flow is the single most critical factor in model's s success. It validates our hypothesis that forcing the model to generate independent, topic-specific findings is crucial for enhancing visual grounding and reducing hallucination.
Anatomy-Aware Awareness Enhances Farameter-Effiiiee Fine-Tuning While adding anatomical masks (LLaVA-TA) provides comparable performance in the full fine-tuning setting, its true value emerges in parameter-efficient fine-tuning (PEFT) regime. We conducted an experiment where only the multimodal MLP projector was trained, keeping the vision encoder and LLM frozen.
As shown in Table 2 under this MLP-only setting, LLaVA-TA dramatically outperforms both LLaVA-T and LL-VAARad For instance, on MIMIC-CXR, LLaVA-TH TA achieves a RadGraph Fl of 31.1, whereas LLaVAs VA-T and LLaVA-Raa lag far behind at 1.6 and 19.8, respectively. This sug- gests that when the L.LM's parameters are frozen, the explicit spatial cues from the anatomical masks are vital. They provide a strong, structured signal that helps the lightweight MLP learn to map vi- sual regions to the LLM's latent space effectively. This finding points promising direction for efficiently adapting large, frozen VLMs to the medical domain by using structured visual inputs.
Table 3: -nttof-distribution generalization performance on the IU-Xray dataset. LLaVA-TA main- tains its significant performance advantage over the VLa-R- baseline, especially on clinical accuracy metrics (RadGraph F1, CheXpert Macro-Fl). This demonstrates the robustness of the topic-guided approach, which learns a stronger vision-language mapping less dependent the lin- guistic priors the training set.
Model
Size
CheXpert
RadGraph
ROUGE-L
Micro-avg
Macro-avg
Micro-avg
Macro-avg
(l) (4)
FI-14
FI-5
FI-14 Fi-5
FI-14+
F1-5+
FI-14+
F1-5+
LLavA-Rad
50.3
38.9
48.2
39.3
30.9
33.5
32.0
22,2
LLaVA-T (topic)*
7B
87.8
92.7
61.0
85.7
87.0
92.3
65.8
85.9
43.8
50.1
28.7
58.1
LLaVA-T trepart)*
76.9
93.2
54.2
86.1
84.0
93.1
62.5
86.2
30.6
40.7
21.3
40.0
LLaVA-TA [toiec
87.4
92.0
60.8
84,5
86.7
91.9
65.9
85.0
45.0
50.8
28.7
57.7
LLaVA-TA treport*
7B
77.4
92.5
54.5
84.7
84.5
92.7
63.4
85.4
31.4
41.8
21.5
40.9
4.3
GENERALIZATION, ROBUSTNESS, AND INTERPRETABILIN
Finally, we assess whether LLaVA-TA's improvements are robust and whether the model's internal mechanisms align with our theory of improved visual grounding.
8

---

Published as a conference paper at ICLR 2026
Table 4: Out-of-distribttion performance on IU-Xray with MLP-only fine-tuning. ALaVA-TA main-
tains its substantial performance lead in this challenging generalization setting. The strong results confirm that the benefits of combining topic disentanglement with anatomy-awar inputs are robust and transfer effectively to new data domains, even with minimal model updates.
CheXpert
BLEU (1)
Model
Size
RadGraph F1
ROUGE-L
Micro-avg
Macro-avg
Micro-avg
Macro-avg
(4)
F1
F1.5
F1
FL.5
Fl
14+
F1
FI-14+
F1.5+
LLaVA-Raa Rad
7B
20.2
18.7
12.9
16.4
20.2
21.0
13.4
18.6
23.5
11.9
3.0
12.1
LLaVA-T (lopicy
43.5
47.2
44.0
51.2
29.5
42.9
7.3
7.8
LLaVA T (report)
7B
19.8
55.8
20.2
33.4
39.7
58.0
24.1
40.4
6.8
2.5
0.3
4.4
LLaVA-Ia TA [[oie))
7B
88.4
93.8
62.9
89.0
87.6
93.5
67.8
88.8
48.2
54.2
32.2
61.4
LLaVA TA (report)
7B
77.9
94.3
57.6
88.5
85.4
94.2
66.0
88.4
34.0
43.5
24.7
43.4
Strong Generalization to nitrof-Distributtin Data. We evaluated our model on the IU-Xray dataset, common fittof--iittibution benchmark. As detailed in Table LLaVA-TA demon- strates superior generalization compared to the strong baseline and our topic-only LLaVA-T abla- tion study under both in-distribution and out-of-distribution settings. This large margin suggests that by reducing the reliance on dataset-specific linguistic priors (i.e., narrative flow), our topic- guided method learns more fundamental and robust mapping between visual evidence and clinical findings, allowing it to adapt more effectively to new data distributions.
Interpretability and Visual Grounding Analysis. To validate the spatial fidelity of LLaVA-TA, we visualize its cross-attention maps during generation. As illustrated in Fig. 3 the model demon- strates consistent focus on relevant anatomical regions when generating topic-specific findings. For instance, when describing cardiac features, attention is strictly concentrated on the cardiac silhou- ette. This qualitative evidence confirms that our topic-guided strategy successfully fosters genuine spatial grounding, ensuring findings are derived from visual signals rather than textual priors.
We further analyze a challenging case in Fig. 4 where the generated report diverges textually from the ground truth. While the reference report notes "bibasal opacities," LLaVA-TA generates "a left retrocardiac opacity is present." Despite this linguistic mismatch, the attention visualization accu- rately highlights the posterior lower lung zones. This indicates that the model is not hallucinating; rather, it is maintaining internal consistency by grounding its specific prediction in valid visual fea- tures. This transparency is vital for clinical trust, as it allows radiologists to distinguish between model hallucinations and valid alternative interpretations of ambiguous visual data.
To move beyond qualitative inspection, we perform rigorous quantification of attention align- ment using the MS-CXR dataset (Boecking et al., 2022). which is a subset of the MIMIC-CXR- JPG dataset. MS-CXR is a phrase grounding benchmark containing curated image regions an- notated with radiologist-verified descriptions of eight pathology categories. We evaluate on all 176
pairs from the test subset using three metrics: (1) Hit Score (Hit) (Saporta et al.) (022), which credits the model if the maximum attention point falls within the ground- truth bounding box (BBox); (2) Contrast-to-noise ratio (CNR) (Boecking et al., (022). defined
02 where A and A represent regions inside and outside the BBox as CNR |M /al/ / + o²,
of the attention map, respectively; and (3) Soft Precision (SP) (Boecking et al.) (022), calculated as SP = (E. S(x, v)) (Ea/((atee S(x, u)), where S(x, is the attention map value the position (x. y)- We also report SP-k, where attention maps are binarized by retaining the top 1% of values before calculating precision.
The results, reported in Table as the mean across all tokens within the sentence associated with the ground truth BBox, reveal dramatic improvement. VL-TA doubles the Hit Score of the baseline (from 29.2% to 59.2%) and achieves consistently higher CNR and Precision scores. This quantitative leap confirms that decomposing reports by topic significantly reduces the model's re- liance on spurious correlations, forcing it to attend to the correct anatomical regions. Examples of visualization are provided in the Appendix
4.4 ABLATION STUDY
To quantify how much of the performance gain comes from dataset simplification and topic splitting, compare the original dataset (OriData) against our topic-decomposed dataset (CleanData) and

---

external segmen- Published as a conference paper at ICLR 2026
Table 5: Attention map quantification on the MS-CXR benchmark. LLaVA-TA demonstrates supe-
rior visual grounding across all metrics, effectively doubling the Hit Rate compared to the baseline.
Method
Hir
CNR(S))
SP(%)
SP-1(%)1
SP-3(%)
SP-5(%))
SP-10(S)1
LLaVA
31.9
29.3
27.0
LLaVA-TA
59.2
43,4
37,5
62.0
55.7
50.3
41.4
compare standard narrative generation (alIGTTopic, the concatenation of all the ground truth topics) against our proposed topic-guided generation (singleToppic, one topic at time).
Table { shows that simply training on the cleaned data improves the RadGraph F1 score by only 1.6%. While this helps, it does not account for the majority of the performance leap. Transition from "CleanData" to "CleanData singleTopic" yield a massive jump in RadGraph F1 from 32.4 to 46.5. This proves that the performance gain is primarily driven by our single-topic query mechanism, not by the simplified data format. The explicit querying forces the model to focus on factual accuracy rather than narrative style. [aalle/]dmoonstrace the necessity of the anatomical mask when the LLM frozen. Now, the model fails to generate accurate reports using the prompt alone ("CleanData singleTopic", RadGraph FI = 5.8). However, adding the mask ("+mask") restores performance to SOTA levels. This confirms that our "Anatomy-Awe -warmmee-rr"" design is crucial for guiding the visual encoder when the language model capacity is constrained.
Table 6: Ablation study with MLP+LLM fine-tuning.
Method
BLEU-T
BLEU-4 ROUGE-L Miero-FI-14 Macro-Fi-14 Micro-F1-5 Macro-FI-S Fi-RadGraph
OriDala
40.0
16.1
30.8
58.1
40.1
57.7
50.3
30.8
CleanData
46.4
26.2
40.0
63.3
40.7
66.0
57.3
32.4
OriData alIGTTopic
30.4
11.2
24.5
71.4
56.8
79.6
74.4
27.2
CleanData Tlpi
80.9
CleanData Tople
$0.9
32.1
60.9
83.7
72.2
90.0
87.9
46.5
CleanData + singleTopic mask
50.9
31.9
60.7
83.6
71.6
89.6
87.3
46.3
Table 7: Ablation study with MLP only fine-tuning.
Method
BLEU-1 BLEU- ROUGE-L Micro-Fi-14
Macro-F Micro-F1-5
Macro-Fi-5 FI-RadGraph
OriData
16.5
3,6
15.3
42.1
23.3
43.5
36.6
19.8
CatanData
9.9
1.4
10.8
31.8
17.7
35.1
24.2
10.4
OriDaaa TlGi
55.8
45,9
Cata + allGTTopic
10.6
1,5
11.9
33.1
20.0
37.6
29.6
12.8
CleanData + singleTopic
2.9
0.4
5.9
27.7
20.4
27.5
22.3
CleanData single Topic mask
46.1
27.4
57.3
82.7
70.0
89.6
87.5
41.4
DISCUSSION
In this work, we identified fundamental limitation in existing approaches Radiology Report Generation: an implicit optimization narrative flow. We provided empirical evidence that this ruto-regressive, sequential generation process encourages models to rely linguistic priors at the expense of strict visual grounding, a primary cause of factual errors and clinical hallucination. To ad- dress this, we introduced ILLVVA-TA novel framework that re-engineers report generation around a topic-gun lopic-guided paradigm. By decomposing the task into independent, maatomically-grounne clin- ical topics, VA-TA breaks the cross-sentence dependency and forces the model ground each finding directly in the relevant visual evidence. Our extensive experiments demonstrate that this approach sets a new SOTA on the MIMIC-CXR-JGG dataset in clinical accuracy (Fl-RadGraph: 30.8 + 46.5, CheXpert ficro-FF-14: 58.1 + 83.7) and OOD generalization (F1 -F1-RaGGraph on the unseen dataset IU-XRay: 19.8 - 45.0), proving that disentangling report's narrative structure to enforce independent, disually-grounder findings is a crucial step toward building more reliable medical AI.
Despite the strong performance, our work has several limitations that open avenues for future re- search. First, our framework uses a static topic list; a dynamic approach that learns topics directly from data could capture more nuanced findings. Second, LLaVA-TA relies tation model; an end-to-end architecture co-learning segmentation and generation could improve vision-language alignment. Finally, currently exclude comparative sentences. Extending this framework to incorporate longitudinal temporal reasoning is critical for real-world clinical utility. LLaVA-TA provides robust foundation for these future explorations.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
We acknowledge that all authors of this work have read and commit to adhering to the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
The processed dataset and model weights will be made publicly available on Hugging Face.
ACKNOWLEDGEMENTS
Our research funded by NIH 4R33-HD 105593. The content in the paper is solely the respon- sibility of the authors and does not necessarily represent the official views of the National Institutes of Health the other funding agencies.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
During the preparation this work the authors used ChatGPT order to improve the readability and language of the manuscript. After using this tool, the authors reviewed and edited the content as needed and take full responsibility for the content of this article.
REFERENCES
Shruthi Bannur, Kenza Bouzid, and al. Maira-2: Grounded radiology report generation, 2024.
URL https: org/ abs/ 6 04449
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel Coelho de Castro, Anton Schwaighofer, Stephanie Hyland, Maria Teodora Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez Valle, et al. Ms-cxr: Making the most of text semantics to improve biomedical vision-language processing. URL: http://xx. doi, org/10. 13026/990jvvb87, 2022.
Zailong Chen, Yingshu Li, Zhanyu Wang, Peng Gao, Johan Barthelemy, Luping Zhou, and Lei Wang. Enhancing radiology report generation via multi-phased supervision. IEEE Transactions on Medical Imaging, 2025.
Zhihong Chen, Yan Song, Tsung- Hui Chang, and Xiang Wan. Generating radiology reports via memory-driven transformer, 2022. URL htttpss//aaai...oo//a/b/////////
16056
foundaaiii Zhihong Chen, Maya Varma, Justin Xu, and al. vision-language foundation model to enhance efficiency of chest x-ray interpretation, 2024. URL. htt.:tpp.//arxivvoor///////2/2 12208
DeepSeck-AI, Aixin Liu, Bei Feng. Bing Xue, and et al. Deepseck-v3 technical report, 2025. URL htttps://axxiv.or//aaaa
Jean-Benoit Delbrouck, Justin Xu, Johannes Moll, Alois Thomas, Zhihong Chen, Sophie Ostmeier, Asfandyar Azhar, Kelvin Zhenghao Li, Andrew Johnston, Christian Bluethgen, et al. Automated structured radiology report generation. In Proceedings of the 63rd Annual Meeting of the Associ- ation Computational Linguistics Volume 1: Long Papers), pp. 26813-26829, 2025.
Dina Bemner-Fushmann Mare Kohli, Mare B Rosemman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing collection of radiol- ogy examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304- 2015.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, and et al. The llama herd of models, 2024.
URL
www.rr///a/////e abw/ 21783
11

---

Published as a conference paper at ICLR 2026
Tiancheng Gu, Dongnan Liu, Zhiyuan Li, and Weidong Cai. Complex organ mask guided radiology report generation. In Proceedings of the IEEE/CVF winter conference on applications computer vision, pp. 7995-8004, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https:
//axxiv.rg/abba. http://ggabba......//// 09685
Stephanie L. Hyland, Shruthi Bannur, Kenza Bouzid, and et al. Maira-l: A specialised large
multimodal model for radiology report generation, 2024. URL
2911.13660
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang. Matthew P. Lungren, Andrew Y. Ng. Curtis Langlotz, and Pranav Ra- jpurkar. Radgraph: Extracting clinical entities and relations from radiology reports. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https: /ptpr//eee/peeeeeeee.e htwp://wwwww.etttfoooornoddd-dee 55KK7V
AE Johnson, TJ Pollard, SJ Berkowitz, NR Greenbaum, MP Lungren, CY Deng, RG Mark, and Horng MIMIC-CXR. a de-identified publicly available database chest radiographs with free- text reports. DOI: https://doi. org/10. 1038/=41577-019-0333 nih. gov/pubmed/318317400, pp. 307,2019a. 2019a.
PMID: https://www. ncbi, nlm.
Alistair Johnson, Matt Lungren, Yifan Peng, Zhiyong Lu, Roger Mark, Seth Berkowitz, and Steven Horng. Mimic-cxr-jpg-chest radiographs with structured labels. PhysioNet, 101:215-220, 2019b.
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang. Tristan Nau- mann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and aision assis- tant for biomedicine in one day, 2023. URL https:ttss..m /ttttps://axiiii /ttpiv.xivvv...////// org/ abs 2206 00890
Chin-Yeu Yew Lin, ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74 81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL htt.s: / /aaaacaaaaa www.claaattoooo...oooooo.o
1013/
Aohan Liu, Yuchen Guo, Jun-hai Yong, and Feng Xu. Multi-grained radiology report generation with sentence-level amage-language contrastive learning. IEEE Transactions on Medical Imaging, 43(7):2657-2699 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-3499 2023.
OpenAI, Josh Achiam, Steven Adler, and al. Gpt-4 technical report, 2024. URL https://
08777
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Sang-Jun Park, Keun-Soo Heo, Dong-Hee Shin, Young- Son, Ji-Hye Oh, and Tae-Eui Kam. Dart: Disease-aware image-tex alignment and self correctrrg re-aligmment for trustworthy radi- ology report generation. In Proceedings of the Computer Vision and Pattern Recognition Confer- ence, pp. 15580-15589, 2025.
Adriel Saporta, Xiaotong Gui, Ashwin Agrawal, Anuj Pareek, Steven QH Truong, Chanh DT Nguyen, Van-Doan Ngo. Jayne Seekins, Francis G Blankenberg, Andrew Y Ng, et al. Bench- marking saliency methods for chest x-ray interpretation. Nature Machine Intelligence, 4(10): 867-878, 2022.
12

---

Published as a conference paper at ICLR 2026
Constantin Seibold, Alexander Jaus, Matthias A. Fink, Moon Kim, Simon ReiB, Ken Herrmann, Jens Kleesick, and Rainer Stiefelhagen. Accurate fine- ine-grained segmentation of human anatomy in radiographs via volumetric pseudo-labelingg 2023. /RL/ttpss//axii..o////// ab3rg//.
03934
Constantin Mare Seibold, Simon ReiB, M. Saquib Sarfraz, Matthias A. Fink, Victoria Mayer, Jan Sellner, Moon Sung Kim, Klaus H. Maier-Hein, Jens Kleesiek, and Rainer Stiefelhagen. Detailed annotations of chest x-rays via ct projection for report understanding. In 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press, 2022.
URL httphtts:: http:://bwwwc22c02 wttp:2222......p.....mmmmmm
mpg. de/ddee/055 påf
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pareek, Andrew Y Ng, and Matthew P Lungren. Chexbert: combining automatic labelers and expert annotations for accurate radiology report la- beling using bert. arXiv preprint arXiv:2004.0P9 09167, 2020.
Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable gooood radiology report generation. In 2023 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 7433-7442. IEEE, June 2023. doi: 10.11099c6) 2023.00718. URL http::/dx> doi wtg/10............ 1010//0.110//PPP.555. 1119/11PR99222 000.7299.22222222 00718
Ryutaro Tanno, David GT Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abi- gail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, et al. Collaboration between clinicians and lasion-laguuage models in radiology report generation. Nature Medicine, 31(2): 599-608, 2025.
Tao Tu, Shekoofeh Azizi, Danny Driess, and et al. Towards generalist biomedical ai, 2023. URL https://arxiv: org/ abs /bt./b//..222222
Peng Wang, Shuai Bai, Sinan Tan, and al. Qwen2-vl: Enhancing visiona model's percep- tion of the world at any resolution, 2024.
12191
Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, et al. A clinically acces- sible small multimodal radiology model and evaluation metrie for chest x-ray findings. Nature Communications, 16(1):3108, 2025.
Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Pre- ston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang, and Hoifung Poon. Biomedelip: mul- timodal biomedical foundation model pretrained from fifteen million scientific image-text pairs, 2025. URttttps::/
wrwia.avvv..... http://aaww....//abaaa.a/a/a/aaa 2303. 00915
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openzeview.ne. wet.:/orommidduuccll.......
13

---

Published as a conference paper at ICLR 2026
A DETAILED ANALYSIS OF THE NARRATIVE FLOW EXPERIMENT
Language Metrics
CheXpert-14 F1
40
BLEU-1
68
BLEU-4 ROUGE-L
30
Mirre1114
15
10
BoPrier
ortion leatk
onting lact
erting lank
blacking NoPricx
bask
Mlacking lastl
blacking
BoPria
lanck
enting kaoth
ortind
lastt
blacktng Nohrior
blackong
lastk
blecking
CheXpart-5 FI
RadGraph Fl
Micro-11:5
PiRadGraph
Macro F1.5
50
Micro F151
Macro F1
18
10.
16
oriteng
Notvior
ortimg
lack
entong hant
arting
blacking
Blacking
Back
backing
lamk
lonk
erting
Noprisa
orling
lavnk
orting
orttog
mask
blacking
blacking
Mack
Iaaek Stankteg
lackk
Figure 5: Detailed performance scores for the narrative flow experiment. The plots show the abso- lute scores for both the orilmg and blackImg conditions across four metric categories. These results illustrate the model's behavior with and without visual input under varying degrees of textual con- text.
We first define "Narrative" Bias as follows: Narrative Bias the tendency of an autoregressive model P(y. (vve, I) to maximize the likelihood of the next token y/ based primarily the linguistic probability P(yelyee) rather than the visual conditional probability P(
To provide empirical evidence for central hypothesis- that optimizing for narrative flow weak- ens a model's visual grounding- conducted a controlled experiment. This section details the experimental setup and provides in-depth analysis of the results.
A. EXPERIMENTAL SETUP
We used pretrained LLaVA-Rad model to generate the final K sentences of radiology report, given the preceding sentences as a textual prompt. This task was performed under two distinct conditions:
oriImg Condition: The model was provided with both the textual prompt and the corre- sponding ground-truth chest X-ray.
blackimg Condition: The model was provided with the same textual prompt but with blank (black) image, forcing it to rely solely on linguistic priors.
Denote the number of sentences for ground truth report is L. We split the front L K sentences to the prompt and let the model generate the last K sentences. We varied the amount of textual context by setting K to 3, 2, and (denoted lastk 3, lasth 2, lastk - 1). We also evaluated baseline NoPrior condition where the model generated the full report from scratch, K = L. The performance under all settings is shown Fig.
A.2 DETAILED PERFORMANCE ANALYSIS
The results in Figure 3 reveal several key trends that support hypothesis: conventional training, by treating the report as a monolithic text sequence, inadvertently teaches models to prioritize lin- guistic coherence over strict visual fidelity. This learned reliance on the textual context of previously
14

---

Published as a conference paper at ICLR 2026
generated sentences could become actively detrimental, allowing it to bias subsequent findings and override direct evidence from the image itself a process that we suspect weakens visual grounding and fosters clinical hallucination.
We analyze the behavior within each condition below.
Analysis of the orilmg condition
Textual Context Degrades Visual Grounding: The model achieves its highest scores in the NoPrior setting, where its generation is primarily guided by the visual evidence. The subsequent drop in performance across most metrics suggests that the preceding sentences introduce a strong linguistic bias, causing the model to generate text that follows the narra- tive, even at the expense of visual accuracy.
Strong Context Leads to Overfitting on Common Diseases: A notable outlier occurs in the CheXpert-144 Fl score lasth - 1. The Micro-FI score (which is weighted by label frequency) remains high, while the Macro-F1 score (unweighted) drops sharply. This divergence indicates that when given strong textual context and short generation target, the model defaults to predicting common, high-frequency diseases that are statistically likely to follow the prompt. relies on learned textual correlations rather than observing the image, leading to overfitting common pathologies.
Analysis of the blackImg Condition: Textual Context as Driver for Hallucination
In the absence of visual imput, the model's generation driven entirely by the textual prompt. This setting allows us to directly measure how linguistic context induces hallucination.
Textual Priors Trigger Plausible, Structured Hallucinations: The most direct evidence hallucination is the jump in RadGraph Fl score when any textual context is provided (lastk settings) compared to none (blackImg NoPrior). RadGraph requires both clinical finding and its anatomical location (e.g., "opacity" the "lungs"). The model's ability generate structurally correct entities from text alone demonstrates that the preceding sentences trigger a chain of plausible, yet entirely fabricated, clinical statements. This is not random guessing; is coherent, context-driven hallucination.
More Context Narrows Hallucinations and Reduces "Lucky Guesses": While it may seem counter-intuitive, performance scores generally decrease as more textual context is provided (from lastk 3 to lastk - 1). This is not because the model is hallucinating less, but because the evaluation becomes stricter. longer prompt creates more specific narrative context, forcing the model generate be incorrect) hallucination. When generating three sentences (lastk - 3), there more opportunity for a random part of the hallucinated text overlap with one of the three ground-truth sentences-a "lucky guess." When generating only one sentence (lastk - 1), the model must match a single, specific target, and any deviation results in score of zero.
specific (and thus more likely to
For Common Diseases, Textual Context Overwrites Visual Evidence: The most strik- ing finding is the trend of the Micro-Fl-14 score in the blackimg condition. Unlike other context is provided, eventually matching the perfor-
metries, this score increases mance of the orilmg condition at the lastk setting. This provides definitive proof of our hypothesis. The Micro-Fl score weighted by the prevalence of each disease label. The rising score shows that with strong textual priors, the model becomes highly proficient hallucinating the most common diseases, to the point where the actual image provides no additional benefit. The linguistic patterns learned from the training data are so strong for these common cases that they completely override the need for visual input.
15

---

Published as a conference paper at ICLR 2026
B TOPIC-SEGMENTATION MAPPING
This mapping is manually constructed between topics at the level disease, organ, and other to the seven segmentation categories provided by the EXAS(Seebbl) al. 2023 system.
Table 8: Mapping between topics and segmentation labels
Topic
Segmentation labels
disease-atelectasis
lung halves
disease-carrdommeally
heart region
conooliaionniii
lung halves
discasc-edema
lung halves
ennaagee cardiomediastinum
mediastinum
disease-fractie
ribs, all vertebrae
disease-lung lesion
lung halves
disease-lung opacity
lung halves
satease-pleural effusion
lung halves
pesease--leurgy other
lung halves
disease- disease-pneumen pncumonia
lung halves
lung halves
organ-heart
heart region
organ-lungs
lung halves
organ-pleura
lung halves mediastinum
organ-mediastinum
organ-bones
ribs, all vertebrae
organ-diaphragm
diaphragm
support devices
all
patient status
all
other
16

---

Published as a conference paper at ICLR 2026
C PROMPT FOR SPLITTING REPORTS INTO SENTENCES BASED ON TOPIC
You are an AI radiology assistant specialized in preprocessing chest X-ray reports for vision- language model training. Your task is extract and reorganize findings into standardized sentences, each aligned with specific topic, while ensuring clarity and suitability for paired image-texl learn- ing.
Rules for Processing Reports:
1. Hierarchical Topic Classification
Classification Priority: Assign each sentence to the most specific applicable topic using this order:
1.1 Pathology-Level Topics: disease-atelectasis, disease-cardiomegally disease-
consolidation, disease-edema, disease-enlarged cardiomediastinum, disease- fracture, disease-lung lesion. disease-lung opacity. aisease-pleural effusion, disease-pleural other, disease-pneumonia,
1.2 Anatomy-Leve Topics (Only if no pathology match): organ-heart, organ-lungs, organ-pleura, argan-mediattium,, organ-bones, organ-diaphragm.
1.3 Special Categories (Only if no previous topics matches): support devices, patient status, other.
If a sentence covers multiple topics, split it into separate sentences (e.g., "Car- diomegaly and pulmonary edema" should be split into "Cardiomegaly is present." and "Pulmonary edema is present.")
support devices: Implanted external hardware directly visible imaging (c.g., pacemakers, catheters, surgical clips). Exclude historical procedures without visible remnants.
patient status: Clinically relevant history (e.g., post-surgical changes, intubation) that may explain imaging findings but does not describe visible devices/anatomy. other for sentences that don't fit predefined topics but are visually relevant.
2. Single Finding per Sentence:
Each sentence must describe only one finding/disease (e.g., "The heart is enlarged with pleural effusion" should be split into "The heart is enlarged." and "Pleural effusion is present.")
3. Handle Comparisons:
If sentence references prior images (e.g.. "Increased opacity compared prior"), reformat it to describe the current study only.
Example:
Before: "The pleural effusion has increased since last study."
After: "Pleural effusion is present."
4. Exclude Visual Content
Remove the following:
Speculation/Intern Interpretation: e.g., "suggestive of pneumonia", "likely due to CHF"
Recommendations: e.g., "Recommend follow-up CT.""
Technical Artifacts: c.g., "Left lung is hard to see in the image."
5. Negations and Absence
Keep negative findings but phrase them affirmatively:
Example:
Before: "No pneumothorax seen."
Alter: "Pneumothorax is absent."
6. Clarity
Use short, concrete phrases that directly map to visual features.
Examples:
Bad example: "TTere some mild interstitial markings."
17

---

Published as a conference paper at ICLR 2026
Good example: "Mild interstitial opacities present."
Output Format:
[ "topic": "disease-lung opacity". "sentences": ["Substantial hyperexpansion of the lungs is present."], "topic": "organ-lungs", "sentences": ["Prominence of central pulmonary arteries is present.". "Severe emphysema present."), "topic": "support devices", "sentences": ["A tra- cheostomy tube present.]]]
Only output the JSON list without additional descriptions.
18

---

Published as a conference paper at ICLR 2026
D PERFORMANCE OF FINE-TUNED MLP AND LLM
The following are the complete performance results for our model LLaVA-TA and the strong base- line model LLa under the settings fine-tuning both the MLP and the language model.
Following LLaVA-Rad Zambrano Chaves et al. (2025), we obtain the median and 95% bootstrap confidence intervals from 500 resampling iterations.
Table Detailed performance on the MIMIC test set when fine-tuning both MLP and LLM
CheXpert
BLEU
Model
Statistic
Micro-avg
Macro-avg
Micro-avg
Macro-avg
RadGraph F1
(I)
(4)
ROUGE-L
F1-14
F1-5
F1-14 FI-5
F1-14+
F1-5+
Fi-14+
F1-S+
median
58.1
57.7
57.7
59.8
42.9
53.0
40.0
16.1
3088
LLaVA-Rad
Cl-low
57.2
56.3
38.6
48.6
56.9
58.6
41.5
51.6
30.2
39.4
15.6
30,3
CI-high
59.0
59.1
41.5
52.1
58.6
61.0
44.2
54.4
31.4
40.5
16.5
31.3
median
90.3
60,9
LLaVA-T (Topicy*
CI-low
83.2
89.5
70.7
87.2
83.2
89.8
77.1
88.1
45.9
50,3
31.6
60.5
Cl-high
84.2
90.6
73.6
88.6
84.3
90.9
79.3
89.4
47.2
51.5
32.7
61.4
median
80.2
88.5
63.0
87.0
81.6
88.7
71.4
87.7
34.2
43,9
25.7
41.8
LLaVA-T (Report)
Cl-low
79.5
87.8
61.4
86.2
81.0
88.
69.9
86.9
33.8
43.4
25.3
41.4
Cl-high
89.1
89.3
73.0
median
83.5
89.6
71.5
87.3
83.6
90.1
77.9
88.3
44.0
50.9
31.8
60.6
LLAA-TA TA (Topicy*
C1-low
83.0
89.0
70.1
86.5
83.1
89.5
76.8
87.5
43.4
50.3
31.3
60.2
Cl-bigh
84.1
90.2
73.0
88.0
84.2
90.6
78.9
89.0
$1.5
32.4
61.2
median
80.0
89.0
62.4
86.8
81.0
88.7
70.7
87.4
34.3
43.7
24.8
42.6
LLaVA-IA "AA (Report)*
Cl-low
79.3
88.4
60.8
86.0
80.5
88.1
69.3
86.7
33.8
43.3
24.4
42,3
CI high
80.7
89.6
64.1
87.5
81.6
89.3
72.2
88.2
34.8
44.1
25.2
43,0
Table 10: Detailed performance on the IU-Xray dataset when fine-tuning both MLP and LLM
CheXpert
BLEU
Model
Statistic
Micro-avg
Macro avg
Micro avg
Macro avg
RadGraph F1
( (4)
ROUGE-L
FI-14
F1-5
FI-14
FI-5
FI-14+
F1-S+
FI-14+
FI-5+
median
50.3
38.9
28.8
29.9
48.2
39.3
30.9
33.5
19.8
32.0
6.4
22.2
LLaVA-Rad
Cl-bigh
51.0
40.4
30,0
32.2
48.9
40.7
32.1
35.2
20.0
32.2
6.5
22.3
median
87.8
92.7
61.0
85.7
87.0
92.3
65.8
85.9
43.8
50.1
28.7
58.1
LLaVA-T (topic)*
Cl-ow
Cl-high
88.2
93.3
62.5
87.1
87.4
92.9
67.4
87.2
44.4
50.7
29.3
58.7
median
76.9
'952
54.2
86.1
"ST.0"
93.1
62.5
$6.2
30.6
40.7
21.3"
70.0
LLaVA-T (report)
low
76.1
92.6
52.4
84.6
83.3
92.4
60.8
84.7
30.3
40.4
21.0
39.8
Cl-high
77.8
93.9
55.8
87.4
84.7
93.6
64.0
87.5
31.0
41.0
21.6
40.3
median
LLaVA-TA (topic)*
Cl-low
87.0
91.4
59.2
82.9
86.2
91.3
64.2
83.5
44.4
50.2
28.2
57.1
Cl-bigh
87.9
92.7
62.5
86.0
87.1
92.6
67.6
86.4
45.6
51.4
29.3
58.2
median
77.4
Ty2.5
34.3
84,7
99.7
63.4
83.4
31.3
41.8
2T.5
70.3
LLaVA-TA (report)*
Cl-low
76.6
91.9
$2.9
K3.2
83.8
92.1
61.7
84.0
31.1
41.5
21.3
40.6
Clihigh
19

---

Published as a conference paper at ICLR 2026
E PERFORMANCE OF FINE-TUNED ONLY MLP
The following are the complete performance results for our model LLaVA-TA and the strong base- line model LLd under the settings fine-tuning only the MLP.
Following LLaVA-Rad Zambrano Chaves et al. (2025), we obtain the median and 95% bootstrap confidence intervals from 500 resampling iterations.
Table 11: Detailed performance on the MIMIC test set when fine-tuning only MLP layers
CheXpert
BLEU
Model
Statistic
Micro-avg
Macro-avg
Micro-avg
Macro-avg
RadGraph F1
(4)
ROUGE-L
F1-14
F1-5
F1-14
F1-5
F1-14+
F1-5+
F1-14+
F1-5+
median
42.1
48.5
23.3
36.6
40.6
48.8
25.1
39.0
19.8
3.6
LLaVA-Rad
Cl-low
41.0
47.0
22.3
35.1
39.4
47.3
24.1
37.5
19.4
16.3
3.5
15.1
Cl-high
43.4
50.1
24.4
38.3
41.7
50.2
26.2
40.6
20.2
16.8
3.8
13.5
median
LLaVA-T (topie)*
Cl-low
27.1
26.6
19.7
21.5
27.0
26.6
21.2
22.3
5.6
2.9
0.3
5.8
Cl-bigh
28.3
21.0
23.1
28.3
28.4
22.6
23.9
2.9
0.4
median
11.1
7.4
7.0
6.6
11.0
7.4
7.3
6.6
1.6
0.8
0.1
1.6
LLaVA-T treport)
Cl-low
10.9
7.1
6.8
6.3
10.7
7.1
7.1
6.4
1.5
0.7
0.1
1.6
Cl-high
7.7
7.2
median
82.7
89.6
70.0
87.5
82.6
90.1
76.8
88.5
41.4
46.1
27.4
57.3
LLMA TA (topic)*
Cl.low
82.2
88.9
68.6
86.7
82.1
89.4
75.7
87.7
40.8
45.4
26.9
56.8
CI-high
83.2
90.2
71.4
88.3
83.2
90.6
77.7
89.1
42.1
46.7
28.0
median
79.6
89.2
61.9
87.2
79.5
89.1
69.6
87.7
31.1
41.1
22.5
38.6
LLaVA-IA TA (report))
Cl-tow
78.9
88.6
60.4
86.5
79.0
88.6
68.2
87.1
30.6
40.5
22.1
38.3
CI high
80.2
89.8
63.4
87.9
80.1
89.7
71.0
88.4
31.6
41.8
23.0
39.0
Table 12: Detailed performance on the IU-Xray dataset when fine-tuning only MLP layers
CheXpert
BLEU
Model
Statistic
Micro-avg
Macro avg
Micro avg
Macro avg
RadGraph F1
(l) (4)
ROUGE-L
FI-14
FI-5
FI-14
FI-5
FI-14+
F1-S+
FI-14+
FI-5+
median
20.2
18.7
12.9
16.4
20.2
21.0
13.4
18.6
23.5
11.9
3.0
12.1
LLaVA-Rad
Cl-bigh
20.8
19.8
13.5
17.6
20.7
22.1
14.0
19.7
23.7
11.9
3.0
12.2
median
43.5
47.2
25.2
38.0
44.0
51.2
29.5
42.9
388
0.3
7.8
LLaVA-T (tupic)*
Clhhggh
44.2
48.5
26.1
39.5
44.7
52.3
30.3
44.5
7.5
3.9
0.3
7.9
median
39.8
555.8
20.2
33.4
39..7
$8.0
27.1
40.4
23
0.3
4.4
LLaVA-T T (report)
low
38.8
54.5
18.8
30.4
38.8
56.6
22.6
36.8
6.5
2.5
0.3
4.3
Cl-high
40.8
57.3
21.8
37.5
40.7
59.5
25.6
43.8
7.0
2.6
0.4
4.4
median
LLaVA-TA (topic)*
Ci-low
88.0
93.2
61.3
87.6
87.2
92.9
66.2
87.4
47.6
53.6
31.7
60.9
Cl-bigh
88.9
94.4
64.5
90.2
88.1
94.1
69.5
89.9
48.9
54.8
32.8
61.9
meddian
77.5
T94.3
37.6
83.5
"85.4
94.2
66.0
88.4
3410
45.3
24.7
73.7
LLaVA-TA (report)*
Cl-low
77.0
93.7
55.7
87.0
84.7
93.7
64.2
87.0
33.6
43.2
24.4
43.1
Cl-high
20

---

Published as a conference paper at ICLR 2026
F ATTENTION MAP QUANTIFICATION EXAMPLES
In this section, we present qualitative comparisons attention maps generated by LLaVA-TA and LLaVA-Rad To ensure representative evaluation across the performance distribution, stratify the test samples based on their Soft Precision scores (specifically using the top-3% binarization threshold, denoted SP-3).
We categorize the results into three performance tiers for each model:
High Performance:
Samples falling within the top 30% of SP-3 scores.
Medium Performance: Samples falling between the 30th and 70th percentiles.
Low Performance: Samples falling within the bottom 30% of SP-3 scores.
We visualize representative example from each tier in Fig- Fig.1 and Fig. respectively.
As observed across all three tiers, LLaVA-TA consistently produces more focused and anatomically
precise attention maps compared LLaVA VA- This qualitative evidence aligns with the quantita- tive results reported in Table reinforcing the superior visual grounding capabilities of our proposed framework.
LaaVWHad Cardiomegary 5h-2: DAR0Z
Salie
Figure 6: High Performance Examples. Attention map visualizations selected from the top 30% of SP-3 scores for each model.
21

---

Published as a conference paper at ICLR 2026
LLaA-T44
Wet
presett
LLWWhRRd Abelectacs SP-3: 0.3655
Ei1
dbe
plee
Ural
U5
knns
and
subsegient
Figure 7: Medium Performance Examples. Attention map visualizations selected from the 30th to thth percentile range of SP-3 scores.
22

---

Published as a conference paper at ICLR 2026
LLaVA-IA | Loeema 3AA-3 U.10b
LLa Edema 5P-3: 0.0826
Figure 8: Low Performance Examples. Attention map visualizations selected from the bottom 30% of SP-3 scores.
23

---

Published as a conference paper at ICLR 2026
G SENSITIVITY TO UPSTREAM ERRORS
G.1 ROBUSTNESS TO SEMANTIC NOISE
We first analyzed the error modes of the DeepSeek-V. pre-processing. We found the dominant eπror was omission (failing to extract a finding) rather than hallucination. To quantify our model's sensitivity this, we conducted Sensitivity Analysis via Topic Dropout.
We intentionally corrupted the training data by randomly dropping 1-2 topics from 5% of the training samples, simulating scenario with significantly higher upstream omission rates.
Table 13: Evaluation of upstream errors (Topic split). We only train the MLP layers and keep both the vision encoder and the L.M frozen.
Method
BLEU-1
BLEU-4
ROUGE-L Micro-F1-14
Macro-F1-14 Micro-F1s 1-5 Macro-F1-5 F1-RadGraphi
LLa
16.3
13.3
42.1
23.3
48.5
36.6
19.8
LLaVA-TA MAA-TTA-randmmDoop
45.4
27.0
57.1
82.7
69.7
89.5
87.3
40.9
87.5
As shown in Table 13 the model trained corrupted data shows negligible performance degrada- tion (RadGraph F1: 41,4 - 40.9). This confirms that our framework is highly robust to upstream errors. Since method treats each topic query independently, upstream omissions act merely reduction in dataset size rather than providing incorrect supervision.
G.2 ROBUSTNESS TO VISUAL NOISE
To evaluate robustness against errors in the upstream segmentation module, we quantified the qual- ity of all 224,105 segmentation masks based on their area ratios. We classified masks into High (Z S 1), Mid 7 < 2), and Low (Z 2 2) quality groups based on their deviation from the population mean, where Z arca_ratio-1 y and a are the mean and standard deviation for the current segmentation mask's 's organ group.
a
(1) The analysis of the segmentation quality is done by calculating the relative mask area ratio across organs, as shown in Table 14
Table 14: Analysis of segmentation quality by organ (area ratio statistics)
Organ
Median
Q1
Q3
Mean Std
Abdomen
All Vertebme
0.114
0.094
0.133 0.113 0.027
Diaphragm
0.241
0.192 0.292 0.243 0.073
Heart Region
0.083
0.072 0.09 0.084 0.019
Lung Halves
0.373
0.311 0.445 0.381 0.094
Mediastinum
0.180 0.160
Ribs
0.256
0.209
0.300 0.253 0.064
(2) Table [5 is the distribution the three quality groups across organs.
Table 15: Distribution of quality groups across organs
Organ
High (Z
Vid (1 2) Low (7 > 2)
Abdomen
131,509
62,124
10,412
All Vertebrae
146,879
69,693
7.533
Diuphrugm
151,038
62,980
10,087
Heart Region
156,214
54,983
12,908
Lung Halves
148.235
67,
8,752
Mediastinum 150,300
64,145
9,660
Ribs
147,662
68,259
(3) Finally, the results of the three mask-quality groups are shown in Table 16 and 17
Table 16 and 17 demonstrate two key findings:
Correlation with Clinical Accuracy: Crucially, observe a positive correlation between mask quality and clinical classification performance (Micro-F1-14), In the MLP+LLM setting, the High- Quality group achieves 84.0, outperforming the Low-Quality group (80. 1). This validates the effec-
24

---

Published as a conference paper at ICLR 2026
Table 16: Model performance different mask-quality groups with Fine-tuning MLP+LLM
Group
BLEUI-1
BLF(U44
ROUGE-I Micro-Fl-14 Macro-FI-14 Micro-Fl-5 Macro-F1-5 Fl-RadGraph
LLaVA Rad
40.0
16.1
30.8
58.1
40.1
57.7
50.3
30.8
(low)
12.5
LLaVA-TA (mid)
53.5
34,9
63.2
82.9
65.5
89.4
87.2
47.9
LLaVA-IA (high)
50.0
30.9
59.7
71.6
89.7
87.4
45.7
LLaVA-TA (all)
50.9
31.8
60.6
83.5
71.5
89.6
87.3
44.0
Table 17: Model performance on different mask- groups with Fine-tuning MLP only
Group
BLEU-1
BLEU-4 ROUGE-L Micro-F1-14 Macro-F1-14 Micro-F1.5 Macro-F1-5 F1-RadGraphi
LLaVA Rad
16.5
LL TA (low)
47.9
26.6
55.6
77.5
53.1
90.8
88.6
37.0
LLaVA-TA (mid)
49.6
31.3
60.4
88.
44.2
LLAVA-TA (high)
44.8
26.3
56.2
83.2
70.8
89.3
87.2
40.5
LLAVA-TA (all)
46.1
27.4
37.3
82.7
70.0
89.6
87.5
41.4
tiveness of our Anatomy-Aware design, demonstrating that accurate anatomical grounding directly contributes to diagnostic precision.
Robustness via Soft Guidance: Even on the "Low Quality" group (masks with extreme area devia- tions), the model achieves RadGraph F1 of 37.0 (MLP setting), which is nearly 2x the performance of the VLLaAA-Ra Rad baseline (19.8). This confirms our Soft Guidance hypothesis: because the seg- mentation acts spatial prompt rather than hard crop, the model retains access to the global context and remains effective even when masks are imperfect.
25


Reviews:
[{"review_id": "debFhJeeEz", "reviewer": "Reviewer_VWZ7", "summary": "This paper identifies a key flaw in current radiology report generation models: their narrative flow causes them to rely increasingly on language context rather than visual evidence, leading to hallucinations and factual errors. To address this, the authors propose LLaVA-TA (Topic-guided and Anatomy-aware), which restructures report generation into independent, topic-specific sentences aligned with corresponding anatomical regions. By guiding the model to generate findings per topic instead of following a continuous narrative, LLaVA-TA significantly improves factual accuracy, visual grounding, and interpretability across multiple radiology benchmarks.", "questions": "Please see the weakness.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper starts from a very insightful and underexplored motivation: rethinking why current radiology report generation models tend to overproduce normal descriptions and underrepresent abnormalities. Instead of merely improving architectures or datasets, the authors identify a fundamental bias in the narrative flow paradigm, where sequential language modeling causes models to ignore visual cues and rely excessively on linguistic priors.\n\nThe proposed topic-guided and anatomy-aware framework is a clear and elegant solution: by decomposing reports into topic-level units and generating findings region by region, the model enforces stronger visual grounding and factual consistency. This design directly addresses the identified failure mode rather than treating it as noise or data imbalance.\n\nThe experiments are comprehensive and convincing, showing substantial gains across factuality and grounding metrics on multiple benchmarks. The presentation is also clear and well-structured, with strong empirical evidence and visual explanations supporting the claims.", "weaknesses": "While the paper presents a clear and well-motivated framework, there are a few important limitations that should be addressed to strengthen the work.\n\nFirst, the method design could be made more comprehensive. Unlike prior work such as RGRG, which explicitly classifies each anatomical region as normal, abnormal, or not described, this paper only generates topic-specific sentences without explicitly distinguishing whether a region is normal or contains pathology. Such classification could make the model more interpretable and closer to real radiology reasoning, where identifying both the presence and absence of findings is equally important. Integrating this step could also prevent redundant or missing descriptions during generation.\n\nSecond, the evaluation setup raises potential fairness concerns. The proposed approach simplifies the report structure by rewriting reports into short, topic-level factual statements, removing stylistic and narrative variations. This preprocessing makes the output text inherently easier to match with reference reports under metrics like BLEU or RadGraph F1. In contrast, existing baselines such as R2Gen or RGRG are evaluated on the original, noisier report format, where stylistic diversity and longer narratives make the task harder. To ensure fair comparison, the authors should retrain or evaluate prior models on the same topic-decomposed version of the dataset, or at least quantify how much of the performance gain comes from dataset simplification versus model improvements.\n\nOverall, while the paper introduces a meaningful direction and shows strong results, its methodological completeness and experimental fairness could be improved. A more balanced evaluation, both in terms of report structure and the explicit modeling of normal/abnormal reasonin, would make the contribution more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KRfO4KME7n", "reviewer": "Reviewer_vVhP", "summary": "This paper addresses a critical issue in vision–language models for radiology report generation: the over-reliance on narrative flow, which leads to factual inaccuracies and weak visual grounding. The authors empirically demonstrate that as textual context increases, models such as LLaVA-Rad rely less on the image itself. To mitigate this, they propose LLaVA-TA, a topic-guided and anatomy-aware fine-tuning framework that decomposes radiology reports into discrete, topic-specific findings (e.g., lungs, heart, pleura) rather than generating a sequential narrative. The framework uses DeepSeek-V3 to split reports by topic, a segmentation model (CXAS) to provide anatomical masks, and trains Vicuna-7B-based models to generate topic-specific sentences conditioned on both the global and local images. Experiments on MIMIC-CXR and IU-Xray show strong improvements in RadGraph F1 and CheXpert metrics, significantly outperforming LLaVA-Rad and other medical VLMs. The authors further analyze model interpretability through attention maps and discuss limitations around static topic ontologies and reliance on external segmentation models.", "questions": "Can the authors quantify how topic decomposition affects semantic completeness or diagnostic recall compared to full-narrative generation?\n\nHow sensitive is the model to the choice of ontology or to errors in DeepSeek-V3’s report splitting?\n\nCould this approach generalize to multimodal datasets (CT, MRI) or longitudinal exams where temporal context is key?\n\nIs there any evidence that radiologists prefer topic-wise generation in practice, or that this improves interpretability during review?\n\nThe experiments assume ground-truth disease labels at inference; how would the model perform without this assumption?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents a clever and well-motivated reformulation of the radiology report generation problem. By decomposing reports into independent topics, it identifies a key failure mode—narrative bias—that has not been explicitly quantified before. The authors provide clear empirical evidence that textual coherence can override visual grounding, demonstrated through the “blank image” experiment. The proposed LLaVA-TA framework is conceptually simple yet effective, improving factual accuracy without resorting to larger model scaling. The paper’s experimental rigor is commendable, including ablations on topic disentanglement and anatomy-aware guidance, parameter-efficient fine-tuning, and out-of-distribution generalization. Visualizations of attention maps lend interpretability and practical relevance for clinical use. Overall, the work offers a valuable diagnostic insight into a core weakness of narrative-based report generation.", "weaknesses": "While the empirical findings are interesting, the novelty and technical depth of the proposed method are somewhat limited. The “topic-guided generation” mainly relies on pre-processing via an external LLM (DeepSeek-V3) and a segmentation model, combined with a straightforward adaptation of LLaVA-Rad; the contribution lies more in data structuring than algorithmic innovation. The reliance on hand-crafted topic ontologies and static segmentation mappings reduces generalizability, especially for unseen pathologies or imaging modalities. Moreover, while the study provides large quantitative gains, it is unclear whether these improvements persist in realistic clinical report generation, where topics and findings are interdependent and contextually nuanced. The “report-level” evaluation remains partially artificial, as the model is prompted with ground-truth topic sets rather than autonomously selecting them, making the benchmark setup favorable to the proposed method.\n\nIn addition, although the attention visualizations are appealing, they are qualitative and anecdotal, lacking rigorous evaluation of interpretability or human expert validation. The paper also misses a more critical discussion of semantic completeness—whether topic-wise independence may lead to omission of cross-topic findings (e.g., cardiopulmonary interactions). Finally, despite the strong experimental section, the writing can feel overly verbose, with extensive metrics and tables that could be summarized more effectively, while the theoretical grounding of the “narrative flow” hypothesis remains intuitive rather than formalized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tQMfx3DG4W", "reviewer": "Reviewer_mzcd", "summary": "This paper addresses a fundamental limitation in current Vision-Language Models (VLMs) for radiology report generation (RRG): their tendency to prioritize narrative coherence over visual grounding. The authors argue that conventional sequential, narrative-based training encourages models to rely excessively on linguistic priors and inter-sentence correlations, which can undermine factual accuracy and lead to clinical hallucinations.", "questions": "see weakness", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is novel, combining with anatomy.\n2. Strong Empirical Evidence for the Hypothesis", "weaknesses": "1. Dependence on External Tools and Heuristics:\nThe framework relies heavily on external systems — DeepSeek-V3 for report decomposition and CXAS for anatomical segmentation. This raises questions about scalability, potential propagation of upstream errors, and domain transferability.\n2. Theoretical grounding is limited:\nThe paper lacks a formal definition of “narrative bias” and a clear explanation of how topic-guided supervision improves representation learning.\n3. Lack of Theoretical Justification for Topic Independence Assumption:\nThe core idea of decomposing radiology reports into independent topics presumes that findings from different anatomical regions are conditionally independent given the image. In reality, many radiological pathologies exhibit inter-regional correlations (e.g., cardiomegaly co-occurring with pulmonary edema).  It may conduct ablation experiments on different partitioned parts to demonstrate the impact of combined inputs and single inputs on the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NZs4p9hKA", "reviewer": "Reviewer_ccY6", "summary": "The paper presents a method for radiology report generation. It it based on an earlier method (LLavaRad), where the major extension is that it performs a segmentation of the radiology images and that the system is trained using these segmented images, instead of the image a a whole. The segments obtained are then linked to textual topics. A significant improvement over the existing method is obtained although less dramatic for the improvement over LlavaRad.", "questions": "- what are exactly the changes compared to the method in the reference i.e. LlavaRad\n- you indicate that you are throwing away uncertain statement like \"there might be ...\". But aren't these important for the doctor who wrote that report? Aren't you throwing away elements that are uncertain but relevant? \n- By modelling the data in terms of topic there seems a clear connection to medical knowledge graphs. Is that indeed the case (you do mention an ontology but it remains a bit vague what that is).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Very well written paper where the method is first motivated by experimentally showing that using longer contextual information on the text side leads to less importance of the image information.\n- There is a clear method defined.\n- The experiments are all answering a specific question for the methodology. \n- The results are significantly better than the baseline method", "weaknesses": "- Somewhat limited in terms of innovation over the existing method\n- The literature could be more recent (limited number of papers from 2025/2024) although this is a hot topic in many venues. Also many refs are basic elements used and not really related work. \n- Ignores uncertain information in the caption while they might be highly relevant. for a clinician / doctor \n- Focus on one family of models (LLava)\n- The rad graph F1 measure is core. Although it is not a contribution of the paper as coming from another reference, it needs more explanation and embedding in the paper. No detailed description needed but at least enough to relate it to the method and the experiments. \n\nSmall typo:\nP4: should be <image> now it has strange symbols", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)