Controllable Chest X-Ray Report Generation from Longitudinal Representations
Authors: Francesco Dalla Serra
Chaoyang Wang
Fani Deligianni
Jeff Dalton
Alison Q O'Neil
Venue: EMNLP 2023 Findings (2023)
Keywords: Controllable Report Generation
Longitudinal Chest X-Rays
Multimodal Transformer

Abstract: Radiology reports are detailed text descriptions of the content of medical scans. Each report describes the presence/absence and location of relevant clinical findings, commonly including comparison with prior exams of the same patient to describe how they evolved. Radiology reporting is a time-consuming process, and scan results are often subject to delays. One strategy to speed up reporting is to integrate automated reporting systems, however clinical deployment requires high accuracy and interpretability. Previous approaches to automated radiology reporting generally do not provide the prior study as input, precluding comparison which is required for clinical accuracy in some types of scans, and offer only unreliable methods of interpretability. Therefore, leveraging an existing visual input format of anatomical tokens, we introduce two novel aspects: (1) longitudinal representation learning ‒ we input the prior scan as an additional input, proposing a method to align, concatenate and fuse the current and prior visual information into a joint longitudinal representation which can be provided to the multimodal report generation model; (2) sentence-anatomy dropout ‒ a training strategy for controllability in which the report generator model is trained to predict only sentences from the original report which correspond to the subset of anatomical regions given as input.
We show through in-depth experiments on the MIMIC-CXR dataset how the proposed approach achieves state-of-the-art results while enabling anatomy-wise controllable report generation.

Controllable Chest X-Ray Report Generation from Longitudinal
Representations
Francesco Dalla Serra 1.2
Chaoyang Wang
Fani Deligianni
Jeffrey Dalton
Alison Q O'Neil
Canon Medical Research Europe, Edinburgh, United Kingdom University of Glasgow, Glasgow, United Kingdom 3 University of Edinburgh, Edinburgh, United Kingdom francesco. aalllaaserrrrrr medi cal. canon
Abstract
Current Scan
Report Generator
Generated Report
Radiology reports are detailed text descriptions of the content of medical scans. Each report describes the presence/absence and location of relevant clinical findings, commonly includ- ing comparison with prior exams of the same patient to describe how they evolved. Radiol- ogy reporting is time-consume consuminn process, and scan results are often subject to delays. One strategy to speed up reporting is to integrate automated reporting systems, however clinical deployment requires high accuracy and inter- pretability. Previous approaches automated radiology reporting generally do not provide the prior study as input, precluding comparison which is required for clinical accuracy in some types of scans, and offer only unreliable meth- ods of interpretability. Therefore, leveraging an existing visual input format of anatomical tokens, we introduce two novel aspects: (1) lon- gitudinal representation learning- we input the prior scan ing a method to align, concatenate and fuse the current and prior visual information into a joint longitudinal representation which can be provided to the multimodal report genera- tion model; (2) sentence-anatomy dropout training strategy for controllability in which the report generator model is trained to predict only sentences from the original report which coree spond to the subset of anatomical regions given as input. We show through in-depth experi- ments the MIMIC-XXE dataset (Johnson et al., b0;,,,b; Goldberger et al., 2000) how the proposed approach achieves state-of-the results while enabling anatomy-wise control- lable report generation.
There low lung volumes.
opacitoos in lung are rescived.
Prior Scan
The right
The-heod appears mildl entorged:
Figure 1: Illustration of controllable automated porting system using longitudinal representations. The report generator trained to generate only sentences corresponding to the selected input anatomical regions. LI. indicates the left lung and RI. the right lung and we colour- match each region with the corresponding sentence. Strikethoough text represents the section of the report that we do not want the generated report to include when only the LL and RL are selected as inputs.
additional input, propos-
ment, 2022), used to visualise and evaluate the lungs, the heart, and the chest wall. Given a CXR, a radiologist or a trained radiographer will diagnose disease (e.g. lung cancer, scoliosis), and assess the position of treatment devices (e.g. tracheostomy tubes, pacemakers). They record their findings in radiology report, writing detailed text description of the presence/absence and location of relevant clinical findings. When prior image available, the radiologist commonly compares the current clinical findings of a patient with the prior clinical findings to assess their evolution over time (e.g. "the heart remains enlarged"', "the catheter has been removed")) this is especially critical in follow-up exams performed for monitoring.
1 Introduction
A chest X-Ray (CXR) is a frequently performed radiology exam (NHS England and NHS improve-
Radiology reporting is a time-consuming pro-

---

cess, and sean results are often subject to delays. In many countries, this reporting backlog only likely to worsen due to increasing demand for imaging studies as the population ages, and to the shortage of radiologists (Rimmer, 2017; Cao et al.,
training the model to associate input anatom- ical regions with the corresponding output sentences, giving controllability over which anatomical regions are reported on;
3. empirically demonstrate oate-of-the-art per- formance of the proposed method both full and partial report generation via extensive ex- periments on the MIMIC-CXR dataset (John- son et al., 2019a,b; Goldberger et al., 2000).
2023). One strategy to speed up reporting is to integrate automated reporting systems. However, to be employed in real-world clinical scenarios, an automated system must be accurate, controllable and explainable; these criteria are difficult to meet on a task requiring sophisticated clinical reasoning across multiple input image features, and targeting an ill-defined and complex text output.
2
Related Works
2.1 Automated Reporting
Previous works on CXR automated reporting have mostly focused on solutions to improve clin- ical accuracy, e.g. Miura al. (2021). However, they generally use a single radiology study input to generate the full report, precluding comparison with prior scans. They also do not allow the end user control over what parts of the image are re- ported on, leading to limited transparency on which image features prompted specific clinical finding description; interpretabilityy is currently achieved by generating heatmaps that are often dubious (Chen al., 2020, 2021). In this work, hence focus on two novel aspects: (1) longitudinal represen- tations - the most recent previous CXR from the same patient is passed as an additional anatomi- cally aligned input to the model to allow effective comparison of current and prior scans; (2) control- lable reporting- to encourage the language model to describe only the subset of anatomical regions presented input: this might be single anatomi- cal regions (e.g. (cardiac silhouette) -> "the cardiac silhouette is enlarged"),, multiple anatom- ical regions (e.g. (left lung, right lung} + "low lung volumes") or the full set of anatomical regions (in which case the target regresses to the full report, as in previous methods). A high-level
The task of generating a textual description from an image is generally referred to as image caption- ing. Advancements in the general domain have often inspired radiological reporting approaches (Anderson et al., 2018; Cornia et al., 2020; Li et al., 2020; Zhang al., 2021). However, the two tasks differ; the target of image captioning is usually a short description of the main objects appearing in a natural image, whereas the target of radiologi- cal reporting is a detailed text description referring to often subtle features in the medical image. Re- search on CXR automated reporting has focused on improving the clinical accuracy of the gener- ated reports by proposing novel model architectures (Chen et al., 2020, 2021), integrating reinforcement learning to reward factually complete and consis- tent radiology reports (Miura et al., 2021; Qin and Song, 2022), grounding the report generation pro- cess with structured data (Liu et al., 2021; Yang et al., 2022; Dalla Serra et al., 2022), and by replac- ing global image-level features extracted from con- volutional neural networks (He et al., 2016; Huang et al., 2017) as the input visual representations with anatomical tokens (Dalla Serra al., 2023).
2.2 Longitudinal CXR Representation
representation is shown in Figure 1. To summarise, our contributions are to:
The problem of tracking how patient's clinical findings evolve over time in CXRs has received limited attention either generally or for the appli- cation of CXR reporting, despite this being a crit- ical component of a CXR report. Ramesh et al. (2022) avoid the problem by proposing a method to remove comparison references to priors from the ground truth radiology reports to alleviate hal- lucinations about unobserved priors when train- ing a language model for the downstream report generation task. Bannur et al. (2023) introduce a supervupere multimodal approach that mod- els longitudinal CXRs from image-level features
1. propose a novel method create a longitudi- nal representation by aligning and concatenat- ing representations for equivalent anatomical regions in prior and current CXRs and project- ing them into a joint representation:
2. propose novel training strategy, sentence- anatomy dropout, in which the model is trained to predict a partial report based on a sampled subset of anatomical regions, thus

---

Visual Anatomical Token Extraction
Report Generator
(End-lo-EEd Training with Sentence-Ansfory Dropout)
Prior Scan
Tokens
Longitudinal Projection Module
Absames
Led Lung
Concatemation
Report Current
Long
Longiuudinl Scane
Aediensi
Store
Right piiuua dials alt,
Current Scan
Faster R-CNN
Anst. Tokera
Gurment
Latiaar
Latg
MLP
Lung
Language Model
The right preumathary has: metrllved Mbart sozo nomool Led
Piper Pipering
Fight
Anaenes
red
ling clar
Piperingg
Spite
Sphre
Indication Fieed
Post drain check postioning
Carrent
Figure 2: Architecture overview. The anatomical region representations the current and prior CXRs are extracted from Faster R-CNN (visual anatomical token extraction). These are aligned, concatenated and projected into a joint representation (Iongitudinal projection module), then passed alongside the tokenised indication field as input to the language model to generate the report for the current scan. The Report Generator is trained end-to-end using centence-anatomy dropout.
as a joint temporal representation to better align text and image. Karwande et al. (2022) have pro- posed an anatomy-awar approach to classifying if a finding has improved or worsened by modelling longitudinal representations between CXRs with graph attention networks (Veličković al., 2018). Similar Karwande et al. (2022), we project longi- tudinal studies into a joint representation based on anatomical representations rather than image-level features. However, we extract the anatomical rep- resentations from Faster R-CNN (Ren et al., 2015), as in Dalla Serra et al. (2023).
a sentence "No evidence of emphysema." requires information from both left and right lungs; this requires us to identify valid subsets of anatomical regions in each CXR report for our dropout training strategy (Section 3.4).
Method
We first extract anatomical visual feature represen- tations 전 Id with d dimensions for N predefined anatomical regions A {a,, J. appearing in a CXR. To model longitudinal relations, perform feature extraction for both the scan under consider- ation and for the most recent prior scan, and then combine region- wiis using our proposed longitudi- nal projection module. We then input the features to the language model (LM) alongside the text in- dication field, which is trained using our anatomy- sentence dropout strategy. We show the proposed architecture in Figure and describe these steps in more detail below.
2.3 Controllable Automated Reporting
We define a controllable automated reporting sys- tem as one which allows the end users to seleet what regions the image they want to report on, giving a level of interpretability. This has par- tially been tackled using hierarchical approaches (Liu et al., 2019), by introducing multi-head transformer with each head assigned specific anatomical region and generating sentences exclu- sively for that region (Wang et al., 2022), and con- temporaneously to our work by Tanida et al. (2023) who (similarly to us) generate sentences based on region-level features extracted through Faster R-
3.1 Visual Anatomical Token Extraction
For the anatomical representations, we extract the bounding box feature vectors from the Region of Interest (Rol) pooling layer of a trained Faster R-CNN model. Faster R-CNN is trained on the tasks of anatomical region localisation in which the bounding box coordinates of the N=36 anatom- ical regions (e.g. abdomen, aortic arch, cardiac si houette) are detected in each CXR image, and finding detection in which presence/absence is predicted each proposed bounding box region
CNN (Ren et al., 2015).
Tanida et al. (2023) make the assumption that each sentence in the report describes at most one anatomical region. Conversely, we acknowledge that there may be multiple anatomical regions which are relevant to the target text output, e.g.,

---

Original Report
Anatomical region
Target output
Mapping type
"The mediastinum is mildly enlarged."
"The mediastinum is mildly enlarged. Blunting of right costophrenic angle noted. No suspicious nodules seen. No pneumothorax or infective consolidation. Bilateral atelectasis, likely post -operative. Degenerative changes seen in both shoulders. NG tube tip positioned correctly in stomach. No free air under diaphragm"
mediastinum
one-to-one
"NG tube tip positioned correctly in stomach. No free air under diaphragm." "Degenerative changes seen in both shoulders." "Blunting of right costophrenic angle noted.
abdomen
one-to-many
left clavicle,
many-to-one
right clavicle
No suspicious nodules seen.
right lung, left lung
No pneumothorax or infective consolidation. Bilateral atelectasis, likely post-operative."
many-to-many
Table 1: Example correspondences between anatomical regions and target output, for synthesised CXR "Original Report" (Findings section). Note the different types of correspondence mapping.
for set of 71 predefined findings (e.g. pleural effusion, lung cancer, scoliosis). Specifi- cally, we augment the standard Faster R-CNN ar- chitecture head comprising an anatomy classi- fication head and bounding box regression head with a multi-labed classification head, following Dalla Serra et al. (2023), to extract finding-aven aware anatomical tokens = { with on E Rd where d=1024. Then, for each anatomical region we select the bounding box representation pro- posal with the highest confidence score. When the anatomical region is not detected, we assign d- dimensional vector of zeros. For more details about the model architecture, the loss term and other im- plementation details, we refer to Dalla Serra et al. (2023).
A Atargtt S A, and we obtain the longitudinal repre- sentation by concatenating the anatomical tokens for each anatomical region an of the CXRs, and passing them through the longitudinal projec- tion module f:
f(TVe. Fp,)) if an E Atarget ((Ō, otherwise.
Ujoint,n
The projection layer f is Multi-Layer Perceptron (MLP) consisting of a stack of a Fully-Connected layer (FC1), a Batch Normalization layer (BN) and another Fully-Connected layer (FC2). We refer to the resulting output the current-prior joint representation Vjoint [ #=1
3.3 Language Model (LM)
3.2 Longitudinal Projection Module
This consists of multimodal Transformer encoder- decoder, which takes the current-pre current-prior joint repre- sentation Vjoint and the indication field the visual and textual input respectively, to generate the output report Y:
Taking the current scan (the most recent scan at specific time point) and the CXR from the most re- cent study (prior scan)?, we extract from Faster R- CNN the anatomical tokens both CXRs. There is one token for each of the anatomical regions: Veurrent = (een J. and Vprior = [Tp.n ) When the current scan is part of an initial exam: p. == e pd Vn I.....N. N. We select indices for the subset of regions that want to report
= LM(V joint: T)
where Y corresponds to the partial report if Atarget C A or the full report if Atarget = A. Similarly to Devlin et al. (2019), the input to the LM corresponds to the sum of the textual and visual
provided Appendix A. anatomical regions and clinical findings are
'Full
'A prior scan is only available if the current scan is not part of an initial exam.
"The indication field contains relevant medical history in the form of free text and is available at imaging time.

---

token embeddings, the positional embeddings (for position of tokens) and the segment embeddings (for modality type: vision or text).
Goldberger al., 2000) and Chest ImaGenome (Wu et al., 2021; Goldberger et al., 2000). The MIMIC-XR dataset comprises CXR image-report pairs. The Chest ImaGenome dataset includes ad- ditional annotations based on the MIMIC-CXR images and reports. In this paper, we train Faster R- CNN with the automatically extracted anatomical bounding box annotations from Chest ImaGenome, provided for 242,072 AnteroPosterior (AP) and PosteroAnterrir (PA) CXR images. Chest Im- aGenome also contains centence-anatomyy pairs an- notations that we use to perform aamtenc dropout. The longitudinal scans of each patient are obtained by ordering different studies based on the annotated timestamp and for each study, taking the most recent previous study as the prior. For this pur- pose, only select AP or PA scans priors (i.e. ignore lateral views). If multiple scans are present in a study, we consider the one with the highest number of non-zero anatomical tokens. In case of a tie, we select it randomly. In all experiments, we follow the train/validation/eess split proposed in the Chest ImaGenome dataset.
3.4 Training with Sentence-Anatomy Dropout
During training, for each instance in each batch, we randomly drop a subset anatomical tokens from the input and omit the corresponding sen- tences from the target radiology report; we term this training strategy sentence-anatomy dropout. In practice, for each training sample not all combi- nations of anatomical regions will be fit for dropout, since they must satisfy the following conditions:
1. Given subset of anatomical tokens as input, the target output must be the full subset of sentences in the report that describe the corre- sponding anatomical regions;
2. Given subset of sentences as the target out- put, anatomical tokens must be input for the full subset of described anatomical regions.
The above conditions are necessary since reject the assumption that each sentence in the report describes only one anatomical region, as made by Tanida et al. (2023). We illustrate with examples of the different mappings in Table 1.
4.2
Data pre-processing
We extract the Findings section of each report as the target text'. For the text input, we extract the Indication field from each report
Let us consider radiology report as set of L sentences S = (= each one describing the findings appearing in different subset of anatomi- cal regions A C A; and P {(sL, A the set of sentence-anatomy pairs of report. To satisfy the two conditions above, we seek to discover the connected components in a graph where sentences are the nodes and an edge between two nodes repre- sents an overlap of described anatomical regions be- tween the two sentences. We describe in Appendix B the algorithm to identify the connected compo- nents for each CXR report and how we group the corresponding sentence-anatomy pairs each con- nected component into Pk: S P. We then define as == k= the set of valid sentence-anatomy subsets (see Appendix C for an example). During training, we randomly select one or more elements of F and then use the anatomical tokens as input and concatenate the corresponding sentences to create the target output.
When training Faster R-CNN, CXRs are resized by matching the shorter dimension to 512 pixels (maintaining the original aspect ratio) and then cropped to resolution of 512 512.
4.3 Metrics
We assess the quality of our model's predicted re- ports by computing three Natural Language Gener- ation (NLG) metrics: BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and METEOR (Banerjee and Lavie, 2005). To better measure the clinical cor- rectness of the generated reports, we also compute Clinical Efficiency (CE) metrics (Smit et al., 2020). derived by applying the CheXbert labeller to the ground truth and generated reports to extract 14 findings - and hence computing F1, precision and recall scores. In line with previous studies (Miura et al., 2021), this is computed by condensing the four classes extracted from CheXbert (posi tive, negative, uncertain, or no mention) into binary
4 Experiments
4.1 Datasets
http://hhhttececmmcmt///////ttt/ files.py
We consider two open-source CXR imaging datasets: MIMIC-CXR (Johnson et al., b019a,b;
'ht http:///ggtuuuu.ccccjacneeeeeee ttolss/mimiccccxrppppprcssss

---

classes of positive (positive, uncertain) versus negative (negative, no mention).
remaining metrics, whilst providing also better con- trollability (and interpretability). Whilst both our method and that proposed by Tanida et al. (2023) tackle the controllability aspect, we show superior results in all metrics.
4.4 Implementation
We adopt the torchvision Faster R-CNN imple- mentation, as proposed in Li et al. (2021). This consists of a ResNet-50 (He et al., 2016) and a Feature Pyramid Network (Lin et al., 2017) as the image encoder. We modify it and select the hyper- parameters following Dalla Serra et al. (2023).
5.2 Ablation Study
We investigate the effect of incorporating prior CXR scans as input (Priors) and adopting sentence- anatomy dropout during training (SA drop).
The two FC layers in the MLP projection layer have input and output feature dimensions equal to 2048 and include the bias term.
Full Reports We evaluate the different configura- tions of our method using the full set of anatomical regions as input (Atarget = A) and the full report as the target text. Table shows the results; can be seen that both adding priors and using sentence- anatomy dropout during training boost most met- ries, with best overall performance obtained when combining the two mechanisms. This is illustrated qualitatively in Figure 3.
The encoder and the decoder of the Report Gen- erator consist of 3 attention layers, each composed of 8 heads and 512 hidden units.
The MLP and the Report Generator are trained end-to-end for 100 epochs using cross-entropy loss with Adam optimiser (Kingma and Ba, 2014) and the sentence-ane sentence-anatomyy dropout training strategy. We set the initial learning rate to 5 x 10 and reduce it every 10 epochs by a factor of 10. The best model is selected based on the highest F1-CE score.
Initial Follow-up Scans We study the effect of the different components by dividing the test set into initial scans versus follow-up scans, resulting in 11,951 and 20,735 CXR report pairs respectively. The results are shown in Table 4. We note the improvement of our method over the baseline on both subsets, with the best results obtained when adding the priors alone or in combination with the sentence-anatomy dropout. It is worth noting that the benefit of including priors is also present for initial studies with no prior scans. We hypothe- sise that since the model can infer which are initial scans (using the fact that the prior anatomical to- kens are all zero-vectors), it will correctly generate more comprehensive report rather than focussing on progression change of known findings.
We repeat each experiment 3 times using dif- ferent random seeds, reporting the average in our results.
4.5 Baselines
We compare our method with previous SOTA works in CXR automated reporting, de- scribed in Section 2.1: R2Gen (Chen et al., 2020), R2GenCMN (Chen et al., 2021), M Transformer+facttrrrvi. (Miura et al., 2021), T (Dalla Serra et al., 2023) and RGRG (Tanida et al., 2023). For all baselines, we keep the hyperparameters as the originally reported values. For a fair comparison, we use the same text and image pre-processing as proposed in this work and we re-train the baselines based on the Chest ImaGenome dataset splits.
Partial Reports To measure the controllability of our method, we evaluate the ability of the differ- ent configurations generate partial reports given subset of anatomical regions. For this purpose, we divide each report in the test set into its set of valid sentence- anatomy subsets F (Algorithm I). We take the anatomical regions contained in each subset X C F as input and the corresponding sen- tences as the target output. We then obtain total of 71 11,698 partial reports. The results in Table 5 show how adopting sentence-anatomy dropout en- ables a controllable method which correctly reports only on the anatomical regions presented as input and does not hallucinate the missing anatomical regions; this is further illustrated in Figure 4.
5 Results
5.1 Automated Reporting
Table 2 shows the effectiveness of the proposed method over the baselines, showing superior perfor- mance for most NLG and CE metrics. Compared to Dalla Serra et al. (2023), our method can achieve similar BLEU metrics and superior scores on the
"Wo did not re-train Dalla Serra et al. (2023) and Tanida ct al. (2023), as they already use that same datasct split.

---

NLG Metries
CE Metries
Method
BI-1
BL.2
BL.3
BL.4
MTR
RG-L
P
R
R2Gien (Chen et al., 2020)
0.381
0.248
0.174
0.130
0.152
0.314
0,431
0.511
0.395
R2GenCMN (Chen et al., 2)21)
0.365
0.239
0.169
0.126
0.145
0.309
0.371
0.462
0.311
M² Tr. factgNTSI (Miura et al., 2021)
0.402
0.261
0.183
0.136
0.158
0,300
0.458
0.540
0.404
1lok
(Dalla Serru et al., 2023)
0.490
0.363
0.288
0.237
0.213
0.406
0.537
0.585
0.496
RGRG (Tanida et ul., 2023)
0.400
0,266
0.187
0.135
0,168
0,461
0.475
0.447
Ours
0.486
0.366
0.295
0.246
0.216
0.423
0.553
0.597
0.516
Table 2: Comparison of our proposed approach with previous methods. We show the best results bold.
Configuration
NLG
Metries
CE Metrics
Priors
SA drop
BL-1
B1-22
BL-3
BL-4
MTR
RG-L
F1
R
0.430
0.327
0.266
0.224
0.202
0.420
0.534
0.593
0.485
0.456
0.347
0.283
0.239
0.210
0.428
0.548
0.577
0.522
0.473
0.358
0.289
0.243
0.213
0.426
0.550
0.597
0.510
0.486
0.366
0.295
0.246
0.216
0.423
0.553
0.597
0.516
Table 3: Ablation study on incorporating prior CXR scans as input and adopting sentence-anatomy dropout during training. We report the NLG and CE results on the MIMIC-CXL test set.
Configuration
NLG Metries
CE Metries
uation is itself limited since the CheXbert labeller reportedly has accuracy of only 0.798 FI (Smit et al., 2020) and thus we do not expect measure perfect scores on our clinical efficiency metrics.
Priors
SA drup
B1.44
MTR
RG-L
F1
P
R
Initial Scans
0.283
0.234
0.479
0.532
0.370
0.499
0,303
0.244
0.490
0.543
0.563
0.524
0.303
0.244
0.485
0.541
0.582
0.306
0.245
0.479
0.542
0.589
0.502
Our method focuses only on CXR and adapting to other types of medical scans might be challeng- ing, First, due to the 2D nature of CXRs compared 0,521
Follow-up
Scans
0.194
0.187
0.377
0.533
0.600
0.479
0.206
0.194
0,383
0.550
0.583
0.210
0.197
0.378
0.552
0,602
0,510
to other types of 3D scans (e.g., CT, MRI). Sec- ond, we strongly rely on the Chest ImaGenome dataset and its annotations. These automati- cally extracted and similar sentence-anatomy notations could be extracted for radiology reports from other types of scans. However, as the same au- thors pointed out, there are some known limitations of their NLP and the region extraction pipelines; for instance, clinical findings may not be properly extracted from a follow-up report which may be as simple as "No change is seen".. Hence, some finement of the pipelines with without additional manual input might be required.
0.216
0.202
0.382
0.557
0.599
0.520
Table 4: NLG and CE results the Initial and the Follow-up subsets of the MIMIC-CXR test set.
Configuration
NLG Metries
CE Metries
Priors SA drop
BL.4
MTR
RG-1.
F1
P
0.113
0.187
0.291
0.549
0.519
0.583
0.127
0.182
0.301
0.587
0.604
0.571
0.225
0.226
0.467
0.667
0.651
0.223
0.225
0.462
0.672
0.663
0.680
Table 5: NLG and CE results computed on partial reports of the MIMIC-CXR test set, by dividing each report into its set of valid sentence-anatomy subsets.
Conclusion
To further assess the quality of our method and each of its components, we measure the length dis- tribution of the predicted reports, similar to Chen et al. (2020), showing that our method more closely matches the ground truth distribution than baseline methods; see results in Appendix D.
This work focussed on two key aspects of CXR automated reporting: controllability and longitudi- nal CXR representation. We proposed a simple yet effective solution to align, concatenate and fuse the anatomical representations of two subsequent CXR scans into joint representation used as the visual input to language model for automated reporting. We then proposed novel training strategy termed sentence- aanitoyyyaammyyy dropout, supervise the model to link each anatomical region to the corresponding output sentences. This gives the user more control and easier interpretability of the model predictions.
6 Limitations
While the proposed method shows state-of-the-ard results on CXR automated reporting, end-to-end re- port generation from CXR images requires further research to reach the clinical accuracy needed to be useful as a diagnostic tool. We note that our eval-

---

Ground Truth
Baseline
Ours
Comments
There parsising small modderad balalann plaura affusions with subjacent alälectasi as well as puimonary vascular engorgement and interalltel septai th thickening The infrshilr opartlyy unchenged. entergement the condiar athoustee unchanged Dilofiaation of sortc arch prosont.
Moderato cardamegaly
Moderate lange riht amall
ropart predicted our mathod ahowes more acturale discripttion of findlops and nigher rocal compored to mo baselire.
alsble Modaniis biliinal pleundl aftusknns ara unchanged. Thera mild
[kunni affaaira arw unchanged. There is mils pulmonary vascuaar congestion and interstitaal eceme. There a moderate
pulmonary odeme. There is no focal consolidation preumothoras.
pleural affusson with sesodated right Millactoon tandomerglly is stable The aottic krob is Billifid. No priumotharae. No acule osceous abnormalttes.
companison the prior chest intlogmnci aill limes luber nemoved. bilateral king improved dramatically Thore left pleural eftusion. There is cutde right bastlar opacity. Heart azee normal The medtastinal hilrr contours normal. pulmoner
Compared to the prior study there no signriiam intervel charge.
Intenval removal of endotracheal tubel The
The report predicted our method cornccly desmmbes the renooval the difarrrn
The feeding tube been jogular cathiter has hilar
has
ramovad The cardoomddazzing contours are within normal limiss The aoria tortuous. There is small left stelectasiss eftuston preumethorss.
pnaumothonax seors
Stable ohron cally elovated right hemidaaprramm with moderate right lower lobs stalectooiss Small itable browe colatication projects adjacent the let: pnoumothorax ploural offusion. Left lung clear. Heart size mediastinal contour and hiss appear normal. No bonry abnormallly
focal consolidation picural eftuston preumathores or pulmonary edema is seen. Heart sioe normel Mechariinll contours normal
Elevation of right hom diaphragm unchanged from prior. The lungs are dear
Our proposed mathod shows Improvement compared to the caseline éut there is stl messing vnformaiiio sunh companson with previous CT
wilhout locil corsokiistion No ploural
preumothorax
Heart
mediseiion
normal limits tartuous aortic contour.
There is elevation of the right hemdappragg.
Figure 3: Qualitative results of full reports generation. We compare the reports generated by the baseline (without adding prior scans and eentecc-annatomy training) and proposed solution with the ground truth. We highlight using different colours the segments of the reports that commented on in the right column.
Anat. Regions
Ground Truth
Baseline
Ours
spino
Moderato antorior osteophytos along the lower thorasio spine aspear unchanged.
The heart is at the upper limits of normal size. The aortic arch Is calcified. The mediastinal and hilar contours appear unchanged. There pleural effusion pnoumothorax. The lungs appear clear.
Mild degenerative changes are sm along the thoracis
sihouelle
Moderate severe enfargement cardiac sthoustte unchanged.
Modersiee entargement re-demonseraaid Mudiastinal and hilar contours are unchanged. Pulmonaary vasculature is not engorged. No focal consolidation plaural effusion pnsumothoraz
silccuele
Moderate bardiomagaly ia mi meddempreddiledd
abdomen. cardiac sihouotte, med mediastiuum, uppar merdisslimmm
The nasogastricc tube been removed. The heart and mediastinmm within normal limits
heart normal. hilar and mediastnnll contours normal There has been interval resclution
The nasogastrc tube has been removed. cardomedassinn sthouetle urnamackable..
previously small lefbeided pneumathoras. There has been alight Interval Improvement the previousty seen small right pleural effusion. There is no evidence of focal consollisatoos conceriing for preumonia.
aroas stroaky stolectasis the bilntoral lung bases. There are parsistant prominent intorsatiaa markings which suggest mid
Lung valumes low. Uneer opocitios the bitateral lower lungs are consistent subsegmental atolectosis. local consolidation pleural ettus pnoumothorax The unchanged.
There pulmonary vasoulor congastion and mild interditaal puimonary odema. Linear bibosita for opacti concistont with abolectasis. There is no pleural effusion or preumotharse.
lnft lover lung zone loft costophronic angle, structures, right una.
chronic interstiiall atnormality intersttial edema. lungs remain hyperitatted. There is no pleural effusion or proumationess No looal comsolibaiion is
most
rightt lower lung zone. right costopheeric angle. hiter structures
Figure 4: Qualitative results of partial reports generation. From left to right: the subset of anatomical regions Atarget we want to report, the ground truth partial reports, the reports generated by the haseline (without adding prior scans and sentence-anatomy training) based on Atarget and those generated by our proposed method. We indicate in red the hallucination on the missing anatomical regions.
We showed the effectiveness of the proposed so- lution on the MIMIC-CXR dataset where it gives state-of-the-ar art results. Moreover, we evaluated through extensive ablations how the different com- ponents help to generate better reports in different setups: full report generation, partial report gener- ation, and Initial vs. Follow-up report generation. In future, this method could be integrated with more advanced language models such as Touvron
et al. (2023) or OpenAl (2023), or alternative tech- niques such as Dalla Serra et al. (2023). Further, when considering the patient history, the prior CXR scan might usefully be augmented with other types of imaging and associated radiology reports, clin- ical notes, clinical letters, and lab results. In fu- ture work, will look to extend our method to broader multimodal approach considering more data inputs.

---

References
physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101 101(2))e215--222)
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recogni- tion. In CVPR, pages 770-778.
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In ACL. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. 2017. Densely connected con- volutional networks. In Proceedings of the IEEE conference computer vision and pattern recogni- tion, pages 1700-4708.
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fer- nando Perez-Garcia, Maximilian llse, Daniel C Cas- tro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, et al. 2023. Learn- ing to exploit temporal structure for biomedi- cal vision-language processing. arXiv preprint arXiv:2301.06 045588...
Alistair EW Jobnson, Tom J Pollard, Seth J Berkowitz. Nathaniel Greenbaum, Matthew Lungren, Chih ying Deng, Roger G Mark, and Steven Horng. 2019a. MIMIC-CRR, de-identified publicly avail- able database chest radiographs with free-text re- ports. Scientific Data, 6(1).
Alistair EW Johnson, Tom J Pollard, Nathaniel Green- baum, Matthew P Lungren, Chih-ying Deng, Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 201 MIMIC-CXR-JCC, a large publicly available database of labeled chest radio- graphs. arXiv preprint 1w0.v:9901...0 190.02.422.
Daniel J Cao. Casey Hurrell, and Michael N Patlas. 2023. Current status of burnout in canadian radiol- ogy. Canadian Association of Radiologists Journal, 7431)337-43.
Zhihong Chen, Yaling Shen, Yan Song. and Xiang Wan. 2021. Cross-modal memory networks for radiol- ogy report generation. In ACL-JUCNLP, pages 5904- 5914.
Gaurang Karwande, Amarachi B Mbakwe, Joy T Wu, Leo A Celi, Mehdi Moradi, and Ismini Lourent- zou. 2022. CheXReINet: An anatomy-awan model for tracking longitudinal relationships between chest X-rays. In Medicul Image Computing and Com- puter Assisted Intervention-MICCA. 2022: 25th In- ternational Conference, Singapore, September 18-22, 2022, Proceedings, Part I. pages 581-591. Springer.
Zhihong Chen, Yan Song. Tsung-Hui Chang, and Xi- ang Wan. 2020. Generating radiology reports via memory-driven transforme.. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439-1449.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 141::641..6900
Marcella Comia,, Mattco Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory trans- former for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pat- lern recognition, pages 10578-10587
Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semantics aligned pre-training for vision-language tasks. In ECCV, pages 121-137. Springer.
Francesco Dalla Serra, William Clackett, Hamish MacK- innon, Chaoyang Wang. Fani Deligianni, Jeff Dalton, and Alison Q Neil 2022. Multimodal generation of radiology reports using Enowledge-groundes ex- traction of entities and relations. In AACL-IJCNLP. pages 615-624.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. 2021. Benchmark- ing detection transfer learning with vision transform- ers. arXiv preprint 1...:22111.14444
Francesco Dalla Serra, Chaoyang Wang, Fani Deli- gianni, Jeffrey Dalton, and Alison Q O'Neil. 2023. Finding-awaad aware anatomical tokens for chest X-ray au- tomated reporting. arXiv preprint arXiv:2308. 15961.
Chin-Yew Lin. 2004. ROUGE: A package for auto- matic evaluation of summaries. In Text summariza- tion branches out, pages 74-81.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie, 2017. Feature pyramid networks for object detection. In CVPR, pages 2117-2125.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In NAACL, pages 4171-4186. Association for Computational Linguistics.
Fenglin Liu, Xian Wu. Shen Ge, Wei Fan, and Yuexian Zou. 2021. Exploring and distilling posterior and prior knowledge for radiology report generation. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pages 13753- 13762.
Ary I. Goldherger, Luis AN Amaral, Leon Glass, Jef- frey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George Moody, Chung-Kang Peng, and H Eugene Stanley. 2000. Physiobank,

---

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDer- mott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest X-ray report generation. In Machine Learning for Healthcare Conference, pages 249-269. PMLR.
Petar Veličković. Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph attention networks. In Proceedings of the International Conference on Learning Represen- tations.
Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Lan- glotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency of image-to-text radiol- ogy report generation. In NAACI, pages 5288-5304.
Lin Wang, Munan Ning, Donghuan Lu, Dong Wei, Yefeng Zheng, and Jie Chen. 2022. An inclusive task- aware framework for radiology report generation. Medical Image Computing and Computer Assisted Intervention -MICoA 2022: 25th International Con- ference, Singapore, September 18-22, 2022, Proceed- ings, Purt VIII, pages 568-577. Springer.
NHS England and NHS improvement. 2022. Diagnostic imaging dataset statistical release. pages 1-17.
OpenAl. 2023. GPT-4 technical report. ArXiv, abs/2303.08774.
Joy T Wu, Nkechinyere Nneka Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexander Paguio, Jasper Seth Yao, Edward Christopher Dee, William G Mitchell, Satyananda Kashyap, Andrea Giovannini, et al. 2021. Chest ImaGenome dataset for clinical reasoning. In NeurlPS: Datasets and Benchmarks Track (Round 2).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: method for automatic evaluation machine translation. ACL, pages 3-1--18.
Shuxin Yang, Xian Wu, Shen Ge, S Kevin Zhou, and Li Xiao. 2022. Knowledge matters: Chest radiology report generation with general and specific knowl- edge. Medical Image Analysis, page 102510.
Han Qin and Yan Song. 2022. Reinforced cross-modal alignment for radiology report generation. In ACL, pages 448-458. Association for Computational Lin- guistics.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jian- feng Gao. 2021. VinVL: Revisiting visual represen- tations in vision-language models. In CVPR, pages 5579-5588.
Vignav Ramesh, Nathan A Chi, and Pranav Rajpurkar. 2022. Improving radiology report generation systems by removing hallucinated references to non-existent priors. In Machine Learning for Health, pages 456- 473. PMLR.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28.
Abi Rimmer. 2017. Radiologist shortage leaves patient care at risk, warns royal college. BMJ: British Medi- cal Journal (Online), 359.
Akshay Smit, Saahil Jain, Pranav Rajpurkar, Anuj Pa- reek, Andrew Ng, and Matthew Lungren. 2020. Com- bining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 1500-1519, Online. Association for Computa- tional Linguistics.
Tim Tamida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2023. Interactive and explainable region- guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1433-7442
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie Aune Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open efficient foundation language models. arXiv preprint arXiv:2302. 13971.

---

A Anatomical Regions& & Findings
Below we show the lists 36 anatomical regions and 71 clinical findings used in this paper.
Anatomical Regions
abdomen
left clavicle
medisstinum
right lower lung 2002
aortic arch
left costophrenic ungle
right upical zone
right lung
cardiac silhouette
left hemidiaphragm
right atrimm
right lung zone
hilar structures
right silhouette
right upper
cavoutrial junetion
left lower
right cardiophranic angle
right upper
descending aceta
Jeft lung
right clavicle
spine
left apical zone
left mid lung zone
right costophrenic angle
svc
left cardiac silbouettee
lett upper abdommnn
right bemidiaplaaag
uacheca
left cardiophrenic ungle
left upper lung zome
right hilar structures
upper mediastinum
Findings
sirspace opacity
eyst/bllac
Innear/patchh stelectasis
pneunothorax
alvenlar hemomhage
disphrugmatic eventration (henign)
lobunkegmenntal collapse
prosthetic valve
nottic grafthrppir
elevated hemidiaphragm
low lung volumes
pulmosary edoma/hazy opacity
artilact
endotracheal tube
lung cancer
rolated
aspirution
enlanged cardiae silhouetre
lung lesion
steloctasis
calarged hilum
lung opacity
shoulde ostecorthiiii
home lesion
enteric tube
mas/nodule otherwise specilied)
degenerative changes
breast/nipple shadows
fluid overload/beur failure
mediastinal displacement
spinal fracture
bronchiectasis
golter
mediastinal drain
sub diaphragmatic air
cabg grafts
gramelomatous discase
mediastinal widening
subelavian line
calcified nodule
berna
multiple masses/aeules
superior modiastinal mass/cllagemeet
pacer
hydropneumothorax
pericardial effusion
swan ganz
chest port
hyperaeration
piec
chest tube
ij line
pigtail calbeter
urchecstomy tube
clavicle fracture
increased reticular markingsiid pattern infiltation
pleural effusion
vascular calcification
consolidation
plouralpareeehymma scarring
vascular congestion
copdlemphysemaa
interstitial lung discase
pacumomediastinum
vascular redistribution
costophrenic angle blunting
intra-aortic halloon pump
pacumoonia
Table 6: Complete set of 36 anatomical regions and 71 findings used to supervise the anatomy localisation and the finding detection tasks, annotated the Chest ImaGenome dataset (https: (https://physiooet. org//ontent/ chest-iman iett-immagnoom//11 0..0010.0..0 0.0/).

---

B Algorithm for discovering valid sentence-anatomy subsets
Algorithm Find set of valid sentence-anatomyy subsets.
Input: set of (sentence, regions) pairs from single CXR report, P Output: set of valid sentence-analomy subsets, JF
1: function FINDVALIDSUBSETS())
2:
F empty set
3:
Pi + set populated with the first (sentence, regions) pair in P
4:
Pr Premaining f- set of (sentence, regions) pairs in P not assigned to P.
5:
R(P:) regions in P:
6:
((Prrmmiiig))) - regions in remainnng
7:
while Py Premainiig + 0 do
8:
while R(P.) n R remainiin)) + ( do for (sentence, regions) E Pomainnng do regions n R(P,) + {} then
9:
10:
11:
P: + include (sentence, reweeeeeegooon regions)
12:
end if end for Premaining set of (sentence, regions) pairs in P not assigned to P3 nor any Pk E F R(P:) regions in R(Premaane remaining) f regions in Premaining
13:
14:
end while
18:
F + include P;
19:
P. < set populated with the first (sentence, regions) pair in Premaining
20:
end while
21: return J.
22: end function

---

C
Example of Report Sentence-Anatony Pairs
Report
The kampe fygedhtlioed with Batleing of the dephaigns augpestiv undeddyng COPD. The hear is miley unterped The wort mnducus and alfowsly colcited Mediasiinl and Nur contoure othenwise are urronaflaele Palroorayy vasculadty not ongorged No focal consoltation aitural eftision or mitinal steasy stelociatic changes
pncsumothoaa
ossecus abromaliies are present digorrective
Chest ImaGenome sentence anatoomes pairs:
P = (111, A
Senrianee
Anatomical Regiona
The unge hyporinfated with fattening the diphrogms sugpestive underrinngg COPD:
Frighs ling,, Tort ling Hight hemisaphragni Teft hemidiashragm)
Purmenary vazcularity engerged"
Fright king", hight structurss, lont lung", hiaar structures]
No fooil conselowton, pleunal eftuxin preumothonee stentfiie'
(right king", contophmenic angli, 'witt king', lilt costopreeiic angle)
There minimal bloaslar atslestatic
Tright
ling', 1oft
Medicstinll hilar caidlied!
unvemarkable"
structuros)
'madiastinnn. mediaddiunt
'aortic
milty entarged!
athoustie)
ossecus abrormallioo are present.
Fright clavice clivicee
Mild muli degenaattive changea thonecic spine.
[xpine]
centence-anatoryy subsets:
F= P
F
Regions
lunge typer lastninng daphragme suggeetive underfynn
Enight king. lung. light hemidiaphmmmi hymodag
Duumoonry sonculivity
[righe king hive structorne left lung!. sefl hine structores]
NNo local consolidation pleural offusion or proumothon orax ideenffed..
Fright lung, trigt costuphronie angle, lift ling;. costophrenic angle]] [right king. hight
changges)
kingi.
"Mediastinal and nilar comtous otherwice are inremaricalle..
[right Niar structures, left hilr structures; "medlastiunn. luppe mediastrum)
The zorta fortuous and ditusely calc
[mediastinnt, 'sortic anch)
wrane ceseruus sénormodiioe are present!
fright clevenw Twrl clavicee, (epinw]]
"Mil muitleowl degeneent
thonacic
[spine]
The heart is mildly enteredd.
[secdiae sihoustio?
Figure 5: Example of sentence-amatomy annotations of report and its set of valid sentence-anatomy subsets.

---

D Report Length
The length of a report corresponds to the number of words contained. We compute this for the full reports generated from the full set of anatomical regions and for the partial reports derived from the set of valid sentence-anatomy subsets. In Figure 6 (left) we see that the distribution of the proposed method is closer to the distribution of the GT reports. In Figure 6 (right), we note that adopting the sentence-region dropout strategy allows the method to generate partial reports with length distribution closer to the GT partial reports. These results provide further evidence of the improvement of the proposed method over the baseline (without adding prior scans and sentence-anatomy training).
Reports
Reports
Models
Models
$1003
SA Drooout
SA Dropout
Add Wiers Cropout
Add Prizes Dropoit
20000
of
15003
5000
20
40
60
100
120
20
60
130
120
Longth
Report Length
Figure 6: Length distribution of the predicted reports compared to the GT reports. The length of report corresponds to the number of words.


Reviews:
[{"review_id": "hH5Z3CyxPV", "reviewer": "Reviewer_kJZG", "summary": "", "questions": "Is there a way to combine this approach with an image-based pretraining strategy beyond multi-label prediction to address the long tail of rare diseases?\n\nDid the authors consider trying an LLM instead of the few transformer layers that were used?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive feedback. We are encouraged they found the paper interesting and the results compelling. Please find our answers to the reviewer’s questions below.\n\n**Is there a way to combine this approach with an image-based pretraining strategy beyond multi-label prediction to address the long tail of rare diseases?**\n- Apart from the use of Faster R-CNN we have only explored using a ResNet-101 (pretrained on ImageNet or the 71 clinical findings) integrated with the Transformer encoder-decoder, similar to what most of the existing automated reporting methods have adopted. However, we are aware of the challenge posed by addressing the long tail of rarer diseases and we thank the reviewer for the insightful suggestion of using a vision-language pretrained image encoder.\n\n**Did the authors consider trying an LLM instead of the few transformer layers that were used?**\n- As we are limited on computational resources (i.e. all our models were trained on a single Nvidia RTX A5000 GPU), we did not try using an LLM or increasing the number of parameters of the Transformer. Also, in the first place, we wanted to keep the setup simple and focused on our proposed methodology. We agree with the reviewer that trying larger LLMs (e.g. GPT, LLaMa) is a good idea and (as we state in our Conclusion) we plan to do this investigation as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper presents a straightforward method to make use of prior images for radiology report generation. The authors present compelling results on public datasets, outperforming other recent baselines. ", "reasons_to_reject": "As the image encoder was trained for multi-label prediction, it may not represent the long tail of rare diseases as well. This is in contrast to a visual-language pretrained image encoder which may encode more rare disease information. ", "questions_to_authors": "Is there a way to combine this approach with an image-based pretraining strategy beyond multi-label prediction to address the long tail of rare diseases?\n\nDid the authors consider trying an LLM instead of the few transformer layers that were used?", "paper_topic_and_main_contributions": "This paper proposes a method to incorporate a previous image with a current image to do radiology report generation. A significant challenges is aligning the previous and current images so that comparisons can be made and this paper proposes using an R-CNN, trained for anatomy localization, to do this. The paper then inputs image tokens and the indication for the current scan into a few transformer layers for predicting the radiology report. ", "excitement": 5, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "aSrBmgrt0A", "reviewer": "Reviewer_pwef", "summary": "", "questions": "A. In Section 3.2, it is stated that there is one token for each of the N anatomical regions, and that the longitudinal representation is obtained by concatenating the respective anatomical tokens, for the two CXRs. It might be clarified as to the outcome were the anatomical region not to be detected, in one or both CXRs.\n\nB. In Section 5.1, For the baseline results as reported in Table 2, it might be clarified as to whether the comparison methods are able to exploit longitudinal data (i.e. additional prior visit images), since it appears that prior data was indeed used with the proposed approach, from the results in Table 3.\n\nC. Moreover, SA dropping is also exploited in achieving the results reported for the proposed method in Table 2. It might be clarified as to how the relevant SA to keep/drop were chosen, for the Chest ImaGenome dataset - were the bounding box annotations used, and if so, were these annotations expected to correspond to the expected output in the radiology text report? In any case, this would seem to attribute part of the superiority of the proposed method, to the usage of additional information not available to comparison methods.\n\nD. In Section 5.2, hallucination on missing anatomical regions is mentioned. Does this imply that the actual scan(s) do not actually contain the anatomical regions reported on? In any case, this might be illustrated with the corresponding images if possible in an expanded Figure 4, for better understanding.\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that they found the ablation experiments comprehensive to support the two main novel aspects of this paper: longitudinal scan integration and SA dropout. \n\nIn response to the comment in the \"Reasons To Reject\" section (*“unclear if the comparisons against existing methods are entirely fair, given possible use of prior and SA-based inputs not available to such methods”*), we agree that the proposed mechanisms could improve existing methods and hope this paper provides evidence to try introducing our ideas and finding the best marriage with other existing strategies. The intention behind our current results in Table 2 is to illustrate how the two principal novel contributions proposed within this paper when integrated into a standard Transformer architecture, yield SOTA outcomes without employing any of the alternative solutions featured in existing methods (architectural design, reinforcement learning, and knowledge grounding, among others). \n\nPlease find our answers to the questions raised by the reviewer below.\n\n**A. In Section 3.2, it is stated that there is one token for each of the N anatomical regions, and that the longitudinal representation is obtained by concatenating the respective anatomical tokens, for the two CXRs. It might be clarified as to the outcome were the anatomical region not to be detected, in one or both CXRs.**\n\n- As stated in Section 3.1 (lines 226-228), when an anatomical region is not detected, the corresponding token embedding is set to a zero vector representation. This is equivalent to the step of removing an anatomical region that is used in our proposed sentence-anatomy dropout strategy. Therefore, we expect the model to omit reporting on any undetected regions.\n\n**B. In Section 5.1, For the baseline results as reported in Table 2, it might be clarified as to whether the comparison methods are able to exploit longitudinal data (i.e. additional prior visit images), since it appears that prior data was indeed used with the proposed approach, from the results in Table 3.**\n\n- Indeed, the prior CXR is used in the results presented in Table 2. This component is a fundamental aspect of our approach and forms the “longitudinal” aspect of our method. Existing methods have largely ignored prior data, and we consider that it is beyond the scope of this paper to attempt to find the best way to incorporate prior data into each existing method, by making ad-hoc adjustments that potentially deviate far from the original methodologies as conceived by their respective authors. However, we hope that our work showing one method of introducing prior data and its impact on performance might encourage other authors to similarly incorporate prior data into their methods.\n\n**C. Moreover, SA dropping is also exploited in achieving the results reported for the proposed method in Table 2. It might be clarified as to how the relevant SA to keep/drop were chosen, for the Chest ImaGenome dataset - were the bounding box annotations used, and if so, were these annotations expected to correspond to the expected output in the radiology text report? In any case, this would seem to attribute part of the superiority of the proposed method, to the usage of additional information not available to comparison methods.**\n\n- As stated in Section 3.4 (lines 287-306), performing SA dropout requires generating the set of valid sentence-anatomy subsets for each report. This is achieved by identifying *\"the connected components in a graph where the sentences are the nodes and an edge between two nodes represents an overlap of described anatomical regions between the two sentences\"*. This step can be straightforwardly performed using the detailed annotations from the Chest ImaGenome dataset, in which each sentence is annotated with the mentioned anatomical regions. During training, this is used to randomly dropout a subset of sentences from the target output (contained in one or more connected components) and the corresponding anatomical regions from the input. Comparatively, the sole other method within the existing literature utilizing the Chest ImaGenome dataset's sentence-anatomy annotations is RGRG (Tanida et al., 2023), which generates sentences focused on individual regions sequentially. In Section 3.4 and Table 1 (lines 272-286), we explain our rationale for rejecting this approach and instead advocating for SA dropout. The results of RGRG (Tanida et al., 2023) are also presented in Table 2 for comparison. We agree that including sentence/anatomy annotations using the method proposed in this paper or alternative supervision strategies, could improve other existing methods, and hope this paper provides evidence to try this. \n\n**D. In Section 5.2, hallucination on missing anatomical regions is mentioned. Does this imply that the actual scan(s) do not actually contain the anatomical regions reported on? In any case, this might be illustrated with the corresponding images if possible in an expanded Figure 4, for better understanding.**\n\n- No, we do not consider the case that the actual scans may not contain some anatomical regions (although this may occasionally be the case and we would hope that the relevant regions are simply not detected if so). In Section 5.2 and Figure 4, our objective is to illustrate the impact of the SA dropout training strategy by showing that at inference time we get different effects when we block the visual information from selected regions.  Namely, we select a subset of regions that we wish to report on by zeroing out the vector representations of the \"missing\" anatomical regions (effectively simulating their non-detection); therefore, no direct visual information is passed to the reporting model about these regions. When the SA dropout training strategy is not used, descriptions are still generated for these \"missing\" regions. We will augment Figure 4 with CXR images, highlighting the selected regions, provided that we obtain the necessary permissions from the MIMIC-CXR authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": " - Comprehensive ablation experiments on respective contribution of prior data, as well as SA dropping.\n", "reasons_to_reject": " - Unclear if the comparisons against existing methods are entirely fair, given possible use of prior and SA-based inputs not available to such methods\n", "questions_to_authors": "A. In Section 3.2, it is stated that there is one token for each of the N anatomical regions, and that the longitudinal representation is obtained by concatenating the respective anatomical tokens, for the two CXRs. It might be clarified as to the outcome were the anatomical region not to be detected, in one or both CXRs.\n\nB. In Section 5.1, For the baseline results as reported in Table 2, it might be clarified as to whether the comparison methods are able to exploit longitudinal data (i.e. additional prior visit images), since it appears that prior data was indeed used with the proposed approach, from the results in Table 3.\n\nC. Moreover, SA dropping is also exploited in achieving the results reported for the proposed method in Table 2. It might be clarified as to how the relevant SA to keep/drop were chosen, for the Chest ImaGenome dataset - were the bounding box annotations used, and if so, were these annotations expected to correspond to the expected output in the radiology text report? In any case, this would seem to attribute part of the superiority of the proposed method, to the usage of additional information not available to comparison methods.\n\nD. In Section 5.2, hallucination on missing anatomical regions is mentioned. Does this imply that the actual scan(s) do not actually contain the anatomical regions reported on? In any case, this might be illustrated with the corresponding images if possible in an expanded Figure 4, for better understanding.\n", "paper_topic_and_main_contributions": "This paper demonstrates longitudinal representation learning on chest X-ray scans by fusing learned feature representations from both a current and a prior image, and further introduces sentence-anatomy dropout for controlling the text report output, by limiting the report output to a subset of anatomical (SA) regions provided as input. Improved performance is claimed on MIMIC-CXR-based datasets.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}, {"review_id": "4TM7dLSROh", "reviewer": "Reviewer_1BQa", "summary": "", "questions": "Question A: The paper seems to be an extension of the paper (Anonymous,  2023) 'Task-aware anatomical tokens for chest x-ray automated reporting'. Based on the comparison in Table 2 and Table 3, I notice that most of the improvement seems to come from the inherited method from the paper. In some aspects, the proposed method seems to be worser than (Anonymous,  2023). If the authors wrote the two papers, why not combine the methods and propose that as a single paper. In addition, it is unclear to me what are the major differences between this two papers and what makes the most significant contributions to performance difference. \n\nQuestion B: Section 3.4 mentions the differences between the current paper and RGPG (Tanida et al.,2023). Table 2 and 3 shows that even without  Priors and SA drop, it is better than RGPG. Thus, I wonder with the same underlying method, what is the difference between base+SA drop and base+RGPG.\n\nQuestion C: Line 341 mentions that you cropped the CXR. Are the cropped-out regions mostly background or they contain valid information relating to reporting?\n\nQuestion D: Line 373 you used 100 epochs  for training. Does it converge? Does it need that many epochs to converge?\n\nQuestion E: Line 381, you mention that you select models  based on the best score. I am not sure what models did you selected from? Did you selected the best model among different random  seeds?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback and their proposed improvements to make the writing clearer. We are also encouraged that they found the technical novelty compelling.\n\nWe begin by addressing the concern raised in the “Reasons to Reject” section regarding Longitudinal Representation Learning; we believe the other concerns are addressed below in response to specific questions. We have indeed taken a similar approach to Karwande et al. (2022), albeit for the target task of text generation and with a Transformer architecture as the workhorse model. Our novelty lies in proposing a region-level mechanism for the use of prior images as context for text report generation (to the best of our knowledge, we are the first to do so). Since we are inputting the visual tokens to a large Transformer model (and training end-to-end), for practical reasons we chose to use a more computationally efficient method than Karwande et al, employing an MLP rather than GCN, and performing *unified* region detection and encoding in Faster R-CNN (instead of cropping the RoIs from the CXR and encoding them separately).\n\nPlease find our answers to the questions raised below.\n\n**Question A: The paper seems to be an extension of the paper (Anonymous, 2023) 'Task-aware anatomical tokens for chest x-ray automated reporting'. Based on the comparison in Table 2 and Table 3, I notice that most of the improvement seems to come from the inherited method from the paper. In some aspects, the proposed method seems to be worser than (Anonymous, 2023). If the authors wrote the two papers, why not combine the methods and propose that as a single paper. In addition, it is unclear to me what are the major differences between this two papers and what makes the most significant contributions to performance difference.**\n\n- This paper proposes methodological improvements (sentence-anatomy dropout training strategy, incorporation of prior scans to enable longitudinal representations) that improve performance compared to (Anonymous, 2023), in particular on the clinical efficiency (CE) metrics which we believe to be the most important and which we use for best parameter model selection. We agree that the NLG metrics are sometimes very close between the 2 methods, however, we consider these must be treated with caution since they measure linguistic rather than semantic similarity to the ground truth reports, and therefore can be contradictory e.g. a generated sentence *“There is severe pulmonary vascular congestion”* vs ground truth of *“There is no pulmonary vascular congestion”* would score highly on NLG metrics but not on CE metrics. Compared to (Anonymous, 2023) our method has the additional disadvantage that it does not use the two-step architecture used by (Anonymous, 2023); this explains why the ablation results in Table 3 are worse than the inherited method. We intend that the two-step architecture could also be used with our method to maximise performance; we plan to do this investigation as future work (as we state in our Conclusion). We further note that our sentence-anatomy dropout training strategy confers controllability, which was not an explicit feature of (Anonymous, 2023) although an emergent learned relationship between tokens and output was observed in some instances - see Figure 2 in (Anonymous, 2023) - and served as motivation for this paper.\n\n**Question B: Section 3.4 mentions the differences between the current paper and RGPG (Tanida et al.,2023). Table 2 and 3 shows that even without Priors and SA drop, it is better than RGPG. Thus, I wonder with the same underlying method, what is the difference between base+SA drop and base+RGPG.**\n\n- Indeed, our method without Priors and SA drop shows better results than RGRG, which can likely be attributed to the broader context offered by providing the model with the full set of anatomical regions when generating the report as opposed to generating a sentence at the time from a single anatomical region. Since the design of RGRG is centered around how to generate a suitable sentence from a given (single) anatomical region (e.g. region selection module), it is difficult to see which elements of the base method are compatible and could be meaningfully combined whilst retaining the contributions of RGRG. However, we did not put extensive effort into attempting to combine the two since Tanida et al. is a recently released paper (17th April) which we became aware of shortly before the submission deadline of June 23rd.\n\n**Question C: Line 341 mentions that you cropped the CXR. Are the cropped-out regions mostly background or they contain valid information relating to reporting?**\n\n- The regions that are cropped out typically consist of the upper and lower sections of the CXRs, as CXRs usually have a vertical orientation. These sections usually comprise the neck and the lower abdominal area of the patient, depending on how the patient is positioned i.e. not the chest region. Consequently, the relevant regions are still present after cropping the image (e.g. lungs, heart, etc.).  However, we agree there is a risk in rare cases that such image pre-processing cuts out some relevant peripheral regions.\n\n**Question D: Line 373 you used 100 epochs for training. Does it converge? Does it need that many epochs to converge?**\n\n- Yes, the model tends to converge around 70 epochs. Conservatively, we train for a further 30 epochs to ensure the model has trained. However, we select the best model from an earlier epoch using the best F1 score on the validation data. \n\n**Question E: Line 381, you mention that you select models based on the best score. I am not sure what models did you selected from? Did you selected the best model among different random seeds?**\n\n- No, as we mention above, here we refer to the process of choosing the best model for 1 run (i.e. at which epoch the training converged). Apologies, we can see that the sentence ordering for Lines 378-381 in Section 4.4 is confusing (the sentence about best model selection should be moved to the previous paragraph), we will clarify this in the camera-ready paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper combines and extends the effective strategy from existing works and demonstrate improved performance on the task of automatic radiology reporting. The major contributions are:\n\n1. Approach for Radiology Reporting: The paper introduces a novel approach for automating radiology reporting by leveraging anatomical representations of prior scans. This shift from traditional image-level features to anatomical representations presents a new perspective and can potentially enhance the accuracy and clinical relevance of automated reports.\n\n2.Sentence-Anatomy Dropout for Controllability: The introduction of sentence-anatomy dropout as a training strategy allows the report generator model to predict sentences corresponding to specific anatomical regions. This fine-grained controllability in report generation is crucial for clinicians who require targeted and interpretable reports, and it addresses a significant limitation of previous automated reporting systems.", "reasons_to_reject": "1. Lack of Clarity on Major Contributions: The paper appears to combine ideas from existing works rather than introducing a novel structure. It is unclear whether the reported performance improvement is solely attributable to this paper or results from the combination of ideas from the previous work (Anonymous, 2023). This ambiguity makes it challenging to ascertain the specific contributions of this paper and may raise concerns about the novelty and originality of the approach.\n\n2. Limited Performance Improvement: While the paper seems to extend the prior work (Anonymous, 2023) and introduce the methods of Longitudinal Representation Learning and Training with Sentence-Anatomy Dropout, the reported performance improvement in table 2 and 3 are not substantial when compared to (Anonymous, 2023). This raises questions about the significance and effectiveness of the proposed extensions.\n\n3. Lack of Sufficient Novelty in Methodologies: The paper introduces Longitudinal Representation Learning as an extension of Karwande et al. (2022) by using Faster R-CNN (Ren et al., 2015). This approach does not seem to offer a significant advancement beyond the existing methods, potentially limiting its contribution to the NLP community.\n\n4. Some of the experiments can be better designed. For example, Figure 4 compare results from with and without adding prior scans and sentence-anatomy training). For sure that if the baseline never sees anat. region-sentence pairs in training, then it will not generate sentence properly if asking for region descriptions. This experiment relating to hallucination can be better designed.", "questions_to_authors": "Question A: The paper seems to be an extension of the paper (Anonymous,  2023) 'Task-aware anatomical tokens for chest x-ray automated reporting'. Based on the comparison in Table 2 and Table 3, I notice that most of the improvement seems to come from the inherited method from the paper. In some aspects, the proposed method seems to be worser than (Anonymous,  2023). If the authors wrote the two papers, why not combine the methods and propose that as a single paper. In addition, it is unclear to me what are the major differences between this two papers and what makes the most significant contributions to performance difference. \n\nQuestion B: Section 3.4 mentions the differences between the current paper and RGPG (Tanida et al.,2023). Table 2 and 3 shows that even without  Priors and SA drop, it is better than RGPG. Thus, I wonder with the same underlying method, what is the difference between base+SA drop and base+RGPG.\n\nQuestion C: Line 341 mentions that you cropped the CXR. Are the cropped-out regions mostly background or they contain valid information relating to reporting?\n\nQuestion D: Line 373 you used 100 epochs  for training. Does it converge? Does it need that many epochs to converge?\n\nQuestion E: Line 381, you mention that you select models  based on the best score. I am not sure what models did you selected from? Did you selected the best model among different random  seeds?\n", "paper_topic_and_main_contributions": "The paper focuses on addressing the challenges in automating radiology reporting, which typically involves detailed text descriptions of medical scans and their clinical findings. The key problem the paper tackles is the need for high accuracy and interpretability in clinical deployment of automated reporting systems. \n\nThe main contributions of this paper can be summarized as follows:\n\n1. Longitudinal Representation Learning: The paper introduces a novel approach for incorporating prior scan information into the reporting process. Unlike previous methods that do not use the prior study as input, this work proposes longitudinal representation learning. It aligns, concatenates, and fuses the current and prior visual information to create a joint longitudinal representation.\n2. Sentence-Anatomy Dropout: This paper introduces a training strategy for controllability in report generation called sentence-anatomy dropout.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Unclear writing which  may be improved:\nLINE 018: 'an existing visual input format of anatomical tokens,'\nLINE 054: 'monitoring' --> 'disease monitoring'\nLINE 136:  'by ...'\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors describe an interesting and well-executed study. The reviewers agree that the quality of the experiments is high and report no issues with clarity. The main concerns relate to the originality of the approach compared to previous work by the same authors, as well as the significance of the results compared to past experiments and baselines. The author response does a good job alleviating some of these concerns and showcasing the novelty and value of the proposed approach. In my view, the main question at this point is whether this paper should be published in its current form or if there should be a bigger paper that combines this approach with the one proposed in prior work and provides comparison to enhanced baselines.

Quality:

Pros:

-	“This paper presents a straightforward method to make use of prior images for radiology report generation. The authors present compelling results on public datasets, outperforming other recent baselines.”

-	“Comprehensive ablation experiments on respective contribution of prior data, as well as SA dropping”

Cons:

-	“As the image encoder was trained for multi-label prediction, it may not represent the long tail of rare diseases as well.”

Originality:

Pros:

-	“This shift from traditional image-level features to anatomical representations presents a new perspective and can potentially enhance the accuracy and clinical relevance of automated reports.”

-	“The introduction of sentence-anatomy dropout as a training strategy allows the report generator model to predict sentences corresponding to specific anatomical regions. This fine-grained controllability in report generation is crucial for clinicians who require targeted and interpretable reports, and it addresses a significant limitation of previous automated reporting systems.”

Cons:

-	“The paper appears to combine ideas from existing works rather than introducing a novel structure. It is unclear whether the reported performance improvement is solely attributable to this paper or results from the combination of ideas from the previous work (Anonymous, 2023). This ambiguity makes it challenging to ascertain the specific contributions of this paper and may raise concerns about the novelty and originality of the approach.” In their response, the authors highlight that there are methodological changes that lead to improvements on the most significant metrics, as well as to some disadvantages shown in the ablation study. I agree with the point raised by the reviewer that a combination between the two studies into one paper will likely provide better value to the community.

-	“The paper introduces Longitudinal Representation Learning as an extension of Karwande et al. (2022) by using Faster R-CNN (Ren et al., 2015). This approach does not seem to offer a significant advancement beyond the existing methods, potentially limiting its contribution to the NLP community.” The authors respond that: “Our novelty lies in proposing a region-level mechanism for the use of prior images as context for text report generation (to the best of our knowledge, we are the first to do so). Since we are inputting the visual tokens to a large Transformer model (and training end-to-end), for practical reasons we chose to use a more computationally efficient method than Karwande et al, employing an MLP rather than GCN, and performing unified region detection and encoding in Faster R-CNN (instead of cropping the RoIs from the CXR and encoding them separately).” Based on my understanding as a metareviewer, the contribution here is mainly in the computational efficiency of the new approach, which I believe is an important advancement.

Significance:

The main issue here relates to the fact that the comparison to some baselines may not be fair since they were not provided the additional data that the main method utilizes.

-	“Unclear if the comparisons against existing methods are entirely fair, given possible use of prior and SA-based inputs not available to such methods”. Another reviewer notes: “For sure that if the baseline never sees anat. region-sentence pairs in training, then it will not generate sentence properly if asking for region descriptions. This experiment relating to hallucination can be better designed.” The authors respond that: “We agree that including sentence/anatomy annotations using the method proposed in this paper or alternative supervision strategies, could improve other existing methods, and hope this paper provides evidence to try this.” The question about fairness of comparison remains – why not compare to these previous methods by providing them with the additional data as well?

Clarity:

No major concerns reported in the reviews.
Decision: Accept-Findings