RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
Authors: Wenjun Hou
Yi Cheng
Kaishuai Xu
Wenjie Li
Jiang Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: radiology report generation
text generation grounded on vision

Abstract: Automating radiology report generation can significantly alleviate radiologists’ workloads. Previous research has primarily focused on realizing highly concise observations while neglecting the precise attributes that determine the severity of diseases (e.g., small pleural effusion). Since incorrect attributes will lead to imprecise radiology reports, strengthening the generation process with precise attribute modeling becomes necessary. Additionally, the temporal information contained in the historical records, which is crucial in evaluating a patient’s current condition (e.g., heart size is unchanged), has also been largely disregarded. To address these issues, we propose RECAP, which generates precise and accurate radiology reports via dynamic disease progression reasoning. Specifically, RECAP first predicts the observations and progressions (i.e., spatiotemporal information) given two consecutive radiographs. It then combines the historical records, spatiotemporal information, and radiographs for report generation, where a disease progression graph and dynamic progression reasoning mechanism are devised to accurately select the attributes of each observation and progression. Extensive experiments on two publicly available datasets demonstrate the effectiveness of our model.

RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning
Wenjun Hou! Yi Cheng", Kaishuai Xu" Wenjie Li Jiang Liu?t 'Department of Computing, The Hong Kong Polytechnic University, HKSAR, China
"Research Institute of Trustworthy Autonomous Systems and
Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen, China
houwenjun@60@gmail. com
kaishuai xu)@connect. polyu. hk cswjli@comp yulyu. edu. hk, liuj@sustech.edu
Abstract
Prior Visit
Precise Attributes
Concise Observations
Automating radiology report generation can significantly alleviate radiologists' workloads. Previous research has primarily focused on re- alizing highly concise observations while ne- glecting the precise attributes that determine the severity diseases (e.g., small pleural ef- fusion). Since incorrect attributes will lead to imprecise radiology reports, strengthening the generation process with precise attribute modeling becomes necessary. Additionally, the temporal information contained in the histor- ical records, which is crucial in evaluating patient's 's current condition (c.g., heart size is unchanged). has also been largely disregarded. To address these issues, propose RECAP, which generates precise and accurate radiology reports via dynamic disease progression rea- soning. Specifically, RECAP first predicts the observations and progressions (i.e., spatiotem- poral information) given consecutive radio- graphs. It then combines the historical records, spatiotemporal information, radiographs for report generation, where disease progres- sion graph and dynamic progression reasoning mechanism are devised to accurately select the attributes of each observation and progression. Extensive experiments on two publicly avail- able datasets demonstrate the effectiveness of our model.!
unchanged
Enlarged Card. / POS
normal
? Cardiomegaly NEG
decreased
@ Lung Lesion POS
Progressions
Stable
unchanged
Enlarged Card./ / POS
top-normal
@ Cardiumegaly NEG
decreased
Lesion POS
Current Visit
An example of follow-up visit record with its prior visit record. Part their observations are listed with their precise attributes. Enlarged Card. denotes Enlarged Cardiomediastinum.
Figure I:
from the research community due to its large po- tential to alleviate radiologists' workloads.
Recent research works (Nooralahzadeh et al.. 2021; Nishino et al., 2022; Delbrouck et al., 2022; Bannur et al.. 2023; Tanida et al., 2023; Hou et al.. 2023) have made significant efforts in improving the clinical factuality of generated reports. De- spite their progress, these methods still struggle to produce precise and accurate free-text reports. One significant problem within these methods is that although they successfully captured the se- mantic information of observations, their attributes still remain imprecise. They either ignored histori- cal records (i.e., temporal information) that are quired for assessing patients' current conditions omitted the fine-grained attributes of observations (i.e., spatial information) that are crucial in quanti- fying the severity diseases, which are far from adequate and often lead to imprecise reports. Both
1 Introduction
Radiology report generation (Rennie et al., 2017; Anderson al., 2018; Chen et al., 2020), aiming to generate clinically coherent and factually accurate free-text reports, has received increasing attention
"Equal Contribution.
'Correspodding authors.
OOur code is available at https: //gi thub. con/wjhouj Recap.

---

temporal and spatial information are crucial for gen- erating precise and accurate reports. For instance, as illustrated Figure 1, the patient's conditions can change from time to time, and the observations become Better and Stable. Only accessing the historical records, the overall conditions could be estimated. In addition, different attributes reflect the severity of an observation, such as normal and top-normal for Cardiomegaly. In order produce precise and accurate free-text reports, we must con- sider both kinds information and apply stronger reasoning to strengthen the generation process with precise attribute modeling.
[X?,y?} or D 0 if the historical record is missing". the task of radiology report genera- tion aims to maximize p( DP). To learn the spatiotemporal information, observations (i.e., spatial information) (Irvin et al., 2019) and progressions P (i.c., temporal information) (Wu et al., 2021) are introduced. Then, the report gen- eration process is divided into two stages in our framework, i.e., observation and progression pre- diction (i.e., Stage 1) and spatiotemporal-awwe -wware aware report generation (i.e., Stage 2). Specifically, the probability of observations and progressions are de- noted as p()]x") and p(P .X"). respectively, and then the generation process is modeled as p D2 O, P). Finally, our framework aims to maximize the following probability:
In this paper, we propose RECAP, which cap- tures both temporal and spatial information for ra- diology REport Generation via Dynamic DiseAse Progression Reasoning. Specifically, RECAP first predicts observations and progressions given two consecutive radiographs. then combines them with the historical records and the current radio- graph for report generation. To achieve precise attribute modeling, we construct a disease progres- sion graph, which contains the prior and current observations, the progressions, and the precise at- tributes. We then devise a dynamic progression reasoning (PrR) mechanism that aggregates infor- mation in the graph select observatioo--ellvaat attributes.
Stage
x X x(OX')) p(PXX XP)) p(1 D",O,P)
Stage
2.2 Progression Graph Construction
Observation and Progression Extraction. For each report, we first label its observations {oi, , with CheXbert (Smit et al., 2020). Similar to Hou et al. (2023), each observation is further labeled with its status (i.c., Positive, Nega- live, Uncertain, and Blank). We convert Positive and Uncertain POS, Negative as NEG, and re- move Blank, shown in Figure 1. Then, extract progression information P of patient with Chest ImaGenome (Wu al., 2021) which provides pro- gression labels (i.c., Better, Stable, or Worse) between two regions interest (ROIs) X" and X respectively. However, extracting ROIs could be difficult, and adopting such ROI-level labels may not generalize well across different datasets. Thus, we use image-level labels, which only indi- cate whether there are any progressions between X² and X As result, patient may have dif- ferent progressions (e.g.. both Better and Worse), The statistics of observations and progressions can be found Appendix A.
In conclusion, our contributions can be summa- rized as follows:
We propose RECAP, which capture both spatial and temporal information for generat- ing precise and accurate free-text reports.
To achieve precise attribute modeling, we construct a disease progression graph con- taining both observations and fine-grained at- tributes that quamtify the severity of discases. Then, devise dynamic disease progres- sion reasoning (PrR) mechanism to select observation/progression -nneean attributes.
We conduct extensive experiments on two pub- licly available benchmarks, and experimental results demonstrate the effectiveness of our model in generating precise and accurate radi- ology reports.
Spatial/Tempora Entity (Attribute) Collection." To model spatial and temporal information, we
2 Preliminary
"There are two kinds records (i.e., first-visit and follow- up-visit). If the first visit patient, the historical record does not exist.
2.1 Problem Formulation
AAtrrbbuees are included in the entity set as provided by Jain et al. (2021). For simplicity, we use "attribute" and "entity" interchangeably in this paper.
Given a radiograph-report pair D (X", Y*). with its record of last visit being either DP

---

Spatial/Tempon Triples
Progression Graph
Prior Observation
Stable
0
Current Ohservation
Spatial
Stable
R5
e
Temporal Enticy
Construct
0
Belter
Sparial Encity
Temporal
Ra
Better
Current Visit
Retrieve
Xx
Predict
Observations
Progressions
p
Vision Transformer
Lung Lesion/
Stable
x
Card./NEG
/Better
XWorre
[CLSS
Prior Viait
(If Extsts)
Pro-Encoder,
Pro-aware Cross-Attn,
CLS
Decoder
x
Obs-Encoder,
Obs-aware Cross-Attn,
Masked Self-Attn
x
y
x
Stage
het
Stage
Figure 2: Overview RECAP framework. Pro-Encoderp is the progression encoder and Obs-Encoder, is the observation-related encoder, respectively. Other modules in the decoder omitted for simplicity.
collect set of entities to represent it. For temporal entities, we adopt the entities provided by (Bannur et al., 2023), denoted as F" For spatial entities ES, we adopt the entities with a relation modify or located_at in RadGraph (Jain et al., 2021), and we also filter out stopwords" and temporal entities from them. Part of the temporal and spatial entities are listed in Appendix A.2.
à to oj where oj c (E", FES. Then, we rank these triples using PMI((o; ry). ) e and select top- K of them as candidates for each (o,, ry). Finally, we use observations as the query to retrieve relevant triples. We consider edges in the graph: ez 2 of No of Imm, ex as shown in the top-right of Figure 2, consistent with the progression direction.
Progression Graph Construction. Our progres- sion graph G =< V. > constructed based purely on the training corpus in an unsupervised manner. Specificially, V = {0, J is the node-set, and R = (S, B, W, Rs. Ro} is the edge set, where S, B, and W denote three pro- gressions Stable, Better, and Worse, connecting observation with an temporal entity. addi- tion, Rg and R, are additional relations connect- ing current observations with spatial entities and prior/current observations, respectively. To extract spatial/temporal triples automatically, we use the proven-efficient statistical tool, i.c., pointwise mu- tual information (PMI; Church and Hanks (1990)), where higher PMT score implies two units with higher co-occurrence, similar to Hou et al. (2023):
Methodology
3.1 Visual Encoding
Given an image X an image processor is first to split it into N patches, and then visual encoder (i.e., VIT (Dosovitskiy et al., 2021)) is adopted to extract visual representations X
X = [[CLS) .f....., xx} = ViT(X"),
where [CLS] E Ra is the representation of the class token [CLS] prepended in the patch sequence, xt R is the i-th visual representation. Simi- larly, the visual representation of image X" is ex- tracted using the same ViT model and represented as X
xf....
Stage 1: Observation and Progression 3.2
p à) log p() PMI [,, .t log p())
Prediction
p(t)
Observation Prediction. As observations can be measured from a single image solely, we only use the pooler output [CLS) of X for observation
Specifically, we & to (os, T5) where rj R and
'https: httpp:://www.lll mltk. org/

---

prediction. Inspired by Tanida et al. (2023), we divide it into two steps, i.c., detection and then clas- sification. Specifically, the detection probability pa(o:) of the i-th observation presented in a report and the probability of this observation Po(o:) being classified as abnormal are modeled as:
or [FoV] for follow-up-visit records is appended distinguish them, represented as [F*V]:
h = [xx: hg] = Encoder, [X: ( [F*V]; O9),
where f hg E I are the visual hidden represen- tations and observation hidden representations of the current radiograph and observations.
(os) a Wa (CLS) bd;), Po(o,) = (We, e, [CLS + be:),
Progression aware Information Encoding. We use another encoder to encode the progression in- formation (i.e., temporal information). Specifically, given X and YP, the hidden states of the prior record are represented as:
where a is the Sigmoid function, Wd,, E R are the weight matrices and ba. beg R are the bi- ases. Finally, the probability of the i-th observation is denoted p(o;) = pa(os) Pe(o:). Note that for observation No Finding On is presented in every sample, i.e., pa(o,) = 1 and p(on) = Po(on). Progression Prediction. Similar to observation prediction, the pooler outputs [CLS]P of XP and (CLS]* of X are adopted for progression predic- tion, and the probability of the j-th progression p(p)) is modeled as:
h² - [1'x h2y Encooer,,(X""; YPI). where h²x hy E RA are the visual hidden rep- resentations and textual hidden representations of prior records, respectively.
Concise Report Decoding, Given he and ht, Transformer decoder is adopted for report genera- tion. Since not every sample has prior record and follow-up records may include new observations, controlling the progression information neces- sary. Thus, we include a soft gate a to fuse the observation-related and progression-related infor- mation, as shown in Figure 2:
[CLS] = [[LSPP [
p(p,) = o (CSS) + bj),
where [:] is the concatenation operation, Wj C 2 is the weight matrix, and by E I are the bias. As we found that learning sparse signals from image-level progression labels is difficult and has side effects on the performance of observation pre- diction, we detach [CLS] from the computational graph while training.
h = Self-Attn(h, h h:re),
h; = Cross-Atth, (hū: he h(),
Decoder
h2 Cross-Atun (hi h², h²)
a = (Wahi + ba):
Training. We optimize these two prediction tasks by minimizing the binary cross-entropy loss. Specifically, the loss of observation detection La is denoted as:
h h. + (1 a) hf: Pv( Softmax (Wyh+ + by),
where Self-Attn is the self-attention module, Cross- Attn is the cross-attention module, hü hf, hff h, E R are self-attended hidden state, observation- related hidden state, progression-related hidden state, and spatiotemporal-avve spectively, W. C R
Σ laa ld, logp:(o;)
Ca = 101
+(1 la;) log(1 pa(o:))),
hidden state, re- Wy C R are weight matrices and ba R. by E glM are the biases. Disease Progression Encoding, As there are dif- ferent relations between nodes, we adopt an I-layer Relational Graph Convolutional Network (R-GCN) (Schlichtkrull et al., 2018) to encode the disease progression graph, similar to Ji et al. (2020):
where an is the weight to tackle the class imbalance issue, ld; denotes the label of i-th observation d,- Similarly, the loss of observation classification Le and progression prediction Cp can be calculated using the above equation. Note that L. and Lp are unweighted loss. Finally, the overall loss of Stage 1 is Ls1 = La + L. + Cp.
3.3 Stage 2: Tpatoolemmpoallawaaa Report Generation
TyE
41 = ReLU
Σ W
wh'h'. h + Wihh.
Observ Visual Encoding. To learn the observation-aware visual representations, jointly encode X and its observations O" using a Transformer encoder (Vaswani et al., 2017). Addi- tionally, a special token [Fiv] for first-visit records
VLEV
where & is the number of neighbors connected to the i-th node, W W! E R Rhxh are learnable weight metrics, and h! 1l1 h E IR are the hidden representations.

---

NLG
Metrics
CE Metrics
Dataset
Model
B-1
B-2
B-3
B-4
MTR
R-L
R
Fi
R2GEN
0.290
0.157
0.093
0.061
0.105
0.208
0.266
0.320
0.272
MIMIC
R2GENCMN
0.264
0.140
0.085
0.056
0.098
0.212
0.290
0.319
0.280
-ABN
ORGAN
0.314
0.180
0.114
0.078
0.120
0.234
0.271
0.342
0.293
RECAP (Ours)
0.321
0.182
0.116
0.080
0.120
0.223
0.300
0.363
0.305
R2GEN
0.353
0.218
0.145
1.103
0.142
0.270
0.333
0.273
0.276
R2GENCMN
0.353
0.218
0.148
0.106
0.142
0.278
0.344
0.275
0.278
M
0.378
0.232
0.154
0.107
0.145
0.272
0.240
0.428
0.308
KNOW MAT
0.363
0.228
0.156
0.115
0.284
0.458
0.348
0.371
CMM-RL
0.381
0.232
0.155
0.109
0.151
0.287
0.342
0.294
0.292
MIMIC
CMCA
0.360
0.227
0.156
0.117
0.148
0.287
0.444
0.297
0.356
-CXR
KiUT
0.393
0.243
0.159
0.113
0.160
0.285
0.371
0.318
0.321
DCL
0.109
0.150
0.284
0.471
0.352
0.373
METrans
0.386
0.250
0.169
0.124
0.152
0.291
0.364
0.309
0.311
ORGAN
0.407
0.256
0.1
0. 121
0.162
0.293
0.416
0.418
0.385
RECAP (Ours)
0.429
0.267
0.177
0.125
0.168
0.288
0.389
0.443
0.393
Table I: Experimental Results of our model and baselines on the MIMIC-ABN and MIMIC-CXR datasets.
The best results are in boldface, and the underlined are the second-best results. The experimental results on the MIMIC-ABN dataset are replicated based their corresponding repositories.
Precise Report Decoding via Progression Rea- soning. Inspired by Ji et al. (2020) and Mu and Li (2022), we devise a dynamic disease progression reasoning (PrR) mechanism to select observation- relevant attributes from the progression graph. The reasoning path of PrR is of 2 Ek. where ry be- longs to either three kinds of progression or Rg. Specifically, given t-th hidden representation h; the observation representation h and the entity representation h of Eg, the progression score (ex) of node ex is calculated as:
to combine py({)) and (G( into p(u)):
p())) Dr Pv(U)) (1 gr) PC(Ut); where W, R and bg c IR are the weight matrix and bias, respectively.
Training. The generation process is optimized using the negative log- oggliiklihood loss, given each token's probability p(ye) and the probability of ge:
I
LNLI
Σ logp(v)),
T
psc(ex) = (Weel
Σ
ohh, Wr, (b: D
Cg =
Σ t=1
Lg logg: + (1 - )og((1 9g)]]
(ow.)))N,
where 1ge indicates t-th token appears in G. Finally, the loss of Stage 2 is Ls2 LNLL + Xhg.
pss(ex) = y (((( W.h!.
where is the Tangent function, is the scale factor, No. is the neighbor collection of eg. and Wr, E h2 and W. E hg are weight ma- trices for learning relation ra and self-connection, respectively. In the PrR mechanism, the relevant scores (i.e., psr (sr(e)) (ex)) of their connected observa- tions are also included in (es) since h contains observation information, and higher relevant scores of these connected observations indicate a higher relevant score of Cę- Then, the distribution over all entities in G is denoted as:
4 Experiments
4.1 Datasets
We use two benchmarks to evaluate our models. MIMIC-ABN (Ni et al., 2020) and MIMIC- C (Johnson et al., 2019). We provide other details of data preprocessing Appendix A.3.
MIMIC-CXR consists of 377.110 chest X- ray images and 227,827 reports from 63,478 patients. We adopt the settings of Chen et al. (2020).
(G(m) Softmax (oftmax(pss,(cx)) (cx))
Finally, soft gate gi = (W h + by) is adopted
https://gituubb.omm/zz//////////
http:////ppissoontooog//nnnneene 2.0.0/

---

Model
Sections
B-2
CE-Fi
Model
B-4
R-L
CE-Fi
TEM
R2GEN
Find. & Imp.
0.212
0.148
CXR-RePaiR-2
0.021
0.143
0.281
0.125
IFCC
Findings
0.217
0.270
BioViL-NNN
0.037
0.200
0.283
0.111
CXR-RePaiR-SS
Impressions
0.050
0.274
BioViL-T-NN
0.045
0.205
0.290
0,130
ViL-
Impressions
0.159
0.348
BioViL-AR
0.075
0.279
0.293
0.138
BioViL-T
Find. & Imp.
0.213
0.359
BioViL-T-AR
0.092
0.296
0.317
0.175
ORGAN
Findings
0.267
0.385
RECAP (Ours)
0.118
0.279
0.400
0.304
RECAP (Ours)
Findings 0.265
0.393
RECAP w/o OP
0.093
0.260
0.256
0.203
RECAP w/o Obs
0.104
0.270
0.307
0.240
Table BLEU score and CheXbert score of our model and baselines the MIMIC-CXR dataset. Results cited from Bannur et al. (2023) and Hou et al. (2023).
RECAP w/ Pro 0.103
0.266
0.395
0.269
RECAP w/o PrR
0.115
0.279
0.403
0.296
Progression modeling performance of our model and baselines on the MIMIC-CXR dataset. *-NN models nearest neighbor search for report generation, and the *-AR models use autoregressive de- coding, indicated Bannur (2023).
Table 3:
MIMIC-ABN is a modified version of MIMIC-CXR and only contains abnormal sentences. The original train/validationntesst split of Ni et al. (2020) is 16,946/3,80177 .804 samples, respectively. To collect patients' historical information and avoid information leakage, we recover the data-split used in MIMIC-CXR according to the subject_idl id". Finally, the data-split used in our experiments is 71,786/546/886 for tain/validation/testt sets, respectively.
4.3 Implementation Details
We use the ViT (Dosovitskiy al., 2021), vision transformer pretrained on ImageNet (Deng et al., 2009), as the visual encoder The maximum de- coding step is set to 64/104 for MIMIC-ABN and MIMIC-CXR, respectively. y is set to and K is set to 30 for both datasets.
For model training. we adopt AdamW (Loshchilov and Hutter, 2019) the optimizer. The layer number of the Transformer encoder and decoder are both set to 3, and the dimension of the hidden state is set to 768, which the same as the one of ViT. The layer number L of the R-GCN is set to 3. The learning rate is set to 5e-5 and le-4 for the pretrained ViT and the rest of the parameters, respectively. The learning rate decreases from the initial learning rate to 0 with linear scheduler. The dropout rate is set to 0. 1, the batch size is set to 32, and À is set to 0.5. We select the best checkpoints based on the BLEU-4 on the validation set. Our model has 160.05M trainable parameters, and the implementations are based on the HuggingFace's Transformers (Wolf et al., 2020). We implement our models on an NVIDIA-3090 GTX GPU with mixed precision. Other details of implementation (e.g., Stage training) can be found in Appendix A.3.
4.2 Evaluation Metrics and Baselines
NLG Metries. BLEU (Papineni et al., 2002), ME- TEOR (Banerjee and Lavie, 2005), and ROUGE (Lin, 2004) are selected as the Natural Language Generation metrics (NLG Metrics), and we use the MS-COCO evaluation tool to compute the results. CE Metrics. For Clinical Efficacy (CE Metries), CheXbert (Smit et al., 2020) adopted label the generated reports compared with disease labels of the references. Besides, we use the temporal entity matching scores (TEM), proposed by Bannur et al. (2023), to evaluate how well the models generate poogression-relater information.
Baselines. For performance evaluation, we com- pare our model with the following state-of-the-at (SOTA) baselines: R2GEN (Chen et al., 2020), R2GENCMN (Chen et al., 2021), KNOWMAT (Yang et al., 2021), 22TR (Nooralahzadeh et al., 2021), CMM-RL (Qin and Song, 2022), CMCA (Song et al., 2022), CXR-RePaiR-Sel/2 (Enndo et al.. 2021), Bio (Bannur al.. 2023), DCL (Li al., 2023), METrans (Wang et al., 2023), KiUT (Huang et al., 2023), and ORGAN (Hou et al., 2023).
Results
5.1 Quantitative Analysis
NLG Results. The NLG results of two datasets are listed on the left side of Table and Table 2. As we can see from Table 1, RECAP achieves the best
"subject id is the anonymized identifier of patient. "https://github catoll.oooottteeeeeeeeeeeeeeeeee wwt.//oooo/oooooeoooooocccoccc caption chttps://gitteee com/ wtt.m///ttaarrrrggggoo///hhctttt
The model card is boogleeviibbus--pnccl6---------

---

NLG Metrics
CE Metrics
Dataset
Model
B-1
B-2
B-3
B-4
MTR
R-L
P
R
Fi
RECAP
0.321
0.182
0.116
0.080
0.120
0.223
0.300
0.363
0.305
RECAP w/o OP
0.303
0.170
0.109
0.074
0.113
0.227
0.289
0.300
0.280
MIMIC
RECAP w/o Obs
0.302
0.174
0.114
0.079
0.114
0.231
0.341
0.314
0.282
-ABN
RECAP w/o Pro 0.306
0.169
0.107
0.072
0.114
0.220
0.298
0.361
0.298
RECAP w/o PrR
0.320
0.180
0.115
0.079
0.120
0.224
0.295
0.365
0.301
RECAP
0.429
0.267
0.177
0.125
0.168
0.288
0.389
0.443
0.393
RECAP w/o OP
0.350
0.219
0.150
0.109
0.140
0.278
0.356
0.259
0.266
MIMIC -CXR
RECAP w/o Obs
0.356
0.224
0.153
0.113
0.144
0.283
0.464
0.281
0.296
RECAP w/o Pro 0.402
0.245
0.161
0.112
0.157
0.278
0.379
0.433
0.386
RECAP w/o PrR
0.415
0.257
0.171
0.119
0.164
0.285
0.381
0.443
0.391
Table Ablation results of our model and its variants. RECAP w/o OP the standard Transformer model, w/o Obs stands for without observation, and w/o Pro stands for without progression.
performance compared with other SOTA models and outperforms other baselines substantially on both datasets.
pared with variant 1, the performance of RECAP w/o Obs increases substantially on the MIMIC- CXR dataset, which demonstrates the importance of historical records in assessing the current condi- tions of patients. In terms of CE metrics, learning from the observation information boosts the perfor- mance of RECAP drastically, with improvement of 12%. In addition, the performance of RECAP increases compared with variant w/o PrR. This in- dicates that PrR can help generate precise and ac- curate reports.
Clinical Efficacy Results. The clinical efficacy re- sults are shown on the right side of Table 1. RECAP achieves SOTA performance on F1 score, leading to 1.2% improvement over the best baseline (i.e., orGan) the MIMIC-ABN dataset. Similarly, on the MIMIC-CXR dataset, our model achieves a score of 0.393, increasing by 0.8% compared with the second-best best. This demonstrates that RECAP can generate better clinically accurate reports. Temporal-related Results. Since there are only 10% follow-up-visits records in the MIMIC-ABN dataset, we mainly focus on analyzing the MIMIC- CXR dataset, as shown in Table and Table 6. RECAP achieves the best performance on BLEU- 4, TEM. In terms of the clinical F,, RECAP w/o PrR outperforms other baselines. This indicates that historical records are necessary for generating follow-up reports.
5.2 Qualitative Analysis
Case Study. We conduct detailed study on how RECAP generates precise and accurate at- tributes of given radiograph in Figure 3, RECAP successfully generates six observations, including five abnormal observations. Regarding attribute modeling, our model can generate the precise de- scription "the lungs are clear without focal con- solidation", which also appears in the reference. while RECAP w/o OP can not generate relevant descriptions. This indicates that spatiotemporal information plays vital role in the generation pro- cess. Additionally, RECAP can learn to compare with the historical records (e.g., mediastinal con- tours are stable and remarkable) so as to precisely measure the observations.
Ablation Results. We perform ablation analysis. and the ablation results are listed in Table 4. We also list the ablation results progression model- ing in Table 6. There are four variants: (I) RECAP w/o OP (i.e., a standard Transformer model, re- moving spatiotemporal information), (2) RECAP w/o Obs (i.e., without observation), (3) RECAP w/o
Pro (i.e., without progression), and (4) RECAP w/o PrR, which does not adopt the disease progression reasoning mechanism.
Error Analysis. We depict error analysis to pro- vide more insights, as shown in Figure 4. There are two major errors, which are falsc-positivy ob- servations (i.e., Positive Lung Opacity and Positive Pleural Effusion) and false-negative observations (i.e., Negative Cardiomegaly). Improving the per- formance of observation prediction could be an im- portant direction in enhancing the quality of gener-
As can see from Table 4, without the spa- tottmmpral information (i.e., variant 1), the perfor- mances drop significantly on both datasets, which indicates the necessity of spatiotemporal modeling in free-text report generation. In addition, com-

---

Radiagraph
Prior Report: low lung volumes are present. this accentuates the size of the cardiac Silhouette which is likely mildly enlarged. ! mediastinal and hilar contours are likely within normal limits. right brachiocephalic venous stent is re-demonstrated. there is crowding of the bronchovascular structures with probable 3 mild pulmonary vascular congestion. ① no pleural effusion or 5 pneumothorax is identified.
Reference: ap and lateral views of the chest. the lungs are + clear of consolidation or 3 effusion.
Current Progression: Stable
(2) the cardiac silhouette is enlarged but unchanged. no acute osseous abnormality is detected. right brachiocephalic venous stent again noted.
Ours:
the lungs are clear without focal consolidation. no pleural effusion pneumotho- is the cardiac silhouette top-normal mildly enlarged. mediastinal contours are stable and unremarkable. there no pulmonary edema.
Ours w/o OP: ap upright and lateral views of the chest provided. lung volumes are low limiting assessment. allowing for this there is ( no focal consolidation ① effusion or 5 pneumothorax. ( the cardiomediastinal silhouctte is normal. imaged osseous structures intact. free air below the right hemidiaphragm seen.
Cardiomapaly//
Edema/Net
Consolidation//
Preumcchrrx//NEG
Plcural Effusion/NEG
Figure 3: Case study of a bollow-up-visit sample, given its prior radiograph and prior report. Attributes of observa- tions in reports are highlighted in boldface, and spans with colors reports indicate mentions of observations.
enhance the performance and Liu et al. (2021b)
Reference: there is no new consolidation, right pncumonia
present significantly improved. there mmmmamdiiscinum prsumath- orax. there pleural effusion. mediastinal and cardiac contours are stable.
proposed to explore posterior and prior knowledge for report generation. Yang et al. (2021); Li et al. (2023); Huang et al. (2023) proposed to utilize the external knowledge graph (i.e., RadGraph (Jain et al., 2021)) for report generation.
esophageal
Ours: heart size snormal, the hilar and
contours
Enlarged /NEG
small right pleural effision. the visalized been interval improvement of the right basi: opacisy.
there pncumothorax.
Cardiomegall/iF NEG
Lung Osacity/F
Other works focused on improving the clinical accuracy and faithfulness of the generated reports. Liu et al. (2019a); Lovelace and Mortazavi (2020); Miura al. (2021); Nishino et al. (2022); Del- brouck et al. (2022) designed various kinds of re- wards (e-g., entity matching score) to improve clin- ical accuracy via reinforcement learning. Tanida et al. (2023) proposed an explainable framework for report generation that could identify the abnor- mal areas of given radiograph. Hou et al. (2023) proposed to combine both textual plans and radio- graphs to maintain clinical consistency. Addition- ally, Ramesh al. (2022) and Bannur al. (2023) focus on handling the temporal structure in radiol- ogy report generation, either removing the prior or learning from the historical records.
Pneumothorax//N
Effusion/F POS
Figure 4: Error case generated by RECAP. The span and the spans denote false negative observation and false positive observation, respectively.
ated reports. In addition, although RECAP aims to model precise attributes of observations presented in the radiograph, it still not cover all the cases. This might be alleviated by incorporating external knowledge.
6 Related Work
6.1 Medical Report Generation
Medical report generation(Jing et al., 2018; Li al., 2018), as one kind of image captioning (Vinyals et al., 20 Rennie et al., 2017: 7; Lu et al., 2017; An- derson et al., 2018), has received increasing atten- tion from the research community. Some works fo- recording key information of the generation process via memory mechanism (Chen al., 2020. 2021; Qin and Song, 2022: Wang et al., 2023). In addition, Liu et (2021c) proposed to utilize con- trastive learning to distill information. Liu et al. (2021a) proposed to use curriculum learning to
6.2 Graph Reasoning for Text Generation
Graph reasoning for text generation (Liu et al., 2019b; Tuan et al., 2022) tries to identify relevant knowledge from graphs and incorporate it into gen- erated text sequences. Huang et al. (2020) proposed to construct knowledge graph from the input doc- ument and utilize to enhance the performance of abstractive summarization. et al. (2020) pro- posed to incorporate commonsense knowledge for

---

language generation via multi-hop reasoning. Mu and Li (2022) proposed to combine both event- level and token-level from the knowledge graph to improve the performance.
China (Grant No. 82272086, 62076212), Guang- dong Provincial Department of Education (Grant No. 2020ZDZXX043), Shenzhen Natural Sci- ence Fund (JCY1200001001 19440020999999999 and the Sta- ble Support Plan Program and the Research Grants Council of Hong Kong (15207920, 15207821, 15207122).
7 Conclusion
In this paper, we propose RECAP, which can cap- ture both spatial and temporal information for gen- erating precise and accurate radiology reports. To achieve precise attribute modeling in the generation process, we construct disease progression graph containing both observations and fine-ggraind at- tributes which quantify the severity of diseases and devise a dynamic disease progression reasoning (PrR) mechanism to select observation-rellvann attributes. Experimental results demonstrate the effectiveness of our proposed model in terms of generating precise and accurate radiology reports.
References
Peter Anderson. Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 6077-6086, Computer Vi- sion Foundation IEEE Computer Society.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An automatic metric for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACI. Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Transla- tion and/or Sammarization, pages 65-72, Ann Arbor, Michigan, Association for Computational Linguis- ties.
Limitations
Our proposed two-stage framework requires pre- defined observations and progressions for training, which may not be available for other types of radio- graphs. In addition, the outputs of Stage I are the prerequisite inputs of Stage 2, and thus, our frame- work may suffer from error propagation. Finally, although prior information is important in generat- ing precise and accurate free-text reports, historical records are not always available, even in the two benchmark datasets. Our framework will still gen- erate misleading free-text reports, conditioning on non-existent priors, as indicated in Ramesh et al. (2022). This might be mitigated through rule-based removal operations.
Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fer- nando Pérez-García, Maximilian Tlse, Daniel C. Cas- tro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle and Ozan Oktay. 2023. Learning to exploit temporal structure for biomedical vision- language processing.
Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021. Cross-modal memory networks for radiology report generation.
Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing, ACI/IICNLP 2021, (Volume 1: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 5904 5914. Association for Computational Linguistics.
Ethics Statement
The MIMIC-ABN(Ni et al., 2020) and MIMIC- CXR (Johnson et al., 2019) datasets are publicly available benchmarks and have been automatically de-identified to protect patient privacy. Although our model improves the factual accuracy of gener- ated reports, its performance still lags behind the practical deployment. The outputs of our model may contain false observations and diagnoses due to systematic biases. In this regard, we strongly urge the users to examine the generated output in real-world applications cautiously.
Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xi- ang Wan. 2020. Generating radiology reports via memory-driven transformer. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing.
Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicog- raphy. Computational Linguistics, 16(1):22 22:2--2-9 29.
Jean-Benoit Delbrouck, Pierre Chambon, Christian Bluethgen, Emily Tsai, Omar Almusa, and Curtis Langlotz. 2022. Improving the factual correctness of radiology report generation with semantic rewards. In Findings of the Association for Computational
A Acknolwedgments
This work was supported in part by General Pro- gram of National Natural Science Foundation of

---

Linguistics: EMNLP 2022, pages 4348-4366 Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
on Empirical Methods in Natural Language Process- ing (EMNLP), pages 725-736, Online. Association for Computational Linguistics.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hier- archical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pages 248 255.
Baoyu Jing. Pengtao Xie, and Eric P. Xing. 2018. On the automatic generation of medical imaging reports. In Proceedings of the 56th Anmual Meeting of the As- sociation for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2577-2587 Association for Com- putational Linguistics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations.
Alistair EW Johnson, Tom J Pollard, Nathaniel Green- baum, Matthew P Lungren. Chih-ying Deng. Yifan Peng, Zhiyong Lu, Roger G Mark, Seth J Berkowitz, and Steven Horng. 2019. Mimic-cxr-jpg. a large pub- licly available database of labeled chest radiographs. arXiv preprint arvi: 00t1::9901.000 1901.07042.
Mark Endo, Rayan Krishnan, Viswesh Krishma, An- drew Y. Ng, and Pranav Rajpurkar. 2021. Retrieval- based chest x-ray report generation using a pre- trained contrastive language-image model. In Pro- ceedings of Machine Learning for Health, volume 158 of Proceedings of Machine Learning Research, pages 209-219. PMLR.
Mingjie Li. Binggian Lin, Zicong Chen, Haokun Lin, Xiaodan Liang, and Xiaojun Chang. 2023. Dynamic graph enhanced contrastive learning for chest x-ray report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pages
Yuan Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2018. Hybrid retrieval- retrieval-generattee generation reinforced agent for medical image report generation. In Advances in Neural Information Processing Systems 31: An- nual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pages 1537-1547.
Wenjun Hou, Kaishuai Xu, Yi Cheng, Wenjie Li, and Jiang Liu. 2023. Organ: Observation-guided radiol- ogy report generation via tree reasoning.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5094-5107, On- line. Association for Computational Linguistics.
Chin-Yew Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- fion Brunches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Zhongzhen Huang. Xiaofan Zhang, and Shaoting Zhang- 2023. Kiut: Knowledger Snowledge-injected u-transformer for radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat-
Fenglin Liu, Shen Ge, and Xian Wu. 2021a. Competence-based multimodal curriculum learning for medical report generation. In Proceedings of the 59th Annual Meeting of the Association for Com- putational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNPP 2021, (Volume Long Papers), Virtual Event, August 1-6, 2021, pages 3001-3012. Associa- tion for Computational Linguistics.
lern Recognition (CVPR), pages 19809-19818.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko. Yifan Yu. Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn L.. Ball, Katie S. Shpan- skaya, Jayne Seekins, David A. Mong, Safwan Ha- labi, Jesse K. Sandberg, Ricky Jones, David B. Lar- son, Curtis P. Langlotz. Bhavik N. Patel, Matthew P. Lungren, and Andrew Y. Ng. 2019. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In The Thirty-Third AAAI Conference on Artificial Intelligence, pages 590-597.
Fenglin Liu, Xian Wa, Shen Ge, Wei Fan, and Yuexian Zou. 2021b. Exploring and distilling posterior and prior knowledge for radiology report generation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 13753-13762. Computer Vision Foundation IEEE.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven Q. H. Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, An- drew Y. Ng. Curtis P. Langlotz. and Pranav Rajpurkar. 2021. Radgraph: Extracting clinical entities and rela- tions from radiology reports. CORR, abs/2106. 14463,
Fenglin Liu, Changehang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. 202 lc. Contrastive attention for automatic chest x-ray report generation. In Find- ings of the Association for Computational Linguis- tics: ACIAJCCLLPP 2021, Online Event, August 1-6, 2021, volume ACL/UCNLP 2021 of Findings of ACL, pages 269 280- Association for Computational Lin- guistics.
Haozhe Ji, Pei Ke, Shaohan Huang, Furu Wei, Xiaoyan Zhu, and Minlie Huang. 2020. Language generation with multi-hop reasoning on commonsense knowl- edge graph. In Proceedings of the 2020 Conference

---

Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew B. A. McDermott, Willie Boag. Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019a. Clini- cally accurate chest x-ray report generation. CoRR, abs/1904.02633.
Language Processing, Online. Association for Com- putational Linguistics.
Farhad Nooralahzadeh. Nicolas Perez Gonzalez. Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer. 2021. Progressive transformer-base based generation of radiology reports. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2824-2832, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Zhibin Liu, Zheng-Y Yu Niu, Hua Wu, and Haifeng Wang. 2019b. Knowledge aware conversation generation with explainable reasoning over augmented graphs. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNAL pages 1782- 1792 Hong Kong, China, Association for Computa- tional Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine tramslation. In Proceedings of the 40th Anmual Meeting of the Association for Compu- tational Linguistics, pages 31 511-318, Philadelphia. Pennsylvania, USA. Association for Computational Linguisties.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Han Qin and Yan Song. 2022. Reinforced cross-men mosaal alignment for radiology report generation. In Find- ings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 448 458. Association for Computational Linguistics.
Justin Lovelace and Bobak Mortazavi. 2020. Learning to generate clinically coherent chest X-ray reports. In Findings of the Association for Computational Linguistics: EMNLP 2020. pages 1235- 1243, Online. Association for Computational Linguistics.
Vignay Ramesh, Nathan Andrew Chi, and Pranav Ra- jpurkar. 2022. Improving radiology report generation systems by removing hallucinated references to non- existent priors.
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when to look: Adaptive attention via a visual sentinel for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 3242-3250. IEEE Computer Society.
Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017, Self-critical se- quence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recog- nition, CVPR 2017, Homolulu, HI, USA, July 21-26, 2017, pages 1 1179-1 195. IEEE Computer Society.
Yasuhide Miura, Yuhao Zhang, Emily Tsai, Curtis Lan- glotz, and Dan Jurafsky. 2021. Improving factual completeness and consistency of image-to-text radi- ology report generation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5288-5304, Online. Association for Computational Linguistics.
Michael Schlichtkrull, Thomas N Kipf. Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In European semantic web confer- ence, pages 593-607, Springer.
Akshay Smit, Saahil Jain, Pranav Rajpurkar. Anuj Pa- reek, Andrew Ng, and Mathhw Lungren. 2020. Com- bining automatic labelers and expert annotations for accurate radiology report labeling using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1500-1519, Online. Association for Computa- tional I. Linguistics.
Feiteng Mu and Wenjie Li. 2022. Enhancing text generation via multi-level knowledge aware reason- ing. In Proceedings of the Thisty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 43 4310-4316. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Jianmo Ni, Chun-Nan Hsu, Amilcare Gentili, and Julian embed- dings for reporting abnormal findings on chest X-rays. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 1954-1960, Online. Association for Computational Linguistics.
Xiao Song, Xiaodan Zhang, Junzhong Ji, Ying Liu, and Pengxu Wei. 2022. Cross-modal contrastive atten- tion model for medical report generation. In Proceed- ings of the 29th International Conference on Com- puteational Linguisticx, pages 2388-2397, Gyeongju, Republic of Korea. International Committee on Com- putational Linguistics.
McAuley. 2020. Learning
Toru Nishino, Yasuhide Miura, Tomoki Taniguchi. Tomoko Ohkuma, Yuki Suzuki, Shoji Kido, and Noriyuki Tomiyama. 2022. Factual accuracy is not enough: Planning consistent description order for radiology report generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural
Tim Tanida, Philip Müller, Georgios Kaissis, and Daniel Rueckert. 2023. Interactive and explainable region- guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7433-7442.

---

Yi-Lin Tuan, Sajjad Beygi. Maryam Pazel-Zaramn Qiaozi Gao, Alessandra Cervone, and William Yang Wang. 2022. Towards large-scale interpretable knowledge graph reasoming for dialogue systems. In Findings of the Association for Computational Lin- guistics: ACI. 2022, pages 383-395, Dublin, Ireland. Association for Computational Linguisties.
5 lists the observation distributions annotated by CheXber((Smit et al., 2020) in the train/valid/test split of two benchmarks and Table 7 shows the progression distributions in our experiments.
#Observation
MIMIC-ABN
MIMIC-CXR
No Fouding/POS
5002/32/22
64,671/514/229
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS 17. page 6000 6010, Red Hook, NY, USA. Curran Associates Inc.
No FoudingNNEG
66,784/514/784
106,133/1.61//43
CardiomegatyIPOS
16,312/118/244
70,561/514/1 1401,602
CardionegalyINEG
804/4/8
85,448/7 14/801
Pleural EBBusion/POSS
10,502/80/186
56,972/477/1 1,379
Pleural Fijusiug/NBG
1. 1,94//18/24
170,98911,100/ 1,0/6,663
Preumothoras/OOS
1,452/24/4
8,707/6/1106
Preumnhoonu/NFG
1,792/10/26
190,356/1 4495/22333
Entarged Card
5,202/40/90
49, 19,806413/1 3/1 140
Entarged Card/NEGG
1,194/10/14
129.360V 209.3601.000/868
Oriol Vinyals. Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In CVPR, pages 3156-3164. IEEE Computer Society.
Consolidaatio/POS
4. 4,104/36/96
14,449/1 19/384
ConsolidationNEG
3,334/20/34
97.197/788964
Lung OpacityPOS
22,598/166/356
67,714/497/
Lang Opacity/NEG
748/10/4
8,15//3//225
Fracture/POS
4,458/2/776
11,070/59/222
Zhanyu Wang, Lingqiao Liu, Lei Wang, and Luping Zhou. 2023. Metransformer: Radiology report gener- ation by transformer with multiple learnable expert tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11558-11567.
Fracture(NEG
330/0/0
9,632/72/53
Lung Lesion/POS
5,612/54/112
11,717/123300
Lung Lesion/NEG
120///2
1,972/21/11
Edema/POS
8,704/76/168
33,034/257/899
EdemaNNG
1,898/16/32
51,639/409/669
Atelectasis/POS
19,132/344/220
68,273/515/1,210
Atelectasis/NEG
116/200
563/5/9
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue. Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Support Devices/POS
9. 9886/58/196
60,455/451V1,358
Support Devices/NBG
394/0/10
1,08177/11
Preumonia/POS
17.826/138/260
23,945/184/503
Preumomia/NECi
3,226/22/34
21,976/165/41
Pleural Other/POS
2,850/30/62
7,296/70/184
Pleural OtheriNEG
8/0/0
63/0/)
Table 5: Observation distribution in train/valid/test split. of two benchmarks. Enlarged Card. refers to Enlarged Cardiomediastium.
A.2 Spatial and Temporal Entity
Joy T Wu, Nkechinyere Nneka Agu, Ismini Lourentzou, Arjun Sharma, Joseph Alexander Paguio, Jasper Seth Yao, Fdwward Christopher Dee, William G Mitchell, Satyamanda Kashyap, Andrea Giovannini, Leo An- thony Celi, and Mehdi Moradi. 2021. Chest im- agenome dataset for clinical reasoning. In Thirty- fith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Here are some of the spatial entities: healed, frac- tured, healing, nondisplaced, top, size, heart, nor- mal, mediastinum, widening, contour, widened, consolidative, collapse, underlying, developing, fi- brosis, thickening, biapical, blunting. indistinct- ness, asymmetrical, haziness, asymmetric, layering, subpulmonic, thoracentesis, trace, small, adjacent, tiny, atypical, developing, supervening. multifo- cal, correct, superimposed, patchy, and borderline. For temporal entities, we use the same settings of Bannur et al. (2023), which are: bigger, change, cleared, constant, decrease, decreased, decreasing, elevated, elevation, enlarged, enlargement, enlarg- ing, expanded, greater, growing, improved, im- provement, improving, increase, increased, increas- ing, larger, new, persistence, persistent, persisting. progression, progressive, reduced, removal, resolu- tion, resolved, resolving, smaller, stability, stable, stably, unchanged, unfolded, worse, worsen, wors-
Shuxin Yang, Xian Wu, Shen Ge, Shaohua Kevin Zhou, and Li Xiao. 2021. Knowledge matters: Radiology report generation with general and specific knowl- edge. CoRR, abs/2112.150099
A
Appendix
A.1 Observation and Progression Statitics
There are 14 observations: No Finding, Enlarged Cardiomediastinum, Cardiomegaly, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pncumothorax, Pleural Effusson,, Pleu- ral Other, Fracture, and Support Devices. Table

---

NLG Metrics
CE Metrics
Dataset
Model
B-1
B-2
B-3
B-4
MTR
R-L
P
R
Fi
W. Historical Record D
RECAP
0.327
0.183
0.117
0.081
0.124
0.227
0.274
0.372
0.297
RECAP w/o OP 0.300
0.164
0.106
0.072
0.110
0.217
0.281
0.274
0.257
MIMIC
RECAP w/o Obs 0.306
0.173
0.110
0.076
0.114
0.233
0.270
0.288
0.259
-ABN
RECAP w/o Pro 0.295
0.158
0.099
0.070
0.109
0.209
0.249
0.361
0.278
RECAP w/o PrR
0.320
0.177
0.112
0.076
0.121
0.218
0.266
0.377
0.292
RECAP
0.423
0.260
0.170
0.118
0.169
0.279
0.387
0.457
0.400
RECAP w/o OP
0.321
0.196
0.131
0.093
0.130
0.260
0.350
0.238
0.256
MIMIC -CXR
RECAP w/o Obs
0.347
0.213
0.144
0.104
0.141
0.270
0.465
0.293
0.307
RECAP w/o Pro 0.396
0.236
0.151
0.103
0.153
0.266
0.383
0.447
0.395
RECAP w/o PrR
0.420
0.257
0.168
0.115
0.166
0.279
0.386
0.459
0.403
w/o Historical Record DP
RECAP
0.319
0.182
0.116
0.080
0.120
0.223
0.306
0.360
0.306
RECAP w/o OP
0.303
0.171
0.109
0.074
0.110
0.217
0.299
0.302
0.283
MIMIC
RECAP w/o Obs
0.301
0.174
0.114
0.079
0.114
0.231
0.353
0.313
0.282
-ABN
RECAP w/o Pro
0.309
0.171
0.109
0.073
0.115
0.222
0.314
0.360
0.302
RECAP w/o PrR
0.320
0.181
0.116
0.079
0.120
0.225
0,299
0.362
0.302
RECAP
0.427
0.268
0.180
0.128
0.168
0.294
0.378
0.417
0.374
RECAP w/o OP
0.371
0.236
0.164
0.121
0.130
0.260
0.357
0.259
0.268
MIMIC -CXR
RECAP w/o Obs
0.363
0.231
0.161
0.119
0.146
0.291
0.415
0.262
0.277
RECAP w/o Pro
0.406
0.251
0.151
0.103
0.153
0.266
0.364
0.405
0.365
RECAP w/o PrR
0.412
0.257
0.172
0.122
0.163
0.289
0.364
0.414
0.368
Table 6: Ablation results of our model and its variants on progression modeling. RECAP w/ OP is the standard Transformer model, w/o Obs stands for without observation, and w/o Pro stands for without progression.
#Progrresioon
MIMIC-ABN
MIMIC-CXR
we first resize an input image to 256 x 256, and then the image is randomly cropped to 224 224, and finally, flip the image horizontally with probability of 0.5, We select the best checkpoint based on the Macro-F, of abnormal observations at this stage.
Better
929/2/19
14,790/1 10/345
Worse
1,219/6/30
18,083/163/431
Stable
4,114/31/99
41,721/334/1,
Total
6,440/48/137
64,498/535/1,55
Ratio
9%%/8.8%/17%
24% 16/40.66%
Table Progression distribution train/valid/test split of two benchmarks.
Hyperparameter Training Epoch
MIMIC-ABN
MIMIC-CXR
10
5
Dropout Rate
0.1
0.1
ened, worsening and unaltered.
Learning Rate
1c 4
le le-4 4 (64, 128) (1,2,3)
Batch Size
{64, 128)
A.3 Other Implementation Details
Sample Weight (aa))
(1,2,3)
Data Preprocessing. We adopt the preprocessing setup used in Chen et al. (2020), and the mini- mum count of each token is set to 3/10 for MIMIC- ABN/MIMIC-CXR, respectively. Other tokens are replaced with special token [UNK].
Table Selected hyperparameters of Stage training. The final hyperparameters in boldface are tuned on the validation set and others are set empirically.
Implementation Details of Stage Training. As the variant w/o OP and the variant w/o Ohs in Table 4 are not trained in Stage 1, they trained with more epochs (i.e., 10 epochs).
Implementation Details of Stage 1 Training. Ta- ble 8 shows the hyperparameters used in Stage 1 training for two datasets. Note that la; is the weight for observation detection, and the weights of obser- vation classification and progression classification are both set to In addition, two data augmenta- tion methods are used during training. Specifically,
A.4 Other Experimental Results
We show experimental results of observation pre- diction and progression prediction during Stage

---

training in Table and Table 10, respectively.
Dataset
D-F1 A-F1
N-F1
MIMIC-ABN
0.539
0.355
0.126
MIMIC-CXR
0.686
0.428
0.759
Table Experimental results of observation prediction after Stage training. D-F,, A-Fi, and N-F, denote the of observation detection, abnormal observation pre- diction, and normal observation prediction, respectively.
Dataset
Better
Worse
Stable
Macro
MIMIC-ABN
0.286
0.468
0.934
0.563
MIMIC-CXR
0.389
0.455
0.896
0.580
Table 10: Experimental results of progression prediction (Fi) after Stage training.
Observation
F
Enlarged Card.
0.323
0.589
0.417
Cardiomegaly
0.585
0.836
0.689
Lung Opacity
0.489
0.499
0.494
Lung Lesion
0.265
0.011
0.075
Edema
0.562
0.587
0.574
Consolidation
0.285
0.233
0.256
Pneumonia
0.242
0.444
0.313
Atelectasis
0.126
0.800
0.556
Pneumothorax
0.265
0.167
0.205
Pleural Effusion
0.691
0.781
0.728
Plcural Other
0.184
0.050
0.078
Fracture
0.155
0.081
0.107
Support Devices
0.720
0.660
0.689
No Finding
0.265
0.429
0.327
Macro Average
0.389
0.443
0.393
Table 11: Experimental results each observation after
Stage training.


Reviews:
[{"review_id": "XDIqz0LEfQ", "reviewer": "Reviewer_Uawd", "summary": "", "questions": "Line 211-212, are there gold labels available for training this task?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable feedback on our work regarding the motivation and effectiveness of introducing longitudinal information. \n\nHistorical records are difficult to collect. Patients are usually first diagnosed with some diseases, and then they will be asked to attend a follow-up check to track their conditions. However, patients with abnormal conditions are the minority in most cases. As a result, studies of patients with historical records become even rarer (i.e., data sparsity). In addition, introducing all the historical records would be time/space-inefficient regarding the performance gain (i.e., efficiency). Considering these two factors, we only use MIMIC datasets for experiments and introduce the last visit in our framework, but we think this could be a good chance for better visualization of progression, and we will conduct related experiments later for a better understanding of such temporal information. \n\nBecause of the data sparsity, there are only a few datasets available for progression modeling and the MIMIC collection is one of them. We hope our work can encourage more research related to this topic and motivate the research community to collect and release other kinds of notes and data.\n\nRegarding the performance of patients who have recent visits, there are two tables (i.e., Table 3 on page 6 and Table 7 on page 13) related to such results (i.e., temporal modeling performance).  Since most baselines did not release their code publicly, we can only compare several models. \n\nFor the question about labels for optimizing progression prediction, we adopt labels provided by Chest ImaGenome, which is also extracted from the MIMIC-CXR dataset and can be assessed at https://physionet.org/content/chest-imagenome/1.0.0/", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Good motivation for using longitudinal data with disease progression information to generate radiology reports. This can potentially be used to generate other types of reports or other clinical NLP/NLG tasks.\n\nThe comparison experiments and ablation studies show promising results of the proposed method, especially the benefit of considering information from the last visit of a patient. The examples are also insightful.\n\nThe paper is overall well-written.", "reasons_to_reject": "The experiments are restricted to the MIMIC dataset only. It is unclear how this method may perform on a different dataset or different types of notes.\n\nThe model only considers the most recent last visit of the patient. What if more historical visits are considered?", "questions_to_authors": "Line 211-212, are there gold labels available for training this task?", "paper_topic_and_main_contributions": "The paper proposes a method RECAP to leverage the data of the last visit and current visit of a patient to generate radiology reports so that the disease progression can be considered in the modelling. RECAP has two stages of training. The model was evaluated on MIMIC-CXR and MIMIC-ABN. Both ablation studies and comparison studies show promising results of the proposed method.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "It can be insightful to demonstrate evaluation metrics on a subset of MIMIC-CXR, where patients have recent visits, i.e. $D^p \\neq \\emptyset$. This may help demonstrate the advantage of using disease progression information better."}}, {"review_id": "KKRUhSyin1", "reviewer": "Reviewer_6ocQ", "summary": "", "questions": "- Is there a reason you didnt use the bertscore?\n- How does your model scores on F1Radgraph? [1]\n- It seems like the Progression Graph is used as input to the model. Could you please clarify which radgraph or label information (or any information originating from the reports) is predicted and which information is used as input to the model? Or more broadly, could you clarify which information you need at test-time to generate a radiology report?\n\n[1] Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards Jean-Benoit Delbrouck | Pierre Chambon | Christian Bluethgen | Emily Tsai | Omar Almusa | Curtis Langlotz, EMNLP22", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We genuinely appreciate the time and effort that you have dedicated to providing valuable feedback on our manuscript, especially on the aspect of spatial/temporal modeling. We understand the concerns about the model's complexity and will improve the presentation for better comprehension. We will include the EMNLP'22 paper for discussion in the related works section.\n\nWe want to emphasize that there is no information/label leakage in our framework. The progression graph is built purely based on the training corpus (i.e., statistical information). Here are the answers to the questions:\n\nFor Q1: We follow previous works and only adopt word-overlap metrics (e.g., BLEU) for evaluation. A typical radiology report consists of several short sentences with each corresponding to one or two observations. Instead of assessing the overall quality of it, which the BERTScore will surely do, we pay more attention to phrases (e.g., n-grams). In this case, we do not adopt it for evaluation. However, we notice that BERT could be a good indication of report quality and will consider including such a score for better comparison.\n\nFor Q2: For the MIMIC-CXR dataset, our model scores ~29% F1 on RadGraph, which demonstrates a substantial improvement over the baselines trained with the NLL objective presented in [1]. Actually, we conduct all our experiments with evaluation based on RadGraph and will include more details in the updated version.\n\nFor Q3: Basically, at test time, the prerequisites are:\n\n(1) historical records (if exist, i.e., prior image and prior report), \n\n(2) the current image, and \n\n(3) the progression graph, or statistical information extracted from the training corpus (i.e., entities from RadGraph, and triples mined using PMI). \n\nThe progression (sub-)graph is built on the fly after we predict the observations and progressions based on the images. There is no other label from the ground truth in our case. Here are some steps in generating a report:\n\n(1) observation/progression prediction based on historical records + the current image, \n\n(2) progression sub-graph retrieval based on the result of step (1), and\n\n(3) report generation based on the results of steps (1) and (2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The Spatial/Temporal component of radiology reports has long been overlooked. This work is a great step toward this direction. \n- Extensive comparison to prior work\n- Use of RadGraph to compute a semantic graph and progression\n- Substantial improvements on CE Metrics.\n- Results are reported in a wide range of metrics\n\n", "reasons_to_reject": "- The model is quiet complex and difficult to understand\n- It seems some information from the ground-truth reports are used as input to the model (at test-time). I might be wrong about this one (see questions for the authors)", "questions_to_authors": "- Is there a reason you didnt use the bertscore?\n- How does your model scores on F1Radgraph? [1]\n- It seems like the Progression Graph is used as input to the model. Could you please clarify which radgraph or label information (or any information originating from the reports) is predicted and which information is used as input to the model? Or more broadly, could you clarify which information you need at test-time to generate a radiology report?\n\n[1] Improving the Factual Correctness of Radiology Report Generation with Semantic Rewards Jean-Benoit Delbrouck | Pierre Chambon | Christian Bluethgen | Emily Tsai | Omar Almusa | Curtis Langlotz, EMNLP22", "paper_topic_and_main_contributions": "- Authors propose RECAP, which can capture both spatial and temporal information for generating precise and accurate free-text reports.\n- Authors construct a disease progression graph containing both observations and fine-grained attributes which quantify the severity of diseases. Then, we devise a dynamic disease progression reasoning (PrR) mechanism to select observation/progression-relevant attributes.\n- Authors conduct extensive experiments on two publicly available benchmarks, and experimental results demonstrate the effectiveness of our model in generating precise and accurate radiology reports.\n\nFor the experiments, authors conduct their research on two datasets with an excellent comparison to prior work. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "In the related work its seems:\nImproving the Factual Correctness of Radiology Report Generation with Semantic Rewards Jean-Benoit Delbrouck | Pierre Chambon | Christian Bluethgen | Emily Tsai | Omar Almusa | Curtis Langlotz, EMNLP22\nfalls into the category of \"Other works focus on improving the clinical accuracy of the generated reports\"\n"}}, {"review_id": "1ac474Yj6F", "reviewer": "Reviewer_7aGE", "summary": "", "questions": "See the reason to reject section", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for R3’s review. After carefully reading R3’s comments, we find that R3 might have some misunderstanding about our experimental setting and might have overlooked some parts of our experiment section. More specific responses to R3’s concerns are as follows. \n\n$\\textbf{Q1. Fails to outperform existing works:}$ The major objective of this paper is to generate precise and accurate radiology reports, so the CE metrics, which measure clinical accuracy, are our primary focus. As for the NLG metrics, they mainly calculate the similarity between the generations and the references in terms of n-gram accuracy. They might reflect the language quality of the generated reports to some extent, but they do not guarantee accuracy and reliability, which is the most critical concern in the medical field. That being said, our method still achieves competitive results on these dimensions. R3 mentioned that the B-4 of our method is lower than the one of ViT-Transformer, but in fact, there is only a 0.001 performance gap between them. Since there have already been over 10 baselines on the MIMIC-CXR dataset, including the most state-of-the-art ones, we do not put ViT-Transformer in the table.\n\n$\\textbf{Q2. What if the patients are first-time visitors? The authors need to evaluate the proposed approach on new patients and old patients, respectively:}$ R3 might skip reading some important parts of the experiment section. In fact, we do provide the related experimental results on temporal modeling (i.e., results of “old patient”) in Table 3 and Table 7. The TEM score reflects how these models can generate progression-related information.\n\n$\\textbf{Q3. Fairness of the experiment setting:}$ R3 might have some misunderstanding about our experimental setting. We want to point out that all the samples are included in the MIMIC-CXR and MIMIC-ABN datasets. Previous baselines treated and learned all the samples (including historical records) independently, and it does not mean that the historical records are unseen to these baselines. In other words, all the samples are seen to the baselines. Instead,  the overall setting is fair and proper, because the relations between consecutive studies play a vital role in generating precise and accurate reports, and again, this meets the objective of our paper.\n\n$\\textbf{Q4. Why not include one more baseline (DeltaNet) published at COLING 2022:}$ We will consider including DeltaNet for discussion in the next version of our paper. Nonetheless, we want to point out that our current version has already considered over 10 baseline methods, including the most state-of-the-art ones published at top conferences, so we believe the comparisons in our experiments are already very sufficient. Moreover, the method mentioned by R3, which focuses on leveraging historical records or retrieved samples as contextual information for better COVID-19 report generation, is not closely related to our work that focuses on precise and accurate radiology report generation.\n\n$\\textbf{Q5. Intermediate results should be evaluated:}$ We will include more qualitative analysis in the later version.\n\n$\\textbf{Q6. Should include the IU dataset:}$ The IU dataset is not a suitable testbed for our method. The reasons are as follows: (1) The IU dataset does not contain historical records of patients, and (2) this dataset is too small for training a report generation system and for evaluating the performance of a model. In our current paper, we have conducted experiments on two commonly-sued datasets, which have already provided sufficient experimental results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Medical report generation is a critical clinical task which can relieve radiologists from the heavy workload.\n\n2. Using historical reports and mining dissease progresses are useful for report generation.", "reasons_to_reject": "1. The proposed approach fails to outperform existing works. For example, in Table 1, the B-4 of proposed approach is lower than the basic baseline ViT-transformer on MIMIC-ABN. Why the ViT-transformer is not evaluated on MIMIC-CXR data set.\n2. What if the patients are the first time visitors without historical reports. The authors need to evaluate the proposed approach on new patients and old patients respectively.\n3. The experiment setting is not fair. For the proposed approach, the historical reports of patients are used to generate reports for current input. While these data are unseen to baseline works. These historical reports should be added into the training data set of baselines.\n4. One existing work which also includes historical reports in modeling should be referenced and discussed.\nDeltaNet: Conditional medical report generation for COVID-19 diagnosis, Coling 2022.\n5. Since the proposed approach targets to mine the progress of diseases to generate better results. Such intermediate results (the disease progress) should be evaluated in experiments.\n6. The IU data set should be included in experiments.", "questions_to_authors": "See the reason to reject section", "paper_topic_and_main_contributions": "In this paper, the authors target the medical report generation problem. Different from existing works, the authors propose to include historical reports in modeling and try to mine the disease progresses by comparing the historical reports with current input. The derived progressions are further used to improve report generation. The authors evaluate the proposed approach on the public MIMIC data set.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "DeltaNet: Conditional medical report generation for COVID-19 diagnosis, Coling 2022"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a method for generating radiology reports by incorporating both the current radiology image and the patient's image from a previous visit. The proposed technique employs a two-stage training approach: initially, the model predicts disease observations and progressions, which are utilized to construct a disease progression graph. Subsequently, a decoder is trained in the second stage to formulate the report based on the data obtained from the first stage.

Reviewers largely concur that the structural modeling of spatial and temporal progression aspects in radiology images is both innovative and intuitive. Robust experiments demonstrate a marked improvement over existing baselines, and in-depth ablation studies have been conducted. Nonetheless, there are reservations concerning the complexity of the proposed method and its potential adaptability to datasets other than MIMIC-CXR.
Decision: Accept-Findings