Normal-Abnormal Decoupling Memory for Medical Report Generation
Authors: Guosheng Zhao
Yan Yan
Zijian Zhao
Venue: EMNLP 2023 Findings (2023)
Keywords: Medical Report Generation
Normal-Abnormal Decoupling
Semantic Extraction
Abnormal Mode Memory
TL;DR: We propose a normal-abnormal semantic decoupling network based on abnormal pattern memory, which optimizes visual extraction through abnormal semantics and alleviates the noise problem in medical reports.

Abstract: The automatic generation of medical reports plays a crucial role in clinical automation.  In contrast to natural images, radiological images exhibit a high degree of similarity, while medical data are prone to data bias and complex noise, posing challenges for existing methods in capturing nuanced visual information. To address these challenges, we introduce a novel normal-abnormal semantic decoupling network that utilizes abnormal pattern memory. Different from directly optimizing the network using medical reports, we optimize visual extraction through the extraction of abnormal semantics from the reports. Moreover, we independently learn normal semantics based on abnormal semantics, ensuring that the optimization of the visual network remains unaffected by normal semantics learning. Then, we divided the words in the report into four parts: normal/abnormal sentences and normal/abnormal semantics, optimizing the network with distinct weights for each partition. The two semantic components, along with visual information, are seamlessly integrated to facilitate the generation of precise and coherent reports. This approach mitigates the impact of noisy normal semantics and reports. Moreover, we develop a novel encoder for abnormal pattern memory, which improves the network's ability to detect anomalies by capturing and embedding the abnormal patterns of images in the visual encoder. This approach demonstrates excellent performance on the benchmark MIMIC-CXR, surpassing the current state-of-the-art methods.

Normal-Abnormal Decoupling Memory for Medical Report Generation
Guosheng Zhao', Yan Yan', Zijian Zhao 1+
School of Control Science and Engineering, Shandong University
DDeparmeent of Computer Science, Illinois Institute of Technology. , wwaizzaazzzzaaeee edu.cn, cn, yyan34@iit. edu
Abstract
Ieport:
The automatic generation of medical reports plays crucial role in clinical automation. In contrast to natural images, radiological images exhibit a high degree of similarity, while med- ical data prone to data bias and complex noise, posing challenges for existing methods capturing nuanced visual information. address these challenges, we introduce novel normal-abnormal semantic decoupling network that utilizes abnormal pattern memory. Differ- ent from directly optimizing the network using medical reports, we optimize visual extraction through the extraction of abnormal semantics from the reports. Moreover, we independently learn normal semantics based on abnormal se- manties, ensuring that the optimization of the visual network remains unaffected by normal semantics learning. Then, we divided the words in the report into four parts: normal/abnormal sentences and normal/abnormal semantics, op- timizing the network with distinct weights for each partition. The two semantic components, along with visual information, are seamlessly integrated to facilitate the generation of pre- cise and coherent reports. This approach mit- igates impact of noisy normal semantics and reports. Morcover, we develop novel encoder for abnormal pattern memory, which improves the network's ability to detect anoma- lies by capturing and embedding the abnormal patterns of images in the visual encoder. This approach demonstrates excellent performance on the benchmark MIMIC-CN surpassing the current state-of-the-un !ethods.!.
cardact nomel, medartie
Stnormat: hone
purmonnry
sppware
sormal surdoomed ustinal nomall pu purmorary
consolissonn ploural offusion. thorool spine
Figure I: Radiological images, medical reports and normal abnormal semantics.
been achieved in this field (Wang et al., 2022c,a,b; Kong et al., 2022; Qin and Song, 2022). Nonethe- less, generating radiological reports remains formidable undertaking. However, several aspects require further exploration, such as: 1) the pres- ence of pronounced visual and textual data biases. Within medical data, normal images predominate the section containing deseriptions of normal utter- ances in the reports. 2) Unlike natural images, radi- ological images frequently exhibit high similarity, posing challenges in extracting fine-graine fine-grained visual details. The presence of noise in reports impacts network optimization. The descriptions of similar semantics may vary among different doctors. For instance, in Figure la, "cardiac" and "mediastinal" are used, whereas in Figure 1b, "cardiomediasti- nal" is employed. Moreover, is plausible for the syntax of the same description to exhibit signifi- cant variation. Some normal descriptions may not be mentioned due to physicians practices. For in- stance, Figure lb, "consolidation" and "effusion" are referenced. whereas in Figure 1a, they are disre- garded. These sources noise significantly impact network optimization, yet only limited research has been dedicated to addressing this issue.
1 Introduction
Report writing is critical responsibility for radi- ologists. Automatic report generation is clinically significant it alleviates the workload of physi- cians. Recently, substantial adyancements have
Our code is available at httpps://gihhh ctt.bb..co///zkk/////
NADM
Correspondiig authors.

---

To address the aforementioned limitations, we propose semantic decoupling network based on abnormal pattern memory for generating reports. In medical reports, the fundamental semantic struc- ture comprises three components: anatomy, obser- vation, and judgment. To mitigate the impact of noise in the reports, we optimize the visual extrac- tor using abnormal semantics. This is based on the observation that abnormal semantics relatively consistent, despite the occasional omission of nor- mal descriptions and variations in similar semantic descriptions across reports from different doctors. We utilize RadGraph (Jain et al., 2021) to extract the semantics, focusing solely on the core semantic components while disregarding others. We em- ploy visual encoder based on higher-order feature interaction attention, enhancing the perception of fine-grained features through a bilinear pool (Kim al., 2016), which has been shown to be effective in fine-grained classification (Lin et al., 2015). Fur- thermore, drawing inspiration from discrete varia- tional autoencoder (Van Den Oord et al., 2017), we store the abnormal modalities of images and incor- porate higher-order feature interaction processes to augment anomaly perception. During the re- port generation phase, fuse the two modalities, semantic and visual, jointly guide the report gen- eration process. In the training phase, we employ convolution for image encoding and grid feature generation. Moreover, we utilize a variational self- encoder to capture anomaly modality memory and introduce two semantic branches atop it to predict both abnormal and normal semantics of the image during the visual attention phase. We argue that different words in the report possess varying lev- els of importance, with abnormal semantics being intuitively more significant than normal semantics. However, previous approaches (Jing et al., 2017; Yuan et al., 2019; Yin et al., 2019; Najdenkoska et al., 2021; Chen et al., 2021b; Wang et al., 2022c) tend to treat all report words equally. Therefore, we divide the reported words into four categories: normal/abnormal sentences and normal/abnormal semantics. Subsequently, automatically learn distinet weighting parameters for optimization. In conclusion, our contributions are outlined as fol- lows:
effectively mitigating the impact of noise and data bias present in reports.
(2) We develop a visual encoder based on anomaly pattern memory, which enhances anomaly perception by explicitly memorizing abnormal pat- terns and incorporating them during higher-order interaction in the visual processing phase.
(3) Our approach shows promising performance on MIMIC-CXR over multiple state-of-the-art methods.
2 Related Work
Image captioning involves generating relevant tex- tual descriptions topics for given image. Early approaches utilized templates retrieval-based methods for caption generation (Hossain al., 2019). In recent years, encoder-decoder frame- works, primarily based on transformer architec- tures (Vaswani et al., 2017), have been widely em- ployed to generate individual descriptive sentences from images (Cornia et al., 2020; Anderson et al., 2018; Chen et al., 2021a; Xu et al., 2021). No- tably, anchor- frame- frase- methods for title gener- ation have demonstrated accurate text description generation (Xu al., 2021). However, a signifi- cant portion of the available data lacks annotations. To address this limitation, (Fang et al., 2022) pro- posed the use of semantic extraction to enrich in- formation and improve caption generation. Further- more, while most caption generation tasks focus on shorter descriptive discourse, existing approaches for very long utterances often overlook issues lated to data bias Keriazi al., 2018). The image captioning task has a more explicit objec- tive, but the presence of complex semantic contexts, judgmental utterances, and noise problems med- ical reports introduces additional challenges.
Medical report generation, which falls under the category of image captioning. predominantly adopts an encoder-decoder framework. Various attention mechanisms and hierarchical LSTM mod- els have been proposed to generate radiology re- ports (Jing et al., 2017; Yuan et al., 2019; Yin et al., 2019). Another approach involves constructing graphs based on medical knowledge and utilizing graph convolutional neural networks to enhance feature extraction (Zhang al., 2020). To ad- dress the issue of modal bias, (Najdenkoska et al., 2021) introduced a set of latent variables as top- ics to guide sentence generation by aligning image and language patterns the latent space. Modal
(1) We introduce network for report genera- tion that leverages anomaly semantic extraction. This approach focuses on optimizing the visual ex- traction network solely using anomaly semantics,

---

alignment was further improved through the in- troduction of cross-modal memory network by (Chen et al., 2021b), and later refined by (Wang et al., 2022a) with the proposition of cross-modal prototype-driven network. Additionally, (Liu et al., 2021a) presented framework that leverages both priori and posteriori data to enhance generative reporting. (Wang et al., 2022c) utilized semantic extraction improve generation, they overlooked the problems of noise and data bias prevalent in the reports. In (Jing et al., 2019), the data bias problem is considered, where the noise of normal statements still affects the optimization of the model despite the separate generation of normal/abnormal state- ments.
Den Oord al., 2017), and introduce discrete potential space called the memory codebook 92, where each entry in corresponds to a potential embedding of an image pattern, and N denotes the total number of memorized patterns as a hyperpa- rameter. In the case of an abnormal image, the features extracted by visual convolution search for match within Ω to find the corresponding location of the memory pattern. While the nearest neigh- bor approach is simple way find a match, it poses the risk of pattern collapse (Chen al., 2022). Thus, we adopt a dichotomous matching approach and apply the Hungarian algorithm (Kuhn, 1955) to resolve this issue. When memorizing, we look for unique pattern in the memory space corre- sponding to each image encoding. The Hungarian algorithm is used to find unique pattern match in the memory matrix by treating the image fea- tures with each pattern of the memory matrix as an ensemble matching problem. Specifically, we construct bipartite graph using all local grid fea- tures Vg and pattern embeddings in Ω ensuring that the sizes of Vg and codebook i are matched using 0, and then determine the arrangement of the l elements GN with the lowest allocation cost.
3 The Proposed Method
The proposed architecture for the memory of anomaly patterns the semantic decoupling net- work, consisting of three critical components: a vi- sual extractor, a semantic extractor, and a decoder, where the visual extractor consists of an image en- coder and a visual encoder. The overall structure of the network is illustrated in Figure 2. The visual extractor is responsible for converting images into detailed features with the goal of detecting anoma- lies and generating precise visual representations. The semantic extractor divided into two parts: abnormal semantic extractor and normal seman- tic extraction. The purpose of abnormal semantic extractor is to identify semantics related to aberra- tions from the preceding visual representation and to capture normal semantics while considering the abnormal semantics. The decoder is responsible for processing the visual and semantic characteristics, and guides the report generation by deep fusion.
arg min TEHN
Σ V 2-00l2
(2)
3.3 Abnormal Pattern Enhancement Multi-Head Attention
The general self-attention approach only lever- ages the interaction of first-order features, which presents certain limitations. To achieve more de- tailed feature extraction, we adopt the bilinear at- tention module designed in accordance with (Pan et al., 2020). In the implementation, global features are queries Q, regional features are keys K, and values V. To incorporate abnormal patterns into higher-order interactions, we extend the key and value sets with mnemonics to encode and collect abnormal patterns from visual convolution. Our keys and values can be defined as K [K, ?2] and V [V, where [.] denotes splicing and Q is all abnormal embeddings of the memory. Then low-rank bilinear pooling is performed ob- tain the joint bilinear query-key Bg and query- (1) value By by B = a KW @ a (W*Q) and By (W Vo (W Q) W., wy. W and W are learnable parameters. o denotes ReL.U unit, and @ represents Hadamard Product. We then compute the attention on the space and channels by
3.1 Image Encoder
To extract visual information from the images, we extracted features using the ResNet50 pre-trained by Bio (Boecking et al., 2022). Given an image I, grid features are obtained by convolution V/2 = ResNet50(T), and combined with global features
Vg with position encoding Epos to compose the visual information Ve.
V. = Concat (Vg, Vi) + Eppos
3.2 Abnormal Mode Memory
To prevent anomalies from being overwhelmed, we are inspired by variational autoencoder (Van

---

Visual Encoder
Concat
Semantic Extractor
Outou Resort
Add Nom
Decader
Softman
Classticrtin
Casssicntion
APE- MHA
Sum
Add & harm
Entbadi +
Nor
Embead
APE MFE-
Food Forward
Feed
Enbeo
Embod
Norm
Add & Norm
MHA
Fusion
Embed
NHV
Ads & Nor
Fusian
Image
MHA
MHH
Target Report
datach
The overall structure of the model includes three modules: visual extractor, semantic extractor, and decoder. visual extractor module a combination an image encoder and visual encoder. images are encoded to obtain embedded features. Fine-grained information is obtained through high-level interactions by the visual encoder. The semantic extractor module decouples normal/abnormal semantics by the two branches. Finally, the report generated through fusion decoding.
Figure 2:
projecting each bilinear query key representation into the corresponding attention weights through two embedding layers, and then normalize using the softmax layer to introduce the spatial bilinear ( i BB) ,then use another lin- ear layer to map B3 from De dimension to di- mension to obtain the ppatial-wise attention weight Meanwille, we perform squeeze-excitation operation (Hu et al., 2018) generating channel attention 3. Sigmoid (W. We are learnable parameters and B is an average pooling of B The basic layers that make up the visual encoder are as follows:
3.4 Semantic Extractor
To extract semantic information, treat it as multi-label classification process, and employ Rad- Graph (Jain et al., 2021) extract pseudo-medical concepts as ground-truth for this task. The trans- former attention block is used to process the in- termediate features V generated by the image encoder. To facilitate multi-label perception (MLP) network prediction, the [CLS] token is added to the image features, and its corresponding output repre- sents the concept token. The vocabulary of concept tokens the same as that used for the headings. Importantly, we predict concepts the token level rather than the label level, to enable the multimodal decoding module to directly employ the top-K to- kens for guidance reporting. The extraction process of anomaly semantics involves the following:
attention B;
APE(V.) Be O a.
V AddLN (FFN (AddLN (V))(
QKT
Att K, V) Softmax
(5)
(4) Where Vj. Vi represents the i-th layer of visual feature extraction. FFN denotes a fully connected feed-forward network, and AddLN represents the composition of a residual connection and a normal- ization layer.
Vd
MHA [Att] A
(6)
fn MHA (V7", VJ", VVT)
(7)
S. - AddLN (FFN (AddLN (fr)))
(8)

---

o - Emb(MLP((.))
(9)
ht. respectively, subsequently, the previous con- textual, semantic, and visual information fused and encoded using Sigmoid to obtain the gating g. Taking the state at time as an example, the equa- tion is as follows:
Where So the generated intermediate semantic prediction, S denotes the output corresponding to the classification flag [CLS], MLP represents the final classification layer, Emb represents the embedding layer, Att represents the attention layer,
h; -aak--Ml A(,,, h,, h4)
(15)
and S represents the top-k abnormal semantic concepts encoded at the end.
h" MHA VG, VD) +HA Sy,
After obtaining the abnormal semantics, con- sider the acquisition of normal semantics con- ditional probability problem under the condition of abnormal semantics. Due to the complexity of the normal semantics, does not serve as a basis for optimal visual extractor, so its update is only related to the normal semantic extractor. We add represeinng the prediction of the normal semantics. The extractor process is as follows:
(16)
g =
(17)
h AddLN + +(1 g)h)))
(18)
3.6 Objective Function
Memory loss: the anomaly memory codebook serves the purpose of storing various anomaly in- formation and interacting with image information. It is important to note that the learning process of
fn = A (S S ()
(10) the codebook should not affect the image coding, We want to use orthogonality to make the codebook (11) learn as much information as possible about dif- ferent modes. We use Sg to represent the gradient (12) cutoff and E to denote the unit matrix. The loss
= =HA( V7")
S, AddIN (FFN (AdaTN (f)))
SK = Emb(MLP((S))
(13) function is defined as follows:
S = Concat(St SK)
(14)
Lm
sg V. l|g
(V2) 12 Ω E
(19)
Where Sp is the generated intermediate semantic prediction, S% denotes the output corresponding the classification flag [slot] SK represents the top-k normal semantic concepts encoded at the end and S represents the final semantic output.
Semantic loss: in the semantic extraction task. our goal is to extract normal and abnormal seman- tics, which entails multi-label classification pro- cess. However, the semantic distribution in medical reports often heavily imbalanced, which can pose a challenge for standard multi-label classification approaches. To address this issue, we employ an asymmetric loss (Fang et al., 2022; Ridnik et al.. 2021; Liu et al., 2021b: bb; Ridnik et al., 2021), which has shown good performance in handling unbal- anced problems.
3.5 Decoder
We adopt the decoder structure introduced in (Li et al.. 2022), which leverages the rich visual to- kens Vj obtained from the visual encoder and the semantic tokens Sy extracted from the semantic extractor. The decoder combines both visual and semantic information provide guidance for gen- erating accurate and coherent sentences. Formally, let R = (ro, ri...., r denote report of words generated. The decoder takes the report R as input and learns to predict the next word automatically and regresssively, conditional on the visual V and semantic tokens Sy. We implemented the decoder Nd-stacked transformer block. It consists of masked multiheaded attention layer Mask-MHA for modeling the overall textual context hi of the previously generated word h, and two crossed mul- tiheaded attention layers that cross the visual and semantic tokens to trigger the generation of the cue
C. = asym (Pa. asym (Pr. vn)
(20)
Pa. P. denotes the predicted probability distribu- tion corresponding to the abnormal/normal seman- tics, respectively, and Na, Un denotes its correspond- ing label.
For the generated reports, use minimizing the negative log-likelihood of the given image features, semantic features train the model parameters. Although our positive anomaly seman- tic decoupling avoids the effect of noise on feature extraction, the effect noise is still unavoidable in
Reports loss:

---

Table 1: The performances our model compared with baselines on MIMIC-XRR dataset. The best results are highlighted in bold.
Dataset
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
S&T
0.299
0.184
0.121
0.084
0.124
0.263
AdaAtt
0.299
0.185
0.124
0.088
0.118
0.266
TopDown
0.317
0.195
0.130
0.092
0.128
0.267
R2Gen
0.353
0.218
0.145
0.103
0.142
0.277
PPKED
0.360
0.224
0.149
0.106
0.149
0.284
M2TR
0.378
0.232
0.154
0.107
0.272
MIMIC-CXR
R2GenCMN
0.353
0.218
0.148
0.106
0.142
0.278
XProNet
0.344
0.215
0.146
0.105
0.138
0.279
GSKET
0.363
0.228
0.156
0.115
0.284
R2GenRL 0.381
0.232
0.155
0.109
0.151
0.287
MSAT
0.373
0.235
0.162
0.120
0.143
0.282
Ours
0.402
0.258
0.179
0.130
0.155
0.289
the decoding phase of report generation. Thus, mea- suring the importance of different words with dif- ferent weights to mitigate the effect noisy normal descriptions makes the network focus more on crit- ical information. We use RadGraph to classify the vocabulary in the report into normal/abnormal sen- tences, and normal/abnormal semantics. Inspired by the multitasking approach (Kendall al., 2018; Liebel and Körner, 2018), the importance of words is measured using uncertainty. Following (Liebel and Körner, 2018), the four weights are learned automatically.
Table 2: The results of clinical efficacy (CE) metrics on the MIMIC-CRR dataset. The best results are high- lighted in bold.
Model
Precision
Recall
F1-Score
S&T
0.249
0.203
0.204
AdaAtt
0.268
0.186
0.181
TopDown
0.320
0.231
0.238
R2Gen
0.333
0.273
0.276
R2GenCMN
0.334
0.275
0.278
R2GenRL
0.342
0.294
0.292
Ours
0.417
0.413
0.415
T
chest x-ray images. In our experiments, for fair comparison, we used the official segmentation of MIMIC-CXR after the work (Chen al., 2020) and excluded all sample reports that did not contain description of medical observations. We only focus on the finding part of the medical report.
C,
log Pe (Re
Ret, Sp)
(21)
C, Σ m==
22 1. + log(1 + am)
(22)
m
Rs denotes the report information at time Vj denotes the image information, Sy denotes the se- mantic information,o,, denotes the different uncer- tainty parameters. LT" denotes category m reports loss. The final losses are as follows:
4.2 Evaluation Metrics
We used the widely used BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), and ROUGE-L (Lin, 2004), computed by the standard evaluation toolkit In particular, BLEU and ME-
L Lm Lv Lr
(23) TEOR are proposed for machine translation eval- uation. ROUGE-L was designed to measure the quality of the summary. For report generation, the predictive accuracy of the disease should also be considered. Therefore, we used clinical efficiency (CE) metrics to express the performance of our model. We used CheXpert (Irvin et al., 2019y to
4 Experiment Settings
4.1
Datasets
We conducted numerical experiments on MIMIC- CXR² (Johnson et al., 2019). MIMIC-CXR is the largest radiology dataset to date, including 473,057
hhttps///githu thub. com/ hww.yyyl//ccco-cptiion hhttps://gi tttp///gggtthhheee comm/MIT-LLPP/iiiicco///tt-eee master/txt/chexper.
8/

---

Table The performances our model compared with the model without different proposed modules on MIMIC CXR dataset. The w/o is the abbreviation of without. The best results highlighted in bold.
MIMIC-CXR
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
Ours
0,402
0.258
0.179
0.130
0.155
0.289
w/o Bio
0.405
0.255
0.171
0.121
0.156
0.286
w/o SE
0.403
0.254
0.171
0.120
0.156
0.284
w/o APE
0.396
0,250
0.169
0.119
0.161
0.284
w/o WE
0.398
0.251
0.172
0.124
0.158
0.289
in both metrics and obtain SOTA results. This proves the superiority of our method.
5.2 Ablation Study
In this section, we performed ablation experiments on the MIMIC dataset to investigate the contribu- tion of each component of our method. Table 3 shows the experimental results. Bio denotes the initialization of the ResNet network weights us- ing BioViL (Boecking al., 2022). DE denotes the truncated reports loss the visually extracted gradients. SE denotes the semantic extractor struc- ture with normal and abnormal separation. APE denotes abnormal memory extraction structure. WE denotes the report loss weight adjustment. We investigated the following variants:
010
Figure 3: BLEU-4 scores and versus memory size when the model is trained and tested on the MIMIC- CXR dataset. The horizontal coordinate indicates the size of the multiplier the base capacity 49.
extract discase labels from real reports and model predictions calculate precision, recall, and Fl scores.
An important note: the implementation details, dataset splits, preprocessing steps, and additional experiments are described in the supplementary materials.
w/o Bio indicates that the model for ResNet50 randomly initializes the weight parameters and leaves the rest of the structure unchanged. w/o SE indicates that instead using decoupled extrac- tion semantics, a single semantic extractor branch is taken and normal and abnormal semantics are predicted simultancously. w/o APE indicates that instead of using the strueture of APE-MHA, the visual encoder is replaced with the structure of self- attention MHA. w/o WE indicates that instead of weight the report loss, but treat all words in the report equally.
5 Results and Analysis
5.1 Comparison with Previous Studies
To further validate the effectiveness of our method, we compare our proposed method with conven- tional image captioning works, e.g. S&T (Vinyals et al., 2015), AdaAtt (Lu et al., 2017), Top- Down (Anderson et al., 201 20, and the ones pro- posed for the medical domain, e.g. R2Gen (Chen al., 2020), PPKED (Liu al., 2021a), M2TR (Nooralahzadeh al., 2021), R2GenCMN (Chen et al., 2021b), XProNet (Wang et al., 2022a), GS- KET (Yang et al., 2022), R2GenRL (Qin and Song, 2022) and MSAT(Wang et al., 2022c). The results on S&T (Vinyals et al., 2015), AdaAtt (Lu et al., 2017), TopDown (Anderson et al., 2018 8) from (Chen et al., 2020), and the rest of the results were cited from the original paper. Table shows the NLG metries and Table shows the CE metrics. As can be seen, thanks to our good framework of positive anomaly separation and excellent feature extraction ability. We achieve a large improvement
Contribution of each component. We con- ducted an analysis to evaluate the importance of different components our proposed method. We found that all key components play a critical role in achieving high performance. Removing any of these components results in significant perfor- mance degradation. Specifically, if we do not ini- tialize visual extractor with Vil, observe reduction in performance. This underscores the importance of Bio in providing prior informa- tion by aligning images with reports, thereby re- ducing the modal differences. This also suggests that some alignment aspects could potentially be

---

Image
heatmeap
abnormal
normal
gt
biittrral cardiomogaly
intersttial conacsson
vescuar silhouede
pulmorary enlange
meciostnal contours
ettusoon ploural
assccus hlar
obnormallyy
Fontaa and iateral Viewo of the chast were octained.. the heart sae milly
ontarged the mediastinl and hilar
contour are unremariable pulmonary vasculatura is normal
pa and iateral views of the chost. the lunge are dear rocal consolldanon pleurai effusion. there are hawever introased intorttial markings troougout the kings and entarged carciac si sihouelle which unchangger from prior. there acuto cssaous aonormality detected.
pleural heart left cardiar acc lowar
edema unchanged coneckiattion pulmonary
there no pleural effus on onnumetthoras. there is no pulmonary odemo. there is no acute asceous abnommalies
pertillr upper
structures normal
contaur preumathoras
Figure 4: Model visualization, where the heatmap shows the anomalies of interest to the modelred text indicates the abnormal semantics related to the report; blue text indicates the normal semantics related to the report.
integrated with our method to further boost perfor- mance. Additionally, we found that the decoupling structure and APE-MHA are crucial components, and their absence results in the largest performance degradation. This highlights the significance of decoupled learning, which mitigates the negative impact of the uncertainty noise contained in normal semantics predicting abnormal semantics. Fur- thermore, our proposed abnormal modal memory structure enhances the perception of fine-grained images and improves model performance. We also observed that WE enhances the report generation process by optimizing the grouping of words in the report and assigning different weights to them. This allows the decoder to focus more on the key semantics, resulting in better quality reports. Over- all, our analysis demonstrates the effectiveness and importance of each component in our proposed method.
dataset. As shown in the Figure 4, we visualized the attention graph produced by the model for the im- ages, as well as for the top 15 predicted semantics. Abnormal semantics related to the report were high- lighted in red, while normal semantics related to the report were highlighted in blue. Our analysis re- vealed that the model pays close attention to abnor- mal regions in the images, and successfully predicts abnormal phenomena such as heart enlargement. In the prediction of normal semanties, the model gives a rich set of normal semantic candidates, con- taining all normal mentions in real reports, such as consolidation, pleural effusion, and osseous ab- normality. This shows that our model can produce relatively accurate and fluent reports. These results suggest that model is capable of generating relatively accurate and fluent reports.In addition, we visualize the final learned report weights such that X 1zo then À corresponding to abnormal normal semantics, abnormal normal sentences is [1 11.502, 1. 199,,..999...1100 999,9999,,....... 993, 0.691]. From the weights, we can see that the abnormal cases have higher weights and the semantics have higher weights than the nor- mal sentences. This indicates that the main body of the report is the anomaly semantics, which tains more information. It is more important to pay attention to the anomaly semantics when the report is generated.
Impact of memory size. To analyze the effect of memory capacity on the model during anomaly modality extraction, we trained the model using different memory capacities. Since the memory modality is matched one-to-one with the local fea- tures of the image, using a multiple of the image information size of 49 we conducted the experi- ments and the results are shown in Figure 3. It can be seen that as the memory capacity increases, the model performance rises. This is due to the fact that larger capacity allows more image informa- tion be stored and may capture more anomalies, which can be used as prior for subsequent bilinear extraction. However, when greater than a certain threshold, the memory of abnormal modes is suffi- cient, and more capacity instead memorizes unim- portant secondary information introducing noise and causing negative effects.
Conclusions
We propose a new framework for extracting nor- mal and abnormal semantics separately. By de- coupling the semantics, the noise that may exist in the normal semantics is avoided. And we propose the structure of anomaly modality enhancement to enhance the extraction of abnormal features. We classify the reported words into different parts automatically learn the weights to measure the im- portance. This eventually makes better semantic and visual features jointly know to generate process accurate reports.
5.3 Qualitative analysis
To evaluate the performance of our model, we per- formed a qualitative analysis on the MIMIC-CXR

---

Limitations
Nori, Javier Alvarez-Valle et al. 2022. Making the most of text semantics to improve biomedical vision-language processing. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Is- rael, October 23-27, 2022, Proceedings. Part XXXVI, pages 1-21. Springer.
While our approach achieves good results, it is not without limitations. The main limitation lies in treating semantic extraction as multi-label classi- fication process. In practice, medical reports con- tain many labeling concepts, including some rare diseases that are seldom mentioned and difficult for the network to identify. And some specific de- scriptions such as "acute obliquely oriented lucency through the right 12th posterior rib" cannot be gen- satted.Additionaally while noise in the report does not affect visual feature extraction, optimizing the report using different weights may only partially alleviate the effects of noise on the decoder, with- out addressing the root cause.In the future, it may be necessary to start at the data level and standard- ize reporting up front. Reduce the interference of uncertainty. For rare diseases, additional med- ical knowledge may need to be introduced into the model. Improve identification of rare diseases through external knowledge enhancement. So at this stuge, it can still only play supporting role in the medical system.
Long Chen, Zhihong Jiang, Jun Xiao, and Wei Liu. 2021a. Human-like controllable image captioning with verb-specific semantic roles. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16846-16856.
Qi Chen, Chaorui Deng, and Qi Wu. 2022. Learning distinct and representative modes for image caption- ing. In Advances in Neural Information Processing Systems, volume 35, pages 9472-9485. Curran Asso- ciates, Inc.
Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021b. Cross- modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 5904 4914,, Online. Association for Computational Linguistics.
Zhihong Chen. Yan Song, I'sng-Huii Chang, and Xi- ang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439-1449, Online. Association for Computational Linguistics.
Ethics Statement
The MIMIC-CXR datasets employed in our study underwent a meticulous de-identification process. Our utilization of the MIMIC-CXR dataset aligns with the PhysioNet Health Data license 1.5.0 7. Importantly, the distributed MIMIC-XE dataset underwent thorough processing to remove all in- stances of protected health information (PHI). This approach underscores our commitment to adhering to ethical standards.
Marcella Cornia, Matteo Stefamini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed- trans- former for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages
Dina Demner-Fushmann Marc D Kohli, Mare B Rosen- man, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDon- ald. 2016. Preparing a collection of radiology ex- aminations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304 310.
References
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Zhiyuan Fang. Jianfeng Wang. Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu. 2022. Injecting semantic concepts into end- to-end image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition, pages 18009-18019.
Satanjeev Banerjee and Alon Lavic. 2005. Meteor; An automatic metric for mt evaluation with improved cor- relation with human judgments. In Proceedings of the act workshop intrinsic and extrinsic evaluation measures for machine translation and/or summariza- tion, pages 65-72.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shi- ratuddin, and Hamid Laga. 2019. A comprehensive survey of deep learning for image captioning. ACM Computing Surveys (CxUR), 11(6)
Jie Hu, Li Shen, and Gang Sun. 2018 Squeeze-and- excitation networks. In Proceedings of the IEEE con- ference on computer vision and pattern recognition. pages 7132-7141.
Benedikt Boecking, Naoto Usuyama, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer, Stephanie Hy- land, Maria Wetscherek, Tristan Naumann, Aditya

---

Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-lleus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, al. 2019. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference artificial intelligence, volume 33, pages 590-597.
Proceedings of the 29th ACM International Confer- ence on Multimedia, pages 3799-3800
Yehao Li, Yingwei Pan. Ting Yao, and Tao Mei. 2022. Comprehending and ordering semantics for image captioning. In Proceedings of the IEFF/CVE Confer- ence on Computer Vision and Pattern Recognition. pages 17990-17999.
Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P Lungren, Andrew Y Ng. et al. 2021. Radgraph: Extracting clinical entities and relations from radiology reports. arXiv preprint arXiv:2106. 14463.
Lukas Liebel and Marco Körner. 2018. Auxil- iary tasks in multi-task learning. arXiv preprint arXiv: 1.XXv:1800.06334
Chin- Yew Lin. 2004, Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Jaehwan Jeong, Katherine Tian, Andrew Li, Sina Har- tung. Subathra Adithan, Fardad Behzadi, Juan Calle, David Osayande, Michael Pohlen, and Pranav Ra- jpurkar. 2023, Multimodal image-text matching im- proves retrieval-based chest x-ray report generation. In Medical Imaging with Deep Learning.
Tsung- Yu Lin, Aruni RoyChowdhury, and Subhransu Maji. 2015. Billnear enn models for fine-grained vi- sual recognition. In 2015 IEEE International Confer- ence Computer Vision (ICCV). pages 1449 1457.
Fenglin Liu, Xian Wu, Shen Ge. Wei Fan, and Yuexian Zou. 2021a. Exploring and distilling posterior and prior knowledge for radiology report generation. Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 13753- 13762.
Baoyu Jing, Zeya Wang, and Eric Xing. 2019. Show, describe and conclude: On exploiting the structure information of chest X-ray reports. In Proceedings of the 57th Annual Meeting of the A Association for Com- putational Linguistics, pages 6570-6580, Florence, Italy, Association for Computational Linguistics.
Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. 2021b. Query2label: A simple transformer way to multi-label classification. arXiv preprint arXiv:2107.10834
Baoyu Jing. Pengtao Xie, and Eric Xing, 2017. On the automatic generation of medical imaging reports. arXiv preprint arXiv: 1711.0819...
Alistair EW Johnson, Tom Pollard, Nathaniel R Green- baum, Matthew P Lungren, Chih-ying Deng, Yifan Peng. Zhiyong Lu, Roger G Mark. Seth J Berkowitz, and Steven Horng. 2019. Mimic-cxr-jpg, large pub- licly available database of labeled chest radiographs. arXiv preprint 10X1:00900.0044..
Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2017. Knowing when look: Adaptive attention via a visual sentinel for image captioning. In P Proceedings of the IEEE conference on computer vision and pattern recognition, pages 375-383.
Luke Melas-Kyriazi, Alexander M Rush, and George Han. 2018. Training for diversity in image paragraph captioning. In proceedings of the 2018 conference on empirical methods in natural language processing, pages 757-761.
Alex Kendall, Yarin Gal. and Roberto Cipolla. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 7482-7491.
Ivona Najdenkoska, Xiantong Zhen, Marcel Worring, and Ling Shao. 2021. Variational topic inference for chest x-ray report generation. In Medical Image Com- puting and Computer Assisted MMCCCA 2021: 24th International Conference, Strasbourg, France, September 27 27-October 1, 2021, Proceedings, Part III 24, pages 625-633, Springer.
Jin-Hwa Kim, Kyoung- Woon On, Woosang Lim, Jeonghee Kim, Jung-Woo Woo Ha, and Byoung-Taie Zhang. 2016. Hadamard product for low-rank bi- linear pooling. arXiv preprint arXiv:1610.0453
Ming Kong, Zhengxing Huang, Kun Kuang, Qiang Zhu, and Fei Wu. 2022. Transq: Transformer-basee semantic query for medical report generation. In Medical Image Computing and Computer Assisted Intervention Intervention-MICCAA 2022: 25th International Con- ference, Singapore, September 18-22, 2022, Proceed- ings, Part VIII, pages 610-620, Springer.
Farhad Nooralahzadeh, Nicolas Perez Gonzalez, Thomas Frauenfelder, Koji Fujimoto, and Michael Krauthammer. 2021. Progressive transformer based generation of radiology reports. In Findings of the Association for Computational Linguistics: EMNIP 2021, pages 2824-2832 Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quar- terly. 2(1-2):83-99
Yingwei Pan, Ting Yao, Yehao Li, and Tao Mei. 2020. X-linear attention networks for image captioning. In Proceedings of the IEEE/CVF conference com- puter vision and pattern recognition, pages 10971 10980.
Yehao Li, Yingwei Pan, Jingwen Chen, Ting Yao, and Tao Mei. 2021. X-modaler: A versatile and high- performance codebase for cross-modal analytics. In

---

Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computa- lional Linguistics, pages 311-318.
Shuxin Yang. Xian Wu, Shen Ge, S. Kevin Zhou, and Li Xiao. 2022. Knowledge matters: Chest radiology report generation with general and specific knowl- edge. Medical Image Analysis, 80: 102510.
Changchang Yin, Buyue Qian, Jishang Wei, Xiaoyu Li, Xianli Zhang, Yang Li, and Qinghua Zheng. 2019. Automatic generation of medical imaging diagnostic report with hierarchical recurrent neural network. In 2019 IEEE international conference on data mining (ICDM), pages 728-737. IEEF.
Han Qin and Yan Song. 2022. Reinforced cross-modal alignment for radiology report generation. In Find- ings of the Association for Computational Linguisties: ACL 2022, pages 448 458.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman, Matan Protter, and Lihi Zelnik-Manor. 2021. Asymmetric loss for multi- label classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 82-91.
Feiyang Yu, Mark Endo, Rayan Krishnan, Ian Pan, Andy Tsai, Eduardo Pontes Reis, Eduardo Kaiser Ururahy Nunes Fonseca, Henrique Min Ho Lee, Zahra Shakeri Hossein Abad, Andrew Ng. et al. 2023. Evaluating progress in automatic chest x-ray radiology report generation. Patterns, 4(9).
Aaron Van Den Oord, Oriol Vinyals, et al. 2017. Neural discrete representation learning. Advances in neural information processing systems, 30.
Ke Yu, Shantanu Ghosh, Zhexiong Liu, Christopher Deible, and Kayhan Batmanghelich. 2022. Anatomy- guided weakly-supervised abnormality localization in chest x-rays. In Medical Image Computing and Computer Assisted Intervention MICCAI 2022, pages 558-668. Cham. Springer Nature Switzerland.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Jianbo Yuan, Haofu Liao, Rui Luo, and Jiebo Luo. 2019. Automatic radiology report generation based multi-view image fusion and medical concept richment. Medical Image Computing and Com- puter Assisted InterventionMIICAA 2019: 22nd In- ternational Conference, Shenzhen, China, October 13-17, 2019, Proceedings. Part VI 22, pages 721- 729. Springer.
Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3156-3164.
Jun Wang, Abhir Bhalerao, and Yulan He. 2022a. Cross- modal prototype driven network for radiology report generation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23- 27, 2022, Proceedings, Part XXXV. pages 563-579. Springer.
Yixiao Zhang, Xiaosong Wang, Ziyue Xu, Qihang Yu, Alan Yuille. and Daguang Xu. 2020. When radiol- ogy report generation meets knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12910-12917.
A
Appendix
Lin Wang, Munan Ning, Donghuan Lu, Dong Wei, Yefeng Zheng, and Jie Chen. 2022b. An inclu- sive task-aware framework for radiology report gen- eration. In Medical Image Computing and Com- puter Assisted InterventionMMCCAI 2022: 25th In- ternational Conference, Singapore, September 18- 22, 2022 Proceedings, Purt VIII, pages 568-577. Springer.
A.1 Dataset and pre-processing
We used the MIMIC-CXR dataset consisting of 377, 110 chest x-ray images. To facilitate compar- ison, we followed the preprocessing methods of other previous work (Chen et al., 2020). We only focus on the finding part of the medical report, ex- clude invalid data in the data, and use lowercase for all reports. Words that appear less than 10 times are ignored and replaced with <unk>. Only the first 100 words are retained for each report. The final sets obtained
Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou, 2022c. A medical semantic semantic-assisted transformer for radiographic report generation. In Medical Image Computing and Computer Assisted Intervention .MCCAII 2022: 25th International Conference, Singapore, September 18-22, 2022, Pro- ceedings, Part III. page 655-664, Berlin, Heidelberg. Springer- Verlag.
assisted
number of training/validationnnesss based on official data division: 269235/21 12/3851. In the training process, we scaled the image data to uniform size of (256.256) and used the follow- ing data enhancements: random image cropping to (224,224) and affine transformation, translation up to +2% of image height/width and rotation up to 100
Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo. Qing Du and Qi Wu. 2021. Towards accurate text-based image captioning with content diversity exploration. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 12637-12646.

---

Table 4: The impact of different semantics and extraction methods on the proposed model on the MIMIC-XR dataset.
Setting
SE
Abnormal
Normal
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
0.396
0.248
0.167
0.120
0.150
0.288
b
7
0.403
0.254
0.171
0.120
0.156
0.284
c
0.402
0.258
0.179
0.130
0.155
0.289
Table 5: dataset.
The proposed model's performance compared to those different memory modules on the MIMIC-CXK
Setting
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
0.354
0.222
0.152
0.109
0.132
0.267
b
0,402
0.258
0.179
0.130
0.155
0.289
To enable semantic tagging, we utilize the Rad- Graph (Jain et al., 2021) method for extracting clinical entities and their relationships from radi- ology reports. Specifically,
normal or abnormal. Description: XX. Keyword: XX." It is important to note that our semantics share the same word list the original report, and do not require any additional tokenization. Furthermore, the sentences where normal semantics are found are categorized as normal sentences, while those containing abnormal semantics are labeled as ab- normal sentences. Overall, we divide the words in the report into four parts: normal/abnormal sen- tences, and normal/abnormal semantics.
focus extract- ing anatomical entities and observation entities from the reports. To perform normal/abnormal semantic classification, we employ a keyword de- tection method. particular, we define a set of normal keywords based on the reference (Yu et al., 2022). These extracted semantics are then used multi-label classification labels for the semantic extraction task. Purely keyword-based detection sometimes produces misclassiffcations due to key- words. For example, "enlarged heart size stable since' may be misclassified as normal. To address these challenges, we have leveraged ChatGPT to enhance our categorization process. Specifically. the formidable semantic understanding capabilities of ChatGPT assist us in discerning between nor- mal and abnormal cases. We the GPT-3.5-turbo model for two-stage assessment. In the first stage, we determine whether the sentence is normal or abnormal. In the second stage, for sentences iden- tified as abnormal, we use the semantic keywords extracted by RadGraph to assess whether the spe- cific semantics are normal or abnormal. In the first stage.prompt: "You are a specialized radiologist. Evaluate the following medical descriptions. Note: any deviation from the imaging presentation of normal person is considered abnormal. No need to explain; answer directly: normal or abnormal. De- scription: XX." In the second stage, prompt: "You are a specialized radiologist. Determine whether the description of a given keyword is abnormal in a medical description. Note: any deviation from the imaging presentation of a normal person is consid- ered abnormal. No need to explain, answer directly:
A.2 Implementation Details
To ensure consistency with the experimental setup of previous work, used one image as input for MIMIC-CXR. The encoder and decoder modules in our framework consist of three basic layers. We set the 8 attention heads, 512 dimensions for hid- den states and initialize randomly respectively. During the report generation process, we set the beam size to 3. The codebook size N is 98, while the value of K used in the semantic generation pro- cess is 25. Semantic branch starts from the first layer APE-MHA. To optimize the model, em- ploy the Adam W optimizer, with model learning rate set 10 We implement our model using Pytorch and X-modaler (Li et al., 2021). We used 2080ti at MIMIC-CXR for 30 epoch of training for total of 26 GPU hours.
A.3 Additional Experiments
A.3.1 Semantic extraction method
We evaluated the effectiveness of our proposed method for semantic extraction, and the results are summarized in Table 4. In the table, SE represents the decoupled extraction of semantics, where Ab- normal and Normal indicate abnormal and normal semantics, respectively. Specifically, a denotes the

---

ties. Specifically, we memorize only the images where anomalies exist, and the model is encouraged to memorize different features by leveraging one- to-one matching and mutual exclusion loss. Figure shows the similarity matrix of the memorized codebooks in our method, and it can be observed that the patterns memorized in the codebooks only have high similarity to themselves. This illus- trates the diversity of our method's memory. Over- all, our abnormal memory approach yields better results than method A, which underscores its effec- tiveness in addressing data bias and improving the performance medical report generation models.
A.3.3 Noise impact experiment
To explore the effect of noise in report and nor- mal semantics, we designed additional experiments with Normal denoting the gradient of truncated normal semantic branches visual extraction and Report_ De denoting the gradient of truncated port decoding on visual extraction. a denotes that report noise with normal semantic noise can af- feet visual extraction, denotes that report noise only affects, and с denotes that normal semantic noise only influence. d indicates our final model, i.e., truncated with all noise gradients. The results in Table show that both noises have a negative impact and cause decrease in the metrics. This also shows that our framework of normal anomaly semantic decoupling can avoid the effect of this complex noise to some extent.
Cosine similarity scores of all patterns memo- rized in the codebook
Figure 5:
extraction of abnormal semantics only, b denotes the extraction of both normal and abnormal seman- ties but without decoupling, and represents proposed decoupling method. Comparing a and we observe that both types of semanties have a posi- tive impact on the final report generation. However, when comparing b and c, we find that the decoupled and separated extraction upproach is more effective, with 0.9% improvement in BLUE4. This under- scores the presence of noise in normal semantics, which can adversely affect the visual extractor and compromise reporting accuracy. On the other hand, our proposed decoupling method avoids the effects of noise and achieves better reporting results.
A.3.4 More detailed evaluation
to comprehensively demonstrate the performance of our model, we employed the RadGraph-Fi (Yu et al., 2023) test model and provided a de- tailed breakdown of disease-specific F1 scores and proportions. Table represents the RadGraph- FI scores, where R2Gen (Chen et al., 2020) and R2GenCMN (Chen et al., 2021b) are obtained by calculations using the officially provided model weights and X-REM (Jeong et al., 2023) are from the original paper. Our metrics achieve the highest scores. Table 8 represents the F1 scores for the 14 diseases. The weight of this disease occupying all the MIMIC-CXR data is labeled after each category. R2Gen (Chen et al., 2020) and R2GenCMN (Chen et al., 2021b) obtained by calculations using the officially provided model weights. can be seen that the best scores are achieved in most categories. And the percentage of anomalics is small in terms of category share. Detection failures can be due
A.3.2 Memory method
We compared our proposed memory approach with the memory modeling approach proposed in method (Cornia al., 2020) to highlight the role of abnormal memory. For the experiments, parameters such as memory capacity constant. The results show that the memory approach in method (Cornia et al., 2020) did not perform well. We be- lieve that this is due to data bias, as method (Cornia et al.. 2020) introduces a memory matrix in the self-attentive phase, which enables it to automat- ically memorize features in the training data and form prior knowledge. However, since data bias may have caused it to memorize more repetitions of normal cases, achieved suboptimal results.
kept
In contrast, our approach provides more useful information by explicitly memorizing abnormali-

---

Table 6: Effect of different noise on the proposed model on the MIMIC-CXR dataset.
Setting
Normal DD
Report_ De
BLEU-1
BLEU-2
BLEU-3
BLEU-4
METEOR
ROUGE-L
a
0.397
0.251
0.172
0.124
0.151
0.289
b
V
0.394
0.253
0.176
0.128
0.1.1
0.290
0.391
0.250
0.173
0.125
0.150
0,289
d
0.402
0.258
0.179
0.130
0.155
0.289
Table 7: The performances of our model compared with baselines on IU-Xray dataset. The best results are highlighted in bold.
Dataset
Model
BLEU-1
BLEU-2
BLEU-3
BLEU-4 METEOR
ROUGE-L
S&T
0.216
0.124
0.087
0.066
0.306
AdaAtt
0.220
0.127
0.089
0.068
0.308
R2Gen
0.470
0.304
0.219
0.165
0.187
0.371
PPKED
0.483
0.315
0.224
0.168
/
0.376
M2TR
0,486
0.317
0.232
0.173
0.192
0.390
IU-Xray
R2GenCMM
0.475
0.309
0.222
0.170
0.191
0.375
GSKET
0.496
0.327
0.238
0.178
/
0.381
R2GenRL 0.494
0.321
0.235
0.181
0.201
0.384
MSAT
0.481
0.316
0.226
0.171
0.190
0.372
Ours 0.506
0.344
0.256
0.198
0.211
0.390
Table 8: The detailed results across all CheXpert categories for previous report generation models and our proposed model based on MIMIC-CXR.
Class(%)
R2Gen
R2GenCMN
Ours
No Finding (33.119%)
0.512
0.513
0.471
Enlarged Cardiomediastinum (7.266%)
0.360
0.335
0.344
Cardiomegaly (22.336%)
0.377
0.405
0.406
Lung Lesion (3.259%)
0.329
0.327
0.322
Lung Opacity (24.297%)
0.350
0.350
0.364
Edema (17.641 %)
0.385
0.428
0.429
Consolidation (6. (6.632%)
0.335
0.354
0.370
Pneumonia (15.295%)
0.347
0.350
0.355
Atelectasis (24.639%)
0.369
0.398
0.341
Pneumothorax (5.044%)
0.338
0.339
0.348
Pleural Effusion (26.366%)
0.383
0.463
0.464
Pleural Other (1.218%)
0.326
0.331
0.331
Fracture (2.171%)
0.322
0.322
0.328
Support Devices (29.318%)
0.506
0.522
0.467
to different reasons, such as little training data (e.g., "fracture". "Lesion") or difficult to learn (e.g., "pneumothorax", which is is also difficult for clinicians determine).
Table 9: Comparison of our model previous report generation models on MIMIC-CXR,
Model
RadGraph-F1
A.3.5 IU-Xray experiment
R2Gen
0.165
The Indiana University chest x-ray collection (IU- Xray²) (Demner-FFushman et al., 2016) is pub- lic radiology examination dataset and common dataset used in medical report generation tasks.
R2GenCMN 0.182
X-REM
0.181
Ours
0.209
hhttps://openi: nlm. http:/pppni...l......oo.o./////

---

The dataset includes 7,470 x-ray images and the corresponding 3,955 radiology reports. However, the IU-Xray dataset does not have standard dataset segmentation, resulting in some the pre- vious methods not performing comparably in terms of metrics. To facilitate comparison with previous work, we used the same preprocessing and segmen- tation as in (Chen et al., 2020). We also compare only the methods that the same segmentation. The training/test/val setting for the entire dataset was 7:1:2.0r 2.Our data processing steps were similar to those of MIMIC-CXE Due to the small number of IU-Xray data, we excluded words with less than occurrences and focused only on the first 60 words.
ours
GT
frontal and lateral views of the obtained no focal consolidation effusion pneumothorax. cardiac silhouelle normal size patchy opacily in the right lower lobe likely due to atelectasls mediastinal and contours unremarkable pulmonary vasculature normal. mild degenerative changes are noted
chest pa and lateral radiograph demonstrates unremarkable mediastinal and hilar contours increased opacity overlying the right diachragm background of right lower lung atelectasis may indicate pneumoniaa pleural effusion or pneumathorax
(a)
and t12 compression fractures slable degenerativee changes the right shoulder
the heart normal size the mediastinum is unremarkable. emphysemalous changes are identified. lungs othervise grossly clear
the heart is normal size the lungs are clear there no pleural effusion preumothorax
(b)
we compare our proposed method with conven- tional image captioning works, e.g. S&T (Vinyals et al., 2015), AdaAtt (Lu et al., 2017), and the ones proposed for the medical domain, c.g. R2Gen (Chen et al., 2020), PPKED (Liu et al., 2021a), M2TR (Nooralahzadeh et al., 2021), R2GenCMN (Chen al., 2021b) GSKET (Yang et al., 2022), R2GenRL (Qin and Song, 2022) and MSAT(Wang al., 2022c). The results on S&T (Vinyals al., 2015), AdaAtt (Lu et al., 2017) from (Chen et al., 2020), and the rest of the results were cited from the original paper.As we can see from the results in Table 7, our method can still achieve good re- sults on the IU-Xray dataset, which indicates the generality of our method.
cardiac silhouette enlarged and accompanied by pulmonary vascular congestion persistent moderate right and small pleural effusions adjacent basilar lung opacities which probably reflect atalectasis although coexisting pneumonia possible in the appropriate clinical setting
volumes low. the heart size is mildly enlarged mediastinal
(c)
contours are unremarkable
Figure 6: Example of a generated report compared to a real report.
may be due to the lack of more detailed labeling in the report.
A.5 More Example Visualizations
A.4 Hallucination samples
order more clearly exemplify the error and hallucination samples, we have listed a detailed comparison in Figure 6. can be seen that our report (a) observes atelectasis but omits fractures. This may be mainly due to the fact that the fractures category is underrepresented in the data. Secondly we not able report specific descriptions like 11 and t12. Although we report degenerative changes, they are not specific to the right shoulder. For some the normal descriptions of (b), we don't 't match well with the real report, we don't 't report no acute bony abnormalities. This has to do with the normal noise in the report, which is mentioned in some doctors' reports and not in others. In (c) we report patchy opacities, due to lung atelectasis, but in real- ity is basilar lung opacities. and report pleural effusion on the left side, but in the actual report is present on the left side well as the right side. is possible that the model discriminated the plcural effusion but was inadequate for orientation. This

---

image
OurS
GT
the heart size and mediastinal contour appear within nomaal limits airspuce consolidution pleural oflusion pueumotborax, acute hony abnoemalities.
beart and mediastinal contours within normal limits, the lungs are clear there is no pleural effusion pneumothorax.
compared to the previous radiograph there no relevant change. new parenchymal opacity unchanged moderate cardiomegaly and unchanged position and course of the sternal wires and clips after cabg pre-existing platelike in the left mid lung has resolved.
obtained. the lungs are clear without focal consolidation effusion or pneumothomxx the cardiae silhouctte mildly colarged. mediastinal and hilar contours are within normal limits. there is no pulmonary edema. there no pleural efftasion pacumothoras, there is mild degenerative changes in the thorucic spine
unchanged area of mild right lateral pleural thickening no pulmonury edema pleural effusions. lung nodules or masses.
POTTULE
lung volumes low, the heart size is mildly enlarged , the medinstinal and hilar contours are unremarkable there mild pulmonary vascular congestion. patchy opacities in the right lower lung base likely due to atelectasis. Iaterta pleural effusions
curdiac silhouette enlarged and accompanied by pulmonary vascular congestion persistent moderate right and small pleural effusions adjacent basilar lung opacities which probably reflect atelectasis although coexisting pocumonia is possible the appropriate clinical setting.
the left. there no pacumethorax there is no pulmonary edema .tere is no acute abnormalities.
the heart is normal size the medinstinum is unremarkable emphysematous changes identified. the lungs are otherwise grossly clear.
heart normal size lungs are there no pleural effusion pneumothorax.
Figure 7: Visualization of prediction results, where GT is the abbreviation of the ground truth, text in red indicates the reported keywords and ground truth keywords generated by our method.


Reviews:
[{"review_id": "0NlaBNIzU4", "reviewer": "Reviewer_6Vvb", "summary": "", "questions": "1. What are the details of GPUs used in the experiment and the number of epochs for training, and GPU hours?\n2. The paper mentions in line 51 that the presence of noise in reports impacts network optimization, and similar mentions of noise has been repeated in the paper a few times. Please elucidate the type of noise being referred to, and how it impacts network optimization\n3. Is the paper using only the Finding section of the radiology report throughout the experiment? It is a bit unclear from the description.\n4. Which language model has been used for embeddings for training this model?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for thoroughly reviewing our paper. I greatly appreciate your attention to detail. I will promptly address any questions you have and augment the content accordingly.\n\nQ1: the type of noise and normal-abnormal semantics concepts. \n\nWe would like to clarify that the noise sources mentioned in the paper encompass two aspects. Firstly, variations in different descriptions of the same normal semantics . Secondly, differences in reporting habits among various doctors can lead to discrepancies in normal descriptions within reports, where some might be mentioned while others are omitted. Importantly, the semantics themselves are inherently noise-free, as we attribute correctness to the physician's expert judgment. The explicit classification of normal and abnormal semantics is outlined in detail in Appendix A.1. To extract anatomical entities with observations, we employ RadGraph for semantic extraction. For the categorization of normal and abnormal semantics, we rely on using easily detectable normal keywords (e.g., \"normal, clear\"), as they offer a straightforward means of classification. These keywords are non-ambiguous, limited in scope, and remain relatively consistent. Statements that do not match normal descriptions are presumed to pertain to anomalies, which has demonstrated utility in previous research [1]. We intend to provide comprehensive coverage of this clarification within the main text of the paper.\n\nQ2: how noise impacts network optimization?\n\nIn Q1, we clarify the concept of noise. We contend that noise contributes to an element of ambiguity during the training process. The normal description generated by the model may be correct, but omitted from its report (or a different description with the same meaning), while it may be mentioned again in other reports. Our assertion is that this inherent ambiguity disrupts the training process. This interpretation aligns with the outcomes of our noise-related experiments presented in Appendix A.3.3. Specifically, these experiments demonstrate performance enhancements when training gradients that encompass normal semantics containing noise or when training gradients derived from reports with noise are eliminated.\n\nQ3: our motivation for developing the model architecture.\n\nWe would like to clarify the motivation of our decoupling structure; as illustrated in Q2, the ambiguity as well as the uncertainty of the normal description can disrupt the training of the model, so we designed this decoupling structure to isolate the interference of this noise. In fact, anomalies are often valued more than normal descriptions in reality. Our decoupling structure realizes this well. We added a separate set of experiments with decoupled structures in our response to other reviewers, and it is exciting to note that without adding any other improvements, our model has reached the level of past SOTA models.\n\nQ4: the types of hallucinations the model and how it differs in its outputs and hallucinations from other state of the art models.\n\nWhen it comes to modeling hallucinations, our approach is designed to mitigate the impact of noise. This is achieved by decoupling normal abnormal and optimizing report generation for these categories separately. This strategic division enhances the model's sensitivity to anomalies while reducing the prominence of normal descriptions. It is important to acknowledge that our model might produce normal descriptions that deviate from true reports. This could stem from the potential for normal descriptions to be confounded within the training data. Nevertheless, in a real-world context, a primary emphasis should rightfully be placed on anomalies. Misidentified anomalies have a more pronounced adverse impact than redundant normal descriptions.Thanks to the integration of abnormal pattern memory and the model's inherent focus on abnormal semantics, our approach achieves greater score in describing certain anomalies compared to earlier models (our F1-RadGraph and clinical efficiency metrics performs better ). For example, \"Consolidation\" was ignored in the reports generated by previous models, while our approach mentions this abnormality. However, there are mistakes in the description of some orientations, e.g., left and right are sometimes confused, which is a challenge for generating accurate medical reports. We will add comparisons of our model with other models in the revision to show more examples of the hallucination.\n\nQ5: more detailed evaluation. For example the distribution of diseases in the dataset, the detailed AUC/F1 per disease,  F1-RadGraph,  metrics of text generation like fluency, coherence.\n\nIn our study, we opted for common NLG metrics (BLEU, METEOR, ROUGE). Because we are comparing the whole report as an output to the real report, rather than a single sentence, as elaborated in previous work these metrics can to some extent respond to the fluency and coherence [2,3]. Furthermore, to comprehensively demonstrate the performance of our model, we employed the RadGraph-F1 test model, in line with suggestions, and provided a detailed breakdown of disease-specific F1 scores and proportions.  The results are presented in Table 1. It's worth noting that due to the nature of our testing methodology, which involves comparing the label extraction of generated reports, accurate calculation of AUC metrics isn't feasible. Consequently, we exclusively report F1 metrics for each disease in this context. Table 1 represents the RadGraph-F1 scores, where [4,5] are obtained by calculations using the officially provided model weights and [6] are from the original paper. Our metrics achieve the highest scores.  Table 2 represents the F1 scores for the 14 diseases. The weight of this disease occupying all the MIMIC-CXR data is labeled after each category. [4,5] obtained by calculations using the officially provided model weights. It can be seen that the best scores are achieved in most categories. And the percentage of anomalies is small in terms of category share. Detection failures can be due to different reasons, such as too little training data (e.g., \"fracture\", \"Lesion\") or too difficult to learn (e.g., \"pneumothorax\", which is is also difficult for clinicians to determine).\n\nQ6: details about the language model used for embeddings.\n\nWe did not use an existing language model for embedding, but only used a general embedding layer for language embedding, which will be trained as the model is trained. We used to use Bio_ClinicalBERT [7] for embedding in the early stage, but there was not much difference in its performance. So we use general embbedding to facilitate the training together.\n\nQ7: what are the details of GPUs used in the experiment and the number of epochs for training, and GPU hours? Missing References. Misspelled format.\n\nWe apologize for not being explicit, we used 2080ti at MIMIC-CXR for 30 epoch of training for a total of 26 GPU hours. Thanks for pointing out the spelling errors in the article, we will correct the spelling errors and missing definitions and add the missing references [8] in the revision. We will also make it clear in the text that we are focusing on the findings section of the report.\n\nQ8: The paper does not have an ethics section,it is prudent to include a section detailing failures and biases of the model.\n\nWe wish to clarify and emphasize our ethical considerations in this context. The MIMIC-CXR datasets employed in our study underwent a meticulous de-identification process. Our utilization of the MIMIC-CXR dataset aligns with the PhysioNet Health Data license 1.5.0 7. Importantly, the distributed MIMIC-CXR dataset underwent thorough processing to remove all instances of protected health information (PHI). This approach underscores our commitment to adhering to ethical standards. Secondly we will add the failure and bias part of the model further to limitation, in fact, data bias is the main cause of failure and bias. For example, fracture accounts for less than 2% of the entire dataset. In the test, it is difficult for the model to detect report the case of fracture. Secondly, the description ''acute oblique translucency through the right posterior twelfth rib'', which is specific to the first rib, could not be given or was reported incorrectly. We will add more negative sample analyses.\n\nTable 1 : Comparison of our model to previous report generation models on MIMIC-CXR.\n\n| Model        | RadGraph-F1 |\n| ------------ | ----------- |\n| R2Gen [4]    | 0.165       |\n| R2GenCMN [5] | 0.182       |\n| X-REM [6]    | 0.181       |\n| Ours         | **0.209**   |\n\nTable 2 : The detailed results across all CheXpert categories for previous report generation models and our proposed model based on MIMIC-CXR.\n\n| Class(%)                             | R2Gen | R2GenCMN  | Ours      |\n| ------------------------------------ | ----- | --------- | --------- |\n| No Finding (33.119%)                 | 0.512 | **0.513** | 0.471     |\n| Enlarged Cardiomediastinum  (7.266%) | 0.360 | 0.335     | **0.344** |\n| Cardiomegaly (22.336%)               | 0.377 | 0.405     | **0.406** |\n| Lung Lesion (3.259%)                 | 0.329 | **0.327** | 0.322     |\n| Lung Opacity (24.297%)               | 0.350 | 0.350      | **0.364** |\n| Edema (17.641%)                      | 0.385 | 0.428     | **0.429** |\n| Consolidation (6.632%)               | 0.335 | 0.354     | **0.370** |\n| Pneumonia (15.295%)                  | 0.347 | 0.350     | **0.355** |\n| Atelectasis (24.639%)                | 0.369 | **0.398** | 0.366     |\n| Pneumothorax (5.044%)                | 0.338 | 0.339     | **0.348** |\n| Pleural Effusion (26.386%)           | 0.383 | 0.463     | **0.464** |\n| Pleural Other (1.218%)               | 0.326 | **0.331** | **0.331** |\n| Fracture (2.171%)                    | 0.322 | 0.322     | **0.328** |\n| Support Devices (29.318%)            | 0.506 | **0.522** | 0.507     |\n\n[1] Ke Yu, Shantanu Ghosh, Zhexiong Liu, Christopher Deible, and Kayhan Batmanghelich. 2022. Anatomyguided weakly-supervised abnormality localization in chest x-rays. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, pages 658–668, Cham. Springer Nature Switzerland.\n\n[2] Hoang Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong, and Li Cheng. 2021. Automated generation of accurate & fluent medical X-ray reports. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3552–3569, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.\n\n[3] Justin Lovelace and Bobak Mortazavi. 2020. Learning to generate clinically coherent chest X-ray reports. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1235–1243, Online. Association for Computational Linguistics.\n\n[4] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. 2020. Generating radiology reports via memory-driven transformer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1439–01449, Online. Association for Computational Linguistics.\n\n[5] Zhihong Chen, Yaling Shen, Yan Song, and Xiang Wan. 2021b. Cross-modal memory networks for radiology report generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5904–5914, Online. Association for Computational Linguistics.\n\n[6] Jaehwan Jeong, Katherine Tian, Andrew Li, Sina Hartung, Subathra Adithan, Fardad Behzadi, Juan Calle, David Osayande, Michael Pohlen, and Pranav Rajpurkar. 2023. Multimodal image-text matching improves retrieval-based chest x-ray report generation. In Medical Imaging with Deep Learning.\n\n[7] Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.\n\n[8] Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The methodology achieves higher than state-of-the-art for MIMIC CXR report generation.\n2. The methodology is novel, and involves an understanding the structure of radiology report findings beyond just caption generation problem for radiographic images. In radiology reports, the radiologists tend to write more about the abnormalities than focus on the normal perceptions in the report, and this architecture utilizes that aspect in their modeling quite well.\n3. The analysis of the model is rigorous and gives us a clear understanding of the types of errors to expect.", "reasons_to_reject": "1. Though one of the core concepts of the paper is normal-abnormal semantics in reports, the classification from the reports is largely dependent on keyword-based method, which, the authors accept is largely noisy. The definition of what is an abnormal semantics in a report varies depending on application, and there is no specification of any such definition followed in this paper. This makes the normal/abnormal semantic section a little weak compared to the other sections. \n-- After rebuttal: Even with the rebuttal, I think this portion could have been handled much better.", "questions_to_authors": "1. What are the details of GPUs used in the experiment and the number of epochs for training, and GPU hours?\n2. The paper mentions in line 51 that the presence of noise in reports impacts network optimization, and similar mentions of noise has been repeated in the paper a few times. Please elucidate the type of noise being referred to, and how it impacts network optimization\n3. Is the paper using only the Finding section of the radiology report throughout the experiment? It is a bit unclear from the description.\n4. Which language model has been used for embeddings for training this model?", "paper_topic_and_main_contributions": "The paper introduces a novel concept to automatically generate radiology reports from radiographic images using normal-abnormal semantic decoupling network with abnormal pattern memory. The authors combine the concepts of abnormal semantics in reports with visual feature extraction to shape the decoder output. Their trained model achieves higher performance than state of the art models in all of the relevant metrics. In addition, the paper clearly analyses the model output by using model visualization which helps in understanding the model better. The writing is crisp, easy to understand and clearly explains the methodologies used in the paper without ambiguity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Please cite Hungarian algorithm in line 221. Please cite the language model that has been used.", "Justification_for_Ethical_Concerns": "The paper does not have an ethics section, and since this pertains to a medical application, it is prudent to include a section detailing failures and biases of the model."}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. The paper focuses on generating the Findings in a radiology report, which is an important detail that is only mentioned in line 871 in the Appendix section of the paper. This can be mentioned in the beginning.\n2. Please correct spacing around period in line 240.\n3. In line 294, Att definition is missing.\n4. In line 326 Cancat needs to be changed to Concat, and defined.\n5. Correct typo : Abnaormal -> Abnormal in table 5"}}, {"review_id": "RHLG5RNoTb", "reviewer": "Reviewer_dM6t", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewers for reviewing the article and pointing out the spelling errors, which we will correct in the revision. Our primary objective is to tackle the challenge posed by noise in the domain of report generation. To the best of our information, we are the first article to make this noise explicit and attempt to address it. In our pursuit, we present a novel approach that effectively mitigates the impact of this ambiguous noise by decoupling normal abnormal semantics. Additionally, we introduce an encoder for abnormal pattern memory to heighten the model's sensitivity to anomalies. The performance of our model substantially outperforms the previous SOTA. Hopefully, our research can contribute to the development of automated assisted diagnosis in medicine. Thanks again for reviewing the manuscript!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of using the codebook in VQVAE to store various anomaly information seems to be a right way to go. \n2. The memory loss in the objective functions is novel and makes sense. \n3. The experimental results seem to be very promising. And the baselines are comprehensive. ", "reasons_to_reject": "I didn't find any obvous reason to reject the paper. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work aims to deal with the problem of medical report generation. The develop a visual encoder based on anomaly pattern memory to enhance anomaly perception. The performance shown on MIMIC-CXR is promising over multiple SOTA methods. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 203, \"Combined with global features $V_l$\" should be \"Combined with global features $V_g$\" "}, "extra_text": {}}, {"review_id": "34uvjucb98", "reviewer": "Reviewer_VhyG", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for carefully reviewing our paper, and we offer clarification to your questions.\n\nQ1: The proposed method is somewhat incremental.\n\nWe think that our work is novel and relevant. The description of normal varies considerably from doctor to doctor report. To the best of our information, we are the first article to make this noise explicit and attempt to address it. The detrimental impact of this noise is effectively demonstrated in Experiment A.3.3, as elaborated in the Appendix. Our novel proposition of employing the concept of decoupling stands as a logical and effective means to circumvent the influence of this noise. While prior research has explored the separation of normal exceptions to generate reports, it's essential to emphasize that our approach significantly differs in both intent and methodology. As in [1], the data bias problem is considered, where the noise of normal statements still affects the optimization of the model despite the separate generation of normal/abnormal statements. We decouple normal and abnormal semantics, normal semantics are generated independently and gradients do not affect visual optimization. Secondly grouping the reports for optimization emphasizes the role of key semantics rather than unimportant noise. We added additional experiments to illustrate the superiority of our approach. We constructed a separate model of the decoupled structure, with the rest of the modules using the normal Transformer structure, and the hyperparameters remaining the same as in our original model. As shown in Table 1, we find that even without using any other improvements, our method can achieve comparable performance to the previous SOTA model just by utilizing the decoupled approach for training. This proves the significance of our method. Moreover, we believe that this decoupling can be equally inspired by large-scale data. And our decoupled structure can be used in parallel with other improvements without conflict.\n\nQ2: the abnormal mode memory part is not quite clear.\n\nDue to data bias, normal images dominate in the data. We mitigate this problem by using abnormal pattern memory to explicitly memorize abnormal regions of an image. Specifically, inspired by VQ-VAE [2], a discrete memory matrix is created, assuming $N\\times D$. The image is encoded by ResNet to obtain the local information of $V \\times D$. When memorizing, we look for a unique pattern in the memory space corresponding to each image encoding ($1 \\times D$). The Hungarian algorithm [3] is used to find a unique pattern match in the memory matrix by treating the image features with each pattern of the memory matrix as an ensemble matching problem. We utilized orthogonality as well as the Hungarian algorithm [3] in order to find as much information about different patterns as possible (Figure 5 shows that we learned different patterns).  And due to gradient truncation, our memory matrix does not affect visual optimization. We will improve the description of this section in the revision to make it clearer.\n\nQ3: This paper only includes results of one dataset in the main text.\n\nDue to space limitation, we only use the MIMIC-CXR dataset in the main text, but in Appendix A.3.4 we also report the performance of the IU-Xray dataset, which also achieves SOTA. The MIMIC-CXR has a much larger amount of data, and the results are more responsive to the performance of the model.\n\nQ4: Missing References. Misspelled format.\n\nThanks for pointing out the missing definitions in the article, we will correct the spelling errors as well as clarify the definitions in the revision, where $V_l$  stands for localized image encoding, $V_g$  refers to global image encoding, and k refers to the prediction scores ranked in the first k. And we will add the missing references [1].\n\nTable 1 : Comparison of our simple decoupling model to previous SOTA models on MIMIC-CXR.\n\n| Model    | BLEU-1    | BLEU-2    | BLEU-3    | BLEU-4    | METEOR    | ROUGE-L   |\n| -------- | --------- | --------- | --------- | --------- | --------- | --------- |\n| MSAT [4] | 0.373     | 0.235     | 0.162     | **0.120** | 0.143     | **0.282** |\n| Ours     | **0.389** | **0.241** | **0.163** | 0.117     | **0.147** | 0.281     |\n\n[1] Baoyu Jing, Zeya Wang, and Eric Xing. 2019. Show, describe and conclude: On exploiting the structure information of chest X-ray reports. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6570–6580, Florence, Italy. Association for Computational Linguistics.\n\n[2] Aaron Van Den Oord, Oriol Vinyals, et al 2017. Neural discrete representation learning. Advances in neural information processing systems, 30.\n\n[3] Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97.\n\n[4] Zhanyu Wang, Mingkang Tang, Lei Wang, Xiu Li, and Luping Zhou. 2022c. A medical semantic-assisted transformer for radiographic report generation. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part III, page 655–664, Berlin, Heidelberg. Springer-Verlag.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. It is interesting to separately model normal and abnormal reports.\n2. The proposed approach could outperform baselines on the benchmark MIMIC-CXR dataset.\n3. The ablation study shows that each of the component could help improve the performance.", "reasons_to_reject": "1. The abnormal mode memory part is not quite clear, which is difficult to interpret. \n2. The proposed method is somewhat incremental.\n3. This paper only includes results of one dataset in the main text.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a normal-abnormal decoupling memory network for medical report generation. The proposed approach optimizes the visual extraction network only based on anomaly semantics but not normal semantics. The experiments show that the proposed method could outperform baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Jing, Baoyu, Zeya Wang, and Eric Xing. \"Show, Describe and Conclude: On Exploiting the Structure Information of Chest X-ray Reports.\" Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019.\n\n[1] is a prior work which also decomposes normal and abnormal reports.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. Section 3.2 is not clear, which needs further presentation improvement.\n2. There are some notation issues. For example, there are some issues with $V_l$ and $V_g$. It is not clear what does $k$ mean in section"}}]

Author rebuttal:

Meta-review: The paper investigates the automatic generation of radiology reports from X-ray images (MIMIC-CXR). The authors' approach centers on optimizing the visual encoder based on anomaly semantics, deliberately excluding normal semantics to mitigate the impact of potential noisy signals.

All reviewers acknowledge the effectiveness of the proposed method in the radiology report generation domain, noting improvements relative to current state-of-the-art benchmarks. Comprehensive ablations and analyses are also presented. Some concern is expressed regarding how to determine abnormal vs. normal semantics (a naive heuristic is used which can be noisy itself).
Decision: Accept-Findings