Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
Authors: Fenglin Liu
Chenyu You
Xian Wu
Shen Ge
Sheng Wang
Xu Sun
Venue: NeurIPS 2021 Poster (2021)
Keywords: Medical Report Generation
Unpaired Data
Unsupervised Learning
Knowledge Graph

Abstract: Medical report generation, which aims to automatically generate a long and coherent report of a given medical image, has been receiving growing research interests. Existing approaches mainly adopt a supervised manner and heavily rely on coupled image-report pairs. However, in the medical domain, building a large-scale image-report paired dataset is both time-consuming and expensive. To relax the dependency on paired data, we propose an unsupervised model Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images and reports in training. KGAE consists of a pre-constructed knowledge graph, a knowledge-driven encoder and a knowledge-driven decoder. The knowledge graph works as the shared latent space to bridge the visual and textual domains; The knowledge-driven encoder projects medical images and reports to the corresponding coordinates in this latent space and the knowledge-driven decoder generates a medical report given a coordinate in this space. Since the knowledge-driven encoder and decoder can be trained with independent sets of images and reports, KGAE is unsupervised. The experiments show that the unsupervised KGAE generates desirable medical reports without using any image-report training pairs. Moreover, KGAE can also work in both semi-supervised and supervised settings, and accept paired images and reports in training. By further fine-tuning with image-report pairs, KGAE consistently outperforms the current state-of-the-art models on two datasets.

Auto-Encoding Knowledge Graph for Unsupervised Medical Report Generation
Fenglin Liu!?, Chenyu You', Xian Wu', Shen Ge Sheng Wang"; Xu Sun
'MOE Key Laboratory of Computational Linguistics, School of EECS, Peking University
"School of ECE, Peking University
Paul G. Allen School of Computer Science and Engineering, University of Washington
*Department of Electrical Engineering, Yale University Tennent
(fenglinliu98, xusun] Opku. edu. en, chenyu. chenyu. you@yale edu (kevinxuu, shenge)@tenccnn com, swang0cs washington edu
Abstract
Medical report generation, which aims to automatically generate a long and coher- ent report given medical image, has been receiving growing research interests. Existing approaches mainly adopt supervised manner and heavily rely on cou- pled image-report pairs. However, in the medical domain, building a large-scale image-report paired dataset is both time-consuming and expensive. To relax the dependency on paired data, we propose an unsupervised model Knowledge Graph Auto-Encoder (KGAE) which accepts independent sets of images and reports in training. KGAE consists of a porstrontreeted knowledge graph, a knowledge- driven encoder and knowledge-drice shared latent space to bridge the visual and textual domains; The knowledge-driver encoder projects medical images and reports to the corresponding coordinates in this latent space and the inwweedeeddriven decoder generates a medical report given coordinate in this space. Since the knowledge-driver encoder and decoder can be trained with independent sets of images and reports, KGAE is unsupervised. The experiments show that the unsupervised KGAE generates desirable medical reports without using any image-report training pairs. Moreover, KGAE can also work in both semi-supervised and supervised settings, and accept paired images and reports in training. By further fine-tuning with image-report pairs, KGAE consistently outperforms the current state-of-the of-the-art models on two datasets.
decoder. The knowledge graph works as the
Introduction
Medical images, such as radiology and pathology images, and their corresponding reports are widely used for clinical diagnosis and treatment [1]. medical report usually a paragraph of multiple sentences which describes both the normal and abnormal findings in the medical image. In clinical practice, writing report can be cime-consuming and tedious for experienced radiologists, and error-prone for inexperienced radiologists [4]. Therefore, given the large volume of medical images, automatically generating reports can improve current clinical practice in diagnostic radiology and assist radiologists in clinical decision-making [5 25 Specifically, it can relieve radiologists from such heavy workload and alert radiologists of the abnormalitics to avoid misdiagnosis and missed diagnosis. Therefore, automatic medical report generation attracts remarkable attention in both artificial intelligence and clinical medicine.
Recently, inspired by the great success of neural machine translation 2
38
image the data-driven deep
50
48
captioning [44 39 46 28 30 29 and medical imaging analysis 47
52
"Corresponding authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Input Image
Knowledge
Predicted
Knowledge Graph
heanearsse crbiges b23
slsgoce spackies efuzion.
proumsiters, Empt yoerartin
charpen Calokee thenc acdss
511
Knowledge-driven Encoder
Knowledge Representations
Reconstructed Report
Bonderize cartler enberpement Enarged califtied hhilacd sora. Ernalipcoms no sulmenary sponddeasis.
Eskarped cakrieed therac aoraa Errpfosama, acsie purmonary
Botterinn cardlat entargement
G
Mill
g
9B
Figure 1: Ilustration of our Knowledge Graph Auto-Encoder, which consists of pre-constructed knowledge graph, knowledge encoder and knowledge-driven decoder. The Green and Red lines denote the data flow in the training process and testing process of report generation, respectively,
neural models, particularly those based on the encoder-decoder frameworks [5] 16 25
45
53 [54 [1, have achieved great success in advancing the state-of of-the-art of medical report generation. However, these models are trained in a supervised learning manner and heavily rely on labeled paired image-report datasets (9 [7], which are not easy to acquire in the real world. Specifically, the medical-related data can only be manually labeled by professional radiologists, and also involves privacy issues. Therefore, the medical report generation datasets are partieularly labor-intensive and expensive to obtain. As a result, the scales of existing widely-used datasets for medical report generation models [16] 25 22 23 i.e., MIMIC-CXR (0.22M samples) (17] and IU X-ray (4K samples) [9] are relatively small compared to image recognition datasets, e.g., ImageNet (14M samples) [TO, and image captioning datasets, e.g., Conceptual Captions (3.3M samples) [6) In addition, the MIMIC-CXR and IU X-Ray datasets only include Chest X-Ray images, for other types of medical images (MRL, Dermoscopy, Retinal, etc.) of other body parts (brain, skin, eye, etc.), the image-repor report pairs could be much less or even unavailable. Therefore, to relax the reliance on the paired data sets, making use of all available data, like independent image or report sets, is becoming increasingly important.
In this paper, we propose unsupervised model Knowledge Graph Auto-Encoder (KGAE), which utilizes independent sets of images and reports in training (the image and report set are separate and have no overlap). KGAE consists of pre-constructed knowledge graph, a knowledge-driver encoder and a knowledge-driven decoder. As shown in Figure I] the knowledge graph works as the shared latent space of images and reports. The knowledge- driven encoder can take either the image I or the report R as queries and project them to corresponding coordinates Gj and GR in the latent space. In this manner, since Gy and GR share the same latent space, we can use positions in latent space to measure the relationship between images and reports which narrows the gap between visual and textual domains. brief, bridge the gap between vision and language domains without training on the pairs of images and reports, we adopt the knowledge graph to create latent space encoder, which includes a common mapping function to project images and reports to the same latent space. As result, our encoder can extract the image and report knowledge representations, i.e., the knowledge related to the image and report, they (image, report knowledge) share the common latent space, which allows our model to bridge the gap between vision and language domains without the training on the pairs of image and report. Next, we introduce the knowledge-drive we estimate the parameters of the decoder by reconstructing the input report R based on GR. i.c., R Gn R nuto-encoding pipeline; In the prediction stage, we directly input G, into the trained decoder to generate the report. In this way, our approach can produce desirable reports without any labeled image-report pairs.
and propose a knowledge-drie
decoder to exploit G7 and GR to generate the report. In the training stage,
Overall, the contributions of this paper are as follows:
In this paper, we make the first attempt to conduct unsupervised medical report generation where the image-report pairs are not available. To this end, we propose the Knowledge Graph Auto-rn (KGAB). By leveraging a pre-constructed knowledge graph, we introduce the knowledge-driven encoder and decoder which are trained with independent sets of

---

images and reports. According to the experimental results, the unsupervised KGAE can even outperform several supervised approaches.
In addition to the unsupervised mode, KGAE can also be applied in semi-supervisee or supervised manner. Under the semii-sppervieed setting, by using only 60% of paired dataset, KGAE is able to achieve competitive results with current state-of-art models; Under the supervised setting, by training on fully paired datasets as in existing works, KGAE can set state-of-the-an performances on the IU X-ray and MIMIC-CXR respectively.
The analysis, both quantitative and qualitative, as well as human evaluation conducted by professional radiologists, further proves the effectiveness of our approach.
2
Related Works
Medical report generation aims to generate a relatively long paragraph to describe given medical image. It is similar to the image captioning task [5], which aims generate a sentence describe given image. In image captioning, the encoder-decoder framework [39], where the encoder [20) 12
computes visual representations for the image and the decoder [T3] 38 generates target sentence
based on the visual representations, has achieved great success [1. 31 35 44
However, instead of
only generating one single sentence, medical report generation aims to generate a long paragraph including multiple structured sentences that describe both the normal and abnormal parts [2 [5]. To this end, given the success of incoder-decoder framework on image captioning. most existing medical report generation models attempt to exploit the hierarchical LSTM (HLSTM) [6] 19] or Transformer [6] to generate an accurate, long and coherent report. However, existing models require the paired image-report datasets, which time-consuming and expensive collect. In this paper, we propose the unsupervised model Knowledge Graph Auto-Encoder (KGAE) which doesn't 't need paired images and reports.
Note that although the knowledge graph has been integrated in existing medical report generation models [54] [22), these approaches are supervised and require paired images and reports. Thus, their objectives and motivations of using knowledge graph are different from our work. detail, existing knowledge- graph based medical report generation methods aim to adopt the knowledge graph to boost the performance of supervised models. However, in our work, we aim to generate a medical report without using any coupled eepertt training pairs, i.e., unsupervised medical report generation. key challenge of unsupervised medical report generation is to bridge the gap between vision and language domains. To this end, we adopt the knowledge graph create latent space and propose knowledge-driven encoder to project image and report to the same latent space.
3 Approach
We first formulate the conventional supervised medical report generation problems; Then, we describe the proposed Knowledge Graph Auto-Encoder for unsupervised medical report generation detail.
3.1
Conventional Supervised Medical Report Generation Models
Given a medical image I, the goal is to generate a descriptive report R. Most models [5 normally include image encoder and report decoder, which can be formulated as:
16
Image Encoder: : 1 1'; Report Decoder: - R,
(1)
where E NRd denotes the image embeddings extracted by the image encoder, e.g., ResNet-50 Then, I' is used to guide the generation of the target report R in the report decoder, e.g., LSTM
12].
13 and Transformer [38]. During training, given the ground truth report for the input image, we train the encoder-decoder model by minimizing supervised training loss, e.g., cross-entropy loss. However, paired image-report (I-R) datasets are particularly labor-intensive and expensive to obtain. Therefore, in this paper, we aim to relax the reliance the paired datasets and make use of the independent sets of images and reports instead.
3.2 Knowledge Graph Auto-Encoder
As shown in Figure | and knowledge-driven decoder, which will be described in detail in the following sections.
the proposed KGAE includes a knowledge graph, a knowledge-driver encoder
Knowledge Graph The motivation of using the knowledge graph that writing medical reports usually requires particular domain knowledge [ 222. Therefore, can utilize the knowledge
3

---

graph, which models the Nomain-specinn knowledge structure. to serve as a bridge to correlate the knowledge representations of the visual and textual domains. In particular, we construct an off-the-shelf global medical knowledge graph G = (V, E) covering the common abnormalities and normalities, where V = J E IR- x is a set of nodes and E' = {ei, (i.j)i, is a set of edges. In detail, based on the report corpus, i.e., MIMIC-CXR [7.. we consider the NKG frequent clinical abnormalities (e.g., "enlarged heart size", "pleural effusion", and "bibasilar consolidation' and normalities (e.g.. "heart size is normal" and "lungs are clear") as nodes. The edge weights are calculated by the normalized co-occurrence of different nodes computed from report corpus. Please note that this knowledge graph is built purely from the training set of MIMIC-CXR, thus there is no label leakage. After that, the knowledge graph is embedded by graph embedding module, i.e., graph convolution network 32
24 which implemented as follows:
NKG
2 0 + ReLU eig W,,)),
where ReLU(-) represents the ReLU activation function and W, Rd is the learnable matrix. As result, we can acquire a set of node embeddings V = 1/.......... NNNaa } E R R.Maxd, It is worth noting that more complex graph structures could be constructed by using more large-scale medical textbooks. Therefore, our model is not limited to the currently constructed graph and could provide good basis for the research of unsupervised medical report generation for other domains.
Knowledge-driven Encoder The knowledge-driven encoder (KE) is designed to utilize the knowl- edge graph G to extract knowledge representations (Gi IR Ny xd and GR E IR-Nexdd of both image and report R to bridge the vision and the language domains, which can be formulated as:
G = KE, 9); GR = (E,(R,
(3)
In implementations, as shown in Figure 1 we first adopt the ResNet-50 12 and the Transformer 38 as the image embedding module and report embedding module to embed the image / and report R. acquiring the image embeddings I' E R and report embeddings R' E TR-Nxdd respectively. Next, to extract the knowledge representations from knowledge graph G, we adopt the attention mechanism 38 to implement the KE The motivation stems from that the attention mechanism can compute the association weights between different features and allows probabilistic many-to-many As a result, we take I' and It' as the queries, and take the knowledge graph G (i.e.. V {0/, 12. d'Kco E R Nau as the lookup matrix:
relations instead of monotonic relations, as in [38 44 22].
= (1, (Altttttnittttniioooo//// (1', V')); GR = KER(R, 9) = ([(Attention<(I1/) V'))
xWa (WWW))
(4)
Attention(2, y) = softmax
yW..
Vd
where x E Noxa y E N, Wg E Rd W1 Rdyxdy and W. Rddxdy (d= = dyy learnable parameters. F is implemented as two fully-commected (FC) layers with a ReLU in between, i.., FC-ReLU-FC. The resulted G7 N, xd and Gn E R Naxd turn out to be a set of attended (i.e., extracted) knowledge related to the image and report. The knowledge representations 9, and Gn share the common latent space which bridges the vision and the language domains. Note that the F used in KE, and KE shares the same parameters and is thus introduced to further boost the bridging capabilities of different modalities. It means that we adopt two KEs, i.e., KE, and KEg, and each KE includes an attention model as well as a common mapping function F. The only shared weights of KE, and KER are the parameters of F, and the parameters of attention models are independent.
Knowledge-driven Decoder The decoder is designed to generate the reports based on the graph representations G7 GR. For clarity, use Ga to represent the Gy and Gj during the training and testing stages, respectively. In implementations, since medical report generation requires generating a long paragraph, we choose the (three-layer) Transformer [8] as the basic module of our decoder and incorporate the proposed Knowledge-driven Attention (KA) to effectively model the long sequences.
At each decoding step i, the decoder takes the embedding of current input word = w. + 6 E 2d as input, where We and et denote the word embedding and fixed position embedding, respectively, and generate each word r2 in report R- (r1,72,... .T). which can be defined as follows:
hg Attention 2111 where Gt- = Gn. Training h1 = KA Gx. B)
; = softmax (FFnmax)FF(())WW)) (5)
G1, Testing
4

---

where FFN stands for Feed-For Forward Network in the original Transformer [38]; W E R2dx (P] is the learnable parameter (ID) denotes the vocabulary size); B represents the knowledge bank in the KA.
In particular, the knowledge-driven attention (KA) is inspired by the success of many works that attempt to augment working memory into the network, e.g., memory network. Memory network pre- serves a dynamic knowledge base for subsequent inference [37 42 43 40
46 33
For example,
37 21 and 42 43 proposed to preserve the context of document and visual knowledge solve language modeling and visual question answering, respectively. In this paper, since the knowledge representations G& are extracted from the off-the-shell global knowledge graph G. we incorporate a knowledge memory mechanism to distill and preserve the fine- greeggr medical knowledge related to report generation task, which encourages our decoder to generate more accurate and desirable reports. Specifically, we introduce the knowledge bank B = {br, b2,... 6NN } E I NB xd where Nn stands for the total number of the knowledge corresponding to report generation. As a result, given the graph representations G: E TR. xd the knowledge memory mechanism is defined as follows:
B softmax (9aB B.
(6)
Through above operation, each feature/vector in the GR R Nx xd can distill the fine-grained knowl- edge preserved in B to acquire B& E R Nu xd for accurate report generation.
Based on the above mechanism, the knowledge-driven attention in Eq.
(5) is defined as:
K = KK(h,,, 9k. B) Attention(h:, (h,, [9x: BB]) Attention (h+, [9x; softmax (G&BT) B)).
(7)
where [ denotes the concatenation operation. It is worth noting that these operations are all differentiable, thus the bank B can be learned in an end-to-end fashion.
In our subsequent analysis, we will show that the introduced knowledge memory mechanism indeed distills and preserves the desired medical knowledge, and thus boost the generation of reports.
3.3 Implementation Details
Unsupervised Training Details To train our KGAE in an unsupervised manner, instead of using the paired megge-report dataset in the conventional supervised model, we only require an image set CheXpert [14], which includes 12, X-ray images, and a separate report corpus MIMIC-CXR 17 IU X-ray which includes 222,758 2,770 = 225,528 reports
In detail, to train our knowledge-driven encoder KE, and KER (see Eq. we feed the Gy and Gr into a common multi-label classification network [54 15 trained with binary cross entropy loss for 14 common radiographic observations classification In this way, our encoder can extract the knowledge representations 91 and GR of both image and report in a common latent space, effectively bridging the vision and the language domains. To train the knowledge-drive bank B (see Eq. (5), since there are no coupled image-report pairs, we propose to reconstruct the report R based on the Gn. Therefore, through Eq. (5). taking the input report R = {r1, ......, PT as the ground truth report, we can train our approach by minimizing the cross-entropy loss:
decoder, as well as the knowledge
LCE = - Σ log (p (re (ra-1))
(8)
In this way, we can train our decoder in the R - GR - R auto-encoding pipeline.
During testing, we first adopt the knowledge-driven encoder extract the knowledge representations Gr of the test image (see Eq. Then, we directly feed Gr into the decoder to generate final report in the I - 91 + pipeline (see Eq. (5)) In this way, our approach can relax the reliance on the image-report pairs. In our following experiments, we validate the effectiveness of our approach, which even outperforms some supervised approaches.
Semi-Supervised and Supervised Training Details To further validate the effectiveness of our approach, we fine-tune the unsupervised KGAE using partial and full image-report pairs to acquire the KGAE-Semil-Supen -Suervisedd and S.peessseee respectively, where the former can evaluate the performance of our approach under limited labeled pairs for training and the latter can compare the
2 'There are no paired image-report samples between CheXpert and MIMIC- CXRICCUC--UUUUU X-ray.
3 Atelecttass. Cardiomegaly, Consolidation. Edema, Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, No Finding, Pleural Effusion, Pleural Other, Pneumonia, Pneumothorax, Support Devices.
5

---

performance of KGAE with ofate-of-the-ar supervised approaches, In the (semi-)superyae setting, given the image-report pairs, i... I-R, we first incorporate the original visual information into the knowledge representation Gr. and then train our KGAF by generating the ground truth report in the - 9r + R pipeline and minimizing the cross-entropy loss in Eq. (8). During testing, we also
follow the unsupervised setting to generate the final report in the I > 91 + R pipeline.
Experiments
We first introduce the datasets, metrics and detailed settings used for evaluation. Then, we present the evaluation of our approach under the unsupervised, semi sumissuper and supervised training settings.
4.1 Datasets, Metries and Settings
Datasets In this paper, we adopt the test sets of IU X-ray [9] and MIMIC-CXR [17 for evaluation. All protected health information (e.g., patient name and date of birth) was de-identified. In particular, the IU X-ray [9] is a widely-used public benchmark dataset for medical report generation and contains 7,470 chest X-ray images associated with 3,955 fully de-identified medical reports. Each report is composed of impression, findings and indication sections, etc. (22). Following [25), our method also focuses on the findings section as is the most important component of reports. Then, following [16] [2 25 6), we randomly select 1096-10%-005 image-report pairs of dataset to form the training- validation-lesting sets. The MIMIC-CXR [7] includes 377, 110 chest x-ray images associated with
227,835 reports. The dataset is officially split into 368,960 images (222,758 reports) for training, 2,991 images (1,808 reports) for validation and 5,159 images (3,269 reports) for testing. It is worth noting that we focus the unsupervised medical report generation, where the image-report pairs are not available, thus the image-report training pairs of both the IU X-ray and MIMIC-CXI datasets are discarded and are not used in unsupervised training stage. Only the training reports of MIMIC-CXR and IU X-ray, i.e., 222,758 + 2,770 = 225,528 reports, are used as the independent report corpus train our unsupervised model. Only under the (semi-) Jsppervsed training setting, will adopt the image-report training pairs to train our approach.
5 to calculate the widely-used natural language generation metrics, i.e., BLEU (34,, METEOR 3 and ROUGE-L (26). which measure the match between the generated reports and ground truth reports, but are not specialized for the abnormalitics in the reports. Therefore, to measure the accuracy of descriptions for clinical abnormalities, we further report clinical efficacy metrics following the work of Chen et al. 16). The clinical efficacy metries are calculated by comparing the generated reports with ground truth reports in 14 different categories related to thoracic diseases and support devices, producing the Precision, Recall and F1 scores.
Metrics To fairly compare with existing models 25 we adopt the evaluation toolkit
The size d is set to 256. For the attention mechanism in Eq. and Eq. (5). we adopt the multi-head attention 38 the number of heads n multi-head attention is set to The intermediate dimension in F, i.e., Eq. ( is set to 1024. Based on the average performance on the validation set, the Nn knowledge bank, i.e., Eq. 6 is set to 10,000. The NkG in the knowledge graph set to 200. In our nhowledge-driven encoder, the image embedding module adopts the ResNet-50 [12 pre- trained on ImageNet [10] and fine-tuned on CheXpert dataset [14] to extract the image embeddings in the shape of 7 x 7 x 2048, which will be projected to d = 256, acquiring I' E 49x9X256 i.e.,
Settings
N₂ = 49; The report embedding module is implemented by the Transformer [38] equipping with the self-attention mechanism provided in Lin al. (27; NR N₁ 49. In both unsupervised and (semi-)supervised settings (see Section (.3), the batch size is set to 16 and Adam optimizer [18] with learning rate of 1c-4 is used for parameter optimization. Before testing the IU X-ray/MIMIC- CXR, we further only employ the training data from IU X-ray/MIMIC-CXT to train the model. All re-implementations and our experiments were on 8 V100 GPUs.
4.2 Automatic Evaluation
We evaluate the performance of our approach under unsupervised. semi-supervised and supervised settings. The results are shown in Table [ Table 2 and Figure 2 We select several supervised methods, including a recently state-of-the-an art model R2Gen [6], for comparison. These models follow the encoder-decoder architecture, trained the full pairs of images and reports.
Unsupervised Setting As shown in Table D and Table 2 our unsupervised model KGAF achieves competitive results with some supervised models in both IU X-ray and MIMIC-CXR datasets, and

---

Table 1: Performance in terms of natural language generation metrics on the IU X-ray and MIMIC- CXR. B-n, M and R-L are short for BLEU-n, METHOR and ROUGE-L, respectively. Higher better in all columns. It is worth noting that the KGAE is trained in an unsupervised manner, KGAE-Semi trained 60% paired data, KGAE-Supen
and existing methods are trained with full paired data.
Ratio of Pairs
IU X
MIMIC- CXR
17
Methods
Yeur
B-1
B-2
B-3
B-4
M
R-L
B-1
B-2
B-3
B-4
M
R-L
39
100%:
0.124
0.087
0.306
0.299
0.184
0.121
0.124
0.263
AdaArt
2017
100%
0.220
0.127
0.089
0.068
0.308
0.299
0.185
0.124
0.088
0.118
0.266
Att2in
85
2017
100%
0.224
0.129
0.089
0.068
0.308
0.325
0.203
0.136
0.096
0.134
0.276
Torns
2020
100%
0.396
0.254
0.179
0.135
0.164
0.342
0.314
0.192
0.127
0.090
0.125
0.265
AMº Trans.
2020
100%
0.437
0.290
0,205
0.152
0.176
0.353
0.238
0.151
0.102
0.067
0.110
0.249
R2Gen
100%:
0.470
0.304
0.219
0.165
0.187
0.371
0.218
0.145
0.103
0.142
0.277
KGAE
0%
0.417
0.263
0.181
0.126
0,149
0.318
0.221
0.144
0.096
0.062
0.097
0.208
KGAE-Semi
60%
0.320
0.232
0.189
0.379
0.219
0.149
0.147
0.290
KGAE-Supervise
100%
0.512
0.327
0.240
0.179
0.195
0.383
0.369
0.231
0.156
0.118
0.153
0.295
1220e mDu Mingo
Table 2: Performance of automatic evalu-
030
0.1
0.40
005
ation in terms of clinical efficacy metrics, which measure the accuracy of descriptions for clinical abnormalities, on the test set of MIMIC-XXI dataset. Higher is better in all columns.
022
000
0.0
400
Ratio of Parn Xenal
Rato X-nuy!
Pargr
Ratio of Pains
MIMIC-CXR H
0.0
Methods
Precision Recall
Fl
0.0
007
0.44
2015
100%
0.249
0,203
0.204
022
AdaAm 31
2017
100%
0.268
0,186
0.181
Al1Zin 35
2017
100%
0.322
0.239
0.249
IMNIC
Ratio (MMII--XX
Up Down
2018
100%
0.320
0,231
0.238
Figure 2: Performance of re-implemented state-of-the-
A Trans. 7
2020
100%
0.197
0,145
0.3
art model R2Gen [6] and our approach on the test sets with respect to various amount of paired data used for training. The margins in different ratios are shown with
Transformer
2020
100%
0.331
0.224
0.228
R2Gen 6
2020
100%
0.333
0,273
0.276
KGAE
0.156
KGAF.Semi
Ours
60%
0.360
0,302
0.307
the polyline and the right y-axis. As we can see, our approach consistently outperforms the R2Gen.
KGAE-Sapervied
100%
0.389
0.362
0.355
even outperforms several supervised models. Specifically, on the IU X-ray dataset, Table shows that KGAE surpasses the NIC [44, AdaAtt [31] and Att2in 335 in terms of all metrics, and the Transformer 6 in terms of BLEU-1,2,3, 1,2,3. On the MIMIC-CXR dataset, Table that KGAE outperforms the M Trans. 7 in terms of Precision, Recall and F1. The competitive results prove the effectiveness of our approach in addressing the unsupervised medical report generation, and thus can generate desirable medical reports without the training the pairs of image and report.
Semi-Supervised Setting To further prove the effectiveness of our approach, we fine-tune the unsupervised KGAE using partial downstream paired mmage-report datasets (see Section resulting in the KGAE-Semi. To this end, in Figure we evaluate the performance of our approach on both IU X-ray and MIMIC-CXR datasets with respect to the increasing amount of paired data. For fair comparison, we also re-train the state-of-the-art model R2Gen (6) using the same amount of pairs. As we can see, our model outperforms the R2Gen under all ratios of paired dataset used for training. It is worth noting that the fewer the image-report pairs, the larger the margins, e.g., under the very limited pairs setting (20% of paired datasets), our approach significantly surpasses the R2Gen by 8.5% absolute BLEU-4 score on IU X-ray and 9.8% absolute Fl score on MIMIC-CRR Intuitively, since our approach can relax the reliance on the paired datasets, we can make use of available unpaired image and report data as a solid bias for medical report generation task. Table T and Table 2 further prove the effectiveness of our approach, which achieves results competitive with current the models by using only 60% of paired dataset.
Supervised Setting We fine-tune the unsupervised KGAE using full image-report pairs, acquiring the KGAF-Supervised model (see Section 3.3). Table[aand [ that KGAF-Supervised sets the new state-of-the- art results the two datasets in all metrics. Moreover, in terms of the clinical
7

---

Table 3: We invite three professional clinicians to conduct human evaluation for comparing our
approach with M Trans. 7 and R2Gen [6] under various amount of pairs for training in terms of the comprehensiveness of the generated true abnormalities and the faithfulness of the generated normalities and abnormalities to the ground truth reports. All values are reported percentage (%).
KGAE (05)
KGAE (0%)
KGAB-Semi (20%) vs. R2Gen (20%)
KGAE-Superviee (100%) vs. R2Gen (100%)
Metrics
vs. M Trans. (100%)
vs. R2Gin (100%)
Loss
Tie
Win
Loss
Tie
Win
Loss
Tie
Win
Loss
Tie
Win
Faithfulness
34
18
48
62
15
23
21
25
22
53
Comprehenssveeness
27
20
53
57
17
26
24
11
65
32
21
47
efficacy metrics, approach achieves 0.389 precision score, 0.362 recall score and 0.355 F1 score, outperforming the state-of-the model R2Gen [6]. The superior clinical efficacy scores demonstrate the capability of approach to produce higher quality descriptions for clinical abnormalities than existing models.
Overall Combining the results of unsupervised, semi- supervisd and supervised settings, the proposed KGAE can relax the dependency on the paired datasets, and thus makes the medical report generation model use the available separate image and report data to boost the performance. The advantages under the scenarios with limited labeled pairs (i.e., semi-supervised setting) show that KGAE might be applied to other medical images (MRI, Dermoscopy, Retinal, etc.), where the coupled images and reports pairs could be much less or even unavailable.
4.3 Human Evaluation
We conduct human evaluations to verify the effectiveness of KGAE in clinical practice. Specifically, to assist radiologists in clinical decision-maker makiin and reduce their workload, it important to generate accurate reports (faithfiinesss), i.e., the model does not generate normalities and abnormalities that does not exist according to doctors, with comprehensive abnormalitics (comprehensiveness)) the model does not leave out the abnormalities. Therefore, we randomly select 100 samples from the MIMIC-XRR and invite three professional clinicians to compare our approach and baselines independently. The clinicians shown in Table 3 As we can see, under the unsupervised setting, our approach achieves competitive results with the supervised model, outperforming the M Trans with winning pick-up percentages. Under the semi-)supervised setting, our method is better than state-of-the-art model R2Gen (6] in all metrics, especially for the semi-super setting (20% of paired dataset), our method substantially surpasses the R2Gin, which is in accordance with the automatic evaluation, by 69 - 21 48 and 65 24 41 points in terms of the faithfulness and comprehensivenes: metrics, respectively.
unaware of which model generates these reports. The results
Analysis
In this section, we conduct several analysis to better understand our proposed approach.
5.1 Knowledge Graph Sensitivity
In this section, to evaluate the knowledge graph sensitivity, we evaluate the performances using different knowledge graphs defined on X-Ray only, MIMIC-CXR only, both MIMIC-CXR and IU X-Ray. Table shows the results of KGAE (0%) and KGAE-Supervised (100%) on the IU X-ray dataset. As we can see, our KGAE using different knowledge graphs can consistently outperform several existing supervised models, i.c., NIC, AdaAtt. Att2in, across all metrics (Table Similarly, our KGAE-SSperving
with different knowledge graphs can also consistently outperform existing state-of-the of-the-art model, i.c.. R2Gen [T. The results prove the robustness of our proposed model to the pre-defined knowledge graph. Therefore, this work could provide good basis or starting point for the research of unsupervised medical report generation in other clinical domains such as MRI and Dermoscopy.
5.2 Ablation Study
In Table 5
we conduct quantitative analysis to better understand our approach under both the unsupervised and supervised training settings. For different ablation settings, the KGAF-Supervised is acquired by further fine-tuning the KGAE using image-report pairs.
8

---

Table 4: Analysis of the knowledge graph sensitivity, Performance of our approach using different knowledge graphs defined on IU X-Ray only, MIMIC-CRR only, both MIMIC-CXR and IU X- X-Ray.
Ratio of
It X-my
P
Methous
Knowledge Graphs
B-1
B-2
B-3
B-4
M
R-L
IU X-Ray
0,425
0.271
0.185
0.114
0.310
KGAB
0%
MIMIC CXR
0.417
0.263
0.181
0.126
0.149
0.318
MIMIC- -CCC X-Ray
0.419
0.260
0.180
IU X-Ray
0.519
0.331
0.235
0.174
0.191
0.376
KGAB-Sapervisd
100%
MIMIC-CXR
0.512
0.327
0.240
0.383
MIMIC- IU X-Ray
0.505
0.323
0.239
0.181
0.198
0.385
Table 5: We conduct the quantitative analysis under both the unsupervised setting (KGAF) and supervised setting (KGAF-Supervised) We analysis the effect of the shared F in the encoder (Eq. (4) ), and the effect of the number of the knowledge in the decoder's knowledge bank B (Eq. (6)
TU X-ray
MIMIC-CXR
17
Methods
Settings F
B
B-1
B-2
B-3
B4
M
R-L
B-1
B-2
B-3
B44
M
R-
0.291
0.170
0.086
0.120
0.095
0.064
0.070
0.174
0,352
0.227
0.154
0.109
0.133
0.313
0.192
0.118
0.079
0.045
0.085
0.187
KGAE (0%)
(c)
5,000
0.403
0,252
0.170
0.117
0,144
0.316
0.211
0.137
0.093
0.058
0.093
0.202
15,000
0.412
0.260
0.178
0.121
0.148
0.320
0.217
0.140
0.092
0.059
0.101
0,205
Full Model
/ 10,000
0.417
0.263
0.126
0.149
0.318
0.221
0.144
0.096
0.062
0.097
0.208
(e)
0.482
0,310
0.219
0.166
0,184
0.373
0.323
0,204
0.128
0.104
0,115
0.266
0.470
0.304
0.217
0.162
0.181
0.368
0.320
0.206
0.129
0.101
0.118
0.267
KGAE Supervised (100%)
(g)
5,000
0,505
0.323
0.232
0.178
0.190
0.379
0.363
0,228
0.152
0.115
0,147
0.290
15,000
0.498
0.315
0.170
0.186
0.374
0.368
0.235
0.158
0.114
0.150
0.293
Model V 10,000 0.512
0.327
0.240
0.179
0.195
0.383
0.369
0.231
0.156
0.118
0.153
0.295
As we can see, under the unsupervised setting, both the introduced shared FF and knowledge bank B significantly boost the performance, which proves our arguments and verifies the effectiveness of our approach in performing the unsupervised medical report generation.
Under the supervised setting, as shown in settings (e.f), applying shared F generates unchanged and impaired performance on the MIMIC-CXR and IU X-ray datasets, respectively. We speculate the reason is that the supervised model no longer requires the J to bridge the vision and the language domains. Therefore, the performance is unchanged on the large dataset MIMIC-CXR However, the increased parameters introduced by F might bring overfitting increase the difficulty in optimization on the small dataset IU X-ray, which somewhat hinders the performance. For the knowledge bank B, we can find that the results of setting (h) outperforms (g) on the MIMIC-CRR dataset, but underperforms (g) on the IU X-ray dataset. We speculate the reason is that the large MIMIC-CXR dataset contains more knowledge than the small IU X-ray dataset, so a larger knowledge bank may learn more knowledge of MIMIC-CXC to boost the performance while introducing more noisy knowledge into the IU X-ray dataset degrade the performance.
5.3 Qualitative Analysis
In Figure 3 we conduct the qualitative analysis to better understand our approach. As we can see, the visualization verifies the effectiveness of our knowledge-driven encoder in extracting the knowledge representations of both image and report. For the generated reports, our unsupervised KGAE generates desirable report, which correctly describes "innumerable nodules are present" and "heart size is normal". When removing the bank B, the model tends to generate plausible general reports with no prominent abnormal narratives and some repeated reports, which shows that the knowledge memory mechanism can indeed distill and preserve the desired medical knowledge to boost the generation of reports. Under the superviiee setting, the R2Gen can not well handle the medical report generation task and generates some repeated sentences of normalities (Underlined text) and fails to depict some rare but important abnormalities, i.e., "nodules"" and "xcoliasis", while our approach can generate fluent report supported by accurate abnormalities. Under the supervised setting, our approach can generate an accurate report showing significant alignment with the ground truth report. It further prove arguments and the effectiveness of our proposed approach.

---

Input Image
11
Unsupervised Setting
KGAE w/o (0%)
KGAE (0%)
no onsumottorax
No pleural effusion. No acule cardiopulmonary abnormality. There is pneumathorax pleural effusion. The lungs are
Heart size is normal. No focal airspace consolidation, Innumerable nodules are demonstrated. There is pneumothorax or pleural effusion.
nodules
dear. Heart Semi-suueerisse Setting
R2Gen (20%)
KGAE-Seml (20%)
normal
The heart size is normal. No evidence pneumothorax pleural effusion. No acute bony abnornal. The lungs are clear. There is no pneumothorax. There pneumothorax.
Heart size and mediastinal contour are within normal limits, Nodules are present. There no focal airspace consolidation. No large pleural effusion or pneumothorax.
Ground Truth
GR
Heart size and pulmonary
vascularity appear
normal limits,, Innumerable bilateral lung nodules present. These are seen diffusely throughout both lungs. No superimpased focal sirspace disease seen. No pleural effusion pnoumothorax identi- fied. Scollosis is present.
prsumothorie
Supervised Setting
nodusiss
R2Gen (100%)
KGAE-Suppeviiss (100%)
There no pneumothorax or large pleural effusion. There mild scoliosis of the spine. The lungs are free of focal ainspace disease. Heart is normal size.
and pulmonary vascularity appear within normal limits. There are small bilateral pulmonary noduless No airspace
heart
No pneumothorax or pleural effusion. Scoliosis is present.
noi
Figure 3: Reports generated by our approach and a sfate-of-the-ar model R2Gen (6]. Under the unsupervised setting, we show four nodes with top-4 attention weights in Eq. (4) to visualize Gj and GR. The Purple and Blue colored text denote the generated wrong sentences (Faithfulness) and true abnormalities (Comprehensivenes)) respectively; Underlined text denotes the repeated sentences.
Conclusions and Discussions
In this paper, we propose the Knowledge Graph Auto-Encoder (KGAB). Without any image-report pairs, KGAE can extract the knowledge representations of both image and report from the knowledge graph to bridge the visual and textual domains, and generate desirable reports by being trained in the auto-encoding pipeline. The experiments verify the effectiveness of our approach, which even exceeds several supervised models. Moreover, by further fine-tuning KGAE using paired datasets, achieve the state-of-the-ar results on two public datasets with the best human preference.
In the future, 1) since we can relax the dependency on paired data, it can be interesting to apply the KGAE to other types of medical images of other body parts, where the image-report pairs could be much less or even unavailable, to assist radiologists in clinical decision-making and reduce their workload; 2) We can replace the explicit pre-defined knowledge graph with an implicit large matrix (e.g., knowledge bank in our decoder) to improve the generalization ability of our approach.
Societal Impacts: In this paper, we target the problem of medical report generation. Although the proposed model outperforms state-of-the- th-the-rr approaches, aims to assist the radiologists instead of replacing them. For experienced radiologists, given a large amount of medical images, our model can automatically generate medical reports, the radiologists only need to make revisions rather than write new report from scratch. However, it is possible that some radiologists direct copy the generated report as the final report. Also for less experienced radiologists, they may not be able correct the errors in machine-generated reports. In order to apply the proposed model in clinical practice, it is required to add process control to avoid unintended use.
Although the proposed KGAE can work in an unsupervised manner. we still need the independent sets of medical images and medical reports which may still be difficult to collect for some types of medical images. In addition, our approach introduces the knowledge graph to bridge visual and textual domain, in the paper, we collect the frequent clinical findings as nodes and build the knowledge graph from the set of reports automatically. When applying to new domains, we need to collect a new set of clinical findings.
Limitations:
A Acknowledgments
We would like to sincerely thank the clinicians (Xiaoxia Xie, Jing Zhang, etc.) of the Harbin Chest Hospital in China for providing the human evaluation. We also sincerely thank all the anonymous reviewers and chairs for their constructive comments and suggestions that substantially improved this paper. Xu Sun and Sheng Wang are the corresponding authors of this paper.
10

---

References
[1] Anderson, X. Hc, C. Buchler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and VQA. In CVPR, 2018.
[2] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In ICLR, 2015.
[3] S. Banerjee and A. Lavie. METEOR: automatic metric for MT evaluation with improved correlation with human judgments. In EEEvallation@&CC. 2005.
[4] A. Brady, R. Ó. Laoide, Mccarthy, and R. Medermott, Discrepancy and error in radiology: Concepts, causes and consequences. The Ulster Medical Journal, 81:3 - 9, 2012.
[5] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv: 100::5504.0333 2015.
[6] Z. Chen, Y. Song. T. Chang, and X. Wan. Generating radiology reports via memory-driven transformer. In EMNLP, 2020.
[7] M. Cornia, M. Stefanini, L. Baraldi, and R. Cucchiara. Weshed-memory transformer for image captioning. In CVPR, 2020.
[8] L. Delrue, R. Gosselin, B. Ilsen, A. Van Landeghem, J. de Mey, and P. Duyck. Difficulties the interpretation of chest radiography. Comparative Interpretation of CT and Standard Radiography of the Chest, pages 27-49, 2011.
[9] D. Antani, G. R. Thoma, and C. J. McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Am. Medical Informatics Assoc., 23(2):304-300, 2016.
M. Kohli, B. Rosenman, Shooshan, L. Rodriguez, S. K.
[10] J. Deng, W. Dong, R. Socher, L.. Li, K. Li, and Li. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[11] S. Goergen, F. Pool, Turner, E. Grimm, M. N. Appleyard, C. Crock, M. C. Fahey, M. F. Fay, N. J. Ferris, S. M. Liew, et al. Evidence- based guideline for the written radiology report: Methods, recommendations and implementation challenges. Journal of medical imaging and radiation oncology, 57(1):1-7, 2013.
[12] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Comput., 98): 1997.
[14] J. Irvin, P. Rajpurkar, M. Ko, Y. Yu, S. Ciurea-llcus, C. Chute, H. Marklund, B. Haghgoo, R. L. Ball, K. S. Shpanskaya, J. Seekins, D. A. Mong. S. Halabi, K. Sandberg, R. Jones, D. B. Larson, C. Langlotz, B. N. Patel, M. P. Lungren, and A. Y. Ng. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. AAAI, 2019.
[15] B. Jing, Xie, and E, Xing. On the automatic generation of medical imaging reports. In ACL, 2018.
[16] B. Jing, Z. Wang, and P. Xing. Show, describe and conclude: On exploiting the structure information of chest x-ray reports. In ACL, 2019.
[17] A. E. W. Johnson, J. Pollard, S. J. Berkowitz, N. R. Greenbaum, M. P. Lungren, C. Deng, R. G. Mark, and S. Horng. MIMIC-CXR: A large publicly available database labeled chest radiographs. arXiv preprint arXiv: 1901.0009 2019.
[18] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In ICLR, 2014.
[19] Krause, J. Johnson, R. Krishna, and Fei-Fei. hierarchical approach for generating descriptive image paragraphs. In CVPR, 2017.

---

[20] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
[21] A. Kumar, O. Irsoy, P. Ondruska, M. lyyer, Bradbury, I. Gulrajani, Zhong, R. Paulus, and R. Socher. Ask me anything: Dynamic memory networks for natural language processing. In ICML, 2016.
[22] C. Li, X. Liang, Z. Hu, and E. P. Xing. Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In AAAI, 2019.
[23] M. Li, Wang, X. Chang, and X. Liang. Auxiliary signal-guided knowledge encoder-decoder for medical report generation. arXiv preprint TrXiv:000.033744, 2020.
[24] Y. Li, D. Tarlow, M. Brockschmidr, and R. S. Zemel. Gated graph sequence neural networks. In ICLR, 2016.
[25] Y. Li, X. Liang, Z. Hu, and E. P. Xing. Hybrid retrieval -eeeeral-gnnnrationn reinforced agent for medical image report generation. In NeurlPS, 2018.
[26] C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In ACL, 2004.
[27] Z. Lin, M. Feng, C. N. dos Santos, M. Yu, B. Xiang, B. Zhou, and Y. Bengio. A structured self-attentive sentence embedding. In ICLR, 2017.
[28] F. Liu, X. Ren, Y. Liu, H. Wang, and X. Sun. simnet: Stepwise image-topic merging network for generating detailed and comprehensive image captions. In EMNLP, 2018
[29] F. Liu, Y. Liu, X. Ren, X. He, and X. Sun. Aligning visual regions and textual concepts for mmmantic-grounded image representations. In NeurlPS, 2019.
[30] Liu, X. Ren, Y. Liu, K. Lei, and X. Sun. Exploring and distilling cross- coosl information for image captioning. In IJCAI, 2019.
[31] J. Lu, C. Xiong, D. Parikh, and R. Socher. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In CVPR, 2017.
[32] D. Marcheggiani and I. Titov. Encoding sentences with graph convolutional networks for semantic role labeling. In EMNLP, 2017.
[33] A. H. Miller, A. Fisch, Dodge, networks for directly reading documents. In EMNLP, 2016.
Karimi, A. Bordes, and J. Weston. Key-value memory
[34] K. Papineni, S. Roukos, T. Ward, and W. Zhu. BLEU: Method for automatic evaluation of machine translation. In ACL, 2002.
[35] S. Rennie, E. Marcheret, Y. Mrouch, J. Ross, and V. Goel. Self-critical sequence training for image captioning. In CVPR, 2017.
[36] R. Soricut, N. Ding, Sharma, and S. Goodman. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[37] S. Sukhbaatar, A. Szlam, J. Weston, and R. Fergus. End-to-end memory networks. In NIPS, 2015.
[38] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Kaiser, and I. Polosukhin. Attention is all you need. In NIPS, 2017.
[39] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan. Show and tell: A neural image caption generator. In CVPR, 2015.
[40] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra. Matching networks for one shot learning. In NIPS, 2016.
[41] X. Wang, Y. Peng. I. Lu, 7. Lu, and M. Summers. Tienet: Text-image embedding network for common thorax disease classification and reporting in chest x-rays. In CVPR, 2018.
12

---

[42] C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. In ICML, 2016.
[43] H. Xu and K. Saenko. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. In ECCV, 2016.
[44] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, S. Zemel, and Y. Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015.
[45] Y. Xue, T. Xu, L. R. Long, Z. Xue, S. K. Antani, G. R. Thoma, and X. Huang. Multimodal recurrent model with attention for automated radiology report generation. MICCAI, 2018.
[46] X. Yang, K. Tang, H. Zhang, and Cai. Auto-encoding scene graphs for image captioning. In CVPR, 2019.
[47] C. You, G. Li, Y. Zhang. X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, W. Cong, et al. CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (gan-circle). IEEE Transactions on Medical Imaging. 2019.
[48] C. You, N. Chen, and Zou. Knowledge distillation for improved accuracy in spoken question answering. In ICASSP, 2021.
[49] C. You, N. Chen, and Y. Zou. MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering. In IJCAI, 2021.
[50] C. You, N. Chen, and Y. Zou. Self-supervised contrastive cross modality representation learning for spoken question answering. arXiv preprint arXiv:2109.00 03381, 2021.
[51] C. You, R. Zhao, L. Staib, and J. S. Duncan. Momentum contrastive voxel-wise represen- tation learning for semi-supervised volumetric medical image segmentation. arXiv preprint arXiv:2/05.07059/ 2021.
[52] C. You, Y. Zhou, R. Zhao, L. Staib, and J. S. Duncan. Simevd: Simple contrastive voxel-wise representation distillation for semi-supervised medical image segmentation. arXiv preprint arXiv:2108.06227, 2021.
[53] J. Yuan, H. Liao, R. Luo, and J. Luo. Automatic radiology report generation based multi-view image fusion and medical concept enrichment. In MICCAI, 2019.
[54] Y. Zhang, X. Wang. Z. Xu, Q. Yu, A. L. Yuille, and D. Xu. When radiology report generation meets knowledge graph. In AAAI, 2020.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See the claimed contributions in Introduction.
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section K
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] We ensured that our paper conforms them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or as a URL)? [Yes] See our supplemental material.
13

---

(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] Error bars not reported because it would be too computationally expensive. Besides, many previous works on medical report generation did not provide error bars well.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Settings in Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See Section 4.1 Specifi-
cally, PhysioNet Credentialed Health Data License 1.5.0 (https: /pyysionet org/
0/) is used for MIMIC-CXR; Cre-
ative Commons Aurrhuuoo--baacCoommmccaa----llr 4.0
International License is used for IU X-
ray.
(c) Did you include any new assets either the supplemental material or as URL? [No] (d) Did you discuss whether and how consent was obtained from people whose data you're 'e using/curating? [Yes] See Section 4.1 The used datasets are publicly available. All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Datasets in Section 4.] All protected health information was de-identified. De-identification was performed in compliance with Health Insurance Portability and Accountability Act (HIPAA) standards in order to facilitate public access the datasets. Deletion of protected health information (PHI) from structured data sources (e.g.. database fields that provide patient name or date of birth) was straightforward. All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rZRebqtrKNs", "reviewer": "Reviewer_cvae", "summary": "The authors propose an unsupervised method for the task of medical report generation. The model can be trained using unpaired images and reports. To bridge the image and language representations, the model uses a pre-constructed knowledge graph to create a latent space and their encoder could map images and reports to the same latent space. \n\nSpecifically, 1) in the knowledge graph, the nodes are N_{KG} (= 200) frequent clinical abnormalities and normalities. The edge weights are calculated by the normalized co-occurrence of different nodes computed from the MIMIC-CXR training set. The KG embedding is trained from a graph neural network. 2) The encoder is fine-tuned on the CheXpert dataset from ResNet-50 pre-trained on ImageNet. The encoder uses the attention mechanism where queries are image or report embeddings and keys/values are KG entity embeddings. The encoder is trained with multi-label classification with binary cross entropy loss for common radiographic observations classification. 3) The decoder is a transformer-based model augmented with a memory network. The decoder is trained from scratch by re-constructing a given report. During inference time, the encoder maps an image into the latent space, and the decoder generates a report from its latent representation. \n\nThe authors evaluated their method on two datasets: IU X-ray and MIMIC-CXR. The model achieved decent performance at the unsupervised setting. Furthermore, fine-tuned using 60% paired data (60% semi-supervised), the model reaches the previous SOTA performance. Fine-tuned using 100% paired data (supervised), the model sets the new SOTA with significant gains. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Motivation**: the paper proposes an unsupervised method for medical report generation, and for this task there may not be many image-report pairs existing. The task is similar to the unsupervised summarization task. \n\n**Pros**: \n- The paper proposes an interesting idea for unsupervised image captioning by using a knowledge graph to define a latent space which bridges image and language. This could be used in other unsupervised conditional text generation tasks. \n- The paper shows extensive experiment results (overlapping metrics, clinical efficacy metrics, and human evaluation). \n- The model achieves good performance even only trained with unpair data, and shows SOTA performance when it is trained with paired data. \n\n**Cons**:\n- This method needs to rely on a pre-designed knowledge graph. This may require re-designing the graph every time when the domain is changed. \n\n**Typos**:\n- Section 3.3 Semi-Supervised and Supervised Training Details: “where the formal can evaluate the performance of our approach” (formal → former)\n\n**Some comments and questions for the authors**:\n- How sensitive is the model to the predefined knowledge graph? Some quick experiments could be ablation studies of different KGs, i.e., KGs defined on IU X-Ray only, both MIMIC-CXR and IU X-Ray. \n- One of the limitations is that the method needs to rely on a pre-designed knowledge graph. In the future study, if you are able to use a general medical KG and demonstrate that the model in this work consistently generalizes well to all tasks such as MRI and Dermoscopy with one KG, it clearly addresses the aforementioned limitation.\n- Section 4.5 Qualitative Analysis, “Under the semi-supervised setting, the R2Gen … generates some repeated sentences of normalities”. In my humble opinion, the sentence repetition problem is because R2Gen is not pre-trained on a relatively large corpus. Therefore it is clearly under-trained when only fed by 20% training data. This is not a fair comparison to KGAE-Semi (20%) which is pre-trained from 225,528 reports, so there are no repeating sentences.\n- Table 4, results with and without F are always compared without B, do you also have results to compare with F with B (10,000) and without F with B (10,000)? \n- If I understand correctly, the models are separately trained for KG embedding, encoder, and decoder. Is it possible to combine them (or some of them) for an end-to-end training? Maybe it helps backpropagate information among KG embeddings, encoder, and decoder?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See “cons” above.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hrs"}, "extra_text": {}}, {"review_id": "RpzMwJyNKuw", "reviewer": "Reviewer_AGjp", "summary": "This paper proposes a knowledge graph based method for medical report generation which can be trained by the unsupervised、semi-supervised and supervised manner. The overall framework consists of a pre-constructed knowledge graph、a knowledge-driven encoder and a knowledge-driven decoder. The authors first construct a medical knowledge graph and embed it by a graph embedding module, after that, they use the knowledge-driven encoder to obtain the knowledge representation of the image (report) by calculating the self-attention of the knowledge graph embeddings and the image (report) embeddings,  and then feed the knowledge representation into the knowledge-driven decoder to generate the report. As a result, the proposed framework outperforms several strong baselines on two datasets, i.e., IU-Xray and MIMIC-CXR.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: This paper proposes a knowledge graph based method for medical report generation and makes the first attempt to train the model in an unsupervised manner, which is certainly of value. However, there are also the following concerns/comments.  First, the authors argue that the paired image-report datasets are particularly labor-intensity and expensive to obtain, which is not very true. Indeed, the image-report pairs could be naturally generated in daily clinical routines when radiologists inspect the images and write the reports.  Although I agree with the importance of unsupervised learning, the unsupervised application scenario for medical report generation seems not as natural as general image captioning. It would be helpful if the authors could provide an example of unsupervised learning in clinic scenario.  Second, a significant claimed contribution is about the knowledge graph. As knowledge graph has been used in medical report generation, the authors need to clarify what their new contribution is in this aspect. This is not clear in the current paper as there is no experiment comparison with those knowledge-graph based medical report generation methods (such as [47, 22]) either, which seem to be close related works. Providing such information could help the reviewer better understands the originality of the paper.\n\nQuality: This paper has two main technical contributions for medical report generation: unsupervised learning and knowledge graph generation. For the first contribution, it is interesting to see such an attempt for medical report generation, which should also be encouraged. For the second contribution, this motivation of incorporating knowledge graph is persuasive and the quantitative results reported in Table 1 are quite competitive (compared with those without using knowledge graph in supervised settings).  However, the comparison fairness needs to be clarified, since the knowledge graph seems to be built upon CheXpert dataset (14-class classicification), which is also used in the supervised manner. However, this external dataset is not used in the comparing methods (except [28]). In this way, I do not think the results in Table 1 (especially for supervised methods) are comparable. If I am wrong, please correct me.  The overall writing of this paper is good, but the clarify could be further improved (see Clarity).\n\nClarity: The clarity of the paper could be further improved with more explanations in motivation (such as an example clinical scenario for unsupervised report generation), contribution (such as the contribution in the knowledge graph part), how the unsupervised model could work (see Limitations), experimental settings (see Limitation), and the fairness of the comparison (see Limitations). \n\nSignificance: This paper studies an important task, medical image report generation, and tries to explore it in an unsupervised manner (unpaired images and reports), which is for the first time, and has referential value. The generated medical reports are regularized by the constructed knowledge graph that is expected to incorporate certain domain knowledge. The proposed model was also tested under semi-supervised and supervised settings although the fairness of the comparison needs further clarification (See Limitation).\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The current paper is limited in the following aspects.\n\n1、\tThis paper lacks the clarification about the new contribution of the proposed knowledge graph with respect to those knowledge-graph based medical report generation methods (except the proposed method is an unsupervised model), such as [47, 22] mentioned in the related works. There also lacks of experimental comparison with these methods.\n\n2、\tThis paper lacks the details of how to construct the knowledge graph. For example, In line 111-114, the authors said they consider the terms of the clinical abnormalities and normalities as nodes: are those nodes are selected manually? \n\n3、\tThe authors should explain more why the unsupervised training method can work. For example, In line 173-177, the authors train a common multi-label classification network with knowledge representation of images and reports, how can this bridge the vision and the language domains? \n\n4、\tIn the experimental validation, the commonly used criteria CIDEr in medical report generation is missing.\n\n5、\tSince the IU-Xray dataset does not have an official training-test split, it is unclear if all the methods in comparison in Table 1 used the same training set on IU-Xray. \n\n6、\tThe experimental comparison reported in Table 1 may not be fair, since the knowledge graph used in the proposed method seems to build upon an external dataset CheXpert, while other methods in comparison did not make use of this dataset (except PPKED[28] that uses CheXpert to finetune ResNet-152). At least the results under supervised setting are not comparable.In the literature, there are some medical report generation models that use the auxiliary CheXpert dataset, such as [R1], however, they were not included in the comparison.\n[R1] Learning to Generate Clinically Coherent Chest X-Ray Reports, EMNLP2020\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "DkP9Pju88zO", "reviewer": "Reviewer_S5sP", "summary": "For automatic medical report generation, this work 1) builds a knowledge graph from the report corpus of MIMIC-CXR, 2) designs a knowledge-driven encoder which uses a ResNet-50 and a BiLSTM trained in a multi-label classification task, 3) designs a knowledge-driven decoder based on Transformer and memory network. The proposed model doesn’t require any image-report training pairs, in other words, it uses independent sets of images and reports in training, but it outperforms several supervised models on MIMIC-CXR and IU X-ray datasets. This method can be further extended to semi-supervised and supervised settings, and outperforms current SOTA.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper embeds the image using a trained ResNet-50, attends the embedding to the knowledge graph embeddings, and finally obtains an image knowledge representation. The same process applies to reports, through a trained BiLSTM. In this way, the paper obtains the attended knowledge related to the image and report, and they (image, report knowledge) share the common latent space, which allows us to train the model using independent sets of images and reports. \n\nThe paper is well-written with no obvious errors. The motivation and method descriptions are clear.\n\nThe knowledge graph nodes are clinical abnormalities and normalities, which can be in the form of phrases, instead of only keywords.  \n\nProposed knowledge-driven attention incorporates a knowledge bank to implement a knowledge memory mechanism, which is similar to a memory network. In this way, the input knowledge representation can distill the fine-grained knowledge preserved in the knowledge bank for better accuracy. The bank can be learned in an end-to-end manner.\n\nBesides the widely-used NLG metrics, the paper also uses clinical efficacy metrics to evaluate the accuracy of descriptions of clinical abnormalities.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It is not clear how both graphs share the common latent space without training text-image together. Further analysis is needed to prove that the knowledge encoder can indeed encode images and reports to a common latent space. \n\nThe unsupervised method, which is the main contribution of this work, does not outperform SOTA.\n\nThe knowledge graph is built from the training set of MIMIC-CXR, hence not applicable to other clinical domains. \n\nMinor:\n\nThe edge weights in the knowledge graph are based on normalized co-occurrence. Sometimes co-occurrence may not represent the correct relations, for example in the case of negations. Maybe a more sophisticated definition of edge weights can further boost the performance.\n\nPage 1, line 36. ‘labeled’ seems misleading.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "4HoUqsgw3w9", "reviewer": "Reviewer_bvmd", "summary": "This manuscript proposes an unsupervised method to generate medical report (a paragraph of textual description of medical image) for a given unlabelled medical image. The scenario is interesting but I have several major concerns for the model, as following comments.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Why use knowledge graph, is it really necessary? I saw the motivation in the beginning of Section 3.2, but it's not convincing. Please justify it better and provide a simple comparison experiment: replace the graph by CNN, see how it works. In detail, only keep the node feature matrix V (Line 110; ignore the edges E) and use CNN to process it to get embedding v_i of each ab/normalities. \n\nBTW, the notation of the knowledge graph is inappropriate. In line 110, the notation V denotes node features (each node has d-dimension feature) but not 'a set of nodes'. Please check some graph related literature and improve the notation.\n\n2. How many KE are used in this model? one or two? Are KE_I and KE_R share all parameters or partial parameters? \n\n3. The learned embedding I' and R' are not under the same distribution although they are in the same latent space. I suggest to add a mapping function (e.g., a Fully-connected layer) on I' and R', respectively, to map them into the same distribution. \n\n4. A major issue: Line 145, if the input is an image, there is no word, so what's w_t? Or will this model still appliable?\n\n5. The concept of knowledge bank (knowledge memory mechanism) is interesting to me. How to get the bank B, is it pre-defined or learnable? \n\n6. Line 173-175, how to get the label of these images or reports in multi-label classification? Suppose the dataset itself contains these labels, it still makes this whole framework a semi-supervised model? At least not a strict unsupervised framework. I understand this issue is not easy to address in this work, please discuss this limitation in Discussion. \n\n\nMinor comments:\n\n- In fig 1, since green line denotes training and red line denotes testing, it seems that the model only accepts report in training and only accept images in testing. But the model will take both images and reports in training, right? Please revise the figure accordingly. \n\n- Please add motivation of the use of knowledge graph in Introduction. I saw the motivation in Section 3.2, but it's too far behind.\n\n\n---------------Update after rebuttal------------------\nThank the authors for the detailed response. The answers addressed most of my concerns and I boosted my score from 5 to 6. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The societal impact provided by the authors is ok for me. I don't have particular suggestions.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overview:
The paper proposes a novel model for generating medical reports using medical reports and medical images that are *not* paired. The coupling between the two domains is achieved via a shared latent space grounded in Knowledge Graph, a reasonable choice since medical reports incorporate substantial amount of prior knowledge from the medical domain. The authors demonstrate the utility of their model with extensive empirical results and improve performance over SOTA.

Review:
Overall, the reviewers appreciated the novelty of the unsupervised medical report generation model. The reviewers had minor concerns which were adequately clarified by the authors in their detailed and meticulous responses. A few illustrative examples include: 
1) Concerns about sensitivity to domain on which the KG was trained. They provide detailed experimental results to show the impact.
2) Ablation study in Table 4 with higher B. The clarifications are consistent with the results in the paper.
3) Request for Cider metric. Provided.
4) Fairness of training KG on CheXpert. Addressed with KG trained w/o CheXpert.
5) Clarification of the motivation for using KG. Adequately addressed.
6) and so on.

Decision: Accept (Poster)