Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose an unsupervised method for the task of medical report generation. The model can be trained using unpaired images and reports. To bridge the image and language representations, the model uses a pre-constructed knowledge graph to create a latent space and their encoder could map images and reports to the same latent space. 

Specifically, 1) in the knowledge graph, the nodes are N_{KG} (= 200) frequent clinical abnormalities and normalities. The edge weights are calculated by the normalized co-occurrence of different nodes computed from the MIMIC-CXR training set. The KG embedding is trained from a graph neural network. 2) The encoder is fine-tuned on the CheXpert dataset from ResNet-50 pre-trained on ImageNet. The encoder uses the attention mechanism where queries are image or report embeddings and keys/values are KG entity embeddings. The encoder is trained with multi-label classification with binary cross entropy loss for common radiographic observations classification. 3) The decoder is a transformer-based model augmented with a memory network. The decoder is trained from scratch by re-constructing a given report. During inference time, the encoder maps an image into the latent space, and the decoder generates a report from its latent representation. 

The authors evaluated their method on two datasets: IU X-ray and MIMIC-CXR. The model achieved decent performance at the unsupervised setting. Furthermore, fine-tuned using 60% paired data (60% semi-supervised), the model reaches the previous SOTA performance. Fine-tuned using 100% paired data (supervised), the model sets the new SOTA with significant gains. 

main_review: **Motivation**: the paper proposes an unsupervised method for medical report generation, and for this task there may not be many image-report pairs existing. The task is similar to the unsupervised summarization task. 

**Pros**: 
- The paper proposes an interesting idea for unsupervised image captioning by using a knowledge graph to define a latent space which bridges image and language. This could be used in other unsupervised conditional text generation tasks. 
- The paper shows extensive experiment results (overlapping metrics, clinical efficacy metrics, and human evaluation). 
- The model achieves good performance even only trained with unpair data, and shows SOTA performance when it is trained with paired data. 

**Cons**:
- This method needs to rely on a pre-designed knowledge graph. This may require re-designing the graph every time when the domain is changed. 

**Typos**:
- Section 3.3 Semi-Supervised and Supervised Training Details: “where the formal can evaluate the performance of our approach” (formal → former)

**Some comments and questions for the authors**:
- How sensitive is the model to the predefined knowledge graph? Some quick experiments could be ablation studies of different KGs, i.e., KGs defined on IU X-Ray only, both MIMIC-CXR and IU X-Ray. 
- One of the limitations is that the method needs to rely on a pre-designed knowledge graph. In the future study, if you are able to use a general medical KG and demonstrate that the model in this work consistently generalizes well to all tasks such as MRI and Dermoscopy with one KG, it clearly addresses the aforementioned limitation.
- Section 4.5 Qualitative Analysis, “Under the semi-supervised setting, the R2Gen … generates some repeated sentences of normalities”. In my humble opinion, the sentence repetition problem is because R2Gen is not pre-trained on a relatively large corpus. Therefore it is clearly under-trained when only fed by 20% training data. This is not a fair comparison to KGAE-Semi (20%) which is pre-trained from 225,528 reports, so there are no repeating sentences.
- Table 4, results with and without F are always compared without B, do you also have results to compare with F with B (10,000) and without F with B (10,000)? 
- If I understand correctly, the models are separately trained for KG embedding, encoder, and decoder. Is it possible to combine them (or some of them) for an end-to-end training? Maybe it helps backpropagate information among KG embeddings, encoder, and decoder?

limitations_and_societal_impact: See “cons” above.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a knowledge graph based method for medical report generation which can be trained by the unsupervised、semi-supervised and supervised manner. The overall framework consists of a pre-constructed knowledge graph、a knowledge-driven encoder and a knowledge-driven decoder. The authors first construct a medical knowledge graph and embed it by a graph embedding module, after that, they use the knowledge-driven encoder to obtain the knowledge representation of the image (report) by calculating the self-attention of the knowledge graph embeddings and the image (report) embeddings,  and then feed the knowledge representation into the knowledge-driven decoder to generate the report. As a result, the proposed framework outperforms several strong baselines on two datasets, i.e., IU-Xray and MIMIC-CXR.
main_review: Originality: This paper proposes a knowledge graph based method for medical report generation and makes the first attempt to train the model in an unsupervised manner, which is certainly of value. However, there are also the following concerns/comments.  First, the authors argue that the paired image-report datasets are particularly labor-intensity and expensive to obtain, which is not very true. Indeed, the image-report pairs could be naturally generated in daily clinical routines when radiologists inspect the images and write the reports.  Although I agree with the importance of unsupervised learning, the unsupervised application scenario for medical report generation seems not as natural as general image captioning. It would be helpful if the authors could provide an example of unsupervised learning in clinic scenario.  Second, a significant claimed contribution is about the knowledge graph. As knowledge graph has been used in medical report generation, the authors need to clarify what their new contribution is in this aspect. This is not clear in the current paper as there is no experiment comparison with those knowledge-graph based medical report generation methods (such as [47, 22]) either, which seem to be close related works. Providing such information could help the reviewer better understands the originality of the paper.

Quality: This paper has two main technical contributions for medical report generation: unsupervised learning and knowledge graph generation. For the first contribution, it is interesting to see such an attempt for medical report generation, which should also be encouraged. For the second contribution, this motivation of incorporating knowledge graph is persuasive and the quantitative results reported in Table 1 are quite competitive (compared with those without using knowledge graph in supervised settings).  However, the comparison fairness needs to be clarified, since the knowledge graph seems to be built upon CheXpert dataset (14-class classicification), which is also used in the supervised manner. However, this external dataset is not used in the comparing methods (except [28]). In this way, I do not think the results in Table 1 (especially for supervised methods) are comparable. If I am wrong, please correct me.  The overall writing of this paper is good, but the clarify could be further improved (see Clarity).

Clarity: The clarity of the paper could be further improved with more explanations in motivation (such as an example clinical scenario for unsupervised report generation), contribution (such as the contribution in the knowledge graph part), how the unsupervised model could work (see Limitations), experimental settings (see Limitation), and the fairness of the comparison (see Limitations). 

Significance: This paper studies an important task, medical image report generation, and tries to explore it in an unsupervised manner (unpaired images and reports), which is for the first time, and has referential value. The generated medical reports are regularized by the constructed knowledge graph that is expected to incorporate certain domain knowledge. The proposed model was also tested under semi-supervised and supervised settings although the fairness of the comparison needs further clarification (See Limitation).



limitations_and_societal_impact: The current paper is limited in the following aspects.

1、	This paper lacks the clarification about the new contribution of the proposed knowledge graph with respect to those knowledge-graph based medical report generation methods (except the proposed method is an unsupervised model), such as [47, 22] mentioned in the related works. There also lacks of experimental comparison with these methods.

2、	This paper lacks the details of how to construct the knowledge graph. For example, In line 111-114, the authors said they consider the terms of the clinical abnormalities and normalities as nodes: are those nodes are selected manually? 

3、	The authors should explain more why the unsupervised training method can work. For example, In line 173-177, the authors train a common multi-label classification network with knowledge representation of images and reports, how can this bridge the vision and the language domains? 

4、	In the experimental validation, the commonly used criteria CIDEr in medical report generation is missing.

5、	Since the IU-Xray dataset does not have an official training-test split, it is unclear if all the methods in comparison in Table 1 used the same training set on IU-Xray. 

6、	The experimental comparison reported in Table 1 may not be fair, since the knowledge graph used in the proposed method seems to build upon an external dataset CheXpert, while other methods in comparison did not make use of this dataset (except PPKED[28] that uses CheXpert to finetune ResNet-152). At least the results under supervised setting are not comparable.In the literature, there are some medical report generation models that use the auxiliary CheXpert dataset, such as [R1], however, they were not included in the comparison.
[R1] Learning to Generate Clinically Coherent Chest X-Ray Reports, EMNLP2020
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript proposes an unsupervised method to generate medical report (a paragraph of textual description of medical image) for a given unlabelled medical image. The scenario is interesting but I have several major concerns for the model, as following comments.
main_review: 1. Why use knowledge graph, is it really necessary? I saw the motivation in the beginning of Section 3.2, but it's not convincing. Please justify it better and provide a simple comparison experiment: replace the graph by CNN, see how it works. In detail, only keep the node feature matrix V (Line 110; ignore the edges E) and use CNN to process it to get embedding v_i of each ab/normalities. 

BTW, the notation of the knowledge graph is inappropriate. In line 110, the notation V denotes node features (each node has d-dimension feature) but not 'a set of nodes'. Please check some graph related literature and improve the notation.

2. How many KE are used in this model? one or two? Are KE_I and KE_R share all parameters or partial parameters? 

3. The learned embedding I' and R' are not under the same distribution although they are in the same latent space. I suggest to add a mapping function (e.g., a Fully-connected layer) on I' and R', respectively, to map them into the same distribution. 

4. A major issue: Line 145, if the input is an image, there is no word, so what's w_t? Or will this model still appliable?

5. The concept of knowledge bank (knowledge memory mechanism) is interesting to me. How to get the bank B, is it pre-defined or learnable? 

6. Line 173-175, how to get the label of these images or reports in multi-label classification? Suppose the dataset itself contains these labels, it still makes this whole framework a semi-supervised model? At least not a strict unsupervised framework. I understand this issue is not easy to address in this work, please discuss this limitation in Discussion. 


Minor comments:

- In fig 1, since green line denotes training and red line denotes testing, it seems that the model only accepts report in training and only accept images in testing. But the model will take both images and reports in training, right? Please revise the figure accordingly. 

- Please add motivation of the use of knowledge graph in Introduction. I saw the motivation in Section 3.2, but it's too far behind.


---------------Update after rebuttal------------------
Thank the authors for the detailed response. The answers addressed most of my concerns and I boosted my score from 5 to 6. 
limitations_and_societal_impact: The societal impact provided by the authors is ok for me. I don't have particular suggestions.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: For automatic medical report generation, this work 1) builds a knowledge graph from the report corpus of MIMIC-CXR, 2) designs a knowledge-driven encoder which uses a ResNet-50 and a BiLSTM trained in a multi-label classification task, 3) designs a knowledge-driven decoder based on Transformer and memory network. The proposed model doesn’t require any image-report training pairs, in other words, it uses independent sets of images and reports in training, but it outperforms several supervised models on MIMIC-CXR and IU X-ray datasets. This method can be further extended to semi-supervised and supervised settings, and outperforms current SOTA.
main_review: This paper embeds the image using a trained ResNet-50, attends the embedding to the knowledge graph embeddings, and finally obtains an image knowledge representation. The same process applies to reports, through a trained BiLSTM. In this way, the paper obtains the attended knowledge related to the image and report, and they (image, report knowledge) share the common latent space, which allows us to train the model using independent sets of images and reports. 

The paper is well-written with no obvious errors. The motivation and method descriptions are clear.

The knowledge graph nodes are clinical abnormalities and normalities, which can be in the form of phrases, instead of only keywords.  

Proposed knowledge-driven attention incorporates a knowledge bank to implement a knowledge memory mechanism, which is similar to a memory network. In this way, the input knowledge representation can distill the fine-grained knowledge preserved in the knowledge bank for better accuracy. The bank can be learned in an end-to-end manner.

Besides the widely-used NLG metrics, the paper also uses clinical efficacy metrics to evaluate the accuracy of descriptions of clinical abnormalities.



limitations_and_societal_impact: It is not clear how both graphs share the common latent space without training text-image together. Further analysis is needed to prove that the knowledge encoder can indeed encode images and reports to a common latent space. 

The unsupervised method, which is the main contribution of this work, does not outperform SOTA.

The knowledge graph is built from the training set of MIMIC-CXR, hence not applicable to other clinical domains. 

Minor:

The edge weights in the knowledge graph are based on normalized co-occurrence. Sometimes co-occurrence may not represent the correct relations, for example in the case of negations. Maybe a more sophisticated definition of edge weights can further boost the performance.

Page 1, line 36. ‘labeled’ seems misleading.
review_title: null

Meta-review: Overview:
The paper proposes a novel model for generating medical reports using medical reports and medical images that are *not* paired. The coupling between the two domains is achieved via a shared latent space grounded in Knowledge Graph, a reasonable choice since medical reports incorporate substantial amount of prior knowledge from the medical domain. The authors demonstrate the utility of their model with extensive empirical results and improve performance over SOTA.

Review:
Overall, the reviewers appreciated the novelty of the unsupervised medical report generation model. The reviewers had minor concerns which were adequately clarified by the authors in their detailed and meticulous responses. A few illustrative examples include: 
1) Concerns about sensitivity to domain on which the KG was trained. They provide detailed experimental results to show the impact.
2) Ablation study in Table 4 with higher B. The clarifications are consistent with the results in the paper.
3) Request for Cider metric. Provided.
4) Fairness of training KG on CheXpert. Addressed with KG trained w/o CheXpert.
5) Clarification of the motivation for using KG. Adequately addressed.
6) and so on.

Decision: Accept (Poster)