Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: First, the paper introduces a dataset for "difference-aware medical visual question answering". It has chest x-ray pairs from the same patient, with associated questions and answer labels for a VQA task. It is scraped from the existing MIMIC dataset. Though this is similar to "image captioning" task, the authors argue that medical imaging comparisons have distinct challenges, so this should be considered a distinct task. 

Second, the paper proposes a model for solving this proposed task. This model uses a detector to identify anatomical regions used for feature extraction, which are then used in a knowledge graph model. 

strength_and_weaknesses: Strengths:
- The dataset is useful and well-motivated. The authors argue that radiologists really do evaluate images by considering differences in pairs of images from the same patient. 
- Furthermore, the dataset is real with clear future applications. Hopefully it can be used as a challenging benchmark beyond existing toy examples in similar tasks like 'image captioning'.
- The method far outperforms the baseline. 

Weaknesses: 
- Concern about dataset correctness: Fig.1 and introduction paragraph 1 argue that prior VQA datasets in medical images have significant errors in their labels due to issues with their text mining strategies (for example, issues with rule-based systems). But paragraph 3 claims that similar approaches are used to generate the new dataset. It is good that some human verification was done (as explained in the appendix), but only 300 randomly chosen pairs were checked out of from 700,821. Also, the dataset is not yet released, so reviewers cannot check. This contribution would be stronger by a more thorough verification.
- For the method, the ablations in Table 2 suggest marginal gains for the spatial and semantic graphs, so the model could be simplified. 
- MCCFormer (Qiu 21) is the only image difference captioning baseline, and it does badly on the proposed task. The authors argued this is because they work patch-wise and so cannot handle mis-aligned pairs (which isn't a problem for datasets like CLEVER-change). But the related work discuss "birds-to-words" dataset which has misaligned pairs, and it mention several works that perform well on it: (Yao 22) and (Yan 21). Therefore these works may be appropriate baselines to include. 


clarity,_quality,_novelty_and_reproducibility: Clarity: 
- Some errors in grammar (e.g. sentences beginning "while the current state-of-the-art ...." is an unfinished sentence). 
- Future versions should place citations inside brackets. Currently it's a little tough to read. 
- I found the model architecture and decsription difficult to follow. 
- In the section `Multi-Relationship Graph Module`, a number of symbols in the graph definition are left undefined, e.g. $\varepsilon_{sp}$, and the symbols in the definition for vertices. (They can be inferred from context, but they should still be defined in the text). 
- CIDEr metric not explained.
- The difference between contributions 2 and 3 is a bit unclear. 
- [minor] "MCCFormer" sometimes written as "MMCFormer"
- [minor] opening quote brackets (") are the wrong way around
- [minor] spelling errors: "relaiton", "perfomrd",
- [minor] should bold "ours" results in Table 3.

Novelty: 
- The dataset novelty is good. 
- Unsure about novelty in the knowledge-graph model component

Reproducibility: 
- Unfortunately I cannot yet access the code or dataset, so I cannot rate reproducibility as a strength.
summary_of_the_review: The dataset is a useful contribution, and I hope it is used more widely, especially by general VQA researchers. This contribution would be stronger if it is verified more thoroughly (as discussed above). I have a concerns regarding baselines and clarity (see above).
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work proposes a medical image difference VQA problem and constructs a MIMIC-Diff-VQA dataset from the existing MIMIC-CXR dataset. To perform the task, this work proposes a knowledge-aware graph and a multi-relationship graph, where the former takes each anatomical structure as a node in the graph and compares the image differences in each anatomical structure, and the latter adopts the spatial relationship, semantic relationship, and implicit relationship to compute the image-difference graph representations. The experiments on the constructed MIMIC-Diff-VQA dataset show that the proposed approach 
strength_and_weaknesses: Strengths:
1. The proposed task/problem is new. The constructed dataset seems to be useful for the community.
sound. 
2. The proposed approach can outperform several constructed baseline models.

Weaknesses:
1. The usefulness of the proposed task should be clarified.
- What benefits can be brought by this task in clinical? And why?
- Otherwise, I may think that this work just applies the techniques/tasks in computer vision (i.e., NS-VQA and image difference captioning) to medical images without understanding the clinical significance of the problem being addressed. 
- More importantly, the proposed task is very similar to the existing NS-VQA task [1], but the authors neither cite nor discuss this paper.
- Meanwhile, no mention is made of the type of abnormalities for which this approach/task is suitable. In fact, since a chest X-ray may show many pre-existing conditions which may be anatomical abnormalities (e.g. scoliosis) but not the target of the chief complaint for which the report was ordered, it is important to quantify the type of abnormalities.

2. The novelty of this work is limited.
- First, as mentioned in Weakness 1, the proposed medical image difference VQA task is very similar to the existing NS-VQA task [1], but the authors neither cite nor discuss this paper. Therefore, it's important to clarify what benefits can be brought by this task in clinical. 
- The proposed approach and the arguments, i.e., "This is consistent with the radiologist’s diagnosis practice that compares the current image with the reference before concluding the report" in the Abstract and "When radiologists make a diagnosis, they usually compare the main one with a reference image to find their differences" in the Introduction, are very similar to the existing work [2].
- Figure 7(b) is directly borrowed from [3]. The authors should note it.

3. The motivation is unclear.
- The motivation for introducing the reference image is unclear.
- Does there are any relationships between the designed questions and the reference image? 
- What are the contributions of the incorporated reference image?
- What are the reasons to adopt a reference image to answer the questions?
- Regarding the proposed approach, why does this work adopt the language model? What is the motivation?

4. The experiments should be improved.
- This paper only includes one baseline model for comparison. I strongly recommend the authors attempt to re-implement existing state-of-the-art methods in NS-VQA on the constructed dataset to show the effectiveness of the proposed approach.
- Could you provide the results of the approach on different types of question types/abnormalities?
-  I am also interested in knowing if the approach brings errors. And what type of errors does it bring? And why?

5. Some details of the pre-processing of the constructed dataset are missing.
- As the main contribution of this paper, it's necessary to provide more details about the dataset.
- Rule-based methods: What is the motivation for adopting rule-based methods? How to adopt the rule-based methods to construct the dataset?
- The MIMIC-CXR dataset includes two views of chest x-rays, how do you process them?
- Does the proposed approach and task could be applied to the lateral view image? 
- Could you give some examples of the lateral view image?

6. (Minor) This paper seems to be written in a bit of a hurry, and there is a lot of scope for improving the presentation of the paper. I strongly recommend the author check and revise the text carefully. 


References:

[1] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018.

[2] Contrastive Attention for Automatic Chest X-ray Report Generation. In ACL, 2021.

[3] When Radiology Report Generation Meets Knowledge Graph. In AAAI, 2020.
clarity,_quality,_novelty_and_reproducibility: The clarity should be improved - there are several typos in the paper. I recommend the author check and revise the text carefully. The novelty is limited: 1) The usefulness of the proposed task should be clarified; 2) An existing work Contrastive Attention in the medical image field and NS-VQA task should be discussed. The reproducibility of this paper may be good, the key resources (i.e., code and data) will be available upon publication as promised in the Abstract. However, the experiments should be improved.
summary_of_the_review: Overall, the paper has provided a new task and a new large-scale dataset, but the usefulness of the proposed task should be clarified. Meanwhile, the novelty of this paper should be discussed. Before the rebuttal, I tend to reject this paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces a new dataset and visual question answering task to find differences between chest xray images of the same patient taken at two different points in time. 

Furthermore, the authors develop a graph representation learning model that uses domain knowledge to solve this task, and provide a baseline for future extensions. This model extracts anatomical/disease features from both images, and passes them as input to a multi-relationship graph module that extracts a final image representation. The features for both images can then be subtracted and used in a language model that generates the final answer. 

 
strength_and_weaknesses:  

**STRENGTHS** 

1. The authors prepared a large scale dataset that can be used to advance multi-modal models working on medical text/images 

2. Most ML applications for chest xray analysis only consider a single chest xray image, which is limiting as lots of medically relevant information can be obtained comparing studies taken at different point in times. This model on the other hand is able to mimic what radiologists do when comparing studies to assess the progression of a disease.  

3. How to embed medical domain knowledge in ML models is still a very open research questions. The knowledge-aware graph representation used in this paper is a novel interesting idea.

 

**WEAKNESSES** 
1. Given the heterogeneity of radiological reports, they are quite hard to process with rule-based methods such as the ones used by the authors. I would have like to see a larger validation set than the 100 examples used in the paper. 
2. The model relies on anatomical parts segmentation, which is quite challenging in chest x-rays (with large opacities/effusions anatomical parts might not even be visible). What is the classification performance of your anatomical parts classifier? How does it perform on diffuse diseases? 
3. Have you tried to use classifiers instead of a language model for some of the tasks 1-6? It would be interesting for example to see a classification baseline for the "abnormality" tasks, that would give an idea of how good your model is.
4. What are the training times of the model? 
 
clarity,_quality,_novelty_and_reproducibility:  
The introduced dataset is the first of its kind, and the ideas introduced in the model are novel to the best of my knowledge. 

The paper is quite clear to read, although I could spot several typos. The citation style used in the paper is often confusing when citations are in the middle of a sentence, as the author's name seems being part of the sentence. For example in page 5 you say: "Each word is tokenized and embedded with Glove Pennington et al. (2014) embeddings.". It would be better to use \citep in latex to get "Each word is tokenized and embedded with Glove (Pennington et al, 2014) embeddings." 
summary_of_the_review: I believe this paper could be relevant for the ICLR community:
1. The paper introduces a interesting large-scale dataset for vision/language medical tasks.
2. The model developed by the authors is novel.

I therefore argue for acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper received borderline ratings overall, with some divergence across reviewers, and was therefore additionally discussed in a virtual meeting with reviewers. Reviewers overall appreciated the value of the new dataset. However, multiple reviewers expressed concerns with the proposed model and experiments, even after the author rebuttal. In particular, although the model was proposed to address medical VQA, it was not sufficiently convincing that the main components of the model that contributed to performance (as presented in ablation studies) were targeted to address medical image-specific challenges, and that it makes sense to distinguish comparison of this model and evaluation from general VQA models and evaluation. Additionally, it was not clear that the work has sufficiently compared against strong VQA baselines that could also be adapted to the medical VQA task, including several references suggested by the reviewers. The author rebuttal partially but did not completely satisfy reviewer concerns about this. Overall, after the reviewer meeting, two reviewers chose to keep their recommendation as reject, while the third reviewer with the highest score clarified that their score was primarily based on evaluation of the dataset and they did not oppose rejection due to the modeling and evaluation concerns. I agree with the majority reviewer opinion that the weaknesses of this paper outweigh the strengths in the current form and do not recommend acceptance at this time.
justification_for_why_not_higher_score: The modeling and evaluation concerns are too significant for acceptance.
Decision: Reject