Anatomical Structure-Aware Image Difference Graph Learning for Difference-Aware Medical Visual Question Answering
Authors: Xinyue Hu
Lin Gu
Qingyu Chen
Liangchen Liu
Kazuma Kobayashi
Qiyuan An
Zhang Mengliang
Tatsuya Harada
Zhiyong Lu
Yingying Zhu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Chest Xray
Difference Image VQA
medical dataset
Graph Neuron Networks

Abstract: To contribute to automating the medical vision-language model, we propose a novel Chest-Xray Different Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions on both diseases and more importantly, the differences between them. This is consistent with the radiologist's diagnosis practice that compares the current image with the reference before concluding the report. For this task, we propose a new dataset, namely  MIMIC-Diff-VQA including 700,821 QA pairs on 109,872 pairs of images. Meanwhile, we also propose a novel expert knowledge-aware graph representation learning model to address this problem. We leveraged the expert knowledge such as anatomical structure prior, semantic and spatial knowledge to construct a multi-relationship graph to represent the image differences between two images for the image difference VQA task. Our dataset and code will be released upon publication. We believe this work would further push forward the medical vision language model.

Under review as a conference paper at ICLR 2023
ANATOMICAL STRUCTURE-AWARE IMAGE DIFFER- ENCE GRAPH LEARNING FOR DIFFERENCE-AWARE MEDICAL VISUAL QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
To contribute to automating the medical vision-language model, we propose novel Chest-Xray Different Visual Question Answering (VQA) task. Given a pair of main and reference images, this task attempts to answer several questions both diseases and, more importantly, the differences between them. This is consis- with the radiologist's 's diagnosis practice that compares the current image with the reference before concluding the report. For this task, we propose a new dataset, namely MIMIC-Diff VQA, including 698,739 QA pairs on 109,790 pairs of im- ages. Meanwhile, we also propose novel expert knowledge-aware graph repre- sentation learning model to address this problem. We leveraged expert knowledge such as anatomical structure prior, semantic and spatial knowledge to construct rolli-relationshipp graph to represent the image differences between two images for the image difference VQA task. Our dataset and code will be released upon publication. We believe this work would further push forward the medical vision language model.
INTRODUCTION
Several recent works focus on extracting text-mined labels from clinical notes and using them to train deep learning models for medical image analysis with several datasets: MIMIC (Johnson al.,
2019), NIH14 (Wang et al., (017) and Chexpert (Irvin et al., 2019). During this arduous journal
on vision-language (VL.) modality, the community either mines per-image common disease label
[ (b)) through Natural Language Processing (NLP), or endeavors report generation (Fig) [
(c) generated from (Nguyen et al. 202I) or even answer certain pre-defined questions (Fig [
(d)).
Despite significant progress achieved on these tasks, the heterogeneity, systemic biases and sub- jective nature of the report still pose many technical challenges. For example, the automatically mined labels from reports in Fig TI. (a) is obviously problematic because the rule-based approach that was not carefully designed did not process all uncertainties and negations well (Johnson et al.) 2019). Training an automatic radiology report generation system directly match the report ap- pears to avoid the inevitable bias in the common NLP-mine thoracic pathology labels. However, radiologists tend to write more obvious impressions with abstract logic. For example, as shown in [ig[ (a), a radiology report excludes many diseases (either commonly diagnosed or intended by the physicians) using negation expressions, e.g., no, free of, without, etc. However, the artificial report generator could hardly guess which disease is excluded by radiologists.
Instead of thoroughly generating of the descriptions, VQA is more plausible as only answers the specific question. As shown in Fig. D the question could be raised exactly for "is there any pneumothorax in the image?" in the report while the answer no doubt "No". However, the ques- tions in the existing VQA dataset ImageCLEF (Abacha et al. 2019) concentrate very few general ones, such as "is there something wrong in the image? what is the primary abnormality in this im- age?", lacking the specificity for the heterogeneity and subjective texture. It often decays VQA into classification. While VQA-RAD (Lau et al. 2018) has more heterogeneous questions covering 11 question types, its 315 images dataset is relatively small.
To bridge the aforementioned gap in the visual language model, we propose a novel medical image difference VQA task which is more consistent with radiologists' practice. When radiologists make

---

Under review as a conference paper at ICLR 2023
Main image
(b) Text Mined Label from MIMIC
(a) Ground Truth Report:
Atelectasis
No change in the positioning of the left: sided chest tube. Re demonstration of fractures of the fifth and sixth left posterior ribs. A pleural line not present on the current study. Residual left lung atelectasis present. The right lung is clear, Heart size is normal No focal consolidation. No pleural line is detected to indicate residual left pneumothorax Left lobe stelectasis persists, the setting of possible splinting given the posterior rib
Fracture
Pneumothorax (incorrect)
(c) Generated Report
compared with the prior study, there new left apical pneumothorax with a small left apical pneumothoras: the left chest tube in unchanged position., there persistent elevation of the left hemidiaphragm and left basilar atelectasis the right lung is clear. there small left pleural effusion (unrelated diseases). heart normal in size.
fractures
(d) ImageCLEF VQA- questions
is there something wrong the image?
Reference image
what the primary abnormality in this image?
(e) Ground Truth Report
Comparison 07:47. Removal the left chest tube. No evidence of pneumothorax. Mild basal atelectasis Mild cardiomegaly. No pneumonia.
(1) Our medical difference VQA questions
What abnormality is seen in this image?
Atelectasis, Fracture.
is there pneumothorax this image? No Where the fracture? Posterior ribs Where the atelectasis? Left lower labe What is the level of heart size? Normal which view this image has changed compared image?
view
reference
The main image has an additional finding of fracture. The main image is missing the finding of cardiomegaly.
Figure I: (a) The ground truth report corresponding to the main(present) image. The red text repre- sents labels incorrectly classified by either text mining or generated reports, while the red box marks the misclassified labels. The green box marks the correctly classified ones. The underlined text is correctly generated in the generated report. (b) The label "Pneumothorax" is incorrectly classified because there is NO evidence of pneumothorax from the chest x-ray. (c) "There is a new left api- cal pneumothorax" - This sentence is wrong because the evidence of pneumothorax was mostly improved after treatment. However, the vascular shadow in the left pulmonary apex is not very ob- vious, so it is understandable why it is misidentified as pneumothorax in the left pulmonary apex. "there is a small left pleural effusion" - It is hard for doctor to tell if the left pleural effusion present or not. (d) The ImageCLEF VQA-MED questions are designed too simple. (e) The ref- crence(past) image and clinical report. (f) Our medical difference VQA questions are designed to guide the model to focus on and localize important regions.
diagnoses, they compare current and previous images of the same patients to check the disease's progress. Actual clinical practice follows patient treatment process (assessment diagnosis inter- vention evaluation) as shown in Figurd A baseline medical image is used as an assessment tool to diagnose clinical problem, usually followed by therapeutic intervention. Then, another follow-up medical image is retaken to evaluate the effectiveness of the intervention in comparison with the past baseline. In this framework, every medical image has its purpose of clarifying the doctor's clinical hypothesis depending on the unique clinical course (e.g., whether the pneumothorax is mit- igated after therapeutic intervention). However, existing methods can not provide a straightforward answer to the clinical hypothesis since they do not compare the past and present images. Therefore, we present chest x-ray image difference VQA dataset, MIMIC-Diff VQA, to fulfill the need of the medical image difference task. Moreover, we propose a system that can respond directly to the information the doctor wants by comparing the current medical image (main) to past visit medical image (reference). This allows to build diagnostic support system that realizes the inherently interactive nature of radiology reports in clinical practice.
MIMIC-DifT-OL. contains pairs of "main" (present) and "reference" images from the same patient's 'ss radiology images at different times from MIMIC[Johnson et al. 2019) (a large-scale public database of chest radiographs with 227,835 studies, each with unique report and images). The question and answer pairs are extracted from the MIMIC report for "main" and "reference" images with ruse- techniques. Similar to (Abacha al. 2019 Lau et al. 2018 He et al. 2020).
2

---

Under review as a conference paper at ICLR 2023
Patient assessment What is the problem?
Diagnosis & Plan
Intervention
Evaluation
How managye the problem?
A Autinng the treatment plan into actior
Did the treatment plan work?
Pneumothorax
thoracostomy
What has changed compared past image?
Is there pneumothorax in this image?
Reference (past) image
Main (present) image
purpose taking present image is to get answer these questions considering the difference from the past.
Our model can directly the doctor's of the study.
medical difference VQA model
Figure 2: Clinical motivation for Image difference VQA.
first collect sets of abnormality names and attributes. Then we extract the abnormality in the images and their corresponding attributes using regular expressions. Finally, we compare the abnormalities contained in the two images and ask questions based the collected information. We designed seven types of questions: 1. abnormality, 2. presence, 3. view, 4. location, 5. type, 6. level, and 7. difference. In our MIMIC-Diff- VQA dataset, 698, 739 QA pairs extracted from 109,790 image pairs. Particularly, difference questions answer pairs inquiry on the clinic progress and change on the "main" image compared the "reference" image as shown in Fig. ([c..
The current mainstream state-of-the-art image difference method only applies to synthetic images with small view variations, (Jhamtani & Berg- Kirk-KKirattiikk 2018 Park at. 2019) as shown Fig. However, real medical image difference comparing is very challenging task. Even the images from the same patient show large variances the orientation, scale, range, view, and nonrigid deformation, which are often more significant than the subtle differences caused by diseases as shown in Fig. 3 Since the radiologists examine the anatomical structure to find the progression of discases, similarly, learning model as shown Fig. We extract the features from different anatomical structures (for example, left lower lung, and right upper lung) nodes in the graph.
propose expert knowledge-awe aware image difference graph representation
Moreover, we construct three different relationships in the graph to encode expert knowledge: 1) Spatial relationship based on the spatial distance between different anatomical regions, such "left lower lung", "right costophrenic angle", etc. We construct this graph based on the fact that radiol- ogists prefer to determine the abnormalities based particular anatomical structures. For plc, "Minimal blunting of the left costophrenic angle also suggests tiny left pleural effusion."; 2) Semantic relationship based on the disease and anatomical structure relationship from knowledge graph (Zhang et al., 2020). We construct this graph because of the fact that diseases from the same or nearby regions could affect each other's existence. For example, "the effusions remain moderate and still cause substantial bilateral areas of basilar atelectasis."; 3) Implicit relationship to model potential implicit relationship beside 1) and 2). The graph feature representation for each image learned as a weighted summation of the graph feature from these three different relationships. The image-difference graph feature representation constructed by simply subtracting the main image graph feature and the reference image graph feature. This graph difference feature is fed into LSTM networks with attention modules for answer cencrationTToutanvaa et al. 2003).
Our contributions summarized as:
1) collect the medical imaging difference question answering problem and construct the first large-scale medical image difference question answering dataset, MIMIC-Dire VQA. This dataset comprises 109,790 image pairs, containing 698,739 question-answering pairs related to various at- tributes, including abnormality, presence, location, level, type, view, and difference.
3

---

Under review as a conference paper at ICLR 2023
Anatomical Structure-Aware Image- Dieferifeee Graph
image
Anattorical call Branuue feature Right
Anatomical feature
Disease feature
Difference feature
Rignteweriing
Relationship Graph Module
Attontion Module
General Image
lauf
What has changed in the right lung areay
M
pairs
Answer Generato
Roference image
Right
Hugrilowe lung
Ariswe
Multi- Relesonshipp Graph Module
the level of pleural effusion
Attertioon
changed from small moderate,
General image
Heart
Figure 3: Anatomical structure-aware image-difference graph for medical image difference visual question answering.
2) We propose an anatomical structure-aware image-difference model to extract the image-differenee feature relevant to disease progression and interventions. We extracted features from anatomical structures and compared the changes in each anatomical structure to reduce the image differences caused by body pose, view, and nonrigid deformations of organs.
3) We develop multi- multii-rlationsshp image- donge-ddfference graph feature representation learning method to leverage the spatial relationship and semantic relationship (extracted from expert knowledge graph) to compute image-difference graph feature representation, generate answers and interpret how the answer generated on different image regions.
METHODS
MIMIC-DiTT-VOA dataset. We introduce our new MIMIC-DifE-VQA dataset for the medical imag- ing difference question-answering problem. The MIMIC-Dif--QQA dataset is constructed follow- ing an Extract-Check-Fix cycle to minimize errors. Please refer Appendix, A.2 for the details on how the dataset is constructed. In MIMIC-Diff VQA, each entry contains two different chest x-ray images from the same patient with a question-answw answer pair. Our question design extended from VQA-RAD, but with an additional question type of "difference". In the end, the questions can be divided into seven types: 1) abnormality, 2) presence, 3) view, location, 5) type, 6) level, 7) difference. Tab. shows examples of the different question types.
Table 1: Selected examples of the different question types. See Table in Appendix A.2 for the full list.
Question type Example
Abnormality
what abnormality is seen in the left lung?
Presence
is there evidence of atelectasis in this image?
View
which view is this image taken?
Location
where in the image is the pleural effusion located?
Type
what type the opacity?
Level
what level is the cardiomegaly? what has changed compared to the reference image?
Difference
The image pairs selected from the MIMIC (Johnson et al. 2019) dataset, and each image in an image pair is from the same patient. total 109,790 image pairs are selected from MIMIC, and 698,739 739 questions constructed. We also balance the "yes" and "no" answers avoid possible
4

---

spaces Under review as a conference paper at ICLR 2023
Ornert of the disease
Pulmonary edema
Candiomagaly
Isital signa pulmonary
Congastion blood
Progresion of pulmonary edama
pushed
Ploural effusian
Pleural effusion
Cardiomegalyy
-Paural space
that uxists
lungs underneath
Widespread opacification
effusion
(a) Progression from cardiomegaly to edema and pleural effusion
evidence pleural suggesting successful
The lungs apocar small because this chest Image was supine postion rather
stancingg
treetment traumatic pneumothorax by the thest tube
evisence pulmonary
5th and rio fractures.
Chost drain tube for the treatment the traumatic preumothorax.
relatively
The central part of the lungs aeprerts clear, succestino no evidence of pulmonary edema.
sharp, supgesting
evidence massive
pleural effusion
Pessibly, previous
traumalic hemothoras
Bilateral basiaar opacty con be soon, porhops sigghly more apparent on the right
Blevstion the left disphregm and opercity stslcctavis, reflect recovery process collapsed left lung due the preumothorax.
was also drained by the tubc.
wagesting the pe presence of the bilateral or right- sided basilar atolectasis, Right sided pleural effusion can be possibtte but no apoa rent evidence bec suse the has opanity could also moditied by the supine position.
suggests
(b) Radiologist's 's annotation example 1
(c) Radiologist's 's annotation example 2
Figure 4: Illustration of the progression of diseases and two X-ray annotation examples.
bias. The statistics regarding each question type can be seen in Fig. in Appendix. The ratio between training, validation, and testing set is 8:1:1.
Problem Statement. Given an image pair (Im, I,), consisting of the main image I, and the refer- ence image I,,, and question q, our goal to obtain the answer a of the question q from image pair. In our design, the main and reference images are from the same patient.
Expert Knowledge-Aware Graph Construction and Feature Learning. As shown in the left 3 previous work on image difference question answers in the general image domain. They create paired synthetic images with identical backgrounds and only move or remove the simple objects from the background. The feature of image difference was extracted by simply comparing the feature on the same image coordinates. Unfortunately, even the medical imaging the same patients shows significant variations due to the pose and nonrigid deformation. The change of pose, scale, and range of the main image and reference image in Fig. 3 are strongly different compared the disease change (pleural effusion changed from small moderate). If image difference methods, the computed image differences related to the pose change will dominate, and the subtle discase changes will be neglected. To better capture the subtle disease changes and eliminate the pose, orientation, and scale changes, we propose to use an expert knowledge-aware image difference graph learning method by considering each anatomical structure as node and comparing the image changes in each anatomical structure just as radiologists, which consist of the following parts:
the general
Anatomical Structure, Disease Region Detection, and Question Encoding, We first extract the anatomical bounding boxes and their features f, from the input images using tre-trained Faster- RCNN on the MIMIC dataset (Ren et al. 2015 Karargyris et al., 2020). Then, we train a Faster- RCNN on the VinDr dataset (Pham et al. 2021) to detect the diseases. Instead of directly detecting diseases on the given input images, extract the features f, from the same anatomical regions using the extracted anatomical bounding boxes. The questions and answers are processed the same way as (Li al. 2019; Norcliffe-Browe Brownnrrwwnnne et al. 2018). Each word is tokenized and embedded with Glove (Pennington et al. 2014)) embeddings. Then we use a bidirectional RNN with GRU (Cho) et al. 2014) and self-attention to generate the question embedding q
5

---

Under review as a conference paper at ICLR 2023
Multi- Multi-RRelaionnship Graph Module. After extracting the disease and anatomical structure, we con- struct an anatomical structure-awan image representation graph for the main and reference image. The multi reltiosplaio graph is defined G = {V, Esp, Eses Eimp), where Exps Eses and Eimp rep- resent the edge sets of spatial graph, semantic graph and implicit graph, each vertex V; E V 1,... 2N can be either anatomical node V& [fa,*|]q] E Rdy+da, , E fa; for k 1,..., N, or disease node Va- [fax [1] E (R Sa. E fa; for k = 1..... N. representing anatomical structures or disease regions, respectively. Both of these two types of nodes are embedded with a question feature as shown in Fig. 33 d is the dimension of the anatomical and disease features. dg is the dimension of the question embedding. N represents the number of anatomical structures of one image. Because each disease feature extracted from the same corresponding anatomical region, the total number of the vertex is 2N.
We construct three types of relationships in the graph for each image: 1) spatial relationship: We construct spatial relationships according to the radiologist's practice of identifying abnormalities based on specific anatomical structures. For example, "the effusions remain moderate and still cause substantial bilateral areas of basilar atelectasis"; "Elevation of the left diaphragm and opacity in the left lower lung suggests remaining left basilar atelectasis" as shown in Fig. 4b; "The central part of the lungs appears clear, suggesting no evidence of pulmonary edema." as shown in Fig. 4c In our MIMIC-DifE MRaIC-Diff-Vie VQA dataset, questions are designed for the spatial relationship. such as "where in the image is the pleural effusion located?" as shown in Tab. [ Following previous work (Yao al.) 2018). we include 11 types of spatial relations between detected bounding boxes, such as "left lower lung". "right costophrenic angle", etc. The 11 spatial relations includes inside (class 1), cover (class2), overlap (class3), and 8 other directional classes. Each class corresponds to a 45-degree of direction. We define the edge between node i and the node j as a.j = c, where e the class of the relationship, c = 1,2,.... ..... K. K is the number of spatial relationship classes, which equals to 11. When d,j > 1, we set ar O, where d,j is the euclidean distance between the center points of the bounding boxes corresponding to the node i and node j. / is the threshold. The threshold t is defined to be (I.- + 1,)3 3 by reasoning and imitating the data given by (Li et al. 2019).
2) Semantic relationship:
The semantic relationship is based on two knowledge graphs, including an anatomical knowledge graph from (Zhang et al. 2020), as shown in Fig. Sa] and a label occur- rence knowledge graph built by ourselves, shown in Fig. Sb If there is an edge linking two labels in the Knowledge graph, we connect the nodes having these two labels in our semantic relationship graph. The knowledge graph can include abstracted expert knowledge and depicts the relationships between diseases. These relationships play a crucial role in disease diagnosis. Multiple diseases could be interrelated to each other during the course of specific disease. For example, in Fig. [4a progression from cardiomegaly to edema and pleural effusion is shown. Cardiomegaly, which refers to an enlarged heart, can start with a heart dysfunction that causes congestion of blood in the heart, eventually leading to the heart's enlargement. The congested blood would be pumped up into the veins of the lungs. As the pressure of the vessels in the lungs increases, fluid is pushed out of the lungs and enters pleural spaces causing the initial sign of pulmonary edema. Meanwhile, the fluid starts to build up between the layers of the pleura outside the lungs, i.e. pleural effusion. Pleural effu- sion can also cause compression atelectasis. As pulmonary edema continues to progress, widespread opacification in the lung can appear. These can be verified in actual diagnostic reports. For exam- plc, "the effusions remain moderate and still cause substantial bilateral areas of basilar atclectasis"; "Bilateral basilar opacity can be seen, suggesting the presence of the bilateral or right-sided basilar atelectasis" as shown in Fig. 4c
3) Implicit relationship: a fully connected graph is applied to find the implicit relationships that are relationships can be grouped as explicit relationships.
not defined by the other two graphs. Among the three types of relationships, spatial and semantic
Relation-Aware Graph Attention Network. As shown in Fig construct the multi- relationship graph for both main and reference images, and use the relation-aware graph attention network (ReGAT) proposed by (Li et al. 2019) learn the graph representation for each image, and embed the image into the final latent feature. In a relation-aware graph attention network, edge la- bels are embedded calculate the attention weights between nodes. Please refer to Appendix. AA.4 for details of the calculation. For simplicity, we use sspa()) (-), Geem(-), and (-mp(-) to represent the spatial graph module, the semantic graph module, and the implicit graph module, respectively.

---

Under review as a conference paper at ICLR 2023
Multi- Nulti-Relationship Graph Module
Implicit Graph Gimpp
Spatial Graph Spa
Semantic Graph sem
Anatomical feature
Hight Lower Lung
Right Lower Lung
Right Lower Lune
Disease feature
Question feature
Pleural effusion
Pleural effusion
Pleural effusion
ReGAT
ReGAT +
ReGAT
What has changed the right lung area?
GAP
Final imnaee feature
Figure 5: Multi-modal relationship graph module.
Given the input feature nodes V of each image, the final graph feature V can be represented as:
GAP(G spa (V) + Geem (V) + Gamp
where GAP(-) means the global average pooling. The image difference graph features Viliff is constructed by subtracting the node feature and edge feature between the main and reference image:
diff
maain
vrel ; ,N,,
(2)
where VHu Aimain vref Rd represent the final feature for i-th node of graphs. Therefore, the final graph features Vrdiff Vmain Vref E RRN xd can be obtained.
Feature Attention and Answer Generation Following previous work (Tu et al. 2021 , the gen- erated main, reference, and difference features vmain vresf wdlff are then fed into Feature Attention Module, which first calculates the attention weights of each node, then output the final feature vectors Im. I,, and Lafft- For details the calculation, please refer to Appendix [A.5) Fi- nally, by feeding the final feature vectors L,. 1,., and Laifs into the Answer Generation module, the final answer is generated. Same as (Tu et al.) 2021) s setting, the Answer Generation module composed of LSTT networks and attention modules. The Part-Of-Speech (POS) information is also considered to help generate the answers. For the calculation details, please also refer to Ap- pendix A.6 We adopt the generative language model because our questions have highly diverse answers. (c.g. the difference type question). A simple classification model is not adequate for our task.
EXPERIMENTS
Datasets. MIMIC-CK- The MIMIC-CXR dataset is large publicly available dataset of chest radiographs with radiology reports, containing 377,110 images corresponding 227,835 radiograph studies from 65,379 patients (Johnson et al. 2019). One patient may have multiple studies, and each study consists of a radiology report and one or more images. Two primary sections of interest in reports findings: natural language description of the important aspects of the image and an impression: short summary of the most immediately relevant findings. Our MIMIC-Diff-V Diff-VQAA is constructed based on the MIMIC- CXR dataset.
Chest ImaGenome. MIMIC-CXR has been added more annotations by eW al. 2021 Goldberger al. 2000) including the anatomical structure bounding boxes. This new dataset named Chest ImaGenome Dataset. We trained the Faster-RCNe to detect the anatomical structures on their gold standard dataset, which contains 26 anatomical structures.

---

Under review as a conference paper at ICLR 2023
VinDr: The VinDr dataset consists of 18,000 images manually annotated by 17 experienced radiolo- gists (Nguyen et al. 2020). Its images have 22 local labels boxes surrounding abnormalities and six global labels of suspected diseases. We used it to train the pre-trained disease detection model.
Baselines Since we are the first to propose this medical imaging difference VQA problem, we have to choose two baseline models from the traditional medical VQA task and image difference captioning task, respectively. One is Multiple Meta-model Quantifying (MMQ) proposed by (Do
al. 202T). The other is Multi-Change Captioning transformers (MCCFormers) proposed by (Qiu 2.2T
et al.
1.MMQ is one of the recently proposed methods perform the traditional medical VQA task with excellent results. MMQ adopted Model Agnostic Meta-Learning (MAML) (Finn et al., 2017) to handle the problem of the small size of the medical dataset. It also relieves the problem of the difference in visual concepts between general and medical images when finetuning.
2.MCCFormers is proposed to handle the image difference captioning task. It achieved state-of- the-art performance on the CLEngeChange dataset (Park et al., 2019). which is a famous image difference captioning dataset. MCCFormers used transformers to capture the region relationships among intra- and inter-image pairs.
3.IDC Yao al. 2022 is the state-of-the-art method performed on the general image difference captioning task. They used the pretraining technique to build the bridge between vision and lan- guage, allowing them to align large visual variance between image pairs and greatly improve the performance on the challenging image difference dataset, Birds-to-Wond -ordd (Forbes et al. (019).
Results and Discussion. We implemented the experiments on the PyTorch platform. We used an Adam optimizer with learning rate of 0.0001 to train model for 30,000 iterations at batch size of 64. The experiments are conducted on two GeForce RTX 3090 cards with a training time of 3 hours and 49 minutes. The bounding box feature dimension is 1024. Each word is represented by 600-dimensional feature vector including a 300-dimensional Glove (Pennington et al., (2014) embedding. We used BLEU (Papineni al. 2002), which popular metrie for evaluating the generated text, as the metric in our experiments. We obtain the results using Microsoft COCO Caption Evaluation (Chen et al. 2015). For the comparison with MMQ. we use accuracy as the metric.
Ablation Study. In Tab. 2 We present quantitative results of ablation studies of our method with different graph settings, including implicit graph-only, spatial graph-only, semantic graph-only, and full model with all three graphs. The studies were performed on our constructed MIMICCDiff-VQL dataset. Although the overall gain on metrics is slight, we visualized the ROIs of our model using different graphs in Appendix [A.8 to demonstrate the interpretability gain in some specific question types, such as the questions related to location, and semantic relationships between abnormalities.
Table 2: Quantitative results of our model with different graph settings performed on the MIMIC- Diff- VQA dataset
Metrics
Implicit
Spatial
Semantic
Full
Blcu-1
0.627
0.616
0.616
0.630
Bleu-2
0.543
0.530
0.533
0.546
Bleu-3
0.480
0.464
0.469
0.482
Bleu-4
0.424
0.408
0.415
0.426
Comparison of accuracy. Due to the nature of MMQ being classification model, MMQ is unable to perform on our difference question type because the diversity of answers. Also, given that the baseline model cannot take in two images simultaneously, we excluded the difference type question from this comparison. Therefore, we compare our method with MMQ only on the other six types of questions, including abnormality, presence, view, location, type, and level. These six types of questions have a limited number of answers. In order to compare with them, we use accuracy as the metric for comparison. Please note that method is still text- rext-generation model. We count the predicted answer as a True answer only when the prediction is fully matched with the ground truth answer.
8

---

Under review as a conference paper at ICLR 2023
The comparison results shown in Tab. We have refined the comparison into open-ended question results and closed-ended question (with only 'yes' or 'no' answers) results. is clear from the results that the current VQA model has difficulty handling our dataset because of the lack of focus on the key regions and the ability to find the relationships between anatomical structures and diseases. Also, even after filtering out the difference questions, there still 9,231 possible answers in total. It is difficult for classification model to localize the optimal answer from such a huge amount of candidates.
Table 3: Accuracy comparison between method and MMQ.
Question
Open
Closed
Total
MMQ
11.5
10.8
11.5
Ours 25.59
74.20
49.29
Comparison of quality of the text. For the difference question, the generated text. The comparison results between our method, MCCFormers, and IDC are shown in Tab. 4 Our method significantly outperforms MCCFormers every metric. IDC performs better but is still not comparable to ours. The CIDEr (Vedantam et al. 2015) metric, a measure of similarity between sentences,
the metrics for evaluating
reached 0 on MCCFormers, which means failed provide any meaningful keywords in the answers. This is because the generated answers of MCCFormers are almost identical, and it failed to identify the differences between images. Although MCCFormers is difference captioning method, it compares patch to patch directly. It may work well in the simple CLVER dataset. However, when comes medical images, most of which are not aligned well, the patch-to-patch method cannot identify which region corresponds to a specific anatomical structure. Furthermore, MCCFormers requires no medical knowledge graphs to find the relationships between different regions. IDC has the ability align significant variances between images. This enables them to have much higher results than MCCFormers. However, they still use pre-trained patch- patch-wise image features, which is not feasible in the medical domain with more fine- fin-grained features.
Table 4: Comparison results between our method and MCCFormers on difference questions of the MIMIC-diff-VOL dataset
Metries
MCCFormers
IDC
Ours
Blcu-1
0,214
0,525
0.641
Bleu-2
0.190
0,464
0.564
Bleu-3
0.170
0,405
0.500
Bleu-4
0.153
0.354
0.441
Visualization. Visualized results can be found in Appendix A.8
CONCLUSION
First, We proposed a medical image difference VQA problem and constructed large-scale MIMIC- DQff-VQA dataset for this task, which valuable to both the research and medical communities. Also, we designed an anatomical structure-aware multi-relation image difference graph to extract image-difference features. We trained an image difference VQA framework utilizing medical knowl- edge graphs and compared it to current state-of-the-ar methods with improved performances. How- ever, our constructed dataset currently only focusing on the common cases and ignoring special ones for example, cases where the same disease appears in more than two places. Our current Key-Info dataset can only take care of, at most, two locations of the same disease. Future work could be extending the dataset consider special cases.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Asma Ben Abacha, Soumya Gayen, Jason Lau, Sivaramakrishnan Rajaraman, and Dina Demner- Fushman. Nim at imageclef 2018 visual question answering in the medical domain. In CLEF (Working Notes), 2018.
Asma Ben Abacha, Sadid Hasan, Vivek Datla, Joey Liu, Dina Demner-Fushman, and Henning Müller. Vqa-med: Overview of the medical visual question answering task at imageclef 2019.
CLEF (Working Notes), 2, 2019.
Xinlei Chen, Hao Fang, Tsung-Y. Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint #rXYv:1504.00322 2015.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprini urXiv::406. 1078, 2014.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. 2018.
Tuong Do, Binh X Nguyen, Erman Tjiputra, Minh Tran, Quang D Tran, and Anh Nguyen. Multiple meta-model quantifying for medical visual question answering. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 64-74. Springer, 2021.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pp. 1126-1133 135. PMLR, 2017.
Maxwell Forbes, Christine Kaeser-Chen, Piyush Sharma, and Serge Belongie. Neural naturalist: generating fine-grained image comparisons. arXiv preprint arXiv: Nttv:9909.000 101, 2019.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.011 018000,0.18,..... 2016.
Any L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung Kang Peng. and H Eugene Stanlcy. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101(23)e211--222 2000.
Haifan Gong, Ricong Huang, Guanqi Chen, and Guanbin Li. Sysu-hep at vqa-med 2021: A data- centric model with efficient training methodology for medical visual question answering. Pro- ceedings http://ceur-ws. org ISSN, 1613:0073, 2021.
Xuchai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003. 10286, 2020.
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2734,2734, 2021.
Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Hluus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Harsh Jhamtani and Taylor Berg- Kirkpatrick. Learning to describe differences between pairs of similar images. arXiv preprint arXiv: 10584, 2018.
Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lun- gren, Chih-ying Deng, Roger Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):1-8, 2019.
10

---

Under review as a conference paper at ICLR 2023
Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy Wu, Matthew Tong, Arjun Sharma, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth Krupinski, et al. Eye gaze data for chest x-rays. PhysioNet https://ooi. 000.//..13022/QFD-220880 2020.
Hoeseong Kim, Jongseok Kim, Hyungseok Lee, Hyunsung Park, and Gunhee Kim. Agnostic change captioning with cycle consistency. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2095-2104, 2021.
Jin-Hwa Kim, Jaehyun Jun, and Syoung-Tak Zhang. Bilinear attention networks. Advances in Neural Information Processing Systems, 31, 2018.
Jason Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman, dataset clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1-10, 2018.
Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for vi- sual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10313-10322 2019.
Zeming Liao, Qingbao Huang, Yu Liang, Mingyi Fu, Cai, and Qing Li. Scene graph with 3d information for change captioning. In Proceedings of the 29th ACM International Conference Multimedia, pp. 5074-5082, 2021.
Fenglin Liu, Changehang Yin, Xian Wu, Shen Ge, Ping Zhang, and Xu Sun. Contrastive attention for automatic chest x-ray report generation. arXiv preprint arXiv:2106.06963 2021.
Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. Scispacy: fast and robust models for biomedical natural language processing. arXiv preprint arXiv:1902.07669 2019.
Binh D Nguyen, Thanh- Toan Do, Binh X Nguyen, Tuong Do, Erman Tjiputra, and Quang Tran. Overcoming data limitation in medical visual question answering. In International Conference on Medical Image Computing and Computer-Assiss
Intervention, pp. 522-530. Springer, 2019.
Ha Q Nguyen, Khanh Lam, Linh T Le, Hieu H Pham, Dat Q Tran, Dung B Nguyen, Dung D Le, Chi M Pham, Hang TT Tong. Diep H Dinh, et al. Vindr-exr: An open dataset of chest x-rays with radiologist's annotations. arXiv preprint 20 /5029, 2020.
Hoang T. N. Nguyen, Dong Nie, Taivanbat Badamdorj, Yujie Liu, Yingying Zhu, Jason Truong. and Li Cheng- Automated generation of accurate & fluent medical x-ray reports. CoRR, abs/2108. 12126, 2021.
Will Norcliffe- Browfn Stathis Vafeias, and Sarah Parisot. Learning conditioned graph structures for interpretable visual question answering. Advances in neural information processing systems, 31, 2018.
Ariyo Oluwasanmi, Enoch Frimpong, Muhammad Umar Aftab, Edward Y Baagyere, Zhiguang Qin, and Kifayat Ullah. Fully convolutional captionnet: Siamese difference captioning attention model. IEEE Access, 7:175929-175939 2019.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4624 4633, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pp. 1532-1543 1543, 2014.
Van-Tien Pham, Cong-Minh Tran, Stanley Zheng, Tri-Minh Vu, and Shantanu Nath. Chest x-ray abnormalities localization via ensemble of deep convolutional neural networks. In 2021 Inter- national Conference on Advanced Technologies for Communications (ATC), pp. 125-130. IEEE, 2021.
11

---

Under review as a conference paper at ICLR 2023
Yue Qiu, Shintaro Yamamoto, Kodai Nakashima, Ryota Suzuki, Kenji Iwata, Hirokatsu Kataoka, and Yutaka Satoh. Describing and localizing multiple changes with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 1971-1980, 2021.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 779-788, 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Lei Shi. Feifan Liu, and Max P Rosen. Deep multimodal learning for medical visual question answering. In CLEF (Working Notes), 2019.
Xiangxi Shi, Xu Yang, Jiuxiang Gu, Shafiq Joty, and Jianfei Cai. Finding it at another side: A viewpoint-adaptee matching encoder for change captioning. In European Conference on Com- puter Vision, pp. 574 57. Springer, 2020.
Yaoqi Sun, Liang Li, Tingting Yao, Tongyv Lu, Bolun Zheng, Chenggang Yan, Hua Zhang, Yongjun Bao, Guiguang Ding, and Gregory Slabaugh. Bidirectional difference locating and semantic consistency reasoning for change captioning. International Journal of Intelligent Systems, 2022.
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Expressing visual relation- ships via language. arXiv preprint MXXiv:1906.PP0 07689, 2019.
Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. Feature-rich part-of- speech tagging with a cyclic dependency network. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pp. 252 259, 2003.
Yunbin Tu, Tingting Yao, Liang Li, Jiedong Lou, Shengxiang Gao, Zhengtao Yu, and Chenggang Yan. Semantic relation-aware difference representation learning for change captioning. In Find- ings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 63-73, 2021.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Xiaosong Wang, Yifan Peng, Le Lu, Zhiyong Lu, Mohammadhadi Bagheri, and Ronald M Sum- mers. Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2097-2106. 2017.
Joy Wu, Nkechinyere Agu, Ismini Lourentzou, Arjun Sharma, Joseph Paguio, Jasper Seth Yao, Edward Christopher Dee, William Mitchell, Satyananda Kashyap, Andrea Giovannini, al. Chest imagenome dataset. PhysioNet, 2021.
An Yan, Xin Eric Wang, Tsu-Jui Fu, and William Yang Wang. L2c: Describing visual differences needs semantic understanding of individuals. arXiv preprint arXiv:2102.01860, 2021.
Xin Yan, Lin Li, Chulin Xie, Jun Xiao, and Lin Gu. Zhejiang university at imageclef 2019 visual question answering in the medical domain. CLEF (Working Notes), 85, 2019.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21-29, 2016.
Linli Yao, Weiying Wang, and Qin Jin, Image difference captioning with pre- pre-training and contrastive learning. arXiv preprint arXiv:2202.0423 04298, 2022.
Ting Yao. Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pp. 684-699, 2018.
12

---

Under review as a conference paper at ICLR 2023
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Li-Ming Zhan, Bo Liu, Lu Fan, Jiaxin Chen, and Xiao-Ming Wu. Medical visual question an- swering via conditional reasoning. In Proceedings of the 28th ACM International Conference Multimedia, pp. 2345-2354 2020.
Yixiao Zhang, Xiaosong Wang. Ziyue Xu. Qihang Yu, Alan Yuille, and Daguang Xu. When diology report generation meets knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 12910-12917, 2020.
Yangyang Zhou, Xin Kang, and Fuji Ren. Employing inception-resnet-v2 and bi-lstm for medical domain visual question answering. In CLEF (Working Notes), 2018.
A APPENDIX
A.1
RELATED WORK
Image Difference Captioning. The analysis of different images has been explored by a number of researchers in the general domain. The exploration of Image Difference Captioning(IDC) can be split into three stages. The beginning of the first stage is marked by the "spot-the-diff" dataset (Jham- tani Berg- Kirkpatrick, 2018), which consists of different frames of the same video surveillance footage. This is also the very first time that the IDC task has been proposed. In this phase. the researchers only focus on the pixel-level difference the same view of the same scene. (Jhamtani & (Berg-Kirkpatricl 2018) use the clusters of differing pixels as a proxy for exposing object-level differences. (Tan et al. 2019; Oluwasanmi et al. 2019) propose to employ encoder-decoder archi- tecture with attention modules to find the relationship between two images. In the second phase, the challenge was upgraded by adding different view angles of the scenes. This demands higher requirement for the analysis of different regions between images. The iconic dataset in this phase is the CLEVR-change dataset (Park et al., 2019). which comprises pictures of a group of objects(cube, sphere, and cylinder) from different views. The attention mechanism is widely employed to ad- dress this challenge (Park et al. 2019 [Shi et al. 2020
Tu et al. 2021 Sun et al. 2022 Kim et al.) propose to use an auxiliary task to enhance the primary task to generate the captions. (Liao et al. 2021> consider 3D information and adopt a scene graph to assist in localizing the changing objects. (Kim al. 2021) also introduces a CLEVR-DC dataset, which is similar to CLEVR-change, but with a larger viewpoint change. In the third phase, more fine- -raiigr visual differences are shown in the image pairs. The Birds-to- Words dataset (Forbes et al. 2019) is composed of variety of bird images, and each image pair is cap- tioned by human observers. Since the species, posture, and background of the birds in each picture vary greatly. this desires Naturalist, which is a transformer-bart based model. (Yan et al. (021) learns to understand the seman- tic structures while comparing the images by leveraging image segmentation with novel semantic pooling and using graph convolutional networks to perform reasoning. (Yao et al.) (022) embrace the pre-trate training technique to align the visual difference and the text descriptions and achieve state- of-the-art performance. We compared our method with theirs and outperformed them on our medical image difference dataset.
2021 Qiu al. (021). (Hosseinzadeh & Wang 2021
method to solve the problem. (Forbes et al. 2019) proposed Neural
Medical Visual Question Answering. Medical visual question answering aims to answer clinical questions given medical images. Medical images span a wide spectrum of modalities, including CT/MRI imaging, histopathology images, angiography, characteristic imaging appearance, ultra- sound, and radiographs (Abacha et al. 2019. Lau et al. 2018 He et al. 2020). Clinical questions mainly ask for modality, plane, organ system, and abnormality (Abacha et al.; 2019). However, large and well-annotated medical VQA datasets are still in scarcity. Previous MED-VQA methods mostly employ a two-stage procedure: 1) extract visual features medical images through a detec- tion model like RCNN (Ren et al. 2015). YOLO (Redmon et al., 2016)). and extract question features via BERT (Devlin et al. 2018); 2) attempt to aggregate visual and question features for pre- dicting the final answer (Zhan et al. 2020; Abacha et al., 2018 Zhou et al. 2018 Shi et al., 2019 Yan et al. 2019). (Lau et al. (2018) deploys existing VQA models, i.e., the stacked attention network
13

---

Under review as a conference paper at ICLR 2023
(SAN) Yang al. 2016) and the multimodal compact bilinear pooling (MCB) (Fukui al. 2016 in general domains to solve MED-VQA, (Nguyen et al. 2019) proposes to mix enhanced visual features framework with different attention mechanisms such bilinear attention network (BAN)
(Kim et al. 2018) and SAN. (Zhan et al.] 2020) proposes separate reasoning modules for differ- questions improve the reasoning medical questions. (Shi et al. (019) integrates question categories and question topic distributions to assist answer prediction. (Yan et al., (219) improves the CNN feature extractor with global average pooling boost classification. (Zhou et al., 2018) applies some image enhancement methods by reconstructing with small random rotations, offsets, scaling, and clipping to boost classification. However, the MED-VQA problem still suffers from lacking fine-grained annotations on images, massive diversity of medical data types, and medical reasoning skills from professions, and is thus far from practical.
Other related work. In the general domain, NS-VQA (Yi et al. 2018) proposed to extract regions of interest(ROIs) with predicted semantic labels and generate scene graphs based on the semantic labels using Mask-RCNN. However, NS-VQA focused on leveraging pre-designed python logical programs to process different questions and interpret(calculate) the answers. NS-VQA's answer generation greatly relies the quality of the object segmentation and labeling by pre-trained Mask- RCNN. Since NS- VQA only evaluated the performance on a simple dataset: CLVER, where all pictures have single color background, each object has fixed number of labels and the same label types. Thus, training Mask-RCNN to detect different objects on this dataset is easy to obtain an ideal performance.
(Liu et al., 2021) proposed to extract abnormality-relatee image features by constructing a pool of normal chest X-ray images and using contrastive learning distill the contrastive features between abnormal and normal images to improve the report generation performance. However, We focus on comparing the past visiting and current visiting images from the same patient track the subtle changes that happened between the two visits. Our method is clinically driven and aims at helping the radiologist validate the hypothesis of what has changed after the intervention for each patient.
A.2 MIMIC-DIFF-VOL DATASET CONSTRUCTION
164405,
Difference,
Abnomality, 152065,
Type, 24665, 4%
Presene, 54999,, 22%
Level, 62690, 9%
Location, $3545, 12%
View, 56380, 8%
Abnormality Presence mooocation Level Type Difference
Figure 6: Statistics by question types
First, we exclude the lateral view and select only the common PA or AP views for comparison. Patients with only one radiology visit are also excluded because no second image is available for differential comparison.
Next, we collect a set of abnormality names, as well as the sets of important attributes including location, level, and type, from the filtered MIMIC-CXR dataset. The lists of abnormality names and the attribute words are collected by iteratively extracting entities from random reports using ScispaCy (Neumann et al. 2019), which is SpaCy model for biomedical text processing. Then manually go through all the extracted entities that haven' been added to the collection list and select the common keywords that appear frequently. Then add these selected keywords to the collection
14

---

Under review as a conference paper at ICLR 2023
Table 5: Full list of examples for each question type.
Question type example
what abnormalities are seen in this image?
what abnormalities are seen in the flocation]?
Abnormality
there evidence of any abnormalities in this image?
is this image normal?
is there evidence of [abnormality] in this image?
Presence
there [abnornality?
is there [abnormality] in the [location]?
which view is this image taken?
View
is this PA view?
is this AP view?
where in the image is the /abnormality) located?
where is the [abnormality]?
Location
the [abnormality] located on the left side right side?
is the [abnormality] in the [location]?
Level
what level the [abnormality]?
Type
what type is the /abnormality??
what has changed compared to the reference image?
Difference
what has changed in the [location] area?
lists of abnormality names and attributes. During this process, different variants that represent the same abnormality are also recorded. Next, for each study, we use regular expressions to localize the abnormality names as well as their variants to detect attribute words near these detected abnormal- ities. (Here, "study" represents single patient visit. Please refer to Section 3 for more context.) Meanwhile, by going through the extracted entities, we manually select the deyroods//xpressions that indicate negation information to localize the negative findings. i.c. cases where the abnormality does not exist. After updating the keyword lists, we keep repeating this Extract-Check-FF cycle until minimum mistakes found.
Thereafter, a dataset of single studies can be constructed accordingly. We call this dataset the Key- Info dataset. As shown in Fig. [ for each study, the Key-Info dataset provides information on every positive finding and its corresponding attributes as well as the negative findings. The full lists of the selected abnormality names and the attribute words shown in Tab. alnd Tab. 7 respectively. The "posterior location" attribute represents the location information that appears after the abnormality keyword in sentence.
Study pairing and question generation
When the abnormality database is constructed, questions for study pairs can be generated accord- ingly. The examples of each question type are shown in Tab. Each image pair contains the main image and a reference image, which are extracted from different studies. Among all the question types, the first six question types are for the main image only, and the difference question is for both images.
A.2.1 DATASET VALIDATION
To further verify the reliability our constructed dataset, 3 human verifiers were assigned 1200 ran- dom sampled question-answer pairs along with the reports and evaluated each sample by annotating "correct" or "incorrect". Finally, the accuracy of the evaluation achieved 97.33%, which is accept- able for training neural networks. Tab. B shows the evaluation results of each verifier. It proves that our approach of constructing a dataset in an Extract-Check Fix cycle works well in ensuring that the constructed dataset has minimum mistakes.
15

---

Under review as a conference paper at ICLR 2023
finding
abnormality name
level
Positive
findings
view
location
Posterior location
Study
Negutive finding
Negative findings
Figure 7: Structure of one study in the Key-Info dataset
A.3 ANATOMICAL STRUCTURE DETECTION
The Anatomical structure detection results shown in Tab. 9 We also tested trained Faster- RCNN on examples diffuse and non-diffuse diseases to verify the robustness of our detection model. We select interstitial edema as the diffuse disease. Diffuse diseases accounted for 5 out of total of 200 examples, and non-diffuse diseases accounted for 195 examples.
A.4 RELATION-AWARY GRAPH ATTENTION NETWORK
For the implicit relationship, each updated node V; Rd in the final graph can be calculated as below:
V: W
(II-M (
Σ aij W))
(3)
JEN;
where Nr is the neighborhood set of the node i, W E [dx(d))dd) is the projection matrix, the dimension of the final node feature, a is the activation function, || represents concatenating the output of the M attention heads, W E Rd The attention weights 0g between the node and node j consider the similarity between node pairs and the relations between the corresponding region locations. The calculation for aij can be formulated as:
0b exp (a!t)
aij ΣK o.y exp (az))
(4)
a (Uv)) (Vv,)
(5)
L max (0, w fo(b.j))
(6)
where U, V e (Rdx(dy+d,) are projection matrices. by is the relative geometry feature between node i and j. and can be calculated by [log( | log( w; logg | w.. logg( (*) loog( A )}, fo is function that embeds the 4-dimensional relative geometry feature into d-dimensional, E Rd vector that transforms the feature into scalar weight. The bounding box coordinates, widths, and heights of the node i and j can be represented by z;, Fy Yi. W;, Wy, h, and ht.
Spatial and semantic graphs, which also be called explicit graphs, can be seen as directed graphs. The updating rule considers the relation directions between node pairs and the labels of the edges. The formulation of single attention head shown below:
16

---

Under review as a conference paper at ICLR 2023
Table 6: Applicable disease names
id
Disease names
0
pleural effusion
1
atelectasis
2
cardiomegaly
3
enlargement of the cardiac silhouette
4
edema
5
hernia
6
vascular congestion
7
hilar congestion
8
pneumothorax
9
heart failure
10
lung opacity
11
pneumonia
12
tortuosity of the descending aorta
13
scoliosis
14
gastric distention
15
hypoxemia
16
hypertensive heart disease
17
hematoma
18
tortuosity of the thoracic aorta
19
contusion
20
emphysema
21
granuloma
22
calcification
23
pleural thickening
24
thymoma
25
blunting of the costophrenic angle
26
consolidation
27
fracture
28
pncumomediastinum
29
air collection
v. a Σ Osj W dir(t.)) + brad(T)))
(7)
JEN.
exp ((Uv;)T V Vair(i.) Qla(i.j))
0ij Σ JEN, exp ((Uv.) Vdir(i.) + Clab(i.)))
(8)
where dir(i,)) represents the direction goes from node i to j. j) is the label assigned to the edge (i,j), W dir(i.)) Vair(i.) E (dx(tr are projection matrices, biab(i.j) Cua(i..)) E Rd are bias terms. The multi-head attention can calculated similarly by concatenating the output features and adding projection matrix W E Rdx Md.
A.5 FEATURE ATTENTION MODULE
The generated main image features Vinai, reference image feature and the difference feature / are then fed into the Feature Attention Module, which is similar to the two modules in (Tu al., 202T) called Cross-semantic Relation Measuring block(CSRM) and Prior Knowledge- guided Change Localizer. In the Feature Attention module, we first calculate the prior knowledge C²,, and C for the main image and the reference image, respectively. Take C% for example, the calculation process is shown below.
Om =0==6( < Symainee + mainn we + 6c)
(9)
Am a (mmain Wa mmmin W b²)
(10)
17

---

Under review as a conference paper at ICLR 2023
Table 7: Attribute keywords for level, location(pre), location(post), and type.
Attribute
level
location(pra pre)
location(post)
type
moderate
mid to lower
the lower lobe the upper lobe the middle lobe the left lung base the right lung base the lung bases the left base the right base
interstitial
acute
left
layering
mild
right
dense
small
retrocardiac
parenchymal
moderately
pericardial
compressive
severe
bibasilar
obstructive
moderate to large moderate to severe mild to moderate moderate to large minimal
bilateral
linear
basilar
plate-like
apicolateral basal
the right upper lung the left upper lung the right middle lung the left middle lung the right mid lung
patchy
ground-glass
left-sided
calcified
mildly
lobe
scattered
subtle
lung
interstitial
massive
area
the left mid lung
focal
minimally
right-sided apical pleural
the right lower lung
multifocal
increasing
the left lower lung
multi-focal
decreasing
the right upper lobe
loculated
minor
upper
the left upper lobe the right middle lobe the left middle lobe the right mid lobe
hazy
trace
lower
middle
mid
rib
the left mid lobe
the right lower lobe the left lower lobe the left upical area the left apical region the right apical area the right apical region the apical region the apical area the right mid to lower lung the left mid to lower lung the medial right lung base the medial left lung base the upper lungs
the lower lungs
the upper lobes
the lower lobes
the right mid to lower hemithorax
the soft tissues the right midlung the left midlung
C¹ = Am 3 Cm
where Cm E R2Nxd is the "candidate change", Am E R2Nxd the "attention gate", We We Wa We E Raxd be, b0,00 ba Rd, @ represents the element-wise multiplication, & is the 9: tanh function, is the sigmoid function. C₁ can be calculated similarly.
18

---

Under review as a conference paper at ICLR 2023
Table 8: Evaluation results by human verifiers(todo))
Verifier
# of examples
# of correctness
Accuracy
Verifier 500
475
95%
Verifier 1000
989
98.9%
Verifier 3
200
193
96.5%
Total 1700
1657
97.4%
Table 9: Anatomical structure detection results. Precision represents when the Intersection over Union(loU) threshold is set to 0.5.
Category
Precision (IoU =0.5)
Diffuse disease Precision
non-diffuse Precision
right lung
97.561
100
97.569
right lower lung zone
88.774
100
88.72
right costophrenic angle
68.294
80.198
68.178
left upper lung zone
95.075
100
95.114
left hilar structures left hemidiaphragm
90.092
100
90.479
76.314
72.277
76,908
left clavicle sve right atrium right upper lung zone
83.859
100
83,808
87.734
100
87.729
80.54
100
80.457
95.55
100
95.562
right hilar structures right hemidiaphragm left mid lung zone
92.887
100
92.877
83.766
100
83.7
87.251
100
87.774
left apical zone trachea
92.654
100
93.312
89.421
100
89.444
aortic arch
90.951
100
90.957
cardiac silhouette
90.643
100
90.812
carina
45.423
30.693
45.821
right mid lung zone
91.776
100
91.754
right apical zone left lung
93.352
100
93.354
96.695
100
96,942
left lower lung zone
82.534
100
83.01
left costophrenic angle
63.95
80.198
64.321
right clavicle
87.384
100
87.393
upper mediastinum
95.216
100
95.26
cavoatrial junction
66.503
100
65.747
Then, guided by the prior knowledge, we calculate the attention weights am and a, for the main image and the reference image respectively. The formulations shown below:
am (FC)(RELU( ([vmain, Viiif, C(,,))
(12)
0,- = a (RFC,(Re) (Vvro, Vdiff; ; C.I)))
(13)
where [:] represents concatenation, FC represents fully-connected layer, represents the sig- moid function.
After obtaining the attention weights am E RRN and a, E IRNN the final image feature vector In and I, for the main image and the reference image can be calculated as follows:
2N
Om, main
(14)
2N
1,
ar,
ref
(15)
i=1
19

---

Under review as a conference paper at ICLR 2023
Lung
Mediazirum
Cardiomeguly
Acctle enkaggement
Ealazged PA
Mediastinal shift
Pleural etfiaion
Pleuri thickening
Pocumothoras
Calcification
Rib fracture
Clavicle fincture
Consolidation
Intilration
Pulmonary fibrosia
NoduleiMdass
Lung cavvity
Atcloctasis
ILD
Lung specity
Other losion
Lung cyst
Emphyysum
(a) Anatomical knowledge graph (compared Zhang 22200, our new added disease types are annotated by red.)
Fracture
Cardiomegaly
Pleural effusson
Edema
Consolidation
Pneumonia
Support devices
Enlarged cardiomediastinum
Lung opacity
Pleural other
Lung lesion
Pneumothorax
Atelectasis
(b) Co-occurance knowledge graph
Figure 8: Knowledge graphs
where 0m E R2N and a, R2N are the attention weights. The difference vector is accordingly
computed as:
laifs -m 1,
(16)
A.6 ANSWER GENERATION
Dynamic Feature Generation. At each time step 4, we first calculate the attention weights o
which for calculating the intermediate dynamic feature (t) in the next step. The (t) can be calculated follows:
dgn
ReLU(W,,, [eLUUWalller bar) (1)) [v; he ((t-111
(17)
(18)
( LST (h (1)) (1)) (1 (1tt-)))
(19)
(8) Softmar (W.,. n!9 + bar)
(20)
where Wa,, Waz bays bas are learnable parameters, LSTM, is a LSTM network used as attention weights generator, Aa is the output of the LSTM, at the time step t, A (t-1) is the output of the answer generator R.STM, the time step 1, which will be explained in more detail later.
Then, the intermediate dynamic feature
(t) dgn can then be calculated as follows:
(t)
Σ
O (t)
(21)
dym
20

---

Under review as a conference paper at ICLR 2023
where i E (bef, fif. aft).
Before calculating the final dynamic feature L'n POS feature p()) needs to be obtained first. The
POS feature is calculated from the hidden embedding of the answer he (t(11) from the last time step. The calculation can be formulated below:
h ReLU(W, p1
((-1)111) h/ + +bp.) p..)
(22)
w (2)
Softmax(Mi (W pz /1 + bpz)
(23)
pl = E
(24)
where W p1: W. pa pp p2 are learnable parameters, Eyp is learnable POS embedding matrix.
(() dgn and the POS feature p()), we can calculated the final
With the intermediate dynamic feature dynamic feature L (1) dyn'
Be = oWW,, [ReLUU(.,(pp! - 1 dym (gg)))) 1)))
(25)
T. L. = 3 1ygn
(26)
where the range of 3e [0, 1], the value of it indicates how much the visual information will be used in the answer generation part.
Answer generator. The answer generated by an LSTM network word by word. The initial word at time step 0 is the slarl >> token.
(t) [E Lygn
(27)
(0(0) (h (c(0) 11 (1:s-1))
(28)
(t) Softmare (Wa (Whl(.) be)
(29)
where word embedding layer, Elu(t-11]] is the word embedding for the word (t--1), Wey b. are learnable parameters.
We adopt the generative language model because our questions have highly diverse answers. (e.g. the difference type question). A simple classification model is not adequate for our task.
A.7 OTHER RESULTS
We evaluated our proposed multi-relationship graph for the general chest X-ray image classification- based VQA problem (14 diseases) and compared it to state of art method SYSU-HCP (Gong al. 2021]. the best team the ImageCLEF VQA-Med 2021 task. As shown in Tab. I0 We use AUC as the metric because answering abnormality questions be considered multi-label classification problem. Our model achieved significant improvement compared to the state-of-the-arr disease classification performance.
We show the results of our model on each question type in Tab. II It is worth noting that, Bleu short,
and Bleu 4 tend to have low scores. This is because the answers most of the questions except for the "difference" questions. For abnormality questions, 72% of the answers have less than or equal to 2 words; for location questions, 79% of the answers have less than or equal to 2 words; 93% of level questions have one-word answers.
A.8 VISUALIZATIONS
To prove the improvement the interpretability of our model by adding the spatial and semantic graphs, visualize the ROIs of our model using different graphs and demonstrate the predictions. As shown in Fig. b)), our model using the only implicit graph missed the regions important for the question and failed to interpret the correct answer. In contrast, shown in Fig. al), with the help
21

---

Under review as a conference paper at ICLR 2023
Table 10: Results of classificattin--usse VQA problem.
Answer
SYSU-HCP
Ours
Pneumothorax
0.806
0.876
edema
0.737
0.893
lung lesion
0,665
0.843
no
0.537
0.951
lung opacity
0.605
0.859
ateletasis
0.645
0.868
plcural other
0.858
0.845
support devices
0.769
0,924
pneumonia
0.715
0.833
pleural effusion
0.796
0.938
enlarged cardiomediastinum
0.725
0.828
yes
0.545
0.944
consolidation
0.708
0.819
cardiomegaly
0,688
0.892
fracture
0.664
0.871
total (micro)
0.792
0.934
total (macro)
0.697
0.879
Results of each question type. "." represents not applicable because no ground truth answer has enough words to trigger the corresponding Bleu metric.
Table 11:
Question type
Bleu
Bleu 2
Bleu 3
Bleu 4
Abnormality
0.482
0.333
0.197
0.109
Presence
0.801
View
0.948
0.941
Location
0.525
0.364
0.210
0.144
Level
0.496
0.101
0.068
Difference
0.641
0.564
0.500
0.441
22

---

Under review as a conference paper at ICLR 2023
of the spatial relationship graph, our model succeeded in finding the critical region and delivering the correct answer.
Fig. 10 demonstrates a similar scenario abnormality question. our model using only the implicit graph detected only one abnormality, atelectasis, missed pleural effusion, and lung opacity. However, with the help of the semantic relationship graph, which emphasizes the relationship be- tween pleural effusion, atelectasis, and lung opacity, our full model detected all three abnormalities and provided the correct answer.
Q: Is the consolidation located on the left side or right side?
GT answer: left side
left lung
left upper lung zone
right hilar
left mid lung
cavbatrial rial junction
hemidiaphragm
(a) Full model
(b) Model with implicit graph only Prediction: right side
Prediction: left side
Figure 9: ROIs Visualization comparison between implicit graph and all graphs on location type question.
Q: what abnormalities are seen in this image?
GT answer: pleural effusion, atelectasis, lung opacity
left upper lung 201e
hemmleephaagm
herridlaph agm
costophrenic angle
(a) Full model
(b) Model with implicit graph only Prediction: atelectasis
Prediction: pleural effusion,
atclectasis, lung opacity
Figure 10: ROIs Visualization comparison between implicit graph and all graphs on abnormality type question.
As shown in Fig. when asking about pleural effusion, which is an abnormality that happens in the lower lung when there is excess fluid between the layers of the pleura outside the lungs, our method highlighted the corresponding regions (left lower lung). Also, by focusing on these regions, method can accurately determine the change in the level of pleural effusion between the
23

---

Under review as a conference paper at ICLR 2023
main and reference image. In Fig. 12 method also highlighted cardiac silhouette, this could be because of the strong semantic relationship between cardiomegaly and pleural effusion as mentioned in Section. and Fig. 4a
Reference
Main
structures
heridiaaptaggm
Question: what has changed in the left lung area?
GT Answer: the level of plcural effusion has changed from moderate small to small.
Prediction: the level of pleural effusion has changed from moderate small small.
Figure 11: Visualization example
Reference
Main
left lung
cardiec silhouette
loft lower lung zone
lung
Question: what has changed in the left lung area?
GT Answer: the level of pleural effusion has changed from moderate to small.
Prediction: the level of pleural effusion has changed from moderate to small.
Figure 12: Visualization example
24


Reviews:
[{"review_id": "egO7Djc7_zA", "reviewer": "Reviewer_eNEk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "First, the paper introduces a dataset for \"difference-aware medical visual question answering\". It has chest x-ray pairs from the same patient, with associated questions and answer labels for a VQA task. It is scraped from the existing MIMIC dataset. Though this is similar to \"image captioning\" task, the authors argue that medical imaging comparisons have distinct challenges, so this should be considered a distinct task. \n\nSecond, the paper proposes a model for solving this proposed task. This model uses a detector to identify anatomical regions used for feature extraction, which are then used in a knowledge graph model. \n", "summary_of_the_review": "The dataset is a useful contribution, and I hope it is used more widely, especially by general VQA researchers. This contribution would be stronger if it is verified more thoroughly (as discussed above). I have a concerns regarding baselines and clarity (see above). \n", "strength_and_weaknesses": "Strengths:\n- The dataset is useful and well-motivated. The authors argue that radiologists really do evaluate images by considering differences in pairs of images from the same patient. \n- Furthermore, the dataset is real with clear future applications. Hopefully it can be used as a challenging benchmark beyond existing toy examples in similar tasks like 'image captioning'.\n- The method far outperforms the baseline. \n\nWeaknesses: \n- Concern about dataset correctness: Fig.1 and introduction paragraph 1 argue that prior VQA datasets in medical images have significant errors in their labels due to issues with their text mining strategies (for example, issues with rule-based systems). But paragraph 3 claims that similar approaches are used to generate the new dataset. It is good that some human verification was done (as explained in the appendix), but only 300 randomly chosen pairs were checked out of from 700,821. Also, the dataset is not yet released, so reviewers cannot check. This contribution would be stronger by a more thorough verification.\n- For the method, the ablations in Table 2 suggest marginal gains for the spatial and semantic graphs, so the model could be simplified. \n- MCCFormer (Qiu 21) is the only image difference captioning baseline, and it does badly on the proposed task. The authors argued this is because they work patch-wise and so cannot handle mis-aligned pairs (which isn't a problem for datasets like CLEVER-change). But the related work discuss \"birds-to-words\" dataset which has misaligned pairs, and it mention several works that perform well on it: (Yao 22) and (Yan 21). Therefore these works may be appropriate baselines to include. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "XrjMHkmPA1p", "reviewer": "Reviewer_FdAD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes a medical image difference VQA problem and constructs a MIMIC-Diff-VQA dataset from the existing MIMIC-CXR dataset. To perform the task, this work proposes a knowledge-aware graph and a multi-relationship graph, where the former takes each anatomical structure as a node in the graph and compares the image differences in each anatomical structure, and the latter adopts the spatial relationship, semantic relationship, and implicit relationship to compute the image-difference graph representations. The experiments on the constructed MIMIC-Diff-VQA dataset show that the proposed approach ", "summary_of_the_review": "Overall, the paper has provided a new task and a new large-scale dataset, but the usefulness of the proposed task should be clarified. Meanwhile, the novelty of this paper should be discussed. Before the rebuttal, I tend to reject this paper.", "strength_and_weaknesses": "Strengths:\n1. The proposed task/problem is new. The constructed dataset seems to be useful for the community.\nsound. \n2. The proposed approach can outperform several constructed baseline models.\n\nWeaknesses:\n1. The usefulness of the proposed task should be clarified.\n- What benefits can be brought by this task in clinical? And why?\n- Otherwise, I may think that this work just applies the techniques/tasks in computer vision (i.e., NS-VQA and image difference captioning) to medical images without understanding the clinical significance of the problem being addressed. \n- More importantly, the proposed task is very similar to the existing NS-VQA task [1], but the authors neither cite nor discuss this paper.\n- Meanwhile, no mention is made of the type of abnormalities for which this approach/task is suitable. In fact, since a chest X-ray may show many pre-existing conditions which may be anatomical abnormalities (e.g. scoliosis) but not the target of the chief complaint for which the report was ordered, it is important to quantify the type of abnormalities.\n\n2. The novelty of this work is limited.\n- First, as mentioned in Weakness 1, the proposed medical image difference VQA task is very similar to the existing NS-VQA task [1], but the authors neither cite nor discuss this paper. Therefore, it's important to clarify what benefits can be brought by this task in clinical. \n- The proposed approach and the arguments, i.e., \"This is consistent with the radiologist’s diagnosis practice that compares the current image with the reference before concluding the report\" in the Abstract and \"When radiologists make a diagnosis, they usually compare the main one with a reference image to find their differences\" in the Introduction, are very similar to the existing work [2].\n- Figure 7(b) is directly borrowed from [3]. The authors should note it.\n\n3. The motivation is unclear.\n- The motivation for introducing the reference image is unclear.\n- Does there are any relationships between the designed questions and the reference image? \n- What are the contributions of the incorporated reference image?\n- What are the reasons to adopt a reference image to answer the questions?\n- Regarding the proposed approach, why does this work adopt the language model? What is the motivation?\n\n4. The experiments should be improved.\n- This paper only includes one baseline model for comparison. I strongly recommend the authors attempt to re-implement existing state-of-the-art methods in NS-VQA on the constructed dataset to show the effectiveness of the proposed approach.\n- Could you provide the results of the approach on different types of question types/abnormalities?\n-  I am also interested in knowing if the approach brings errors. And what type of errors does it bring? And why?\n\n5. Some details of the pre-processing of the constructed dataset are missing.\n- As the main contribution of this paper, it's necessary to provide more details about the dataset.\n- Rule-based methods: What is the motivation for adopting rule-based methods? How to adopt the rule-based methods to construct the dataset?\n- The MIMIC-CXR dataset includes two views of chest x-rays, how do you process them?\n- Does the proposed approach and task could be applied to the lateral view image? \n- Could you give some examples of the lateral view image?\n\n6. (Minor) This paper seems to be written in a bit of a hurry, and there is a lot of scope for improving the presentation of the paper. I strongly recommend the author check and revise the text carefully. \n\n\nReferences:\n\n[1] Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018.\n\n[2] Contrastive Attention for Automatic Chest X-ray Report Generation. In ACL, 2021.\n\n[3] When Radiology Report Generation Meets Knowledge Graph. In AAAI, 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3CCVq1Y4G6n", "reviewer": "Reviewer_NUDS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new dataset and visual question answering task to find differences between chest xray images of the same patient taken at two different points in time. \n\nFurthermore, the authors develop a graph representation learning model that uses domain knowledge to solve this task, and provide a baseline for future extensions. This model extracts anatomical/disease features from both images, and passes them as input to a multi-relationship graph module that extracts a final image representation. The features for both images can then be subtracted and used in a language model that generates the final answer. \n\n ", "summary_of_the_review": "I believe this paper could be relevant for the ICLR community:\n1. The paper introduces a interesting large-scale dataset for vision/language medical tasks.\n2. The model developed by the authors is novel.\n\nI therefore argue for acceptance.", "strength_and_weaknesses": " \n\n**STRENGTHS** \n\n1. The authors prepared a large scale dataset that can be used to advance multi-modal models working on medical text/images \n\n2. Most ML applications for chest xray analysis only consider a single chest xray image, which is limiting as lots of medically relevant information can be obtained comparing studies taken at different point in times. This model on the other hand is able to mimic what radiologists do when comparing studies to assess the progression of a disease.  \n\n3. How to embed medical domain knowledge in ML models is still a very open research questions. The knowledge-aware graph representation used in this paper is a novel interesting idea.\n\n \n\n**WEAKNESSES** \n1. Given the heterogeneity of radiological reports, they are quite hard to process with rule-based methods such as the ones used by the authors. I would have like to see a larger validation set than the 100 examples used in the paper. \n2. The model relies on anatomical parts segmentation, which is quite challenging in chest x-rays (with large opacities/effusions anatomical parts might not even be visible). What is the classification performance of your anatomical parts classifier? How does it perform on diffuse diseases? \n3. Have you tried to use classifiers instead of a language model for some of the tasks 1-6? It would be interesting for example to see a classification baseline for the \"abnormality\" tasks, that would give an idea of how good your model is.\n4. What are the training times of the model? \n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject