Pretraining Vision-Language Model for Difference Visual Question Answering in Longitudinal Chest X-rays

Authors: yeongjae  cho
taehee  kim
sungzoon  cho
t  cho
h  kim
s  shin
d  cho
  shin
Venue: MIDL_io_2024_Conference
Published: 2024-01-29 06:38:03

Abstract: Difference visual question answering (diff-VQA) is a challenging task that requires answering complex questions based on differences between a pair of images. This task is particularly important in reading chest X-ray images because radiologists often compare multiple images of the same patient taken at different times to track disease progression and changes in its severity in their clinical practice. However, previous works focused on designing specific network architectures for the diff-VQA task, missing opportunities to enhance the model's performance using a pretrained vision-language model (VLM). Here, we introduce a novel VLM called PLURAL, which is pretrained on natural and longitudinal chest X-ray data for the diff-VQA task. The model is developed using a step-by-step approach, starting with being pretrained on natural images and texts, followed by being trained using longitudinal chest X-ray data. The longitudinal data consist of pairs of X-ray images, along with question-answer sets and radiologist’s reports that describe the changes in lung abnormalities and diseases over time. Our experimental results show that the PLURAL model outperforms state-of-the-art methods not only in diff-VQA for longitudinal X-rays but also in conventional VQA for a single X-ray image. Through extensive experiments, we demonstrate the effectiveness of the proposed VLM architecture and pretraining method in improving the model’s performance.

Main paper: 
 introduction: 
 Assessing longitudinal medical images is an essential daily task for radiologists. It involves reading and comparing multiple images of the same patient taken at different times. The purpose of this assessment is to track disease progression and changes in its severity. For example, radiologists review longitudinal CT images to determine whether a patient has any malignant nodules with increased sizes, which is a common indication of lung cancer Figure 1: The training process of the PLURAL model. We adopted a Transformer-based network that takes a single image and an instruction as an input (see blue and green boxes) in the first stage. In the second and third stages, we added a new input branch for a past image (see red boxes) to utilize longitudinal chest X-ray data, including chest X-ray reports and difference VQA data. (Callister et al., 2015) . In the case of tuberculosis, they regularly compare longitudinal chest X-ray images to monitor the effect of treatment and check for any improvement in pulmonary findings (Lee et al., 2020) .Visual question answering (VQA) is a challenging task that involves answering a set of complex questions about an image. With the increasing popularity of vision-language models (VLMs) (Alayrac et al., 2022; OpenAI, 2023; Liu et al., 2023) , VQA is gaining attention as an effective method to interpret medical images (Lin et al., 2023) , assist visually impaired individuals (Bigham et al., 2010) , and so on. In particular, a recent study (Hu et al., 2023) has attempted to solve a VQA task based on longitudinal images, detecting the changes between two X-ray images taken at different time points. This task, called difference VQA (diff-VQA), is especially helpful in assisting radiologists in their clinical practice.In previous works (Qiu et al., 2021; Yao et al., 2022; Hu et al., 2023) , specific network architectures were designed to solve the diff-VQA task effectively because this task requires an AI model to differentiate between common and distinguished features in two images. However, customizing neural networks has caused them to miss valuable opportunities to enhance the model's performance by utilizing VLMs pretrained on a vast number of texts and images, in general, (Chen et al., 2022; Wang et al., 2022; Bao et al., 2022) and/or medical domains (Zhang et al., 2023; Moor et al., 2023; Li et al., 2023) . As a result, no studies have proposed an effective pretraining pipeline for diff-VQA or investigated the impact of pretraining settings, such as pretraining steps and data composition.In this study, we introduce a novel VLM called PLURAL, which stands for pretrained vision-language model based on natural and longitudinal chest X-ray data for diff-VQA.The training process of this model involves using a Transformer-based network (Wang et al., 2022) that has been pretrained on natural images and texts (see Stage 1 in Figure 1 ) and then optimizing it using longitudinal chest X-ray data (see Stage 2 and 3 in Figure 1 ). The longitudinal data consists of pairs of longitudinal X-rays, QA sets, as well as X-ray reports that provide detailed descriptions of the changes in lung abnormalities and diseases over time. This step-by-step training approach allows the PLURAL model to benefit from the transferred knowledge of a pretrained VLM, while also taking advantage of the rich temporal information in the X-ray reports.We have demonstrated the effectiveness of the PLURAL method by comparing its performance in the diff-VQA data with that of previous state-of-the-art (SOTA) methods. We conducted several ablation studies to confirm the impact of each pretraining component, such as natural data pretraining, longitudinal data pretraining, inputs of past images, and some sections in the X-ray reports (i.e., Findings and Impression). Through extensive experiments, we showed that the PLURAL model outperformed the other SOTA methods not only in diff-VQA for longitudinal X-rays but also in conventional VQA for a single X-ray image. 
method: 
 The training process of the PLURAL model includes a series of three training stages as follows (see Figure 1 ): pretraining using natural images and texts, pretraining using longitudinal chest X-ray data, and finetuning using diff-VQA data. To train the network using single or longitudinal images according to the different stages, we adopted a Transformerbased network (Wang et al., 2022) in the first stage and modified its inputs in the second and third stages (see Stage 2 and 3 in Figure 1 ).The VLM architecture we used in the second and third stages is shown in Figure 2 . Based on a Transformer encoder-decoder architecture proposed by Wang et al. (2022) , we designed our own architecture by attaching a new input branch for a past image (i.e., input branch for i past in Figure 2 ). This design allowed the model to take two longitudinal images as input simultaneously.Both past (i past ∈ R H×W where H = 384 and W = 384) and current (i cur ∈ R H×W ) images are encoded using the same image encoder (E img (i past ) ∈ R Ĥ× Ŵ ×E where Ĥ = 24, Ŵ = 24, and E = 1024), which is a ResNet101 (He et al., 2016) , and were flattened(v past = F (E img (i past )) ∈ R N ×E where N = Ĥ • Ŵ = 576).Then, the flattened feature vector for each image is linearly projected (P img (v past ) ∈ R N ×D where D = 768), followed by the addition of the positional (p img enc ∈ R N ×D ) encoding. After that, to differentiate the time points of the two input images, we separately added the time encoding for each image (t past enc ∈ R N ×D for past; t cur enc for current) as trainable parameters:v past trans = P img (v past ) + p img enc + t past enc (1) v cur trans = P img (v cur ) + p img enc + t cur enc (2)The text instruction (s txt ) was encoded using the text encoder (l txt = E txt (s txt ) ∈ R Nt×D where N t is the length of an instruction sequence; byte-pair encoding (Sennrich et al., 2015) and word embedding used), followed by the addition of the positional encoding (p txt enc ∈ R Nt×D , l txt trans = l txt + p txt enc ). Finally, all the processed features of the input images and instruction were aggregated, producing the concatenated input feature (i trans ∈ R (2•N +Nt)×D ) for the Transformer encoder:EQUATIONThe Transformer encoder (6 layers) subsequently encoded the combined feature (i tras ), and the Transformer decoder (6 layers) disentangled the output of the Transformer encoder to generate an answer to the instruction. The loss function was a cross-entropy loss as follows:L = - |y| j=1 log P θ (y j |y <j , i past , i cur , s txt ),where L is a cross-entropy loss, y is an output of the Transformer decoder, and θ is trainable model parameters.In the first stage of the training process of PLURAL, the network was pretrained using various tasks (eight tasks including image captioning, image-text matching, VQA, etc.) and datasets of natural images and texts (e.g., CC12M dataset (Changpinyo et al., 2021) , SBU dataset (Ordonez et al., 2011) , COCO dataset (Lin et al., 2014) , etc.) as described in the work of Wang et al. (2022) . Since all the tasks in the first stage consider a single image (or no image) as an input, in this stage, the network architecture included neither an input branch nor time encoding for a past image (i.e., no input branch for i past in Figure 2 ). The loss function was defined only to consider a single input image as follows: L = -|y| j=1 log P θ (y j |y <j , i cur , s txt ), which is different from the loss function used in the second and third stages. In our implementation, we utilized a pretrained model weight available on the web1 (OFA base model with 184M parameters) instead of training the model from scratch by ourselves. Detailed training parameters (e.g., a learning rate, an optimizer, and a dropout ratio) can be found in Wang et al. (2022) .In the second stage, we continued to pretrain the model from the first stage using longitudinal chest X-ray data. To do that, we added a new input branch for a past image (see Stage 2 in Figure 1 ), modifying the network architecture as shown in Figure 2 .We utilized two sets of longitudinal data for pretraining, consisting of longitudinal chest X-rays with QA pairs (MIMIC-Diff-VQA dataset (Hu et al., 2023) ; see Section 2.5 for details) and radiologist's reports (MIMIC-CXR dataset (Johnson et al., 2019) ). Two main sections, Findings and Impression, in the chest X-ray reports were used for pretraining. We introduced separate instructions for each section, "What does the image describe?" for the Findings section and "What is the summary of the image?" for the Impression section. These sections include valuable information about the longitudinal changes in chest X-ray images, which can serve as important clues to solve the diff-VQA task effectively. For example, as shown in the 'Label' box in Figure 2 , they imply many indications for the additional findings, such as consolidation, opacity, and pneumonia (see underlying words in Figure 2 ) that are the key answers to the difference VQA question (e.g., "What has changed compared to the reference image?").In the last training stage of PLURAL, the model from the second stage was finetuned using only the diff-VQA data without any modification of the network architecture (see Stage 3 in Figure 1 ). Details of training configurations are summarized in Appendix C.) is a large-scale chest X-ray dataset comprising 377,110 chest X-ray images and 227,827 reports from 63,478 patients. Each report contains two main sections: Findings, which contains a detailed description of the image, and Impression, which provides a concise summary of the findings. We utilized the training and validation data based on the official split to train the PLURAL model in the second stage. We extracted the Findings sections from the reports and performed preprocessing (e.g., eliminating special characters, typos, and scarce words), following the work of Chen et al. (2020) . To extract the Impression sections, we followed the method of Endo et al. (2021) . We only used chest X-ray images of posterior-anterior or anterior-posterior views, discarding those of other views such as lateral. For each pair of an X-ray image and a report (i.e., Findings or Impression), a chest X-ray of the same patient taken at a prior visit was selected and allocated to construct a longitudinal image dataset. We also utilized a single X-ray image to pretrain the PLURAL model in cases where there were no prior visits. We excluded all the images (and corresponding reports) included in the test sets of the VQA dataset (i.e., MIMIC-Diff-VQA). In other words, the final test set for diff-VQA was totally independent of this dataset.MIMIC-Diff-VQA (Hu et al., 2023 ) is a VQA dataset containing 700,703 QA pairs about 164,324 sets of longitudinal chest X-ray images. The images are a subset of those of the MIMIC-CXR dataset. QAs were created using an automatic algorithm based on radiologist's reports. Those QA pairs are categorized as difference, presence, abnormality, view, location, level, and type based on the forms of questions. We used 131,563 and 16,372 QA pairs of 'difference' form to train and validate the PLURAL model during the second and third stages (see Figure 1 ) and 16,389 pairs as an independent test set to measure the performance of the model, based on the official split of the dataset. 
difference visual question answering: 
 As shown in Table 1 , PLURAL achieved the highest scores in all the natural language generation (NLG) metrics, including BLEU (Papineni et al., 2002) , METEOR (Banerjee and Lavie, 2005) , ROUGE-L (Lin, 2004) , and CIDEr (Vedantam et al., 2015) , compared to previous state-of-the-arts methods, such as MCCFormers (Qiu et al., 2021) , IDCPCL (Yao et al., 2022) , and EKAID (Hu et al., 2023) . The differences between the scores were significant (e.g., CIDEr: 1.832 for PLURAL vs. 1.027 for EKAID). Additionally, Figure 3 showcases selected examples to qualitatively compare the AI outputs of EKAID and PLU-RAL. We discovered that, on average, PLURAL was better at capturing the longitudinal change of multiple lung findings (see Figure 3 We performed ablation studies to assess the impact of pretraining steps in PLURAL. We removed each pretraining stage individually to check how it affected performance. When we removed all pretraining stages and only performed the finetuning stage (Stage 3 in Figure 1 ), the model reported the lowest performance (see (a) in Table 2 ). However, incorporating each pretraining step significantly improved model performance. For instance, adding the pretraining stage using longitudinal chest X-ray data led to a significant increase in performance (e.g., CIDEr: 1.088 for model (a) vs. 1.796 for model (c) in Table 2 ). The best performance was achieved when both pretraining stages were combined (see the PLURAL model in Table 2 ). We also conducted ablation experiments to investigate how the structure of inputs in the pretraining stage affects the PLURAL model's performance. Specifically, we removed past 1 ). As shown in Table 3 , the best performance in terms of BLEU scores was achieved when we used past images and all report sections together as inputs. We observed a significant improvement in performance compared to the baseline model (see (a) in Table 3 ) across all NLG metrics (e.g., BLEU-4: 0.496 for baseline vs. 0.520 for PLURAL). This indicates that temporal information in past images and reports was indeed useful in solving the diff-VQA task.We conducted further training and the evaluation of the PLURAL model with two other SOTA methods, MMQ (Do et al., 2021) and EKAID (Hu et al., 2023) , on six categories of questions other than the 'difference' category (i.e., 'location', 'views', etc.; see Section 2.5). These questions did not require the consideration of longitudinal differences between two X-ray images (i.e., non-difference VQA). The questions were of two types: open and closed. For open questions, answers were in free form, while for closed questions, answers were either 'yes' or 'no'. We calculated exact-match accuracy between the model outputs and ground-truths in both cases. Table 3 summarizes the results of the evaluation of non-difference VQA. The PLURAL model achieved the highest accuracy compared to PLURAL without pretraining (e.g., 65.0% vs. 87.3% in closed questions), and the two SOTA methods (e.g., 26.4% for EKAID vs. 51.2% for PLURAL in open questions). These results indicate that the PLURAL model is effective not only in answering questions about the changes in longitudinal images but also in answering other types of questions, such as the ones related to the location of abnormalities or X-ray view (see Appendix B for examples). 
conclusion: 
 In this study, we present a Transformer-based VLM called PLURAL that is pretrained on natural images and texts, and longitudinal chest X-ray data to solve the difference VQA problem. Our approach allows the PLURAL model to benefit from both pretrained VLM knowledge and temporal information in the X-ray reports that describe the changes in lung abnormalities and diseases over time. Throughout the experiments, we have demonstrated that the PLURAL method outperformed previous SOTA methods in both diff-VQA for longitudinal X-rays and conventional VQA for a single X-ray image. We believe that this new model and training pipeline would be beneficial in promoting the application of the difference VQA task to assist radiologists in reading longitudinal chest X-rays. 
 
 Appendix: https://github.com/OFA-Sys/OFA 


Decision: The paper introduces a novel Vision-Language Model (VLM) called PLURAL for the task of difference visual question answering (diff-VQA) in longitudinal chest X-ray images. The model is developed through a multi-step approach, starting with pre-training on natural images and texts, followed by pre-training/fine-tuning on longitudinal chest X-ray images and radiological reports. Ablation studies are conducted to analyze the impact of each pre-training step on the model's performance. The proposed model demonstrates superior performance compared to some state-of-the-art (SOTA) models for natural language generation metrics.

Overall, the paper presents a promising approach for diff-VQA in longitudinal chest X-ray images, with a clear methodology and experimental setup. However, improvements in the discussion of results, including variability assessment and error analysis, would strengthen the paper's contribution to the field.

Reviews: review: paper_summary: paper_summary: This paper presents a vision language model to address the task of difference visual question answering (diff-VQA) in the context of longitudinal chest X-ray images. The authors propose a multi-step approach to pre-train the model on natural images and texts and then pre-train/fine-tune on longitudinal chest X ray images and radiological reports. Ablation studies were conducted to understand the impact of each pre-training step of the model. Results show that the proposed model outperforms some SOTA for natural language generation metrics.
main_review: main_review: The authors have added error analysis to better understand the model's errors and their clinical relevance. The work is well presented and it would be of good relevance for the community working on the difference VQA task.
strength_weakness: strength_weakness: - Missing variability assessment or statistic tests to verify the significance of the gain in each metric.
- Authors do not present an error analysis part in the qualitative results to check the clinical relevance of the errors.
- Authors do not discuss the limitations of their work.
questions: questions: - What are some examples of errors made by the model and do they have any clinically relevant pattern? for example, describing the same abnormality but with different words.
- Did you test base pre-trained models other than OFA (CLIP, for example)?
- Did you perform any statistical tests or variance analysis to verify the significance of the results?
paper_summary: paper_summary: This work introduces a novel VLM called PLURAL, which is pre-trained on natural and longitudinal chest X-ray data for the diff-VQA task. The model is developed using a step-by-step approach, starting with being pre-trained on natural images and texts, followed by being trained using longitudinal chest X-ray data. The longitudinal data consist of pairs of X-ray images, along with question-answer sets and radiologist’s reports that describe the changes in lung abnormalities and diseases over time.
main_review: main_review: Thanks for the author's response. The experiment results show the method performs well on the MIMIC-Diff-VQA dataset. Based on the work and the response, I think this work makes progress in the CXR image difference VQA task, and this paper deserves accepted.
strength_weakness: strength_weakness: 1. The 3.1 section says the model performs better on disease severity, please clarify it in Fig. 3 (c).
2. Lack of specific examples of non-difference VQA tasks.
3. Due to the specific nature of medical images and reports, it may be better to include physician comments.
questions: questions: 1. Provide some specific examples of non-difference VQA tasks.
2. Due to the specific nature of medical images and reports, it may be better to include physician comments.
paper_summary: paper_summary: This paper introduces a novel VLM called PLURAL, which is pretrained on natural and longitudinal chest X-ray data for the diff-VQA task. This draft primarily builds upon EKAID to further explore pretrained Vision-Language Model (VLM) models. The motivation is sufficient, the methodology is moderately innovative, but the experimental results demonstrate state-of-the-art performance.
main_review: main_review: Thanks for the author's efforts to reply, I am willing to revise the opinion. 
The authors compare similar methods and show differences.
But I still have the problem of lack of data sets. I hope it can be reflected in my future work.
strength_weakness: strength_weakness: 1. Code availability is not specified.
2. It seems that the MIMIC-Diff-VQA dataset has already been used in step 2, and considering it is derived from the MIMIC-CXR dataset, it might be more beneficial for the fine-tuning in step 3. 
3. The method in Figure 2 lacks sufficient innovation, as its primary framework is derived from Wang et al. 2022.
questions: questions: See Detailed Comments. If the author can better respond to the shortcomings, I may change the grade.

Comments: comment: Thank you for your response. I have no further questions.
Thanks for your clarification of my doubts. I have no further questions.
We are grateful for the time and effort you put into your detailed review. The points you highlighted have been immensely helpful in revising our paper.

## Q1 Examples of errors and clinically relevant pattern

Thanks for your valuable opinion. As you suggested, we have included an error analysis in Supplementary Appendix A. As you pointed out, we found a clinically relevant case where PLURAL generated an answer for describing the change of an abnormality in time-reverse order.

## Q2 Test on the other base pre-trained model

Thank you so much. We think that the OFA model is the best fit as a baseline based on the following reasons: First, it is easy to train the OFA model using multiple tasks, such as report generation and VQA. Second, the network architecture is simple, which permits us easily to add a new input branch for a past image. Third, a pre-trained model weight is available on the web, reducing the burden of making a pre-trained model by ourselves. There had been other candidates to be used as baselines [1-2] but these are limited in some of the criteria above.

## Q3 Statistical tests or variation analysis

Thanks for your comment. We are also eager to test the statistical significance of our experiments and check the impact on performance improvement, such as in the ablation study. Unfortunately, the proposed methodology requires training and finetuning a model through three stages, preventing us from repeating experiments, and requiring a lot of computational resources and costs. Therefore, we regret to say that checking the statistical significance is really hard for us under the current circumstances.

[1] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." *International conference on machine learning*. PMLR, 2021.

[2] Alayrac, Jean-Baptiste, et al. "Flamingo: a visual language model for few-shot learning." *Advances in neural information processing systems* 35 (2022): 23716-23736.
Thank you for the considerable time and effort you have devoted to your review.

## Q1 Examples of non-difference VQA
Thank you so much. We have attached some examples of non-difference VQA tasks in Appendix B Fig. 5, as you proposed.

## Q2 Clarification in Fig. 3 (c).
Thanks for the comment. As you suggested, we have elaborated on the severity and location of the abnormality in the caption of Fig. 3c for further clarification.

## Q3 Physician comments
We really appreciate your feedback. Based on your suggestions, we added some texts and figures in Supplementary Appendix A after having the radiologist’s comments on some examples.  


In this revision, we have been able to articulate more precise statements regarding disease severity and incorporate non-difference VQA examples and physician comments. We have applied all the comments you mentioned to our paper.
Thank you for investing your time and effort in writing a truly valuable review. In this revision, we have made the code available for access (see https://github.com/yjch00/PLURAL).

## Q1 Other datasets for step 3

If we understand correctly, you're asking if there are alternative datasets that could be used during the finetuning process of step 3. We had been also eager to utilize other datasets in this study; however, to the best of our knowledge, there is no other dataset that satisfies the following criteria: Longitudinal chest X-ray dataset with a particular focus on analyzing the differences between two images. A large-scale dataset that meets these specific requirements was MIMIC-Diff-VQA only. We hope to find another dataset in the near future, training and testing our model as soon as it becomes available.

## Q2 Any improvements from OFA (Wang et al. 2022) [1]

We believe that the PLURAL method has the following improvement or distinct features compared to OFA: Firstly and most importantly, from the perspective of an architecture design, PLURAL has a new input branch for a past chest X-ray to capture the differences between two input images, whereas OFA doesn’t. Secondly, to process a past image more effectively, we combined a time-encoding scheme which was not used in the OFA method. Lastly, OFA adopted a simple pretraining-finetuning method to address each downstream task, while PLURAL uses a three-step pretraining pipeline which is optimized to improve the performance of AI in terms of solving diff-VQA tasks in chest X-rays.

## Q3  Other datasets available for testing

As we discussed in our answer to Q1, we have really wanted to find another dataset that could be used in either the training or testing processes but failed. In line with our survey, all the previous works [2-4] utilizing longitudinal chest X-ray data to detect temporal changes have also exclusively utilized the MIMIC dataset. This is because other datasets, such as the IU Dataset [5], do not contain historical patient records, including past images, making them not practical to use in our study.

[1] Wang, Peng, et al. "Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sequence learning framework." *International Conference on Machine Learning*. PMLR, 2022.

[2] Hou, Wenjun, et al. "RECAP: Towards Precise Radiology Report Generation via Dynamic Disease Progression Reasoning." *Findings of the Association for Computational Linguistics: EMNLP 2023*. 2023.

[3] Zhu, Qingqing, et al. "Utilizing Longitudinal Chest X-Rays and Reports to Pre-fill Radiology Reports." *International Conference on Medical Image Computing and Computer-Assisted Intervention*. Cham: Springer Nature Switzerland, 2023.

[4] Bannur, Shruthi, et al. "Learning to exploit temporal structure for biomedical vision-language processing." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2023.

[5] Demner-Fushman, Dina, et al. "Preparing a collection of radiology examinations for distribution and retrieval." *Journal of the American Medical Informatics Association* 23.2 (2016): 304-310.