Context-Guided Medical Visual Question Answering

Venue: MICCAI_org_2024_Workshop_MSB
Published: 2024-06-29 04:30:24

Abstract: Given a medical image and a question in natural language,
medical VQA systems are required to predict clinically relevant answers.
Integrating information from visual and textual modalities requires com-
plex fusion techniques due to the semantic gap between images and text,
as well as the diversity of medical question types. To address this chal-
lenge, we propose aligning image and text features in VQA models by
using text from medical reports to provide additional context during
training. Specifically, we introduce a transformer-based alignment mod-
ule that learns to align the image with the textual context, thereby in-
corporating supplementary medical features that can enhance the VQA
model’s predictive capabilities. During the inference stage, VQA operates
robustly without requiring any medical report. Our experiments on the
Rad-Restruct dataset demonstrate a significant impact of the proposed
strategy and show promising improvements, positioning our approach as
competitive with state-of-the-art methods in this task.

Main paper: 
 introduction: 
 Medical Visual Question Answering systems can streamline healthcare workflow efficiency by allowing quick retrieval of relevant information from medical images. This can save valuable time for healthcare providers and offer additional insights into diagnostic procedures while assisting in clinical decision-making. Despite holding such potential, the application in the medical field has been minimal due to the small-scale of available datasets, the complex nature of medical images, the diversity of questions and the high level reasoning required to answer them.Initial research [5, 11] attempted to transfer advances in general VQA to the medical domain. However, Medical VQA faces unique constraints related to the acquisition and processing of data [16] . Usually, constructing medical VQA datasets require costly expert annotation and professional knowledge, e.g., extracting Question-Answer (QA) pairs directly from a medical image needs domain specific expertise. This limitation often restricts the data collection process leading to small size datasets. Unlike general-domain VQA datasets, such as VQA [3] , that contains hundreds of thousands of samples, medical VQA datasets are limited to tens of thousand images and QA pairs [4, 6, 7, 14] .Recent research focuses on harnessing the potential of attention-based pretrained biomedical visual language models such as BioGPT [18] and BLIP-2 [15] in a generative strategy. For instance, Van Sonsbeek et al. [24] proposed mapping visual features to a set of tokens that prompt a GPT-XL decoder [23] . For classification-based VQA, capturing high-quality medical features is crucial. For this purpose, Mixture Enhanced Visual Features model (MEVF) [19] uses the Denoising Auto-Encoder (DAE) as a visual extractor to enhance the quality of visual features extracted from medical images. With a typical focus on either visual feature extraction or fusion models, recent works [12, 2] have seen a surge in the deployment of vision-language models as image encoders for their significant capability of producing high-quality features. However, such approaches rely heavily on transformers and attention mechanism for feature aggregation and fusion [25] . While attention has proven to be effective in cross-modal settings [2] and text-image fusion [1] , this ties the advancement of medical VQA to the development status of attention mechanism and restricts the progress to fusion models. In contrast, this work posits that incorporating additional information to provide textual context during training could enhance both feature extraction and feature fusion.Inspired by the NLP Question Answering task [20] , we propose to use freetext medical reports as additional context to enhance visual feature extraction during training. Specifically, medical reports provide context we use to align image features with the same embedding space as the questions. This alignment improves the quality of image features, enabling accurate responses to questions during the inference stage without requiring additional context input. Our focus is on a closed-ended classification VQA task, where the targets are a predefined set of answers. In our pipeline, we first summarize medical reports using the GPT model [8] to effectively clean and pre-process the raw free-text for better encoding. Then, our model incorporates context features from summarized medical reports through a trainable alignment module that learns meaningful correlations between the textual and visual features. This enhances the image content as a reference for the question and helps identify fine-grained visual details guided by the medical text. To generate the final classification input of the VQA model, a fusion module is trained to combine the aligned image-context features and the encoded question features. Moreover, we use a pre-trained visual language encoder as an image encoder and an LLM-based encoder for text encoding without extra finetuning. By doing so, we significantly reduce the number of parameters and the computational complexity of our model. Experiments on the Rad-Restruct dataset [21] show an increase in performance outperforming the baseline model, with a state-of-the-art accuracy.In summary, our research contributes in three distinct aspects. First, we propose the first approach to use free-text medical reports as context to guide the prediction of answers in medical VQA. Second, we harness the power of pretrained models throughout our model while reducing computational complexity by 90%. Finally, we prove through experiments that our strategy outperforms the baseline methods on different metrics with state-of-the-art accuracy.  
methodology: 
 Figure 1 presents an overview of our architecture. The primary objective of this method is to utilize free-text reports to enhance the visual reference space, thereby providing a more comprehensive context for answering questions. To achieve this, we first encode the medical report associated with each image to generate textual context, and introduce a module to align this context with image features. Subsequently, we fuse the context-image features with the question vector and feed them into an MLP classifier for final answer prediction. Our framework has 5 main components: 1) a text encoder for encoding the question and context, 2) an image encoder for encoding image features, 3) an alignment module that learns to align and fuse the context and image features, 4) a fusion model that learns to combine the aligned context-image vector with the question features, and 5) a classifier that is trained to predict the answer.The majority of free-text medical reports are divided into sections i.e., Impression and Findings. Following prior literature on report generation [13] , we focus on the Impression section as it provides less general information, and more detailed medical findings. The extracted text is summarized using a prompted GPT model and the output summary is used as context. The summarizing process aims not only to potentially shorten the reports but also to clean and simplify the text descriptions. Herein, we denote the training data as follows:D = (c i , q i , I i , y i ) N i=1where c i represents the context (text), q i the question (text), I i is the image, and y i the corresponding ground truth label (text).For an input image I i , we extract the visual features V I using a pre-trained image encoder E image . Formally,EQUATIONSimilarly, we obtain feature vectors of the question V q and context V c through a text encoder E text as follows:EQUATIONTo learn correlations between the image and context, we integrate a transformerbased alignment network f align that produces a final context embedding U c as follows:EQUATION)f align is a stack of three multi-head self-attention layers that compute crossattention scores between textual and image features. Here, we consider context as query, and image embeddings as key-value pairs. Each layer learns to attend to both image and textual context, producing context-aligned image features.The final aligned representation is obtained via the upper attention layer. The aligned image features implicitly represents image regions that are most relevant to the textual content. Following the work of Pellegrini et al. [21] , we adopt an autoregressive strategy and incorporate history information as input. At each step, the history vector H i contains the previous (higher-level) questions with their answers. For example, the question "In which part of the body?" requires prior knowledge from higher level questions such as "Is there an opacity in the lung?". The history vector is constructed by concatenating previous and current question tokens.Next, a multi-modal transformer-based fusion model f f usion is trained to combine the question and the image-context features to generate a final embedding vector U i for classification. Formally,EQUATIONwhere H i is the history information for the current question q i . Finally, the MLP classifier predicts the answer y i given the features of the resulting U i .During the training phase, the alignment module is trained using both the context and the image features to produce a vision-context vector. Similar to [21] , we train our model with a cross-entropy loss in an autoregressive manner taking in consideration the question history. For inference, our approach assumes an absence of context and takes as input the image and questions. The alignment module creates a context vector using the image through the learned textual representation during the training process.  
experiments: 
 Dataset. We use the Rad-Restruct dataset [21] , which contains 3720 Xray images, 3597 structured reports, and 180k questions. For each image, a corresponding free-text report is retrieved from the IU-Xray dataset [9] . Images were normalized and cropped to 224×224. We use the original 80-10-10 split for training, validation, and testing, respectively. Rad-Restruct questions are categorized into 3 levels. Level 1, the highest level, contains general-purpose questions such as "Are there any foreign objects?". Level 2 questions ask for more specific findings, like "Is there pneumonia in the lung?", and level 3 questions are related to detailed findings, for example, asking for the degree or a description of a disease, "What are the attributes?". This hierarchical structure allows an autoregressive parsing of the questions, making lower-level questions depend on higher-level ones, which aims to provides more context for difficult questions in level 3 question. In addition, there are 96 classes in total. The questions can be single-choice or multiple-choice, with each having a defined set of possible options. To the best of our knowledge, at the time of the experiment, Rad-Restruct is the only available dataset that enables our experimental design by providing access to all three components, medical images, free-text reports, and QA pairs. Context Extraction. To extract the context from radiology reports, we use the prompted GPT 3.5 Turbo with the following prompt: 'Please summarize the following X-ray report while keeping the medical terms.' Figure 2 shows an example of a summarized report by the GPT API. The GPT model aims to generate natural language text comprehensively. In doing so, it rephrases reports into simplified vocabulary, excluding sensitive medical details that could be beneficial. We address this concern by requesting the retention of medical terms. The length of generated summaries varies based on the original report and content nature. We impose a maximum size of 512 tokens, truncating when needed. These summaries serve as a direct context for training our model.As an image encoder, we use PubMedClip [10] , a variant of the CLIP model [22] designed for medical VQA, it is pre-trained Table 1 . Performance comparison on Rad-Restruct dataset. We compare 5 methods, a visual baseline [21] , hi-VQA, both the published results and the released dataset results, re-VQA (our reproduced hi-VQA results) and our method Context-VQA.Visual baseline [21] 31 [17] , and was trained on a large corpus of radiology reports. To encode text inputs, we leverage RadBert's pre-trained embeddings, which capture domain-specific semantics and contextual. The encoders are frozen, preserving their pre-trained weights and preventing further parameter modification. As a classifier, we use a 5-layer MLP with batch normalization and a dropout rate of 0.2. The model is trained for 200 epochs on a NVIDIA RTX A6000 GPU. We use Adam Optimizer with a learning rate of 1e-5 and a batch size of 32. 
results: 
 Table 1 shows the performance of our Context-VQA method and the state-of-theart method hi-VQA [21] on the Rad-Restruct dataset. For a fair comparison, we use the same evaluation approach as hi-VQA and report our results accordingly using the conventional metrics Accuracy, F1, Precision, and Recall. As accuracy, we provide the report accuracy, a metric that is specific to the dataset. Rad-Restruct was built to structure the text reports in the IU-Xray dataset into a form populated by hierarchical questions. Thereby, each image is accompanied by its set of questions referred to as a structured report. The report accuracy metric represents the accuracy of reports that were fully predicted correctly. Thus, a report is considered correct if all questions at all levels for a given image are perfectly answered. Context-VQA outperforms hi-VQA on this specific metric with almost a 10% increase. This translates to an increase of fully predicted reports' questions by 10%. The evaluation was done using the script provided in [21] , allowing direct comparison to hi-VQA. Our results significantly outperform the hi-VQA on the other metrics, with precision improved by 29%.Furthermore, Table 2 shows the performance on each level of questions. Context-VQA steadily achieves state-of-the-art accuracy on all levels. Although, the F-1 score shows a slight decrease for some type of questions in level 2 and level 3, level 1 and level 2 (all) show higher overall performance. In addition, it is noteworthy that our approach, with only 16M trainable parameters was able to perform closely or even better than hi-VQA (164M parameters). We also note that we compare to the released dataset results instead of the scores reported in the paper. Figure 3 illustrates qualitative examples of predictions of Context-VQA compared to the baseline predictions. Questions are arranged sequentially from left to right, level-wise, indicating their hierarchical dependency. In the first instance, hi-VQA predicts a negative response to the initial question, consequently influencing subsequent questions to also receive negative predictions. In examples 2 and 3, we observe that for these cases Context-VQA accurately predicts lower-level questions, which are often challenging and influenced by preceding questions. This explains the ability of our model to correctly predict sequential questions leading to the improvement in report accuracy, as previously reported.Computational complexity By leveraging the pretrained encoders, we achieve a substantial reduction in the number of trainable parameters,from 230M to 16M, against hi-VQA's 164M parameters. Consequently, the training time is reduced.Experiments on a single A6000 GPU record the training time of 1h30min per epoch for hi-VQA against 35min required by Context-VQA, as demonstrated in Table 3 . Notably with fewer parameters and less computational time, Context-VQA still performs well on the Rad-Restruct dataset.We conducted ablation studies to assess the impact of contextual information on the prediction capabilities of our Context-VQA model.In the first experiment, we integrated the context as an input in the baseline hi-VQA model by tokenizing and feeding it into the text encoder alongside the question. The textual features were then concatenated with the visual features and processed in the fusion module. This experiment aimed to demonstrate the impact of incorporating textual information. The results, labeled as "hi-VQA + context," are presented in Table 3 , showing a performance improvement compared to the baseline model. To further examine the significance of the alignment module, we adopted a similar approach of incorporating contextual information as an additional input in our model without further alignment. The results are reported as "our -f align " in Table 4 . We note that the recall and F1 scores exhibited a slight decrease, primarily attributed to the lack of encoder updates, which limited improvements in feature extraction. 
conclusion: 
 In this work, we proposed a novel approach to enhance medical Visual Question Answering (VQA) systems by leveraging free-text medical reports as contextual information. We use the context by incorporating an adaptive text-image alignment module that learns to align textual and visual features. Through extensive evaluation on Rad-Restruct dataset, we validated the efficacy of integrating context-based information alongside images and questions that provides richer medical features for VQA with significant performance gains. Given the rise of datasets with additional medical reports and EHR, this work paves the way for further exploration and refinement of context-based approaches in advancing the capabilities of medical VQA systems and ultimately improving medical AI systems' outcomes. 
 
 Appendix:  


Decision: This work presents a technically sound VQA approach using radiology reports as context with good results. The reviewers agree that the paper is clear and well-written.The authors have reasonably addressed the reviewer’s concerns and incorporating these changes will strengthen the paper. Additionally, considering the limitations of data for real-world use would be beneficial as it might not be what doctors actually need. The problem is bigger than this one study, and for this to be useful in a clinical setting, it needs more work. The authors are recommended to include more stronger baseline methods and report statistical significance for a more robust evaluation.

Reviews: review: main_review: main_review: The proposed work is interesting for the audience and has its merits. The paper is well-written with good presentation. However, there are some questions related to the method and experimental settings (see weaknesses) about this work, addressing them will help understanding this work more clearly.
strength_weakness: strength_weakness: I would like the authors to address the following questions:

1. How the f_align module handles the absence of context at inference needs clarification. How the (context) query is initialized at inference time in this module? An illustration of the f_align module during training and inference will be helpful. 
2. This is unclear if open access to the model will be provided or not.
3. The impact of history vector H_i on the final model is not reported in Table 4. It is unclear to me how much beneficial keeping this history vector is. What is the dependency between H_i and a single image-question pair?
4. How often does the context truncation to 512 tokens happen in this dataset? What is the maximum length of the summarized reports?  
5. The failure analysis for the proposed Context-VQA method is not provided. It will help in identifying the limitations of this work that require further exploration.
6. The authors claim that Rad-Restruct is the only available VQA dataset with free-text reports. I am aware that the PMC-VQA[1] dataset has QA pairs generated from free text reports. I would like the authors to comment on why did they not consider evaluating their method on this dataset in their experimental setup. What are the limitations?

[1]  PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering, X. Zhang et al. (https://github.com/xiaoman-zhang/PMC-VQA)
review_summary: review_summary: The paper proposes to solve the VQA task by employing radiology reports as context to image-question pairs during the training phase. The VQA task is formulated as a close-vocabulary answer classification task. A new VQA architecture is introduced that learns to align context with image features to learn better image features by mapping them into the same feature space as the question. The proposed approach is evaluated on Rad-Restruct dataset demonstrating better performance compared to the baseline method hi-VQA notably in terms of the report accuracy, and precision.
main_review: main_review: Although there are still some points to improve in the paper, I think it fits well within the scope of the workshop, which aims to provide a framework for in-depth discussions to further enhance the work.
strength_weakness: strength_weakness: Major: 
1. The number of baseline methods is a bit sparse. Advanced medical VLMs such as RadFM are missing.
2. The paper mentions that during inference, the model does not receive context information. Please provide a more detailed description of how the module handles this missing information.
3. To enhance reproducibility, it would be beneficial to publish the code.

Minor:
1. Page 2:  “Moreover, we use a pre-trained visual language encoder as an image encoder” – I guess, the authors mean that a vision encoder of a pretrained vision language model is used?
2. Page 2: “By doing so, we significantly reduce the number of (*trainable*)parameters and the computational complexity of our model.”
3. Table 4: Include hi-VQA (Its annoying to switch to Table 1), and add information about which version of hi-VQA was used.
review_summary: review_summary: The paper proposes to include medical reports to train an alignment module to generate context-aligned image features. Combined with pretrained Image and Text encoder, this module improves the performance compared to previous methods on the Rad-Restruct dataset..
main_review: main_review: The paper presents an interesting idea, with clear benefits regarding the model size (on GPU during training) and training time. However, there are major shortcomings in the evaluation. I still think that this paper is relevant for the workshop, hence I rate it weak accept.
strength_weakness: strength_weakness: - If I understand correctly, the GPT API is used to summarize the original report ("Context Extraction"). In a realistic setting, sensitive data would be leaked, although the entire purpose of this step is to avoid data leakage. Perhaps this step was delegated to GPT 3.5 to facilitate the prototyping, but the manuscript should discuss whether a local model could also be employed for this task.
- Since the pipeline is still quite complex, not only the training time but also the inference time (and FLOPS) matters and should be reported. How fast is the inference of a single test sample (wall clock time)?
- Apparently, the authors created a fully trained and partially trained version of their model (Table 3, Section "Computational Complexity"). However, it is not 100% clear which of these versions are investigated in the ablation study or the baseline comparison, although I strongly assume it is the partially trained (frozen encoders) as this is a central contribution of the paper. It would still be interesting to include both fully and partially trained architectures in all evaluations; I ask myself: "what would I gain with a full training, what do I sacrifice by relying on pre-trained encoders?".
- Source code is not provided.
review_summary: review_summary: The authors propose a pipeline for Visual Question Answering on chest X-ray imaging, that comprises several steps to extract image context from reports and align it with image features. The proposed method can rely on pre-trained/finetuned models, significantly beating SOTA on the same task.
main_review: main_review: This work proposes techniques based on what is available of the dataset and the main trends of task formulation in AI, rather than what is really needed in clinical settings. This is not good research as it only follows the technical trends but lacks critical thinking of the common practice in the technical community and ignores the true research problems. I would like to kindly remind the authors that technical trends can easily be misleading and cannot always justify themselves. For example, "Historically grown practices are not always justified" (Understanding metric-related pitfalls in image analysis validation https://www.nature.com/articles/s41592-023-02150-0) I encourage the authors to exercise their critical thinking skills in the problem formulation and justification, and conduct good future research.
strength_weakness: strength_weakness: The problem formulation of the closed-ended medical VQA and the proposed technique exhibits vague clinical values, utilities, and benefits. It is unclear how it can contribute to the usage in clinical settings, other than the narrow motivation to "show promising improvements" and "competitive with state-of-the-art methods in this task." The authors do not provide justifications that improving performance of the closed-ended medical VQA problem is on the right track to clinical usage. If the whole research direction of the closed-ended medical VQA problem is not on the right track towards clinical utility, how can the authors justify that such performance improvement on this wrong track is valid and exhibits research significance?

It is difficult to see that the formulated problem of the closed-ended medical VQA is on the right research track. Think about the clinicians using the model, how can the model be useful to them if it can only answer questions based on the predefined n sets of multiple choice options, where the clinicians questions can be any questions? If such closed-ended answers are useful for clinicians somehow, since it's closed ended, the ultimate information the model outputs is similar to a simple classifier outputting the n sets of answers. If so, how do the authors justify the clinical utility of the proposed closed ended model, compared to a simple classifier outputting the same sets of answers without clinicians' efforts to ask questions, something like the simple classifier outputting a structured medical report? Wouldn't that simple classifier be more useful for clinicians than the proposed closed ended VQA? 

If the authors assume that closed ended VQA can be the prerequisite for open-ended VQA, they did not justify this assumption in the paper. In fact, it is hard to be convinced that open and close ended VQA are the same problem, rather than two distinct ones that require different assumptions and modelling approaches.
review_summary: review_summary: This work proposes a technique on the problem of closed-ended medical VQA by utilizing medical report and image-text alignment.

Comments: title: Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
Rebuttal by Authors
comment: We thank the reviewer for the thorough review and valuable feedback. We greatly appreciate the time and effort they have invested in evaluating our work. We would like to address some of the concerns raised and provide clarifications on several points to highlight the contribution and clinical relevance of our research.

Closed-ended medical VQA: In clinical usage, while closed-ended questions may seem limited, they offer a controlled and consistent way to validate and integrate AI in clinical settings. This ensures that the answers are reliable and align with predefined clinical standards, which is crucial for building a trustworthy VQA. In contrast, open-ended questions offer flexibility but require robust reasoning. Striking the right balance between these question types is essential for optimizing clinical utility. Although the primary focus of our work does not center on assessing the validity of closed-ended medical VQA in relation to its open-ended counterpart, we believe the formulation of the closed-ended VQA problem is a steppingstone towards more complex and open-ended systems. Our motivation lies in overcoming challenges such as data sparsity and leveraging available information to enhance VQA, ultimately bridging the gap between research and clinical practice.

Clinical Utility: In both open-ended and closed-ended tasks, the set of questions is predefined within the datasets. The key distinction lies in the nature of the answers: free-text or a set of candidate answers. Notably, this difference would not pose a significant issue in terms of the clinicians’ questions. Specifically, within practical medical contexts, questions posed by clinicians are limited by the image and test type and have a finite range of potential answers. Thus, close-ended questions can be as relevant as open-ended ones.

Vision Classifiers: While a simple visual classification may yield predictions, it remains confined to providing a general observation for specific diseases. In contrast, medical VQA represents a superior multimodal task. By jointly combining both images and questions, it extends its predictive capacity to encompass a broader spectrum of potential answers.

Our model enables sequential question answering. If deployed as a clinical system, users would only need the image and questions without the need for context. In a real scenario, a radiologist may initially ask, "Is there an abnormality in the image?" The system responds affirmatively and supports subsequent detailed inquiries such as "What is the nature of the abnormality?", then "What are the specific signs observed?", "Where is the abnormality located?", and "What are its characteristics (attributes)?" Or “What is its degree?”. This progressive questioning enables the construction of a comprehensive diagnostic report based on the findings within the image. Moreover, beyond clinicians, patients and other healthcare workers could also benefit from such a system.

Additionally, while a simple classifier could predict answers, it lacks the interactive element that our VQA model offers. Our model's ability to respond to specific questions adds an extra layer of utility, enabling clinicians to query the system dynamically and receive precise answers relevant to their immediate concerns. This interaction can lead to more personalized and efficient clinical workflows. 

Context Processing: In our approach, context extraction serves as a preprocessing step to eliminate ambiguous tokens, including privacy-related replacements. Numeric values are typically transcribed into alphabetic representations, Fig 2, and the measurements we consider pertain to image dimensions and other specifics of the test procedure—not the medical measurements themselves. Importantly, the context does not directly influence model predictions; its primary role is aligning visual features. Consequently, our model exhibits insensitivity to minor variations in context language. We posit that minor modifications to non-medical language would have negligible impact on the extracted visual features.

Dataset and Evaluation: Level1 questions are yes/no questions, while level 2 and 3 can vary. Notably, the attributes in level 3 question refer to the description of the abnormality, example answers include ‘round’ and ‘patchy’. For a comprehensive understanding of the questions and evaluation metrics, please check the original dataset paper [21]. Subsequently, please refer to the dataset GitHub repository for more details on the discrepancy between the released dataset and the results originally published in the paper [21]. 

We thank the reviewer for highlighting the medical errors in Fig 1 and we are committed to revising and addressing the concerns raised to improve the paper. While acknowledging the existing areas for improvement in our work, we maintain that it aligns with the objectives of the workshop. We eagerly anticipate engaging in profound discussions to delve into its details.
We thank the reviewer for the thoughtful and constructive feedback on our work. We appreciate your recognition of the strengths in our organization, clarity, and comprehensive evaluation. We also value the suggestions for improving our paper. 

Private Data Handling: Our model is trained on public data, which eliminates the risk of privacy exposure. In case sensitive data is exclusively available, a simple preprocessing of data would be sufficient to anonymize personal information such as names and dates before utilizing GPT for further processing. Moreover, Open AI claims following strict privacy and data security protocols for data processed through the GPT API, ensuring that they do not collect or utilize this data to train their language models. 

Inference Time: Considering real settings, evaluation on a CPU yields an average inference time of 15 minutes for the entire test set of Rad-Restruct dataset, which is equivalent to an average of 2 seconds per image, including all the questions. This demonstrates the necessary efficiency in handling images with varying numbers of questions. On a GPU, the evaluation is two times faster with approximately 8 min on average. We agree with the reviewer that the inference time is important as well, and commit to including more details, such as FLOPS, during the revision of this paper for the final submission. 

Clarity on Model Versions: As the reviewer noted, we reported the results of the partially trained model with frozen encoders. The encoders were robustly pre-trained on large domain specific data, and retraining increases the risk of overfitting on our small dataset. By freezing the encoders, we mitigate this risk and preserve the strong representations already encoded during pre-training. Moreover, retraining the encoders results in training the full architecture which takes time and is computationally costly. Additionally, the partially trained model achieves strong performance, while significantly reducing computational time and complexity, and reducing the number of trainable parameters by 90%. 

We recognize that there are areas for improvement in our work, which is one of the reasons we are submitting it to the workshop. Our goal is to engage with professionals to receive valuable insights and feedback that can contribute to refining and advancing our research.
We thank the reviewer for the detailed and constructive feedback. We appreciate their recognition of the strengths in our methodology, pipeline description, and the improvements shown in our results. We are also grateful for the refinement suggestions, which will undoubtedly help us enhance our work. We would like to address some major concerns. 

Baseline Methods: Due to time constraints and our commitment to ensuring fair comparisons, the number of baseline methods included in this study is limited. We plan to extend our experiments to incorporate additional methods, such as advanced medical Visual Language Models (VLMs), to provide a more comprehensive comparison. 

Inference Context Handling: The attention in f_align’ is trained to learn correlations between the image and context, thereby enabling the extraction of visual features that correspond closely to the context. During the inference stage, in the absence of context, f_align utilizes the learned correlations to align the image feature vectors. Technically, during inference, the trained self-attention mechanism treats the image features as both the value and the query and recalls the learned representations to navigate the image and extract the vision-context vector. 

Clinical Application: Our model enables sequential question answering based on an X-ray image. Assuming our method is trained on a large enough dataset, when deployed in clinical settings as a VQA system, users would only need the image and relevant questions without the need of context. In a straightforward scenario, a radiologist may initially inquire, "Is there an abnormality in the image?" The system responds affirmatively and supports subsequent detailed inquiries such as "What is the nature of the abnormality?", "What are the specific signs observed?", "Where is the abnormality located?", and "What are its characteristics (attributes)?" Or “What is its degree?”. This progressive questioning enables the construction of a comprehensive diagnostic report based on the findings within the image. Moreover, beyond clinicians, patients could also benefit from such a system. 

The comments will be considered during the refinement of this paper. We hope our clarifications and revisions will address the concerns mentioned and highlight the significance of our contributions and we note that we are considering releasing the source code. 
We are encouraged by the reviewer’s positive recommendation and confidence in our work. We believe addressing these points will greatly improve this paper's quality and impact, and we look forward to having this work deeply discussed during the workshop.
We thank the reviewer for the constructive feedback. We appreciate the positive remarks on the clarity, novelty, and experimental setup of our work. We also value the detailed suggestions for improvement and would like to address the major points raised. 

Inference Details:  f_align’s attention is trained to learn correlations between the image and context, thereby enabling the extraction of visual features that correspond closely to the context. During the inference stage, in the absence of context, f_align utilizes the learned correlations to align the image feature vectors. Technically, during inference, the trained self-attention mechanism treats the image as both the value and the query and recalls the learned representations to navigate the image and extract the vision-context vector. 

The History Vector, Contex truncation and Failure Cases: The history vector is a list of tokens of previous questions and their answers. For a given image, the history vector is empty for the first question and subsequently updates after each step by concatenating tokens from the preceding question and its answer. Thereby, a single image-question pair is treated similar to the first question with an empty history vector. This autoregressive parsing mechanism enables the model to maintain continuity with previous knowledge (answers) and avoid contradictions in its responses. We are planning additional ablation studies to evaluate the impact of this history vector within our experimental framework. As for the context, most of the reports contain fewer tokens, therefore the truncation rarely occurs on less than 1% of the context during training.  

In Figure 3, we present examples of successful cases where our model surpasses previous methods, demonstrating how the quantitative results translate into improved predictions. This illustration aims to provide a clear understanding of the practical implications of our results. It is important to note that our model does encounter failures in other instances, and we are prepared to provide examples of such cases upon request. 

The PMC-VQA dataset: Our core motivation is to explore the impact of context on VQA performance. Our choice of close-ended classification on Rad-Restruct dataset was deliberate, as it limits the number of variables that could influence the performance. This task enables experimental validation of our method’s efficacy. The PMC-VQA dataset is designed for generative tasks with significant reliance on text decoders, especially Large Language Models (LLMs) in recent methods. This requires changing the model architecture and complicates attributing improvements to context alone. We are currently in the process of adapting our proposed method to generative VQA. We aim to expand this work and anticipate achieving promising results in the near future. Additionally, we are considering making our source code open to facilitate further research and collaboration. 

While we recognize that our work has potential for further refinement, we are eager to engage in discussions during the workshop. We hope these interactions will provide valuable insights and feedback that will help us improve and advance our approach.