From Sights to Insights: Towards Summarization of Multimodal Clinical Documents (2024)

Abstract: The advancement of Artificial Intelligence is pivotal in reshaping healthcare, enhancing diagnostic precision, and facilitating personalized treatment strategies. One major challenge for healthcare professionals is quickly navigating through long clinical documents to provide timely and effective solutions. Doctors often struggle to draw quick conclusions from these extensive documents. To address this issue and save time for healthcare professionals, an effective summarization model is essential. Most current models assume the data is only textbased. However, patients often include images of their medical conditions in clinical documents. To effectively summarize these multimodal documents, we introduce EDI-Summ, an innovative Image-Guided Encoder-Decoder Model. This model uses modality-aware contextual attention on the encoder and an image cross-attention mechanism on the decoder, enhancing the BART base model to create detailed visual-guided summaries. We have tested our model extensively on three multimodal clinical benchmarks involving multimodal question and dialogue summarization tasks. Our analysis demonstrates that EDI-Summ outperforms state-of-the-art large language and vision-aware models in these summarization tasks. Disclaimer: The work includes vivid medical illustrations, depicting the essential aspects of the subject matter.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13117–13129 August 11-16, 2024 ©2024 Association for Computational Linguistics
Disclaimer: The work includes vivid medical illustrations, depicting the essential aspects of the subject matter.

heading: 1 Introduction
text: One of the most impactful sectors where AI advancements can bring a significant revolution is healthcare. The latest WHO report highlights a drastic doctor-to-patient ratio, emphasizing the need for automation in various healthcare tasks. This imbalance, coupled with strides in information and communication technologies (ICTs) and the impact of COVID-19, has led to a significant rise in telehealth practices (Nittari et al., 2020). Recent
studies (Sahoo et al., 2022a), (Abacha et al., 2023), (Xue et al., 2018), (Sahoo et al., 2022b) demonstrate that AI models can be effectively employed in various healthcare settings and can drastically increase the productivity of healthcare professionals.
In this regard, a significant development is that a considerable number of patients now discuss their medical history and conditions in various clinical forums before attending in-person appointments. Distilling key insights from these clinical documents not only boosts the efficiency of AI-based healthcare systems but also accelerates responses to patients, providing valuable time savings for healthcare professionals and facilitating a more prompt and beneficial exchange of information between both patients and doctors (Abacha and DemnerFushman, 2019).
Most research in the field of medical document summarization has focused on unimodal, textbased approaches. Wei et al. (2023) explored medical query summarization using entity-driven contrastive learning. Similarly, (Joshi et al., 2020) addressed text-based medical dialogue summarization, while (DeYoung et al., 2019) focused on summarizing multiple medical documents. Recognizing the adage that an image is worth a thousand
13117
words, a discernible shift is underway, with patients increasingly incorporating images of their medical conditions alongside their queries. This inclination arises from the fact that a considerable portion of the population may not be well-versed in medical terminology. This challenge becomes more pronounced when dealing with related symptoms, such as distinguishing between eye redness and eye inflammation. Integrating text and images provides a more comprehensive context, enriching the analysis beyond what textual examination alone might capture, which is underscored in the work (Kumar et al., 2018). Motivated by the research gap (Ghosh et al., 2024a), (Tiwari et al., 2023), and (Ghosh et al., 2024b) have contributed datasets along with their proposed models, utilizing techniques such as prompting and end-to-end finetuning to incorporate visual information alongside textual content. Each of these works has established its own model, but no single model has been developed that is well-suited for all these multimodal clinical summarization benchmark datasets. Our exploration culminates in proposing a model named EDI-Summ, designed to work across all the mentioned multimodal clinical summarization datasets. EDI-Summ integrates contextual multimodal fusion within the encoder and employs a cross-attention mechanism in the decoder of the BART architecture.
Research Questions: We aim to investigate the following research questions in this work:
R1) How is the performance of EDI-Summ compared to the baselines?
R2) How different pre-trained vision models like ResNet (He et al., 2016), VGGNet (Simonyan and Zisserman, 2014), VIT (Dosovitskiy et al., 2020) affect the quality of generated summary in terms of various evaluation metrics?
R4) How much does the performance vary if we change the order of image fusion in the encoder and decoder layers?
R5) How effective is EDI-Summ in handling Hindi-English codemixed text ?
R6) Do we need an Image Fusion module in both the encoder and decoder? What is the impact of Decoder Visual cross-attention?
Contributions: The key contributions of our work are stated below:
1) We present an innovative encoder-decoder visual-infused transformer model: EDI-Summ, which includes a contextual image fusion layer
on the encoder side and an image cross-attention layer on the decoder side. Our proposed model exhibits remarkable performance gains, substantially outperforming baseline models and unimodal architectures in comprehensive evaluations.
2) We undertook an exhaustive automated and human evaluation, complemented by in-depth qualitative analysis and risk assessments. This meticulous scrutiny was essential to guarantee the safety and reliability of the model’s outputs, affirming its readiness for deployment in real-world healthcare scenarios with high confidence.

heading: 2 Related Works
text: The following works have been relevant to the following two research areas, namely (a) Works on Medical Document Summarization and (b) Works on Multimodal Summarization.
Works on Medical Document Summarization: Wang et al. (2020) was the first to conduct research on the summarization of COVID-19 documents. DeYoung et al. (2021) introduced a multi-document biomedical document summarization dataset. Wallace et al. (2021) introduced a dataset on the summarization of factual queries. In 2019, the MeQSum dataset (Abacha and Demner-Fushman, 2019) revolutionized Medical Question Summarization (MQS), establishing itself as a dedicated resource for the field. There was a competition on the task in 2021 by Abacha (Abacha et al., 2021). Researchers initially utilized various pre-trained models like BART (Lewis et al., 2019), Pegasus (Zhang et al., 2020), and Prophetnet (Qi et al., 2020). Goff and Loehfelm (2018) proposed a novel method of using pre-trained models for summarizing the impression section of radiology reports. Joshi et al. (2020) proposed a method to exploit local structures for summarizing medical dialogues. Chintagunta et al. (2021) showed how GPT-3 can be utilized for generation of quality training data for medical dialogue summarization. Poornash et al. (2023) proposed a novel solution for layout summarization of biomedical articles.
Works on Multimodal Data: Previous research has demonstrated the benefits of incorporating multimodal information in various tasks (Ghosh et al., 2024c). For instance Jha et al. (2022) showed how combining both visual and language representations can help in patch-based identification of lexico-semantic relations. Suman et al. (2022) showed how multimodal information can be used
for personality prediction. Jha et al. (2024) showed multimodal explanations can be useful for enhanced cyberbullying understanding. Maity et al. (2022) showed how multimodal information from memes helps in combating cyberbullying. Tiwari et al. (2022) highlighted how multimodal information improves the performance of Disease Diagnosis Virtual Assistants. Delbrouck et al. (2021) showed that integrating images leads to better summarization of radiology reports. Kumar et al. (2023) showed multimodality helps in summarizing news articles. Verma et al. (2023) developed a large multilingual multimodal dataset. Kumar et al. (2023) illustrated how multimodal summarization can be useful for extracting insights from opinions. Joshi et al. (2020) and Molenaar et al. (2020) worked on the task of handling dialogues between patients and doctors and how to summarize their conversations. Goff and Loehfelm (2018) and Cai et al. (2021) are some of the first to propose the task of radiology report summarization.

heading: 3 Methodology
text: This section delineates the problem statement and outlines the various components of our proposed model.

heading: 3.1 Problem Formulation
text: Formally given an input textual document (D) represented by D = {d0, d1, . . . , dn} where n is the sequence length of the document, and a visual image corresponding to a textual query represented by (V ), the task is to generate a clinically nuanced summary guided by the visual cue. The proposed architecture and its components are shown in Figure 2. The proposed model EDI-Summ has three main components, namely: i) Representation of different Modalities ii) Encoder Contextual Image Fusion iii) Decoder Image Cross Attention. A thorough explanation of the functionality of each section is provided in the following sections.

heading: 3.2 Representation of Different Modalities
text: Visual Representation: Our experimentation involved ResNet, VGG-Net, and VIT for generating image embeddings for the multimodal fusion layer. Since these pre-trained models produced image embeddings of dimension 2048, we introduced an additional neural network atop them to transform its model dimension to align with the textual dimension. The weights of all layers, excluding the
last three, were frozen, and we derived a vector representation of the image by pooling the output from the last layer, ultimately achieving a reduced embedding dimension of 768.
Textual Document Representation: In our study, a patient’s medical document is represented as a sequence of words. We utilize the pretrained BART-base language model to generate 768-dimensional embeddings for textual data. With six layers in its encoder and decoder, BART captures bidirectional contextual information, producing a fixed-size textual representation. BART tokenizers are instrumental in breaking down input text into tokens, ensuring effective encoding for our multimodal clinical query summarization.

heading: 3.3 Encoder Contextual Image Fusion
text: The various components of the Encoder Contextual Image Fusion are elucidated in the following mentioned points:
a) Multimodal Context Aware Self Attention: The effectiveness of the aggregated representation hinges on the seamless integration of multiple information sources. Inspired by the findings of Yang et al. (2019), we proposed a multimodal contextaware self-attention that produces conditioned key (K) and value (V ) vectors that become instrumental in generating modality-aware attention vectors. This is elaborated in Figure 2.
Upon obtaining the intermediate representation H = {h0, h1, . . . , hn} generated by the BART encoder at a particular layer, we proceed to compute the query (Q), key (K), and value (V ) ∈ Rn×d vectors, where d is the model dimension. This calculation is shown in Equation 1.
  Q K V   = H   WQ WK WV   (1)
Here WQ,WK , and Wv ∈ Rd×d are the learnable parameters corresponding to the query, key, and value vectors, respectively.
In assessing the interrelation between textual query and the visual information, we create conditioned key (K̂) and value (V̂ ) vectors tailored to textual and visual contexts. These attention vectors transpose the query vector derived from the dialogue transcript, generating a versatile, imageinfused information vector. The key and value pairs are calculated as shown in Equation 2. Here, we pass visual representation through a transformer
and project its sequence length1 to be equal to textual sequence length in order to perform multimodal fusion.
[ K̂
V̂
] = [ λk λv ] (G [ Uk Uv ] ) + (1− [ λk λv ] ) [ K V ] (2)
where G ∈ Rn×d indicates visual representation, Uk and Uv ∈ Rd×d are the learnable parameters.
The vector λ = [ λk λv ] controls how much information to retain from visual modality. This parameter can be calculated by the Equation 3.
[ λk λv ] = σ( [ K V ][ Wk1 Wv1 ] +G [ Uk Uv ][ Wk2 Wv2 ] ) (3)
where Wk1 ,Wk2 ,Wv1 , and Wv2 ∈ Rd×1 are learnable parameters. The final scaled dot product attention can be calculated as shown in Equation 4.
Hv = Softmax( QK̂T√
dk )V̂ (4)
where Hv represents visual information fused vector.
b) Unified Multimodal Integration: To regulate and modulate the flow of the visual-infused
1We transform image sequence length (1) to textual sequence length (n). Image sequence length is one because it is a single image corresponding to a given textual query.
information, we utilize visual gates. The transmission of contextual information occurs through these gates according to Equation 5:
gv = σ([H ⊕Hv]Wv + bv) (5)
Here, Wv ∈ R2d×d and bv ∈ Rd×1 are learnable parameters. The ultimate amalgamated representation, denoted as Ĥ , is obtained according to the Equation 6.
Ĥ = H + gv ⊙Hv (6)
The contextualized image-infused vector (Ĥ) is passed to the upper layer of the transformer and sent to the decoder.

heading: 3.4 Decoder Image Cross Attention
text: In the BART Decoder, we inject an Image crossattention block for performing image fusion. We pass the image representation obtained from an image encoder to a transformer, then project its sequence length to the textual sequence length and obtain image representation I ∈ Rm×d, m is the sequence length at the decoder. Let the intermediate representation at the input of Image cross-attention be F ∈ Rm×d. We obtain query (Qd) from F and key (Kd) and value (Vd) vectors from I as shown in Equation 7.
  Qd Kd Vd   =   FWQd IWKd IWVd   (7)
Here WQd , WKd , and WVd ∈ Rd×d are trainable parameters. Then, we perform multi-head crossattention as shown in Equation 8 to obtain a new image-infused vector FI .
FI = Softmax( QdKd
T
√ dk )Vd (8)
We combine the image-infused vector FI with intermediate representation F using a gate mechanism shown in Equation 9 and Equation 10.
gd = σ([F ⊕ FI ]WFd + bFd) (9) Here WFd ∈ R2d×d and bFd ∈ Rd×1 are trainable parameters. We pass the fused representation F̂ to the upper layers of the decoder and generate the summary.
F̂ = F + gd ⊙ FI (10)

heading: 4 Experimental Results and Analysis
text: The following section outlines the datasets used for analysis, experimental setup, and evaluation discussion of the performance of the proposed model across a spectrum of both automatic and human evaluation metrics. A qualitative analysis of the generated summaries is conducted under various model configurations.

heading: 4.1 Datasets Used
text: To ensure the effectiveness of the proposed model, we conducted experiments on three multimodal clinical datasets: MMQS (Ghosh et al., 2024a), MMCQS (Ghosh et al., 2024b), and MM_CliConSummation (Tiwari et al., 2023). These datasets cover topics related to multimodal question summarization and multimodal dialogue summarization in the context of healthcare. The MMQS and MMCQS datasets contain approximately 3,015 data points and cover 18 medical symptoms. In comparison, the MM_CliConSummation dataset includes around 1,668 samples and encompasses 266 symptoms.

heading: 4.2 Experimental Setup
text: Our experimentation employed RTX 2080 Ti GPU, with each model requiring an average runtime of 20-30 minutes. Leveraging the PyTorch (Paszke
et al., 2019) and Hugging Face (Wolf et al., 2019) libraries, we executed both baselines and our proposed architecture. The foundation of our proposed model rests on the BART (Lewis et al., 2019). The dataset was partitioned into training, validation, and test sets in an 80:5:15 ratio. Hyperparameters included maximum epochs (30), maximum sequence length (360), visual embedding size (786), Adam optimizer, learning rate (5e-05), and batch size (32). We have performed the most extensive experiments in the case of the MMCS dataset. For the case of MMCS, our textual baselines featured T5 (Raffel et al., 2020), GPT-3.5 (OpenAI, 2023a), and BART (Lewis et al., 2019). For multimodal baselines, we used KM-CliConSummation (Tiwari et al., 2023) and GPT-4V (OpenAI, 2023b). Both GPT-3.5 Turbo and GPT-4V are used in the fewshot setting. In (Sahoo et al., 2024b), it is clearly stated that shot prompting is more effective in general than zero shot. Our proposed multimodal model, named EDI-Summ, is built on top of BART, where image fusion is done in both the encoder and decoder parts of the BART model. It is the first work where image fusion is performed in both the encoder and decoder portions of the BART model. EI-Summ is a simpler version of EDI-Summ where image fusion is done only in the encoder part of BART, inspired by (Kumar et al., 2022). We experimented with ResNet, VGG-Net, and VIT to generate image embeddings for our proposed multimodal model2. For the task of multimodal dialogue summarization, the chosen baselines were GPT-3.5, GPT-4V, and KM-CliConSummation (Tiwari et al., 2023). To evaluate the capability of handling codemixed text, we used the MMCQS dataset. The baselines selected for this task were GPT-3.5, GPT4, and the MedSumm models (Ghosh et al., 2024b) of LLAMA2 and Zephyr versions. We utilized automated metrics like BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) scores alongside human evaluation metrics. We collaborated with a healthcare professional and several medical students to conduct human evaluations. Our approach encompasses the assessment of three unique and medically nuanced metrics: clinical evaluation score, factual recall (Abacha et al., 2023), and omission rate (Abacha
2We conducted experiments with varying image embeddings on the MMQS dataset. However, we observed that the results did not differ significantly across these embeddings. Therefore, we did not extend similar experiments to the MMCQS and MM_CliConSummation datasets.
et al., 2023).

heading: 4.3 Automated Evaluation
text: Tables 1, 3, and 4 present the comprehensive evaluation of our proposed model, EDI-Summ, across three different multimodal medical summarization datasets: MMQS, MMCQS, and MM_CliConSummation. MMQS and MMCQS focus on multimodal medical query summarization, while MM_CliConSummation focuses on multimodal medical dialogue summarization.
R1) Comparison with baselines: From Table 1, we can infer that T5’s performance lagged behind the BART base model among the range of textual baseline models, leading to the conclusion that BART is a more advantageous choice for multimodal extension. In the case of multimodal baselines, KM-CliConSummation performed the best, followed by GPT-4V. For the task of medical dialogue summarization, as shown in Table 3, KM-CliConSummation achieved the highest performance. MedSumm(LLAMA-2) version was the top performer for code-mixed query summarization. The selection of these baselines is based on the works by (Ghosh et al., 2024b), (Ghosh et al., 2024a), and (Tiwari et al., 2023).
R2) Influence of Visual Cues: The impact of incorporating visual cues with textual queries for generating more nuanced summaries is evident across all metrics from Table 1, 3 and 4. The encoder image fusion model: EI-Summ and EDI-Summ exhibit significant superiority over models that rely solely on textual queries as input.
R3) Impact of Varied Pre-trained Vision Models for Generating Embeddings: In our exploration, as shown in Table 2, we investigated the impact of employing different pre-trained vision models to generate image embeddings for integration into the BART model. Our findings indicate that embeddings from ResNet and VGG tend to exhibit slightly superior performance compared to VIT, albeit the distinction is not markedly significant.
R4) Ablation study with different modality infusion orders: Table 2 elucidates how performance varies with changes in the infusion order within the encoder and decoder of the BART model employing VGG embeddings. Our experimental findings underscore a consistent trend, revealing that optimal results are consistently achieved when fusion occurs at layer 3 in both the encoder and
decoder of EDI-Summ. This pattern holds true across all pre-trained vision embeddings. In the scenario of the EI-Summ, the most favorable outcomes are attained through fusion at layer 3 of the encoder. 3
R5) Effective in handling codemixed text : Table-4 clearly suggests the effectiveness of EDISumm in handling Hinglish text. It performed much better than the baseline models that were proposed in (Ghosh et al., 2024b) .
R6) Impact of Decoder Visual Cross Attention: A consistent trend emerges with the enhancement of ROUGE, BLEU, and METEOR scores when using EDI-Summ instead of EI-Summ, as shown in Tables 1, 3, and 4. Our evaluation across these three multimodal clinical summarization datasets supports our intuition that decoder attention improves the alignment of multimodal information.

heading: 4.4 Human Evaluation
text: As hallucination is a big problem in multimodal generation (Sahoo et al., 2024a), we have conducted a rigorous human-level analysis of the generated summaries. A team of medical students, under the guidance of a doctor, conducted the human evaluation on 55 data samples from each of the generated summaries from the three datasets. For the baseline for human evaluation we decided to go with the generated summaries of GPT-4v as it is the state-of-the-art multimodal baseline. To evaluate the significance of decoder attention, we compare the results of EI-Summ and EDI-Summ across all three different datasets. The evaluation metrics included: Clinical Evaluation Score: The doctor and the team assigned ratings ranging from 1 (poor) to 5 (good), evaluating the summaries based on overall relevance, consistency, fluency, and coherence, Medical Fact-Based Metrics: Factual Recall (Abacha et al., 2023) and Omission Recall metrics (Abacha et al., 2023) were employed to measure how well the generated summary captured medical facts compared to the gold standard annotated summary. Table-5 presents the results of EDI-Summ, which outperforms all baseline methods like GPT-4V across the selected human evalua-
3In Table 2, we present our experiments on the MMQS dataset. We conducted similar experiments on the MMCQS and MM_CliConSummation datasets and observed consistent trends across all three datasets.
tion metrics on all three datasets.4

heading: 4.5 Qualtitative Analysis of Generated Summaries
text: In Figure 3, we examine two examples from MMQS dataset comparing the gold summary with
4One key observation is that GPT-4V sometimes misses important medical concepts in the generated summaries.
the BART (Text only) model, EI-Summ and EDISumm. In both instances, the text-only model exhibits the poorest performance. Among the remaining two models, it is evident that the Encoder-only model overlooks certain intricate clinical terms in the final summary, which is potentially crucial for analyzing a patient’s case. On the other hand, EDISumm consistently generates the most clinically
nuanced summaries, as shown in the highlighted text in red color in Figure 3. More qualitative analysis is shown in the Appendix section

heading: 4.6 Statistical Analysis of the Generated Summaries
text: The median number of words in gold summary is 24, and in generated summaries is around 23, so they are pretty close. The golden and predicted summaries’ word clouds are shown below in Figure 4.

heading: 5 Risk Analysis
text: Through comprehensive automated, human, and qualitative analyses, we have discerned that visual cues significantly contribute to generating more clinically nuanced summaries. However, particularly in sectors as critical as healthcare, the stakes are high for any potential misinformation or misinterpretation. In our human evaluation, instances were identified where the model, due to visual information, missed crucial keywords from the textual query. This occasional divergence underscores the need to carefully balance visual and textual inputs to ensure a more comprehensive summary. Moreover, our model is currently limited to recognizing 18 main symptoms. If presented with an image con-
taining symptoms outside this set, the model might inadvertently generate misinformation in the final summary. Additionally, the quality of the image plays a pivotal role in generating accurate summaries. Our experimentation with a low-quality image revealed a lack of proper detail capture. Hence, the involvement of a medical expert becomes im perative, particularly in high stakes, to mitigate the risks associated with potential inaccuracies or oversights.

heading: 6 Conclusion
text: In this paper, we introduced a multimodal model EDI-Summ, which incorporates contextual multimodal attention in both the encoder and decoder of the BART model. To evaluate the effectiveness of our proposed model, we conducted automated and human evaluations on three clinical multimodal summarization benchmark datasets. Our analysis suggests that incorporating contextual attention in the encoder and cross-attention in the decoder enhances the performance of multimodal summarization.

heading: 7 Limitations
text: There are some noticeable limitations of our work. Those are enumerated in the points below:
1) Although our comprehensive evaluation indicates that incorporating attention in both the encoder and decoder of the BART model achieves state-of-the-art performance for multimodal summarization, this claim lacks strong theoretical support.
2) In our experiments, we primarily focused on English and code-mixed settings. However, the effectiveness of this framework in handling multilingual text remains an open question. We plan to explore this research question in future work.
3) All the datasets MMQS, MMCQS, and MM_CliConSummation are multimodal, handling only text and images as inputs. However, the effectiveness of our proposed EDI-Summ in handling other modalities, such as videos and speech, remains to be explored in future work.

heading: 8 Ethics Statement
text: Our collaborative efforts extend to close coordination with a medical expert, who also holds the position of co-authorship on this paper. The execution of the entire task, spanning from experiment
design to human validation and qualitative analysis, involved the dedicated participation of three MBBS students, who volunteered for the project. As a testament to ethical considerations, these students were compensated following the prevailing minimum wage guidelines in India as outlined5. Furthermore, to reinforce the ethical integrity of our work, we are in the process of obtaining Institutional Review Board (IRB) approval for this project. It is important to note that the proposed model is designed solely for the task of summarization and does not include any predictive functionalities that could unfairly impact the user. This deliberate design choice underscores our commitment to ethical practices and responsible model usage.

heading: 9 Acknowledgements
text: Akash Ghosh and Sriparna Saha extend their sincere appreciation to the SERB (Science and Engineering Research Board) POWER scheme, Department of Science and Engineering, Government of India, for generously funding this research.

heading: A Appendix
text: A.1 Dataset and Code The code of our proposed model and dataset will be shared in this GitHub account https://github.com/AkashGhosh/From-Sights-toInsights-Towards-Summarization-of-MultimodalClinical-Documents.

heading: B Qualitative analysis
text: We conducted more analysis of the samples from the test set. Our exploration revealed that among the various baselines examined, EDI-Summ stood out remarkably in its ability to encapsulate intricate medical concepts. Notably, it excelled in accurately capturing the precise medical terminology derived from the images (highlighted in red), thus providing a more nuanced summary generation.
B.1 FAQs
1. Why BART is chosen over T5 as the base model? Ans: We chose BART-Base as the foundational model for our proposed method after thorough experimentation. Our extensive evaluations consistently revealed that BART-Base outperforms T5-Base across various metrics and perspectives. We favored BART-Base for its encoder-decoder architecture, allowing the encoding of diverse information before summary generation. Additionally, to align with our available computing resources, we concentrated on the Base versions of all models for our comparative analysis.
2. What are the qualifications of the doctor and the other annotators? How senior are they?
Ans: To uphold ethical standards, the entire process was conducted under the guidance of a senior medical doctor—an additional professor at a government medical college and a co-author of our paper. The team included three medical postgraduate students who were closely supervised by the doctor throughout the entire process.
3. Why do we use cross-attention instead of contextual cross-attention in the decoder of EDISumm?
Ans: We use cross-attention instead of contextual cross-attention in the decoder of EDI-Summ because while generating the tokens, the sequence length at the decoder increases by one every time we generate a single token. Due to this, we can’t calculate contextualized vector as given in Equation 2 because it requires the addition of key and value vectors from text and image, but since the sequence length of text keeps on increasing while generating tokens and the sequence length of the image representation is fixed, we can’t add these two vectors.
4. Are the results in our experiments statistically significant?
Ans: We conducted five runs for each of the top-performing text baseline (BART), multimodal baseline (EI-Summ), and our proposed model, EDISumm, followed by a t-statistical test. The resulting p-value, calculated at 0.004, signifies statistical significance with a confidence level of 95%. Consequently, we concluded that the observed differences are statistically significant.
5. How EDI is different from other multimodal baselines?
Ans: In our paper, EDI-Summ is constructed on the foundation of the BART model, where we
have integrated Multimodal Context-Aware Attention in the encoder and Image Cross Attention in the decoder. The EI-Summ model implements image fusion solely in the encoder, a methodology similar to that of Kumar et al. (). We conducted extensive experiments utilizing various pre-trained image embedding techniques, including VGG, VIT, and Resnet, observing performance enhancements compared to our text and other multimodal baselines (EI-Summ). Similarly, EDI-Summ performs the best in human evaluation metrics too. To the best of our knowledge, our work represents the inaugural attempt at incorporating image fusion in both the encoder and decoder of the BART model.
6. Why only multimodal baselines are used for analysis of results of MMCQS and MM_CliConSummation?
Ans :The insights gained from the experiments on the MMCS dataset allowed us to streamline the experiments for the MMCQS and MM_CliConSummation datasets. For these later datasets, we only experimented with multimodal baselines, as text-based baselines were not a good baseline for our model.
7.Definition of relevance, consistency and fluency in our work.
Ans: Certainly, here’s how each criterion would be defined in the context of human evaluation for medical passage summarization:
1. Fluency: The smoothness and clarity of the summary, evaluating how effectively the medical information is communicated coherently and understandably, without jargon or ambiguity.
2. Adequacy: The accuracy and comprehensiveness of the summary in encapsulating all crucial details and main points of the medical passage, ensuring that no vital information is omitted or misrepresented.
3. Informativeness: The degree to which the summary provides valuable and relevant medical insights, assessing its ability to convey essential information succinctly and clearly while avoiding extraneous or redundant details.
4. Persuasiveness: The convincing presentation of medical findings or recommendations in the summary, evaluating its ability to influence the reader’s understanding or decision-making process regarding the medical topic, potentially leading to further action or consideration.

heading: Future Work
text: We plan to explore this research question in future work. 3) All the datasets MMQS, MMCQS, and MM_CliConSummation are multimodal, handling only text and images as inputs. However, the effectiveness of our proposed EDI-Summ in handling other modalities, such as videos and speech, remains to be explored in future work