<figure><img src="image_1.png" /></figure>

HAL
open science

# Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence

Wei Sun, Mingxiao Li, Damien Sileo, Jesse Davis, Marie-Francine Moens

► To cite this version:

Wei Sun, Mingxiao Li, Damien Sileo, Jesse Davis, Marie-Francine Moens. Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence. ACM Transactions on Computing for Healthcare, 2025, 6 (2), pp.23. 10.1145/3712296. hal-05290617

HAL Id: hal-05290617

https://hal.science/hal-05290617v1

Submitted on 6 Oct 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence

WEI SUN and MINGXIAO LI, CS Department, KU Leuven, Leuven, Belgium

DAMIEN SILEO, Inria Lille, Villeneuve-d'Ascq, France

JESSE DAVIS and MARIE-FRANCINE MOENS, CS Department, KU Leuven, Leuven, Belgium

Medical Question Answering (medical QA) systems play an essential role in assisting healthcare workers in finding answers to their questions. However, it is not sufficient to merely provide answers by medical QA systems because users might want explanations, that is, more analytic statements in natural language that describe the elements and context that support the answer. To do so, we propose a novel approach for generating natural language explanations for answers predicted by medical QA systems. As high-quality medical explanations require additional medical knowledge, so that our system extracts knowledge from medical textbooks to enhance the quality of explanations during the explanation generation process. Concretely, we designed an Expectation-Maximization approach that makes inferences about the evidence found in these texts, offering an efficient way to focus attention on lengthy evidence passages. Experimental results, conducted on two datasets MQAE-diag and MQAE, demonstrate the effectiveness of our framework for reasoning with textual evidence. Our approach outperforms state-of-the-art models, achieving a significant improvement of 6.13 and 5.47 percentage points on the Rouge-L score; 6.49 and 5.28 percentage points on the Bleu-4 score on the MQAE-diag and MQAE datasets.

CCS Concepts: • **Computing methodologies** → **Natural language generation**; • **Applied computing** → **Health care information systems**;

Additional Key Words and Phrases: Expectation Maximization, Medical Question Answering, Explanation Generation

**ACM Reference format:**

Wei Sun, Mingxiao Li, Damien Sileo, Jesse Davis, and Marie-Francine Moens. 2025. Generating Explanations in Medical Question-Answering by Expectation Maximization Inference over Evidence. *ACM Trans. Comput. Healthcare* 6, 2, Article 23 (April 2025), 23 pages.

https://doi.org/10.1145/3712296

# 1 Introduction

Medical Question Answering (medical QA) is an essential component of modern healthcare. By automating the process of answering medical queries, it spares users the effort of manual searches. Despite the development of advanced medical QA systems [1, 20], the ideal medical QA systems should not only provide

This article is supported by AIDAVA (Horizon Europe: EU HORIZON-HLTH-2021-TOOL-06-03) and CHIST-ERA projects ANTIDOTE (ERA-NET CHIST-ERA IV FET PROACT JTC 2019).

Authors' Contact Information: Wei Sun (corresponding author), CS Department, KU Leuven, Leuven, Belgium; e-mail: sun.wei@kuleuven.be; Mingxiao Li, CS Department, KU Leuven, Leuven, Belgium; e-mail: mingxiao.li@kuleuven.be; Damien Sileo, Inria Lille, Villeneuve-d'Ascq, France; e-mail: damien.sileo@inria.fr; Jesse Davis, CS department, KU Leuven, Leuven, Belgium; e-mail: jesse.davis@kuleuven.be; Marie-Francine Moens, CS department, KU Leuven, Leuven, Belgium; e-mail: sien.moens@kuleuven.be.

<figure><img src="image_3.png" /></figure>

This work is licensed under Creative Commons Attribution International 4.0.

© 2025 Copyright held by the owner/author(s).

ACM 2637-8051/2025/4-ART23

https://doi.org/10.1145/3712296
accurate answers but also explanations that describe the elements and context that support a choice [7]. Ex-
planations serve as a linchpin for promoting the reliability of answers and facilitating fact-checking [7, 9], so
that users, i.e., patients and clinicians, can trust medical QA systems by addressing their concerns. An alter-
native explanation generation approach is to generate concise, self-contained natural language explanations
that are more comprehensible and accessible to users [4]. This approach can empower clinicians and patients
to better understand the medical QA system's behavior, facilitating informed decisions and improved patient
care [10].

In this article, we will explore what we call **Question-Answering Explanation (QAE)**, where the goal is to provide a natural language explanation of the answer or decision in a medical QA system. QAE provides explanations for the returned answer given a question, a patient's case description, and possibly external data. Prior works [2, 6] demonstrate that evidence retrieved from an external knowledge base helps to improve the quality and effectiveness of NLP systems. For medical explanation generation, a key challenge to improve the quality of explanations lies in how to fuse medical knowledge from multiple lengthy documents. This is difficult because it is often unclear which evidence pieces are relevant and each document is quite long. Classical multi-head attention mechanisms struggle to deal with multiple lengthy documents. It is expensive to conduct full attention across concatenated evidence because their time complexity increases exponentially as the number of input tokens increases. To overcome this challenge, we propose **Expectation-Maximization Inference over Evidence (EMIN)**, implemented as an encoder-decoder model with a dual encoder for encoding question and answer, and evidence texts. This approach iteratively optimizes both the relevance weights of each piece of evidence and the text generation model's parameters. This permits attending over separated evidence rather than performing full attention across concatenated evidence.

Our main contributions are the following:

- We introduce a novel task known as QAE in the medical domain. We present two new datasets, **Medical Diagnosis QAE (MQAE-diag)** and **Medical QAE (MQAE)**, curated from publicly available medical resources.

- We propose a novel framework, EMIN, designed for generating contextualized explanations. EMIN enables inference of the relevance of each piece of evidence and leverages attention in a scalable way.

- Experimental results show that our model outperforms state-of-the-art **Large Language Model (LLM)** based and multihop reasoning baselines by a large margin on the MQAE and MQAE-diag datasets and on other benchmark datasets across different evaluation scores. Notably, our model achieved improvements by 6.86 and 9.43 percentage points on the Rouge-1 score; 8.23 and 7.82 percentage points on the Bleu-4 score on the MQAE-diag and MQAE datasets.

This article is organized as follows: Section 2 details the EMIN model. Section 3 presents results on two MQAE datasets and an error analysis. Finally, Section 4 describes related works and Section 5 summarizes this article.

## 2 Methods

The goal of **QAE** is to generate a natural language explanation of the predicted answer to a given question. We utilize the ground-truth answer for training purposes and rely on the predictions generated by a trained QA system during the inference phase. We will approach this as a text-to-text generation task. More formally, for training we assume access to a dataset $((Q_1, A_1, E_1), Exp_1), \dots, (Q_N, A_N, E_N), Exp_N)$, where $Q_i$ is a question, $A_i$ is the answer to this question, $E_i$ refers to the set of relevant evidence paragraphs and $Exp_i$ is the corresponding explanation. The goal is to learn a question-answering system $\mathbf{f}': Q_i \to A_i$ and an explanation generator $\mathbf{f}: \{Q_i, A_i, E_i\} \to Exp_i$ where the $Exp_i$ is a text composed of natural language sentences.
<figure><img src="image_4.png" /><figcaption>Fig. 1. The overall architecture of the EMIN model (training stage). The evidence query is the question $Q_i$'s answer. Model parameters consist of the parameters of the question, evidence encoders, and dual cross-attention decoder. We use of ground-truth answers during the training phase, while relying on the predictions generated by a trained QA system during the inference phase.</figcaption></figure>

Figure 1 shows the overall architecture of our approach, which works as follows. First, we automatically identify a limited set of potential pieces of evidence in a document collection, typically represented as relevant paragraphs corresponding to each question. Second, we employ a text generation model based on a BART encoding-decoding neural architecture [11]. We fine-tune this model using an **Expectation-Maximization (EM)** approach. This iterative process refines not only the model's parameters, including attention parameters but also updates the importance weights associated with each potential evidence piece. During inference, we employ an analogous EM-based approach to generate explanations for question-answer pairs that have not been previously encountered. In this phase, we rely on a trained question-answering system to predict the answers. The following subsections describe each component of the approach in detail.

## 2.1 Evidence Selection

The initial step involves selecting a set of evidence paragraphs from a document collection that are likely to be relevant for generating the explanation. To do so, we employ the **Dense Passage Retrieval (DPR)** model [8], which retrieves documents related to the answer from a medical document collection (in our case medical textbooks). The retrieval query is the answer $A_i$. Before initiating the retrieval process, both documents and retrieval queries undergo preprocessing. This involves tokenization into words, removal of stop words, and conversion of all remaining words to lowercase. The DPR algorithm subsequently returns a set of evidence documents based on their relevance scores. Following the retrieval of documents, the next step involves breaking them down into evidence paragraphs. These evidence paragraphs are then ranked using the DICE similarity score concerning the textual retrieval query. The top-k ranked textual evidence paragraphs are selected, taking into account normalized word overlap.<fnref n="1" />

## 2.2 Neural Backbone

The dual encoder-decoder model is based on BART [11]. It has two encoders: one encodes the question-answer and the other encodes the evidence paragraphs. We initialize the dual encoder-decoder model using the pretrained parameters of the BART base model.

<footnote n="1">¹This approach resulted in better explanation generation results than first splitting the documents into evidence paragraphs and then ranking them with DPR.</footnote>
### 2.2.1 Encoder.
The BART encoder has a transformer architecture characterized by applying self-attention to the representations at a specific layer *l* of the transformer encoder. Let *X*<sub>*i*</sub> be the concatenation of the question string *Q*<sub>*i*</sub> and answer string *A*<sub>*i*</sub>. We pass *X*<sub>*i*</sub> to the encoder and use *X*<sub>*i*</sub><sup>*l*</sup> ∈ ℝ<sup>*L*<sub>*x*</sub>×*d*</sup> to represent its output at a specific layer *l* where *L*<sub>*x*</sub> is the sequence length of *X*<sub>*i*</sub> and *d* is the dimension of the hidden states. Evidence features *E*<sub>*i*</sub> are obtained by feeding each selected evidence paragraph *E*<sub>*i*,*j*</sub> ∈ ℝ<sup>*L*<sub>*e*</sub>×*d*</sup> into the evidence encoder. *E*<sub>*i*</sub><sup>*l*</sup> refers to the encoded evidence paragraphs at a specific layer *l* of the transformer encoder. *L*<sub>*e*</sub> represents the evidence's sequence length in terms of tokens.

### 2.2.2 Dual Cross Attention Decoder.

*Vanilla BART decoder.* The BART decoder has a transformer architecture characterized by self-attention and cross-attention over the tokens of the encoder. The decoder is composed of different layers. Given the hidden state of the *l*th layer, **H**<sup>*l*</sup> ∈ ℝ<sup>*L*<sub>*d*</sub>×*d*</sup>, which represents the hidden states of the decoder sequence. *L*<sub>*d*</sub> is the length of the decoder sequence in terms of tokens. The hidden state of the next layer **H**<sup>*l*+1</sup> is computed as follows:

$$SA_i^l = \text{Self-Attn}(H_i^l, H_i^l), \qquad (1)$$

$$CA_i^l = \text{Cross-Attn}(SA_i^l, X_i^l), \qquad (2)$$

where Self-Attn and Cross-Attn operations compute the scaled dot-product attention [19]. The decoder autoregessively generates explanations. That is, at each timestep of the decoding process, the input of the decoder is the embedding of the previous token (embedding of the ground truth token during training and embedding of the generated token during inference).

*EMIN Decoder.* In EMIN, we explicitly model evidence weights as *z*<sub>*i*</sub> ∈ ℝ<sup>*k*</sup> that capture the relevance of each selected evidence paragraph for instance *i*. As we do not have ground truth evidence weights, we infer them using an EM algorithm. The inferred evidence weights make the attention of many texts or on a long text split in paragraphs scalable for realistic applications.

The EMIN decoder follows the general architecture of the vanilla BART decoder. This entails that the hidden state of each layer is updated with the context vector that contains the encoded information of the evidence tokens weighted by the cross-attention weights obtained by Cross-Attn($CA_i^l, E_i^l$). In EMIN, each evidence paragraph $E_{i,j}$ is also weighted with its evidence weight $z_{i,j}$ by element-wise multiplication of their respective vectors.

The intuition is that every transformer layer of the decoder cross-attends to relevant features extracted by the corresponding encoder layer builds a representation to be decoded that captures different linguistic information. The representation of each layer can be used to update the evidence weight $z_{i,j}$. Equation (3) becomes:

$$H_i^{l+1} = z_i \odot \text{Cross-Attn}(CA_i^l, E_i^l). \qquad (3)$$

## 2.3 EMIN

The evidence weights $z_i \in ℝ^k$ capture the importance of the evidence paragraphs $E_i$. The evidence weights are the hidden variables in the EM algorithm with $\text{sum}(z_i) = 1$. During training for each question-answer input $X_i$, the weights of the evidence paragraphs $z_i$ and parameters $\Theta$ of the dual encoder-decoder model are iteratively optimized by the EM algorithm. In the E-step, we update the evidence weights $z_i$ using the ground-truth explanations. In the M-step, we use the back-propagation algorithm to optimize the dual encoder-decoder model's parameters $\Theta$ given the expected values of $z_i$. During inference, we only iteratively update the evidence weights $z_i$ based on the trained explanation generation model to gain insight into the contribution of each evidence paragraph in generating the explanation. The weights of the evidence paragraphs $z_i$ of an input example are uniformly initialized as $\frac{1}{k}$ unless mentioned otherwise.
### 2.3.1 EM during Training. E-Step.
In the E-step at each time step t, given input features $X_i$, the parameters of the dual encoder-decoder model at time step t − 1, and evidence representation $E_i = [E_{i,1}, E_{i,2}, \dots, E_{i,k}]$, we generate explanation $\overline{Exp}_{i,j}^{t-1}$. We first compute the inverse Cross-Entropy (CE) between the generated and ground truth explanations:

$$c_{i,j}^{t} = \frac{1}{-\sum_{m=1}^{L_d} P(Exp_{i,m}^{t-1}) \log(P(\overline{Exp}_{i,j,m}^{t-1}))}, \quad (4)$$

where $P(\overline{Exp}_{i,j,m})$ is the generated token probability at position m of sample i when only evidence j is used, and $P(Exp_{i,m})$ denotes the ground truth token at the same position of sample i, which is just a one-hot encoding in our case.

The expected values of $z_i^t$ are then computed as follows:

$$z_{i,j}^{t} = \frac{\exp^{\frac{c_{i,j}^{t}}{\lambda}}}{\sum_{j=1}^{k} \exp^{\frac{c_{i,j}^{t}}{\lambda}}}, \quad (5)$$

where $\lambda$ is a temperature parameter that controls the softness of the evidence weights distribution. We gradually reduce $\lambda$ using $\lambda = e^{-0.01 \times \text{iteration}}$, so that weights will increasingly concentrate on the most important pieces of evidence.

Intuitively, a piece of evidence should have a high weight if using it produces a generated explanation that is close to the ground truth one. We leverage the inverse CE to measure the similarity. The lower the value of this metric is, the closer the distance between the generated explanation and the ground-truth is, meaning that the model should pay more attention to this evidence paragraph by assigning it a larger weight.

### M-Step.
In the M-step, at iteration step t given the input features $X_i$, and the output ground truth explanation $Exp_i^t$, we optimize the dual encoder-decoder model's parameters $\Theta^t$ with backpropagation using the CE loss commonly used in text generation:

$$\mathcal{L} = - \sum_{i=1}^{N} \sum_{m=1}^{L_d} P(Exp_{i,m}^{t}) \log P(\overline{Exp}_{i,m}^{t}), \quad (6)$$

where $L_d$ refers to the length of the ground truth sequence. N is the number of training instances. Note that during explanation generation, the decoder integrates the evidence weights computed in the E-step in the way described in Section 2.2.2.

The EM iteration stops when the KL divergence between the evidence weights of iteration t and those of iteration t − 1 is smaller than a defined threshold $\epsilon$. We summarize the EM algorithm of the training stage in Algorithm 1.

We opted for the EM algorithm for MLE because it is particularly well-suited for handling latent variable models, offering both computational efficiency and relative simplicity in convergence guarantees for our problem's scale and structure. EM's iterative approach to estimating evidence weights via the E-step, followed by generator's parameter optimization in the M-step, proved effective given the underlying data distribution.

### 2.3.2 Inference.
When generating an explanation at test time, the model parameters $\Theta^t$ are fixed, but we still need to estimate the weights of the evidence paragraphs $z_i$ for the test question. Here, we use the generated explanations to infer the weights of the evidence paragraphs. In each iteration t, we calculate the expected weight of the evidence in a manner similar to the E-step used during training. However, we modify this step by evaluating the CE of the generated explanation and using the predicted answer from a trained question-answering system, namely BioLinkBERT [24]. The global explanation generated at t − 1: We assume generated explanations of the trained model are high-quality, so that good explanations can provide our EM algorithm a good starting point
**Algorithm 1:** EM algorithm in the training stage

**Data:** Input $X_i$; Model Parameters $\Theta$; Evidence Weights $z_i$; Evidence $E_i$; Ground-truth Explanation $Exp_i$; KL divergence threshold $\epsilon$; Convergence iteration $T$

$t = 1$;

Initialize evidence weights $z_i$ with uniform values;

Initialize model parameters $\Theta^{t-1}$ as the pretrained parameters of the BART encoder-decoder;

**repeat**

Compute evidence weights $z_i^t$ using the predicted explanations per evidence $E_{i,j}$ with model parameters $\Theta^{t-1}$;

/* E-step */

Optimize model parameter $\Theta^t$ with the explanation generation loss while integrating $z_i^t$ in the decoding;

/* M-step */

$t = t + 1$;

**until** $|z_i^{t-1} - z_i^t| \le \epsilon$;

Output: final model parameter $\Theta^T$

**Algorithm 2:** EM Algorithm in the Testing Stage

**Data:** Input $X_i$; Trained Model Parameters $\Theta$; Evidence Weights $z_i$; Evidence $E_i$; KL divergence threshold $\epsilon$; Convergence iteration $T'$

$t = 1$;

Initialize evidence weights $z_i^{t-1}$ with uniform values;

**repeat**

Generate $\overline{Exp}_i^{t-1}$ as reference using model parameters $\Theta$ while integrating $z_i^{t-1}$ in the decoding;

Compute the predicted explanation per evidence $E_{i,j}$: $\overline{Exp}_{i,j}^{t-1}$ while integrating $z_i^{t-1}$ in the decoding;

Update evidence weights $z_i^t$ according to equations 7 and 8;

$t = t + 1$;

**until** $|z_i^{t-1} - z_i^t| \le \epsilon$;

Output: final explanation $\overline{Exp}_i^{T'}$ using model parameters $\Theta$ while integrating final evidence weights $z_i^{T'}$

during the inference stage. Experimental results show that the inference EM can improve results across all evaluation metrics.

$$c_{i,j}^t = \frac{1}{-\sum_{m=1}^{L_d} P(\overline{Exp}_{i,m}^{t-1}) \log P(\overline{Exp}_{i,j,m}^{t-1})}, \quad (7)$$

$$z_{i,j}^t = \frac{\exp^{-\frac{c_{i,j}^t}{\lambda}}}{\sum_{j=1}^{k} \exp^{-\frac{c_{i,j}^t}{\lambda}}}. \quad (8)$$

The stopping criterion is the same as in the training phase. After $z_i$ has converged, we generate the final explanation. Algorithm 2 shows the overall working of the algorithm in the inference stage. Through EM inference, the model at inference time learns which pieces of evidence have contributed the most to the explanation, and which contributed less.
# 3 Experiments

## 3.1 Datasets

In our study, we utilize two medical QAE datasets, namely the MQAE-diag and the MQAE datasets, in conjunction with a repository of textual medical evidence documents. The distinction between MQAE-diag and MQAE lies in the nature of their questions. MQAE-diag is tailored for questions related to patient diagnosis, whereas MQAE is designed to evaluate fundamental medical knowledge.<fnref n="2" />

**MQAE-diag.** The MQAE-diag dataset's samples are derived from the MedMCQA dataset [15], and two medical examination databases, i.e., the Spanish medical residency exam (MIR)<fnref n="3" /> and the **United States Medical Licensing Examination (USMLE).**<fnref n="4" /> In this dataset, there are 8038 samples for training, 995 instances for validation and 987 for testing.

MedMCQA gathers 194k multiple-choice questions paired with the correct answer and explanation, crawled in medical textbooks from various sources. We use textbooks from Curso MIR Asturias which are learning material for the Spanish medical residency exam but translated to English. The books are composed of medical multiple choice questions from the 2012–2020 exams, annotated with local explanations (each choice is motivated individually). The USMLE sample questions are publicly available each year which amounts to 1,083 annotated English questions, annotated by a medical doctor. These multiple-choice questions mostly deal with diagnoses determination, causes of findings, or appropriate procedure determination. Each sample of the MQAE-diag dataset contains a question, an answer, and a ground-truth explanation. The question consists of descriptions about the patient’s case and a clinical question. Instances from the MQAE-diag dataset are close to real medical diagnosis scenarios. Given the descriptions of the situation of a patient, a clinical question is asked, and the answer is provided. The ground-truth explanations explain why this answer is correct given the patient’s clinical history.

**MQAE.** The MQAE dataset is based on the MedMCQA dataset [15]. We extract 15,993, 2,000, and 1,999 clinical instances for training, validation and testing.

**CasiMedicos.** The CasiMedicos dataset [15] contains 434 samples for training, 125 samples for testing, and 63 samples for development.

**Document Collection with Evidence.** We rely on a human evaluation of the credibility of the sources that the retriever uses. Having such a human control of the sources of evidence is an advantage compared to relying on LLMs for evidence collection. In our experiments we use StatPearls<fnref n="5" /> as a source of evidence. StatPearls is a well-known and credible source [22, 23]. Our evidence collection contains 9070 medical textual documents. Then, we utilize the DPR retriever to retrieve relevant evidence from textual documents and use this collection to retrieve top-k (k = 10). We removed stopwords and numbers from retrieval queries and external evidence paragraphs. We used the DPR algorithm to retrieve evidence documents. Evidence retrieval is discussed in Section 2.1.

## 3.2 Experimental Set-Up

**Model Settings.** The trained QA system is the Med-PaLM [18]. We employ the system on the multiple options question-answering tasks and it achieves an accuracy of 71.13% on the MQAE-diag dataset, 70.49% on the MQAE dataset, and 71.19% on the CasiMedicos dataset. The maximum length of questions and evidence spans is 944 and the explanations' maximum length is 128. The size of the hidden representations of the model is 768. The number of selected evidence paragraphs *k* is set to 10 (we evaluate the influence of the value of this hyperparameter). For the ChatGPT, we set its temperature as 0.5 to provide outputs with balanced creativity and coherence.

**Training and Inference Details.** (1) For the training, all baseline models and our framework are trained with fp16. All of the models utilize the AdamW optimizer [12] with a learning scheduler initialized at $2e^{-5}$ and linearly

<footnote n="2">²https://github.com/VRCMF/EMIN.git</footnote>

<footnote n="3">³https://www.curso-mir.com/index.html</footnote>

<footnote n="4">⁴https://www.usmle.org/</footnote>

<footnote n="5">⁵https://www.statpearls.com/.</footnote>
<table><caption>Table 1. Experimental Results in Terms of ROUGE (R) and BLEU scores (%) on the MQAE-diag Dataset</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>T5<sub>q</sub></td><td>15.76±0.87</td><td>3.03±0.59</td><td>12.38±0.67</td><td>1.76±0.54</td><td>78.36±0.49</td></tr><tr><td>BART<sub>q</sub></td><td>18.32±1.69</td><td>4.30±0.88</td><td>12.76±1.19</td><td>1.17±0.51</td><td>80.78±0.47</td></tr><tr><td>T5<sub>q+a</sub></td><td>15.46±1.28</td><td>4.72±0.84</td><td>12.21±1.04</td><td>2.21±0.81</td><td>79.91±0.38</td></tr><tr><td>BART<sub>q+a</sub></td><td>18.56±1.99</td><td>5.72±1.16</td><td>13.13±1.26</td><td>0.81±0.10</td><td>80.52±0.39</td></tr><tr><td>SIMI<sub>q+a+e</sub></td><td>18.86±2.44</td><td>6.20±1.93</td><td>13.80±2.26</td><td>1.51±0.63</td><td>80.30±0.41</td></tr><tr><td>MHOP<sub>q+a+e</sub></td><td>21.35±2.54</td><td>6.79±2.32</td><td>14.77±1.91</td><td>1.83±0.57</td><td>81.89±0.45</td></tr><tr><td>MEAN<sub>q+a+e</sub></td><td>22.96±2.51</td><td>7.59±1.88</td><td>16.14±1.52</td><td>2.67±0.96</td><td>81.47±0.37</td></tr><tr><td>ChatGPT<sub>q+a</sub></td><td>30.41±3.27</td><td>8.56±3.83</td><td>17.75±3.79</td><td>4.40±2.01</td><td>82.06±0.37</td></tr><tr><td>EMIN</td><td><b>32.83±3.34</b></td><td><b>12.53±3.01</b></td><td><b>21.61±3.07</b></td><td><b>9.23±3.02</b></td><td>83.69±0.45</td></tr></tbody></table>

"q," "a," and "e" represent questions, answers, and evidence, respectively. The best results are highlighted in bold.

<table><caption>Table 2. Experimental Results in Terms of ROUGE (R) and BLEU Scores (%) on the MQAE Datasets</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>T5<sub>q</sub></td><td>9.47±1.12</td><td>2.47±0.60</td><td>8.83±1.36</td><td>1.79±0.37</td><td>78.61±0.51</td></tr><tr><td>BART<sub>q</sub></td><td>15.83±1.85</td><td>6.12±1.53</td><td>13.22±1.74</td><td>0.36±0.24</td><td>77.23±0.39</td></tr><tr><td>T5<sub>q+a</sub></td><td>14.66±1.43</td><td>5.93±1.02</td><td>13.31±1.51</td><td>0.32±0.28</td><td>79.35±0.40</td></tr><tr><td>BART<sub>q+a</sub></td><td>15.63±1.44</td><td>6.61±1.44</td><td>13.44±1.35</td><td>0.71±0.26</td><td>79.18±0.31</td></tr><tr><td>SIMI<sub>q+a+e</sub></td><td>18.28±2.01</td><td>7.83±1.66</td><td>15.27±1.92</td><td>0.87±0.35</td><td>80.14±0.36</td></tr><tr><td>MHOP<sub>q+a+e</sub></td><td>20.05±2.21</td><td>8.15±1.88</td><td>16.82±1.95</td><td>1.34±0.62</td><td>81.23±0.31</td></tr><tr><td>MEAN<sub>q+a+e</sub></td><td>21.26±1.96</td><td>9.26±2.07</td><td>17.74±1.83</td><td>1.64±0.47</td><td>82.07±0.39</td></tr><tr><td>ChatGPT<sub>q+a</sub></td><td>24.55±2.14</td><td>10.34±3.76</td><td>18.62±2.47</td><td>3.62±1.93</td><td>82.45±0.37</td></tr><tr><td>EMIN</td><td><b>28.05±2.72</b></td><td><b>12.53±3.13</b></td><td><b>21.61±2.27</b></td><td><b>9.23±2.81</b></td><td>83.55±0.43</td></tr></tbody></table>

"q," "a," and "e" represent questions, answers, and evidence, respectively. The best results are highlighted in bold.

decreased to 0. We apply the early stopping strategy on the baseline models by monitoring the loss on the validation set while the patience number is set to 4. The KL divergence threshold $\epsilon$ for our approach in the training stage is 0.01. (2) During the inference stage, we use beam search to generate text and set the width of the beam to 5. $\epsilon$ in the inference stage is also 0.01.

## 3.3 Baseline Models

Due to the maximum length of the BART encoder and other language model based encoders, we cannot use the concatenation of the question, answer and all selected evidences as the input. Hence the simple baselines below do not use retrieved evidence paragraphs. Their results are mentioned as a kind of underbound of the performance. The generated explanation here only relies on the pretrained language model.

T5 (q) [16] takes questions as input to generate explanations by the T5 model.

T5 (q+a) [16] passes the concatenation of questions and answers to the T5 language model to generate explanations.

BART (q) [11] inputs questions into the BART model to generate explanations.
<table><caption>Table 3. Experimental Results in Terms of ROUGE (R) and BLEU Scores (%) on the CasiMedicos Dataset</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>T5<sub>q</sub></td><td>18.02±0.65</td><td>4.07±0.13</td><td>13.73±1.03</td><td>1.73±0.24</td><td>84.42±0.52</td></tr><tr><td>BART<sub>q</sub></td><td>18.29±0.84</td><td>4.21±0.30</td><td>13.70±1.15</td><td>1.81±0.44</td><td>83.82±0.47</td></tr><tr><td>T5<sub>q+a</sub></td><td>18.66±0.64</td><td>4.12±0.13</td><td>14.12±1.07</td><td>1.76±0.28</td><td>84.54±0.50</td></tr><tr><td>BART<sub>q+a</sub></td><td>19.35±0.59</td><td>4.44±0.51</td><td>14.31±1.28</td><td>1.82±0.57</td><td>84.91±0.59</td></tr><tr><td>SIMI<sub>q+a+e</sub></td><td>20.02±0.81</td><td>4.80±0.43</td><td>14.71±0.52</td><td>1.77±0.40</td><td>85.33±0.61</td></tr><tr><td>MHOP<sub>q+a+e</sub></td><td>20.10±0.76</td><td>5.27±0.47</td><td>15.62±1.12</td><td>2.32±0.53</td><td>85.21±0.73</td></tr><tr><td>MEAN<sub>q+a+e</sub></td><td>21.32±0.96</td><td>5.08±0.55</td><td>14.85±1.69</td><td>2.96±0.43</td><td>85.61±0.67</td></tr><tr><td>ChatGPT<sub>q+a</sub></td><td>23.40±1.18</td><td>5.47±0.58</td><td>16.21±2.00</td><td>3.36±0.47</td><td>85.77±0.58</td></tr><tr><td>EMIN</td><td><strong>25.30±1.34</strong></td><td><strong>7.07±0.79</strong></td><td><strong>17.30±2.09</strong></td><td><strong>3.87±0.51</strong></td><td>86.28±0.73</td></tr></tbody></table>

The best results are highlighted in bold.

BART (q+a) [11] simply concatenates questions and answers as input, which is fed into the BART model to generate explanations.

The following models retrieve paragraphs in the same way as done in our model (see 2.1), and the evidence are individually encoded with the BART encoder (see 2.2.1).

SIMI (q+a+e) removes EMIN's E-step and computes the evidence weights $z'_i$ with the dot product similarity between the encoder outputs of question-answer representations and evidence features normalized by a softmax function. SIMI leverages the EMIN's decoder to incorporate evidence. The SIMI's neural backbone is the BART model while its inputs are questions, answers, and evidence. The SIMI model provides evidence weights by multiplying two encoders' features. We apply mean operation on two encoders' features and multiply two output features to provide the evidence weights, which is denoted as:

$$z'_{i} = \text{Mean}(\mathbf{X}_{i}) * \text{Mean}(\mathbf{E}_{i})^{T}, \qquad (9)$$

$$z'_{i} = \text{Softmax}(z'_{i}), \qquad (10)$$

where the Mean operation average features (average calculation ignores padding locations) along the sequence length dimension. Softmax represents the softmax function. The model's optimization follows the M-step in the Section 2.3.1.

MHOP (q+a+e) Following the literature on Key-Value Memory Networks [13], MHOP encodes the evidence paragraphs with the BART encoder and attends in multiple hops to this encoded "memory". The MHOP iteratively (iterations of *H* hops) looks for relevant evidence from the evidence memory (composed of the encoding of each evidence paragraph), and then fuses evidence features into the question and answer representation $\mathbf{X}_i$ through summation. The updated question and answer representation again attends over the memory and retrieves updated evidence features. This process lasts for *H* iterations. The final question-answer representation is then input into the decoder to autoregressively generate the explanation. MHOP uses the vanilla BART model for decoding (see Section 2.2.2). MHOP relies solely on attention weights to find relevant evidence tokens in the evidence memory, and attention weights and other model parameters are updated during the *H* hops. We test the MHOP model with different numbers of hops and present the results for the best model, which is 2-hop in our case.

MEAN (q+a+e) discards the E-step in EMIN, and uses uniform evidence weights to fuse the evidence features, so evidence weights are not learned with EM inference. The dual encoder-decoder is the same as for EMIN.
The following models retrieve paragraphs in the same way as done in our model (see Section 2.1), and the evidence
are individually encoded with the BART encoder (see Section 2.2.1).

ChatGPT (*q+a*) Because it is computationally heavy to directly fine-tune the GPT-3.5 Turbo-06-13, thus we leverage
a more effective technique, i.e., zero-shot in-context learning [3] to prompt predictions for question-answering
and explanation generation tasks. For the multiple options question-answering tasks, the ChatGPT achieves an
accuracy of 45.71% on the MQAE-diag dataset and 47.36% for on the MQAE dataset.

3.4 Results and Discussion

To compare with baseline models, we report the results of models on the ROUGE (recall oriented metric and
coded further as R) and BLEU-4 scores (precision oriented metric) that compare the ground-truth explanation
with the generated explanation. The numbers in the metric names refer to the size of n-gram of words considered.
ROUGE-L (RL) takes into account the length of the longest common subsequence of words. Tables 1–3 report test
set performance for all models on both datasets.

MQAE-diag. Our model outperforms the baseline models by large margins across all evaluation metrics. Compared to the baseline models that use encoded evidence paragraphs to generate explanations (i.e., SIMI, MHOP, and MEAN), EMIN improves the BLEU-4 score by 8.35, 8.14, and 7.82 percentage points, respectively. MEAN is the best evidence-enhanced model, and EMIN outperforms it by 9.67, 6.10, 7.15, and 8.23 percentage points on the R1, R2, RL and BLEU scores, respectively. EMIN beats the baselines that do not consider evidence paragraphs by even larger margins. Compared to the LLM, i.e., ChatGPT, our method increases by 2.14, 4.56, 6.13, and 6.49 percentage points on the R1, R2, RL, and BLEU scores, respectively.

MQAE. On the MQAE dataset, EMIN improves all evaluation scores, especially the BLEU score. Our approach outperforms the best no-evidence model, i.e., BART (q+a), by 12.42, 6.28, 7.61, and 8.49 percentage points on the R1, R2, RL and BLEU scores, respectively. In particular, our framework improve the R1 scores by 9.88, 8.00, and 6.86 percentage points compared with the evidence-enhanced models (SIMI, MHOP, and MEAN). The results show that EMIN yields large gains over all competitors and the evidence based approaches are all much better than the non evidence based ones. We compare our framework to the ChatGPT, the EMIN increases by 3.38, 5.52, 5.47, and 5.28 percentage points on the R1, R2, RL and BLEU-4 scores, respectively.

Discussion. we compare the results presented in Tables 1–3 to emphasize the significance of our proposed EM
inference method. Unlike the MEAN model, which employs uniform weights throughout, EMIN iteratively updates
evidence weights. Essentially, MEAN can be seen as the starting point of EMIN, where there is no distinction in
assigning importance to each evidence paragraph when generating explanations. While both the MHOP and
EMIN models iteratively adjust attention weights on the tokens within individual evidence paragraphs within the
dual encoder-decoder architecture, it is worth noting that MHOP does not successfully capture the importance
of evidence paragraphs. Here, EM inference offers an elegant solution for reasoning at the paragraph level. To
demonstrate the efficiency of EM inference, we conducted a comparison between EM inference and a model using
full attention across concatenated evidence paragraphs. Assume that we have *m* paragraphs and each paragraph
has *n* words, the time complexity of the EM inference and the full attention one are *O*(*m*²·*n*) and *O*(m²·n²),
respectively.<fnref n="6" /> Consequently, EM inference proves to be more efficient than the full attention model.

3.5 Analysis

3.5.1 EMNIN and MHOP Convergence during Inference. Figure 2 shows the BLEU-4 scores and entropy of the EMIN and MHOP models’ evidence weights for samples that converged in four iterations on two MQAE datasets. The entropy indicates that EMIN results in faster convergence of the evidence weights compared with

<footnote n="6">⁶It is impossible to compute full attention across concatenated evidence paragraphs with given the available computational power. The length for each evidence text is 95 ± 59 (mean±standard deviation) words.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 2. Comparison of the EMIN and MHOP's evidence weights obtained on the MQAE datasets. The bars represent the BLEU-4 scores of two models while the curves are the entropy. (e) and (b) are abbreviations for Entropy and BLEU-4.</figcaption></figure>

the multi-hop reasoning (MHOP), while obtaining better BLEU-4 scores. For all evaluation metrics, the difference between the EMIN and MHOP's evaluation results are statistically significant (paired *t*-test, and *p* < 0.001).

The goal of EM inference during testing is to assign evidence weights to the selected evidence paragraphs to increase the transparency of the generated explanation. We do not expect that the inference leads to better explanations as during testing we do not have any supervisory signal of the ground truth explanation. The EM algorithm very slightly improves the R1, R2, RL, and BLEU-4 scores by 0.52, 0.33, 0.33, and 0.37 percentage points, respectively, in the MQAE-diag test set. For the MQAE test set, the algorithm very slightly improves the BLEU-4, R2, RL scores by 0.03, 0.08, 0.07 percentage points, respectively. The difference between the evaluation scores of EMIN (uniform evidence weights) and EMIN (EM inference) are not statistically significant (paired *t*-test, and 0.4 < *p* < 0.5). The results confirm our expectation.

### 3.5.2 Influence of the Number of Evidence Paragraphs.
EMIN and the baselines consider the same number of retrieved evidence paragraphs. Here we study the relation between this number and the model performance. Tables 4 and 5 show the results for the three best evidence-enhanced QAE models (i.e., MHOP, MEAN, and EMIN) where we vary $k \in \{3, 5, 10, 12\}$. Specially, Figures 3 and 4 show changes of evaluation scores by setting different number of evidence paragraphs. Regardless of the model, considering more evidence improves performance. Increasing the number of evidence paragraphs generally means that the model has access to more and richer information, possibly offering different perspectives and details. This can help the model understand the question more fully when generating explanations, thereby improving the quality of the generated explanation. Increasing the number of evidence paragraphs can also increase the model's learning ability: During the training process, the model can better capture patterns in the data by learning the relationships and content of multiple evidence documents. We also note the improvement of EMIN's score when the amount of evidence increases. This shows that EMIN is able to use the additional evidence effectively, thereby improving the quality of the generated explanation. This performance may be related to the design of the EMIN model in terms of information integration and feature selection, which makes it perform better when dealing with a large amount of evidence. In contrast, the scores of MHOP and MEAN improve more slowly, probably because they fail to effectively use the newly added information. This shows that they are less robust when dealing with multi-evidence scenarios.

### 3.5.3 Influence of the Evidence Weight Temperature.
The temperature value of Equation (5) controls the entropy of the evidence weights' distribution during training. As mentioned in Section 2.3.1, we reduce the temperature
<table><caption>Table 4. Top-3 Evidence-Enhanced Models’ Results (in %) on the MQAE-diag Dataset by Setting Different Numbers of Evidence Documents</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>MHOP<sub>3</sub></td><td>20.77</td><td>6.23</td><td>14.02</td><td>1.64</td><td>81.96</td></tr><tr><td>MHOP<sub>5</sub></td><td>21.09</td><td>6.52</td><td>13.98</td><td>1.58</td><td>81.34</td></tr><tr><td>MHOP<sub>10</sub></td><td>21.35</td><td>6.79</td><td>14.77</td><td>1.83</td><td>81.89</td></tr><tr><td>MHOP<sub>12</sub></td><td><strong>21.69</strong></td><td><strong>6.97</strong></td><td><strong>15.17</strong></td><td><strong>2.03</strong></td><td>82.35</td></tr><tr><td>MEAN<sub>3</sub></td><td>20.29</td><td>5.53</td><td>14.21</td><td>1.69</td><td>81.25</td></tr><tr><td>MEAN<sub>5</sub></td><td>21.36</td><td>6.87</td><td>14.25</td><td>1.77</td><td>81.46</td></tr><tr><td>MEAN<sub>10</sub></td><td>22.96</td><td>7.59</td><td>16.14</td><td>2.67</td><td>81.47</td></tr><tr><td>MEAN<sub>12</sub></td><td><strong>22.87</strong></td><td><strong>7.13</strong></td><td><strong>16.52</strong></td><td><strong>2.85</strong></td><td>81.93</td></tr><tr><td>EMIN<sub>3</sub></td><td>22.93</td><td>7.47</td><td>16.93</td><td>3.09</td><td>82.91</td></tr><tr><td>EMIN<sub>5</sub></td><td>27.38</td><td>10.59</td><td>20.71</td><td>7.40</td><td>83.58</td></tr><tr><td>EMIN<sub>10</sub></td><td>32.83</td><td>12.53</td><td>21.61</td><td>9.23</td><td>83.69</td></tr><tr><td>EMIN<sub>12</sub></td><td><strong>32.78</strong></td><td><strong>13.17</strong></td><td><strong>22.69</strong></td><td><strong>10.79</strong></td><td>83.62</td></tr></tbody></table>

"3, 5, 10, and 12" correspond to the evidence number identifiers. The best results are highlighted in bold.

<table><caption>Table 5. Top-3 Evidence-Enhanced Models’ Results (in %) on the MQAE Dataset by Setting Different Numbers of Evidence Documents</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>MHOP<sub>3</sub></td><td>18.27</td><td>7.56</td><td>15.18</td><td>0.87</td><td>79.64</td></tr><tr><td>MHOP<sub>5</sub></td><td>19.73</td><td>7.88</td><td>16.67</td><td>0.98</td><td>79.81</td></tr><tr><td>MHOP<sub>10</sub></td><td>20.05</td><td>8.15</td><td>16.82</td><td>1.34</td><td>81.23</td></tr><tr><td>MHOP<sub>12</sub></td><td><strong>20.29</strong></td><td><strong>8.14</strong></td><td><strong>16.92</strong></td><td><strong>1.32</strong></td><td>81.64</td></tr><tr><td>MEAN<sub>3</sub></td><td>18.53</td><td>8.06</td><td>15.39</td><td>0.80</td><td>81.30</td></tr><tr><td>MEAN<sub>5</sub></td><td>20.26</td><td>8.53</td><td>16.58</td><td>1.15</td><td>81.65</td></tr><tr><td>MEAN<sub>10</sub></td><td>21.26</td><td>9.26</td><td>17.74</td><td>1.64</td><td>82.07</td></tr><tr><td>MEAN<sub>12</sub></td><td><strong>21.96</strong></td><td><strong>9.89</strong></td><td><strong>18.17</strong></td><td><strong>1.76</strong></td><td>82.69</td></tr><tr><td>EMIN<sub>3</sub></td><td>20.19</td><td>9.02</td><td>16.46</td><td>3.72</td><td>82.30</td></tr><tr><td>EMIN<sub>5</sub></td><td>24.57</td><td>10.44</td><td>18.87</td><td>6.94</td><td>82.81</td></tr><tr><td>EMIN<sub>10</sub></td><td>28.05</td><td>12.53</td><td>21.61</td><td>9.23</td><td>83.55</td></tr><tr><td>EMIN<sub>12</sub></td><td><strong>28.43</strong></td><td><strong>12.62</strong></td><td><strong>21.43</strong></td><td><strong>9.43</strong></td><td>83.86</td></tr></tbody></table>

"3, 5, 10, and 12" correspond to the evidence number identifiers. The best results are highlighted in bold.

with increasing epochs, that is, $t' = e^{-0.01 \times epoch}$. Tables 6 and 7 show that annealing with this temperature value outperforms using a fixed temperature value especially with regard to the BLEU-4 score.

### 3.5.4 EM during Testing.
The goal of EM inference during testing is to assign evidence weights to the selected evidence paragraphs to increase the transparency of the explanation generation. We do not expect that the inference leads to better explanations as during testing we do not have any supervisory signal of the ground truth explanation. Anyway, it is interesting to investigate how the EM algorithm affects EMIN’s performance during the inference stage. The EM algorithm improve the R1, R2, RL, and BLEU-4 scores by 0.52, 0.33, 0.33, and 0.37 in the medical diagnosis QAR test set. For the MQAE test set, the algorithm improves the BLEU-4, R2, RL scores by
<figure><img src="image_6.png" /><figcaption>Fig. 3. Curves for top-3 evidence-enhanced models' results (in %) on the MQAE-diag dataset by setting different numbers of evidence documents. "3, 5, 10, and 12" correspond to the evidence number identifiers</figcaption></figure>

<table><caption>Table 6. Experimental Results (in %) in Terms of ROUGE and BLEU Scores Obtained on the MQAE-diag Dataset by Using Different Evidence Weight Temperature</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>EMIN<sub>0.5</sub></td><td>32.36</td><td>12.35</td><td>21.42</td><td>8.79</td><td>83.41</td></tr><tr><td>EMIN<sub>0.7</sub></td><td><strong>32.90</strong></td><td>12.26</td><td>21.43</td><td>8.81</td><td>83.65</td></tr><tr><td>EMIN<sub>1.0</sub></td><td>31.63</td><td>11.41</td><td>20.45</td><td>8.53</td><td>83.14</td></tr><tr><td>EMIN<sub>tmp</sub></td><td>32.83</td><td><strong>12.53</strong></td><td><strong>21.61</strong></td><td><strong>9.23</strong></td><td>83.69</td></tr></tbody></table>

"0.5, 0.7, and 1.0" denote the value of fixed temperature method and "tmp" refer the temperature annealing decreasing by epoch approach. The best results are highlighted in bold.
<figure><img src="image_7.png" /><figcaption>Fig. 4. Curves for top-3 evidence-enhanced models' results (in %) on the MQAE dataset by setting different numbers of evidence documents. “3, 5, 10, and 12” correspond to the evidence number identifiers</figcaption></figure>

<table><caption>Table 7. Experimental Results (in %) in Terms of ROUGE and BLEU Scores Obtained on the MQAE Dataset by Using Different Evidence Weight Temperature</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>EMIN<sub>0.5</sub></td><td>27.51</td><td>12.16</td><td>20.74</td><td>8.76</td><td>83.12</td></tr><tr><td>EMIN<sub>0.7</sub></td><td>27.73</td><td>12.39</td><td>20.94</td><td>8.75</td><td>83.83</td></tr><tr><td>EMIN<sub>1.0</sub></td><td>27.13</td><td>11.96</td><td>20.75</td><td>8.56</td><td>83.10</td></tr><tr><td>EMIN<sub>tmp</sub></td><td><strong>28.05</strong></td><td><strong>12.53</strong></td><td><strong>21.61</strong></td><td><strong>9.23</strong></td><td>83.35</td></tr></tbody></table>

“0.5, 0.7, and 1.0” denote the value of fixed temperature method and “tmp” refer the temperature annealing decreasing by epoch approach. The best results are highlighted in bold.
<figure><img src="image_8.png" /><figcaption>Fig. 5. The KL divergence and evaluation scores of the EMIN's evidence weights varies with iterations on two MQAE dataset.</figcaption></figure>

0.03, 0.08, 0.07. Figure 5(a) and (b) illustrate variances of the KL divergence of evidence weights, ROUGE-2 and the BLEU-4 scores for the samples that converged in five iterations on two MQAE datasets. The KL divergence of evidence weights between the previous and current states is continuously decreasing while its variances become decreased. Stable KL divergence and small variance reveal that our EM algorithm converges in this inference stage, confirmed by an improvement of the quality of the generated explanations.

### 3.5.5 *Assessment of Generalization Capability.*

## 3.6 Case Study

Table 8 presents a case study on explanation generation in the context of a trained QA system making a wrong prediction. From the table, we can observe that the ground-truth explanation and EMIN explanation (based on the ground-truth answer) are all indicate “metformin increase insulin output”. Even though the trained QA systems make an incorrect prediction, i.e., “b/l oophorectomy and post op radiotherapy,” the EMIN explanation (based on the incorrect answer) still suggests the “oophorectomy on the involved side” is the correct drug for the type 2 diabetes patients. Tables 9 and 10 present a case studies on explanation generation in the context of a trained QA system making a correct prediction.

Based on the comparisons, the explanation provided by the ChatGPT is concrete and convincing, but the explanations are lengthy. It is hard to catch the key points of explanations, at this point, EMIN holds great promising by providing relevant and brief explanations.

## 3.7 Computational Efficiency

Computational efficiency of a trained model is important for practical applications. In this respect, we tested inference latency on a single NVIDIA GeForce RTX 3090. We conducted 100 runs and report the average latency across them. For the MHOP, MEAN, and EMIN methods, the latencies were 940 ms, 1,111 ms, and 1,163 ms, respectively, measured on the MQAE-diag dataset. We found that EMIN and two strong baselines have little difference in latency performance, but our model achieves better performance. The EMIN model introduces additional computations in two places during inference: one is to calculate the weighted evidence, and the other is to inject the weighted evidence into the decoder.
<table><caption>Table 8. As an Example from the MCQAE Dataset, Consider a Clinical Question with Four Options: “b/l oophorectomy and post op radiotherapy,” “oophorectomy on the involved side,” “only chemotherapy,” “hysterectomy with bilateral salpingopherectomy”</caption><tr><td>Question.</td><td>18-year-old female presents with an ovarian mass, her serum lactate dehydrogenase (LDH) and placental alkaline phosphate are found to be elevated. What is the treatment of choice in this condition?</td></tr><tr><td>Ground-Truth Explanation.</td><td>The given scenario suggests the diagnosis of Dysgerminoma Most common germ cell malignancy High risk: dysgenetic gonads, testicular feminisation syndrome Tumour markers: LDH, placental alkaline phosphatase (PLAP), (hcg may be raised in some but AFP is normal) In Dysgerminoma, conservative surgery (U/L Salphingo-Oophorectomy) recommended as most patient are Young girls Post operative Chemotherapy gives 90% success NOTE: bilateral salpingo-oophorectomy is not done here in order to conserve ovarian function and feility unless mentioned that the presentation is</td></tr><tr><td>EMIN Explanation (CA).</td><td>Oophorectomy on the involved side. In the given case, patient is having Ovarian mass, her serum LDH and placental alkaline phosphate are found to be elevated. The treatment of choice is oophorectomy on the involved side, so as to rule out of option option there is total resection of ovary.</td></tr><tr><td>EMIN Explanations (WA).</td><td>Oophorectomy and post op radiotherapy. In the question patient is presenting with ovarian mass, her serum LDH and placental alkaline phosphate level is found. The best treatment of choice is oophorectomy on the involved side</td></tr><tr><td>ChatGPT Explanations (CA).</td><td>The scenario you described raises concerns about a potential ovarian germ cell tumor, particularly a type known as dysgerminoma. Elevated serum markers such as LDH and PLAP are associated with this type of ovarian tumor. Dysgerminomas are malignant germ cell tumors that typically occur in young females. They are considered the ovarian counterpart of testicular seminomas in males. These tumors are highly sensitive to radiation therapy and chemotherapy. However, the primary treatment of choice is surgical removal of the affected ovary, a procedure known as oophorectomy on the involved side. The reasons for choosing oophorectomy include: Localized Disease Control: Oophorectomy allows for the removal of the affected ovary, which is crucial for controlling the disease locally. It helps to eliminate the tumor and prevent its spread within the affected ovary. Pathological Diagnosis: Surgical removal of the ovarian mass allows for a thorough pathological examination of the tissue. This is essential to confirm the diagnosis, determine the tumor type, and assess the extent of the disease. Preservation of Function: In cases where the tumor is confined to one ovary, removing only the affected ovary (oophorectomy) may spare the contralateral ovary. This is important for preserving hormonal function and fertility, especially in young patients. Adjuvant Therapy Planning: The pathological findings from the surgical specimen guide further treatment decisions. If the tumor is more advanced or shows certain characteristics, adjuvant therapies such as chemotherapy or radiation therapy may be recommended. Improved Survival Rates: Early detection and surgical intervention contribute to better overall outcomes and survival rates for patients with ovarian germ cell tumors, including dysgerminomas. It's important to note that the specific treatment plan may vary based on individual patient characteristics, the stage of the disease, and the overall health of the patient. Multidisciplinary collaboration involving gynecologic oncologists, pathologists, and other specialists is typically involved in managing such cases.</td></tr></table>

The correct answer for this question is "oophorectomy on the involved side," but the trained QA system predicts "b/l oophorectomy and post op radiotherapy" as the answer. In this context, "CA" refers to the explanation based on the ground-truth answer, and "WA" represents the explanation based on the wrong answer. The italic words in EMIN explanation explanations (WA) indicated that the "oophorectomy on the involved side" is the best treatment of choice for this case.
<table><caption>Table 9. As an Example from the MCQAE Dataset, the Trained QA System Correctly Predicts “Sta SSRI Treatment Based on Side Effect Profile” as the Answer</caption><tr><td>Question. A female patient presented with depressed mood, loss of appetite, and no interest in surroundings. There is associated insomnia. The onset of depression was preceded by a history of business loss and soon after it she developed the following symptoms for the past 1 year. True statement regarding management of this patient is?</td></tr><tr><td>Ground-Truth Explanation. Selective Serotonin Reuptake Inhibitor (SSRI) are the drug of choice now. As TCAs have anticholinergic, cardiovascular and neurological side effects. They have low safety margin and have a lag time of 2-4 weeks before anti depressant action manifest. Hence SSRI are used based on side effect profile.</td></tr><tr><td>EMIN Explanation (CA). Sta SSRI treatment based on side effect profile. Depression associated with reactive disorders usually does not call for drug therapy and can be managed by psychotherapy and the passage of time. In this case, the onset of depression was preceded by an episode of loss and soon after it she developed the symptoms suggestive of depression. Drugs used in depression are usually drug of choice for the following reasons: 1. Subcoical symptoms; 2. Depressive mood.</td></tr><tr><td>ChatGPT Explanations (CA). The information provided describes a patient with symptoms consistent with Major Depressive Disorder (MDD). The patient's depressed mood, loss of appetite, anhedonia (lack of interest in surroundings), and associated insomnia are typical features of depression. Additionally, the history of a business loss and the presence of symptoms for the past year suggest a chronic or recurrent nature of the depressive episode. The recommended management for this patient, as mentioned in the answer, is to start SSRI treatment based on the side effect profile. Here's an explanation of why this choice is appropriate: Effectiveness of SSRIs: SSRIs are commonly used as first-line treatment for MDD. They are considered effective in managing depressive symptoms and have a relatively favorable side effect profile compared to some other classes of antidepressant medications. Side effect profile: SSRIs are generally well-tolerated, with common side effects being mild and transient. These may include nausea, headache, and insomnia. Importantly, SSRIs have a lower risk of certain side effects, such as anticholinergic effects (dry mouth, blurred vision, constipation) and cardiovascular effects, which can be seen with other classes of antidepressants. Safety: SSRIs have a favorable safety profile, and the risk of overdose is relatively lower compared to some other antidepressant classes. This is an important consideration in the management of depressive disorders, as safety is a key concern. Evidence-based practice: The use of SSRIs in the treatment of depression is supported by a robust body of evidence from clinical trials. They have been shown to be effective in improving mood and overall functioning in individuals with MDD. It's important to note that the choice of antidepressant can vary based on individual patient factors, and the decision should be made in consultation with a healthcare professional. Monitoring for potential side effects and adjusting the treatment plan as needed is a standard practice in the management of depression. Additionally, psychotherapy and other supportive interventions may be considered as part of a comprehensive treatment approach.</td></tr></table>

In this context, “CA” refers to the explanation based on the ground-truth answer.

## 4 Related Work

Recently, significant advancements in medical question-answering systems have been driven by the introduction of novel models and datasets [1, 15, 20]. However, within the domain of medical inquiry, the role of explanations has become increasingly critical. Approaches such as segmenting highlighted textual spans [5] and knowledge-base paths [25] have limitations when they come to providing explanation which requires substantial mental effort, particularly because out-of-context spans may be difficult to interpret. An alternative approach is to
<table><caption>Table 10. As an Example from the MCQAE Dataset, the Trained QA System Correctly Predicts “Epidural Block” as the Answer</caption><tr><td>Question.</td><td>A 35-year-old pregnant female at 40 weeks gestational age presents with pain and regular uterine contractions every 4–5 min. ON arrival, the patient is in a lot of pain and requesting relief immediately. Her cervix is 5 cm dilated. What is the most appropriate method of pain control for this patient?</td></tr><tr><td>Ground-Truth Explanation.</td><td>When complete relief of pain is needed throughout labour, epidural analgesia is safest and simplest. It provides sensory as well as various degrees of motor blockade over the region of body.</td></tr><tr><td>EMIN Explanation (CA).</td><td>Epidural block. When complete relief of pain is needed throughout labor, epidural analgesia is safest and simplest.</td></tr><tr><td>ChatGPT Explanations (CA).</td><td>The most appropriate method of pain control for the 35-year-old pregnant female at 40 weeks gestational age with 5 cm cervical dilation and regular uterine contractions every 4–5 minutes is an epidural block. Epidural analgesia is a commonly used method for pain relief during labor and delivery. Here’s why an epidural block is often chosen in this scenario: Effective Pain Relief: Epidural anesthesia provides excellent pain relief by blocking the transmission of pain signals from the nerves in the spine to the brain. This can significantly alleviate the discomfort associated with labor contractions. Regional Anesthesia: Epidurals offer regional anesthesia, meaning they target a specific region of the body—in this case, the lower half of the body where the uterus and cervix are located. This allows the woman to remain awake and actively participate in the birthing process while being relieved of pain. Controlled Administration: Epidurals allow for the administration of medication in a controlled and continuous manner. This helps in maintaining a stable level of pain relief throughout the labor process. Flexibility in Dosage: The dosage of the epidural medication can be adjusted according to the individual’s pain tolerance and the progression of labor. This flexibility allows for tailored pain management. No Impact on Mental Alertness: Unlike systemic medications that may affect the entire body, epidurals primarily act locally and do not cause sedation. This means the mother can stay mentally alert and be an active participant during the birthing process. Reduced Stress: Effective pain control can reduce the overall stress response associated with labor, potentially improving maternal and fetal outcomes. It’s important to note that while epidural analgesia is a common and effective method of pain control, the decision to use it should be made in consultation with the healthcare provider, taking into consideration the individual’s medical history and any potential contraindications.</td></tr></table>

In this context, “CA” refers to the explanation based on the ground-truth answer.

generate natural language explanations, as we proposed and evaluated in article. Explanations serve as a linchpin for promoting the reliability of answers and facilitating fact-checking [7, 9], so that users can trust medical QA systems by addressing their concerns. Nevertheless, the manner in which explanations are presented warrants careful consideration. Approaches such as segmenting highlighted textual spans, as exemplified in [4], have limitations when it comes to providing insightful explanations through feature importance techniques [5] or when leveraging a knowledge base [25]. This approach has the potential to yield explanations that are more comprehensible and accessible to readers [4].

The EM approach, known for its utility in learning latent variable models, has found applications in various downstream NLP tasks, including document classification and question-answering. For example, Singh et al. [17] developed an EM-based framework for question-answering by iteratively optimizing the parameters of a reader and a retriever. The framework encodes information from multiple documents to improve question-answering performance. Nigam et al. [14] employed the EM method to augment limited labeled data with information from unlabeled data, improving model performance. LLMs have achieved remarkable performance on medical question-answering datasets, emphasizing the significance of accurate predictions. However, user acceptance,
particularly among clinicians, is contingent upon the model's ability to provide transparent explanations of its decision-making process. To address this need, **Chain-of-Thought (CoT)**-based techniques [21, 26] are proposed. Due to CoT-based approaches require complex computations, we proposed a light-weight QAE model designed to generate explanations for a medical QA system. In the medical domain, applications deployed in hospitals must prioritize data privacy, necessitating local server deployment. However, the sheer size of LLMs precludes their local deployment in such contexts. Additionally, our model introduces the concept of evidence weights, computed using EMIN, offering insights into the relative contributions of different pieces of evidence to the explanation in a transparent manner.

## 5 Conclusions

We propose an approach to collectively reason over all textual evidence while harnessing the potent attention mechanisms of a transformer architecture. Our method employs an EM algorithm to deduce the weights of evidence paragraphs as latent variables, enhancing the model parameters for explanation generation. This gives rise to the EMIN framework, which notably outperforms existing models, such as those utilizing multi-hop reasoning, in terms of generating explanations. The explicit evidence weights we derive unveil which pieces of evidence bolster the generated explanations. These weights complement the attention weights assigned to tokens within the evidence paragraphs, all in a scalable manner. This ensures that users gain comprehensive insights into the significance of the evidence within the generated explanation.

## References

[1] Asma Ben Abacha and Pierre Zweigenbaum. 2015. MEANS: A medical question-answering system combining NLP techniques and semantic web technologies. *Information Processing & Management* 51, 5 (2015), 570-594.

[2] Tian Bai and Slobodan Vucetic. 2019. Improving medical code prediction from clinical text via incorporating online knowledge sources (WWW '19). *ACM*, 72-82.

[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. *Advances in Neural Information Processing Systems* 33 (2020), 1877-1901.

[4] Wei-Lin Chen, An-Zi Yen, Hen-Hsen Huang, and Hsin-Hsi Chen. 2022. Learning to generate explanation from E-hospital services for medical suggestion. In *Proceedings of the 29th COLING*, 2946-2951.

[5] Shuyuan Hu, Fei Teng, Lufei Huang, Jun Yan, and Haibo Zhang. 2021. An explainable CNN approach for medical codes prediction from clinical text. *BMC Medical Informatics and Decision Making* 21, 9 (2021), 1-12.

[6] Xinyu Hua and Lu Wang. 2018. Neural argument generation augmented with externally retrieved evidence. In *Proceedings of the 56th ACL*. Association for Computational Linguistics, Melbourne, Australia.

[7] Qiao Jin, Zheng Yuan, Guangzhi Xiong, Qianlan Yu, Huaiyuan Ying, Chuanqi Tan, Mosha Chen, Songfang Huang, Xiaozhong Liu, and Sheng Yu. 2022. Biomedical question answering: A survey of approaches and challenges. *ACM Computing Surveys (CSUR)* 55, 2 (2022), 1-36.

[8] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In *Proceedings of the 2020 EMNLP*. ACL, Online.

[9] Neema Kotonya and Francesca Toni. 2020. Explainable automated fact-checking for public health claims. In *Proceedings of the 2020 EMNLP*. ACL, Online, 7740-7754.

[10] Minsuk Lee, James Cimino, Hai Ran Zhu, Carl Sable, Vijay Shanker, John Ely, and Hong Yu. 2006. Beyond information retrieval-medical question answering. In *AMIA Annual Symposium Proceedings*, Vol. 2006. American Medical Informatics Association, 469.

[11] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In *Proceedings of the 58th ACL*. ACL, Online, 7871-7880.

[12] Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In *Proceedings of the ICLR*.

[13] Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In *Proceedings of the 2016 EMNLP*. ACL, Austin, Texas.

[14] Bhawna Nigam, Poorvi Ahirwal, Sonal Salve, and Swati Vamney. 2011. Document classification using expectation maximization with semi supervised learning. arXiv:1112.2028. Retrieved from https://arxiv.org/abs/1112.2028

[15] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In *Conference on Health, Inference, and Learning*. PMLR, 248-260.
[16] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. *Journal of Machine Learning Research* 21, 140 (2020), 1–67.

[17] Devendra Singh, Siva Reddy, Will Hamilton, Chris Dyer, and Dani Yogatama. 2021. End-to-end training of multi-document reader and retriever for open-domain question answering. *Advances in Neural Information Processing Systems* 34 (2021), 25968–25981.

[18] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. *Nature* 620, 7972 (2023), 172–180.

[19] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. *NeuralIPS* 30 (2017).

[20] Hadi Veisi and Hamed Fakour Shandi. 2020. A persian medical question answering system. *International Journal on Artificial Intelligence Tools* 29, 06 (2020), 2050019.

[21] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. *NeuralIPS* 35 (2022), 24824–24837.

[22] Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. 2024. Benchmarking retrieval-augmented generation for medicine. In *Findings of the Association for Computational Linguistics (ACL '24)*. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.), Association for Computational Linguistics, Bangkok, Thailand, 6233–6251. DOI: https://doi.org/10.18653/v1/2024.findings-acl.372

[23] Yue Yang, Mona Gandhi, Yufei Wang, Yifan Wu, Michael S. Yao, Chris Callison-Burch, James C. Gee, and Mark Yatskar. 2024. A textbook remedy for domain shifts: Knowledge priors for medical image analysis. arXiv:2405.14839. Retrieved from https://arxiv.org/abs/2405.14839

[24] Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining language models with document links. In *Proceedings of the 60th ACL*. ACL, Dublin, Ireland.

[25] Yingying Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2019. Multi-modal knowledge-aware hierarchical attention network for explainable medical question answering. *Proceedings of the MM '19*. ACM, New York, NY, USA.

[26] Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. 2023. Least-to-most prompting enables complex reasoning in large language models. In *Proceedings of the 11th ICLR*.
# Appendix

## A.1 How the Evidence Weights Affect Explanation Generation?

The evidence weights control the importance of retrieved evidence. To analyze their influence on the generated explanations, we conduct a case study shown in Table A1. In this study, we progressively introduce more critical evidence and observe the corresponding changes in the generated explanations. We find that as more relevant evidence is incorporated, the explanations becomes progressively more accurate.

<table><caption>Table A1. A Case Study Where We Progressively Introduced More Critical Evidence to Observe the Corresponding Changes in the Generated Explanations</caption><tr><td>Inject Top 9–10 Evidence. The use of an inferior vena cava (IVC) filter is indicated in patients with venous thrombosis who are hemodynamically stable. The other alternative to using an IVC filter is inappropriate because of the possible adverse effects on the cardiovascular system. (Comments: The description contains some inaccuracies. IVC filters are generally not the preferred treatment option for hemodynamically stable patients with venous thrombosis.)</td></tr><tr><td>Inject Top 7–10 Evidence. The use of an IVC filter is indicated in patients with venous thrombosis who are hemodynamically stable. (Comments: Not entirely accurate. Routine use of IVC filters is generally not recommended for hemodynamically stable patients with VTE.)</td></tr><tr><td>Inject Top 5–10 Evidence. The use of an IVC filter has been associated with an increased risk of venous thrombosis and pulmonary embolus with mild hypoxemia. The VC filter should be used as well as other anti-inflammatory agents such as anticoagulant medications such as coumadin.</td></tr><tr><td>Inject Top 3–10 Evidence. The use of an IVC filter has been associated with many cases of venous thrombosis (e.g., pulmonary embolus with severe hypoxemia and edema). The VVC filter is indicated in patients who have a desire to stop bleeding but do not want to risk the consequences of further surgery. (Comments: There is some confusion regarding the description. IVC filters are indicated for patients who cannot take anticoagulants or in situations where anticoagulation is ineffective or inappropriate.)</td></tr><tr><td>Inject Top 1–10 Evidence. The use of an IVC filter is indicated in patients with venous thrombosis or who have a pulmonary embolus that is hemodynamically stable. The most common adverse effect of the VC filter is vasoconstriction, but tamponade and antiplatelet therapy can also be beneficial. The other choices are not associated with pulmonary emboli.</td></tr><tr><td>Ground-Truth Explanation. In general, failure (or inability) to continue anticoagulants is an indication to insert an IVC filter to minimize the possibility of serious and possibly fatal pulmonary embolus.</td></tr></table>

## A.2 In-Depth Analysis of Failure Cases or Limitations in Explanation Coherence

We discuss some of the key patterns in failure cases where the generated explanations tend to lack coherence.

In our analysis of the coherence and effectiveness of model-generated explanations, we find that the model's ability to generate coherent or useful explanations is significantly reduced when the medical information contained in the question is not detailed enough. In this case, the model tends to retrieve general evidence related to the medical context expressed in the question and often fails to focus on key, specific medical knowledge. For example, for the following question: *A 60-year-old male diabetic and hypertensive patient was found unconscious in the morning. On examination, pulse rate is 120/min, BP=160/100 mm Hg. What is the next step to be done for management?*, the evidence retrieved by the model focuses on general treatments for diabetes and hypertension, but lacks further guidance on the treatment of this specific case. This results in the lack of specificity and coherence
<table><caption>Table A2. Experimental Results Showing ROUGE (R) and BLEU Scores (%) on the CasiMedicos Dataset to Evaluate the EMIN's Transfer Abilities</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>EMIN<sub>(trained on CasiMedicos training set)</sub></td><td>25.30</td><td>7.07</td><td>17.30</td><td>3.87</td><td>86.28</td></tr><tr><td>EMIN<sub>(trained on the MQAE dataset and then do zero-shot)</sub></td><td>24.56</td><td>6.07</td><td>16.41</td><td>3.15</td><td>84.17</td></tr></tbody></table>

<table><caption>Table A3. Experimental Results in Terms of ROUGE (R) and BLEU Scores (%) on the MQAE-diag Dataset</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERTScore</th></tr></thead><tbody><tr><td>EMIN</td><td>32.83</td><td>12.53</td><td>21.61</td><td>9.23</td><td>83.69</td></tr><tr><td>EMIN<sub>(SNOMED CT)</sub></td><td>31.73</td><td>11.59</td><td>22.93</td><td>10.13</td><td>84.18</td></tr></tbody></table>

<table><caption>Table A4. Experimental Results in Terms of ROUGE (R) and BLEU Scores (%) on the MQAE Dataset</caption><thead><tr><th>Models</th><th>R1</th><th>R2</th><th>RL</th><th>BLEU-4</th><th>BERScore</th></tr></thead><tbody><tr><td>EMIN</td><td>28.05</td><td>12.53</td><td>21.61</td><td>9.23</td><td>83.55</td></tr><tr><td>EMIN<sub>(SNOMED CT)</sub></td><td>28.11</td><td>12.68</td><td>20.33</td><td>9.12</td><td>83.62</td></tr></tbody></table>

in the generated explanations. The shows that the coherence and informativeness of the generated explanation depend on the level of detail of the medical information in the question, revealing the limitations of our method in dealing with low-detail problems.

## A.3 Assessment of Generalization and Transfer Capabilities on Unseen Data

For the transfer capacities on unseen, we conducted an experiment to evaluate the model's generalization capability. Using a zero-shot setting on an unseen dataset, the model maintained robust performance, indicating effective transfer learning. This result (see Table 8) suggests that the model's sophisticated design and dataset specificity did not lead to overfitting but instead enhanced its adaptability to new contexts, demonstrating resilience and generalization beyond the initial training data.

## A.4 Effectiveness of Integrating Specialized Medical Ontologies

The retriever extracts relevant knowledge from the external medical textbooks. In this experiment we expand these with highly technical medical terms and their definitions, and use this knowledge to support the generation of medical explanations. The goal is to improve the retrieval of evidence by adding knowledge of medical terminology. More specifically, we add the SNOMED CT ontology as additional evidence into the evidence document collection. The results and effect on explanation generation are shown in Tables A3 and A4. Based on experimental results, the SNOMED CT ontology does not further enhance the model performance. We assume that the concepts in SNOMED CT are limited in their capability of generating complex, context-rich medical explanations. The concepts in SNOMED CT are often simplified, structured codes and terms, which may not be sufficient to support natural explanation generation.
### A.5 Real-World Applicability of the Proposed Model

We have considered integrating the proposed model with existing electronic health record systems. Due to the privacy considerations and in order to measure real-time performance, we have deployed the model on a local server as an interactive system. We have designed runs under docker-compose orchestration, basically running under the Nginx proxy to avoid dealing with multiple open ports, which will bring security risks. Therefore, in order to avoid any problems, we have designed a REST API-a WSGI server. Finally, we have packaged our model into a REST API and then turned the model into an independent service through a RESTful API, enabling it to communicate seamlessly with the front end or other modules of the EHR system.

Received 8 March 2024; revised 1 November 2024; accepted 9 December 2024