M-QALM: A Benchmark to Assess Clinical Reading Comprehension and Knowledge Recall in Large Language Models via Question Answering (2024)

Abstract: There is vivid research on adapting Large Language Models (LLMs) to perform a variety of tasks in high-stakes domains such as healthcare. Despite their popularity, there is a lack of understanding of the extent and contributing factors that allow LLMs to recall relevant knowledge and combine it with presented information in the clinical and biomedical domain—a fundamental pre-requisite for success on downstream tasks. Addressing this gap, we use Multiple Choice and Abstractive Question Answering to conduct a large-scale empirical study on 22 datasets in three generalist and three specialist biomedical sub-domains. Our multifaceted analysis of the performance of 15 LLMs, further broken down by sub-domain, source of knowledge and model architecture, uncovers success factors such as instruction tuning that lead to improved recall and comprehension. We further show that while recently proposed domain-adapted models may lack adequate knowledge, directly fine-tuning on our collected medical knowledge datasets shows encouraging results, even generalising to unseen specialist sub-domains. We complement the quantitative results with a skilloriented manual error analysis, which reveals a significant gap between the models’ capabilities to simply recall necessary knowledge and to integrate it with the presented context. To foster research and collaboration in this field we share M-QALM—our resources, standardised methodology, and evaluation results—with the research community to facilitate further advancements in clinical knowledge representation learning within language models.

heading: 1 Introduction
text: Recent success in the application of proprietary large language models in the knowledge-intensive medical domain (Singhal et al., 2023a,b) has sparked vivid research interest in applying smaller,
*Work done during an internship at AICS
more readily available open-source LLMs to various settings in the clinical and biomedical domains. Examples of tasks include summarization of clinical text (Veen et al., 2023), automatic note generation for physicians (Ben Abacha et al., 2023b), and condensation of doctor-patient dialogues (Ben Abacha et al., 2023a; Toma et al., 2023). More broadly, open-source LLMs have been adapted to the domain to serve as foundational clinical models (Han et al., 2023; Wu et al., 2023; Toma et al., 2023; Bolton et al., 2022; Li et al., 2023).
The success of such adaptation is typically established by measuring the performance on downstream tasks, by means of token overlap or semantic similarity-based metrics (Lin, 2004; Zhang et al., 2020). To address their inherent weaknesses (Schlegel et al., 2022; Gatt and Krahmer, 2018), research attempts to incorporate specific dimensions, such as factuality or faithfulness (Umapathi et al., 2023). Two important problems remain, however. Firstly, Natural Language Generation (NLG) evaluation metrics are merely approximations of the phenomena they aim to measure, and their effectiveness is typically established by the degree of correlation to human judgements of the evaluated criteria
4002
(Huang et al., 2021). Secondly, an (offline) evaluation setup is functionally grounded and serves as a proxy of a real-world application scenario, but the transferability of insights from functionallygrounded to application-grounded evaluation is barely discussed (Doshi-Velez and Kim, 2017). Taken together, these problems might taint the credibility of conclusions about the successful adaptation of LLMs drawn from such experiments.
Given such difficulties, we approach the problem of evaluating LLM adaptation from a complementary angle. Specifically, we ask: Do LLMs possess the necessary pre-requisites to succeed in the clinical and medical domains? Without an established theory of how knowledge is acquired and organised in LLMs, the present work is guided by the established theories of knowledge acquisition in humans (Adams, 2015). Typical NLG tasks, such as summarisation, are higher-level cognitives that require the understanding of learned knowledge and its application in new contexts (Bloom, 1956). They build on the most fundamental capability of reading comprehension (Kintsch, 1988): the construction of a text-base and its integration with previously acquired background knowledge. In NLP research, this process is evaluated by openbook Question Answering (QA), the task of either generating (abstractive, AQA) or selecting among presented options (multiple-choice, MCQA) the correct answer for a question, where potentially not all necessary information is included in the question or
the presented context. MCQA evaluation does not suffer from the issues pertaining to NLG metrics, as performance is established by exact match. Thus, conclusions obtained from such evaluations tend to be more robust, if the quality of the benchmark is sufficient.
Therefore, in this paper we focus on the task of QA, to evaluate knowledge recall and comprehension pre-requisites of LLMs for successful adaptation to the medical domain. We present an exhaustive, publicly available QA benchmark called M-QALM including 16 MCQA datasets. To enable future research on NLG-based QA, we complement M-QALM by 6 high-quality AQA datasets, where the ground-truth answer is an unconstrained string. With such a standardized benchmark, we conduct an extensive evaluation of the capabilities of openly available general-purpose and medical LLMs, both “out-of-the-box” and after fine-tuning on M-QALM. Our findings provide insights into the strengths and weaknesses of different LLMs across a range of datasets, question categories and QA tasks. Overall, we find their performance lacking, both compared to humans and to proprietary LLMs. Further analysis reveals promising tendencies of domain-specific pre-training and fine-tuning to bridge this gap and to generalise to new QA datasets.

heading: 2 Related Work
text: Large open-domain QA benchmarks The availability of QA datasets from multiple domains and
sources has enabled the curation of large and diverse QA benchmarks (Dua et al., 2019; Fisch et al., 2019; Talmor and Berant, 2019). Such resource collections enable researchers to perform large-scale empirical studies to understand how well language models can generalise to new questions from new domains or sources, or how fine-tuning can impact this performance. While multiple studies exist in the general domain, to the best of our knowledge, no such large-scale study has been carried out for QA in the clinical domain. In this paper we aim to address this gap.
Evaluation in the clinical domain Datasets that evaluate the lowest-level cognitive task of knowledge recall and reading comprehension in the medical domain have been proposed before (Jin et al., 2021; Vilares and Gómez-Rodríguez, 2019; Pal et al., 2022). They feature questions commonly found in examinations like the US Medical Licensing Exam (USMLE). M-QALM unifies the existing literature by incorporating licensing exam questions from diverse regions, such as India and Spain. We go beyond the scope of the general medical domain, covering specialist topics such as ophthalmology and Alzheimer’s disease.
Beyond factual recall and comprehension, Fries et al. (2022) collected a unified bio-medical benchmark, featuring NLP primitives such as sentence(-pair) classification or entity recognition and linking. Aiming at higher, more task-specific cognitives, Singhal et al. (2023a) introduced MultiMedQA, including HealthSearchQA, which requires models to generate high-quality free-form answers. Similarly, He et al. (2023) introduced a multi-domain benchmark for evaluating generation and classification capabilities on a diverse set of in-hospital downstream tasks. Other researchers looked to evaluate the quality and factuality of generation (Umapathi et al., 2023) and synthesised general-purpose medical instructions (Fleming et al., 2023). Our work is complementary, because we evaluate knowledge recall and comprehension as a pre-requisite of higher-level cognitive tasks, such as understanding and application—the focus of previously discussed works.

heading: 3 M-QALM Datasets
text: The primary goal of M-QALM is to develop a comprehensive, open-source repository of medical QA datasets to assess the recall of medical knowledge in LLMs. To obtain such a collection, we perform
an exhaustive literature and resource search using the terms “clinical OR medical”, “Question Answering OR QA” and include a dataset or resource if it satisfies the following criteria: (i) The language is English, as medical documents are usually written in English, even in non-English-speaking countries; (ii) The questions and answers are on general, specialist, or consumer-facing medical topics; (iii) The resource is openly available without restrictive licensing or data agreements; (iv) The resource evaluates the task of MCQA or AQA; (v) The ground truth is collected or reviewed by domain experts.
The result is M-QALM—a comprehensive collection of 22 datasets designed to thoroughly evaluate the clinical knowledge of LLMs. Table 1 gives an overview of the collected MCQA and AQA datasets, including task formulation, size and domain. Refer to the Appendix for further details on each dataset.
Knowledge source categorization The MCQA datasets within the M-QALM benchmark cover a diverse range of medical domains. To be able to perform fine-grained analysis of both the topics covered in these datasets as well as model performance, we categorise the MCQA datasets into eleven highlevel categories, representing different facets of medical knowledge. To do so, we leverage available meta-data from the source datasets MEDMCQA, HEADQA, MMLU and BIOASQ-MCQ. We categorize the PROCESSBANK, PUBMEDQA and BIOMRC datasets into a distinct twelfth Within Context category, as the relevant knowledge is presented in the context. USMLE and QA4MRE lack the necessary meta-data, thus we train a BioBERT-based classifier (Lee et al., 2019) to assign questions into one of the eleven elicited categories using the labels from the other datasets. The classifier achieves 71.56% (micro-)averaged F1 score on a held-out test set, which we deem sufficient.
Table 4 shows that nearly half of all questions (47%) fall into the Basic and Life Sciences and General Medicine category. Diagnostic Sciences, Women’s and Children’s Health and Pharmacology and Anesthesia account for a further 30% of questions.

heading: 4 Empirical Evaluation
text: We investigate how well existing, open-source LLMs are able to recall clinical knowledge and integrate it into a given context in order to succeed on
our benchmark. Specifically, we focus on performance in the zero-shot setting, and after fine-tuning on M-QALM training portions. In the Zero-shot setting: • RQ1. How well do open-source LLMs recall nec-
essary clinical knowledge when they are tested on M-QALM? • RQ2. Does open-domain instruction fine-tuning of LLMs improve their ability to do so? • RQ3. Does domain-specific fine-tuning improve performance on M-QALM? In the Fine-tuned setting: • RQ4. Does finetuning on M-QALM improve per-
formance on unseen data from datasets seen during training? • RQ5. Does fine-tuning improve performance on unseen M-QALM datasets?

heading: 4.1 Study Setup
text: To seek evidence for RQs 1-3 empirically, we evaluate several LLMs and their instruction-tuned versions on the test splits of M-QALM in zero-shot1 manner. To answer RQ4 and RQ5, we fine-tune LLMs on the training portion of M-QALM and evaluate on test splits of datasets both seen and unseen during training. We complement our evaluation with additional automated and manual error analyses to identify causes for model successes and failures.
Models: To assess the zero-shot capabilities of models (RQ1 and RQ2), we include a diverse array of open-source decoder-only models with parameter scales ranging from 3B-13B. We use models from MPT and MPT-Instruct (7B) (MosaicML, 2023), Falcon and Falcon-Instruct (7B) (Almazrouei et al., 2023), LLaMA 1 (7B and 13B) (Touvron et al., 2023a), LLaMA 2 and LLaMA 2-chat (7B and 13B) (Touvron et al., 2023b). In addition to these models, we also use two instruction finetuned encoder-decoder models: Flan-T5 (3B and 11B) (Wei et al., 2021). Models with Instruct or Chat appended to their names are instruction finetuned (Ouyang et al., 2022) versions of their base models. The details of the models are given in Table 10. To address RQ3, we evaluate ChatDoctor (7B) (Li et al., 2023), MedAlpaca (7B) (Han et al., 2023) and PMC-LLama (Wu et al., 2023). To ad-
1For MCQA evaluation in the zero-shot setting (where models are not explicitly fine-tuned for MCQA tasks), we use a 1-shot prompt—giving an example to the model, and find that it adheres better to the MCQA format and the standard 5-shot prompt for MMLU datasets.
dress RQ4, we fine-tune models using the training set of the M-QALM datasets. When official validation splits are unavailable, we employ a random split of up to around 20% of the data for validation purposes. If no training datasets are available, we do not use this dataset for fine-tuning and only consider the test split of the respective datasets to answer RQ5. For evaluating AQA, we use a subsampled version of the test sets of MASHQA (500 questions) and MEDQUAD (200 questions by sampling 100 questions from the two holdout websites), while we use the other datasets as they are. For MCQA, similar to Singhal et al. (2023a), we evaluate all models on the validation set of MEDMCQA since the answers for the test set are not released publicly.
Finetuning and hyperparameters: Since the number of parameters for most of our models is in the billions, we follow a more accepted practice of using parameter-efficient fine-tuning, specifically QLora and 4-bit quantization (Dettmers et al., 2023). We utilize 8-bit quantization for evaluating Flan-T5 (11B), LLaMA 1 (13B), LLaMA 2 (13B) and LLaMA 2-Chat (13B) (Dettmers et al., 2022). We use A100-40G GPUs for all our experiments. The other hyper-parameters used to train our models are reported in the Appendix (Table 11).
Evaluation measures: We use Accuracy to measure the performance of the model on MCQA datasets; for AQA datasets, we use ROUGE-L (Lin, 2004), BERTScore (Zhang et al., 2020) (based on deberta-xlarge-mnli) and METEOR (Banerjee and Lavie, 2005), which is found to correlate better with human judgments than other metrics on AQA (Chen et al., 2019).

heading: 5 Results and Analysis
text: In this section, we report and analyse the findings of our empirical study.

heading: 5.1 Zero-shot Evaluation Results
text: Table 2 shows the dataset-averaged scores of the zero-shot evaluation of language models as evidence towards RQs 1-3. Note that in this way, each dataset contributes equally to the average, regardless of its size. Micro-averaged MCQA accuracy scores are reported in Table 4. However, these are biased towards datasets with more examples (i.e., MEDMCQA). While the results between microand by-dataset-averaged metrics might differ in detail (consult Appendix D for a break-down), the
mean absolute difference between the metrics for all models is 4.2, which suggests that reported trends do not depend on the averaging method.
Table 2 highlights that LLMs exhibit strong zeroshot capability on MCQA and AQA datasets, corroborating the findings of Singhal et al. (2023a). Considering LLMs of the same size (e.g., 7B), LLaMA 2 performs best, possibly due to larger diversity in pre-training data—LLaMA 2 is trained on the most tokens. Another difference is the mixture of datasets used for pre-training, which is not revealed in some cases (c.f. Table 10 in Appendix).
Unsurprisingly, across all models of the same architecture, scale predicts model performance, even without domain-specific adaptation of LLMs on the medical domain. For example, LLaMA 2 (13B) performs better on MCQA (+4.2 Accuracy improvement) compared to the 7B version. Figure 5 in the Appendix shows the relationship between the number of parameters and performance.
To address RQ2, we investigate whether improvements from instruction fine-tuning also apply to the clinical domain of M-QALM. The results are reported in the middle part of Table 2.
Surprisingly, instruction fine-tuned models perform better than their corresponding Base versions, despite the fact that the instruction set used for fine-tuning contains only tasks in the general
domain—see Table 10 and compare *-Instruct/Chat (middle) with their base versions (top). Among them, Flan-T5 models exhibit the best zero-shot performance on MCQA, outperforming comparable decoder-only models. Seemingly, instruction finetuning enables models to obtain representations of question and context which are beneficial for fact recall.
We note that bigger models are not always better—the choice of model architecture and dataset for instruction fine-tuning can have a bigger impact on performance than model size alone. For example the encoder-decoder Flan-T5 (3B) model outperforms LLaMA 2-chat (13B) on the MCQA task, despite being four times smaller.
The performance of domain-adapted models is reported in Table 2 (bottom), as evidence for RQ3. For MCQA, both MedAlpaca and ChatDoctor indeed exhibit improvements in Accuracy over their respective 7B and 13B LLaMA 1 base versions; however they fail to reach the strong zero-shot performance of Flan-T5 (11B).
In contrast, PMC-LLama performs well due to continued pre-training on biomedical corpora before instruction tuning on biomedical and clinical datasets. The latter results in exceptionally high scores on the MEDINFO AQA dataset (See Table 20 in Appendix). This dataset, along with LIVEQA, was used as part of the instruction tuning process, leading to evaluation on these dataset not being “zero-shot”2. Scores on LIVEQA, however, are not inflated, compared to LLaMA 2(-chat) (13B). This is possibly because we use a filtered version of LIVEQA which contains only challenging answers with sufficiently good expert quality rating. PMC-LLama demonstrates significant improvements over other open-source LLMs on MCQA datasets such as USMLE, MEDMCQA and MMLU.
In summary, we conclude that while available LLMs adapted to the medical domain successfully improve performance of the adapted models, they appear to have no improved domain knowledge compared to other available open-domain models. Evaluating these adaptation techniques on stronger base models is an exciting avenue for future research.
Importantly, none of the evaluated open-source LLMs outperform humans: While the passing score
2https://huggingface.co/datasets/axiong/pmc_ llama_instructions
for USMLE is 60% 3, we observe the best zeroshot scores for USMLE are 43% for LLaMA 2, and 54% for the domain-adapted PMC-LLama, both below the passing score. Meanwhile, GPT-4 (OpenAI, 2023) with a customized prompting strategy labeled MedPrompt (Nori et al., 2023) achieves 90.2%, while Med-PALM 2 (Singhal et al., 2023b) achieves scores of 86.5% on USMLE. Similarly, for the PubmedQA dataset, human performance is 78% (Jin et al., 2019), compared to 72.4% of Flan-T5. To summarize: While available LLMs exhibit performance significantly higher than random chance “out-of-the-box”, there is still a substantial gap compared to humans and proprietary LLMs (Singhal et al., 2023a,b) (see Appendix B).

heading: 5.2 Impact of Fine-tuning
text: Given the scale of M-QALM, we are able to finetune models on parts of the data, to address RQ4 and RQ5. We fine-tune four models on MCQA and AQA separately, given the different nature of these datasets, but joint fine-tuning on both MCQA and AQA did not yield significantly different results.
We fine-tune the models only on the MCQA subset of datasets first (cf. Table 3). We find that the fine-tuned models perform better compared to their non-fine-tuned counterparts. Decoder-only models like MPT (7B) benefit more than others (+25.6 Accuracy improvement). Fine-tuning models on the data seems to close the gaps introduced by different model architectures and pre-training data: The standard deviation of the evaluated models’ accuracies reduces from 9.0 in the zero-shot setting to 1.7 after fine-tuning. This suggests that LLMs can benefit from task-specific fine-tuning to address seemingly sub-optimal architecture or pretraining conditions. For AQA, Flan-T5 benefits
3https://www.usmle.org/bulletin-information/ scoring-and-score-reporting
more from fine-tuning compared to the decoderonly models, possibly by better aligning generated outputs to the expected format of the answer. Decoder models present inconsistent results with improvements in ROUGE-L and BERTScore at the expense of lower METEOR scores, which raises concerns about the reliability of the AQA metrics.
Scaling up models introduces practical problems of deploying the model in real-world scenarios— smaller models may be preferred to larger ones due to faster inference times and lower memory footprints. We find that fine-tuning helps compensate for scale. Fine-tuned LLaMA 2 (7B) significantly outperforms zero-shot LLaMA 2 (13B) (+6.4 Accuracy gain on MCQA, +2.7 ROUGE-L gain and +4.4 BERTScore gain on AQA). Similarly, fine-tuned Flan-T5 (3B) outperforms zeroshot LLaMA 2 (13B) on 8 out of 16 MCQA datasets (see Tables 13 and 15).
In summary, we conclude that task-specific finetuning improves performance, mitigating weaknesses due to size, architecture and training data.
Finally, we report the potential of LLMs finetuned on in-domain data to generalize to medical datasets unseen during training to answer RQ5. To this end, during fine-tuning, we hold out ten MCQA and four AQA datasets presented in Figures 2 and 3.
Figure 2 shows the performance of LLaMA 2 (7B) and Flan-T5 (3B) models on the four held-out AQA evaluation sets. While LLaMA 2 does not appear to generalise to unseen AQA datasets, Flan-T5’s scores improve across the board. We note however, that this result might depend on the choice of metric, as Figures 6 and 7 in the Appendix paint a more mixed picture. Indeed, across all conducted experiments, only ROUGE-L scores show a statistically significant Spearman rank correlation with the reliable MCQA accuracy measure (r = 0.616, p = 0.008, more details in Appendix C). This suggests that other metrics used are either a sub-optimal choice or that they measure another, complementary aspect captured neither by Accuracy nor ROUGE-L. These findings highlight the low robustness of overlap-based NLG metrics discussed in the introduction.
Investigating the more robust MCQA setting, Figure 3 (comparing blue ZS with orange AQA-FT bars) shows that fine-tuning on AQA does not improve performance on unseen MCQA datasets. This suggests that higher scores on unseen AQA datasets might stem from better aligning genera-
tions to the expected answer form of AQA answers, which shows improvements in some of the AQA metrics, rather than acquiring additional medical knowledge during fine-tuning. While this could also due to a domain shift between the training and holdout datasets, this is not supported by the performance drop on MedQuaD, which, by this theory, should exhibit improved performance, since its domain is “General Medical”, and would thus be in-domain.
Figure 3 (comparing blue ZS with green MCQFT) suggests that models indeed can learn to acquire domain-specific knowledge during finetuning, as MCQA-tuned models consistently perform better than their zero-shot counterparts. This seemingly contradicts the previous finding that models fail to acquire additional medical knowledge when fine-tuned on the AQA datasets.
Further analysis indicates that the reported generalisation capabilities might be over-stated, as evaluation questions from the unseen datasets have se-
mantically similar counterparts in the fine-tuning data. However, a manual analysis of the cases where fine-tuned models outperform their zero-shot counter-parts reveals that only about 60% of the improvement can be explained by the presence of such similar examples. Details of this analysis are reported in Appendix D.
Based on these findings, we conclude that finetuning can serve as a partial solution for achieving generalisable adaptation to the medical domain.

heading: 6 Error Analysis
text: In this section, we analyze the errors of LLMs on MCQA datasets.

heading: 6.1 Category-wise and Manual Error Analysis
text: To better understand the performance of zero-shot and fine-tuned models across MCQA, we analyze
them broken down by sub-domain and knowledge source. We calculate the accuracy of the models in their zero-shot and fine-tuned settings for each category, as shown in Table 4.
Models tend to perform better on the biological and biomedical sub-domains. We posit as the reason for this that biomedical information is more readily available in the pre-training corpora of the models, e.g., in the form of biomedical abstracts (see also Table 10 in the Appendix). Furthermore, fine-tuning improves performance for all categories, but the gaps between medical and biomedical domains persist, indicating that medical questions are indeed harder to answer, even though they prevail in the training set. Perhaps more worryingly, the Consumer Health AQA scores do not improve as much as for other domains, even after fine-tuning (see Appendix, Table 19).
For knowledge sources, fine-tuned Flan-T5 (3B) excels in Within Context and Supportive and Preventive Services, also showing strong zeroshot capabilities in these categories, perhaps due to architecture or pre-training data. Similarly, finetuned MPT (7B) and LLaMA 2 (7B) show superior performance across categories. However, despite fine-tuning benefits, models still underperform in areas like General Medicine, Basic and Life Sciences, and Dental and Oral Health, which form the majority of the benchmark. Overall, we conclude that fine-tuning improves model performance in sub-domains, but knowledge gaps still persist across different domains and knowledge sources.
Finally, we sample 200 MCQA-questions from M-QALM evaluation data, and annotate the type of reasoning required to solve the problem: we distinguish three broad categories: Recall questions, which only require to recall necessary knowledge, Reading Comprehension questions, which require recall of knowledge and its combination with a given context—and Quantitative/Arithmetic questions, which require the calculation of quantities, such as probabilities or dosages. The majority of analyzed questions fall into the Recall category.
Together with the Reading Comprehension category, these questions account for 95% of annotated questions. These two categories probe the capabilities required for reading comprehension (Kintsch, 1988), validating the use of M-QALM for the stated purpose of evaluating comprehension and recall.
Table 5 describes the accuracy of the four base
and fine-tuned models: we find that Recall questions dominate the sample and models tend to perform best in this category, but even after finetuning on M-QALM, their performance hardly surpasses 50%, indicating that they may yet lack the necessary knowledge. Additionally, models perform worse on Reading Comprehension questions, suggesting that it is indeed harder to integrate necessary knowledge rather than just recalling it. Fine-tuning improves performance for all models for both types of reasoning. Quantitative/Arithmetic are the worstperforming category, even after fine-tuning. This is unsurprising, as arithmetic capabilities are observed to emerge with larger model scale (Wei et al., 2022).

heading: 6.2 Error Analysis of LLama-2
text: We perform a manual error analysis of the fine-tuned LLaMA 2 (7B) model on MCQA. We examine 200 non-Within Context questions where the model erred, and assign them to the Recall, Reading Comprehension and Quantitative/Arithmetic categories, as done previously. The model incorrectly answered 134 Recall, 52 Reading Comprehension, and 14 Quantitative/Arithmetic questions (Table 6). Comparing these errors to the earlier sample of 200 questions we analyze from the test set in Table 5, reveals that the distribution of errors for each category mirrors their general distribution in the overall test set. The prevalence of Recall questions in errors aligns with their dominance in medical exams like MEDMCQA, USMLE, and HEADQA. While fine-tuning on extensive medical corpora may enhance Recall question performance, improving on Reading Comprehension and Quantitative/Arithmetic questions might require different fine-tuning approaches, as these categories demand comprehension skills rather than mere knowledge recall.

heading: 7 Conclusions
text: In this work, we introduce M-QALM, a comprehensive collection of clinical datasets comprising 16 multiple-choice and 6 abstractive questionanswering datasets. Our study encompasses an extensive empirical investigation of open-source language models with up to 13 billion parameters. We assess their clinical and biomedical knowledge, their capacity to acquire such knowledge through training on M-QALM, and their ability to generalize to previously unseen datasets.
Our results highlight the strengths and limitations of LLMs on MCQA and AQA: while performing significantly better than a random guess baseline, they still fall significantly short in performance compared to proprietary language models and humans. This is true even after fine-tuning on M-QALM, which demonstrates potential improvements, especially in the context of instruction finetuned models like Flan-T5. Finally, we show inconsistencies arising from the use of different AQA metrics—in future work we will supplement the automated metrics by fine-grained expert-driven manual evaluation of LLM’s answers on M-QALM to learn to automate (some dimensions of) these expert judgments.
Based on our findings, we caution on the unconstrained use of open-source LLMs (Li et al., 2023; Han et al., 2023) as assistants to help perform medical tasks or provide answers to medical queries, to experts or lay people alike, as they seem to lack the necessary medical domain knowledge.
We make the dataset, experiment code and evaluation protocol publicly available4 to allow future developers of medical LLMs to assess the foundations of their models’ knowledge, as our evaluation shows that architecture of language models, the choice of datasets for pre-training and instruction fine-tuning can greatly impact their performance to the extent it can be assessed by M-QALM.
Limitations
In this paper, we evaluate the medical or clinical knowledge of LLMs by measuring their capability of answering test questions. While this can be a useful proxy measure of a model’s domain knowledge, it is insufficient to gauge its potential application in a real-world scenario. A multi-dimensional analysis of a model’s behaviour, including judging
4https://github.com/anand-subu/m-qalm
the completeness, harmlessness and usefulness of generated answers, is required in addition to solely evaluating their correctness.
Furthermore, the aggregated resource presented in this paper might be seen as lacking diversity, as all collected datasets are in English. To make inferences about the capabilities of evaluated models in other languages, a more diverse dataset with examples in other languages is required.
For our finetuning experiments, we only use parameter-efficient finetuning methods (PEFT) with QLora due to the high compute requirements for full-finetuning. We have not investigated the impact of the full-finetuning of these LLMs on our benchmark.

heading: A Datasets Used
text: In this section, we explain the MCQA and AQA datasets we used in detail. The dataset characteristics are presented in Table 1.
1. USMLE - English: We incorporate the USMLE dataset obtained from the MedQA dataset (Jin et al., 2021), comprising MCQA questions from the Medical Licensing Exam conducted in the US. We retain this dataset’s original training, validation, and test set divisions.
2. MEDMCQA: We incorporate the MEDMCQA dataset from (Pal et al., 2022), which comprises medical MCQA from Indian Medical Entrance Exams. We retain this dataset’s original training, validation, and test set splits. Similar to Singhal et al. (2023a), we evaluate all models on the validation set since we do not have answers for the test set.
3. MMLU: Following the design of Singhal et al. (2023a), we incorporate a subset of the MMLU datasets (6 datasets) (Hendrycks et al., 2021) which are MCQA specifically curated to assess medical domain knowledge. The subsets used are the anatomy, clinical knowledge, college medicine, medical genetics, professional medicine and college biology questions from MMLU. We utilize these datasets only for evaluating models.
4. MEDIQA-ANS: The MEDIQA 2019 shared task introduced the MEDIQA-QA dataset (Savery et al., 2020) for answer-ranking, comprising consumer health questions and passages from reputable online sources. The dataset was curated by extracting passages from the text of web pages, and includes manually generated single and multi-document summaries in both extractive and abstractive forms. We employ the multi-document abstractive summary as our questions’ ground truth reference answer. We specifically filter for questions and answers marked as excellent and utilize this as an AQA dataset solely for evaluating models.
5. HEADQA: We include the HEADQA dataset (Vilares and Gómez-Rodríguez, 2019), which comprises graduate-level MCQA about various
fields of medicine used for examinations to apply for specialization positions in the Spanish public healthcare system. We use the English version of the dataset and retain the original train, validation, and test split.
6. PubmedQA: The PubMedQA dataset (Jin et al., 2019) is a biomedical questionanswering dataset comprising 1,000 expertannotated QA instances. Each instance necessitates reasoning over a biomedical paper’s abstract to answer a relevant question. While the dataset provides long and short answers (yes, no, or maybe), we focus exclusively on the short answers for our evaluation, thereby generalizing the task as MCQA. We retain the original test split of 500 questions. Additionally, we allocate 100 questions from the training set to serve as a validation set, facilitating standardized training and validation in future studies.
7. BioMRC: The BIOMRC dataset (Pappas et al., 2020) focuses on machine reading comprehension within the biomedical domain. It is structured in a cloze-style MCQA format, where questions are based on biomedical abstracts where biomedical entities are replaced with pseudo-identifiers. The task is to correctly identify the masked entity in the title from a list of masked entities. We utilize two compact versions of BioMRC: Tiny A (Setting A) and Tiny B (Setting B). The BIOMRC dataset comprises a large training corpus, where masked entities share the same pseudo-identifier across the entire corpus. Setting A, retains the same pseudo-identifiers used for masked biomedical entities in the training corpus. This setup is beneficial when testing models trained using the BioMRC training set, allowing them to draw on previously seen patterns. Setting B, conversely, changes the pseudo-identifiers for every single question. This means that a model must rely solely on the information in the text of the question and the passage it refers to, without any help from repeated exposure to the same placeholders. While we maintain the original format for Setting B, assessing Setting A as is, is difficult as since we do not utilize the BioMRC training set, it is functionally the same as Setting B. To address this
limitation, we modify Setting A to include the original entity names and their corresponding pseudo-identifiers in the answer options, based on how the original paper (Pappas et al., 2020) assesses the performance of experts and non-experts. This aims to assess whether the model can accurately answer when provided with the information about their original entity names.
8. Processbank: The Processbank dataset (Berant et al., 2014) is designed for machine reading comprehension, featuring questions based on paragraphs describing biological processes. Each question, associated with a particular paragraph, has two answer options (MCQA). The dataset comes with a predefined split of 435 questions (150 files) for training and 100 questions (50 files) for testing. We allocate 25 files from the training set to create a validation set while retaining the original test set for model evaluation.
9. QA4MRE - Alzheimer’s disease QA: The dataset proposed by Morante et al. (Morante et al., 2012) contains MCQA questions regarding Alzheimer’s disease, aimed at assessing machine reading systems’ ability to answer questions about the disease by parsing relevant documents. We have adapted this dataset as an open-ended MCQA task to evaluate LLMs’ ability to answer these questions based on inherent knowledge. This dataset is employed solely for model evaluation purposes.
10. BioASQ: The BioASQ dataset (Tsatsaronis et al., 2015; Krithara et al., 2023) features biomedical questions crafted by experts. We utilize the BioASQ 2022 dataset for our benchmark. The BioASQ dataset is divided into two parts: for MCQA and another for AQA. For the MCQA part, we filter out the yes/no questions from BioASQ, converting them into an MCQ format to create a new subset, which we term BioASQ-MCQ. We manually create a training-validation (train-val) split of roughly 85%-15% from the filtered questions, resulting in 975 training questions and 173 validation questions and retaining a test set of 123 questions. For the AQA part, BioASQ provides fact, list, and bullet-type questions. We compile these into an AQA dataset, ensur-
ing a balanced representation of all question types in training and validation sets. The trainvalidation split results in 4733 training and 697 validation questions, with approximately 15% of all question types in the validation set.
11. MASH-QA: The MASH-QA dataset (Zhu et al., 2020) was designed for answering medical questions based on paragraphs where answers may span multiple text segments. Initially intended for extractive answering tasks, we repurpose it as an AQA task, utilizing the extractive answers as the reference ground truth.
12. MedQUAD: The MedQUAD dataset (Ben Abacha and Demner-Fushman, 2019) encompasses medical question-answer pairs extracted from various National Institute of Health (NIH) websites, covering topics on diseases, drugs, and other medical entities. Only nine of the twelve websites contributing to the original dataset have answers. We segregate questions from these nine websites and devise a train-validation-test split (AQA), assigning data from six websites for training, one website for validation, and two websites for testing.
13. TREC-2017 LiveQA: We employ the TREC2017 LiveQA dataset (Abacha et al., 2017) for evaluation purposes. Specifically, we leverage the rankings provided within the MedQUAD evaluation process (Ben Abacha and DemnerFushman, 2019) to keep question-answer pairs that have answer rating as excellent. We utilize this as an AQA dataset for evaluating the model.
14. British Ophthalmology Practice Tests: We employ sample questions from the Fellowship of the Royal College of Ophthalmologists (FRCOphth) exams, as provided by the Royal College of Ophthalmologists on their website (Raimondi et al., 2023; RCOphth, 2022a,b). These MCQA questions, geared towards testing ophthalmology-related knowledge, are used for evaluation.
15. MEDINFO: The MEDINFO dataset, introduced by Abacha et al. (Ben Abacha et al., 2019), consists of real consumer questions concerning medications and drugs. It en-
compasses 674 question-answer pairs (AQA), which we employ solely for evaluation.

heading: B Performance of other methods for MCQA datasets
text: We report the prior and current best scores on MCQA datasets from current literature in Table 9. GPT-4 combined with a prompting strategy labeled MedPrompt performs the best currently on USMLE, MEDMCQA, and the MMLU datasets. Of the 16 datasets, we can obtain comparable scores for 12. For HEADQA, the results reported by (Vilares and Gómez-Rodríguez, 2019) and (Liu et al., 2020) are across individual sections, whereas we calculate the scores overall across all questions. The method proposed by (Liu et al., 2020), named MurKe achieves average scores of 45.5% on Biology questions, 42.4% on Medicine questions, 42.3% on Nursing Questions, 48.0% on Pharmacology questions, 44.3% on Psychology questions and 44.3% on Chemistry Questions, with an overall macro-average of 44.4% across all the sections. Similarly, for the OPHTH dataset, the results reported by (Raimondi et al., 2023) are separate for Part 1 and Part 2 questions. Bing Chat performs the best on Part 1 questions, achieving a performance of 78.9%, and GPT-4 with prompting obtains a performance of 88.4% on Part 2 questions (Raimondi et al., 2023). We could not find directly comparable scores for the BioASQ MCQ datasets as the test sets are provided in different batches, with the results on the BioASQ leaderboard also reported separately in terms of batches. We combine the questions across all the batches into one combined test set. For BIOMRC - Tiny A, we do not have directly comparable scores from prior works as we provide the names of the original entities along with the pseudo-identifiers to the LLMs, similar to how (Pappas et al., 2020) evaluate the performance of experts and non-experts. In contrast, when evaluating the performance of systems/deep learning models, (Pappas et al., 2020) first fine-tune models on the BIOMRC-Lite dataset and evaluate performance on BIOMRC - Tiny A, without providing names of the original entities to the system.

heading: C Correlation between AQA and MCQA metrics
text: We use ROUGE-L, BERTScore and METEOR for evaluating the performance of LLMs for AQA. We try to understand which of the three metrics might be the
most reliable for evaluation. Assuming that MCQA evaluations give a more robust estimate of models’ capabilities due to the exact nature of evaluation, we calculate the correlation between the MCQA accuracy and each of the AQA metrics. Removing the Flan-T5-ZS models as outliers, we calculate the Spearman Rank Correlation and obtain the following results:
The scores indicate that only ROUGE-L scores show a reliable and statistically significant correlation to MCQA Accuracy scores, suggesting that this might be the more reliable metric of the three. However, we wish to stress that these results must not be taken as definitive because the underlying assumption is that models performing better on MCQA should also perform better on AQA.

heading: D Analysis of the causes of generalisation to unseen datasets
text: We aim to discriminate whether MCQA fine-tuned models’ performance on unseen MCQA datasets can be attributed to their ability to generalize in answering medical questions, or if their performance is influenced by memorization of questions from the training set. To this end, we examine three evaluation-only MCQ datasets not used in the training split of M-QALM: Clinical Knowledge Tests (MMLU-CK) and Medical Genetics (MMLUMG) from MMLU and the OPHTH dataset. We utilize semantic similarity algorithms to retrieve questions in the training sets that closely resemble those in these test sets and manually filter the retrieved results. We identify 6 out of 92, 12 out of 265, and 17 out of 100 questions in the OPHTH, MMLU-CK, and MMLU-MG datasets, respectively, that have similar counterparts in the MEDMCQA dataset which was used to fine-tune the LLaMA 2 model This suggests that scores might be inflated due to train-test leakage.
Next, we focus on questions that the LLaMA 2 (7B) model answered wrongly, but which were corrected by MCQA-fine-tuning. We then crossreference these with the closest equivalent ques-
tions in the MEDMCQA dataset. This allows us to categorize the correct answers from near-duplicate memorization or the model’s generalized learning capabilities. We find 5, 2, and 5 questions in the three investigated datasets, respectively, where the MCQA-fine-tuned model outperformed its zero-shot counterpart and identified closely related questions in MEDMCQA. Of these, 7 questions were nearduplicates with identical answers, while the remaining 5 would have required some level of clinical understanding for the model to answer them correctly. This suggests that the improved performance of instruction-tuned models on unseen datasets can be partially attributed to exposure to near-identical questions during training.

heading: E Prompts utilized
text: In this section, we outline all prompts used for finetuning and evaluating the LLMs. We define Single Context MCQA Prompt as the prompt for the PROCESSBANK dataset with a single paragraph context, Multi-Context MCQA Prompt as the prompt for the PUBMEDQA dataset with multiple paragraph contexts, Cloze MCQA Prompt for the BIOMRC Setting A and B datasets, MCQA Prompt for all other MCQA datasets, and AQA Prompt for all AQA datasets.

heading: Future Work
text: Additionally, we allocate 100 questions from the training set to serve as a validation set, facilitating standardized training and validation in future studies. 7. BioMRC: The BIOMRC dataset (Pappas et al. , 2020) focuses on machine reading comprehension within the biomedical domain. It is structured in a cloze-style MCQA format, where questions are based on biomedical abstracts where biomedical entities are replaced with pseudo-identifiers. The task is to correctly identify the masked entity in the title from a list of masked entities. We utilize two compact versions of BioMRC: Tiny A (Setting A) and Tiny B (Setting B). The BIOMRC dataset comprises a large training corpus, where masked entities share the same pseudo-identifier across the entire corpus. Setting A, retains the same pseudo-identifiers used for masked biomedical entities in the training corpus. This setup is beneficial when testing models trained using the BioMRC training set, allowing them to draw on previously seen patterns. Setting B, conversely, changes the pseudo-identifiers for every single question. This means that a model must rely solely on the information in the text of the question and the passage it refers to, without any help from repeated exposure to the same placeholders. While we maintain the original format for Setting B, assessing Setting A as is, is difficult as since we do not utilize the BioMRC training set, it is functionally the same as Setting B. To address this
limitation, we modify Setting A to include the original entity names and their corresponding pseudo-identifiers in the answer options, based on how the original paper (Pappas et al. , 2020) assesses the performance of experts and non-experts. This aims to assess whether the model can accurately answer when provided with the information about their original entity names. 8. Processbank: The Processbank dataset (Berant et al. , 2014) is designed for machine reading comprehension, featuring questions based on paragraphs describing biological processes. Each question, associated with a particular paragraph, has two answer options (MCQA). The dataset comes with a predefined split of 435 questions (150 files) for training and 100 questions (50 files) for testing. We allocate 25 files from the training set to create a validation set while retaining the original test set for model evaluation. 9. QA4MRE - Alzheimer’s disease QA: The dataset proposed by Morante et al. (Morante et al. , 2012) contains MCQA questions regarding Alzheimer’s disease, aimed at assessing machine reading systems’ ability to answer questions about the disease by parsing relevant documents. We have adapted this dataset as an open-ended MCQA task to evaluate LLMs’ ability to answer these questions based on inherent knowledge. This dataset is employed solely for model evaluation purposes. 10. BioASQ: The BioASQ dataset (Tsatsaronis et al. , 2015; Krithara et al. , 2023) features biomedical questions crafted by experts. We utilize the BioASQ 2022 dataset for our benchmark. The BioASQ dataset is divided into two parts: for MCQA and another for AQA. For the MCQA part, we filter out the yes/no questions from BioASQ, converting them into an MCQ format to create a new subset, which we term BioASQ-MCQ. We manually create a training-validation (train-val) split of roughly 85%-15% from the filtered questions, resulting in 975 training questions and 173 validation questions and retaining a test set of 123 questions. For the AQA part, BioASQ provides fact, list, and bullet-type questions. We compile these into an AQA dataset, ensur-
ing a balanced representation of all question types in training and validation sets. The trainvalidation split results in 4733 training and 697 validation questions, with approximately 15% of all question types in the validation set. 11. MASH-QA: The MASH-QA dataset (Zhu et al. , 2020) was designed for answering medical questions based on paragraphs where answers may span multiple text segments. Initially intended for extractive answering tasks, we repurpose it as an AQA task, utilizing the extractive answers as the reference ground truth. 12. MedQUAD: The MedQUAD dataset (Ben Abacha and Demner-Fushman, 2019) encompasses medical question-answer pairs extracted from various National Institute of Health (NIH) websites, covering topics on diseases, drugs, and other medical entities. Only nine of the twelve websites contributing to the original dataset have answers. We segregate questions from these nine websites and devise a train-validation-test split (AQA), assigning data from six websites for training, one website for validation, and two websites for testing. 13. TREC-2017 LiveQA: We employ the TREC2017 LiveQA dataset (Abacha et al. , 2017) for evaluation purposes. Specifically, we leverage the rankings provided within the MedQUAD evaluation process (Ben Abacha and DemnerFushman, 2019) to keep question-answer pairs that have answer rating as excellent. We utilize this as an AQA dataset for evaluating the model. 14. British Ophthalmology Practice Tests: We employ sample questions from the Fellowship of the Royal College of Ophthalmologists (FRCOphth) exams, as provided by the Royal College of Ophthalmologists on their website (Raimondi et al. , 2023; RCOphth, 2022a,b). These MCQA questions, geared towards testing ophthalmology-related knowledge, are used for evaluation. 15. MEDINFO: The MEDINFO dataset, introduced by Abacha et al. (Ben Abacha et al. , 2019), consists of real consumer questions concerning medications and drugs. It en-
compasses 674 question-answer pairs (AQA), which we employ solely for evaluation