MedREQAL: Examining Medical Knowledge Recall of Large Language Models via Question Answering (2024)

Abstract: In recent years, Large Language Models (LLMs) have demonstrated an impressive ability to encode knowledge during pre-training on large text corpora. They can leverage this knowledge for downstream tasks like question answering (QA), even in complex areas involving health topics. Considering their high potential for facilitating clinical work in the future, understanding the quality of encoded medical knowledge and its recall in LLMs is an important step forward. In this study, we examine the capability of LLMs to exhibit medical knowledge recall by constructing a novel dataset derived from systematic reviews – studies synthesizing evidence-based answers for specific medical questions. Through experiments on the new MedREQAL dataset, comprising questionanswer pairs extracted from rigorous systematic reviews, we assess six LLMs, such as GPT and Mixtral, analyzing their classification and generation performance. Our experimental insights into LLM performance on the novel biomedical QA dataset reveal the still challenging nature of this task.

heading: 1 Introduction
text: The field of natural language processing (NLP) has been transformed with the advent of pre-trained large language models (LLMs). During their process of pre-training to predict the next token on massive amounts of text data, these models learn and construct an internalized representation of world knowledge (Zhang et al., 2023). A popular domain of application of LLMs is healthcare, where they have the potential to democratize medical knowledge and facilitate access to healthcare, but also introduce risks of misinformation and lack of transparency (Clusmann et al., 2023). Recent work has hinted at the fact that LLMs encode clinical knowledge rather well (Singhal et al., 2023).
In the medical world, systematic reviews are overview studies that synthesize the best available
studies on a clearly defined medical research question. In them, the studies on a topic are analyzed, critically appraised, and their interpretations summarized by experts into a refined evidence-based conclusion (Pollock and Berge, 2018). Considering they synthesize the best available knowledge for a medical question into a concise answer, we see systematic reviews as a very well-suited proxy for testing the level of knowledge recall in LLMs.
When prompted in a zero-shot setting, instruction-tuned LLMs also tend to generate a conclusion addressing the question based on their recalled knowledge from diverse sources. To examine how well the LLMs perform medical knowledge recall, we constructed a question-answering (QA) dataset originating from systematic reviews and evaluated the performance of six LLMs, probing their medical knowledge recall. Based on experiments conducted on the newly constructed MedREQAL dataset, we analyze both the classification and generation performance, discussing insights about individual capabilities and limitations.
Our contributions include: (1) a novel dataset of biomedical question-answer pairs originating from rigorous systematic reviews, (2) experiments testing the zero-shot medical knowledge recall of six LLMs, and (3) deeper analysis of the dataset and experimental results. To ensure reproducibility, we provide the dataset and code in a public repository.1

heading: 2 Related Work
text: Medicine is a common domain of application for NLP tasks (Thirunavukarasu et al., 2023). It is marked by the highly complex biomedical language and terminology. Previous studies have highlighted the potential of LLMs in medical knowledge recall and exposed key gaps indicating the importance of further method development for creating safe
1https://github.com/jvladika/MedREQAL
14459
and effective LLMs for health-related applications (Liévin et al., 2022; Singhal et al., 2023).
Biomedical question answering (BQA) can be split into four main categories: scientific, clinical, consumer, and examination (Jin et al., 2022). Our dataset best fits into the scientific category, where questions and answers originate directly from biomedical research publications, and the most well-known datasets are PubMedQA (Jin et al., 2019) and BioASQ (Krithara et al., 2023). While the usual QA setting works with provided documents, recent efforts have started advancing the open-domain QA setting, where evidence first has to be discovered in order to answer the question (Jin et al., 2021; Vladika and Matthes, 2024a,b; Tian et al., 2024).
Also related is the task of scientific factchecking, focusing on assessing the veracity of claims based on relevant scientific evidence (Wadden et al., 2020; Vladika and Matthes, 2023). The most similar dataset to MedREQAL is HealthFC (Vladika et al., 2024), which also uses systematic reviews as its evidence source for answers but focuses on popular health inquiries and retells the reviews in lay language. Systematic reviews are most commonly associated in NLP with the task of their automatic construction (Marshall and Wallace, 2019; van Dijk et al., 2023), which is a highly complex task eliciting advanced biomedical knowledge and refined reasoning skills. To the best of our knowledge, we introduce the first BQA dataset directly generated from systematic reviews, all adhering rigorously to a predefined research protocol.

heading: 3 Dataset
text: Data Source Our data originates from systematic reviews conducted by the Cochrane Collaboration. Systematic reviews synthesize the best available evidence on a clearly defined medical research question and then provide a concise evidence-based conclusion (Pollock and Berge, 2018). The Cochrane Collaboration is a global charitable organization formed with the aim of improving evidence-based healthcare decision-making through systematic reviews of the effects of healthcare interventions (Henderson et al., 2010). Cochrane Reviews are done by 30,000 volunteer medical experts, who follow a strict methodology and a highly structured format (with same seven sections in every abstract) for easier publishing in the Cochrane database (Cumpston and Chandler, 2022). We deem that
Objective: To assess the effects of listening to music on sleep in adults with insomnia and to assess the influence of specific variables that may moderate the effect. Generated question: Can listening to music improve sleep in adults with insomnia? Conclusion: The findings of this review provide evidence that music may be effective for improving subjective sleep quality in adults with symptoms of insomnia. (...) Verdict: Supported (Jespersen et al., 2022) Objective: To assess the effects of alpha-lipoic acid as a disease-modifying agent in people with diabetic peripheral neuropathy. Generated question: Does alpha-lipoic acid have a disease-modifying effect on diabetic peripheral neuropathy? Conclusion: Our analysis suggests that ALA probably has little or no effect on neuropathy symptoms or adverse events at six months, and may have little or no effect on impairment at six months. (...) Verdict: Refuted (Baicus et al., 2024)
Table 1: Example of two instances from the MEDREQAL dataset. The question is generated from the original objective. The original conclusion is used for the answer generation task and the verdict is used for the classification task.
the focus on important healthcare interventions, wide domain coverage, standardized structure, and rigorous process followed by medical experts to construct these reviews are all factors that made them a highly suitable source for the construction of a novel biomedical question-answering dataset.
Dataset Construction To construct the dataset, we first scraped the abstracts of all the available Cochrane systematic reviews in the database PubMed, published from 2018 to 2023,2 using the Python library Beautiful Soup. The final QA dataset consists of (1) questions, (2) labels, and (3) long answers. Questions were generated by leveraging the Objectives section present in every review. The objective text (usually one declarative sentence) was given to an LLM and instructed to form a question from it using GPT-3.5 (Turbo-0125) (see Table 6 in Appendix for all prompt details). Long answers are original words taken directly from the Authors’ conclusions section present in each review. On top of these answers, we also generated a discreet label for each of these conclusions, which was one of the three: SUPPORTED, REFUTED, NOT ENOUGH INFORMATION. The motivation for this
2https://pubmed.ncbi.nlm.nih.gov/?term= %22Cochrane+Database+syst+rev%22%5BJournal% 5D&filter=years.2018-2023
is two-fold: (1) better alignment with existing comparable biomedical fact-checking and QA datasets that have these three labels, and (2) this allows for an easier evaluation of experiments with classification metrics like F1. The generation of these labels was also done with GPT-3.5 (see prompt in Table 6 in Appendix). Ultimately, the generated questions were sorted into various health areas using a classification prompt featuring 13 options, with a fourteenth category created afterward to consolidate questions concerning the renal and urinary systems. The classification process was performed using the same GPT-3.5 model with the temperature parameter set to 0. Two authors evaluated the generated questions and labels by randomly selecting 100 examples (∼4% of all questions) and found 94% of questions and 92% of labels to be correct, which we deem satisfying. The most common error in question generation was generating a question about the study itself (e.g., "Did the study analyze interventions to (...)?"), while the most common label error was mislabeling a refuting verdict with a low level of evidence as "not enough information".
Dataset Description The constructed dataset comprises a total of 2,786 generated questions, each averaging 16.6 words in length with a std. dev. of 5.4. The distribution of labels shows that a significant portion of the medical questions (2057) lack sufficient information for a conclusive verdict, while 543 questions are supported by evidence, and 186 are refuted based on available data. While the proportion of NEI answers seems high (74%) as opposed to supported & refuted (26%), this is the result of strict guidelines followed by Cochrane reviewers for giving a conclusive verdict. A similar distribution was found by Howick et al. (2022), where 2,428 Cochrane reviews were manually analyzed, and the authors note "that only 26% of interventions had effects that were supported by moderate quality evidence according to GRADE3".
As visualized in Figure 1, the distribution of questions across the health categories in our dataset exhibits a relatively balanced distribution, with most topics having a comparable percentage, indicating an even spread and thus contributing to the dataset’s quality. Being the majority class, cognitive and mental health comprises the largest proportion at 23%. Respiratory, cardiovascular, and cancer-related inquiries each represent approx-
3https://bestpractice.bmj.com/info/toolkit/ learn-ebm/what-is-grade/
imately every tenth question, highlighting significant attention to these vital areas. Other important areas include musculoskeletal health, sexual and reproductive health, and immune system-related questions. The dataset also covers questions concerning digestive health, nutrition, sensory organs, and other health topics, reflecting the diversity of topics encountered in clinical studies.

heading: 4 Experimental Setup
text: We conduct our experiments using a variety of large language models. The models were instructed to answer the question (thus producing a long answer) and also predict a final label (one of the S/R/NEI). For evaluation, predicted labels were collected and evaluated using classification metrics accuracy and macro-averaged F1 score, while the long answers were compared to the ground truth authors’ conclusions with NLG evaluation metric ROUGE (Lin, 2004). Since this metric focuses on lexical overlaps, we also use the BERTScore metric, which captures semantic similarity (Zhang et al., 2020).
We chose three general-purpose LLMs and three biomedical LLMs. As a commercial state-of-theart LLM, we include GPT-4 (Turbo-0125) (OpenAI, 2023) in our comparison since it has demonstrated remarkable zero-shot performance on various NLP tasks. Consequently, it is often used as a benchmark for comparing LLMs. We also include two recent open-source models that achieved impressive performance, namely the Mistral-7B (Instruct-v2) (Jiang et al., 2023); and Mixtral (Jiang et al., 2024), based on a sparse mixture-ofexperts architecture (Fedus et al., 2022).
We found three domain-specific biomedical models that fit our use case: (1) PMC-LLaMa 13B (Wu et al., 2023), which is an extension of LLaMa (Touvron et al., 2023) additionally pre-trained on biomedical research articles from the database
PubMed (Canese and Weis, 2013) and later finetuned on various datasets; (2) MedAlpaca 7B (Han et al., 2023), a biomedical extension of Alpaca (instruction-tuned version of LLaMa, Taori et al., 2023) fine-tuned on medical texts, encompassing resources such as medical flashcards, wikis, and dialogue datasets; and (3) ChatDoctor (7B) (Li et al., 2023), another extension of LLaMa finetuned on patient-physician conversation datasets.
Since the models were instruction-tuned with different templates, we used for each of them the base prompt reported by authors in technical reports. The prompt additionally instructed the models to predict one of the three classes and write an explanation for its answer (which we deem the "long answer"). All prompts can be found in Table 7. GPT-4 was prompted through the OpenAI API, and Mixtral through the FastChat API4 platform, which replicates OpenAI’s chat completion API endpoint. The other four models were small enough to be loaded onto a single A100 GPU w/ 80 GB VRAM, and took around two computation hours each to generate all 2786 answers. We set the token limit to 256 and the temperature parameter to 0 for GPT-4 and Mixtral, maximizing deterministic generation by favoring high-probability words.
All the models were tested only in a zero-shot setting with just the question provided. Providing the full text of the abstract would make the experiments focus on machine reading comprehension, which is not the goal of our study. We are interested in medical knowledge recall, which is best evaluated with a zero-shot setup. Initially, we also experimented with few-shot learning, but it only slightly helped in learning the formatting of answers and did not affect F1 performance since answers are still dependent on internal knowledge.

heading: 5 Results and Discussion
text: The performance metrics and results are summarized in Table 2. The three general-purpose models performed better on average, both in the classification and the generation task. To our surprise, GPT-4 performed worse than expected and was beaten in classification by the smaller Mistral and MedAlpaca models, as well as Mixtral. In total, Mixtral exhibited the best performance.
The difference in final classification results is apparent in Table 3 – our dataset contains mostly NEI labels and the best-performing models tended
4FastChat: https://github.com/lm-sys/FastChat
to predict this label frequently. On the other hand, GPT-4 was very prone to answering questions affirmatively and saying they are supported by the latest medical research. Conversely, Mixtral was more reluctant to give definitive affirmative or negative verdicts, but correctly predicted in most cases that there is not enough reliable evidence to answer the question. This is demonstrated in an example in Table 4. When predicting NEI, the models did not refuse to answer but rather gave some background to the question and hints of positive or negative results in the past while finishing by saying the conclusive evidence is missing.
Among the three biomedical LLMs, ChatDoctor had by far the worst performance and resorted to answering almost every question affirmatively (we observed this even when accounting for different values of hyperparameters temperature or top p). Interestingly enough, it was also the model with the highest BERTScore value of its generated text, showing that its fine-tuning on patient-doctor conversations most likely led to more authentic phrasing and word choices when delivering medical information. On the opposite side of the spectrum, PMC-LLaMa was the model that had by far the most REFUTED predictions (which were basically nonexistent for GPT-4 and Mixtral).
The disparity between REFUTED and NEI predictions is a consequence of another common pattern we observed – the models struggle to differentiate between these two classes. Questions labeled with NEI will usually have a conclusion saying there is not enough high-quality evidence to definitively answer the hypothesis. On the other hand, the REFUTED questions will conclude that reliable studies show no effect of the healthcare intervention on the outcome, i.e., there is no difference between the tested intervention and a placebo drug or treatment. Since both of these classes contain "negative" phrasing, they were commonly mislabeled by models even when they were able to recall correct clinical evidence.
Another interesting finding is shown in Table 5. The models were prompted in a zero-shot setting with no reference to reviews, and yet in hundreds of answers, the models referred to reviews and meta-analyses related to the posed question, sometimes even referring to Cochrane itself. This was most evident in Mistral-7B, followed by Mixtral and MedAlpaca. This clearly demonstrates the encoded internal medical knowledge and the ability of models to refer to systematic reviews as the highest type of clinical evidence to answer given questions.
Example response of all six LLMs for the same question is shown in Table 9. Both Mistral and Mixtral referred to (different) systematic reviews
on the given topic, while PMC-LLaMa referred to a specific randomized control trial. GPT-4 and MedAlpaca mentioned that the answer is based on clinical research, and ChatDoctor gave a plain affirmative answer. Still, it is evident that models tend to cite studies that are sometimes rather old. This can also lead to incorrect predictions and quoting outdated knowledge. An example is in Table 8, where the systematic review from 2016 did not have enough information but the updated version of the same review from 2023 in our dataset did have enough new studies to give a positive verdict. How to update the outdated knowledge contained within LLMs is an ongoing challenge of knowledge editing (Yao et al., 2023; Cohen et al., 2024).
In the future, we see the dataset MedREQAL as a challenging testbed for medical knowledge recall in LLMs, but also envision its potential for testing tasks like multi-document summarization, evidence retrieval, or retrieval-augmented generation.

heading: 6 Conclusion
text: We constructed a new biomedical QA dataset for testing the medical knowledge recall of LLMs. The dataset originates from systematic reviews, synthesized evidence-based studies on research questions, a well-suited proxy for knowledge recall probing. We tested the performance of three general LLMs and three biomedical LLMs, showing that the scale of the model or its domain-tuning is not always tied to better performance. Deeper insights show that models have a moderately high level of recall and awareness of systematic reviews as quality evidence but still struggle with decisively concluding when there is not enough evidence to answer the question, leaving space for future improvements.
Limitations
Our dataset was built semi-automatically and relied a lot on using automated generative methods for its construction, which could have led to certain incorrect labeling. While the questions are usually just a simple interrogative rewording of the original objective and the long answers are original authors’ conclusions, labels were completely generated from the given question and conclusion text. Our manual analysis of a 4% subset of data showed that the performance is above 92% correct. Keeping in mind that even human annotation is not perfect and always has inconsistencies in labeling, we deemed this a satisfying performance.
Our comparative analysis has certain limitations. We focus solely on a zero-shot setting and direct medical knowledge recall probing, and we acknowledge that settings of machine reading comprehension and in-context learning are also worthy of exploring in the future. We do not benchmark all relevant biomedical LLMs; some, like MedPaLM, were computationally too expensive for us to run. Our study also lacks human evaluation of generated model responses, which could have shed more light on the qualitative performance and userfriendliness of the answers.
Ethical Considerations
Our dataset and experiments deal with the highly sensitive domain of healthcare and medical NLP. While we probe the models in a zero-shot setting to elicit their internal medical knowledge for our research purposes, this is not a recommended way of their usage by end users or patients. Some responses still contained hallucinations and misleading medical advice that should be taken with a grain of salt and always manually checked within reliable sources or consulted with medical professionals.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their helpful suggestions. This research has been supported by the German Federal Ministry of Education and Research (BMBF) grant 01IS17049 Software Campus 2.0 (TU München).

heading: A Appendix
text: The Appendix provides supplementary material about this study, including the model prompts in full length (Tables 6 and 7) and example questions and model answers (Tables 8 and 9).

heading: Future Work
text: Deeper insights show that models have a moderately high level of recall and awareness of systematic reviews as quality evidence but still struggle with decisively concluding when there is not enough evidence to answer the question, leaving space for future improvements. Limitations
Our dataset was built semi-automatically and relied a lot on using automated generative methods for its construction, which could have led to certain incorrect labeling