Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review
Authors: Zihan Xu, Haotian Ma, Yihao Ding, Gongbo Zhang, Chunhua Weng, Yifan Peng
Accepted at: acl25

Natural Language Processing in Support of Evidence-based Medicine: A Scoping Review

Zihan Xu1,*, Haotian Ma1,*, Gongbo Zhang2, Yihao Ding3, Chunhua Weng2, Yifan Peng1

1Weill Cornell Medicine, 2Columbia University, 3University of Sydney
Correspondence: yip4002@med.cornell.edu
*Authors contributed equally

Abstract
Evidence-based medicine (EBM) is at the forefront of modern healthcare, emphasizing the use of the best available scientific evidence to guide clinical decisions. Due to the sheer volume and rapid growth of medical literature and the high cost of curation, there is a critical need to investigate Natural Language Processing (NLP) methods to identify, appraise, synthesize, summarize, and disseminate evidence in EBM. This survey presents an in-depth review of 129 research studies on leveraging NLP for EBM, illustrating its pivotal role in enhancing clinical decision-making processes. The paper systematically explores how NLP supports the five fundamental steps of EBM – Ask, Acquire, Appraise, Apply, and Assess. The review not only identifies current limitations within the field but also proposes directions for future research, emphasizing the potential for NLP to revolutionize EBM by refining evidence extraction, evidence synthesis, appraisal, summarization, enhancing data comprehensibility, and facilitating a more efficient clinical workflow.

1 Introduction
Evidence-based medicine (EBM) is at the forefront of modern healthcare, emphasizing the use of the best available scientific evidence to guide clinical decisions (Sackett et al., 1996). By integrating clinical expertise, patient values, and the most up-to-date research data, EBM facilitates healthcare decisions by patients and the general public, clinicians, guideline developers, administrators, and policymakers (Mehta et al., 2022; Kwaan and Melton, 2012; Van de Vliet et al., 2023).

The foundation of EBM heavily relies on comprehensive research data from detailed textual sources such as clinical trial publications, cohort studies, and case reports (Blunt, 2022; Ratnani et al., 2023). Navigating this evidence hierarchy necessitates the use of advanced Natural Language Processing (NLP) techniques, which are crucial for streamlining literature searches and extracting PICO (Patient/Population, Intervention, Comparison, Outcomes) elements (Peng et al., 2023; Nye et al., 2018). From the early utilization of statistical machine learning (Arora et al., 2019) and recurrent neural networks (Guan et al., 2019), there has been a significant shift towards more advanced technologies such as transformer-based frameworks and large language models (LLMs). These modern approaches employ self-supervised pretraining and instruct-tuning (Rohanian et al., 2024) to capture domain-specific knowledge (Kalyan et al., 2022), enhancing the accuracy and scalability of medical information processing (Thirunavukarasu et al., 2023). Particularly, the recent advancements in LLMs have further propelled NLP capabilities within EBM, excelling in more complex tasks such as appraising and synthesizing evidence (Góraska and Tacconelli, 2024), differentiating and ranking evidence (Datta et al., 2024), generating human-like responses, answering complex clinical questions (Shiraishi et al., 2024), and identifying relevant clinical trials (Devi et al., 2024a).

Despite these significant advancements, a comprehensive review summarizing NLP development and applications in EBM is still in demand. This paper seeks to fill the gap by offering a thorough review of essential NLP tasks in EBM, with a focus on evidence generation, such as evidence retrieval, extraction, synthesis, and summarization, as well as evidence adoption and evidence-based research, such as question-answering, clinical trial design and identification, and other cutting-edge studies across various clinical specialties.

Furthermore, we outline key benchmarks to facilitate the development of future NLP models. Finally, we explore several potential avenues for future research. To better support both clinicians and researchers in making more informed clinical decisions and producing more comprehensive review
literature, we have made these resources publicly available.1

2 Scope and Literature Selection

Our scoping review adheres to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA2) guidelines, as illustrated in Figure 1.

2.1 Information sources

We searched 4 databases, including PubMed3, IEEE Xplore4, ACM Digital Library5, and ACL Anthology6. The search included studies from the past 5 years, spanning 2019 to 2024.

2.2 Search strategy

Our search strategy was meticulously designed to capture the most relevant studies at the intersection of NLP and EBM (Supplementary File A.3). We targeted key NLP concepts and technologies by including terms such as ‘natural language processing’, ‘language model’, ‘large language model’, ‘computational linguistics’, ‘information extraction’, ‘information retrieval’, ‘clinical trial retrieval’, ‘text summarization’, ‘question answering’, ‘sentence segmentation’, ‘named entity recognition’, ‘tokenization’ and the abbreviations like ‘NLP’ and ‘LLM’. In the domain of EBM, we included terms like ‘Evidence-Based Medicine’, ‘Evidence-Based Practice’, ‘Clinical Trial’ and their abbreviations like ‘EBM’ and ‘EBP’, also limited to appearances in the title or abstract. We used the Boolean operator to combine any word from the NLP domain and any work from the EBM domain in our search terms.

2.3 Study selection and metadata extraction

The references of all eligible studies were imported into Covidence7, and duplicates were removed. We then screened the articles by title and abstract. Inclusion criteria were defined as (1) Studies published in English, (2) research applying NLP techniques specifically for EBM, and (3) Studies focusing on applications for humans. Exclusion criteria were defined as (1) articles unrelated to NLP for EBM, (2) non-English publications, and (3) secondary literature such as systematic reviews, retracted papers, survey papers, case studies, and descriptive papers lacking experimental results.

After the screening, the metadata was extracted from each paper, including models, disease, tasks involved, results, and limitations. Two annotators cross-verified the study selection and metadata extraction processes and consulted a third in cases of disagreement.

2.4 Study Statistics

From an initial pool of 601 papers retrieved from databases and 9 additional sources, we removed 8 duplicates. Subsequently, 386 papers were excluded during the initial screening based on predefined exclusion criteria, and 88 more were removed during full-text screening due to misaligned objectives or lack of relevance to EBM tasks. Ultimately, 129 studies met the inclusion criteria and form the basis of this review, with detailed metadata provided in Supplementary Table 1.

Figure 2 illustrates the distribution of research papers across different years (2019–2024) and their corresponding NLP tasks. There has been a rapid growth of papers over the years, peaking in 2023. The most common tasks throughout the years are Entity Extraction, Classification, and Evaluation, showing their foundational role in NLP for EBM research. Emerging tasks like Question Answering and Quality Assessment have appeared more prominently in recent years, reflecting evolving research directions.

![PRISMA flow diagram showing the inclusion and exclusion process for studies](page_1012_120_420_370.png)

Figure 1: PRISMA flow diagram.
Figure 2: Distribution of papers in different EBM tasks over time. The color schema is the same as Supplementary Table 1.

3 NLP Techniques for EBM

The entire EBM process consists of five steps, commonly referred to as the ‘5A’s: Ask, Acquire, Appraise, Apply, and Assess (Ratnani et al., 2023). NLP can be leveraged at each step to enhance the process (Table 1). For example, in the Ask step, clinicians or patients formulate precise clinical questions to address specific healthcare concerns. During the Acquire step, NLP can be employed to extract evidence, often leveraging the PICO framework. In the Appraise step, NLP tools can assist in evaluating and ranking the quality, validity, and relevance of the retrieved information to ensure its applicability to clinical decision-making. For the Apply and Assess steps, NLP can streamline the design and identification of relevant clinical trials and facilitate their integration into practice, enabling continuous assessment and refinement of patient care strategies. Detailed trends and advancements for each step of the EBM process are discussed in the following sections.

4 Ask - Searching & Selecting Studies

EBM can help researchers and clinicians draft a successful systematic review. After the scope and questions have been determined, the first step is to search for studies to include in the reviews and ensure they remain up to date.

This step is typically achieved using NLP-based information retrieval techniques, which extract relevant information from large text corpora based on user queries. Early heuristic methods involved structured, keyword-based queries to retrieve articles from repositories like MEDLINE or PubMed. These methods, while foundational, are limited by the high cost of expert annotation, maintenance, and domain sensitivity (Névêol et al., 2011). Despite these limitations, recent methods often rely on predefined rule-based strategies, e.g., SR[pt] and CQrs (Navarro-Ruan and Haynes, 2022), to filter and compare the retrieved results for systematic reviews. In addition, while statistical machine learning and context-aware models (Kamath et al., 2021; Samuel et al., 2021) have been widely adopted, they often lack scalability and struggle with less representative text embeddings.

Recent advancements are leaning towards transformer-based deep learning frameworks (Ramprasad et al., 2023a; Jin et al., 2022) due to their scalability and the ability to integrate medical ontologies, improving domain-specific text representation through self-supervised pretraining. For example, (Lokker et al., 2023) used BioBERT’s (Lee et al., 2020) embeddings and attention mechanisms to improve query representation and biomedical literature retrieval in clinical practice. Furthermore, the integration of generative AI models has advanced literature retrieval despite challenges like hallucination. For example, Gwon et al. (2024)

<table>
  <tr>
    <th>EBM cycle</th>
    <th>Description</th>
    <th>NLP tasks</th>
  </tr>
  <tr>
    <td>Ask</td>
    <td>Search & select studies</td>
    <td>Question answering, Information retrieval</td>
  </tr>
  <tr>
    <td>Acquire</td>
    <td>Collect data</td>
    <td>Named entity recognition and normalization, Relation extraction</td>
  </tr>
  <tr>
    <td>Appraise</td>
    <td>Examine relevance, validity, and results</td>
    <td>Quality assessment, Evidence ranking and screening, Evidence synthesis, Evidence summarization</td>
  </tr>
  <tr>
    <td>Apply & Asses</td>
    <td>Apply EBM in practice and research and evaluate their effectiveness</td>
    <td>Clinical trial identification and design, Question Answering, Domain-specific applications</td>
  </tr>
</table>

Table 1: Mapping of EBM cycle to corresponding NLP tasks.
compared Microsoft Bing AI and ChatGPT in accelerating the systematic literature search for a clinical review on Peyronie disease treatment, finding both can speed up the search process.

5 Acquire - Collecting Data

EBM is designed to identify all studies relevant to their research questions and synthesize data regarding the study design, risk of bias, and results. Therefore, the findings of EBM heavily depend on decisions about which data from these studies are presented and analyzed. The data collected should be accurate, complete, and accessible for future review, updates, and data-sharing purposes. Here we describe NLP approaches used to extract data directly from journal articles and other studies’ reports.

5.1 Entity Extraction and Normalization

Initially, entity (e.g., PICO) extraction relied on rule-based approaches, which utilize predefined lexical, syntactic, and contextual rules for extracting entities from clinical trial data (Chen et al., 2019c; Borchert et al., 2022). These methods are simple, transparent, and customizable, making them practical for high-precision tasks in structured contexts. Although they face challenges with complex or ambiguous data, their interpretability and ease of adaptation remain valuable for PICO extraction (Dhrangadhariya and Müller, 2023).

RNN/LSTM-based frameworks lacked long-term memory capabilities. Nevertheless, they have been used for sequential sentence classification to enhance context utilization and improve classification accuracy in unstructured or less structured medical abstracts (Jin and Szolovits, 2018).

The current trend is towards the dominance of transformer-based frameworks due to their domain-aware pertaining benefits. For instance, models such as SciBERT and PubMedBERT have been specifically developed for extracting ‘Intervention’ (‘I’ in PICO) (Tsubota et al., 2022), SrBERT (Aum and Choe, 2021) for classifying articles into “included” or “excluded” categories based on predefined inclusion criteria.

5.2 Relation Extraction

Following the identification of PICO elements, relation extraction approaches can be used to link these elements within studies.

Initially, rule-based and machine-learning methods were used to extract meaningful relationships from medical literature (Alodadi and Janeja, 2019; Borchert et al., 2022). By 2021, transformative methodologies were developed, integrating deep learning frameworks like BERT and Augment Mining (AM). For example, srBERT built-in (Aum and Choe, 2021), identified key elements and defines interrelations from the titles of articles. Stylianou and Vlahavas (2021) classified the relationships between argumentative components within the texts, such as claims and evidence. Their relationships were labeled as ‘supporting’ or ‘opposing’.

In the systematic review process, understanding the connections between different study results can influence the review outcomes. However, besides systematic reviews, automated relation extraction has shifted towards more structured approaches, such as schema-based relation extraction. For example, Sanchez-Graillet et al. (2022) utilized a richly annotated corpus that aligns with the C-TrO ontology. Complementing these advances, graph-based approaches offer a novel way to encode complex relationships between clinical entities. A knowledge graph is a structured representation of information where entities (e.g., symptoms, treatments, drugs) are represented as nodes and their relationships as edges. Graph-based approaches have emerged as an effective method to encode relationships. For example, a knowledge graph was used to organize and visualize relationships among clinical trial entities such as symptoms, treatments, and drug outcomes by structuring data into nodes and edges (Pan et al., 2021).

6 Appraise, Synthesize, and Summarize Evidence

This task screens the included studies for risk of bias and appraises them for quality to ensure that healthcare decisions are informed by the most reliable and relevant evidence. Once the appraisal is complete, the next step is synthesizing evidence by combining findings from multiple studies, often using meta-analyses. Finally, these synthesized insights are summarized into concise, actionable conclusions.

6.1 Quality Assessment

Developing tools to assess evidence is crucial in EBM, such as the fully automated tool that combines machine learning and rule-based techniques by Brassey et al. (2021). It assessed the evidence from randomized clinical trials and systematic re-
views by sentiment analysis, indication of bias, and sample size calculation, and used them to estimate the potential effectiveness of the intervention. Besides, deep learning models such as BERT (Devlin et al., 2019) have been used to evaluate the quality of evidence by analyzing article titles and abstracts. For example, different variations like BioBERT, BlueBERT, and BERT_{BASE} were fine-tuned to classify the articles based on their adherence to methodological quality criteria (Lokker et al., 2023).

6.2 Evidence Ranking and Screening

After the quality assessment, the next step is to screen and rank the evidence. Several ranking methods are available, with statistical-based methods being among the earliest used. For example, Norman et al. (2019b) developed a method to rank references by their likelihood of relevance. Compared with randomized screening, their study showed that prioritization methods (with technological assistance) allow for fewer studies to be screened while still producing reliable results, which effectively reduces both the time and cost associated with the screening process. Rybinski et al. (2020a) introduced the platform A2A, which used Okapi Best Match 25 (BM25) that assigned scores to documents based on term frequency and document length and Divergence from Randomness (DFR) that quantified informativeness as the divergence of a term’s distribution from randomness for document ranking. Additionally, machine learning methods are implemented. Rybinski et al. (2020b) designed a search system with a simple query formulation strategy for initial ranking and used pre-trained BERT models (SciBERT, BioBERT, and BlueBERT) for re-ranking in clinical trial searches, which improved the robustness.

6.3 Evidence Synthesis

Evidence synthesis combines data from included studies to draw conclusions about a body of evidence. While the most common method used is meta-analysis, which statistically combines results from studies to estimate overall effect sizes, NLP-based approaches have also been applied to synthesize studies or findings. Mutinda et al. (2022b) proposed a method to reproduce meta-analysis, computing summary statistics (e.g., risk ratio) and visualizing results using forest plots by extracting and normalizing PICO elements from breast cancer randomized controlled trials. However, this method is built on a small amount of data. Górska and Tacconelli (2024) developed a system to continuously update summary statistics from key publications, further improving the meta-analysis process. However, only binary outcomes were supported in both methods, limiting the applicability to broader meta-analysis needs. Besides meta-analysis, EvidenceMap (Kang et al., 2023) effectively synthesized medical findings by employing a structured and hierarchical representation comprising Entities, Propositions, and Maps that enhances the interpretability and retrievability of evidence through its sophisticated semantic relational retranslation.

6.4 Evidence Summarization

Finally, EBM must present a clear statement of findings or conclusions to help people make better-informed decisions and increase usability. This summary should include information on all important outcomes, evidence certainty, and the intervention’s desirable and undesirable consequences.

From an NLP technical perspective, evidence summarization uses extractive and abstractive strategies. Extractive summarization selects the most important sentences from the original text. Gulden et al. (2019) generated a new dataset from clinicaltrials.gov to test various algorithms (e.g., LexRank, TextRank, and Latent Semantic Analysis), identifying TextRank as the best performer in creating summaries directly from the source texts without altering the original wording. However, these algorithms suffered from inefficiency and high computational complexity when processing large datasets. Sarker et al. (2020) developed a lightweight system that leverages Maximal Marginal Relevance (MMR) and pre-trained word embeddings trained on PubMed and PMC texts to integrate semantic relevance and reduce redundancy. Similarly, Xie et al. (2022) proposed a knowledge infusion training framework called KeBioSum, which incorporated PICO into pre-trained language models (PLMs). It utilized lightweight knowledge adapters to reduce computational costs while improving semantic understanding and contextual representation.

Abstractive summarization focuses on the most critical information and creates new text for the summary; usually, more advanced techniques are used. Lalitha et al. (2023) have implemented sophisticated techniques such as neural network-based model T5 (Text-to-Text Transfer Transformer), BART (Bidirectional Auto-Regressive Transformer), and PEGASUS (Pre-training with
Extracted Gap-sentences for Abstractive Summarization Sequence-to-sequence) to resolve the challenge of obtaining useful information from a vast amount of clinical documents.

With the increased demand for user-interacted summarization, Ramprasad et al. (2023b) presented TrialsSummarizer, a system that helps automate summarizing the most relevant evidence in a set of randomized controlled trials by a multi-headed architecture, enabling each token in the generated summary to be explicitly linked to specific input aspects (e.g., population, intervention, or outcome). It introduces template-infilling capabilities, allowing users to correct or adjust generated summaries dynamically. Moreover, the application of LLMs has evolved to address these tasks with growing precision and depth. Hamed et al. (2023) explored ChatGPT’s capabilities in synthesizing diabetic ketoacidosis (KDA) guidelines by comparing, integrating, and abstracting content. Unlu et al. (2024) employed a Retrieval-Augmented Generation (RAG) framework with GPT-4 for generating responses to clinical trial eligibility questions based on retrieved patient data. Furthermore, TriSum (Jiang et al., 2024) stood out by using structured rationale-based abstractive summarization, where large language models generate aspect-triple rationales that are distilled into smaller models through a dual-scoring selection mechanism and curriculum learning.

7 Apply and Assess: adoption, refinement, and research

Transitioning from the evidence generation and synthesis, the next critical step is its adoption and refinement, facilitated by an ‘Evidence-based Research’ approach. Adoption and refinement are crucial to consistently reassessing and enhancing clinical evidence, particularly when existing evidence gaps lead to unmet needs of clinicians and patients. Evidence-based research further ensures that these gaps inform future clinical studies. Here, we summarize several applications identified from our literature review that align with this topic.

7.1 Specialty-specific adoption

In addition to general applications, we observed that NLP for EBM has been applied within specific medical specialties. Here, we summarized common specialties featured in the papers, such as oncology for conditions like Non-small cell lung cancer (NSCLC) and cardiovascular events such as heart failure. Other diseases are detailed in Supplementary Table 1.

Oncology. Cancer is a central topic in EBM, as it demands continuous integration of new research findings to guide evidence-based decisions for accurate diagnosis, effective treatment, and long-term patient management. Saiz et al. (2021) introduced Watson Oncology Literature Insights (WOLI), an AI system, by automatically identifying, prioritizing, and extracting relevant oncology research, which facilitated the translation of evidence into clinical practice. Similarly, the Clinical Trial Matching (CTM) system (Alexander et al., 2020) was evaluated at a cancer center in Australia with an overall accuracy of 92% for screening lung cancer patients. These tools highlight how AI-driven systems are increasingly embedded in hospital workflows.

Cardiology. Cardiology demands robust evidence to support the decision due to the high prevalence and the critical consequences of diagnostic errors, which can result in severe harm or loss of life. For example, the hybrid model proposed by Tun et al. (2023) exemplifies clinical practice by automating patient eligibility assessment directly within clinical workflows. In a real-world application on a dataset of 40,000 patients across several clinical care pathways, such as heart failure with reduced and preserved ejection fraction and atrial fibrillation, this model was deployed and achieved an impressive accuracy of 87.3%.

7.2 Clinical trial design and identification

Not all medical specialties are fully addressed by current research, and even in those with significant focus, the integration of findings into real-world guidance remains insufficient. Automating clinical trial procedures is critical for instant reaction to pandemics or public health emergencies. A crucial step in advancing future clinical trials or experiments is the design phase, where NLP plays a pivotal role. Effective clinical trial design involves structuring and optimizing trials to ensure they align with patient needs and research objectives. NLP tools can enhance the efficiency of clinical trial design by facilitating the automated matching of patients to suitable trials and ensuring trials are aligned with the right patient cohorts. This capability supports a more effective and timely deployment of research resources in emergency health
situations.

Eligibility matching and cohort identification is a process of matching patients to clinical trials based on their eligibility and identifying groups of patients (cohorts) who meet specific criteria for inclusion in clinical trials. There are several applications. Vydiswaran et al. (2019) proposed a hybrid approach to identify patient cohorts for clinical trials, which combines pattern-based, knowledge-intensive, and feature-weighting techniques to determine if patients meet specific selection criteria. Segura-Bedmar and Raez (2019) explored the use of deep learning models for cohort selection, framing it as a multi-label classification task. By employing CNNs and RNNs to process free-text eligibility criteria, this method allows for automatic learning of representations directly from text. Building on these foundations, Liu et al. (2022) developed Criteria2Query (C2Q) to extract and transform free-text eligibility criteria into structured, queryable data for cohort identification. More recently, Murcia et al. (2024) proposed the “Trial-Matcher” algorithm to match veterans for clinical trials using existing information within EHRs. It extracted attributes from patient profiles and eligibility criteria from trial profiles and compared them using the Sørensen-Dice Index (SDI). These applications show the potential of streamlining the process of recruitment and improving future clinical trial design. Now, researchers try to add LLMs to the studies. LLMs like GPT-3.5 or GPT-4 enhance clinical trial workflows by processing complex natural language data, such as patient profiles and trial eligibility criteria. The examples include AutoTrial (Wang et al., 2023b), focusing on trial design, specifically generating eligibility criteria using multi-step reasoning and hybrid prompting and TrialGPT (Jin et al., 2024), implementing a comprehensive framework for large-scale patient-trial matching, emphasizing real-world deployment and time-saving efficiency.

7.3 Drug repurposing

Another frontier application in this field is drug repurposing, which utilizes NLP to analyze existing medical literature and uncover new therapeutic applications for established drugs. By automating the analysis of large datasets such as clinical trials and research papers, NLP speeds up the identification of potential treatments, offering a faster and more cost-effective alternative to traditional drug discovery methods. During the COVID-19 pandemic, there is an urgent need for drugs for treatment. To quickly meet this requirement, the CovidX Network Algorithm Gates and Hamed (2020) was developed, which utilized NLP to analyze vast COVID-19 biomedical literature. It ranked potential drug candidates for repurposing, highlighting NLP’s power in automating and accelerating evidence synthesis during critical times. Alzheimer’s disease (AD), a progressive neurodegenerative disorder, remains a major global health challenge with limited treatment options and no definitive cure. Despite significant investment in drug development, the failure rate for Alzheimer’s-specific drugs in clinical trials remains exceedingly high. To address this, Daluwatumulle et al. (2022) employed knowledge graph embeddings to predict AD drug candidates by linking textual data and generating hypotheses from unstructured information.

7.4 Question Answering

While EBM is taught according to the five steps: ask, acquire, appraise, apply, and evaluate, a recent trend of application with the advancement in LLMs focuses on treating the entire process as a question-answering (QA) task. Xie et al. (2023) experimented with the consultation of rhinoplasty questions to ChatGPT, which pre-learned knowledge and summarized texts to respond, testing the potential of LLMs to offer valuable feedback. Moreover, Mohammed and Fiaidhi (2024) added bootstrapping to BioBERT and BioGPT so that they could better understand PICO questions from physicians and find potential answers from publications. Expanding on this trend, Chuan and Morgan (2021) introduced Chatbot SOPHIA, which helps users understand their eligibility for clinical trials by answering questions based on trial criteria. Addressing rare cancers, Jang et al. (2022) fine-tuned SAPBERT for QA and NER tasks, ultimately summarizing potential drugs ranked by relevance, such as bevacizumab, temozolomide, lomustine, and nivolumab.

8 EBM Benchmark dataset

Here, we summarize the benchmarks used in NLP and EBM (Supplementary Table 2). The tasks frequently involved with these benchmarks are Evidence Retrieval, Evidence Extraction, and Clinical Trial Identification. There is a notable gap in datasets specifically tailored for Evidence Synthesis and Appraisal, as well as Question Answering.
The existing datasets are often built upon general texts rather than medical-specific content. For example, CNN-DailyMail (See et al., 2017) is used for Evidence Summarization, but it is not medical-related. We also noticed that the primary data sources for these benchmarks are scholarly articles from PubMed and clinical trials.

9 Challenges and Future Directions

EBM is an important, rewarding, and dynamic field that organizes current data to improve healthcare decision-making. By integrating the best available evidence with a healthcare professional’s experience and the patient’s values, EBM aims to optimize health outcomes. Our focus here is on retrieving, extracting, appraising, synthesizing, and summarizing evidence from biomedical literature such as clinical trials, cohort studies, and case reports. However, conducting these analyses can be both demanding and time-consuming. In this study, we explore key NLP techniques that can streamline and facilitate this process.

Our review indicates that NLP-based systems or pipelines have achieved impressive results in EBM, such as extracting entities like PICO, enhancing the information retrieval engines, automating the evidence synthesis, assessing evidence quality, ranking the evidence with the highest confidence, summarizing the information, and answering questions. At the same time, as in any other evolving area, there remain challenges ahead. For example, generative models in EBM tasks have demonstrated impressive fluency and scalability, yet their tendency to hallucinate facts, lack source attribution, and sensitivity to prompt phrasing remain significant limitations for clinical use. A core challenge is the validation and trustworthiness of generated outputs, especially in high-stakes domains like medicine. Mechanisms such as RAG offer potential mitigations but require further development and evaluation.

From another perspective, particularly in handling diseases with limited literature or annotated data (Ge et al., 2023). Few-shot learning holds significant potential, as it enables models to generalize effectively from a small number of examples, reducing the dependency on large, annotated datasets. This data-efficient approach is crucial for EBM tasks in under-researched areas, such as rare diseases, where annotated resources are scarce. Few-shot learning can help these models adapt quickly to specific clinical needs, allowing for more accurate information extraction, question answering, and evidence synthesis, even with minimal training data.

Additionally, there is a pressing need for more benchmark datasets, especially for Evidence Synthesis and Appraisal and Question Answering. Current resources often rely on general corpora rather than those specifically oriented toward medical content, limiting the development of specialized NLP applications. Researchers can consider and build more meaningful datasets. Moreover, NLP-based tools have not yet been widely applied across all medical specialties, such as Urology and Hepatology, indicating room for expansion in these areas.

Another future direction for NLP in EBM involves incorporating real-world data from various sources, such as mobile devices, social media, and genomics. These data sources capture rich and diverse information beyond traditional clinical records, offering valuable insights into patient behaviors, lifestyle, environmental factors, and genetic predispositions. For example, data from mobile health apps and wearable devices can provide real-time health metrics. At the same time, social media posts may reveal patient self-reported outcomes or experiences that are often missed in clinical settings. Integrating genomic data adds another layer, enabling family history and personalized genomic code into disease risk and treatment response.

Furthermore, the “black box” nature of many NLP models limits their interpretability and accountability. Biases within training data can restrict NLP’s effectiveness and fairness across diverse patient demographics. Additionally, the high computational demands and the need for domain expertise in both NLP and healthcare are resource-intensive.

To fully realize the potential of NLP for EBM in real-world clinical workflows often involve interdisciplinary scenarios that span multiple conditions, comorbidities, and patient subpopulations. To address these complexities, NLP systems for EBM must evolve toward more holistic, adaptable frameworks capable of reasoning across diverse clinical questions and integrating heterogeneous data sources.

Addressing these limitations is important for enabling efficiency and ultimately contributing to a safer, more equitable healthcare landscape.
10 Conclusion

Our comprehensive review of over 600 papers resulted in the selection of 129 studies that focus on critical aspects of NLP within EBM. We first provide an overview of EBM, followed by a survey of NLP methods and techniques that address each step of the EBM process. We also explore use cases that demonstrate the application of EBM in various scenarios. Additionally, we review popular datasets and benchmarks. Finally, we present open challenges and future directions for research in this field. As NLP technologies evolve, they offer promising prospects for harnessing vast amounts of unstructured data, thus supporting clinical and research applications.

Limitations

Our study primarily focuses on English-language publications, potentially overlooking important research published in other languages. The inclusion criteria may have excluded studies indirectly related to EBM and NLP that could provide valuable insights. Additionally, our analysis only covers articles published between 2019 and 2024, which may have led to the omission of significant earlier works that contributed to the foundation of this field. Furthermore, the databases and search engines used in this review are limited, and it is possible that some relevant studies on NLP for EBM during the specified period were not identified.

Acknowledgments

This project was sponsored by the National Library of Medicine grants R01LM014344 and R01LM014573.

References

Md Abdullah Al Hafiz Khan, Md Shamsuzzaman, Sadid A. Hasan, Mohammad S. Sorower, Joey Liu, Vivek Datla, Mladen Milosevic, Gabe Mankovich, Rob van Ommering, and Nevenka Dimitrova. 2019. Improving disease named entity recognition for clinical trial matching. In 2019 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 2541–2548.

Marliese Alexander, Benjamin Solomon, David L Ball, Mimi Sheerin, Irene Dankwa-Mullan, Anita M Preininger, Gretchen Purcell Jackson, and Dishan M Herath. 2020. Evaluation of an artificial intelligence clinical trial matching system in australian lung cancer patients. JAMIA Open, 3(2):209–215.

Mohammad S. Alodadi and Vandana P. Janeja. 2019. Linking knowledge discovery in clinical notes and massive biomedical literature repositories. In 2019 IEEE International Conference on Big Data. IEEE.

Paul Arora, Devon Boyne, Justin J. Slater, Alind Gupta, Darren R. Brenner, and Marek J. Druzdzel. 2019. Bayesian networks for risk prediction using real-world data: A tool for precision medicine. Value in Health, 22(4):439–445.

Sungmin Aum and Seon Choe. 2021. srBERT: automatic article classification model for systematic review using BERT. Syst. Rev., 10(1):285.

Jacob Beattie, Sarah Neufeld, Daniel Yang, Christian Chukwuma, Ahmed Gul, Neil Desai, Steve Jiang, and Michael Dohopolski. 2024. Utilizing large language models for enhanced clinical trial matching: A study on automation in patient screening. Cureus, 16(5):e60044.

J Thaddeus Beck, Melissa Rammage, Gretchen P Jackson, Anita M Preininger, Irene Dankwa-Mullan, M Christopher Roebuck, Adam Torres, Helen Holtzen, Sadie E Coverdill, M Paul Williamson, Quincy Chau, Kyu Rhee, and Michael Vinegra. 2020. Artificial intelligence tool for optimizing eligibility screening for clinical trials in a large community cancer center. JCO Clin. Cancer Inform., 4(4):50–59.

Chris Blunt. 2022. The pyramid schema: The origins and impact of evidence pyramids. SSRN Electron. J.

Florian Borchert, Christina Lohr, Luise Moderosohn, Thomas Langer, Markus Follmann, Jan Philipp Sachs, Udo Hahn, and Matthieu-P. Schaprano. 2020. GG-POINC: A corpus of German medical text with rich metadata based on clinical practice guidelines. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 38–48, Online. Association for Computational Linguistics.

Florian Borchert, Laura Meister, Thomas Langer, Markus Follmann, Bert Arnrich, and Matthieu-P Schaprano. 2022. Controversial trials first: Identifying disagreement between clinical guidelines and new evidence. In AMIA Annual Symposium Proceedings, pages 237–246.

Jon Brassey, Christopher Price, Jonny Edwards, Markus Zlabinger, Alexandros Bampoulidis, and Allan Hanbury. 2021. Developing a fully automated evidence synthesis tool for identifying, assessing and collating the evidence. BMJ Evid. Based Med., 26(1):24–27.

Austin J. Brockmeier, Meizhi Ju, Piotr Przybyla, and Sophia Ananiadou. 2019. Improving reference prioritisation with pico recognition. BMC Medical Informatics and Decision Making, 19:256.

Tianrun Cai, Fiona Cai, Kumar P. Dahal, Gabrielle Cremona, Ethan Lam, Charlotte Golnik, Thany Seyok, Chuan Hong, and Katherine P. Liao. 2021. Improving the efficiency of clinical trial recruitment using an
ensemble machine learning to assist with eligibility screening. ACR Open Rheumatology, 3:593–600.

Leonardo Campillos-Llanos, Ana Valverde-Mateos, Adrián Caplönch-Carrión, and Antonio Moreno-Sandoval. 2021. A clinical trials corpus annotated with umls entities to enhance the access to evidence-based medicine. BMC Medical Informatics and Decision Making, 21(69). This article has been corrected. See BMC Med Inform Decis Mak. 2021 Apr 7;21:118.

Boyu Chen, Hao Jin, Zhiwen Yang, Yingying Qu, Heng Weng, and Tianyong Hao. 2019a. An approach for transgender population information extraction and summarization from clinical trial text. BMC Med. Inform. Decis. Mak., 19(Suppl 2):62.

Chi-Jen Chen, Neha Warikoo, Yun Chun Chang, and Chen. 2019b. Medical knowledge infused convolutional neural networks for cohort selection in clinical trials. Journal of the american medical informatics association, 26(11):1227–1236.

Long Chen, Yu Gu, Xin Ji, Chao Lou, Zhiyong Sun, Haodan Li, Yuan Gao, and Yang Huang. 2019c. Clinical trial cohort selection based on multi-level rule-based natural language processing system. Journal of the American Medical Informatics Association, 26(11):1218–1226.

Ching-Hua Chuan and Susan Morgan. 2021. Creating and evaluating chatbots as eligibility assistants for clinical trials: An active deep learning approach towards user-centered classification. ACM Trans. Comput. Healthc., 2(1):1–19.

Jonathan W Cunningham, Pulkit Singh, Christopher Reeder, Brian Claggett, Pablo M Marti-Castelote, Emily S Lau, Shaan Khurshid, Puneet Batra, Steven A Lubitz, Mahnaz Maddah, Anthony Philipakis, Akshay S Desai, Patrick T Ellinor, Orly Varady, Scott D Solomon, and Jennifer E Ho. 2024. Natural language processing for adjudication of heart failure in a multicenter clinical trial: A secondary analysis of a randomized clinical trial. JAMA Cardiol., 9(2):174–181.

Geesa Daluwatumulle, Rupika Wijesinghe, and Ruvan Weerasinghe. 2022. In silico drug repurposing using knowledge graph embeddings for alzheimer’s disease. In Proceedings of the 9th International Conference on Bioinformatics Research and Applications, pages 61–66, New York, NY, USA. ACM.

Surabhi Datta, Kyeryoung Lee, Hunki Paek, Frank J. Manion, Nneka Ofoegbu, Jingcheng Du, Ying Li, Liang-Chin Huang, Jingqi Wang, Bin Lin, Hua Xu, and Xiaoyan Wang. 2024. Autocriteria: a generalizable clinical trial eligibility criteria extraction system powered by large language models. Journal of the American Medical Informatics Association, 31(2):375–385. Published: 11 November 2023.

Yang Deng, Yaliang Li, Ying Shen, Nan Du, Wei Fan, Min Yang, and Kai Lei. 2019. Medtruth: A semi-supervised approach to discovering knowledge condition information from multi-source medical data. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM ’19, page 719–728, New York, NY, USA. Association for Computing Machinery.

Arti Devi, Shashank Uttrani, Aryansh Singla, Sarthak Jha, Nataraj Dasgupta, Sayee Natarajan, Rajeshwari S Punekar, Larry A Pickett, and Varun Dutt. 2024a. Quantitative analysis of GPT-4 model: Optimizing patient eligibility classification for clinical trials and reducing expert judgment dependency. In Proceedings of the 2024 8th International Conference on Medical and Health Informatics, pages 230–237, New York, NY, USA. ACM.

Arti Devi, Shashank Uttrani, Aryansh Singla, Sarthak Jha, Nataraj Dasgupta, Sayee Natarajan, Rajeshwari S. Punekar, Larry A. Pickett, and Varun Dutt. 2024b. Automating clinical trial eligibility screening: Quantitative analysis of GPT models versus human expertise. In Proceedings of the 17th International Conference on PErvasive Technologies Related to Assistive Environments, New York, NY, USA. ACM.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Jay DeYoung, Iz Beltagy, Madeleine van Zuyljen, Bailey Kuehl, and Lucy Wang. 2021. MS^2: Multi-document summarization of medical studies. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7494–7513, Stroudsburg, PA, USA. Association for Computational Linguistics.

Houssein Dhayne, Rima Kilany, Rafiqul Haque, and Yehia Taher. 2021. Emr2vec: Bridging the gap between patient data and clinical trial. Computers & Industrial Engineering, 156:107236.

Anjani Dhrangadhariya, Roger Hilfiker, Roger Schaer, and Henning Müller. 2020. Machine learning assisted citation screening for systematic reviews. Digital Personalized Health and Medicine.

Anjani Dhrangadhariya, Gaetano Manzo, and Henning Müller. 2024. Pico to picos: Weak supervision to extend datasets with new labels. Digital Health and Informatics Innovations for Sustainable Health Care Systems, 316.

Anjani Dhrangadhariya and Henning Müller. 2023. Not so weak pico: leveraging weak supervision for participants, interventions, and outcomes recognition for systematic review automation. JAMIA Open, 6(1).
Nhan V Do, Danne C Elbers, Nathanael R Fillmore, Samuel Ajarapu, Steven J Bergstrom, John Bihn, June K Corrigan, Rupali Dhond, Svitlana Dipietro, Arkady Dolgin, Theodore C Feldman, Sergey D Goryachev, Linden B Huhmann, Jennifer La, Paul A Marcantonio, Kyle M McGrath, Stephen J Miller, Vinh Q Nguyen, George R Schneeloch, Feng-Chi Sung, Kaithin N Swinnerton, Amelia H Tarren, Hannah M Tosi, Danielle Valley, Austin D Vo, Cenk Yildirim, Chunlei Zheng, Robert Zwolinski, Gisele A Sarosy, David Loose, Colleen Shannon, and Mary T Brophy. 2024. Matching patients to accelerate clinical trials (MPACT): Enabling technology for oncology clinical trial workflow. Stud. Health Technol. Inform., 310:1086–1090.

Nicholas J. Dobbins, Bin Han, Weipeng Zhou, Kristine F. Lan, H. Nina Kim, Robert Harrington, Özlem Uzuner, and Meliha Yetisgen. 2023. LeafAI: query generator for clinical cohort discovery rivaling a human programmer. Journal of the American Medical Informatics Association, 30(12):1954–1964.

Nicholas J Dobbins, Tony Mullen, Özlem Uzuner, and Meliha Yetisgen. 2022. The leaf clinical trials corpus: a new resource for query generation from clinical trial eligibility criteria. Sci. Data, 9(1):490.

Jingcheng Du, Qing Wang, Jingqi Wang, Prerana Ramesh, Yang Xiang, Xiaoqian Jiang, and Cui Tao. 2021. COVID-19 trial graph: a linked graph for COVID-19 clinical trials. J. Am. Med. Inform. Assoc., 28(9):1964–1969.

Abdelazeem Eldawlatly, Hussain Alshehri, Abdullah Alqahtani, Abdulaziz Ahmad, Fatma Al-Dammas, and Amir Marzouk. 2018. Appearance of population, intervention, comparison, and outcome as research question in the title of articles of three different anesthesia journals: A pilot study. Saudi Journal of Anaesthesia, 12(2):283–286.

Yilu Fang, Jae Hyun Kim, Betina Ross Idnay, Rebecca Aragon Garcia, Carmen E. Castillo, Yingcheng Sun, Hao Liu, Cong Liu, Chi Yuan, and Chunhua Weng. 2021. Participatory design of a clinical trial eligibility criteria simplification method. Studies in Health Technology and Informatics, 281:984–988.

Lyndsey Elaine Gates and Ahmed Abdeen Hamed. 2020. The anatomy of the SARS-CoV-2 biomedical literature: Introducing the CovidX network algorithm for drug repurposing recommendation. J. Med. Internet Res., 22(8):e21169.

Yao Ge, Yuting Guo, Sudeshna Das, Mohammed Ali Al-Garadi, and Abeed Sarker. 2023. Few-shot learning for medical text: A review of advances, trends, and opportunities. J. Biomed. Inform., 144(104458):104458.

Madhusudan Ghosh, Shrimon Mukherjee, Asmit Ganguly, Partha Basuchowdhuri, Sudip Kumar Naskar, and Debasis Ganguly. 2024a. AlpaPICO: Extraction of PICO frames from clinical trial documents using LLMs. Methods, 226:78–88.

Madhusudan Ghosh, Shrimon Mukherjee, Payel Santra, Girish Na, and Partha Basuchowdhuri. 2024b. BLINKtextsubscriptLSTM: BioLinkBERT and LSTM based approach for extraction of PICO frame from clinical trial text. In Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD), New York, NY, USA. ACM.

Meijian Guan, Samuel Cho, Robin Petro, Wei Zhang, Boris Pasche, and Umit Topaloglu. 2019. Natural language processing and recurrent network models for identifying genomic mutation-associated cancer treatment change from patient progress notes. JAMIA Open, 2(1):139–149.

Christian Gulden, Melanie Kirchner, Christina Schütter, Marc Hinderer, Marvin Kampf, Hans-Ulrich Prokosch, and Dennis Toddenroth. 2019. Extractive summarization of clinical trial descriptions. Int. J. Med. Inform., 129:114–121.

YN Gwon, JH Kim, HS Chung, EJ Jung, J Chun, S Lee, and SR Shim. 2024. The use of generative ai for scientific literature searches for systematic reviews: Chatgpt and microsoft bing ai performance evaluation. JMIR Medical Informatics, 12:e51187.

Anna Góraska and Evelina Tacconelli. 2024. Towards autonomous living meta-analyses: A framework for automation of systematic review and meta-analyses. Stud. Health Technol. Inform., 316:378–382.

Ehab Hamed, Ahmad Eid, and Medhat Alberry. 2023. Exploring ChatGPT’s potential in facilitating adaptation of clinical guidelines: A case study of diabetic ketoacidosis guidelines. Cureus, 15(5):e38784.

Hamed Hassanzadeh, Sarvnaz Karimi, and Anthony Nguyen. 2020. Matching patients to clinical trials using semantically enriched document representation. Journal of Biomedical Informatics, 105:103406.

Hendrik Ter Horst, Nicole Brazda, Jessica Schira-Heinen, Julia Krebbers, Hans-Werner Müller, and Philipp Cimiano. 2023. Automatic knowledge graph population with model-complete text comprehension for pre-clinical outcomes in the field of spinal cord injury. Artificial Intelligence in Medicine, 137:102491.

Yan Hu, Vipina K Keloth, Kalpana Raja, Yong Chen, and Hua Xu. 2023. Towards precise PICO extraction from abstracts of randomized controlled trials using a section-specific learning approach. Bioinformatics, 39(9):btad542.

Andy S. Huang, Kyle Hirabayashi, Laura Barna, Deep Parikh, and Louis R. Pasquale. 2024. Assessment of a large language model’s responses to questions and cases about glaucoma and retina management. JAMA Ophthalmology, 142(4):371–375.

Bum-Sup Jang, Andrew J Park, and In Ah Kim. 2022. Exploration of biomedical knowledge for recurrent glioblastoma using natural language processing deep learning models. BMC Med. Inform. Decis. Mak., 22(1):267.
Pengcheng Jiang, Cao Xiao, Zifeng Wang, Parminder Bhatia, Jimeng Sun, and Jiawei Han. 2024. TriSum: Learning summarization ability from large language models with structured rationale. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2805–2819, Mexico City, Mexico. Association for Computational Linguistics.

Di Jin and Peter Szolovits. 2018. PICO element detection in medical text via long short-term memory neural networks. In Proceedings of the BioNLP 2018 workshop, Stroudsburg, PA, USA. Association for Computational Linguistics.

Qiao Jin, Chuanqi Tan, Mosha Chen, Ming Yan, and Xiaozhong Liu Ningyu Zhang, Songfang Huang. 2022. State-of-the-art evidence retriever for precision medicine: Algorithm development and validation. JMIR Medical Informatics, 10(12):e40743.

Qiao Jin, Zifeng Wang, Charalampos S Floudas, Fangyuan Chen, Changlin Gong, Dara Bracken-Clarke, Elisabetta Xue, Yifan Yang, Jimeng Sun, and Zhiyong Lu. 2024. Matching patients to clinical trials with large language models. Nat. Commun., 15(1):9074.

Tom H Johnston, Alix M B Lacoste, Paula Ravenscroft, Jin Su, Sahar Tamadon, Mahtab Seifi, Anthony E Lang, Susan H Fox, Jonathan M Brotchie, and Naomi P Visanji. 2024. Using artificial intelligence to identify drugs for repurposing to treat L-DOPA-induced dyskinesia. Neuropharmacology, 248(109880):109880.

Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2022. Ammu: A survey of transformer-based biomedical pretrained language models. Journal of Biomedical Informatics, 126:103982.

Sowmya Kamath, Veena Mayya, and Priyadarshini. 2021. A probabilistic precision information retrieval model for personalized clinical trial recommendation based on heterogeneous data. In 2021 12th International Conference on Computing Communication and Networking Technologies (ICCCNT), pages 1–5. IEEE.

Lara J Kanbar, Benjamin Wissel, Yizhao Ni, Nathan Pajor, Tracy Glauser, John Pestian, and Judith W Dexheimer. 2022. Implementation of machine learning pipelines for clinical practice: Development and validation study. JMIR Medical Informatics, 10(12):e37833.

Tian Kang, Adler Perotte, Youlan Tang, Casey Ta, and Chunhua Weng. 2021. Umls-based data augmentation for natural language processing of clinical research literature. Journal of the American Medical Informatics Association, 28(4):812–823.

Tian Kang, Yingcheng Sun, Jae Hyun Kim, Casey Ta, Adler Perotte, Kayla Schiffer, Mutong Wu, Yang Zhao, Nour Moustafa-Fahmy, Yifan Peng, and Chunhua Weng. 2023. EvidenceMap: a three-level knowledge representation for medical evidence computation and comprehension. J. Am. Med. Inform. Assoc., 30(6):1022–1031.

Tian Kang, Shirui Zou, and Chunhua Weng. 2019. Pre-training to recognize PICO elements from randomized controlled trial literature. Stud. Health Technol. Inform., 264:188–192.

Samuel Kaskovich, Kirk D. Wyatt, Tomasz Oliwa, Luca Graglia, Brian Furner, Jooho Lee, Anoop Mayamapurath, and Samuel L. Volchenboum. 2023. Automated matching of patients to clinical trials: A patient-centric natural language processing approach for pediatric leukemia. JCO Clinical Cancer Informatics, 7.

Jenna Kefeli and Nicholas Tatonetti. 2024. Tega-reports: A machine-readable pathology report resource for benchmarking text-based ai models. Patterns, 5(3):100933. Published online February 21, 2024.

AH Khan, A Abbe, B Falissard, P Carita, C Bachert, J Mullol, M Reaney, J Chao, LP Mannent, N Amin, P Mahajan, G Pirozzi, and L Eckert. 2021. Data mining of free-text responses: An innovative approach to analyzing patient perspectives on chronic rhinosinusitis with nasal polyps in a phase iia proof-of-concept study for dupilumab. Dove Medical Press, 2021(15):2577–2586.

Jeongeun Kim, Mitchell Izower, and Yuri Quintana. 2023a. Parsable clinical trial eligibility criteria representation using natural language processing. In AMIA Annual Symposium Proceedings, pages 616–624. American Medical Informatics Association.

Jeongeun Kim, Mitchell Izower, and Yuri Quintana. 2023b. Parsable clinical trial eligibility criteria representation using natural language processing. In AMIA Annual Symposium Proceedings, pages 616–624. American Medical Informatics Association.

Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yencken. 2024. Nicita-pibos dataset. https://doi.org/10.57702/ne4r48m1. Dataset consists of 1,000 medical abstracts manually annotated with semantic tags based on the PICO criteria to support the automatic classification of sentences.

Bevan Koopman, Tracey Wright, Natacha Omer, Veronica McCabe, and Guido Zuccon. 2021. Precision medicine search for paediatric oncology. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, page 2536–2540, New York, NY, USA. Association for Computing Machinery.

Bevan Koopman and Guido Zuccon. 2022. Cohort-based clinical trial retrieval. In Proceedings of the 25th Australasian Document Computing Symposium, ADCS ’21, New York, NY, USA. Association for Computing Machinery.
Fabricio Kury, Alex Butler, Chi Yuan, Li-Heng Fu, Yingcheng Sun, Hao Liu, Ida Sim, Simona Carini, and Chunhua Weng. 2020. Chia, a large annotated corpus of clinical trial eligibility criteria. Sci. Data., 7(1):281.

Mary R Kwaan and Genevieve B Melton. 2012. Evidence-based medicine in surgical education. Clin. Colon Rectal Surg., 25(3):151–155.

Evani Lalitha, Kasarapu Ramani, Dudukula Shahida, Esikela Venkata Sai Deepak, M Hima Bindu, and Diguri Shaikshavali. 2023. Text summarization of medical documents using abstractive techniques. In 2023 2nd International Conference on Applied Artificial Intelligence and Computing (ICAAIC), pages 939–943. IEEE.

Mengfei Lan, Mandy Cheng, Linh Hoang, Gerben Ter Riet, and Halil Kilicoglu. 2024. Automatic categorization of self-acknowledged limitations in randomized controlled trial publications. J. Biomed. Inform., 152:104628.

Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240.

Kyeryoung Lee, Zongzhi Liu, Yun Mai, Tomi Jun, Meng Ma, Tongyu Wang, Lei Ai, Ediz Calay, William Oh, Gustavo Stolovitzky, Eric Schadt, and Xiaoyan Wang. 2024. Optimizing clinical trial eligibility design using natural language processing models and real-world data: Algorithm development and validation. JMIR AI, 3:e50800.

Chao Li, Harsha Gurulingappa, Prathamesh Karmalkar, Jana Raab, Aastha Vij, Gerard Megaro, and Christian Henke. 2021a. Automate clinical evidence synthesis by linking trials to publications with text analytics. In 2021 International Symposium on Electrical, Electronics and Information Engineering, New York, NY, USA. ACM.

Jianfu Li, Qiang Wei, Omid Ghasvand, Miao Chen, Victor Lobanov, Chunhua Weng, and Hua Xu. 2022. A comparative study of pre-trained language models for named entity recognition in clinical trial eligibility criteria from multiple corpora. BMC Med. Inform. Decis. Mak., 22(Suppl 3):235.

Xinhang Li, Hao Liu, Fabricio Kury, Chi Yuan, Alex Butler, Yingcheng Sun, Anna Ostropolets, Hua Xu, and Chunhua Weng. 2021b. A comparison between human and nlp-based annotation of clinical trial eligibility criteria text using the omop common data model. In AMIA Joint Summits on Translational Science Proceedings, pages 394–403. AMIA.

Yizhen Li, Zhongzhi Luan, Yixing Liu, Heyuan Liu, Jiaxing Qi, and Dongran Han. 2024. Automated information extraction model enhancing traditional chinese medicine rct evidence extraction (evi-bert): algorithm development and validation. frontiers artificial intelligence, 7(1454945):not listed.

Cong Liu, Hao Liu, Casey Ta, James Roger, Alex Butler, Junghwan Lee, Jaehyun Kim, Ning Shang, and Chunhua Weng. 2022. Evaluation of Criteria2Query: Towards augmented intelligence for cohort identification. Stud. Health Technol. Inform., 290:297–300.

Cong Liu, Chi Yuan, Alex M Butler, Richard D Carvajal, Ziran Ryan Li, Casey N Ta, and Chunhua Weng. 2019. Dquest: dynamic questionnaire for search of clinical trials. Journal of the American Medical Informatics Association, 26(11):1333–1343.

Hao Liu, Yuan Chi, Alex Butler, Yingcheng Sun, and Chunhua Weng. 2021. A knowledge base of clinical trial eligibility criteria. Journal of Biomedical Informatics, 117:103771.

Cynthia Lokker, Elham Bagheri, Wael Abdelkader, Rick Parrish, Muhammad Afzal, Tamara Navarro, Chris Cotoi, Federico Germiini, Lori Linkins, R Brian Haynes, Lingyang Chu, and Alfonso Iorio. 2023. Deep learning to refine the identification of high-quality clinical research articles from the biomedical literature: Performance evaluation. J. Biomed. Inform., 142(104384):104384.

Khalid Mahmood Malik, Madan Krishnamurthy, Pawel Marcinek, and Ghaus M Malik. 2020. Impact of size, location, symptomatic-nature and gender on the rupture of saccular intracranial aneurysms. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, ASONAM ’18, page 995–1001. IEEE Press.

Iain J Marshall, Benjamin Nye, Joël Kuiper, Anna Noel-Storr, Rachel Marshall, Rory Maclean, Frank Soboczenski, Ani Nenkova, James Thomas, and Byron C Wallace. 2020. Trialstreamer: A living, automatically updated database of clinical trial reports. J. Am. Med. Inform. Assoc., 27(12):1903–1912.

Iain J Marshall, Thomas A Trikalinos, Frank Soboczenski, Hye Sun Yun, Gregory Kell, Rachel Marshall, and Byron C Wallace. 2023. In a pilot study, automated real-time systematic review updates were feasible, accurate, and work-saving. J. Clin. Epidemiol., 153:26–33.

Tobias Mayer, Santiago Marro, Elena Cabrio, and Serena Villata. 2021. Enhancing evidence-based medicine with natural language argumentative analysis of clinical trials. Artificial Intelligence in Medicine, 118:102098.

Chirag Mehta, David Cohen, Priya Jaisinghani, and Payal Parikh. 2022. Internal medicine resident adherence to evidence-based practices in management of diabetes mellitus. J. Med. Educ. Curric. Dev., 9:23821205221076659.

Stéphane M. Meystre, Paul M. Heider, Andrew Cates, Grace Bastian, Tara Pittman, Stephanie Gentilin, and Teresa J. Kelechi. 2023. Piloting an automated clinical trial eligibility surveillance and provider alert system based on artificial intelligence and standard
data models. BMC Medical Research Methodology, 23(88).

Rashmi Mishra, Andrea Burke, Bonnie Gitman, Payal Verma, Mark Engelstad, Ilias Alevizos, William A. Gahl, Michael T. Collins, Janice S. Lee, and Murat Sincan. 2019. Data-driven method to enhance cranio-facial and oral phenotype vocabularies. The Journal of the American Dental Association, 150(11):933–939.e2.

Sabah Mohammed and Jinan Fiaidhi. 2023. Investigation into scaling-up the soap problem-oriented medical record into a clinical case study. In 2023 IEEE 11th International Conference. IEEE.

Sabah Mohammed and Jinan Fiaidhi. 2024. Generative AI for evidence-based medicine: A PICO GenAI for synthesizing clinical case reports. In ICC 2024 - IEEE International Conference on Communications, volume 3, pages 1503–1508. IEEE.

Sabah Mohammed, Jinan Fiaidhi, and Rahul Kudadiya. 2023. Integrating a PICO clinical questioning to the QL4POMR framework for building evidence-based clinical case reports. In 2023 IEEE International Conference on Big Data (BigData), volume 4, pages 4940–4947. IEEE.

Victor M Murcia, Vinod Aggarwal, Nikhil Pesaladinne, Ram Thammineni, Nhân Do, Gil Alterovitz, and Rafael B Fricks. 2024. Automating clinical trial matches via natural language processing of synthetic electronic health records and clinical trial eligibility criteria. AMIA Summits Transl. Sci. Proc., 2024:125–134.

Faith Mutinda, Kongmeng Liew, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2022a. PICO corpus: A publicly available corpus to support automatic data extraction from biomedical literature. In Proceedings of the first Workshop on Information Extraction from Scientific Publications, pages 26–31, Online. Association for Computational Linguistics.

Faith Wavinya Mutinda, Kongmeng Liew, Shuntaro Yada, Shoko Wakamiya, and Eiji Aramaki. 2022b. Automatic data extraction to support meta-analysis statistical analysis: a case study on breast cancer. BMC Med. Inform. Decis. Mak., 22(1):158.

Joshua J. Myszewski, Emily Klossowski, Patrick Meyer, Kristin Bevil, Lisa Klesius, and Kristopher M. Schroeder. 2022. Validating gan-biobert: A methodology for assessing reporting trends in clinical trials. Frontiers in Digital Health, 4.

Tamara Navarro-Ruan and R. Brian Haynes. 2022. Preliminary comparison of the performance of the national library of medicine’s systematic review publication type and the sensitive clinical queries filter for systematic reviews in pubmed. Journal of the Medical Library Association, 110(1).

Aurélie Névêl, Rezarta Islamaj Doğan, and Zhiyong Lu. 2011. Semi-automatic semantic annotation of pubmed queries: a study on quality, efficiency, satisfaction. Journal of biomedical informatics, 44(2):310–318.

Abigail Newbury, Hao Liu, Betina Idnay, and Chunhua Weng. 2023. The suitability of UMLS and SNOMED-CT for encoding outcome concepts. J. Am. Med. Inform. Assoc., 30(12):1895–1903.

Vincent Nguyen, Sarvnaz Karimi, and Brian Jin. 2019. An experimentation platform for precision medicine. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR’19, page 1357–1360, New York, NY, USA. Association for Computing Machinery.

Yizhao Ni, Monica Bermudez, Stephanie Kennebeck, Stacey Liddy-Hicks, and Judith Dexheimer. 2019. A real-time automated patient screening system for clinical trials eligibility in an emergency department: Design and evaluation. JMIR Medical Informatics, 7(3):e14185.

Mauro Nievas, Aditya Basu, Yanshan Wang, and Hrituraj Singh. 2024. Distilling large language models for matching patients to clinical trials. Journal of the American Medical Informatics Association, 31(9):1953–1963.

Christopher Norman, Mariska Leeflang, René Spijker, Evangelos Kanoulas, and Aurélie Névêl. 2019a. A distantly supervised dataset for automated data extraction from diagnostic studies. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 105–114, Florence, Italy. Association for Computational Linguistics.

Christopher R Norman, Mariska M G Leeflang, Raphaël Porcher, and Aurélie Névêl. 2019b. Measuring the impact of screening automation on meta-analyses of diagnostic test accuracy. Syst. Rev., 8(1):243.

Elvira Nurumbetova, Jie Pan, Zilong Zhang, Seungwon Lee, Danielle A Southern, Elliot A Martin, Guosong Wu, Chester Ho, and Cathy A Eastwood. 2023. Developing an inpatient electronic medical record phenotype for hospital-acquired pressure injuries: Case study using natural language processing models. JMIR AI, 2(2023):e41264.

Benjamin Nye, Junyi Jessy Li, Roma Patel, Yinfei Yang, Iain Marshall, Ani Nenkova, and Byron Wallace. 2018. A corpus with multi-level annotations of patients, interventions and outcomes to support language processing for medical literature. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 197–207, Melbourne, Australia. Association for Computational Linguistics.

Zhenhe Pan, Shuang Jiang, Juntao Su, Muzhe Guo, and Yuanlin Zhang. 2021. Knowledge graph based platform of COVID-19 drugs and symptoms. In Proceedings of the 2021 IEEE/ACM International Conference
on Advances in Social Networks Analysis and Mining, New York, NY, USA. ACM.

Yifan Peng, Justin F Rousseau, Edward H Shortliffe, and Chunhua Weng. 2023. AI-generated text may have a role in evidence-based medicine. Nat. Med., 29(7):1593–1594.

Sanjana Ramprasad, Iain J. Marshall, Denis Jered McInerney, and Byron C. Wallace. 2023a. Automatically summarizing evidence from clinical trials: A prototype highlighting current challenges. In Proceedings of the Conference of the Association for Computational Linguistics Meeting, pages 236–247.

Sanjana Ramprasad, Jered McInerney, Iain Marshall, and Byron Wallace. 2023b. Automatically summarizing evidence from clinical trials: A prototype highlighting current challenges. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 236–247, Stroudsburg, PA, USA. Association for Computational Linguistics.

Iqbal Ratmani, Sahar Fatima, Muhammad Mohsin Abid, Zehra Surani, and Salim Surani. 2023. Evidence-based medicine: History, review, criticisms, and pitfalls. Cureus, 15(2):e35266.

Omid Rohanian, Mohammadmahdi Nouriborji, Samaneh Kouchaki, Farhad Nooralahzadeh, Lei Clifton, and David A. Clifton. 2024. Exploring the effectiveness of instruction tuning in biomedical language processing. Artificial Intelligence in Medicine, 158:103007.

Mohammad Abu Tareq Rony, Mohammad Shariful Islam, Tipu Sultan, Samah Alshathri, and Walid El-Shafai. 2023. Medigpt: Exploring potentials of conventional and large language models on medical data. IEEE Access, 12.

Maciej Rybinski, Sarvnaz Karimi, Vincent Nguyen, and Cecile Paris. 2020a. A2A: a platform for research in biomedical literature search. BMC Bioinformatics, 21(Suppl 19):572.

Maciej Rybinski, Jerry Xu, and Sarvnaz Karimi. 2020b. Clinical trial search: Using biomedical language understanding models for re-ranking. J. Biomed. Inform., 109(103530):103530.

David L. Sackett, William M. C. Rosenberg, J. A. Muir Gray, R. Brian Haynes, and W. Scott Richardson. 1996. Evidence based medicine: what it is and what it isn’t. BMJ, 312(7023):71–72.

Jawad Sadek, Alex Inskip, James Woltmann, Georgina Wilkins, Christopher Marshall, Maria Pokora, Amey Vedpathak, Anastasija Jadrevska, Dawn Craig, and Michael Trenell. 2023. Scanmedicine: An online search system for medical innovation. Contemporary Clinical Trials, 125:107042.

Fernando Suarez Saiz, Corey Sanders, Rick Stevens, Robert Nielsen, Michael Britt, Leemor Yuravlivker, Anita M Preininger, and Gretchen P Jackson. 2021. Artificial intelligence clinical evidence engine for automatic identification, prioritization, and extraction of relevant clinical oncology research. JCO Clin. Cancer Inform., 5(5):102–111.

Hamman Samuel, Osmar Zaiane, and Francois Bolduc. 2021. Evaluation of applied machine learning for health misinformation detection via survey of medical professionals on controversial topics in pediatrics. In Proceedings of the 5th International Conference on Medical and Health Informatics, pages 1–6. ACM.

Olivia Sanchez-Graillet, Christian Witte, Frank Grimm, and Philipp Cimiano. 2022. An annotated corpus of clinical trial publications supporting schema-based relational information extraction. J. Biomed. Semantics, 13(1):14.

Abeed Sarker, Yuan-Chi Yang, Mohammed Ali Al-Garadi, and Aamir Abbas. 2020. A light-weight text summarization system for fast access to medical evidence. Front. Digit. Health, 2:585559.

Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368.

Isabel Segura-Bedmar and Pablo Raez. 2019. Cohort selection for clinical trials using deep learning models. J. Am. Med. Inform. Assoc., 26(11):1181–1188.

Makoto Shiraishi, Yoko Tomioka, Ami Miyakuni, Saaya Ishii, Asei Hori, Hwayoung Park, Jun Ohba, and Mutsumi Okazaki. 2024. Performance of ChatGPT in answering clinical questions on the practical guideline of blepharoptosis. Aesthetic Plast. Surg., 48(13):2389–2398.

Irena Spasic, David Krzeminski, Paul Corcoran, and Alexander Balinsky. 2019. Cohort selection for clinical trials from longitudinal patient records: Text mining approach. JMIR Medical Informatics, 7(4):e15980.

Nikolaos Stylianou, Gerasimos Razis, Dimitrios G. Goulis, and Ioannis Vlahavas. 2020. Ebm4: Advancing evidence-based medicine via two level automatic identification of populations, interventions, outcomes in medical literature. Artificial Intelligence in Medicine, 108:101949.

Nikolaos Stylianou and Ioannis Vlahavas. 2021. TransformED: End-to-end transformers for evidence-based medicine and argument mining in medical literature. J. Biomed. Inform., 117(103767):103767.

Davide Testa, Emmanuele Chersoni, and Alessandro Lenci. 2023. We understand elliptical sentences, and language models should too: A new dataset for studying ellipsis and its interaction with thematic fit. In
Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3340–3353, Toronto, Canada. Association for Computational Linguistics.

Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan. Elangovan, et al. 2023. Large language models in medicine. Nature Medicine, 29:1930–1940.

Shubo Tian, Arslan Erdengasileng, Xi Yang, Yi Guo, Yonghui Wu, Jinfeng Zhang, Jiang Bian, and Zhe He. 2021. Transformer-based named entity recognition for parsing clinical trial eligibility criteria. ACM BCB, 2021.

Shubo Tian, Pengfei Yin, Hansi Zhang, Arslan Erdengasileng, Jiang Bian, and Zhe He. 2023. Parsing clinical trial eligibility criteria for cohort query by a multi-input multi-output sequence labeling model. In 2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), pages 4426–4430.

Hegler C. Tissot, Anoop D. Shah, David Brealey, Steve Harris, Ruth Agbakoba, and Anos Folarin. 2020. Natural language processing for mimicking clinical trial recruitment in critical care: A semi-automated simulation based on the leopards trial. IEEE Journal of Biomedical and Health Informatics, 24(10):2950–2959.

Tadashi Tsubota, Danushka Bollegala, Yang Zhao, Yingzi Jin, and Tomotake Kozu. 2022. Improvement of intervention information detection for automated clinical literature screening during systematic review. J. Biomed. Inform., 134(104185):104185.

Pyae Phyto Tun, Jiawen Luo, Jiecheng Xie, Sandi Wibowo, and Chen Hao. 2023. Automatic assessment of patient eligibility by utilizing nlp and rule-based analysis. In 2023 45th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), page 10340494, Sydney, Australia. IEEE.

Ali Turfaha, Hao Liu, Latoya A Stewart, Tian Kang, and Chunhua Weng. 2022. Extending pico with observation normalization for evidence computing. In MEDINFO 2021: One World, One Health – Global Partnership for Digital Innovation, pages 268–272, New York, New York, USA. International Medical Informatics Association and IOS Press, IOS Press.

Ozan Unlu, Jiyeon Shin, Charlotte J Mailly, Michael F Oates, Michela R Tucci, Matthew Varughese, Kavishwar Wagholikar, Fei Wang, Benjamin M Scirica, Alexander J Blood, and Samuel J Aronson. 2024. Retrieval augmented generation enabled generative pre-trained transformer 4 (GPT-4) performance for clinical trial screening. medRxiv.

Peter Van de Vliet, Tobias Sprenger, Linde FC Kampers, Jennifer Makalowski, Volker Schirrmacher, Wilfried Stücker, and Stefaan W Van Gool. 2023. The application of evidence-based medicine in individualized medicine. Biomedicines, 11(7).

Bianca Vora, Denison Kuruvilla, Chloe Kim, Michael Wu, Colby S. Shemesh, and Gillie A. Roth. 2023. Applying natural language processing to clinicaltrials.gov: mRNA cancer vaccine case study. Clinical and Translational Science, 16:2417–2420.

V G Vinod Vydiswaran, Asher Strayhorn, Xinyan Zhao, Phil Robinson, Mahesh Agarwal, Erin Bagazinski, Madia Essiet, Bradley E Iott, Hyeon Joo, Pingjiu Ko, Dahee Lee, Jin Xiu Lu, Jinghui Liu, Adharsh Murali, Koki Sasagawa, Tianshi Wang, and Nalingna Yuan. 2019. Hybrid bag of approaches to characterize selection criteria for cohort identification. J. Am. Med. Inform. Assoc., 26(11):1172–1180.

Kunyuan Wang, Hao Cui, Yun Zhu, Xiaoyun Hu, Chang Hong, Yabing Guo, Lingyao An, Qi Zhang, and Li Liu. 2024. Evaluation of an artificial intelligence-based clinical trial matching system in chinese patients with hepatocellular carcinoma: a retrospective study. BMC Cancer, 24(1):246.

Yu Wang, Yuan Wang, Zhenwan Peng, Feifan Zhang, Luyao Zhou, and Fei Yang. 2023a. Medical text classification based on the discriminative pre-training model and prompt-tuning. Digit. Health, 9:20552076231193213.

Zifeng Wang and Jimeng Sun. 2022. Trial2Vec: Zero-shot clinical trial document similarity search using self-supervision. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6377–6390, Stroudsburg, PA, USA. Association for Computational Linguistics.

Zifeng Wang, Cao Xiao, and Jimeng Sun. 2023b. AutoTrial: Prompting language models for clinical trial design. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12461–12472, Singapore. Association for Computational Linguistics.

Christian Witte, David M Schmidt, and Philipp Cimiano. 2024. Comparing generative and extractive approaches to information extraction from abstracts describing randomized clinical trials. J. Biomed. Semantics, 15(1):3.

Qianqian Xie, Jennifer Amy Bishop, Prayag Tiwari, and Sophia Ananiadou. 2022. Pre-trained language models with domain knowledge for biomedical extractive summarization. Knowl. Based Syst., 252(109460):109460.

Shiyao Xie, Wenjing Zhao, Guanghui Deng, Guohua He, Na He, Zhenhua Lu, Weihua Hu, Mingming Zhao, and Jian Du. 2024. Utilizing chatgpt as a scientific reasoning engine to differentiate conflicting evidence and summarize challenges in controversial clinical questions. Journal of the American Medical Informatics Association, 31(7):1551–1560. Published online: 17 May 2024.

Yi Xie, Ishith Seth, David J Hunter-Smith, Warren M Rozen, Richard Ross, and Matthew Lee. 2023. Aesthetic surgery advice and counseling from artificial
intelligence: A rhinoplasty consultation with Chat-GPT. Aesthetic Plast. Surg., 47(5):1985–1993.

Quan Xu, Yueyue Liu, Dawei Sun, Xiaoqian Huang, Feihong Li, Jincheng Zhai, Yang Li, Qiming Zhou, Niansong Qian, and Beifang Niu. 2023. OncoCT-Miner: streamlining precision oncology trial matching via molecular profile analysis. Database (Oxford), 2023:baad077.

Yumeng Yang, Soumya Jayaraj, Ethan Ludmir, and Kirk Roberts. 2023. Text classification of cancer clinical trial eligibility criteria. AMIA Annu. Symp. Proc., 2023:1304–1313.

Xiaoxi Yao, Zach I. Attia, Emma M. Behnken, Kelli Walvatne, Rachel E. Giblon, Sijia Liu, Konstantinos C. Siontis, Bernard J. Gersh, Jonathan Graff-Radford, Alejandro A. Rabinstein, Paul A. Friedman, and Peter A. Noseworthy. 2021. Batch enrollment for an artificial intelligence-guided intervention to lower neurologic events in patients with undiagnosed atrial fibrillation: rationale and design of a digital clinical trial. American Heart Journal, 239:73–79.

Fatin Syafiqah Yazi, Wan-Tze Vong, Valliappan Ramam, Patrick Hang Hui Then, and Mukulraj J Lunia. 2021. Towards automated detection of contradictory research claims in medical literature using deep learning approach. In 2021 Fifth International Conference on Information Retrieval and Knowledge Management (CAMP), pages 116–121.

Jiayi Yuan, Ruixiang Tang, Xiaoqian Jiang, and Xia Hu. 2024. Large language models for healthcare data augmentation: An example on patient-trial matching. In AMIA Annual Symposium Proceedings, volume 2024, pages 1324–1333. AMIA.

Kun Zeng, Zhiwei Pan, Yibin Xu, and Yingying Qu. 2020. An ensemble learning strategy for eligibility criteria text classification for clinical trial recruitment: Algorithm development and validation. JMIR Medical Informatics, 8(7):e17832.

Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G. Nestor, Matthew E. Spotnitz, Ali Soroush, Thomas R. Campion Jr, Zhiyong Lu, Chunhua Weng, and Yifan Peng. 2024a. Closing the gap between open source and commercial large language models for medical evidence summarization. NPJ Digital Medicine, 7(1):239.

Gongbo Zhang, Yiliang Zhou, Yan Hu, Hua Xu, Chunhua Weng, and Yifan Peng. 2024b. A span-based model for extracting overlapping PICO entities from RCT publications. J. Am. Med. Inform. Assoc.

Yixuan Zhang, Junzhen Liu, and Wei Lu. 2023. Medict-gp: An accurate entity recognition model combining medical domain knowledge and globalization ideas. In Proceedings of the 2023 9th International Conference on Computing and Artificial Intelligence, ICCAI ’23, page 477–483, New York, NY, USA. Association for Computing Machinery.

Ce Zheng, Hongfei Ye, Jinming Guo, Junrui Yang, Ping Fei, Yuanzhi Yuan, Danqing Huang, Yuqiang Huang, Jie Peng, Xiaoling Xie, Meng Xie, Peiquan Zhao, Li Chen, and Mingzhi Zhang. 2024. Development and evaluation of a large language model of ophthalmology in chinese. Br. J. Ophthalmol., 108(10):1390–1397.

A Appendix
A.1 Included Studies

Supplementary Table 1: Overview of Included Studies. P - Precision, R - Recall, Acc - Accuracy, NDCG - Normalized Discounted Cumulative Gain.

(S) - Abstractive Summarization, (T) - Clinical Trial Design, (N) - Entity Extraction and Classification, (E) - Evaluation of Performance, (I) - Evidence Ranking and Screening, (S) - Extractive Summarization, (Y) - Evidence Synthesis, (R) - Information Retrieval, (Q) - Quality Assessment, (A) - Question Answering, (R) - Relation Extraction.

<table>
  <tr>
    <th>Study</th>
    <th>Model</th>
    <th>Disease</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>Al Hafiz Khan et al. (2019)</td>
    <td>RNN</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Alexander et al. (2020)</td>
    <td>Statistical</td>
    <td>Lung Cancer</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Alodadi and Janeja (2019)</td>
    <td>Rules</td>
    <td>–</td>
    <td>N E I R</td>
  </tr>
  <tr>
    <td>Aum and Choe (2021)</td>
    <td>Transformer</td>
    <td>Cognitive Impairment</td>
    <td>N E I R</td>
  </tr>
  <tr>
    <td>Beattie et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Beck et al. (2020)</td>
    <td>Rules</td>
    <td>Breast Cancer</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Borchert et al. (2022)</td>
    <td>Rules</td>
    <td>Cancer</td>
    <td>N R</td>
  </tr>
  <tr>
    <td>Brassey et al. (2021)</td>
    <td>Rules</td>
    <td>–</td>
    <td>N Y A</td>
  </tr>
  <tr>
    <td>Brockmeier et al. (2019)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N L</td>
  </tr>
  <tr>
    <td>Cai et al. (2021)</td>
    <td>Random Forest, Logistic LASSO</td>
    <td>Rheumatoid Arthritis</td>
    <td>T E</td>
  </tr>
  <tr>
    <td>Campillos-Llanos et al. (2021)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Chen et al. (2019a)</td>
    <td>Rules</td>
    <td>–</td>
    <td>S T N E R</td>
  </tr>
  <tr>
    <td>Chen et al. (2019b)</td>
    <td>CNN</td>
    <td>Myocardial infarction</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Chen et al. (2019c)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Chuan and Morgan (2021)</td>
    <td>CNN</td>
    <td>Cancer</td>
    <td>T N E Q</td>
  </tr>
  <tr>
    <td>Cunningham et al. (2024)</td>
    <td>Transformer</td>
    <td>Heart Failure</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Daluwatumulle et al. (2022)</td>
    <td>Graph</td>
    <td>Alzheimer’s Disease (AD)</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Datta et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>N E I R</td>
  </tr>
  <tr>
    <td>Devi et al. (2024b)</td>
    <td>LLM</td>
    <td>Non-Small Cell Lung Cancer (NSCLC)</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>DeYoung et al. (2021)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>S N E R</td>
  </tr>
  <tr>
    <td>Deng et al. (2019)</td>
    <td>Graph, Statistical</td>
    <td>Coronary heart disease, Chest pain, Bronchitis</td>
    <td>E Q</td>
  </tr>
  <tr>
    <td>Do et al. (2024)</td>
    <td>Rules, Statistical</td>
    <td>Cancer</td>
    <td>T N</td>
  </tr>
  <tr>
    <td>Dobbins et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>T N E R</td>
  </tr>
  <tr>
    <td>Dobbins et al. (2023)</td>
    <td>Transformer, Rules</td>
    <td>–</td>
    <td>T N E I R</td>
  </tr>
  <tr>
    <td>Dhrangadhariya et al. (2020)</td>
    <td>Statistical, Learning to Rank</td>
    <td>–</td>
    <td>N E Y A</td>
  </tr>
  <tr>
    <td>Dhrangadhariya and Müller (2023)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Dhrangadhariya et al. (2024)</td>
    <td>Rules, Statistical</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Dhayne et al. (2021)</td>
    <td>SVM, CNN, RNN, Transformer</td>
    <td>Viral Infection</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Du et al. (2021)</td>
    <td>Graph</td>
    <td>COVID-19</td>
    <td>E I</td>
  </tr>
  <tr>
    <td>Fang et al. (2021)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T N</td>
  </tr>
  <tr>
    <td>Gates and Hamed (2020)</td>
    <td>Graph, Statistical</td>
    <td>SARS</td>
    <td>N L</td>
  </tr>
  <tr>
    <td>Ghosh et al. (2024a)</td>
    <td>LLM</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Ghosh et al. (2024b)</td>
    <td>Transformer, RNN</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Górnska and Tacconelli (2024)</td>
    <td>Transformer, LLM</td>
    <td>–</td>
    <td>N L Y</td>
  </tr>
  <tr>
    <td>Gulden et al. (2019)</td>
    <td>Graph</td>
    <td>–</td>
    <td>N E S</td>
  </tr>
  <tr>
    <td>Gwon et al. (2024)</td>
    <td>LLM</td>
    <td>Peyronie Disease</td>
    <td>N E I A</td>
  </tr>
  <tr>
    <td>Hamed et al. (2023)</td>
    <td>LLM</td>
    <td>Diabetic Ketoacidosis</td>
    <td>S B Q</td>
  </tr>
  <tr>
    <td>Hassanzadeh et al. (2020)</td>
    <td>Support Vector Machine (SVM), Random Forest (RF), Logistic Regression (LR), and Stochastic Gradient Descent (SGD), RNN</td>
    <td>–</td>
    <td>T N E I</td>
  </tr>
  <tr>
    <td>Horst et al. (2023)</td>
    <td>Graph, CRF, Statistical</td>
    <td>Spinal Cord Injury</td>
    <td>T N E Y</td>
  </tr>
  <tr>
    <td>Hu et al. (2023)</td>
    <td>Transformer</td>
    <td>COVID-19, AD</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Huang et al. (2024)</td>
    <td>LLM</td>
    <td>Glaucoma</td>
    <td>E Q</td>
  </tr>
  <tr>
    <td>Jang et al. (2022)</td>
    <td>Transformer</td>
    <td>Recurrent Glioblastoma</td>
    <td>N E Q</td>
  </tr>
  <tr>
    <td>Jiang et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>S E Q</td>
  </tr>
</table>

Continued on next page
<table>
  <tr>
    <th>Study</th>
    <th>Model</th>
    <th>Disease</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>Jin et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>E I A</td>
  </tr>
  <tr>
    <td>Jin et al. (2024)</td>
    <td>Transformer, LLM</td>
    <td>–</td>
    <td>T E I I</td>
  </tr>
  <tr>
    <td>Johnston et al. (2024)</td>
    <td>Vector Space Model, Statistical</td>
    <td>I-DOPA-induced dyskinesia in Parkinson</td>
    <td>N L</td>
  </tr>
  <tr>
    <td>Kamath et al. (2021)</td>
    <td>Statistical</td>
    <td>–</td>
    <td>T I I</td>
  </tr>
  <tr>
    <td>Kanbar et al. (2022)</td>
    <td>SVM, Naive Bayes, Random Forest</td>
    <td>Epilepsy</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Kang et al. (2019)</td>
    <td>RNN</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Kang et al. (2021)</td>
    <td>Transformer, RNN</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Kang et al. (2023)</td>
    <td>Transformer</td>
    <td>COVID-19</td>
    <td>N E P Y</td>
  </tr>
  <tr>
    <td>Kaskovich et al. (2023)</td>
    <td>SVM</td>
    <td>Pediatric Leukemia</td>
    <td>T N E L</td>
  </tr>
  <tr>
    <td>Kefeli and Tatonetti (2024)</td>
    <td>BERT</td>
    <td>Cancer</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Khan et al. (2021)</td>
    <td>Statistical</td>
    <td>Chronic Rhinosinusitis with Nasal Polyps</td>
    <td>N</td>
  </tr>
  <tr>
    <td>Kim et al. (2023a)</td>
    <td>L.R, NB, kNN, SVM, CNN, RNN, FastText, Transformer, ERNIE</td>
    <td>Hepatocellular Carcinoma</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Kim et al. (2023b)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Koopman et al. (2021)</td>
    <td>Graph</td>
    <td>Cancer</td>
    <td>N I</td>
  </tr>
  <tr>
    <td>Koopman and Zuccon (2022)</td>
    <td>MMR</td>
    <td>Cancer</td>
    <td>T L I</td>
  </tr>
  <tr>
    <td>Kury et al. (2020)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T E</td>
  </tr>
  <tr>
    <td>Lalitha et al. (2023)</td>
    <td>T5(Text-to-Text Transfer Transformer), BART (Bidirectional Auto-Regressive Transformer) and PEGASUS (Pre-training with Extracted Gap-sentences for Abstractive Summarization Sequence-to-sequence)</td>
    <td>–</td>
    <td>S E</td>
  </tr>
  <tr>
    <td>Lan et al. (2024)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E A</td>
  </tr>
  <tr>
    <td>Lee et al. (2024)</td>
    <td>RNN</td>
    <td>Cancer</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Li et al. (2021a)</td>
    <td>Vector Space model</td>
    <td>–</td>
    <td>I I</td>
  </tr>
  <tr>
    <td>Li et al. (2021b)</td>
    <td>Rules</td>
    <td>–</td>
    <td>N</td>
  </tr>
  <tr>
    <td>Li et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Li et al. (2024)</td>
    <td>Transformer</td>
    <td>Stroke, Colorectal Cancer, Coronary Heart Disease, Heart Failure, Chronic Obstructive Pulmonary Disease, Diabetes, Diabetic Nephropathy, Osteoarthritis, Obesity, Rheumatoid Arthritis, and Diarrhea</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Liu et al. (2019)</td>
    <td>RNN</td>
    <td>–</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Liu et al. (2021)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T N E R</td>
  </tr>
  <tr>
    <td>Liu et al. (2022)</td>
    <td>Rules, RNN</td>
    <td>–</td>
    <td>T N R</td>
  </tr>
  <tr>
    <td>Lokker et al. (2023)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>F I A</td>
  </tr>
  <tr>
    <td>Malik et al. (2020)</td>
    <td>Statistical</td>
    <td>–</td>
    <td>N</td>
  </tr>
  <tr>
    <td>Marshall et al. (2023)</td>
    <td>Transformer</td>
    <td>COVID-19</td>
    <td>N Y I</td>
  </tr>
  <tr>
    <td>Mayer et al. (2021)</td>
    <td>GRU, CRF, RNN, Transformer</td>
    <td>–</td>
    <td>N E R</td>
  </tr>
  <tr>
    <td>Meystre et al. (2023)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T N I</td>
  </tr>
  <tr>
    <td>Mishra et al. (2019)</td>
    <td>Statistical</td>
    <td>Craniofacial Abnormalities</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Mohammed and Fiaidhi (2023)</td>
    <td>Graph</td>
    <td>–</td>
    <td>N I</td>
  </tr>
  <tr>
    <td>Mohammed et al. (2023)</td>
    <td>Graph</td>
    <td>–</td>
    <td>N E S</td>
  </tr>
  <tr>
    <td>Mohammed and Fiaidhi (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>N E Y Q</td>
  </tr>
  <tr>
    <td>Murcia et al. (2024)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>T N</td>
  </tr>
  <tr>
    <td>Mutinda et al. (2022b)</td>
    <td>Rules, Statistical, Transformer</td>
    <td>Breast cancer</td>
    <td>N E P Y</td>
  </tr>
  <tr>
    <td>Myszewski et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E A</td>
  </tr>
  <tr>
    <td>Navarro-Ruan and Haynes (2022)</td>
    <td>Rules, Statistical</td>
    <td>–</td>
    <td>L I</td>
  </tr>
  <tr>
    <td>Newbury et al. (2023)</td>
    <td>Rules</td>
    <td>–</td>
    <td>N E</td>
  </tr>
</table>
<table>
  <tr>
    <th>Study</th>
    <th>Model</th>
    <th>Disease</th>
    <th>Task</th>
  </tr>
  <tr>
    <td>Nguyen et al. (2019)</td>
    <td>Transformer, RNN, SVM</td>
    <td>–</td>
    <td>Q</td>
  </tr>
  <tr>
    <td>Ni et al. (2019)</td>
    <td>Rules</td>
    <td>Respiratory Tract Infection, Traumatic Brain Injury, and Serious Bacterial Infections</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Nievas et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>T E</td>
  </tr>
  <tr>
    <td>Norman et al. (2019b)</td>
    <td>Statistical</td>
    <td>–</td>
    <td>L</td>
  </tr>
  <tr>
    <td>Nurmambetova et al. (2023)</td>
    <td>Random Forest, XGBoost</td>
    <td>Acquired Pressure Injuries</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Pan et al. (2021)</td>
    <td>Transformer, Graph</td>
    <td>COVID-19</td>
    <td>N E S Q R</td>
  </tr>
  <tr>
    <td>Ramprasad et al. (2023a)</td>
    <td>Transformer, Longformer</td>
    <td>–</td>
    <td>S I</td>
  </tr>
  <tr>
    <td>Rony et al. (2023)</td>
    <td>LLM</td>
    <td>–</td>
    <td>E Q</td>
  </tr>
  <tr>
    <td>Rybinski et al. (2020a)</td>
    <td>Rules</td>
    <td>–</td>
    <td>I</td>
  </tr>
  <tr>
    <td>Rybinski et al. (2020b)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E I T</td>
  </tr>
  <tr>
    <td>Sadek et al. (2023)</td>
    <td>Knowledge</td>
    <td>–</td>
    <td>N I</td>
  </tr>
  <tr>
    <td>Saiz et al. (2021)</td>
    <td>Gradient-boosted Trees</td>
    <td>Cancer</td>
    <td>T N E L</td>
  </tr>
  <tr>
    <td>Samuel et al. (2021)</td>
    <td>Statistical</td>
    <td>Autism</td>
    <td>E I</td>
  </tr>
  <tr>
    <td>Sanchez-Graillet et al. (2022)</td>
    <td>Transformer, Schema</td>
    <td>Glaucoma, Type 2 diabetes mellitus</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Sarker et al. (2020)</td>
    <td>Statistical</td>
    <td>COVID-19</td>
    <td>E S</td>
  </tr>
  <tr>
    <td>Segura-Bedmar and Raez (2019)</td>
    <td>CNN, RNN</td>
    <td>–</td>
    <td>T E</td>
  </tr>
  <tr>
    <td>Shiraishi et al. (2024)</td>
    <td>LLM</td>
    <td>Blepharoptosis</td>
    <td>E Q</td>
  </tr>
  <tr>
    <td>Spasic et al. (2019)</td>
    <td>SVM, Logistic Regression, Naive Bayes, Gradient Tree Boosting, Rules, Decision Trees, Random Forests</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Stylianou et al. (2020)</td>
    <td>Transformer, RNN</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Stylianou and Vlahavas (2021)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Tian et al. (2021)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>T N A</td>
  </tr>
  <tr>
    <td>Tian et al. (2023)</td>
    <td>Statistical, Transformer</td>
    <td>AD</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Tissot et al. (2020)</td>
    <td>Rules</td>
    <td>Organ dysfunction in septic shock</td>
    <td>T N E I</td>
  </tr>
  <tr>
    <td>Tsubota et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Tun et al. (2023)</td>
    <td>Rules</td>
    <td>Cardiovascular Events</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Turfaha et al. (2022)</td>
    <td>Rules</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Unlu et al. (2024)</td>
    <td>LLM</td>
    <td>Heart Failure</td>
    <td>E Q</td>
  </tr>
  <tr>
    <td>Vora et al. (2023)</td>
    <td>Rules</td>
    <td>mRNA Cancer</td>
    <td>N I</td>
  </tr>
  <tr>
    <td>Vydiswaran et al. (2019)</td>
    <td>Rules</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Wang and Sun (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Wang et al. (2023a)</td>
    <td>Statistical, Transformer</td>
    <td>–</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Wang et al. (2023b)</td>
    <td>LLM</td>
    <td>–</td>
    <td>T E Y R</td>
  </tr>
  <tr>
    <td>Wang et al. (2024)</td>
    <td>CNN, Graph, Rules</td>
    <td>Hepatocellular Carcinoma</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Witte et al. (2024)</td>
    <td>Transformer</td>
    <td>Glaucoma, Type II Diabetes</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Xie et al. (2022)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N S</td>
  </tr>
  <tr>
    <td>Xie et al. (2023)</td>
    <td>LLM</td>
    <td>Rhinoplasty</td>
    <td>Q</td>
  </tr>
  <tr>
    <td>Xie et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>S E Y A</td>
  </tr>
  <tr>
    <td>Xu et al. (2023)</td>
    <td>Graph</td>
    <td>Cancer</td>
    <td>N L A</td>
  </tr>
  <tr>
    <td>Yang et al. (2023)</td>
    <td>Transformer</td>
    <td>Cancer</td>
    <td>T N</td>
  </tr>
  <tr>
    <td>Yao et al. (2021)</td>
    <td>Not Specify</td>
    <td>Atrial Fibrillation</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Yazi et al. (2021)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>E A</td>
  </tr>
  <tr>
    <td>Yuan et al. (2024)</td>
    <td>LLM</td>
    <td>–</td>
    <td>T N E</td>
  </tr>
  <tr>
    <td>Zeng et al. (2020)</td>
    <td>Transformer</td>
    <td>–</td>
    <td>N A</td>
  </tr>
  <tr>
    <td>Zhang et al. (2023)</td>
    <td>Transformer, RNN</td>
    <td>–</td>
    <td>N E</td>
  </tr>
  <tr>
    <td>Zhang et al. (2024b)</td>
    <td>Transformer</td>
    <td>COVID-19, AD</td>
    <td>N E I</td>
  </tr>
  <tr>
    <td>Zheng et al. (2024)</td>
    <td>LLM</td>
    <td>Glaucoma</td>
    <td>E Q</td>
  </tr>
</table>
A.2 Benchmark Dataset

Supplementary Table 2: Overview of recent benchmark datasets. P - Population, I - Intervention, C - Comparison, O - Outcome (Eldawlatly et al., 2018). RCT - Randomized controlled trial. ETC - Anything that doesn’t fit into the categories above. GPG - GNU Privacy Guard. CMS - Content Management System.

<table>
  <tr>
    <th>Dataset</th>
    <th>Avail.</th>
    <th>Label</th>
    <th>Annotation</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Alzheimer’s disease RCT (<a href="https://doi.org/10.1093/bioinformatics/btac476">Hu et al., 2023</a>)<br>Chia (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Kury et al., 2020</a>)</td>
    <td>Public</td>
    <td>P, I, C, O<br>Non-query-able, Post-eligibility, Informed consent, Pregnancy considerations, Parsing error, Non-representable, Competing trial, Context error, Subjective judgment, Not a criteria, Undefined semantics, Intoxication considerations</td>
    <td>Manual</td>
    <td>150 Alzheimer’s disease RCT abstracts<br>Eligibility statements from 1000 clinical trials and the dataset includes 12,409 annotated eligibility criteria</td>
  </tr>
  <tr>
    <td>Clinical trials on cancer (<a href="https://doi.org/10.1093/bioinformatics/btac476">Rony et al., 2023</a>)</td>
    <td>Private</td>
    <td>Eligible, Not eligible</td>
    <td>Manual</td>
    <td>6M eligibility statements in clinical trials</td>
  </tr>
  <tr>
    <td>COVID-19 corpus (<a href="https://doi.org/10.1093/bioinformatics/btac476">Hu et al., 2023</a>)</td>
    <td>Public</td>
    <td>P, I, C, O</td>
    <td>Manual</td>
    <td>150 COVID-19 RCT abstracts</td>
  </tr>
  <tr>
    <td>CT-EBM-SP (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Campillos Llanos et al., 2021</a>)</td>
    <td>Public</td>
    <td>Anatomy, pharmacological and chemical substances, pathologies, and lab tests, diagnostic or therapeutic procedures</td>
    <td>Manual</td>
    <td>1,200 texts about clinical trials with entities</td>
  </tr>
  <tr>
    <td>EBM-COMET (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Ghosh et al., 2024b</a>)</td>
    <td>Public</td>
    <td>Physiological or clinical, Death, Life impact, Resource use, Adverse events</td>
    <td>Manual</td>
    <td>300 RCT abstracts</td>
  </tr>
  <tr>
    <td>EBM-NLP (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Nye et al., 2018</a>)</td>
    <td>Public</td>
    <td>P, I, O</td>
    <td>Manual</td>
    <td>4,993 medical abstracts from literatures on PubMed</td>
  </tr>
  <tr>
    <td>EliIE (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Testa et al., 2023</a>)</td>
    <td>Public</td>
    <td>Condition, observation, drug/substance, and procedure or device</td>
    <td>Manual</td>
    <td>230 Alzheimer’s disease RCT documents</td>
  </tr>
  <tr>
    <td>GGPONC (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Borchert et al., 2020</a>)</td>
    <td>Public</td>
    <td>Recommendation creation date, Type of recommendation, Recommendation grade, Strength of consensus, Total vote in percentage, Literature references, Expert opinion, Level of evidence, Edit State</td>
    <td>Automatic</td>
    <td>25 GPGs with 8,414 text segments from the CMS</td>
  </tr>
  <tr>
    <td>LCT (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Dobbins et al., 2022</a>)</td>
    <td>Public</td>
    <td>Clinical, Demographic, Logical, Qualifiers, Temporal and Comparative, Other</td>
    <td>Manual</td>
    <td>1,000+ eligibility statements</td>
  </tr>
  <tr>
    <td>Limsi-Cochrane dataset (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Norman et al., 2019a</a>)</td>
    <td>Public</td>
    <td>Systematic reviews, Included studies, Data forms, Text entries, Excluded studies, Diagnostic tests, Test results, Study IDs, Numerical, Summary scores</td>
    <td>Manual</td>
    <td>1,939 meta-analyses from 63 systematic reviews of diagnostic test accuracy from the Cochrane Library</td>
  </tr>
  <tr>
    <td>MedReview (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Zhang et al., 2024a</a>)</td>
    <td>Private</td>
    <td>medical related topics (e.g. Wounds, Urology)</td>
    <td>Manual</td>
    <td>Meta-analysis results and narrative summaries from the Cochrane Library</td>
  </tr>
  <tr>
    <td>MS^2 (<a href="https://doi.org/10.1093/bioinformatics/btaa522">DeYoung et al., 2021</a>)</td>
    <td>Public</td>
    <td>Background, Goal, Methods, Detailed findings, Further study, Recommendation, Evidence quality, effect, ETC</td>
    <td>Manual</td>
    <td>470k documents and 20K summaries from the scientific literature</td>
  </tr>
  <tr>
    <td>NICTA-PIBOSO (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Kim et al., 2024</a>)</td>
    <td>Public</td>
    <td>P, I, O, Background, Study Design, Other</td>
    <td>Manual</td>
    <td>1,000 biomedical abstracts</td>
  </tr>
  <tr>
    <td>PICO-Corpus (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Mutinda et al., 2022a</a>)</td>
    <td>Public</td>
    <td>P, I, C, O</td>
    <td>Manual</td>
    <td>1,011 breast cancer RCT abstracts</td>
  </tr>
  <tr>
    <td>RedHOT (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Ghosh et al., 2024b</a>)</td>
    <td>Public</td>
    <td>P, I, O</td>
    <td>Manual</td>
    <td>22,000 social media posts from Reddit spanning 24 health conditions</td>
  </tr>
  <tr>
    <td>Illness dataset (<a href="https://doi.org/10.1093/bioinformatics/btaa522">Rony et al., 2023</a>)</td>
    <td>Private</td>
    <td>Alzheimer’s, Parkinson’s, Cancer, and Diabetes domains</td>
    <td>Manual</td>
    <td>22,660 tweets</td>
  </tr>
</table>
<table>
  <tr>
    <th>Dataset</th>
    <th>Avail.</th>
    <th>Label</th>
    <th>Annotation</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>Symptom2Disease dataset (<a href="https://doi.org/10.48550/OSF.IO/6JY7S">Rony et al., 2023</a>)</td>
    <td>Private</td>
    <td>24 diseases, each described by 50 symptom profiles</td>
    <td>Manual</td>
    <td>1,200 data points</td>
  </tr>
  <tr>
    <td>Trialstreamer (<a href="https://doi.org/10.1093/bioinformatics/btaa102">Marshall et al., 2020</a>)</td>
    <td>Public</td>
    <td>P, I, O, RCT classifiers</td>
    <td>Manual</td>
    <td>191 RCT publications</td>
  </tr>
</table>
A.3 Queries

{nlp_keywords} = natural language processing
    OR nlp
    OR language model
    OR large language model
    OR llm
    OR computational linguistics
    OR information extraction
    OR information retrieval
    OR clinical trial retrieval
    OR text summarization
    OR question answering
    OR sentence segmentation
    OR ner
    OR named entity recognition
    OR tokenization)
{ebm_keywords} = evidence-based medicine
    OR ebm
    OR evidence-based practice
    OR ebp
    OR clinical trial

PubMed
({{nlp_keywords}[Title/Abstract]) AND ({ebm_keywords}[Title/Abstract]))

IEEE Xplore
(("Abstract":{nlp_keywords}) AND ("Abstract":{ebm_keywords}))
    OR (("Title":{nlp_keywords}) AND (("Title": {ebm_keywords})))

ACM
(Abstract:{nlp_keywords} AND Abstract:{ebm_keywords})
    OR (Title:{nlp_keywords} AND Title:{ebm_keywords})

ACL
(Abstract:{nlp_keywords} AND Abstract:{ebm_keywords})
    OR (Title:{nlp_keywords} AND Title:{ebm_keywords})

Reviewer paper summaries: This paper seeks to fill the gap by offering a thorough review of essential NLP tasks in EBM, with a focus on evidence generation, such as evidence retrieval, extraction, synthesis, and summarization, as well as evidence adoption and evidence-based research, such as question-answering, clinical trial design and identification, and other cutting-edge studies across various clinical specialties.

The paper presents a comprehensive review of 129 research studies exploring the application of Natural Language Processing (NLP) techniques in Evidence-based Medicine (EBM). It systematically outlines NLP's role across the five core EBM steps: Ask, Acquire, Appraise, Apply, and Assess, emphasizing methods such as information retrieval, entity and relation extraction, evidence ranking, synthesis, and summarization. The review highlights the transition from traditional machine learning and heuristic methods to advanced transformer-based models and large language models, significantly improving accuracy and efficiency in handling medical literature. Despite substantial progress, the paper identifies ongoing challenges, including the need for better benchmark datasets specific to medical applications and addressing model interpretability, biases, and resource-intensive processes. Finally, the authors propose future research directions such as few-shot learning for rare diseases, integration of real-world data sources, and expansion of NLP applications across diverse medical specialties to further enhance clinical decision-making processes.
Summary of strengths: The review paper identifies current limitations within the field and proposes directions for future research. 
It emphasizes the potential for NLP to revolutionize EBM by refining evidence extraction, evidence synthesis, appraisal, summarization, enhancing data comprehensibility, and facilitating a more efficient clinical workflow.

1. The paper reviews 129 relevant research studies, providing extensive coverage of Natural Language Processing (NLP) applications across all stages of the Evidence-based Medicine (EBM) lifecycle, ensuring a holistic presentation of the research papers to the readers.

3. The authors follow the PRISMA guidelines for literature selection and evaluation, thus enhancing the transparency and reproducibility of the review process. They effectively map NLP techniques to the five core EBM steps—Ask, Acquire, Appraise, Apply, and Assess—facilitating clarity and ease of reference for readers seeking information about specific NLP applications.

3. Beyond summarizing existing research, the paper thoughtfully highlights limitations and clearly identifies areas needing further exploration.

4. Table 2 and Table 3 can be really useful to quickly lookup relevant research and choose related works according to requirements (data, model) for the EBM  research problems.
Summary of weaknesses: The scope of this review is limited to NLP in the bio-medical domain.

1. While the paper systematically covers numerous studies, it primarily provides very high-level summaries rather than critically analyzing the comparative strengths, weaknesses, or practical effectiveness of different NLP techniques, limiting its usefulness for readers seeking detailed methodological insights. Descriptions in the main body of the paper (sections 4 - 7) often emphasize broad overviews of methods without clearly illustrating practical scenarios or detailed examples, reducing the applicability of the findings for practitioners who aim to implement NLP techniques in clinical contexts. The readers might want to know how the different methods that are presented within these sections perform on the tasks that are discussed. Hence the descriptions need more structure in terms of comparative analysis and the application scenarios.

2. The paper does not thoroughly address the methodological challenges, such as data quality issues, biases in NLP models, and interpretability problems, which are essential considerations when translating NLP techniques into clinical practice.
Comments, suggestions and typos: The challenge of adopting EBM from the standpoint of clinical medics also deserves to be included in the discussion.

1. Figure 2 presents the distribution of papers in different EBM tasks over time. This is a very interesting figure but does not get much attention in the main paper. Some analysis regarding why some topics are gaining more and more traction over time would be really interesting. Such as, `relation extraction` related papers were fairly common until 2024 but in 2024 there were little to no papers on this topic, why?, `question answering` related papers have experienced a surge in 2024, could this be due to the advancements of LLMs?

2. I would want to see more attention on generative AI related papers. A different section on how generative AI based papers are revolutionizing the EBM could be very ornamental to the paper.
Overall assessment scores: 3.5

3.5
Metareview score: 3
Metareview: The paper is a survey paper on the application of NLP techniques in Evidence-based Medicine (over 129 research studies). It outlines the role of NLP across the five core EBM steps, highlighting NLP methods such as information retrieval, entity and relation extraction, evidence ranking, synthesis, and summarization. The review identifies ongoing challenges, including the need for better benchmark datasets specific to medical applications and addressing model interpretability, biases, and resource-intensive processes.
Summary of reasons to publish: - Well written survey paper, providing extensive coverage of Natural Language Processing (NLP) applications across all stages of the Evidence-based Medicine (EBM) lifecycle
- the paper thoughtfully highlights limitations and clearly identifies areas needing further exploration