# Electronic Health Record based recruitment support systems: A scoping review to improve patient inclusion within clinical trials

Morgan Vaterkowski, Christel Daniel, Ambre La Rosa, Dipak Kalra, Marie-Christine Jaulent, Guillaume Lamé, Emmanuelle Kempf

<figure><img src="image_1.png" /></figure>

## To cite this version:

Morgan Vaterkowski, Christel Daniel, Ambre La Rosa, Dipak Kalra, Marie-Christine Jaulent, et al. Electronic Health Record based recruitment support systems: A scoping review to improve patient inclusion within clinical trials. International Journal of Medical Informatics, 2025, 201, pp.105900. 10.1016/j.ijmedinf.2025.105900. hal-05011707

HAL Id: hal-05011707

https://hal.science/hal-05011707v1

Submitted on 10 Jun 2025

HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_3.png" /></figure>

## Review article

# Electronic Health Record based recruitment support systems: A scoping review to improve patient inclusion within clinical trials

Morgan Vaterkowski<sup>a,b</sup>, Christel Daniel<sup>a,c,*</sup>, Ambre La Rosa<sup>a</sup>, Dipak Kalra<sup>d,e</sup>, Marie-Christine Jaulent<sup>a</sup>, Guillaume Lame<sup>f</sup>, Emmanuelle Kempf<sup>a,g</sup>

<figure><img src="image_4.png" /></figure>

## ARTICLE INFO

### Keywords:

Clinical trials (as topic)

Electronic Health Record

Evaluation technology

Patient recruitment

Prescreening

Scoping review

## ABSTRACT

**Objective:** Electronic Health Record (EHR) data is increasingly used to support patient recruitment in clinical trials (CTs). This scoping review focused on articles reporting an evaluation of EHR-based CT recruitment support systems (CTRSS).

**Methods:** Querying 3 biomedical literature databases using keywords as "clinical trial as topic", "patient recruitment", and "EHR", research articles describing evaluations of EHR-based CTRSS and published in English from 2013 to 2024 were included. Abstract screening was performed by 5 reviewers. Two reviewers performed full-text screening, data extraction and synthesis following 3 major themes: clinical workflow integration, technical aspects, and evaluation of EHR-based CTRSS.

**Results:** Of 927 screened articles, 44 qualified in scope, providing an evaluation of EHR-based CTRSS conducted within a diverse set of use cases, settings, and technologies. Results showed that 75% of the systems were CT rather than patient centric, 20% provided patient eligibility information at clinically relevant timepoints in the patient journey and 9% sent notifications to the targeted users. EHR data was transformed according to standard models in 32% of the studies. Eligibility criteria transformation was computer assisted in 48% of the studies. The main patient-trial matching technique was querying structured data in 55% of the cases. Finally, the majority (61%) of the evaluations were done retrospectively and led to a complete system evaluation from data transformation to patient - trial matching in 23%.

**Conclusion:** Despite the diversity of system scopes and designs, metrics for effective evaluation throughout the entire cycle of system implementation across user, technology, and society levels are emerging. Research effort is still needed to extend the scope, quality, and computability of both the source EHR-data and EC and to conduct harmonized evaluation of these systems to demonstrate their impact on recruitment rates, delays, and costs.

## 1. Introduction

Recruiting participants in clinical trials (CTs) remains a key determinant of research efficiency but is highly challenging. Studies commonly report low rates of patient recruitment into CTs establishing this as the most important reason for CT delays. [1,2]

Recruitment failure represents an immense waste of time, money, and human effort in the clinical research life cycle. Thus, healthcare professional policy-makers have to postpone their decisions or base them on an inferior level of evidence. Above all, from the patient perspective, low rates of enrollment in CTs delays access to new treatments. [3,4] Barriers and facilitators to CT enrollment have been largely reported in reviews in the

<footnote><sup>a</sup> Sorbonne Université, INSERM, Université Sorbonne Paris-Nord, Laboratoire d'informatique médicale et d'ingénierie des connaissances en e-santé, LIMICS, 75006 Paris, France</footnote>

<footnote><sup>b</sup> Clinical Science and Operations, Sanofi Research and Development, Gentilly, France</footnote>

<footnote><sup>c</sup> Assistance Publique - Hôpitaux de Paris, Henri Mondor Teaching Hospital, Department of Medical Information, Créteil, France</footnote>

<footnote><sup>d</sup> European Institute for Innovation through Health Data, Ghent, Belgium</footnote>

<footnote><sup>e</sup> Unit of Medical Informatics, Department of Public Health and Primary Care, Ghent University, Ghent, Belgium</footnote>

<footnote><sup>f</sup> Paris Saclay University, Centrale Supélec, Laboratoire de génie industriel, Gif-sur-Yvette, France</footnote>

<footnote><sup>g</sup> Paris Est Créteil University, Assistance Publique - Hôpitaux de Paris, Henri Mondor Teaching Hospital, Department of Medical Oncology, Créteil, France</footnote>

<footnote>* Corresponding author at: Sorbonne Université, Inserm, université Paris 13, Laboratoire d'informatique médicale et d'ingénierie des connaissances en e-santé, LIMICS, 15 rue de l'école de médecine, F-75006 Paris, France.</footnote>

<footnote>E-mail address: christel.daniel@aphp.fr (C. Daniel).</footnote>
literature. [4–6] Many reviews provide insights on the different strategies used to improve patient recruitment including the deployment of digital services. [3,7–15] According to a recent review, the number of published studies on digital patient recruitment approaches has doubled in the past decade. [16] Tools and systems supporting patient recruitment into CTs – designated as “Clinical Trial Recruitment Support System” (CTRSS) – may target different steps of the enrollment process, therefore addressing different use cases: protocol optimization, site feasibility assessment and patient eligibility determination at the pre-inclusion step designated as “prescreening”. Site feasibility assessment occurs before the opening of a CT at a given institution and consists in careful consideration of the cohort of potential trial participants, institutional strengths (resources of the site, burden of trial procedures, etc.), site personnel’s interest and commitment (including whether there is an unmet but well-recognized clinical need and the existing portfolio of potentially competing trials). During CT execution, the patient enrollment process consists of multiple rounds of patient eligibility determination while maintaining communication between the patient’s care team and study coordinator to ensure the availability of slots in CTs. After the first round of patient eligibility determination during the prescreening phase, a pre-screened patient is contacted and offered the option to enroll. Interested patients undergo a final screening for evaluation of modifiable health criteria (status “screened”) followed by a consenting process (status “enrolled” or “recruited”). The electronic health record (EHR) housing clinical, genomic, and treatment-related data providing an up-to-date knowledge of the patient’s disease state may be a useful tool for addressing the aforementioned challenges of CT enrollment. The aim of this study was to identify the available published literature focusing on evaluation studies of CTRSS using patient EHR data designated as “EHR-based CTRSS”, and to synthesize (1) how they were integrated in the clinical workflow; (2) which technical challenges were needed to be addressed, and (3) how they were assessed and what the assessments revealed in terms of benefit over traditional methods.

## 2. Methods

This scoping review followed the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for Scoping Reviews (PRISMA-ScR) guideline. [17].

### 2.1. Study selection

The databases employed in literature search were PubMed, IEEE and EMBASE. The following eligibility criteria were applied to identify articles (1) published in English with abstracts available; related to (2) CTs or multicenter studies or randomized controlled trials as topic; (3) focused on patient recruitment (4) reusing EHR data from EHR systems, clinical data warehouses (CDW) or disease specific registries and (5) published between January 1, 2013 and December 31, 2024. search strings are available in Supplementary Methods. The screening process took part in two steps. Title and abstract were screened by 5 reviewers (AL, CD, EK, GL, MV) to identify articles with CT recruitment and EHR focus and to exclude reviews. Full text were screened by 2 reviewers (CD and MV) to exclude articles describing EHR-based CTRSS only at their early design phase without any implementation and articles without any detailed evaluation. Average agreement at each step was calculated. Disagreements were discussed among reviewers until reaching a consensus.

### 2.2. Conceptual framework, data extraction and analysis

Based on previous work, a conceptual framework of the end-to-end enrollment process was built for guiding data extraction. [14,18] General characteristics about the EHR-based CTRSS of interest were first extracted from the articles: targeted use cases, diseases of interest, number of trials in focus, number of sites, EHR data source and year of

publication. An additional set of data elements following 15 scoping questions (SQ) was extracted and grouped in 3 themes: (1) integration of the EHR-based CTRSS to the clinical or research workflows, (2) technical challenges to be addressed for matching EHR data and CT eligibility criteria (EC), and (3) EHR-based CTRSS evaluation (Table 1). Two reviewers (MV, CD) extracted the data elements from the included articles while 2 reviewers (MCJ, EK) verified the extraction in case of discordance. Rayyan.ia<fnref n="1" /> was used for the collaborative study selection, Microsoft Excel 2016 for the data collection.

## 3. Results

### 3.1. Study selection and characteristics

Fig. 1 displays the PRISMA flow chart for article selection.

The first selection step averaged 97 % of agreement among reviewers and the second one 72 % of agreement among reviewers. Of 927 articles found, 111 were included in the full-text screening step, and 44 articles were included for full analysis (Table 2). In terms of targeted use cases, they were mainly used in the context of prescreening (87 %) or feasibility (14 %). Five papers considered that an EHR-based CTRSS should interestingly provide additional functionalities beyond feasibility or prescreening such as screening, patient retention in CTs, establishing and maintaining a CT database, defining different target populations using real world data and extracting baseline research data from EHR. [19–23] The selected studies were conducted in various medical domains including cancer (41 %), cardiovascular disease (39 %) or diabetes (23 %). The systems were deployed in single (55 %) or multiple (25 %) sites and evaluated in the context of multiple (52 %) or single (32 %) CTs.

### 3.2. Scoping questions

Tables 3, 4 & 5 provide a high-level summary of the data related to the 15 scoping questions extracted among all the included articles. Key results for each scoping question are provided below and categorized into the 3 themes. Sup. Table 1a, b, c, d provide detailed research characteristics for each of the 44 articles.

#### 3.2.1. Theme 1: Integration into the clinical and/or research workflows

In 25 articles (57 %), the interaction model of the EHR-based CTRSS was described in sufficient detail to qualify if it was an active or passive system (SQ 1.1). Four EHR-based CTRSS (9 %) were active systems i.e., a system sending notifications to the targeted users. These active systems, used in various therapeutic areas in the context of prescreening, relied on queries on structured data for patient-trial matching (2 out of 4) as much as query on structured data and natural language processing (NLP) and/or machine learning (ML) on unstructured data (2 out of 4) and were evaluated for a single CT (2 out of 4) as much as multiple trials. They provided various types of notifications about potentially eligible patients to targeted users such as emails, text messages or alerts on dedicated mobile Apps. Most (75 %) of the EHR-based CTRSS were CT centered (SQ1.2), 11 systems (25 %) were designed as patient centered systems. Of those patient centered systems, 6 were developed in the cancer domain and relied mostly on queries (7 out of 11) than machine learning and/or NLP techniques as patient-trial matching methods. They had been often evaluated prospectively (7 out of 11) and during the execution of multiple CTs (7 out of 11). Nine papers described how the EHR-based CTRSS was integrated into the clinical workflow (SQ 1.3). The eligibility status was computed in anticipation of either a scheduled visit or a procedure (6 out of 9), [28,37,43,44,49,53] a multidisciplinary meeting, especially in cancer domain (2 out of 9), [34,56] or at the opening of the patient chart (1 out 9). [55] The description of the user

<footnote n="1">¹ Online version available April 21, 2024.</footnote>
<caption>**Table 1**</caption>

Scoping questions addressing the 3 research themes: workflow integration, technical characteristics, and evaluation of the EHR-based CTRSS.

<table><thead><tr><th>Scoping question</th><th>Explanation and possible values</th></tr></thead><tbody><tr><td colspan="2">Theme 1: EHR-based CTRSS integration to the clinical or research workflow</td></tr><tr><td>SQ 1.1: Active or passive system</td><td>How did the CTRSS interact with its users? Did the CTRSS push notification(s) (active) or wait to be used by a dedicated user (passive)?</td></tr><tr><td>SQ 1.2: Patient or CT centered</td><td>Did the CTRSS scan a set of open CTs to identify those corresponding to a given patient or cohort of patients (patient centered)? Or was the tool designed for identifying patients eligible to a single or few trials or to answer trial questions (as a feasibility questionnaire) (CT centered)?</td></tr><tr><td>SQ 1.3: Triggering/targeted event</td><td>What was the event(s) setting the CTRSS services in motion (new diagnosis or change in disease state collected during patient visit, new test results, others)? What was the targeted point(s) of care considered to notify the care provider/patient (scheduled medical visit or procedure, multidisciplinary meeting, others)?</td></tr><tr><td>SQ 1.4: Targeted users</td><td>By whom was the CTRSS used? (CT investigator and/or research staff, caregivers, both care and research staff)</td></tr><tr><td>SQ 1.5: Delay of execution and data refresh</td><td>How long did the CTRSS need to operate to provide an answer and/or on what frequency the system was refreshing itself?</td></tr><tr><td colspan="2">Theme 2: Technical characteristics addressed for matching EHR data and CT EC</td></tr><tr><td>SQ 2.1: Application source (a) and level of structuration of EHR data (b)</td><td>What was the application source of EHR data (CDW / EHR / Registry / synthetic data)? Were the EHR data structured at data entry, unstructured data, unstructured data transformed into structured data thanks to NLP/ML?</td></tr><tr><td>SQ 2.2: Data quality</td><td>Did the publication mention EHR data quality issues (mentioned) or describe data quality assessment and control (described in detail)? Or not (not specified)?</td></tr><tr><td>SQ 2.3: EC processing</td><td>What was the level of assistance provided by the system in the processing of the EC? Were the EC manually, semi (assisted) or fully automatically transformed?</td></tr><tr><td>SQ 2.4: Data model</td><td>Was there a description of a data model implemented to support the patient-CT matching? If yes, was the data model standard (HL7, OMOP, etc)</td></tr><tr><td>SQ 2.5: Patient-trial matching method</td><td>Which patient-trial matching method was used? (Queries on structured data, queries on unstructured data, ML on structured data, ML on unstructured data, none)</td></tr><tr><td colspan="2">Theme 3: EHR-based CTRSS evaluation</td></tr><tr><td>SQ 3.1: Retrospective or prospective study</td><td>What was the design of the evaluation study? (retrospective or prospective)</td></tr><tr><td>SQ 3.2: Performance assessment in real-world conditions</td><td>Was the CTRSS evaluated in RWC (in the context of running CTs)?</td></tr><tr><td>SQ 3.3: Evaluation category</td><td>Was the evaluation a retrospective study not conducted in RWC (proof of concept), a retrospective study conducted in RWC or a prospective study conducted in RWC?</td></tr><tr><td>SQ 3.4: Component under evaluation?</td><td>What component was under evaluation? (EHR data or EC transformation pipeline, patient-trial matching, whole CTRSS)</td></tr><tr><td>SQ 3.5: Gold standard?</td><td>Which comparator was considered for eligibility criteria or eligibility status? (expert annotation, chart review, CT recruitment status)</td></tr></tbody></table>

profiles targeted by EHR-based CTRSS was available in 24 out of 44 articles (SQ 1.4). The targeted users were mainly the research staff followed by caregivers. Research and care teams were both targeted in 6 articles. [19,30,34,46,49]. The execution time of the patient-trial matching process (SQ1.5) was available in 11 articles and the frequency of the (manual or automatic) refreshment of the patient-trial matching list was specified in 5 articles. Four articles provided a description of both information. [29,34,49,53].

### 3.2.2. Theme 2: Technical characteristics of EHR-based CTRSS (EHR data, EC processing and patient-trial matching)

The main application sources of the EHR data were the EHR itself directly (55 %) followed by CDW (48 %). The use of disease specific registry and synthetic data was reported in only 2 articles. EHR and CDW were used together on 3 occasions (7 %). [27,29,62] The CTRSS used structured EHR data in 22 cases and unstructured data in 9 cases. The use of both structured and unstructured EHR data at the same time occurred 13 times. In 5 cases unstructured EHR data were structured using NLP. More than half of the articles (59 %) at least mentioned EHR data quality issues. Data quality control in place for ensuring that the EHR data fit the purpose of patient-trial matching was described in detail in 16 articles (36 %) (SQ2.2).

Regarding the EC transformation pipeline, the manual approach was the most used one for EC processing (SQ 2.3) with 36 % of the occurrences. Almost one third of the articles (30%) reported an assisted processing of EC while 18 % reported a fully automatic processing. [32,33,35,42,43,48,58,59]. Overall, 32 % of the articles (14 out of 44) provided an explicit description of the underlying data model of the patient-trial matching system (SQ2.4). In most cases (9 out of 14), the system relied on international standard models such as Health Level Seven (HL7) Fast Healthcare Interoperability Resources (FHIR) (3/14), [46,55,56] non-FHIR HL7 standard (3/14), [25,34,61] or Observational Medical Outcomes Partnership (OMOP) (1/14). [54]. Some papers reported the use of an Informatics for Integrating Biology & the Bedside (i2b2) ontology (2 out of 14), [24,38] or other common data models (6 out of 14). [19,28,36,50,53,61] Two papers described using two data models in the same studies. [54,61] Most of the articles not mentioning nor describing a specific data model stated that they used standard terminologies to represent clinical information to ensure interoperability at different scales.

The predominant patient-trial matching method used by EHR-Based CTRSS (SQ 2.5) relied on queries on structured data (59 %). Two articles described non SQL-based queries using other query languages such as SPARQL, [34] or CQL. [55] Two systems used queries on both structured and unstructured data. [50,62]. Various algorithms of ML and/or NLP were used. In 10 (23 %) cases they used unstructured data only, in 4 (9 %) cases both unstructured and structured data and in 1 (2 %) case only structured data. Queries on structured data and ML and/or NLP on unstructured data were used together in 3 articles (7 %). [28,35,53].

### 3.2.3. Theme 3: EHR-based CTRSS evaluation

The study design adopted for the EHR-based CTRSS evaluation (SQ 3.1) was balanced between retrospective and prospective studies with respectively 61 % (27 out of 44) and 39 % (17 out of 44) of the articles. The articles describing an evaluation of the CTRSS in real world conditions (SQ 3.2), i.e., in the context of running CTs, represented 64 % (28 out of 44) of cases. Following our classification based on the Evaluation in Life Cycle of Information Technology (ELICIT) framework, [63] 34 % (15 out of 44) of the studies were classified as evaluations of proofs of concept, 27 % (12 out of 44) were retrospective evaluations of implemented systems conducted in real world conditions and 39 % (17 out of 44) were prospective studies of operational systems. Overall, 29 studies evaluated at least the ability of the patient-trial matching tool to identify potential participants in terms of accuracy, sensitivity, specificity, positive predictive value, negative predictive value, F1-score. 10 of them reported in addition its impact on the recruitment. Some articles (15 out

<footnote>* Not mutually exclusive categories.</footnote>

<footnote>Abbreviations: CDW, Clinical Data Warehouse; CT, Clinical Trial; CTRSS, Clinical Trial Recruitment Support System; EC, Eligibility Criteria; EHR, Electronic Health Record; HL7, Health Level 7; T, Information Technology; ML, Machine Learning; NLP, Natural Language Processing; OMOP, Observational Medical Outcomes Partnership; RWC, real world conditions.</footnote>
<figure><img src="image_5.png" /><figcaption>Fig. 1. PRISMA diagram for article selection. Abbreviations: CTRSS, clinical trial recruitment support system; EHR, Electronic Health Record.</figcaption></figure>

of 44, 34 %) focused on the evaluation of specific sub-components of EHR-based CTRSS such as the EHR data transformation pipeline, [48] or the EC transformation pipeline (SQ3.4).

# 4. Discussion

## 4.1. Summary of evidence

This scoping review focused on articles evaluating digital solutions using patient EHR data to support patient recruitment into CTs, designated as "EHR-based CTRSS". Two recent scoping reviews addressed a broader scope considering the use of patient EHR data at any step of the whole life cycle of CTs in US, [14] and at an international scale. [15] Compared to the single study also dedicated to CTRSS published by Cuggia et al., [13] 13 years ago this review addressed a more focused scope considering only the articles reporting an evaluation of the CTRSS. Insights from included articles were mapped to a conceptual framework based on the end-to-end analysis of the complex recruitment process to summarize the current state of EHR-based CTRSS according to three themes (1) society and users (how CTRSS were integrated in the clinical workflow); (2) information technology (the technical challenges that were addressed), and (3) evaluation methodology (how they were assessed).

Regarding the clinical integration (Theme 1), findings pointed that EHR-based CTRSS systems were usually developed and evaluated within a single use case – mostly for assisting the prescreening step – even though the other steps of the recruitment could benefit from a patient-

trial matching tool. Still few institutions reported successful deployment of EHR-based CTRSS as part of a global large scale centralized recruitment service addressing multiple use cases to support clinical research. [61,62] Like any other clinical decision support systems, CTRSS are expected to provide the right message, to the right person at the right point of care and time. Still a minority of systems were active systems providing notifications to users ("the right message"). Examples of types of notifications included emails or smartphone alerts. [28,44,49,53] One paper stated that those users had to be assisted by technical experts to execute queries, to leverage non-structured data or to set up alerts. [53] Consistently with Cuggia et al.'s review, the CTRSS were still predominantly trial-centered systems (75 %), not primarily designed to support a care giver to find a CT for her patient.

In patient-centered systems caregivers are explicitly identified as targeted users of the CTRSS, which was the case in only 25 % of the articles. Patient-centered systems support the vision of considering CTs as care options and promote a better integration of care and research processes. Although the great importance of delivering the notification of patient eligibility status at the "right place", still few EHR-based CTRSS (21 %) explicitly informed caregivers in the context of targeted points of care such as scheduled visit or a procedure or a multidisciplinary meeting. Among 34 % of the articles providing data on the execution time of the patient-trial matching process, only 4 provided in addition the frequency of refreshment which is necessary to make the CTRSS clinically relevant. [29,34,49,53] In summary, EHR-based CTRSS still often failed at supporting the multiple rounds of eligibility screening beyond just finding patient-trial matches and at facilitating
<table><caption>**Table 2**

Overall descriptive statistics of the included articles.</caption><thead><tr><th>Clinical and IT setting</th><th>Reference</th><th>Number of articles</th><th>%</th></tr></thead><tbody><tr><td><strong>Targeted use case</strong></td><td></td><td></td><td></td></tr><tr><td>Feasibility</td><td>[24-27]</td><td>4</td><td>9%</td></tr><tr><td>Prescreening</td><td>[19-23,28-60]</td><td>36</td><td>82%</td></tr><tr><td>Feasibility & Prescreening</td><td>[61,62]</td><td>2</td><td>5%</td></tr><tr><td>Screening</td><td>[23]</td><td>1</td><td>2%</td></tr><tr><td>Protocol design</td><td>[22]</td><td>1</td><td>2%</td></tr><tr><td><strong>Number of CT</strong></td><td></td><td></td><td></td></tr><tr><td>Multiple</td><td>[19,24-28,30,31,34,36,40-43,46,53,54,56-59,61,62]</td><td>23</td><td>52%</td></tr><tr><td>Single</td><td>[20-23,29,35,37-39,44,45,49,50,55]</td><td>14</td><td>32%</td></tr><tr><td>None</td><td>[32,33,47,48,51,52,60]</td><td>7</td><td>16%</td></tr><tr><td><strong>Number of sites</strong></td><td></td><td></td><td></td></tr><tr><td>Single</td><td>[19-21,24,28-31,34-36,38-43,45,46,53,54,57,58,62]</td><td>24</td><td>55%</td></tr><tr><td>Multiple</td><td>[22,23,25-27,37,44,49,50,56,61]</td><td>11</td><td>25%</td></tr><tr><td>None</td><td></td><td>9</td><td>20%</td></tr><tr><td><strong>Disease of interest*</strong></td><td></td><td></td><td></td></tr><tr><td>Cancer</td><td>[19,24,25,28,30,31,34,36,38,41,42,53,56-61]</td><td>18</td><td>41%</td></tr><tr><td>Cardiovascular disease</td><td>[20,21,23,25,27,32,33,35,40,41,44,46-48,51,52,54]</td><td>17</td><td>39%</td></tr><tr><td>Diabetes</td><td>[22,25,32,33,40,41,47,51,52,60]</td><td>10</td><td>23%</td></tr><tr><td>Other</td><td>[26,29,37,39,43,49,50,54,55,60]</td><td>10</td><td>23%</td></tr><tr><td>Neurological disease</td><td>[25,27,40]</td><td>3</td><td>7%</td></tr><tr><td>None / unspecified</td><td>[19,45]</td><td>2</td><td>5%</td></tr><tr><td><strong>EHR data source</strong></td><td></td><td></td><td></td></tr><tr><td>EHR</td><td>[19,20,23,30,31,34,35,37,39-46,49,50,55-57]</td><td>21</td><td>48%</td></tr><tr><td>CDW</td><td>[21,22,24,25,28,32,33,36,38,47,48,51-54,58,60,61]</td><td>18</td><td>41%</td></tr><tr><td>EHR & CDW</td><td>[27,29,62]</td><td>3</td><td>7%</td></tr><tr><td>Synthetic data</td><td>[59]</td><td>1</td><td>2%</td></tr><tr><td>EHR sourced registry</td><td>[26]</td><td>1</td><td>2%</td></tr><tr><td><strong>Publication date</strong></td><td></td><td></td><td></td></tr><tr><td>2013</td><td>[24,28,36]</td><td>3</td><td>7%</td></tr><tr><td>2014</td><td>[62]</td><td>1</td><td>2%</td></tr><tr><td>2015</td><td>[25,34,41-43,48,61]</td><td>7</td><td>16%</td></tr><tr><td>2016</td><td>[37,53]</td><td>2</td><td>5%</td></tr><tr><td>2017</td><td>[26,35]</td><td>2</td><td>5%</td></tr><tr><td>2019</td><td>[19-21,32,33,40,47,49,51,52]</td><td>10</td><td>23%</td></tr><tr><td>2020</td><td>[30,31,45,50]</td><td>4</td><td>9%</td></tr><tr><td>2021</td><td>[22,23,38,39]</td><td>4</td><td>9%</td></tr><tr><td>2022</td><td>[27,29,44,46,60]</td><td>5</td><td>11%</td></tr><tr><td>2023</td><td>[54]</td><td>1</td><td>2%</td></tr><tr><td>2024</td><td>[55-59]</td><td>5</td><td>11%</td></tr></tbody></table>

Abbreviations: CDW, Clinical Data Warehouse; CT, Clinical Trial; EHR, Electronic Health Record; IT, Information Technology.

* Not mutually exclusive category.

<table><caption>**Table 3**

Summary of research characteristics among all included articles, for theme 1.</caption><thead><tr><th>Theme 1 – EHR-based CTRSS integration in clinical/research workflows</th><th>N</th><th>%</th><th>References</th></tr></thead><tbody><tr><td>1.1 Active or passive system</td><td></td><td></td><td></td></tr><tr><td>Passive</td><td>21</td><td>48%</td><td>[19-21,23-27,29,30,34,37,39,40,45,46,54-56,61,62]</td></tr><tr><td>Active</td><td>4</td><td>9%</td><td>[28,44,49,53]</td></tr><tr><td>Not specified</td><td>19</td><td>43%</td><td></td></tr><tr><td>1.2 Patient or trial centered</td><td></td><td></td><td></td></tr><tr><td>Clinical trial centered</td><td>33</td><td>75%</td><td>[20-27,29,32,33,35,36,38-42,45-48,50-52,54,57-62]</td></tr><tr><td>Patient centered</td><td>11</td><td>25%</td><td>[19,28,30,31,34,37,43,44,49,53,55,56]</td></tr><tr><td>1.3 Triggering/targeted event</td><td></td><td></td><td></td></tr><tr><td>Medical visit</td><td>6</td><td>14%</td><td>[28,37,43,44,49,53]</td></tr><tr><td>Multidisciplinary meeting</td><td>2</td><td>5%</td><td>[34,56]</td></tr><tr><td>At the hospital opening of patient chart</td><td>1</td><td>2%</td><td>[55]</td></tr><tr><td>Not specified</td><td>35</td><td>79%</td><td></td></tr><tr><td>1.4 Targeted users</td><td></td><td></td><td></td></tr><tr><td>Research staff</td><td>13</td><td>30%</td><td>[20,21,27-29,35,54,56,62]</td></tr><tr><td>Both care and research staff</td><td>6</td><td>14%</td><td>[19,30,34,46,49,53]</td></tr><tr><td>Caregivers</td><td>5</td><td>11%</td><td>[23,25,37,44,55]</td></tr><tr><td>Not specified</td><td>20</td><td>45%</td><td></td></tr><tr><td>1.5 Delay of execution and data refresh</td><td></td><td></td><td></td></tr><tr><td>Matching duration</td><td>11</td><td>25%</td><td>[23,25,27,30,31,35,50,54,55,57,61]</td></tr><tr><td>Execution frequency / refresh frequency</td><td>5</td><td>11%</td><td>[19,28,36,45,46]</td></tr><tr><td>Both</td><td>4</td><td>9%</td><td>[29,34,49,53]</td></tr><tr><td>Not specified</td><td>24</td><td>55%</td><td></td></tr></tbody></table>

Abbreviations: CT, Clinical Trial; CTRSS, Clinical Trial Recruitment Support System; EHR, Electronic Health Record.
<table><caption>**Table 4**
Summary of research characteristics among all included articles, for theme 2.</caption><thead><tr><th>Theme 2 – Technical characteristics of the EHR-based CTRSS</th><th>N</th><th>%</th><th>References</th></tr></thead><tbody><tr><td>2.1.a Application source and level of structuration of EHR data</td><td></td><td></td><td></td></tr><tr><td>EHR</td><td>21</td><td>48 %</td><td>[19,20,23,30,31,34,35,37,39–46,49,50,55–57]</td></tr><tr><td>CDW</td><td>18</td><td>41 %</td><td>[21,22,24,25,28,29,32,33,36,38,47,48,51–54,61,62]</td></tr><tr><td>EHR & CWD</td><td>3</td><td>7 %</td><td>[27,29,62]</td></tr><tr><td>Synthetic data</td><td>1</td><td>2 %</td><td>[59]</td></tr><tr><td>Registry</td><td>1</td><td>2 %</td><td>[26]</td></tr><tr><td>2.1.b Level of structuration of EHR data</td><td></td><td></td><td></td></tr><tr><td>Structured EHR data</td><td>24</td><td>55 %</td><td>[19–22,24–27,29,34,36–40,44–46,49,54–56,58,61]</td></tr><tr><td>Structured and unstructured EHR data</td><td>11</td><td>25 %</td><td>[23,28,31,35,41–43,50,53,57,62]</td></tr><tr><td>Unstructured data transformed into structured data thanks to NLP</td><td>6</td><td>14 %</td><td>[41,43,46,50,53,57]</td></tr><tr><td>Unstructured EHR data</td><td>9</td><td>20 %</td><td>[30,32,33,47,48,51,52,59,60]</td></tr><tr><td>2.2 Data quality issues and/or control</td><td></td><td></td><td></td></tr><tr><td>Described in detail</td><td>16</td><td>36 %</td><td>[23–26,28–30,34,36,38,41,43,44,46,56,58]</td></tr><tr><td>Mentioned</td><td>10</td><td>23 %</td><td>[22,27,40,48–51,53,57,59]</td></tr><tr><td>Not specified</td><td>18</td><td>41 %</td><td></td></tr><tr><td>2.3 Eligibility criteria processing</td><td></td><td></td><td></td></tr><tr><td>Manual</td><td>16</td><td>36 %</td><td>[20,22,26,28,34,38–40,44–46,49,50,52,60,62]</td></tr><tr><td>Assisted</td><td>13</td><td>30 %</td><td>[19–23–25,27,29,30,37,53–56,61]</td></tr><tr><td>Automatic</td><td>8</td><td>18 %</td><td>[32,33,35,42,43,48,58,59]</td></tr><tr><td>None</td><td>4</td><td>9 %</td><td>[36,41,47,51]</td></tr><tr><td>Not specified</td><td>3</td><td>7 %</td><td></td></tr><tr><td>2.4 Data model</td><td></td><td></td><td></td></tr><tr><td>Standard data model</td><td>14</td><td>32 %</td><td>[19,24,25,28,34,36,38,46,50,53–56,61]</td></tr><tr><td>HL7 (FHIR)</td><td>3</td><td>7 %</td><td>[46,55,56]</td></tr><tr><td>HL7 (other than FHIR)</td><td>3</td><td>7 %</td><td>[25,34,61]</td></tr><tr><td>I2B2</td><td>2</td><td>5 %</td><td>[24,38]</td></tr><tr><td>OMOP & HL7 (FHIR)</td><td>1</td><td>2 %</td><td>[54]</td></tr><tr><td>other</td><td>6</td><td>14 %</td><td>[19,28,36,50,53,61]</td></tr><tr><td>Absence of standard data model</td><td>30</td><td>68 %</td><td>[20–23,26,27,29–33,35,37,39–45,47–49,51,52,57–60,62]</td></tr><tr><td>2.5 Patient-trial matching method</td><td></td><td></td><td></td></tr><tr><td>Query</td><td>26</td><td>59 %</td><td>[19–22,24–29,31,34,35,37–40,44–46,49,50,53–56,58,61,62]</td></tr><tr><td>Query on structured data</td><td>24</td><td>55 %</td><td>[19–22,24–29,31,34,35,37–40,44–46,49,53–56,58,61]</td></tr><tr><td>Query on structured and unstructured data</td><td>2</td><td>4 %</td><td>[50,62]</td></tr><tr><td>ML and/or NLP</td><td>15</td><td>34 %</td><td>[23,28,30,32,33,35,41–43,47,48,51–53,57]</td></tr><tr><td>Machine learning and/or NLP on unstructured data</td><td>10</td><td>23 %</td><td>[23,30,32,33,47,48,51,52,59,60]</td></tr><tr><td>Machine learning and/or NLP on structured and unstructured data</td><td>4</td><td>9 %</td><td>[41–43,57]</td></tr><tr><td>Machine learning on structured data</td><td>1</td><td>2 %</td><td>[36]</td></tr><tr><td>Query on structured data and machine learning and/or NLP on unstructured data</td><td>3</td><td>7 %</td><td>[28,35,53]</td></tr></tbody></table>

Abbreviations: CTRSS, Clinical Trial Recruitment Support System; CDW, Clinical Data Warehouse; EHR, Electronic Health Record; FHIR, Fast Healthcare Interoperability Resources; HL7, Health Level Seven International; ML, Machine Learning; NLP, Natural Language Processing; OMOP, Observational Medical Outcomes Partnership.

<table><caption>**Table 5**
Summary of research characteristics among all included articles, for theme 3.</caption><caption>Theme 3 – EHR-based CTRSS evaluation</caption><tbody><tr><td>3.1 Study design</td><td></td><td></td><td></td></tr><tr><td>Retrospective</td><td>27</td><td>61 %</td><td>[22–26,30–34,36,38,41–43,47,48,50–52,54,56–61]</td></tr><tr><td>Prospective</td><td>17</td><td>39 %</td><td>[19–21,27–29,35,37,39,40,44–46,49,53,55,62]</td></tr><tr><td>3.2 Performance assessment in RWC</td><td></td><td></td><td></td></tr><tr><td>Yes</td><td>29</td><td>66 %</td><td>[19–23,25–30,35,37–46,49,50,53–55]</td></tr><tr><td>No</td><td>15</td><td>34 %</td><td>[24,31–34,36,47,48,51,52,56–60]</td></tr><tr><td>3.3 Evaluation category</td><td></td><td></td><td></td></tr><tr><td>Prospective studies in RWC</td><td>17</td><td>39 %</td><td>[19–21,27–29,35,37,39,40,44–46,49,53,55,62]</td></tr><tr><td>Retrospective studies not in RWC (POC)</td><td>15</td><td>34 %</td><td>[24,31–34,36,47,48,51,52,56–60]</td></tr><tr><td>Retrospective studies in RWC</td><td>12</td><td>27 %</td><td>[22,23,25,26,30,34,38,41–43,50,61]</td></tr><tr><td>3.4 Component under evaluation</td><td></td><td></td><td></td></tr><tr><td>Patient-trial matching</td><td>19</td><td>43 %</td><td>[22,25,26,29,31,34,36,41,46,49,50,53–57,59,60,62]</td></tr><tr><td>Data (EHR data and EC) transformation pipeline</td><td>15</td><td>34 %</td><td>[19,24,27,32,33,37,40,43,45,47,48,51,52,58,61,19,24,27,32,33,37,40,43,45,47,48,51,52,61]</td></tr><tr><td>Whole EHR-based CTRSS</td><td>10</td><td>23 %</td><td>[20,21,23,28,30,35,38,39,42,44]</td></tr><tr><td>3.5 Gold standard</td><td></td><td></td><td></td></tr><tr><td>Recruitment status</td><td>15</td><td>34 %</td><td>[20,21,24,30,31,36,39–43,46,50,54,62]</td></tr><tr><td>Expert annotation</td><td>12</td><td>27 %</td><td>[19,29,32–34,47,48,51–53,58,60]</td></tr><tr><td>Chart review</td><td>7</td><td>16 %</td><td>[25,35,38,49,56,57,59]</td></tr><tr><td>Chart review & recruitment status</td><td>1</td><td>2 %</td><td>[23]</td></tr><tr><td>Not applicable</td><td>9</td><td>21 %</td><td></td></tr></tbody></table>

Abbreviations: CTRSS, Clinical Trial Recruitment Support System; EC, Eligibility Criteria; EHR, Electronic Health Record; POC, Proof Of Concept; RWC, Real World Condition
collaborative workflow between the patient, the physician, the trial investigator, and clinical research staff. In this context, a paper emphasized the need for regular training to facilitate the use of the CTRSS and its sub-modules (EC and EHR data processing pipelines). [44].

Regarding the technical aspects (Theme 2), similarly to Cuggia et al.'s review, this review showed that patient EHR data were accessed by the CTRSS in various ways. Some systems had a direct access to the EHR or hospital information system (55%), while others had an indirect access through a CDW (48%). Two systems got the patient data from EHR-based disease specific registries or from synthetic data. Findings of this review showed that EHR systems were more accessed for prescreening while CDWs and the EHR-based registry were more used for site feasibility assessment. In contrast with Cuggia et al.'s review, this review observed an increasing use of unstructured EHR data thanks to the recent development of NLP/ML tools. Therefore, in many studies, CTRSS included tools and services to automatically extract from textual reports EHR data such as past medical history, problem lists, reason for admission, pre admission medication, examination findings, diagnoses, care plans and event timeline. [31,41,43,50,53,57] Overall, in 20% of the articles (9 out of 44), the CTRSS processed both structured and unstructured EHR data to provide patient-trial matching. Consistently with Cuggia et al.'s review, the quality and utility of the EHR data was an issue for most initiatives (59%). But only 36% of the articles provided a description of data quality assessment and control. Some authors assumed that a core data set relevant for patient eligibility determination – including demographic characteristics, diagnosis, laboratory results, medication orders, and free-text clinical notes – could be defined upstream to patient-trial matching and guide the data quality checks. [41,56] The PENELOPE (Patients ENrollment within clinical trials from rEal Life databases using COnmon Patient Data ModEls) initiative also proposed to specify at national scale a minimum data set dedicated to patient-trial matching in oncology. [64] Twenty-nine initiatives dedicated effort and time to formalize EC to be computable by their CTRSS. [19,20,22-30,34,37-40,44-46,49,50,52-56,60-62] In contrast with Cuggia et al.'s review, this study pointed out that there was a good balance across articles between manual EC processing (36%) and assisted or fully automatic EC processing (respectively 30% and 18%). Those results underly the impact of research efforts such as Criteria2-Query to support the EC processing into computable elements at clinical sites and the development of phenotyping algorithms. [65,66] The process of EC formalization being time and resources consuming, some authors reused phenotype definitions shared by organizations such as from PheKB (https://phekb.org/) or OHDSI (https://www.ohdsi.org/resources/libraries/phenotype-library/). One paper had to refine such algorithms. [29] At least one paper focused on the specific issue of categorizing the EC and identifying those that were relevant for the prescreening. [54] The recent progress of the NLP techniques and contributions of initiatives such as the National NLP Clinical Challenge (N2C2) challenges, [32,33,51,52] and more recently the emergence of the use of Large Language Models, [66] supported the development of automatic processing of EC. Furthermore, as stated by Cuggia et al., this study confirmed that interoperability was still challenging, with only a third of the selected papers explicitly reported the use of a standard data model, although more than half of the papers explicitly cited the use of shared vocabularies or ontologies. Lastly, similarly to Cuggia et al.'s review, results showed that various methods were adopted for patient-trial matching including rule-based or query-based methods, ML approaches and case-based reasoning methods. While supervised ML approaches and case-based reasoning methods bring great promise (no EC processing and better generalizability of rules), one paper showed that highly imbalanced data sets induced unique challenges. [51] With this approach, the formalization of EC is no longer necessary, but a delay of manual recruitment is needed to build the training set before setting up the CTRSS.

Compared to the scoping review from Cuggia et al., findings of this

study showed that twice more CTRSS were reaching effective implementation and deployment in clinical setting. This can be explained by the increasing number of published contributions and evidence available on the subject including from National, European, or worldwide initiatives such as EHR4CR (Electronic Healthcare Records For Clinical Research (https://www.ihi.europa.eu/projects-results/project-factsheets/ehr4cr), EHR2EDC (https://eithealth.eu/product-service/ehr2edc/) in Europe, or OHDSI (https://www.ohdsi.org/resources/libraries/phenotype-library/) that have been advancing and providing publicly available insights. However, the findings described a great heterogeneity in the evaluation methodology and metrics reported across all included articles. Relying on the ELICIT evaluation framework, this study categorized the evaluations and harmonized the evaluation metrics. The papers showed that sound evaluations could be conducted at any phase of the system development: with 34% of articles reporting evaluation of proofs of concept and 66% at implementation phase (in real world conditions). Notably, 39% of the articles reported prospective evaluations of EHR-based CTRSS deployed in clinical settings to support on-going CTs. In 27% of the articles, reporting retrospective studies, the evaluation of the CTRSS was qualified as performed in real-world conditions since the metrics, although collected retrospectively, were captured in the time frame corresponding to the recruitment period of CTs. Consistently with Cuggia et al., the evaluation metrics used to assess CTRSS were still very heterogeneous across initiatives which is not fully explained by the different contexts. Lastly, only two studies described a medico-economic analysis of the deployment of the system.

This review has several limitations. First considering full text articles available in English as eligibility criteria precluded in part insights from national initiatives. Second, the scope of this research being focused on prescreening, this study didn't encompass articles describing CTRSS systems evaluated in the context of protocol design though optimizing EC is a critical activity for improving efficiency, and mitigating risks of premature closure. Third, the scope didn't include CTRSS processing patient data sources such as disease specific registries (not automatically fed by EHRs), patient registries or online questionnaires even though representing interesting and promising initiatives for patient eligibility determination.

The strength of this scoping review is to highlight the characteristics of the most mature EHR-based CTRSS deployed during the last decade in various clinical settings. These findings may guide the design and development of future CTRSS. Ideally, an EHR-based CTRSS should be patient centered, providing to both care and research staff real-time notifications of patient eligibility at targeted points of care. Technically, the processing of the EC should be at least assisted otherwise significant human resources should be allocated to the set up and maintenance of the CTRSS over time. The CTRSS should use both structured and unstructured EHR data from several data source if needed. The end users should be informed of the quality of the EHR data and associated limitations. The conformance to standard data models reduces the configuration workload and increases interoperability and reuse. Lastly, the adoption of an evaluation framework applicable throughout the development cycle of the EHR-based CTRSS and addressing the society and user level (how is the system integrated in the clinical workflow?) as well as the IT level (which technical challenges need to be addressed?) shall contribute to the design of impactful solutions and to ensure stability, transparency, and trust from users.

## 5. Conclusion

This scoping review focused on published evaluations of digital tools using patient EHR data to facilitate patient recruitment into CTs, labelled as "EHR-based CTRSS" and conducted during the last decade in various use cases, clinical settings and countries. A framework was proposed to synthesize how these systems were integrated in the clinical workflow, which technical challenges needed to be addressed, and how
they were assessed. Observations showed that the EHR-based CTRSS were still predominantly trial centered and not primarily designed to support a care provider to find a CT for her patient and therefore usually did not properly support the collaborative workflow of actors beyond the notification of patient-trial matches. Although largely integrating recent progresses of the big data and AI era, the technology pillar of EHR-based CTRSS is developing and many challenges regarding EHR data and EC processing, system interoperability or patient-trial matching still require continuous research efforts. Ultimately, while EHR-based CTRSS show great promise, inadequate evaluations may result in poor usability, adoption, or impact. This review categorized EHR-based CTRSS evaluations according to the ELICIT evaluation framework. Despite the diversity of the scope and design of the EHR-based CTRSS evaluated, common metrics, usable throughout their development cycle, are emerging and address the society, user, and IT levels. To efficiently bring impactful EHR-based CTRSS to market at large scale, it is critical to use such a multi-level multi-phase evaluation framework for conducting their systematic evaluations.

## Summary table.

What was already known on the topic:

* Recruiting participants in clinical trials (CTs) remains a key determinant of research efficiency but is highly challenging

* The number of published studies on Clinical Trial Recruitment Support System (CTRSS) has doubled in the past decade

* Previous reviews on the use of real-world data and especially EHR data addressed a broad scope considering any step of the whole life cycle of CTs. A single review published 13 years ago focused on CTRSS highlighted the heterogeneity of system design, scope, maturity, assessment and impact between solutions.

## What this study added to our knowledge:

* This scoping review focused on published evaluation of EHR-based CTRSS and defined a framework to synthesize how CTRSS are integrated in the clinical workflow, which technical challenges need to be addressed, and how CTRSS are assessed.

* Clinical workflow integration and clinician acceptance remain critical for CTRSS which are still predominantly clinical trial-centered, and not properly supporting the collaborative workflow of actors beyond the notification of patient-trial matches.

* Although largely integrating recent progresses of the big data and AI era, the technology pillar of EHR-based CTRSS is still developing. Many challenges regarding EHR data and EC processing, system interoperability and patient-trial matching still require continuous research efforts.

* Despite the diversity of system scopes and designs, metrics for effective evaluation throughout the entire cycle of EHR-based CTRSS implementation are emerging and address the user, technology, and society levels.

## Authors' contributions

MV contributed to the design of the study, abstract and full text screening, data extraction, data analysis, drafting the initial manuscript, EK contributed to the design of the study, abstract screening, drafting the initial manuscript and manuscript revisions. AL, GL contributed to abstract screening. MJ and DK contributed to idea generation and manuscript revisions. CD contributed to idea generation, design of the study, abstract and full text screening, data analysis, manuscript revisions, and supervised the research. As corresponding author, CD is the guarantor of the article and attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Ethics approval.

This study did not involve human subjects.

## CRediT authorship contribution statement

**Morgan Vaterkowski:** Writing – review & editing, Writing – original draft, Methodology, Investigation, Conceptualization.
**Christel Daniel:** Writing – review & editing, Writing – original draft, Supervision, Methodology, Investigation, Conceptualization.
**Ambre LA Rosa:** Investigation, Conceptualization.
**Dipak Kalra:** Writing – review & editing.
**Marie-Christine Jaulent:** Writing – review & editing, Methodology.
**Guillaume Lame:** Writing – review & editing, Methodology.
**Emmanuelle Kempf:** Writing – review & editing, Writing – original draft, Validation, Supervision, Methodology, Data curation,

Conceptualization.

## Declaration of competing interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: [Morgan Vaterkowski was supported by a doctoral funding from Sanofi.].

## Acknowledgements

Morgan Vaterkowski was supported by a doctoral funding from Sanofi.

## Appendix A. Supplementary data

Supplementary data to this article can be found online at https://doi.org/10.1016/j.ijmedinf.2025.105900.

## References

[1] C.A. van den Bogert, P.C. Souverein, C.T.M. Brekelmans, S.W.J. Janssen, G. H. Koëter, H.G.M. Leufkens, L.M. Bouter, Recruitment failure and futility were the most common reasons for discontinuation of clinical drug trials, Results of a Nationwide Inception Cohort Study in the Netherlands, Journal of Clinical Epidemiology 88 (2017) 140–147, https://doi.org/10.1016/j.jclinepi.2017.05.001.

[2] S.J. Walters, I.B.D.A. Henriques-Cadby, O. Bortolami, L. Flight, D. Hind, R. M. Jacques, C. Knox, B. Nadin, J. Rothwell, M. Surtees, S.A. Julious, Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme, BMJ Open 7 (2017) e015276, https://doi.org/10.1136/bmjopen-2016-015276.

[3] S. Treweek, J.J. Francis, D. Bonetti, K. Barnett, M.P. Eccles, J. Hudson, C. Jones, N. B. Pitts, I.W. Ricketts, F. Sullivan, M. Weal, G. MacLennan, A primary care Web-based Intervention Modeling Experiment replicated behavior changes seen in earlier paper-based experiment, J. Clin. Epidemiol. 80 (2016) 116–122, https://doi.org/10.1016/j.jclinepi.2016.07.008.

[4] T. Adewole, B. Albrecht, S. Beg, O. Brawley, I.-F. Chang, B. Daly, A. de Claro, J. Dent, M. Dickler, A. Ene-Obong, L. Fashoyin-Aje, J. Finnegan, S. Galbraith, M. Gospodarowicz, J. Gralow, T. Haddad, C. Kalidas, A. Katz, K. Kaucic, H. Keane, M. Lanasa, J. Legos, B.T. Li, J. Mills, S. Oelrich, S. Olopade, J. Palma, R. Pazdur, P. Pilarski, J. Qian, K. Rudd, S. Shokrpour, C. Tendler, M. Wilson, V.W. Smart, Y.-L. Wu, J. Yan, L. Zhang, D. Li, J. Shen, A. Silverstein, G. Abou-Alfa, S. Anderson, P. Beyrer, D. Carnival, S. Clarke, T. Conneran, L.M. DeAngelis, C. Diakos, L. Diaz, C. H. dos Anjos, J.A. Drebin, D. Eisenmen, R. Flannery, D. Fredrickson, J.M. Isbell, D. R. Jones, A. Joseph, C. Kennedy, T.P. Kingham, A. Lee, N.Y. Lee, L. Leigh, Z.-Z. Li, A.-W. Liu, D. Liu, S.-Y. Liu, S.-Y.-M. Liu, V. Makker, M. Mallafore-Larrosa, S. McBride, I. Mestres, S. Mondaca, P. Nigro, L. Norton, J.V. Oyler, N. Pavlakis, S. Powell, B. Robinson, Y. Qin, P. Razavi, J. Reis-Filho, G. Rocco, C.M. Rudin, O. Schell, H.I. Scher, D. Schrag, G. Simon, D. Smith, M. Speece, H.-Y. Tu, A. Umar-Sadiq, B.-H. Wang, H. Wheeler, C.-R. Xu, L.-X. Yan, F. Yang, B. Yoon, C. Young, J.-T. Zhang, Q. Zhou, Advancing Global Health Equity in Oncology Clinical Trial Access, Cancer Discov. (2024) OF1–OF7, https://doi.org/10.1158/2159-8290-cd-24-1288.

[5] A.M. McDonald, R.C. Knight, M.K. Campbell, V.A. Entwistle, A.M. Grant, J.A. Cook, D.R. Elbourne, D. Francis, J. Garcia, I. Roberts, C. Snowdon, What influences recruitment to randomised controlled trials? A Review of Trials Funded by Two UK Funding Agencies, Trials 7 (2006) 9, https://doi.org/10.1186/1745-6215-7-9.

[6] M. Briel, K.K. Olu, E. von Elm, B. Kasenda, R. Alturki, A. Agarwal, N. Bhatnagar, S. Schandelmaier, A systematic review of discontinued trials suggested that most reasons for recruitment failure were preventable, J. Clin. Epidemiology 80 (2016) 8–15, https://doi.org/10.1016/j.jclinepi.2016.07.016.

[7] J.M. Watson, D.J. Torgerson, Increasing recruitment to randomised trials: a review of randomised controlled trials, BMC Méd. Res. Methodol. 6 (2006) 34, https://doi.org/10.1186/1471-2288-6-34.

[8] J. Fletcher, W. Hogg, B. Farrell, K. Woodend, S. Dahrouge, J. Lemelin, W. Dalziel, Effect of nurse practitioner and pharmacist counseling on inappropriate medication use in family practice, Can. Fam. Physician Med. Fam. Can. 58 (2012) 862–868.

[9] B. Parkinson, R. Meacock, M. Sutton, E. Fichera, N. Mills, G.W. Shorter, S. Treweek, N.L. Harman, R.C.H. Brown, K. Gillies, P. Bower, Designing and using incentives to support recruitment and retention in clinical trials: a scoping review and a checklist for design, Trials 20 (2019) 624, https://doi.org/10.1186/s13063-019-3710-z.

[10] M.J. Lamberti, Z. Smith, R. Henry, D. Howe, M. Goodwin, A. Williams, K. Getz, Benchmarking Patient Recruitment and Retention Practices, Ther. Innov. Regul. Sci. 55 (2021) 19–32, https://doi.org/10.1007/s43441-020-00186-4.

[11] D. Townsend, N. Mills, J. Savović, J.L. Donovan, A systematic review of training programmes for recruiters to randomised controlled trials, Trials 16 (2015), https://doi.org/10.1186/s13063-015-0908-6.
[12] R.J. Lacey, R. Wilkie, G. Wynne-Jones, J.L. Jordan, E. Wersocki, J. McBeth, Evidence for strategies that improve recruitment and retention of adults aged 65 years and over in randomised trials and observational studies: a systematic review, Age Ageing 46 (2017) 895-903, https://doi.org/10.1093/ageing/afx057.

[13] M. Cuggia, P. Besana, D. Glasspool, Comparing semi-automatic systems for recruitment of patients to clinical trials, Int. J. Méd. Inform. 80 (2011) 371-388, https://doi.org/10.1016/j.ijimedinf.2011.02.003.

[14] J.R. Rogers, J. Lee, Z. Zhou, Y.K. Cheung, G. Hripcsak, C. Weng, Contemporary use of real-world data for clinical trial conduct in the United States: a scoping review, J. Am. Méd. Inform. Assoc. 28 (2020) 144-154, https://doi.org/10.1093/jamia/joa224.

[15] L.R. Kalankesh, E. Monaghesh, Utilization of EHRs for clinical trials: a systematic review, BMC Méd. Res. Methodol. 24 (2024) 70, https://doi.org/10.1186/s12874-024-02177-7.

[16] G.K. Frampton, J. Shepherd, K. Pickett, G. Griffiths, J.C. Wyatt, Digital tools for the recruitment and retention of participants in randomised controlled trials: a systematic map, Trials 21 (2020) 478, https://doi.org/10.1186/s13063-020-04358-3.

[17] A.C. Tricco, E. Lillie, W. Zarin, K.K. O'Brien, H. Colquhoun, D. Levac, D. Moher, M. D.J. Peters, T. Horsley, L. Weeks, S. Hempel, E.A. Akl, C. Chang, J. McGowan, L. Stewart, L. Hartling, A. Aldcroft, M.G. Wilson, C. Garrity, S. Lewin, C. M. Godfrey, M.T. Macdonald, E.V. Langlois, K. Soares-Weiser, J. Morarty, T. Clifford, Ö. Tunçalç, S.E. Straus, PRISMA Extension for Scoping Reviews (PRISMA-ScR): Checklist and Explanation, Ann. Intern. Med. 169 (2018) 467-473, https://doi.org/10.7326/m18-0850.

[18] N.M. Jain, A. Culley, T. Knoop, C. Micheel, T. Osterman, M. Levy, Conceptual Framework to Support Clinical Trial Optimization and End-to-End Enrollment Workflow, JCO Clin. Cancer Inform. 3 (2019) CCI.19.00033, https://doi.org/10.1200/cci.19.00033.

[19] J. Zeng, M.A. Shufean, Y. Khotskaya, D. Yang, M. Kahle, A. Johnson, V. Holla, N. Sánchez, K.R.M. Shaw, E.V. Bernstm, F. Meric-Bernstm, OCTANE: Oncology Clinical Trial Annotation Engine, JCO Clin. Cancer Inform. 3 (2019) 1-11, https://doi.org/10.1200/cci.18.00145.

[20] C.R. Horowitz, T. Sabin, M. Ramos, L.D. Richardson, D. Hauser, M. Robinson, K. Fei, Successful recruitment and retention of diverse participants in a genomics clinical trial: a good invitation to a great party, Genet. Med. 21 (2019) 2364-2370, https://doi.org/10.1038/s41436-019-0498-x.

[21] E. Pfaff, A. Lee, R. Bradford, J. Pae, C. Potter, P. Blue, P. Knoep, K. Thompson, C. L. Roumie, D. Crenshaw, R. Servis, D.A. DeWalt, Recruiting for a pragmatic trial using the electronic health record and patient portal: successes and lessons learned, J. Am. Méd. Inform. Assoc. 26 (2019) 44-49, https://doi.org/10.1093/jamia/ocy138.

[22] R. Pradhan, D. Abrahami, H. Yin, O.H.Y. Yu, V. Sahasrabudhe, E.B. Andre, L. Azoulay, Defining Clinically Relevant Target Populations Using Real-World Data to Guide the Design of Representative Antidiabetic Drug Trials, Clin. Pharmacol. Ther. 109 (2021) 1219-1223, https://doi.org/10.1002/cpt.2213.

[23] W.B. van Dijk, A.T.L. Fiolet, E. Schuit, A. Sammani, T.K.J. Groenhof, R. van der Graaf, M.C. de Vries, M. Alings, J. Schaap, F.W. Asselbergs, D.E. Grobbee, R.H. H. Groenwold, A. Mosterd, Text-mining in electronic health records can be used as efficient tool for screening and data collection in cardiovascular trials: a multicenter validation study, J. Clin. Epidemiology 132 (2021) 97-105, https://doi.org/10.1016/j.jclinepi.2020.11.014.

[24] J.W. London, B. Balestrucci, D. Chatterjee, T. Zhan, Design-phase prediction of potential cancer clinical trial accrual success using a research data mart, J. Am. Méd. Inform. Assoc. 20 (2013) e260-e266, https://doi.org/10.1136/amiajnl-2013-001846.

[25] I. Soto-Rey, B. Trinczek, Y. Girardeau, E. Zapletal, N. Ammour, J. Doods, M. Dugas, F. Fritz, Efficiency and effectiveness evaluation of an automated multi-country patient count cohort system, BMC Méd. Res. Methodol. 15 (2015) 44, https://doi.org/10.1186/s12874-015-0035-9.

[26] G. Thibault-Halman, C.S. Rivers, C.S. Bailey, E.C. Tsai, B. Drew, V.K. Noonan, M. G. Fehlings, M.F. Dvorak, D. Kuerban, B.K. Kwon, S.D. Christie, T.R. Network, Predicting Recruitment Feasibility for Acute Spinal Cord Injury Clinical Trials in Canada Using National Registry Data, J. Neurotrauma 34 (2017) 599-606, https://doi.org/10.1089/neu.2016.4568.

[27] S.J. Nelson, B. Drury, D. Hood, J. Harper, T. Bernard, C. Weng, N. Kennedy, B. LaSalle, R. Gouripeddi, C.H. Wilkins, P. Harris, EHR-based cohort assessment for multicenter RCTs: a fast and flexible model for identifying potential study sites, J. Am. Méd. Inform. Assoc. 29 (2022) 652-659, https://doi.org/10.1093/jamia/ocab265.

[28] M.H. Eubank, D.M. Hyman, A.D. Kanakamedala, S.M. Gardos, J.M. Wills, P. D. Stetson, Automated eligibility screening and monitoring for genotype-driven precision oncology trials, J. Am. Méd. Inform. Assoc. 23 (2016) 777-781, https://doi.org/10.1093/jamia/ocw020.

[29] N.V.J. Alexander, C.A. Brunette, E.T. Guardino, T. Yi, B.J. Kerman, K. MacIsaac, E. J. Harris, A.A. Antwi, J.L. Vassy, Performance of EHR classifiers for patient eligibility in a clinical trial of precision screening, Contemp. Clin. Trials 121 (2022) 106926, https://doi.org/10.1016/j.cct.2022.106926.

[30] J.T. Beck, M. Rammage, G.P. Jackson, A.M. Preininger, I. Dankwa-Mullan, M. C. Roebuck, A. Torres, H. Holtzen, S.E. Coverdill, M.P. Williamson, S. Chau, K. Rhee, M. Vinegра, Artificial Intelligence Tool for Optimizing Eligibility Screening for Clinical Trials in a Large Community Cancer Center, JCO Clin. Cancer Inform. 4 (2020) 50-59, https://doi.org/10.1200/cci.19.00079.

[31] D. Calaprice-Whitty, K. Galil, W. Salloum, A. Zariv, B. Jimenez, Improving Clinical Trial Participant Prescreening With Artificial Intelligence (AI): A Comparison of the

Results of AI-Assisted vs Standard Methods in 3 Oncology Trials, Ther. Innov. Regul. Sci. 54 (2020) 69-74, https://doi.org/10.1007/s43441-019-00030-4.

[32] L. Chen, Y. Gu, X. Ji, C. Lou, Z. Sun, H. Li, Y. Gao, Y. Huang, Clinical trial cohort selection based on multi-level rule-based natural language processing system, J. Am. Méd. Inform. Assoc. 26 (2019) 1218-1226, https://doi.org/10.1093/jamia/ocz109.

[33] C.-J. Chen, N. Warikoo, Y.-C. Chang, J.-H. Chen, W.-L. Hsu, Medical knowledge infused convolutional neural networks for cohort selection in clinical trials, J. Am. Méd. Inform. Assoc. 26 (2019) 1227-1236, https://doi.org/10.1093/jamia/ocz128.

[34] M. Cuggia, B. Campillo-Gimenez, G. Bouzille, P. Besana, W. Jouini, J.-C. Dufour, O. Zekri, I. Gibaud, C. Garde, R. Duvaufierer, Automatic Selection of Clinical Trials Based on A Semantic Web Approach, Stud. Health. Technol. Inform. 216 (2015) 564-568, https://doi.org/10.3233/978-1-61499-564-7-564.

[35] S.R. Jonnalagadda, A.K. Adupa, R.P. Garg, J. Corona-Cox, S.J. Shah, Text Mining of the Electronic Health Record: An Information Extraction Approach for Automated Identification and Subphenotyping of HPEF Patients for Clinical Trials, J. Cardiovasc. Transl. Res. 10 (2017) 313-321, https://doi.org/10.1007/s12265-017-9752-2.

[36] F. Köpcke, D. Lubgan, R. Fietkau, A. Scholler, C. Nau, M. Stürzl, R. Croner, H.-U. Prokosch, D. Toddtenroth, Evaluating predictive modeling algorithms to assess patient eligibility for clinical trials from routine data, BMC Méd. Inform. Decis. Mak. 13 (2013) 134, https://doi.org/10.1186/1472-6947-13-134.

[37] N. Mastellos, G. Bližniuk, D. Czopnik, M. McGilchrist, A. Misiaszek, P. Bródka, V. Curcin, J. Car, B.C. Delaney, A. Andresson, Feasibility and acceptability of TRANSFORM to improve clinical trial recruitment in primary care, Fam. Pr. 33 (2016) 186-191, https://doi.org/10.1093/fampra/cmv102.

[38] G. Melzer, T. Maiwald, H.-U. Prokosch, T. Ganslandt, Leveraging Real-World Data for the Selection of Relevant Eligibility Criteria for the Implementation of Electronic Recruitment Support in Clinical Trials, Appl. Clin. Inform. 12 (2021) 017-026, https://doi.org/10.1055/s-0040-1721010.

[39] H.N. Miller, J. Charleston, C. Wu, K. Gleason, C.R. White, C.R.D. Himmelfarb, D. E. Ford, T.B. Plante, A.C. Gelber, L.J. Appel, E.R. Miller, S.P. Juraschek, Use of electronic recruitment methods in a clinical trial of adults with gout, Clin. Trials 18 (2021) 92-103, https://doi.org/10.1177/1740774520956969.

[40] H.N. Miller, K.T. Gleason, S.P. Juraschek, T.B. Plante, C. Lewis-Land, B. Woods, L. J. Appel, D.E. Ford, C.R.D. Himmelfarb, Electronic medical record-based cohort selection and direct-to-patient, targeted recruitment: early efficacy and lessons learned, J. Am. Méd. Inform. Assoc. 26 (2019) 1209-1217, https://doi.org/10.1093/jamia/ocz168.

[41] R. Miotto, C. Weng, Case-based reasoning using electronic health records efficiently identifies eligible patients for clinical trials, J. Am. Méd. Inform. Assoc. 22 (2015) e141-e150, https://doi.org/10.1093/jamia/ocu050.

[42] Y. Ni, J. Wright, J. Perentesis, T. Lingren, L. Deleger, M. Kaiser, I. Kohane, I. Solti, Increasing the efficiency of trial-patient matching: automated clinical trial eligibility Pre-screening for pediatric oncology patients, BMC Méd. Inform. Decis. Mak. 15 (2015) 28, https://doi.org/10.1186/s12911-015-0149-3.

[43] Y. Ni, S. Kennebeck, J.W. Dexheimer, C.M. McAneney, H. Tang, T. Lingren, Q. Li, H. Zhai, I. Solti, Automated clinical trial eligibility prescreening: increasing the efficiency of patient identification for clinical trials in the emergency department, J. Am. Méd. Inform. Assoc. 22 (2015) 166-178, https://doi.org/10.1136/amiajnl-2014-002887.

[44] A.E. Peters, M.O. Ogunniyi, S.M. Hegde, C. Bianco, S. Ghafghi, A.F. Hernandez, A.D. DeVore, A multicenter program for electronic health record screening for patients with heart failure with preserved ejection fraction: Lessons from the DELIVER-EHR initiative, Contemp. Clin. Trials 121 (2022) 106924, https://doi.org/10.1016/j.cct.2022.106924.

[45] T.B. Plante, K.T. Gleason, H.N. Miller, J. Charleston, K. McArthur, C. D. Himmelfarb, M. Lazo, D.E. Ford, E.R. Miller, L.J. Appel, S.P. Juraschek, N. Cronin, S. McClure, J. Miskimon, C.M. Mitchell, R.R. Kalyani, D.L. Roth, J. A. Schrack, S.L. Szanton, J. Urbanek, J. Walston, A. Wanigatunga, S.N. Baksh, A. L. Blackford, S. Chattopadhyay, L.T. Drye, J. Dodge, C. Ewing, S. Haider, S. C. Holland, R. Jackson, A. Lears, C. Meinert, D. Shade, M. Smith, A.L. Sternberg, J. Tonascia, M.L.V. Natta, A. Wagoner, G.W. Comstock, E.D. Michos, J.D. Bennett, P. Bowers, J. Coresh, P. Crowley, T. Crunkleton, B. Dick, R. Evans, M. Godwin, L. Hammann, D. Hawks, K. Horning, E. Hull, B. Mills, M. Minotti, L. Raley, A. Reed, R. Reeder, C. Reid, M. Shuda, A. Spikes, R. Stouffer, K. Weicht, C. Abbas, B. Carey, S. Davis, N. DeRoche-Brown, D. Gayles, S. Holland, I. Glenn-Smith, D. Johnson, M. Johnson, E. Keyes, S. Santiago, C. Sapun, V. Sneed, L. Swartz, L. Thomas, H. Christenson, S.-H. Duh, H. Rebuck, C. Rosen, T. Cook, P. Duncan, C. Hansen, A. Kenny, S. Saphes, J. Hannah, S. Romashkan, C.D. Davis, C.T. Sempos, J. M. Guralnik, J.C. Gallagher, Recruitment of trial participants through electronic medical record patient portal messaging: A pilot study, Clin. Trials 17 (2020) 30-38, https://doi.org/10.1177/1740774519873657.

[46] C. Scherer, S. Endres, M. Orban, S. Kääb, S. Massberg, A. Winter, M. Löbe, Implementation of a Clinical Trial Recruitment Support System Based on Fast Healthcare Interoperability Resources (FHIR) in a Cardiology Department, Stud. Health. Technol. Inform. 29 (2022) 440-444, https://doi.org/10.3233/shti220497.

[47] I. Segura-Bedmar, P. Raez, Cohort selection for clinical trials using deep learning models, J. Am. Méd. Inform. Assoc. 26 (2019) 1181-1188, https://doi.org/10.1093/jamia/ocz139.

[48] C. Shivade, C. Hebert, M. Lopetegui, M.-C. de Marneffe, E. Fosler-Lussier, A.M. Lai, Textual inference for eligibility criteria resolution in clinical trials, J. Biomed. Inform. 58 (2015) S211-S218, https://doi.org/10.1016/j.jbi.2015.09.008.

[49] L.E. Simon, A.S. Rauchwerger, U.K. Chettipally, L. Babakhanian, D.R. Vinson, E. M. Warton, M.E. Reed, A.B. Kharbanda, E.O. Kharbanda, D.W. Ballard, Text
message alerts to emergency physicians identifying potential study candidates increase clinical trial enrollment, J. Am. Méd. Inform. Assoc. 26 (2019) 1360-1363, https://doi.org/10.1093/jamia/ocz118.

[50] H.C. Tissot, A.D. Shah, D. Brealey, S. Harris, R. Agbakoba, A. Folarin, L. Romao, L. Roguski, R. Dobson, F.W. Asselbergs, Natural Language Processing for Mimicking Clinical Trial Recruitment in Critical Care: ASemi-Automated Simulation Based on the LeoPARDS Trial, IEEE J. Biomed. Heal. Inform. 24 (2020) 2950-2959, https://doi.org/10.1109/jbhi.2020.2977925.

[51] V.G.V. Vydiswaran, A. Strayhorn, X. Zhao, P. Robinson, M. Agarwal, E. Bagazinski, M. Essiet, B.E. Iott, H. Joo, P. Ko, D. Lee, J.X. Lu, J. Liu, A. Murali, K. Sasagawa, T. Wang, N. Yuan, Hybrid bag of approaches to characterize selection criteria for cohort identification, J. Am. Méd. Inform. Assoc. 26 (2019) 1172-1180, https://doi.org/10.1093/jamia/ocz079.

[52] Y. Xiong, X. Shi, S. Chen, D. Jiang, B. Tang, X. Wang, Q. Chen, J. Yan, Cohort selection for clinical trials using hierarchical neural network, J. Am. Méd. Inform. Assoc. 26 (2019) 1203-1208, https://doi.org/10.1093/jamia/ocz099.

[53] V.I. Petkov, L.T. Penberthy, B.A. Dahman, A. Poklepovic, C.W. Gillam, J. H. McDermott, Automated determination of metastases in unstructured radiology reports for eligibility screening in oncology clinical trials, Exp. Biol. Med. 238 (2013) 1370-1378, https://doi.org/10.1177/1535370213508172.

[54] N.J. Dobbins, B. Han, W. Zhou, K.F. Lan, H.N. Kim, R. Harrington, Ö. Uzuner, M. Yetisgen, LeafAI: query generator for clinical cohort discovery rivaling a human programmer, J. Am. Méd. Inform. Assoc. 30 (2023) 1954-1964, https://doi.org/10.1093/jamia/ocad149.

[55] A. Schwab, A.M. Lin, S. Winkler, Automated Clinical Trial Cohort Definition and Evaluation with CQL and CDS-Hooks, Stud. Heal. Technol. Inform. 313 (2024) 149-155, https://doi.org/10.3233/shti240028.

[56] S.P. Shriver, W. Arafat, C. Potteiger, D.L. Butler, M.S. Beg, M. Hullings, S. Semy, Z. Lister, L. Khosama, S. Armstrong, D. Hadley, J. Pappa, M.E. Fleury, Feasibility of institution-agnostic, EHR-integrated regional clinical trial matching, Cancer 130 (2024) 60-67, https://doi.org/10.1002/cncr.35022.

[57] K. Wang, H. Cui, Y. Zhu, X. Hu, C. Hong, Y. Guo, L. An, Q. Zhang, L. Liu, Evaluation of an artificial intelligence-based clinical trial matching system in Chinese patients with hepatocellular carcinoma: a retrospective study, BMC Cancer 24 (2024) 246, https://doi.org/10.1186/s12885-024-11959-7.

[58] K. Lee, Y. Mai, Z. Liu, K. Raja, T. Jun, M. Ma, T. Wang, L. Ai, E. Calay, W. Oh, E. Schadt, X. Wang, CriteriaMapper: establishing the automatic identification of clinical trial cohorts from electronic health records by matching normalized eligibility criteria and patient clinical characteristics, Sci. Rep. 14 (2024) 25387, https://doi.org/10.1038/s41598-024-77447-x.

[59] S. Ghosh, H.M. Abushukair, A. Ganesan, C. Pan, A.R. Naqash, K. Lu, Harnessing explainable artificial intelligence for patient-to-clinical-trial matching: A proof-of-concept pilot study using phase I oncology trials, PLoS One 19 (2024) e0311510, https://doi.org/10.1371/journal.pone.0311510.

[60] Y. Xiong, W. Peng, Q. Chen, Z. Huang, B. Tang, A Unified Machine Reading Comprehension Framework for Cohort Selection, IEEE J. Biomed. Heal. Inform. 26 (2022) 379-387, https://doi.org/10.1109/jbhi.2021.3095478.

[61] R. Alonso-Calvo, D. Perez-Rey, S. Paraiso-Medina, B. Claerhout, P. Hennebert, A. Bucur, Enabling semantic interoperability in multi-centric clinical trials on breast cancer, Comput. Methods Programs Biomed. 118 (2015) 322-329, https://doi.org/10.1016/j.cmpb.2015.01.003.

[62] M. Weisskopf, G. Bucklar, J. Blaser, Tools in a clinical information system supporting clinical trials at a Swiss University Hospital, Clin. Trials 11 (2014) 673-680, https://doi.org/10.1177/1740774514546702.

[63] P.V. Kukhareva, C. Weir, G.D. Fiol, G.A. Aarons, T.Y. Taft, C.R. Schlechter, T. J. Reese, R.L. Curran, C. Nanjo, D. Borbolla, C.J. Staes, K.L. Morgan, H.S. Kramer, C.H. Stipelman, J.H. Shakib, M.C. Flynn, K. Kawamoto, Evaluation in Life Cycle of Information Technology (ELICIT) framework: Supporting the innovation life cycle from business case assessment to summative evaluation, J. Biomed. Inform. 127 (2022) 104014, https://doi.org/10.1016/j.jbi.2022.104014.

[64] E. Kempf, M. Vaterkowski, D. Leprovost, N. Griffon, D. Ouagne, S. Breant, P. Serre, A. Mouchet, B. Rance, G. Chatellier, A. Bellamine, M. Frank, J. Guerin, X. Tannier, A. Livartowski, M. Hilka, C. Daniel, How to Improve Cancer Patients ENrollment in Clinical Trials From rEal-Life Databases Using the Observational Medical Outcomes Partnership Oncology Extension: Results of the PENELOPE Initiative in Urologic Cancers, JCO Clin. Cancer Inform. 7 (2023) e2200179, https://doi.org/10.1200/ccci.22.00179.

[65] Y. Fang, B. Idnay, Y. Sun, H. Liu, Z. Chen, K. Marder, H. Xu, R. Schnall, C. Weng, Combining human and machine intelligence for clinical trial eligibility querying, J. Am. Méd. Inform. Assoc. 29 (2022) 1161-1171, https://doi.org/10.1093/jamia/ocac051.

[66] J. Park, Y. Fang, C. Ta, G. Zhang, B. Idnay, F. Chen, D. Feng, R. Shyu, E.R. Gordon, M. Spotnitz, C. Weng, Criteria2Query 3.0: Leveraging generative large language models for clinical trial eligibility query generation, J. Biomed. Inform. 154 (2024) 104649, https://doi.org/10.1016/j.jbi.2024.104649.