Enhancing Real-World Data Extraction in Clinical Research: Evaluating the Impact of the Implementation of Large Language Models in Hospital Settings
Authors: Wang Bin , Lai Junkai , Cao Han , Jin Feifei , Li Qiang , Tang Mingkun , Yao Chen , Zhang Ping 
Published: November 2023

Background
The application of artificial intelligence (AI) and large language models (LLMs) in the medical sector has become increasingly common. The widespread adoption of electronic health record (EHR) platforms has created demand for the efficient extraction and analysis of unstructured data, which are known as real-world data (RWD). The rapid increase in free-text data in the medical context has highlighted the significance of natural language processing (NLP) with regard to extracting insights from EHRs, identifying this process as a crucial tool in clinical research. The development of LLMs that are specifically designed for biomedical and clinical text mining has further enhanced the capabilities of NLP in this domain. Despite these advancements, the utilization of LLMs specifically in clinical research remains limited.
Objective
This study aims to assess the feasibility and impact of the implementation of an LLM for RWD extraction in hospital settings. The primary focus of this research is on the effectiveness of LLM-driven data extraction as compared to that of manual processes associated with the electronic source data repositories (ESDR) system. Additionally, the study aims to identify challenges emerging in the context of LLM implementation and to obtain practical insights from the field.
Methods
The researchers developed the ESDR system, which integrates LLMs, electronic case report forms (eCRFs) and EHRs. The Paroxysmal Atrial Tachycardia Project, a single-center retrospective cohort study, served as a pilot case. This study involved deploying the ESDR system on the hospital local area network (LAN). Localized LLM deployment utilized the Chinese open-source ChatGLM model. The research design compared the AI-assisted process with manual processes associated with the ESDR in terms of accuracy rates and time allocation. Five eCRF forms, predominantly including free-text content, were evaluated; the relevant data focused on 630 subjects, in which context a 10% sample (63 subjects) was used for assessment. Data collection involved electronic medical and prescription records collected from 13 departments.
Results
While the discharge medication form achieved 100% data completeness, some free-text forms exhibited data completeness rates below 20%. The AI-assisted process was associated with an estimated efficiency improvement of 80.7% in eCRF data transcription time. The AI data extraction accuracy rate was 94.84%, and errors were related mainly to localized Chinese clinical terminology. The study identified challenges pertaining to prompt design, prompt output consistency, and prompt output verification. Addressing limitations in terms of clinical terminology and output inconsistency entails integrating local terminology libraries and offering clear examples of output format. Output verification can be enhanced by probing the model's reasoning, assessing confidence on a scale, and highlighting relevant text snippets. These measures mitigate challenges that can impede our understanding of the model's decision-making process with regard to extensive free-text documents.
Conclusions
This research enriches academic discourse on LLMs in the context of clinical research and provides actionable recommendations for the practical implementation of LLMs for RWD extraction. By offering insights into LLM integration in the context of clinical research systems, the study contributes to the task of establishing a secure and efficient framework for digital clinical research. The continuous evolution and optimization of LLM technology are crucial for its seamless integration into the broader landscape of clinical research.