The “David Vs Goliath” Study: Application of Large Language Models (LLM) for Automatic Medical Information Retrieval from Multiple Data Sources to Accelerate Clinical and Translational Research in Hematology
Authors: Delleani Mattia , D'Amico Saverio , Sauta Elisabetta , Asti Gianluca , Zazzetti Elena , Campagna Alessia , Lanino Luca , Maggioni Giulia , Grondelli Maria Chiara , Forcina Barrero Alessandro , Morandini Pierandrea , Ubezio Marta , Todisco Gabriele , Russo Antonio , Tentori Cristina Astrid , Buizza Alessandro , Bonometti Arturo , Lancellotti Cesare , Di Tommaso Luca , Rahal Daoud , Bicchieri Marilena , Savevski Victor , Santoro Armando , Santini Valeria , Sole Francesc , Platzbecker Uwe , Fenaux Pierre , Diez-Campelo Maria , Komrokji Rami S. , Garcia-Manero Guillermo , Haferlach Torsten , Kordasti Shahram , Zeidan Amer M. , Castellani Gastone , Della Porta Matteo Giovanni 
Published: December 2024

Background. The innovation process in hematology requires access to a large amount of healthcare data. However, 97% of patient data produced by hospitals remains unused (Source: Deloitte, Health Data, 2023), primarily due to privacy limitations, lack of data harmonization from different sources, and the unstructured and dispersed nature of the information.

Large Language Models (LLM) are computational models capable of performing general-purpose language generation and other natural language processing tasks. These models acquire these abilities by learning statistical relationships from vast amounts of text through a computationally intensive self-supervised and semi-supervised training process. LLMs have been increasingly utilized in healthcare to enhance diagnostics, streamline patient interactions, and improve overall clinical workflows. In this project, we analyze the potential of Artificial Intelligence (AI) solutions based on LLM for data retrieval, extraction and generation to create standardized datasets to accelerate clinical and translational research in blood diseases in hematology.

Aims. The “David vs Goliath” study was conducted by Synthema EU consortium with the following aims to: 1) develop AI solution leveraging LLM for information retrieval, extraction and generation of research-ready datasets from multiple medical sources; 2) evaluate clinical and statistical fidelity of AI-retrieved dataset through a specific Validation Framework (VF); 3) validate the reliability of AI-retrieved dataset to build personalized prognostic models.

Methods. We proposed ARISTOTELES an Automatic Retrieval Information System TO acceleraTE clinical and translationaL research in hEmatological malignancieS. This solution has three main components: a Retrieval-Augmented Generation (RAG) system for information retrieval; an LLM for data extraction and a Generative Pretrained Transformer (GPT) for missing data inference. The RAG component, also leveraging a dedicated LLM model, was implemented to search information across multiple data sources from the Humanitas Research Hospital DataLake (a fully privacy compliant environment) and to enhance the quality of retrieved information. The information provided by RAG was then extracted by a second hematological-tuned LLM model into a structured dataset in common data model format. Finally, the GPT model, trained on hematological data, was then used to generate complete data, conditioned on partially extracted patients' information.

Results. The original dataset (Goliath) comprises 1167 patients with myeloid neoplasms (MN) from Humanitas Research Hospital, including multiple layers of information with comprehensive demographic, clinical and genomic data (cytogenetics and mutational screening) alongside treatment and outcome. The AI-retrieved dataset (David) was generated by applying ARISTOTELES on medical records from the same MN patients.

The comparison of the two datasets was performed by a specific validation framework (based on PMID:38875514). Distributions and correlations for clinical, demographic, genomic and cytogenetic in both datasets were comparable with 91% of fidelity. Mutation distribution and pairwise association among genes and/or cytogenetics abnormalities resulted in 90.1% of fidelity.

No significant statistical difference between the two datasets has been observed by comparing the survival curves with a Kaplan-Meier model with a log-rank test in patients stratified according to clinical labels. Finally, we performed Cox proportional hazards analyses (Cox-PH) including clinical and genomic information from the David vs Goliath datasets to compare their performance (concordance index, CI). Considering overall survival as a clinical endpoint the CI of Cox-PH models was 0.75 and 0.74 respectively.

Conclusion. ARISTOTELES solution allows automatic information retrieval, extraction and structuring of complex multimodal healthcare data. AI-retrieved data (David) resulted in high clinical and statistical fidelity with respect to the original dataset (Goliath). Overall, this results in increasing access to healthcare data and reducing human effort required for data collection tasks, thereby accelerating clinical research in hematology.