Towards global model generalizability: independent cross-site feature evaluation for patient-level risk prediction models using the OHDSI network

Authors:
raw_author_name: Behzad Naderalvojoud
raw_affiliation_string: Department of Medicine, Stanford University , Stanford, CA 94305, United States
raw_author_name: Chen Yanover
raw_affiliation_string: KI Research Institute , Kfar Malal, 4592000, Israel
raw_author_name: Tal El-Hay
raw_affiliation_string: KI Research Institute , Kfar Malal, 4592000, Israel
raw_author_name: Byungjin Choi
raw_affiliation_string: Department of Biomedical Informatics, Ajou University Graduate School of Medicine , Suwon, 16499, Korea
raw_author_name: Rae Woong Park
raw_affiliation_string: Department of Biomedical Informatics, Ajou University Graduate School of Medicine , Suwon, 16499, Korea
raw_author_name: Mary Pat Reeve
raw_affiliation_string: Institute for Molecular Medicine Finland (FIMM), HiLIFE, University of Helsinki , Helsinki, 00014, Finland
raw_author_name: Keith Humphreys
raw_affiliation_string: Center for Innovation to Implementation, Veterans Affairs Palo Alto Health Care System , Palo Alto, CA 94304, United States; Department of Psychiatry and the Behavioral Sciences, Stanford University , Stanford, CA 94305, United States
raw_author_name: Tina Hernandez-Boussard
raw_affiliation_string: Department of Medicine, Stanford University , Stanford, CA 94305, United States

Abstract: Abstract Background Predictive models show promise in healthcare, but their successful deployment is challenging due to limited generalizability. Current external validation often focuses on model performance with restricted feature use from the original training data, lacking insights into their suitability at external sites. Our study introduces an innovative methodology for evaluating features during both the development phase and the validation, focusing on creating and validating predictive models for post-surgery patient outcomes with improved generalizability. Methods Electronic health records (EHRs) from 4 countries (United States, United Kingdom, Finland, and Korea) were mapped to the OMOP Common Data Model (CDM), 2008-2019. Machine learning (ML) models were developed to predict post-surgery prolonged opioid use (POU) risks using data collected 6 months before surgery. Both local and cross-site feature selection methods were applied in the development and external validation datasets. Models were developed using Observational Health Data Sciences and Informatics (OHDSI) tools and validated on separate patient cohorts. Results Model development included 41 929 patients, 14.6% with POU. The external validation included 31 932 (UK), 23 100 (US), 7295 (Korea), and 3934 (Finland) patients with POU of 44.2%, 22.0%, 15.8%, and 21.8%, respectively. The top-performing model, Lasso logistic regression, achieved an area under the receiver operating characteristic curve (AUROC) of 0.75 during local validation and 0.69 (SD = 0.02) (averaged) in external validation. Models trained with cross-site feature selection significantly outperformed those using only features from the development site through external validation (P &amp;lt; .05). Conclusions Using EHRs across four countries mapped to the OMOP CDM, we developed generalizable predictive models for POU. Our approach demonstrates the significant impact of cross-site feature selection in improving model performance, underscoring the importance of incorporating diverse feature sets from various clinical settings to enhance the generalizability and utility of predictive healthcare models.