Developing machine learning algorithms incorporating patient reported outcome measures to predict acute care events among patients with cancer.
Authors: Roberts Thomas J , McGuire Jessica , Temel Jennifer S. , Kamdar Mihir , Greer Joseph A. , Mulvey Therese Marie 
Published: June 2023

1516 Background: Clinical trials have shown that collecting patient-reported outcome measures (PROMs) can improve outcomes among oncology patients. However, there remains uncertainty about how to best collect and use PROMs in routine clinical care. Since 2016 we have collected PROMs for patients receiving cancer treatment within a large health system containing academic and community sites. We sought to examine whether combining PROMs with electronic health record (EHR) data improves the performance of machine learning algorithms predicting patients’ risk of 30-day acute care events. Methods: The sample included patients with solid tumors receiving curative- or palliative-intent systemic therapy between 9/1/20 and 7/31/22 who completed at least one PROMs survey. PROMs were collected via a 12-item questionnaire based on the Common Terminology Criteria for Adverse Events measurement system. Patients completed questionnaires via the EHR’s patient portal or on tablets in waiting rooms. EHR data included 176 variables describing demographics, laboratory values, vital signs, diagnoses, medications, and prior healthcare encounters. A binomial outcome was calculated for each patient encounter to indicate whether the patient had an acute care event (ED visit or hospitalization) in the subsequent 30 days. Data were randomly assigned to training (70%) and test (30%) sets and were used to train and test four models: EHR + PROMs, EHR alone, PROMs alone, and ‘PROMs plus,’ which included PROMs and minimal EHR data such as age, sex, and site of cancer treatment, which served as an accessible proxy for cancer type. For each model, we compared multiple architectures and selected the best-performing model for comparison. Results: Between September 2020 and June 2022, 4,193 patients completed 20,359 PROM surveys during outpatient encounters. There were 2,118 30-day acute care events within the cohort. The area under the receiver operating curves (AUCs) were 0.84 (95% CI 0.81 to 0.86) for the EHR + PROMs model, 0.82 (95% CI 0.80 to 0.84) for the EHR alone model, 0.67 (95% CI 0.65 to 0.69) for the PROMs alone model, and 0.79 (95% CI 0.77 to 0.82) for the PROMs plus model. In the EHR alone and EHR + PROMs models, the most important variables were last albumin, white blood cell count and hemoglobin. In the PROMs plus model, the most important variables were age, treatment site, patient-reported pain, and patient-reported fatigue. Conclusions: Incorporating PROMs into machine learning models to predict risk of acute care events can improve model performance. Notably, the model combining PROMs with a small number of easily accessible EHR variables performed nearly as well as models including all EHR data. These models demonstrate the predictive value of PROMs in oncology patients and they lay the foundation for future interventions aiming to reduce acute care events among high-risk patients.