Sensitivity, Specificity, Generalizability, and Reusability Aspirations for Machine Learning (ML) Models in mHealth

Mobile sensor big data collected from smartphones, smartwatches, fitness trackers, and other wearables can be mined for signatures (called mHealth biomarkers) of subtle changes in daily behaviors (e.g., mobility, gait, sleep, etc.) and/or physiology (e.g., heart function, breathing, sweating, etc.). Clinical adoption of these mHealth biomarkers can lead to potent temporally-precise interventions, enabling patients to initiate and sustain the healthy lifestyle choices and treatment regimes that are necessary to prevent and/or successfully manage the growing burden of multiple chronic conditions. However, for any new biomarker to be successfully used for clinical diagnosis or treatment, its clinical utility must be established. mHealth biomarkers are usually derived by training a machine learning (ML) algorithm on mobile sensor data. The published models differ in feature construction (domain-derived features fed to a supervised ML model vs. data-driven features discovered by a deep learning (DL) model), data collection setting (lab vs. field), data selection and preparation (covering all twenty-four hours of the day vs. awake hours, vs. only when performing certain tasks), data labeling (retrospective self-reported aggregate labels vs. time-synchronized labels from first-person video), data size and diversity (e.g., number of participants, gender, ethnicity, age group, number of days, hours per day, etc.), experiment design (cross-validation vs. cross-subject validation), and performance (e.g., accuracy, F1 score, confusion matrix, AUC, etc.). As a result, there is wide diversity in published models on their potential for reusability, generalizability, and eventual clinical utility. This talk will describe an aspirational framework for specificity, sensitivity, generalizability, and reusability of mHealth biomarkers with some concrete performance targets so that they have a higher chance of widespread clinical utility. It will draw upon the presenter's decade-long transdisciplinary research experience in developing machine learning models to detect a wide variety of daily behaviors such as stress, speaking, smoking, and brushing from wearable physiological and inertial sensors. The talk will use the analogy of five nines (i.e., 99.999%) paradigm in the area of service-level agreements to quantify high-availability. Similar to how these managed services are expected to be available 24-7-365, with the downtime limited to 5.26 minutes per year, we can express the performance requirements of mHealth biomarkers that are expected to detect subtle signs of health and behaviour deterioration anytime and anywhere. Five nines guarantee for the detection of a health event (e.g., fall, stress) translates to one false positive every 100 person-days, if the model runs on 1,000 minutes of sensor data collected each day. Achieving five nines to claim the detection of non-event (e.g., smoking abstinence) is even more challenging, as there are several other failure scenarios for missing an event, in addition to model failure, such as the non-wearing of sensors when performing the event of interest, poor data quality, mismatch of model to where (on the body) and how the sensor is worn (e.g., smartwatch on non-dominant hand), battery failure, and data loss. To achieve generalizability, the model performance must be achieved on independent test data that covers all aspects of daily life, without any data selection. Finally, for the model to be used by others for real-life, the model should not only be accessible to the community, it should also be possible to train and test the model on different datasets by independent non-ML-expert researchers.