Countdown Regression: Sharp and Calibrated Survival Predictions

1 INTRODUCTION . Having patient-specific predictions of time to an event such as mortality or bone fracture allows caregivers to make better informed decisions around patient care . Historically , prognosis scores have served as simple tools to stratify patient risk within a predefined time window ( Lau et al. , 2006 ; Cardona-Morrell & Hillman , 2015 ) . However , such models tend to be too simplistic to be widely useful . They are often estimated from a large population of patients , and do not take into account patient-specific information to make individualized predictions ( Yu et al. , 2011 ) . Meanwhile , the adoption of Electronic Health Record ( EHR ) systems over the past few decades has resulted in the collection of observational data on millions of patients spanning multiple years . This data enables development of patient-specific prediction models using machine learning . Such models are applicable to the larger patient population without being specific to a disease type or demographic , and this makes it possible to develop novel workflows in care delivery . For example , a high predicted probability of 3-12 month mortality could proactively notify palliative care teams of otherwise overlooked patients with end-of-life needs ( Avati et al. , 2017 ) . One way to obtain patient-specific survival predictions is to treat the problem as probabilistic classification ; that is , training a binary classifier to predict outcomes of event by a particular time of interest ( Avati et al. , 2017 ; Rajkomar et al. , 2018 ) . However , such an approach has drawbacks . First , the model is specific to the time of interest it was trained upon – it is not straightforward how to take a model that was trained to predict probabilities of 1-year mortality and obtain predictions of 6-month mortality from it . Second , it is not usually possible to use data on all patients – for example , if a patient has only 3 months of history in the EHR system , it is neither possible to include that patient as a positive case nor a negative case in the 1-year mortality prediction task . Third , the process of constructing the data set implicitly conditions on the future outcome to select prediction times – evaluation is performed only at times looking backward from the event of interest . It has been shown that evaluation metrics can be overly optimistic relative to real world performance as a result ( Sherman et al. , 2017 ) . An alternative approach to the problem is survival prediction ; that is , predicting time to event by estimating a distribution over future time . In this setting , traditional survival analysis methods such as the Cox proportional hazards model ( R. , 1972 ) or accelerated failure time models ( J . ) are capable of handling data with censored observations ( cases in which the event was not observed , but we know that the event did not occur up to a certain time ) . This addresses concerns raised by the classification approach , but there are a few nuances . First , traditional models typically make strong assumptions , such as proportional hazards or linearity . Second , challenges of low prevalence often arise when these methods are applied to large-scale observational datasets with heavy censoring , which is the case in real EHR data . Third , these survival analysis methods are typically evaluated as point estimates of risk , such as 10-year probabilities of events , rather than holistic measures of quality of the predicted distributions ( Goff et al. , 2014 ; Ranganath et al. , 2016 ; Lee et al. , 2018 ) . Common metrics of evaluation include the C-statistic ( Uno et al. , 2007 ) , log- ` 1 loss ( Yu et al. , 2011 ) , and mean-squared-error ( Katzman et al. , 2018 ) . While useful for the purposes of relative risk stratification , model comparisons made using point estimates leaves the quality of uncertainty in predicted distributions left unmeasured . If a point prediction is way off , it is penalized by the same amount whether the model was confident or not ( that is , whether the predicted distribution had low or high variance ) . In contrast , forecasts in the field of meteorology are typically made as full prediction distributions over all weather conditions given past and current observations ( Gneiting et al. , 2008 ) . Evaluation of predictive performance is assessed by the paradigm of maximizing the sharpness of the predictive distribution , subject to calibration ( Gneiting & Katzfuss , 2014 ) . The intuition behind this paradigm is that probabilities have to be calibrated in order to be correct . However , that does not necessarily make them useful ( one could always predict the marginal probability of an outcome without looking at the data , and still be well calibrated ) . The usefulness of a prediction distribution lies in its sharpness , or how well its mass concentrates . In summary , uncalibrated predictions ( sharp or not ) are useless , calibrated but non-sharp predictions are correct but less useful , and calibrated and sharp distributions are most useful . To improve the sharpness of prediction distributions in the survival setting , we propose the use of proper scoring rules beyond maximum likelihood as the training objective . Proper scoring rules are known to measure calibration , and any model trained with a proper scoring rule will tend to maintain calibration ( Gneiting & Katzfuss , 2014 ) . For our purposes , we focus on the continuous ranked probablity score ( CRPS ) which has been used as an objective in the regression setting ( Gneiting et al. , 2008 ; Mohammadi et al. , 2016 ; 2015 ) . We generalize the CRPS for the survival setting , called Survival-CRPS , with right-censored and interval-censored extensions . Our work is one among many recent works Chapfuwa et al . ( 2018 ) in using non-MLE training objectives for survival prediction . Summary of contributions . ( 1 ) We introduce the proper scoring rule Survival-CRPS , a generalization of CRPS , as an objective in survival prediction . We present its right-censored and interval-censored variants . ( 2 ) We propose a new metric , Survival-AUPRC , inspired by the paradigm of maximizing sharpness subject to calibration , to holistically measure the quality of a prediction distribution with respect to a possibly censored outcome . ( 3 ) We give practical recommendations for the mortality prediction task , by recommending use of the log-normal parameterization and interval censoring when training . ( 4 ) We employ the above techniques and demonstrate their efficacy by training a deep recurrent neural network model for accurate survival prediction of patient mortality using EHR data . 2 COUNTDOWN REGRESSION . Parametric survival prediction methods model the time to an event of interest with a family of probability distributions , indexed by the distribution parameters . The survival function , denoted S ( t ) : [ 0 , ∞ ) → [ 0 , 1 ] , is a monotonically decreasing function over the positive reals with S ( 0 ) = 1 and limt→∞ S ( t ) = 0 . The survival function represents the probability of an individual not having the event of interest up to a given time . Every survival function has a corresponding cumulative density function ( CDF ) , denoted F ( t ) = 1− S ( t ) , and probability density function ( PDF ) , denoted f ( t ) = ddtF ( t ) . The choice of the family of probability distributions implies assumptions made about the nature of the data generating process . We denote the medical record of a patient i as ( { ( x ( i ) t , a ( i ) t ) } T ( i ) t=1 , d ( i ) , c ( i ) ) , where t ∈ { 1 . . . T ( i ) } denotes the interaction number of this patient with the health record , x ( i ) t ∈ RD is the set of features corresponding to the t-th interaction , a ( i ) t ∈ R+ is age at time t , d ( i ) ∈ R+ is the age of death or age of last known ( alive ) encounter , and c ( i ) ∈ { 0 , 1 } is a censoring indicator where c ( i ) = 0 means the age of death is d ( i ) , and c ( i ) = 1 means the age of death is at least d ( i ) . For each x ( i ) t we define the quantity y ( i ) t = d ( i ) − a ( i ) t which represents the corresponding time to event or time to censoring . Traditional methods in survival analysis are designed to handle right-censored outcomes , but we observe that in many common scenarios outcomes are actually interval-censored . In the context of mortality prediction , for example , we know that humans almost never live past 120 years of age . Therefore , we assume that the true age of death lies between d ( i ) and A = 120 years , implying that the true time to death lies between 0 and T ( i ) t = A− a ( i ) t . We omit patient superscripts i and interaction subscripts t for succinctness where possible . We note that although our notation focuses on the problem of mortality prediction , our techniques generalize to any time to event task of interest . 2.1 SURVIVAL-CRPS : PROPER SCORING RULE OBJECTIVES . A scoring rule is a measure of the quality of a probabilistic forecast . A forecast over a continuous outcome is a probability density function over all possible outcomes , f̂ with corresponding cumulative density function F̂ . In reality , we observe some actual outcome , y . A scoring rule S takes a predicted distribution and an actual outcome , and returns a loss S ( F̂ , y ) . It is considered a proper scoring rule if for all possible distributions G , Ey∼F̂ [ S ( F̂ , y ) ] ≤ Ey∼F̂ [ S ( G , y ) ] , and strictly proper when equality holds if and only if F̂ = G ( Gneiting et al. , 2008 ) . A proper scoring rule is one in which the expected score is minimized by the distribution with respect to which the expectation is taken . Intuitively , it encourages a model for being honest by predicting what it actually believes ( Savage , 1971 ) . When a proper scoring rule is employed as a loss function , it naturally forces the model to output calibrated probabilities ( Gneiting & Katzfuss , 2014 ) . There are many commonly used proper scoring rules . Perhaps the most widely used is the logarithmic scoring rule , equivalent to the maximum likelihood objective : SMLE ( F̂ , y ) = − log f̂ ( y ) . In the presence of possibly censored data , we maximize the density for observed outcomes , and tail or interval mass for censored outcomes , and this is a proper scoring rule ( Dawid & Musio , 2014 ) . SMLE-RIGHT ( F̂ , ( y , c ) ) = − log ( ( 1− c ) f̂ ( y ) + cŜ ( y ) ) SMLE-INTVL ( F̂ , ( y , c , T ) ) = − log ( ( 1− c ) f̂ ( y ) + c ( F̂ ( T ) − F̂ ( y ) ) ) However , the logarithmic scoring rule is asymmetric , and harshly penalizes predictions that are wrong yet confident . This results in the training process becoming sensitive to outliers , and in general conservative in prediction-making ( that is , hesitant to make sharp predictions ) ( Gneiting & Raftery , 2007 ) . ( a ) Uncensored ( b ) Right-censored ( c ) Interval-censored Another proper scoring rule for forecasts over continuous outcomes is the CRPS ( Gneiting et al. , 2007 ) , defined as SCRPS ( F̂ , y ) = ∫ ∞ −∞ ( F̂ ( z ) − 1 { z ≥ y } ) 2 dz = ∫ y −∞ F̂ ( z ) 2dz + ∫ ∞ y ( 1− F̂ ( z ) ) 2dz . The CRPS has been used in regression as an objective function that yields sharper predicted distributions compared to maximum likelihood , while maintaining calibration ( Gneiting et al. , 2008 ) . Intuition for the CRPS is better understood by analyzing the latter expression and noting that the two integral terms correspond to the two shaded regions in Figure 2a . The CRPS score is completely reduced to zero when the predicted distribution places all the mass on the point of true outcome , or equivalently , when the shaded region completely vanishes . In the context of time to event predictions we propose the Survival-CRPS which accounts for the possibility of right-censored or interval-censored data : SCRPS-RIGHT ( F̂ , ( y , c ) ) = ∫ y 0 F̂ ( z ) 2dz + ( 1− c ) ∫ ∞ y ( 1− F̂ ( z ) ) 2dz , SCRPS-INTVL ( F̂ , ( y , c , T ) ) = ∫ y 0 F̂ ( z ) 2dz + ( 1− c ) ∫ T y ( 1− F̂ ( z ) ) 2dz + ∫ ∞ T ( 1− F̂ ( z ) ) 2dz . Note that when c = 0 , both of the above expressions are equivalent to the original CRPS . Again , the intuition behind the Survival-CRPS is better understood by mapping each of the integral terms to the corresponding shaded region in Figure 2b and Figure 2c . The Survival-CRPS behaves like the original CRPS when the time of event is uncensored . For censored outcomes , it penalizes the predicted mass that occurs before the time of censoring and , if interval censored , also the mass after time by which the event must have occurred . Both variants of the Survival-CRPS are proper scoring rules . They are special cases of the threshold weighted CRPS ( Gneiting & Ranjan , 2011 ) , where the weighting function is an indicator over the uncensored regions .

Summary: The paper proposes the use of Survival Continuous Ranked Probability score instead of maximum likelihood estimation for personalised probabilistic forecasts of time-to-event data, thus estimating a distribution over future time. The authors describe the evaluation their method using (1) proper scoring rule objectives; (2) evaluation of calibration using sharpness as a metric; (3) the survival precision recall curve. The authors then apply these techniques to predicting time-to-mortality using an RNN that takes EHR patient records to predict the probability of death at a given time point. It’s not clear how this is related to the Survival CRPS model or how this model is incorporated into the RNN.