When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting

1 Introduction . Infectious diseases like seasonal influenza and COVID-19 are major global health issues , affecting millions of people [ 14 , 34 ] . Forecasting disease time-series ( such as infected cases ) at various temporal and spatial resolutions is a non-trivial and important task [ 34 ] . Estimating various indicators e.g . future incidence , peak time/intensity and onset , gives policy makers valuable lead time to plan interventions and optimize supply chain decisions , as evidenced by various Centers for Disease Control ( CDC ) prediction initiatives for diseases like dengue , influenza and COVID-19 [ 33 , 16 , 30 ] . Statistical approaches [ 5 ] for the forecasting problem are fairly new compared to more traditional mechanistic approaches [ 13 , 38 ] . While valuable for ‘ what-if ’ scenario generation , mechanistic models have several issues in real-time forecasting . For example , they can not easily leverage data from multiple indicators or predict composite signals . In contrast , deep learning approaches in this context are a novel direction and have become increasingly promising , as they can ingest numerous data signals without laborious feature engineering [ 37 , 33 , 1 , 8 ] . However , there are several challenges in designing such methods , primarily with the need to handle uncertainty to give more reliable forecasts [ 14 ] . Decision makers need to understand the inherent uncertainty in the forecasts so that they can make robust decisions [ 32 ] . Providing probabilistic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . forecasts and interpreting what signals cause the model uncertain is also helpful to better communicate the situation to the public . Due to the inherent complexity of the prediction problem , just like weather forecasting , so-called ‘ point ’ forecasts without uncertainty are increasingly seen as not very useful for planning for such high-stake decisions [ 14 , 33 ] . Uncertainty quantification in purely statistical epidemic forecasting models is a little explored area . Most traditional methods optimize for accuracy of ‘ point-estimates ’ only . Some approaches that model the underlying generative distribution of the data naturally provide a probability distribution of the outputs [ 4 , 5 , 44 , 32 ] , but they do not focus on producing calibrated distributions [ 12 , 22 ] as well . Another line of research addresses this problem with the use of simple methods such as an ensemble of models to build a sample of forecasts/uncertainty bounds [ 34 , 6 ] . Recent attempts for deep learning forecasting models use ad-hoc methods such as bootstrap sampling [ 37 ] ; while others disregard this aspect [ 42 , 36 ] . As a result these can produce wildly wrong predictions ( especially in novel/atypical scenarios ) and can be even confident in their mistakes . In time-series analysis , while a large number of deep learning models [ 1 ] have been proposed , little work has been done to quantify uncertainty in their predictions . Bayesian deep learning [ 28 , 3 , 27 ] ( and approximation methods [ 10 , 25 , 43 ] ) and deep ensembling [ 24 ] are two directions that may mitigate this issue , but their applicability and effectiveness are still largely limited by factors such as intractable exact model inference [ 3 , 27 ] , difficulty of specifying proper parameter priors [ 26 ] , and uncertainty underestimation [ 21 , 19 ] . Neural Process ( NP ) [ 11 ] and Functional Neural Process ( FNP ) [ 26 ] are recent frameworks developed to incorporate stochastic processes with DNNs , but only for static data . Our work aims to close these crucial gaps from both viewpoints . We propose a non-parametric model for epi-forecasting by ‘ marrying ’ deep sequential models with recent development of neural stochastic processes . Our model , called EPIFNP , leverages the expressive power of deep sequential models , while quantifying uncertainty for epidemic forecasting directly in the functional space . We extend the idea of learning dependencies between data points [ 26 ] to sequential data , and introduce additional latent representations for both local and global views of input sequences to improve model calibration . We also find that the dependencies learned by EPIFNP enable reliable interpretation of the model ’ s forecasts . Figure 1 shows an example of a wellcalibrated forecast due to EPIFNP in flu forecasting . CDC is interested in forecasting weighted Influenza-likeillness ( wILI ) counts , where ILI is defined as “ fever and a cough and/or a sore throat without a known cause other than flu . Figure 1 ( a ) shows the historical ILI data with abnormal seasons highlighted ; Figure ( b ) shows how our method EPIFNP , in contrast to others , is able to react well to a particularly novel event ( in this case , introduction of a symptomatically simi- lar COVID-19 disease ) , giving both accurate and well-calibrated forecasts . Our main contributions are : • Probabilistic Deep Generative Model : We design a neural Gaussian processes model for epidemic forecasting , which automatically learns stochastic correlations between query sequences and historical data sequences for non-parametic uncertainty quantification . • Calibration and Explainability : EPIFNP models the output forecast distribution based on similarity between the current season and the historical seasons in a latent space . We introduce additional latent variables to capture global information of historical seasons and local views of sequences , and show that this leads to better-calibrated forecasts . Further , the relations learned between the current season and similar patterns from previous seasons enable explaining the predictions of EPIFNP . • Empirical analysis of accurate well-calibrated forecasting : We perform rigorous benchmarking on flu forecasting and show that EPIFNP significantly outperforms strong baselines , providing up to 2.5x more accurate and 2.4x better calibrated forecasts . We also use outlier seasons to show the uncertainty in EPIFNP makes it adapt well to unseen patterns compared with baselines . 2 Problem and Background . We focus on epidemic disease forecasting in this paper . Our goal is to predict the disease incidence few week into the future given the disease surveillance dataset containing incidence from the past seasons as well as for the past weeks of the current season . This is formulated as a supervised time-series forecasting problem as follows . Epidemic Forecasting task : Let the incidence for season i at week t be x ( t ) i . During the current season N + 1 and current week t , we first have the snippet of time-series values upto week t denoted by x ( 1 ... t ) N+1 = { x ( 1 ) N+1 , . . . , x ( t ) N+1 } . We are also provided with data from past historical seasons 1 to N denoted by H = { x ( 1 ... T ) i } Ni=1 where T is number of weeks per season . In real-time forecasting , intuitively our goal is to use all the currently available data , and predict the next few future values ( usually till 4 weeks in future ) . That is to predict the value y ( t ) N+1 = x ( t+k ) N+1 , k week in future where k ∈ { 1 , 2 , 3 , 4 } given x ( 1 ... t ) N+1 and H . Formally , our task is : given ( a ) the dataset of historical incidence sequences H and ( b ) snippet of incidence for current season N + 1 till week t , x ( 1 ... t ) N+1 , estimate an accurate prediction for y ( t ) N+1 and a well-calibrated probability distribution p̂ ( y ( t ) N+1|x ( 1 ... t ) N+1 , H ) . There are several ways to evaluate such forecasts [ 40 ] , which we elaborate later in our experiments . 3 Our Methodology . Overview : EPIFNP aims to produce calibrated forecasting probabilistic distribution . One popular choice is to use BNNs [ 3 , 9 ] which impose probability distributions for weight parameters . However , as Deep Sequential Models ( DSMs ) have an enormous number of uninterpretable parameters , it is impractical to specify proper prior distributions in the parameter space . Existing works usually adopt simple distributions [ 3 , 35 ] , e.g. , independent Gaussian distribution , which could severely under-estimate the true uncertainty [ 21 ] . To solve this issue , we propose EpiFNP , which combines ( 1 ) the power of DSMs in representation learning and capturing temporal correlations ; and ( 2 ) the power of Gaussian processes ( GPs ) in non-parametric uncertainty estimation directly in the functional space similar to [ 26 ] , instead of learning probability distributions for model parameters . During training phase of our supervised learning task , EPIFNP is trained to predict x ( t+k ) i given x ( 1 ... t ) i as input for i ≤ N . Therefore , we define the training set M as set of partial sequences and their forecast ground truths from historical data H , i.e , M = { ( x ( 1 ... t ) i , y ( t ) i ) : i ≤ N , t + k ≤ T , y ( t ) i = x ( t+k ) i } . For simplicity , let XM be set of the partial sequences in M and yM the set of ground truth labels . Following GPs for non-parametric uncertainty quantification , EPIFNP constructs the forecasting distribution on the historical sequences . Since the number of possible sequences that can be extracted from H is prohibitively large , we narrow down the set of candidates into a set of sequences that comprehensively represents H , called the reference set R. We choose the set of full sequences of T incidence values for each season as reference set , i.e , R = { x ( 1 ... T ) i } NR i=1 . We refer elements of M as { xMi , yMi } NM i=1 and R as { xMi } NR i=1 when we don ’ t need to specify the week and season . Also let XD = { xMi } NM i=1 ∪ { xRi } NR i=1 , the union of reference and training sequences . The generative process of EPIFNP includes three key steps ( also see Figure 2 and Eq . 1 ) : ( a ) Probabilistic neural sequence encoding ( Section 4.2 ) . The first step of the generative process is to use a DSM to encode the sequence xi ∈ XD into a variational latent embedding ui ∈ UD . The representation power of DSM helps us to model complex temporal patterns within sequences , while the probabilistic encoding framework enables us to capture the uncertainty in sequence embedding . ( b ) Stochastic correlation graph construction ( Section 4.3 ) . The second step is to capture the correlations between reference ( UR ) and training ( UM ) data points in the latent embedding space ( i.e . seasonal similarity between epidemic curves ) . We use a stochastic data correlation graph G , which plays a similar role to the covariance matrix in classic GPs . It encodes the dependencies between reference and training sequences , enabling non-parametric uncertainty estimation . ( c ) Final predictive distribution parameterization ( Section 4.4 ) . Finally , we parameterize the predictive distribution with three stochastic latent variables : ( 1 ) The global stochastic latent variable v , which is shared by all the sequences . This variable captures the overall information of the underlying function based on all the reference points . ( 2 ) The local stochastic latent variables ZM = { zMi } NM i=1 . This term captures the data correlation uncertainty based on the stochastic data correlation graph G. ( 3 ) The stochastic sequence embeddings UM = { uMi } NM i=1 . This term captures the embedding uncertainty and provide additional information beyond the reference set . Hence , putting it all together from the generative process , we factorize the predictive distribution of the training sequences into three corresponding parts ( θ is the union of the parameters in EPIFNP ) : p ( yM |XM , R ) = ∑ G ∫ pθ ( UD|XD ) ︸ ︷︷ ︸ ( a ) p ( G|UD ) ︸ ︷︷ ︸ ( b ) pθ ( ZM , |G , UR ) pθ ( v|UR ) pθ ( yM |UM , ZM , v ) ︸ ︷︷ ︸ ( c ) dUDdZMdv . ( 1 ) Compared to existing recurrent neural process ( RNP ) [ 31 ] for sequential data ( and its related predecessors [ 11 , 17 ] ) , our EPIFNP process has stronger representation power and is more interpretable . Specifically , RNP uses a single global stochastic latent variable to capture the functional uncertainty , which is not flexible enough to represent a complicated underlying distribution . In contrast , EPIFNP constructs three stochastic latent variables to capture the uncertainty from different perspectives and can interpret the prediction based on the correlated reference sequences .

Summary: This paper proposes a method for probabilistic time series forecasting for univariate real-valued data (they just consider Influenza-Like Illness) using a semi-parametric neural network. The basic idea is to embed the prefix of observations for the current season into a fixed-sized latent vector using a GRU, and to embed all past full-year sequences into latent vectors, and then to compute a weighted average of the most similar embeddings from the past and to use this feature inside an MLP to predict the future, k-weeks ahead. They evaluate on the CDC dataset and report improved point estimates and calibration scores.