When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
Authors: Harshavardhan Kamarthi
Lingkai Kong
Alexander Rodriguez
Chao Zhang
B. Aditya Prakash
Venue: NeurIPS 2021 Poster (2021)
Keywords: Epidemic Forecasting
Uncertainty Quantification
Time-Series forecasting
Deep Probabilistic Models

Abstract: Accurate and trustworthy epidemic forecasting is an important problem for public health planning and disease mitigation. Most existing epidemic forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations; e.g., it is difficult to specify proper priors in Bayesian NNs, while methods like deep ensembling can be computationally expensive. In this paper, we propose to use neural functional processes to fill this gap. We model epidemic time-series with a probabilistic generative process and propose a functional neural process model called EpiFNP, which directly models the probability distribution of the forecast value in a non-parametric way. In EpiFNP, we use a dynamic stochastic correlation graph to model the correlations between sequences, and design different stochastic latent variables to capture functional uncertainty from different perspectives. Our experiments in a real-time flu forecasting setting show that EpiFNP significantly outperforms state-of-the-art models in both accuracy and calibration metrics, up to 2.5x in accuracy and 2.4x in calibration. Additionally, as EpiFNP learns the relations between the current season and similar patterns of historical seasons, it enables interpretable forecasts. Beyond epidemic forecasting, EpiFNP can be of independent interest for advancing uncertainty quantification in deep sequential models for predictive analytics.

When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting
Harshavardhan Kamarthi
Lingkai Kong
Alexander Rodríguez
Chao Zhang
B. Addiiyy Prakash
College of Computing
Georgia Institute of Technology
[harsha, pk, 1.konkk,,,... arodriguezc, chaozhang, badityap1 edu
Abstract
Accurate and trustworthy epidemic forecasting is an important problem for public health planning and disease mitigation. Most existing epidemic forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-await have several limitations; e.g., it is difficult to specify proper priors in Bayesian NNs, while methods like deep ensembling can be computationally expensive. In this paper, we propose to use neural functional processes to fill this gap. We model epidemic time- with probabilistic generative process and propose a func- tional neural process model called EPIENP, which directly models the probability distribution of the forecast value non-parametric way. In EPIFNP, we use a dynamic stochastic correlation graph to model the correlations between sequences, and design different stochastic latent variables to capture functional uncertainty from different perspectives. Our experiments in real-time flu forecasting set- ting show that EPIFNP significantly outperforms sfate-of-thh-rtt models in both accuracy and calibration metrics, up to 2.5x in accuracy and 2.4x in calibration. Additionally, as EPIFNP learns the relations between the current season and similar patterns of historical seasons, it enables interpretable forecasts. Beyond epidemic forecasting, EPIENP can be of independent interest for advancing uncertainty quantification in deep sequential models for predictive analytics.
time-series forecasting also
Introduction
Infectious diseases like seasonal influenza and COVID-19 are major global health issues, affecting millions of people [14] (34.. Forecasting disease time-series (such as infected cases) various temporal and spatial resolutions is a non-trivial and important task (34). Estimating various indicators e.g. future incidence, peak time/intensity and onset, gives policy makers valuable lead time to plan interventions and optimize supply chain decisions, evidenced by various Centers for Disease Control (CDC) prediction initiatives for diseases like dengue, influenza and COVID-19 33 16 30).
Statistical approaches [5] for the forecasting problem are fairly new compared to more traditional mechanistic approaches [13 38). While valuable for 'what-if" scenario generation, mechanistic models have several issues in real-time forecasting. For example, they cannot easily leverage data from multiple indicators or predict composite signals. contrast, deep learning approaches this context are a novel direction and have become increasingly promising, they can ingest numerous data signals without laborious feature engineering [37 33 8.
However, there are several challenges in designing such methods, primarily with the need to handle uncertainty give more reliable forecasts [14]. Decision makers need understand the inherent uncertainty in the forecasts so that they can make robust decisions 32 Providing probabilistic
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

forecasts and interpreting what signals cause the model uncertain is also helpful to better communicate the situation to the public. Due the inherent complexity of the prediction problem, just like weather forecasting, so-called 'point' forecasts without uncertainty are increasingly seen as not very useful for planning for such high-stake decisions
[14 33].
Uncertainty quantification in purely statistical epidemic forecasting models is little explored area. Most traditional methods optimize for accuracy of "point-estimates" only. Some approaches that model the underlying generative distribution of the data naturally provide probability distribution of
the outputs 5 44 32], but they do not focus on producing calibrated distributions [2 22
as well.
Another line of research addresses this problem with the use of simple methods such as an ensemble
of models to build a sample of forerasts/uncerraintyy bounds [34 6). Recent attempts for deep learning forecasting models use ad-hoc methods such as bootstrap sampling [37]; while others disregard this
aspect (42] 336. As a result these can produce wildly wrong predictions (especially in novel/atypical scenarios) and can be even confident in their mistakes. In time-series analysis, while a large number of deep learning models [ have been proposed, little work has been done to quantify uncertainty
in their predictions. Bayesian deep learning [28] 27 (and approximation methods [10
25
43)
and deep ensembling (24) are two directions that may mitigate this issue, but their applicability and
effectiveness are still largely limited by factors such as intractable exact model inference 27
difficulty of specifying proper parameter priors [26). and uncertainty underestimation [21, 19). Neural
Process (NP) [IT] and Functional Neural Process (FNP) (26) recent frameworks developed to incorporate stochastic processes with DNNs, but only for static data.
Our work aims to close these crucial gaps from both viewpoints. We propose non-parametric model for epi-forecasting by 'marrying' deep sequential models with recent development of neural stochastic processes. Our model, called EPIFNP, leverages the expressive power of deep sequential models, while quantifying uncertainty for epidemic forecasting directly in the functional space. We extend the idea of learning dependencies between data points (26) to sequential data, and introduce additional latent representations for both local and global views of input sequences to improve model calibration. We also find that the dependencies learned by EPIFNP enable reliable interpretation of the model's s forecasts.
Figure T shows an example of a well- calibrated forecast due to EPIENP in flu forecasting. CDC is interested in forecasting weighted Influenza- like- illness (wILI) counts, where ILI is defined as "fever and a cough and/or sore throat without known cause other than flu. Figure I (a) shows the historical ILI data with abnormal seasons highlighted; Figure (b) shows how our method EPIFNP, in contrast to others, is able to react well to par- ticularly novel event (in this case, in- troduction of a symptomatically simi- lar COVID-19 19 disease), giving both accurate and well-calibrated forecasts.
2015:00 Farecaate week
Cimmine
Very carter
Burfi
Pead
EIIEN prodictions Imean)
Week
Week No.
(a) Historical wILI seasons se- (b) Probabilistic predictions of all quences, 2003-20
methods
Figure 1: EPIFNP (red) is the only model reacting reliably for the atypical 3rd peak of 2019/20 season and whose 95 % confidence bounds completely encloses the ground truth.
Our main contributions are:
Probabilistic Deep Generative Model: We design neural Gaussian processes model for epi- demic forecasting, which automatically learns stochastic correlations between query sequences and historical data sequences for non- con-parametic uncertainty quantification.
Calibration and Explainability: EPIENP models the output forecast distribution based on simi- larity between the current season and the historical seasons in a latent space. We introduce additional latent variables to capture global information of historical seasons and local views of sequences, and show that this leads to calterecal forecasts. Further, the relations learned between the current season and similar patterns from previous seasons enable explaining the predictions of EPIFNP. Empirical analysis of accurate well-calibrated forecasting: We perform rigorous benchmarking on flu forecasting and show that EPIFNP significantly outperforms strong baselines, providing up to 2.5x more accurate and 2.4x better calibrated forecasts. We also use outlier seasons show the uncertainty in EPIENP makes it adapt well to unseen patterns compared with baselines.
2

---

2
Problem and Background
We focus on epidemic disease forecasting in this paper. Our goal is to predict the disease incidence few week into the future given the disease surveillance dataset containing incidence from the past seasons as well as for the past weeks of the current season. This is formulated as supervised time-series forecasting problem as follows.
Epidemic Forecasting task: Let the incidence for season i at week t be 2!!)) During the current season N 1 and current week t, first have the snippet of time-series values upto week t denoted by XN+1 (1...)) = {2 f 2 N N We are also provided with data from past historical
(8)
seasons to N denoted by H (x! Jim where
number of weeks per season. real-time
forecasting, intuitively our goal is to use all the currently available data, and predict the next few
future values (usually till 4 weeks in future). That to predict the value VN+1
(t+R) k week (t) =N++1
(L...) and H. Formally, our task is: given (a) the dataset of in future where k E (1,2,3,4) 4} given XN+1
historical incidence sequences H and (b) snippel of incidence for current season N + till week
N+11 estimate an accurate prediction for VN+1 and well-calibrated probability distribution (t) (1...)
(I...)
(8)
P(NN+1 [xN+1 H). There are several ways to evaluate such forecasts [40]. which we elaborate later
in our experiments.
3 Our Methodology
Overview: EPIENP aims to produce calibrated forecasting probabilistic distribution. One popular
choice is to use BNNS 3 9 which impose probability distributions for weight parameters. However, as Deep Sequential Models (DSMs) have an enormous number of uninterpretable parameters, it is impractical specify proper prior distributions in the parameter space. Existing works usually adopt simple distributions [3 35). e.g., independent Gaussian distribution. which could severely under-estimate the true uncertainty (21). To solve this issue, we propose EpiFNP, which combines (1) the power of DSMs in representation learning and capturing temporal correlations; and (2) the power of Gaussian processes (GPs) in non-parana con-aaraetrri uncertainty estimation directly in the functional space similar to (26), instead of learning probability distributions for model parameters.
During training phase of our supervised learning task, EPIFNP is trained to predict x; (t++) given x (l....) as input for < N. Therefore, we define the training set M as set of partial sequences and their forecast ground truths from historical data H, i.c, MM- = {(x} (l...) (ft)) : i S N,t
T. My (1) x; ((+t)), ). For simplicity, let X M be set of the partial sequences in M and ym the set of ground truth labels. Following GPs for non-parametric uncertainty quantification, EPIENP constructs the forecasting distribution on the historical sequences. Since the number of possible sequences that can be extracted from H is prohibitively large, we narrow down the set of candidates into set of sequences that comprehensively represents H, called the reference set R. We choose the set of full
sequences of T incidence values for each season as reference set, i.e, R = {x:
(1...T) Ng Ji=1- We refer
elements of M as
{x,M, y Nu and R as My Jizi when we don't 't need to specify the week and {xx
season. Also let X [xx
M NM 1=1
U (x the union of reference and training sequences.
The generative process of EPIENP includes three key steps (also see Figure and [
(a) Probabilistic neural sequence encoding (Section 4.2). The first step of the generative process is use DSM to encode the sequence x, X into variational latent embedding u, Up. The representation power of DSM helps us to model complex temporal patterns within sequences, while the probabilistic encoding framework enables us to capture the uncertainty in sequence embedding. (b) Stochastic correlation graph construction (Section 4.3). The second step is to capture the correlations between reference (U and training (U M) data points in the latent embedding space (i.e. seasonal similarity between epidemic curves). We use a stochastic data correlation graph G, which plays similar role the covariance matrix in classic GPs. It encodes the dependencies between reference and training sequences, enabling non-parametric uncertainty estimation.
(c) Final predictive distribution parameterization (Section 4.4). Finally, we parameterize the predictive distribution with three stochastic latent variables: (1) The global stochastic latent variable which is shared by all the sequences. This variable captures the overall information of the
3

---

Reference
(a) Probabiiistic Sequence Encoder
Ifuil indidence past seasors)
(c) Predictive
Distribution
Parameterization
Forecast Distribution
SA
Traini M (Preft pastiurrett
Go
Legend
GRU
and nu mimation
Semaull
Sample from
(b) Stochastic Correlation Graph
Figure Pipeline of proposed EPIFNP model. (i) Three main components (a), (b) and (c) correspond to the terms in Equation 7 (ii) Variables highlighted in Red correspond to steps specific to inference of sequence x3
underlying function based on all the reference points. (2) The local stochastic latent variables ZM - [z, M Nus This term captures the data correlation uncertainty based on the stochastic data
correlation graph G. (3) The stochastic sequence embeddings U = (u²*) Nx This term captures the embedding uncertainty and provide additional information beyond the reference set.
Hence, putting it all together from the generative process, factorize the predictive distribution of the training sequences into three corresponding parts (0 the union of the parameters EPIFNP):
plyM MM, R)
Σ/
po (U Xx) DG
(ZM, G. (v) Un)Po(yM UM. ZM. dUpdZadv.
Compared to existing recurrent neural process (RNP) [31] for sequential data (and its related prede- cessors (7). our EPIFNP process has stronger representation power and is more interpretable. Specifically, RNP uses a single global stochastic latent variable to capture the functional uncertainty, which is not flexible enough represent complicated underlying distribution. In contrast, EPIENP constructs three stochastic latent variables to capture the uncertainty from different perspectives and can interpret the prediction based on the correlated reference sequences.
3.1 Probabilistic Neural Sequence Encoder
The probabilistic neural sequence encoder po (UD (D) aims to model the complex temporal correla- tions of the sequence for accurate predictions of y, while capturing the uncertainty in the sequence embedding process. To this end, we design the sequence encoder as RNN and stack a self-attention layer to capture long-term correlations. Moreover, following Variational auto-encoder (VAE) (18), we model the latent embedding u, Gaussian random variable capture embedding uncertainty.
We encode all the sequences, including reference sequences and training sequences, independently. Taking one sequence x; example, first feed x, into Gated Recurrent Unit (GRU) (7):
(h. h ( GRU((=) (1) ..... (......()))
where (*) denotes the hidden state at time step t. To obtain the embedding of x,, the simplest way is to directly use the last step hidden state, (t), However, using the last step embedding is inadequate for epidemic forecasting as the estimates for ILI surveillance data are often delayed and revised multiple times before they stabilize (T]. Over-reliance over the last step hidden state would harm the predictive ability of the model. Therefore, we choose use self-attention layer to aggregate the information of the hidden states across all the time steps:
(1) (a ...,0 a
(8)
Self-Atten(
h (())),
h
h
(3)

---

where h, is the summarized hidden state vector. Compared with the vanilla attention mechanism (2) self-attention is better at capturing long-term temporal correlations [41). Though h, has encoded the temporal correlations, it is deterministic and cannot represent embedding uncertainty. Inspired by VAE, we parameterize each dimension the latent embedding u, as Gaussian random variable:
pe ([( [x,) = =V (h.)le, exp (h,))))),
(4)
where 91 and 92 are two multi-layer perceptrons (MLPs), Ha is the k-th dimension of the variable.
3.2 Stochastic Data Correlation Graph
The stochastic graph G is used to model the correlations among sequences, which is central to the non-parametric uncertainty estimation ability of EPIFNP. is realized by constructing a bipartite graph from the reference set R to the training set M based on the similarity between their sequence embeddings. With this graph, we aim model the dynamic similarity among epidemic curves as in 1 but in a stochastic manner, which allows to further quantify the uncertainty coming from our latent representations of the sequences. Note that the similarity with reference sequence embeddings dynamically changes across the current season since different periods of the season may be similar to different sets of reference sequences (as we illustrate in Section 4.4
We first construct a complete weighted bipartite graph G. from R to M, where the nodes are the sequences. The weight of each edge is calculated as similarity belween two sequences in the embedding space using the radial basis function kernel k(u," u = exp(->il|t l M [[²). Modeling such a similarity in the embedding space more accurate than the input space by leveraging the representation power of the neural sequence encoder.
Though we can directly G. to encode the data correlations, such dense complete graph requires heavy computations and does not scale to large dataset. Therefore, we choose to fur- ther sample from this complete graph to obtain stochastic binary bipartite graph G shown in Figure(3 This graph can be represented as a ran- dom binary adjacency matrix, where G.,- means the reference sequence xf is parent of the training sequence x.M. We then parameter- ize this binary adjacency matrix using Bernoulii distributions:
Reference set
Reference
G
G
Training set
Training set
Figure 3: We sample the (sparse) binary graph G from the complete weighted (dense) graph Ge-
( (G) = II II
(5)
iE JEM
Intuitively, the edges in G. with higher weights are more likely to be kept after sampling. This sampling process leads sparse correlations for each sampled graph, which can speed up training due to sparsity.
3.3 Parameterizing Predictive Distribution
Here we introduce how to parameterize the final prediction based on the three latent variables mentioned in Section 4.1, which capture the functional uncertainty from different perspectives.
Local latent variable z:M: summarizes the information of the correlated reference points for each training point and captures the uncertainty of data correlations. We generate z,M based on the structure of the data correlation graph, and each dimension & follows Gaussian distribution:
N(C Σ hn )x, K
M
Cha(u))w)
(6)
f.G
j.G,.=1
where h2 & h2 are two MLPs and C, E, Gú, is for normalization. As can see from Equation 6 the sequence has lower probability to be connected with the reference sequences, z,M becomes standard Gaussian distribution which is an uninformative prior. This property imposes similar inductive bias as in the GPs with RBF kernel.
Global latent variable v. It encodes the information in all the reference points, computed as:
NR
Bi,...,Bryy = Self-Atten(u",.. ,,
V
3
5

---

In contrast with the local variable z,M the global latent variable summarizes the overall information of the underlying function. It is shared by all the training sequences which allows us to capture the functional uncertainty/ from global level.
Sequence embedding u The above two latent variables are both constructed from the embeddings of the reference sequences, which may lose novel information present in the training sequences. Therefore, we add a direct path from the latent embedding u, of the training sequence to the final prediction to enable the neural network to extrapolate beyond the distribution of the reference sequences. This useful in novel/unprecedented patterns where the imput sequence can not rely only on reference sequences from historical data for prediction.
We concatenate the three variables together into single vector e; and obtain the final predictive distribution (where dj and d2 are MLPs):
e; = con Vi uz),
pluile v. u (N(( exp(d2(e:))).
(8)
3.4 Learning the distribution
We now introduce how to learn the model parameters efficiently during training and forecast for new unseen sequence at test time. Directly maximizing the data likelihood is intractable due to the summation and integral in Equation [ Therefore, choose to use the amortized variational inference and approximate the true posterior pl Up G, ZM, v R, M) with go (UD, G, ZM. viR, M), similar to [26). as
go (UD, G, ZM, RRR M) pa (UD (D))((GU (ZM)
We design lo as single layer of neural network parameterized by é, which outputs mean and variance of the Gaussian distribution go [Zas] Xxr).
We then use a gradient-based method, such as Adam (18), to maximize the evidence lower bound (ELBO) of the log likelihood. After canceling redundant terms, the ELBO can be written as:
L = - F-z G, G.UD,V~q, (Zxr (G. (o.. (log
UM,
(10)
+ log P(ZM| G. Un) - gol (M (Ma)].
We use the reparameterization trick to make the sampling procedure from the Gaussian distribution differentiable. Moreover, as sampling from the Bernoulli distribution in Equation 7 leads to discrete correlated data points, we make use of the Gumbel softmax trick [13] to make the model differentiable.
At test time, with the optimal parameter Oopt. we base the predictive distribution of new unseen partial sequence x" on the reference set as:
p(y" xxx") =Po (Un, u X X, x" (p(a"Inn,," u²) u
(11)
PPops (z" aa)a", UR, u² P9.p. (u"|u", zz, Rdz dv.
where a" is the binary vector that denotes which reference sequences are the parents of the new sequence. u² and zt are latent embedding and local latent variable for the new sequence, respectively.
4 Experiments
All experiments were done Intel i5 4.8 GHz CPU with Nvidia GTX 1650 GPU. The model typically takes around 20 minutes to train. The code is implemented using Pytorch and will be released for research purposes. Supplementary contains additional details and results (e.g. hyperparameters, results on additional metrics (MAPE), additional case and ablation studies).
In our experiments, we focus on flu forecasting. The CDC uses the ILINet surveillance system to gather flu information from public health labs and clinical institutions across the US. It releases weekly estimates of weighted influenza-like illness (WILI)T out- out-patients with flu-like symptoms aggregated for US national and 10 different regions (called HHS regions). Each flu season begins at week 21 and ends of 2003 and ends on week 20 of 2004. Following the guidelines of CDC flu challenge [1]3)), we predict from week 40 till the end of season next year. We evaluate our approach using wIL. data of 17 seasons from 2003/04 to 2019/20
Dataset:
week 20 of the next year e.g. Season 2003/04 begins on week 21

---

Our experiments were designed evaluate the following. Q1: Accuracy and calibration of EPIENP's forecasts. Q2: Importance of different components of EPIENP. Q3: Utility of uncertainty estimates for other related tasks?. Q4: Adaptability of EPIENP to novel behaviors during real-time forecasting. Q5: Explainability of predictions.
Goals:
Evaluation metrics: Let a|/4' be given partial wILI test sequence with observed ground truth ((t)
(1+-) U/4+1 i.e., for a k-week-aheaad task UN+1 is just 2: NN+1 For a model/method M let PN+1.M8 (Y) be
the output distribution of the forecast with mean (L...)) To measure the predictive accuracy, we use Root Mean Sq. Error (RMSE). Mean Abs. Per. Error (MAPE) and Log Score (LS) which are commonly used in CDC challenges (4.. To evaluate the calibration of the predictive distribution introduce new metric called Calibration Score (CS). For model M we define function kAs : [0. [0, 1] as follows. For each value of confidence e [0. 1]. let k (c) denote the fraction of observed ground truth that lies inside the confidence interval of predicted output distributions of
M. For a perfectly calibrated model M we would expect ky- (c) c. CS measures the deviation
of kM from kM- Formally, we define CS as:
CS(M)= -f" [Mar(c) - clde 0.01
H (c) - c/.
(12)
OE (U...........11
For all metrics, lower is better. We also define the Calibration Plot (CP) as the profile of kM (c) vs for all c E [0, 1].
We compare EPIENP with standard and state-of-art models used for flu forecasting before, as well as methods typically used for learning calibrated uncertainty quantification.
Baselines:
Flu forecasting related: SARIMA: Seasonal Autoregressive Integrated Moving-Aveen is a auto- regressive time series model used baseline for forecasting tasks 44). Gated Recurrent Unit (GRU); A popular deep learning sequence encoder, used before as a baseline for this problem (T] Empirical Bayes (EB): Utilizes bayes framework and has won few epidemic forecasting compe- titions in past 4 Delta Density (DD): A probabilistic modelling approach that learns distribution of change in successive wIL.I values given changes from past weeks
Epideep (ED) [T] Recent state-of-thh-art NN flu prediction model based on learning similarity between seasons. Gaussian Process (GP) [44) Recently proposed statistical flu prediction model using GPs. Note that ED, SARIMA and GRU can only output point estimates and we use the ensemble approach to obtain their
uncertainty estimates following [34 [].
General ML Uncertainty related: Monte Carlo Dropout (MCDP) 10 MCDP applies dropout at
testing time for multiple times to measure the uncertainty. We use MCDP on a GRU as a baseline. Bayesian neural network (BNN) (3]: BNN imposes and learns from probability distributions over model parameters. We used L.TM as the architecture for BNN Recurrent Neural Process (RNP) 31 This method builds on Neural Process framework to learn from sequential data.
Note: We need train EPIENP only once at start of season using data from all past seasons unlike some baselines (ED, EB, GP, SARIMA, DD) which require retraining each week.
4.1 Q1 & Q2: Forecast Accuracy, Calibration and Model Ablation
Table 1: Average US National Performance: week ahead forecasting for seasons 2014/15-2019/00.
RMSE
MAPE
LS
CS
Model
k=2
k=3
k=
k=2
k=3
k=4
k=2
k=
k=4
k=2
k=3
k=4
ED
0.73
1.3
1.81
0.14
0,23
0.33
4,26
6.37
8.75
0.24
0.15
0.42
GRU
1.72
1.87
2.12
0.28
0.31
0.356
7.98
8.21
8.95
0.16
0.2
0.22
MCDP
2,24
241
2.61
0.46
0.51
0.6
9.62
10
10
0.24
0.32
0.34
GP
1,28
1,36
1.45
0.21
0.22
0.26
2.02
2.12
2.27
0.24
0.25
0.28
BNN
1.89
2.05
2.43
0.34
0.46
0.51
6.92
7.56
8.03
0.18
0.22
0.25
SARIMA
1.43
1.81
2.12
0.28
0.35
0.42
3.11
3.4
3.81
0.43
0.38
0.34
RNP
0.61
0.98
1.18
0.13
0.22
0.29
3.34
3.61
3.89
0.43
0.46
0.45
EB
1.21
1.23
1.25
0.57
0.58
0.58
6.92
7
7.12
0.07
0.082
0.085
DD
0.6
0.79
0.94
0.35
0.41
0.45
3,56
3.87
4.02
0.12
0.12
0.13
EPIENE
0.48
0.79
0.78
0.089
0.128
0.123
0.56
0.84
0.89
0.068
0.081
0.035
Prediction Accuracy: We first compare the accuracy of EPIFNP against all baselines for real-time forecasting in Table EPIFNP significantly outperforms all other baselines for RMSE, MAPE, L.S (which measure forecast accuracy). We notice around 13% and 42% improvement over the second
7

---

best baseline in RMSE and MAPE respectively. Impressively LS of EPIENP is 2.5 to 3.5 times less than closest b This is because the intervals y! + 0.5 of ground truth consistently fall inside high probability regions of our forecast distribution due to better accuracy (of mean) in general. Even during weeks of uncertainty (like around the peaks) most baselines badly calibrated forecasts don't t sufficiently cover the interval, EPIFNP's distribution are wide enough to capture this interval thanks its superior representation power. We also observed similar results for the 10 HHS regions as well where EPIENP outperforms the baselines where we show 16% and 7% improvements in RMSE ans I.S respectively showing EPIENP's proficiency over large variety different regions and seasons.
Calibration Quality: We measure how well-calibrated EPIENP's uncer- tainty bounds (Figure 4 are via CS. EPIENP was again the clear winner both for national forecasts (Table D and regional forecasts. Calibration Plots (CPs) (Figure (5) show EPIENP much closer to the diagonal line (ideal calibration) compared to even the most competitive baselines. We also observed that applying post-hoc calibration methods [23 39) doesn't t effect the significance of EPIFNP's calibration performance (Appendix Table EPIENP is clearly significantly superior to all other buselines in predicting both better calibrated and more accurate forecast distribution.
Otserved EpIRNP ANP
Fraction of grounds trath
EDIFNP RNP
GP Spideep
Epideop
64
Week No.
Contidence
Figure 4: Forecasts and 95%
Figure 5: CPs for EPIENP and next 3 accurate base- lines, k=4
confidence bounds on 2017/18 season.
Ablation studies:
We found all three of our EPIFNP components important for performance, with the data correlation graph the most relevant in determining uncertainty bounds. Refer to supplementary for complete results and further discussion.
4.2 Q3: Effective uncertainty estimates: Autoregressive inference
Motivation: We further show the usefulness and quality of uncertainty estimates by leveraging the so-called 'auto-regressive' inference (ARI) task. It is common to perform such forecasting in real-time epidemiological settings, especially as accuracy and training data typically drops with increasing & week-ahead in future (37). In this task, the model forecast input (multiple samples) to predict k
its output for k forecasts and so on to derive k-week ahead prediction. Hence an inaccurate and badly calibrated initial model's forecasts propagate their errors subsequent predictions as well.
We perform forecasting for k 2, 2,3,4 week ahead as described above using the k- = 1 trained model. The pseudocode for Au- toregressive inference is given in the Appendix.
Table 2: Evaluation scores for ARI task.
RMSE
LS
Model
k=3
k=4
k=2
K=3
k=
k=2
k=
km4
ED
2.21
3.13
3.82
6.03
8.84
10
0.42
0.45
0.48
MCDP
3.62
4.03
4.39
10
10
0.47
0.46
0.49
BNN
3.41
4.23
4.78
10
10
0.39
0.41
0.42
Results:
See Table 2 Only base-
GP
1.24
1.31
1.38
4.62
5.17
5.51
0.37
0.36
0.37
lines not trained autoregressively by default (as EPIFNP already outperforms them (Q1)) are considered. EPIFNP outperforms all and
EPIENP
0.6
0.85
0.99
0.64
0.96
1.14
0.063
0.074
0.048
comparable even to the non AR trained original EPIFNP scores (Table whereas we observed significant deterioration in scores for other baselines, as anticipated.
4.3
Q4: Reacting to abnormal/novel patterns
Motivation: A major challenge in real-time epidemiology (36] is the presence of novel patterns e.g. consider the impact the COVID-19 pandemic on the 2019720 wILII values (see Figure la). such cases, trustworthy real-time forecasting model to anticipate, quantify and adapt is needed to such abnormal situations. We studied our performance for the 2009/10 and 2019/20 seasons, which are well known abnormal seasons (due to the HINI strain and the COVID-19 pandemic respectively).
Our results are statistically significant using the Wilcox signed ranked test (a 0.05) with runs.

---

While we discuss results for k week ahead forecasting of 2019/20 season, the results for 2009/10 season and for & 1, lead to similar conclusions.
In short, EPIENP reacts re- liably and adapts novel scenarios. EPIENP outperforms other basclines in all metries. We observed 18% and 31% % reduction in RMSE and MAPE respectively compared best baseline (RNP) and 3.7 times lower LS com- pared best baseline (GP). Figure ((a) shows the prediction and uncer- tainty bounds of EPIENP and top baselines. GP and most other base- lines (except RNP) fail to capture the unprecedented third peak around week 24. Calibration Plot Figure bb) shows that EPIENP is better calibrated.
Results:
Observed EpIEN RNP GP
EpiFNP
Fraction ground
RNP
Week No.
Confidence
(a) 2019/20 predictions
(b) 2019/20 CP
Figure 6: EPIENP outperforms top 2 baselines during ab-
normal COVID-19 season 2019/20.
4.4
Q5: Explainable Predictions
Lack of explainability is major challenge in many ML models, which becomes even more acute in critical domains like public health. Since the Stochastic data correlation graph (SDCG) of EpIFNP (recall Section 3.2 explicitly learns relate each test sequence with relevant historical seasons' sequences, we can leverage this to provide useful explanations for predictions and model uncertainty. Knowing which past seasons are similar is very helpful for epidemiologica understanding of the prevalent strain behavior [T]. We sample SDCGs multiple times and compute average edge probability for every edge between each given historical season and test sequences weeks ahead forecasting on season 20155/6 but the observations hold for other seasons and k 1,2, 4 too.
Motivation:
during real-time forecasting for all weeks. We perform this for
Obs 1: EPIFNP automatically chooses most similar historical seasons relevant at time of prediction.
2006
2010
We leverage the edge probabilities from the SDCG to examine the seasons that are more likely sampled at at each week. We observed that the seasons with higher prob- abilities showed similar patterns to that of the current test sequence. Consider week of season 201 5/16 during 3 weeks ahead forecasting. The most likely sampled sea- sons are 2005, 2006 and 2010 (Figure 8). Figure 7 shows these seasons and 2015/T6
54
2013110
0.38
2006
Figure 7: 2015/16 snip- Figure 8: Average edge pet & most similar sea- probabilities for week 21 of sons chosen by EPIENP. 2015/16 season.
snippet; clearly they have very similar wILI patterns.
Obs 2: EpiFNP explains uncertainty bounds of predictions via distribution probabilities in the SDCG.
As seen in Section 4.3 EPIFNP reacts
k=3
liably to abnormal situations and chang- ing trends (e.g. around peaks) by produc- ing larger uncertainty bounds around those events. For example, in Figure 9 uncer- tainty estimates around peak weeks 12 and 22 are higher than for rest of the weeks. To examine the source of changing uncertainty bounds of prediction, we look average edge probabilities generated in SDCG (Fig- TOp and find that around the peak weeks the edge probabilities are lower than in surrounding weeks. This promotes larger variety of small subsets of the reference set to sampled during inference that increases the variance of local latent variable z:M thereby increasing the variance of the output forecast distribution.
Observed
EpIFNP
WILI
Daak havo edge griiaiiiiio
Week
2
20
Figure 9: Higher uncer- Ancer-Figure 10: SDCG Avg. edge tainty around peaks
probabilities

---

5 Conclusion
We introduced EPIFNP, a novel deep probabilistic sequence modeling method which generates well calibrated, explainable and accurate predictions. We demonstrated its superior performance in the problem of real-time influenza forecasting by significantly outperforming other non-trivial baselines (more than 2.5x in accuracy and upto 2.4x in calibration). Importantly. it was the only one capable of reliably handling unprecedented scenarios e.g. HINI and COVID19 seasons. We also showcased its explainability as it automatically retrieves the most relevant historical sequences matching its current week's predictions using the SDCG. All these highlight the usefulness of EPIFNP for the complex challenge of trustworthy epidemiological forecasting, which directly impacts public health policies and planning. However EPIENP can be affected by any systematic biases in data collection (for example, some regions might have poorer surveillance and reporting capabilities). There is limited potential for misuse our algorithms and/or data sources though the dataset is public/anonymized without any sensitive patient information.
We believe our work opens up many interesting future questions. Our setup can be easily extended to handle other diseases and our core technique can be adapted for other general sequence modeling problems. Further, we can extend EPIFNP to also use heterogeneous data from multiple sources. We can also explore incorporating domain knowledge of prior dependencies between different sources/features (e.g. geographically close regions are more likely to have similar disease trends).
Acknowledgments: We thank the anonymous reviewers for their useful comments. This work was supported in part by the NSF (Expeditions CCF-1918770, CAREER IIS-2028586, RAPID IIS-202786) Medium 1SS-195888, Medium IIS-2106961, CCF-2115126 Small III-2008334), CDC MInD program, ORNL, ONR MURI N00014-17-1-2601 faculty research awards from Facebook, Google and Amazon and funds/computing resources from Georgia Tech.
References
[1] Bijaya Adhikari, Xinfeng Xu, Naren Ramakrishnan, and B. Aditya Prakash. "EpiDeep: Ex- ploiting Embeddings for Epidemic Forecasting". In: Proceedings of the 25th ACM SIGKDD. July 2019, pp. 577-586.
[2] Dzmitry Bahdanau, Kyung Hyun Cho, and Yoshua Bengio. "Neural machine translation by jointly learning to align and translate". In: ICLR 2015. 2015.
[3] C. Blundell, Julien Cornebise, K. Kavukcuoglu, and Daan Wierstra. "Weight Uncertainty in Neural Network". In: ICML. 2015.
[4] Logan C. Brooks, David C. Farrow, Sangwon Hyun, Ryan J. Tibshirani, and Roni Rosen- feld. "Flexible Modeling of Epidemics with an Empirical Bayes Framework". In: PLOS Computational Biology 8 (Aug. 2015).
[5] Logan C. Brooks, David C. Farrow, Sangwon Hyun, Ryan J. Tibshirani, and Roni Rosenfeld. "Nonmechanistic forecasts of seasonal influenza with iterative nee-week-a distributions". In: PLOS Computational Biology 14.6 (June 2018).
[6] Prithwish Chakraborty, Pejman Khadivi, and Bryan Lewis ct.al. "Forecasting a Moving Target: Ensemble Models for ILI Case Count Predictions". In: Proceedings of the 2014 SIAM International Conference on Data Mining. Proceedings. Apr. 2014, pp. 262-270.
[7] Kyunghyun Cho and Bart van Merriënboer et.al. "Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation". In: EMNLP. 2014, pp. 1724-1734.
[8]
Facebook. The COVID-19 Symptom Data Challenge. 2020.
[9] Meire Fortunato, Charles Blundell, and Oriol Vinyals. "Bayesian recurrent neural networks".
In: arXiv preprint arXiv: 1n:::::0440 704.02798 (2017).
[10] Yarin Gal and Zoubin Ghahramani. "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning". In: ICML. 2016, pp. (050-1059)
[11] Marta Garnelo, Jonathan Sehwarz, Dan Rosenbaum, Fabio Viola, Danilo Rezende, S. M. Ali Eslami, and Yee Whye Tch. "Neural Processes". In: Advances in Neural Information Processing Systems 32. 2018
[12] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. "On calibration of modern neural networks". In: ICML. 2017, pp. 1321-1330.
10

---

[13] Herbert W. Hethcote. "The Mathematics of Infectious Diseases". en. In: SIAM Review 42.4 (Jan. 2000). pp. 199-653, ISSN: 0036-1445, 1095-7200. 10. 1137 0555//000000144407779777
URL: http: / /pubss siam org / doi / 10. 1137 5003611450037199777 (visited on 08/31/2020).
[14] Inga Holmdahl and Caroline Buckee. "Wrong but Useful - What Covid-19 Epidemiologic Models Can and Cannot Tell Us". In: NE/M 383.4 (2020), pp. 303-305.
[15] Eric Jang, Shixiang Gu, and Ben Poole. "Categorical Reparameterization with Gumbel- Softmax". In: ICLR. 2017.
[16] Michael A Johansson, Karyn M Apfeldorf, Scott Dobson, Jason Devita, et al. "An open challenge to advance probabilistic forecasting for dengue epidemies", In: Proceedings of the National Academy of Sciences 1 116.48 (2019), pp. 24268-24274
[17] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosen- baum, Oriol Vinyals. and Yee Whye Teh. "Attentive neural processes". In: arXiv preprint
arXiv:1901.00 91.0066616/1 (2019).
[18] Diederik P Kingma and Max Welling. "Auto-encoding variational bayes". In: ICLR (2013). [19] Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. "Cali- brated Language Model Fine-Tuning For In- And Out-Of-Distribttion Data". In: Proceedings of the Conference on Empirical Methods Natural Language Processing. 2020, pp. 1326- 1340. 10.:10..8653/v/ /00/00/22222//22 wwnee-mmamaa....
102
[20] Lingkai Kong, Haoming Jiang, Yuchen Zhuang, Jie Lyu, Tuo Zhao, and Chao Zhang. "Cali- brated Language Model Fine- Tuning for In- and Out-of-Distrro Disttribiitiittoonn Data". In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Nov. 2020, pp. 1326-1340. DOI: 10. 10653/863/vv1 v220000. emn smmlp-maing mamm.ainn.... 102
[21] Lingkai Kong, Jimeng Sun, and Chao Zhang. "SDE-Net: Equipping Deep Neural Networks with Uncertainty Estimates". In: ICML. 2020, pp. 5405-5415.
[22] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. "Accurate Uncertainties for Deep Learning Using Calibrated Regression". In: Proceedings of the 35th International Conference on Machine Learning. 2018, pp. 2796-2804.
[23] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. "Accurate uncertainties for deep learning using calibrated regression". In: International Conference on Machine Learning. PMLR. 2018, pp. 2796-2804.
[24] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. "Simple and scalable predictive uncertainty estimation using deep ensembles". In: NeurlPS 2017, pp. 6405- 6416.
[25] Chunyuan Li. Changyou Chen, David Carlson, and Lawrence Carin. "Preconditioneed stochas- tic gradient Langevin dynamics for deep neural networks". : In: Proceedings of the AAAI Conference on Artificial Intelligence. 2016.
[26] Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. "The Functional Neural Process". In: Advances in Neural Information Processing Systems 32. 2019, pp. 8746-8757. (Visited on 10/03/2020).
[27] Christos Louizos and Max Welling. "Multiplicative normalizing flows for variational bayesian (2017).
neural networks". In: arXiv preprint
[28] David JC MacKay. "A practical Bayesian framework for backpropagation networks". In: Neural computation (1992), pp. 448-472.
[29] Alexandru Niculescu-Mizil and Rich Caruana. "Predicting good probabilities with supervised learning". In: ICML 2005. 2005, pp. 625-632.
[30] Zhaozhi Qian, Ahmed M Alaa, and Mihaela van der Schaar. "When and How to Lift the Lockdown? Global COVID-19 Scenario Analysis and Policy Assessment using Compartmental Gaussian Processes". In: Advances in Neural Information Processing Systems 33 (2020).
[31] Shenghao Qin, Jiacheng Zhu, Jimmy Qin, Wenshuo Wang, and Ding Zhao. "Recurrent Atten- tive Neural Process for Sequential Data". In: arXiv:1910.09933 (2019).
[32] Evan L Ray, Krzysztof Sakrejda, Stephen A Lauer, Michael Johansson, and Nicholas G Reich. "Infectious discase prediction with kernel conditional density estimation". In: Statistics in medicine 36.30 (2017), pp. 4908 4929.
[33] Evan I. Ray, Nutcha Wattanachit, Jarad Niemi, et al. "Ensemble Forecasts of Coronavirus Discase 2019 (COVID-19) 19) in the US". In: MedRXiv (2020).
11

---

[34] Nicholas G. Reich, Logan C. Brooks, Spencer J. Fox, et al. "A collaborative multiyear, multimodel assessment seasonal influenza forecasting in the United States". In: PNAS (2019), pp. 3146-3154.
[35] Hippolyt Ritter, Aleksandar Botev, and David Barber. "A Scalable Laplace Approximation for Neural Networks". In: ICLR. 2018.
[36] Alexander Rodríguez, Nikhil Muralidhar, Bijaya Adhikari, Anika Tabassum, Naren Ramakr- ishnan, and B Aditya Prakash. "Steering a Historical Disease Forecasting Model Under Pandemic: Case of Flu and COVID-19". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 6. 2021, pp. 4855-4863.
[37] Alexander Rodríguez, Anika Tabassum, Jiaming Cui, Jiajia Xie, Javen Ho, Pulak Agarwal, Bijaya Adhikari, and B Aditya Prakash. "DeepCOVID: An Operational Deep Learning-driver driven Framework for Explainable Real-time COVID-19 Forecasting", " In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35, 17. 2021, pp. 15393-15400.
[38] Jeffrey Shaman and Alicia Karspeck. "Forecasting seasonal outbreaks of influenza". In: PNAS
109.50 (2012), pp. 20425-20430.
[39] Hao Song, Tom Diethe, Meelis Kull, and Peter Flach. "Distribution calibration for regression". In: International Conference on Machine Learning. PMLR. 2019, pp. 5897-5906.
[40] Farzaneh Tabataba, Prithwish Chakraborty, Naren Ramakrishnan, Srinivasan Venkatramanan, Jiangzhuo Chen, Bryan Lewis, and Madhav Marathe. "A framework for evaluating epidemic forecasts". In: BMC Inf. Diseases 17.1 (2017), 345.
[41] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. "Attention is All you Need". In: Advances in Neural Information Processing Systems (2017), pp. 5998-6008.
[42] Lijing Wang, Jiangzhuo Chen, and Madhav Marathe. "DEFSI: Deep learning based epidemic forecasting with synthetic information". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. 01. 2019, pp. 9607-9612.
[43] Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. "Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning". In: ICLR 2020. 2020.
[44] Christoph Zimmer and Reza Yaesoubi. "Influenza Forecasting Framework based on Gaussian Processes". In: International Conference on Machine Learning. 2020, pp. 11671-11679.
12


Reviews:
[{"review_id": "xwS_Y43BjP", "reviewer": "Reviewer_ENNP", "summary": "This paper a functional neural process model called EPIFNP for accurate and truth-worthy epidemic forecasting. EPIFNP is empirically shown with fast speed, accurate results and explainability.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I am not sure whether EPIFNP is specifically designed for epidemic forecasting. If so, what is the main difference between epi-forecasting and others and what is your main point towards this difference. If not, what if EPIFNP tested on other datasets, such as demand and weather forecasting?\n\nWhat is the computation efficiency? EPIFNP uses variational inference and a correlation graph, which seems computational expensive. \n\nFor the baselines, they are all specially designed for epidemic forecasting. Back to the aforementioned problem, if EPIFNP is not designed for epidemic forecasting, some more general baseline should be included here, such as: calibration methods, post-hoc methods using isotonic regression, deep ensemble. I have seen some of them are included in your baselines, but they looks specially designed for epidemic forecasting. What is their differences?\n* Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in neural information processing systems, pages 5574–5584, 2017. \n* BalajiLakshminarayanan,AlexanderPritzel,andCharlesBlundell.Simpleandscalablepredic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pages 6402–6413, 2017. \n* Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. International Conference on Machine Learning, pages 2796–2804, 2018. \n* Peng Cui, Wenbo Hu, Jun Zhu. Calibrated Reliable Regression using Maximum Mean Discrepancy. Advances in neural information processing systems, 2020. \n* Song H, Diethe T, Kull M, et al. Distribution calibration for regression[C]//International Conference on Machine Learning. PMLR, 2019: 5897-5906.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "YES", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "dcXFCJ-xIn", "reviewer": "Reviewer_mtym", "summary": "This paper proposes a method for probabilistic time series forecasting for univariate real-valued data (they just consider Influenza-Like Illness) using a semi-parametric neural network. The basic idea is to embed the prefix of observations for the current season into a fixed-sized latent vector using a GRU, and to embed all past full-year sequences into latent vectors, and then to compute a weighted average of the most similar embeddings from the past and to use this feature inside an MLP to predict the future, k-weeks ahead. They evaluate on the CDC dataset and report improved point estimates and calibration scores. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper feels like it's using a sledge hammer to open a walnut: the dataset they are working with is tiny (52 scalars per year, 17 years  / seasons of data), but the model they fit is a very complex semi-parametric neural net, combining GRUs, VAEs, self-attention, and functional neural processes (oh my!). The potential for overfitting is very high!\n\nIt is not clear exactly how the authors did their evaluation. It seems (from table 1) they evaluated predictions on years 2014-2019, and therefore presumably trained on 2003-2013, which is fine. But if the reference set is all the sequences in 2003-2013, then wouldn't a simpler baseline like K-nearest neighbors work as well? (e.g.,  I can match the prefix for the current year with the prefix for all past years, using something like dynamic time warping as my similarity metric). It seems like the model is learning to do this kind of similarity matching, but it would be more enlightening to show the behavior of the model on some synthetic data, where we can see where it works and where it fails.\n\nThe evaluation metrics seem a bit limited. RMSE, MAPE and Log score are standard.  But I am concerned the authors needed to invent a \"new metric\" for calibration. (The one they proposed seems standard to me.) There is a large body of work on assessing the predictive accuracy of probabilistic forecasts (see eg Bracher'21 for a recent method),  it would be good to include some of these metrics.\n\nThe use of a Gaussian observation model for a time series of positive counts seems inappropriate. A negative-binomial may be a better choice (if integer valued). Or some kind of quantile predictor.\n\nJ. Bracher, E. L. Ray, T. Gneiting, and N. G. Reich, “Evaluating epidemic forecasts in an interval format,” PLoS Comput. Biol., 2021 [Online]. Available: http://arxiv.org/abs/2005.12881\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation is that the method seems over-engineered to one specific dataset (ILI from CDC), and it is not clear how robust or general purpose it is. The focus on uncertainty estimation, however, is good.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "P7oOMbC96G9", "reviewer": "Reviewer_wNUL", "summary": "The paper develops a deep probabilistic model for forecastic stochastic time series with application to epidemic forecasting. The proposed algorithm encodes data points (weekly flu records) comprising a time series of arbitary size into a fixed-sized embedding by passing it through a GRU and binding the hidden states via self attention. Another round of self attention is used to relate whole time series (seasons). Next, it calculates a probabilistic cross attention graph, the edges of which match the whole time series of full seasons of past years to snippets from the current season. The data generating process where all building blocks come together is inferred by a standard variant of amortized variational Bayes. The paper conducts a rigorous set of experiments on a public flu statistics database to analyze multiple aspects of the proposed method and the benchmarks, such as prediction accuracy, uncertainty calibration, and explainability. The proposed method improves the state of the art with respect to all these aspects with a clear margin.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nSignificance of Novelty:\n\nThe studied application, epidemic forecasting, is obviously impactful. From the applied research standpoint, the methodological contribution of the proposed algorithm is sufficiently novel, as it is carefully engineered towards epidemics data. From the fundamental machine learning research standpoint, it is rather incremental. One can view the proposed method as an \"Attentive Neural Process\" [Rv1], with the cross attention part replaced by the random graph used in the \"Functional Neural Process\" design [22]. The inference algorithm is also a direct adaptation from [22].\n\nTechnical Flaws:\n\nThe paper is technically fully solid. Despite the proposed solution consists of a rather complex network of multiple compartments, they follow intelligible design choices.\n\nPresentation: \n\nThe presentation clarity of the paper is sufficient, though not outstanding. The content has a few circular dependencies. \n\ni) The variable \"v\" used in Eq 1 does not mean much, given the introduction in Line 143. It becomes much clearer in Eq 7, but at the cost of going back to Eq 1 once more. Its introduction in 143 could be detailed a little more to justify its necessity.\n\nii) Figure 2 is too complicated to be descriptive. I recommend extending the caption with more details about the essential properties and novel aspects of the building blocks of the proposed method.\n\niii) Figure 4 to 9 are way too small to be readable without zooming in electronically. Some further material (or a subset of these figures) could be moved to the supplement and the remaining can be presented at least double as big as their current size.\n\niv) [MINOR] Even though the paper cites the \"Recurrent Attentive Neural Process\" [26], perhaps it is better to cite also the seed \"Attentive Neural Process\" paper [Rv1]\n\nExperimental Evaluation:\n\nThe paper conducts a detailed and systematically designed series of experiments that investigates various important aspects of the proposed method, spanning prediction accuracy, uncertainty quantification, and explainability. The method stands out among all the baselines in all experiments by a clear margin.\n\nOverall:\n\nThis is solid work that introduces a novel time series forecasting method engineered tediously for epidemic data. The experimental evaluation is also rigorous and the results are very strong. My initial recommendation is an accept. \n\n[Rv1] Kim et al., Attentive Neural Processes, ICLR, 2019", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not provide an investigation of the limitations of the proposed algorithm. The only weakness mentioned in Section 5 is a feature of data, not the algorithm.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "The paper touches the ethical implications of the introduced method only in Line 408 at a rather superficial level. I do not agree with the statement that the potential misuse of their algorithm is too limited to raise awareness about it. The same algorithms can well also be used for prediction of other types of time series, such as crime or credit return forecasting, where the fairness of the model's prediction could be a matter of concern for its users."}}]

Author rebuttal:

Meta-review: There was significant interest in this paper, as the topic of epidemic forecasting is indeed one that has the attention of the community at the current time.  Reviewers had mixed opinions about this paper, but overall found that the most important concerns raised were indeed addressed by the author responses.  

In particular, reviewer mtym's concern that the model may be prone to overfitting was reasonably addressed by the empirical evaluation.  (And indeed, this would not be the first model in ML literature to show that over-parameterization does not necessarily lead to overfitting.)

Reviewer EENP's questions around whether the model is specifically designed for this application area were also well addressed in rebuttal, along with useful additional detail around computational efficiency and the potential impact of post-hoc calibration.

Overall, I can see no reason not accept this paper as a poster at this point, with the clear expectation of course that the authors will incorporate the feedback from reviewers into revising their paper, and include the additional results from the responses into the final paper.
Decision: Accept (Poster)