Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper a functional neural process model called EPIFNP for accurate and truth-worthy epidemic forecasting. EPIFNP is empirically shown with fast speed, accurate results and explainability.
main_review: I am not sure whether EPIFNP is specifically designed for epidemic forecasting. If so, what is the main difference between epi-forecasting and others and what is your main point towards this difference. If not, what if EPIFNP tested on other datasets, such as demand and weather forecasting?

What is the computation efficiency? EPIFNP uses variational inference and a correlation graph, which seems computational expensive. 

For the baselines, they are all specially designed for epidemic forecasting. Back to the aforementioned problem, if EPIFNP is not designed for epidemic forecasting, some more general baseline should be included here, such as: calibration methods, post-hoc methods using isotonic regression, deep ensemble. I have seen some of them are included in your baselines, but they looks specially designed for epidemic forecasting. What is their differences?
* Alex Kendall and Yarin Gal. What uncertainties do we need in Bayesian deep learning for computer vision? Advances in neural information processing systems, pages 5574–5584, 2017. 
* BalajiLakshminarayanan,AlexanderPritzel,andCharlesBlundell.Simpleandscalablepredic- tive uncertainty estimation using deep ensembles. Advances in neural information processing systems, pages 6402–6413, 2017. 
* Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. International Conference on Machine Learning, pages 2796–2804, 2018. 
* Peng Cui, Wenbo Hu, Jun Zhu. Calibrated Reliable Regression using Maximum Mean Discrepancy. Advances in neural information processing systems, 2020. 
* Song H, Diethe T, Kull M, et al. Distribution calibration for regression[C]//International Conference on Machine Learning. PMLR, 2019: 5897-5906.
limitations_and_societal_impact: YES
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a method for probabilistic time series forecasting for univariate real-valued data (they just consider Influenza-Like Illness) using a semi-parametric neural network. The basic idea is to embed the prefix of observations for the current season into a fixed-sized latent vector using a GRU, and to embed all past full-year sequences into latent vectors, and then to compute a weighted average of the most similar embeddings from the past and to use this feature inside an MLP to predict the future, k-weeks ahead. They evaluate on the CDC dataset and report improved point estimates and calibration scores. 
main_review: This paper feels like it's using a sledge hammer to open a walnut: the dataset they are working with is tiny (52 scalars per year, 17 years  / seasons of data), but the model they fit is a very complex semi-parametric neural net, combining GRUs, VAEs, self-attention, and functional neural processes (oh my!). The potential for overfitting is very high!

It is not clear exactly how the authors did their evaluation. It seems (from table 1) they evaluated predictions on years 2014-2019, and therefore presumably trained on 2003-2013, which is fine. But if the reference set is all the sequences in 2003-2013, then wouldn't a simpler baseline like K-nearest neighbors work as well? (e.g.,  I can match the prefix for the current year with the prefix for all past years, using something like dynamic time warping as my similarity metric). It seems like the model is learning to do this kind of similarity matching, but it would be more enlightening to show the behavior of the model on some synthetic data, where we can see where it works and where it fails.

The evaluation metrics seem a bit limited. RMSE, MAPE and Log score are standard.  But I am concerned the authors needed to invent a "new metric" for calibration. (The one they proposed seems standard to me.) There is a large body of work on assessing the predictive accuracy of probabilistic forecasts (see eg Bracher'21 for a recent method),  it would be good to include some of these metrics.

The use of a Gaussian observation model for a time series of positive counts seems inappropriate. A negative-binomial may be a better choice (if integer valued). Or some kind of quantile predictor.

J. Bracher, E. L. Ray, T. Gneiting, and N. G. Reich, “Evaluating epidemic forecasts in an interval format,” PLoS Comput. Biol., 2021 [Online]. Available: http://arxiv.org/abs/2005.12881

limitations_and_societal_impact: The main limitation is that the method seems over-engineered to one specific dataset (ILI from CDC), and it is not clear how robust or general purpose it is. The focus on uncertainty estimation, however, is good.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper develops a deep probabilistic model for forecastic stochastic time series with application to epidemic forecasting. The proposed algorithm encodes data points (weekly flu records) comprising a time series of arbitary size into a fixed-sized embedding by passing it through a GRU and binding the hidden states via self attention. Another round of self attention is used to relate whole time series (seasons). Next, it calculates a probabilistic cross attention graph, the edges of which match the whole time series of full seasons of past years to snippets from the current season. The data generating process where all building blocks come together is inferred by a standard variant of amortized variational Bayes. The paper conducts a rigorous set of experiments on a public flu statistics database to analyze multiple aspects of the proposed method and the benchmarks, such as prediction accuracy, uncertainty calibration, and explainability. The proposed method improves the state of the art with respect to all these aspects with a clear margin.
main_review: 
Significance of Novelty:

The studied application, epidemic forecasting, is obviously impactful. From the applied research standpoint, the methodological contribution of the proposed algorithm is sufficiently novel, as it is carefully engineered towards epidemics data. From the fundamental machine learning research standpoint, it is rather incremental. One can view the proposed method as an "Attentive Neural Process" [Rv1], with the cross attention part replaced by the random graph used in the "Functional Neural Process" design [22]. The inference algorithm is also a direct adaptation from [22].

Technical Flaws:

The paper is technically fully solid. Despite the proposed solution consists of a rather complex network of multiple compartments, they follow intelligible design choices.

Presentation: 

The presentation clarity of the paper is sufficient, though not outstanding. The content has a few circular dependencies. 

i) The variable "v" used in Eq 1 does not mean much, given the introduction in Line 143. It becomes much clearer in Eq 7, but at the cost of going back to Eq 1 once more. Its introduction in 143 could be detailed a little more to justify its necessity.

ii) Figure 2 is too complicated to be descriptive. I recommend extending the caption with more details about the essential properties and novel aspects of the building blocks of the proposed method.

iii) Figure 4 to 9 are way too small to be readable without zooming in electronically. Some further material (or a subset of these figures) could be moved to the supplement and the remaining can be presented at least double as big as their current size.

iv) [MINOR] Even though the paper cites the "Recurrent Attentive Neural Process" [26], perhaps it is better to cite also the seed "Attentive Neural Process" paper [Rv1]

Experimental Evaluation:

The paper conducts a detailed and systematically designed series of experiments that investigates various important aspects of the proposed method, spanning prediction accuracy, uncertainty quantification, and explainability. The method stands out among all the baselines in all experiments by a clear margin.

Overall:

This is solid work that introduces a novel time series forecasting method engineered tediously for epidemic data. The experimental evaluation is also rigorous and the results are very strong. My initial recommendation is an accept. 

[Rv1] Kim et al., Attentive Neural Processes, ICLR, 2019
limitations_and_societal_impact: The paper does not provide an investigation of the limitations of the proposed algorithm. The only weakness mentioned in Section 5 is a feature of data, not the algorithm.
review_title: null

Meta-review: There was significant interest in this paper, as the topic of epidemic forecasting is indeed one that has the attention of the community at the current time.  Reviewers had mixed opinions about this paper, but overall found that the most important concerns raised were indeed addressed by the author responses.  

In particular, reviewer mtym's concern that the model may be prone to overfitting was reasonably addressed by the empirical evaluation.  (And indeed, this would not be the first model in ML literature to show that over-parameterization does not necessarily lead to overfitting.)

Reviewer EENP's questions around whether the model is specifically designed for this application area were also well addressed in rebuttal, along with useful additional detail around computational efficiency and the potential impact of post-hoc calibration.

Overall, I can see no reason not accept this paper as a poster at this point, with the clear expectation of course that the authors will incorporate the feedback from reviewers into revising their paper, and include the additional results from the responses into the final paper.
Decision: Accept (Poster)